COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation
Authors: Jongmin Lee
Cosmin Paduraru
Daniel J Mankowitz
Nicolas Heess
Doina Precup
Kee-Eung Kim
Arthur Guez
Venue: ICLR 2022 Spotlight (2022)
Keywords: Offline Reinforcement Learning
Offline Constrained Reinforcement Learning
Stationary Distribution Correction Estimation

Abstract: We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset. This problem setting is appealing in many real-world scenarios, where direct interaction with the environment is costly or risky, and where the resulting policy should comply with safety constraints. However, it is challenging to compute a policy that guarantees satisfying the cost constraints in the offline RL setting, since the off-policy evaluation inherently has an estimation error. In this paper, we present an offline constrained RL algorithm that optimizes the policy in the space of the stationary distribution. Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for actual constraint satisfaction. Experimental results show that COptiDICE attains better policies in terms of constraint satisfaction and return-maximization, outperforming baseline algorithms.

Published as a conference paper at ICLR 2022
COPTIDICE: OFFLINE CONSTRAINED REINFORCE- MENT LEARNING VIA STATIONARY DISTRIBUTION CORRECTION ESTIMATION
Jongmin Lee'; Cosmin Paduraru', Daniel J. Mankowitzz, Nicolas Heess", Doina Precup", Kee-Eung Kim', Arthur Guez² KAIST, *DeepMind
ABSTRACT
We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset. This problem setting appealing in many real-world scenarios, where direct interac- tion with the environment is costly or risky, and where the resulting policy should comply with safety constraints. However, it is challenging to compute a policy that guarantees satisfying the cost constraints in the offline RL setting, since the off- policy evaluation inherently has an estimation error. In this paper, we present an offline constrained RL algorithm that optimizes the policy in the space of the sta- tionary distribution. Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while con- straining the cost upper bound, with the goal of yielding cost-conservative policy for actual constraint satisfaction. Experimental results show that COptiDICE at- tains better policies in terms of constraint satisfaction and return-maximization outperforming baseline algorithms.
INTRODUCTION
Reinforcement learning (RL) has shown great promise in wide range of domains, such com- plex games (Mnih et al.) 2013 Silver et al. 2017) and robotic control (Lillicrap et al., 2016 Haarnoja al. (008)). However, the need to interact with the environment during learning hinders its widespread application to many real-world problems for which executing exploratory behavior in the environment costly or dangerous. Offline RL (also known batch RL) (Lange al.. 2012 Levine et al. 2020) algorithms sidestep this problem and perform policy optimization solely from set of pre-collected data. The use of existing (offline) data can make offline reinforcement learning applicable to real world systems and and has led to sharp increase in interest in this paradigm.
Recent works on offline RL, however, mostly assume that the environment modeled as a Markov decision process (MDP), and standard offline RL algorithms focus on maximiiaiaiii only (Fujimoto al. 2019 Wu et al. 20199 Kumar et al. 2019 Siegel et al. 2020 Wang et al. 2020) Kumar et al.) 2020). In contrast, in real-world domains it is common that the behavior of the agent is subject to additional constraints beyond the reward. Consider, for example, autonomous driving or industrial robot in factory. Some behaviors may damage the agent itself or its surroundings, and safety constraints should thus be considered as part of the objective of a suitable reinforce- ment learning system. One of the ways mathematically characterize a constrained RL problem is through the formalism of constrained Markov Decision Processes (CMDP) (Altman] 1999). In CMDPs taking an action incurs cost as well as a reward, and the goal is to maximize the expected long-term reward while satisfying a bound on the expected long-term cost. In this work, we aim to solve the constrained decision making problem in the offtine RL setting, to enable deployment in various safety-critical domains where direct learning interactions are infeasible.
Offline constrained RL inherits the difficulties of offline unconstrained RL, while introducing ad- ditional challenges. First, since the target policy being optimized deviates from the cola-coole
Work done during an intersship DeepMind.

---

Published as a conference paper at ICLR 2022
policy with no further data collection, distribution shift becomes the central difficulty. To mitigate the distributional shift, existing offline RL methods frequently adopt the pessimism principle: either by explicit policy (and critic) regularization that penalizes deviation from the data-collection policy (Jaques et al. 2019; Wu et al.
2019, Kumar et al. 2019; (iieel et al. 2020, Wang et al. 2020 Lee
2021) by reward penalty the uncertain state-action regions (Kumar et al. 2020 Kidambi et al. 2020 Yu et al. 2020). Second, in offline constrained RL, the computed
al. 2020 Kostrikov et al.
policy should satisfy the given cost constraints when it is deployed to the real environment. Unfor- tunately off- -fflccl policy evaluation inherently has estimation errors, and it is therefore difficult to ensure that a policy estimated from a finite dataset will satisfy the constraint when executed in the en- vironment. In addition, constrained policy optimization usually involves an additional optimization for the Lagrange multiplier associated with the cost constraints. Actor-critie-basee constrained RL. algorithms thus have to solve triple (i.e. critic, actor, Lagrange multiplier) intertwined optimization problems [Borkar 2005 Tessler et al. 2019). which can be very unstable in practice.
In this paper, we present an offline constrained RL algorithm that optimizes the state-action sta- tionary distribution directly, rather than the Q-function or the policy. We show that such treatment obviates the need for multiple estimators for the value and the policy, yielding single optimiza- tion objective that is practically solvable. Still, naively constraining the cost value may result in severe constraint violation in the real environment, we demonstrate empirically. We thus propose a method to constrain the upper bound of the cost value, aiming to compute a policy more robust in constraint violation, where the upper bound is computed in a way motivated by a recent advance in off-policy confidence interval estimation (Dai et al. (020).. Our algorithm. Offline Constrained Policy Optimization via stationary DIstribution Correction Estimation (COptiDICE), estimates the stationary distribution corrections of the optimal policy that maximizes rewards while constraining the cost upper bound, with the goal of yielding cost-conservative policy for better actual constraint satisfaction. COptiDICE computes the upper bound of the cost efficiently by solving an additional minimization problem. Experimental results show that COPtiDICE attains a better policy in terms outperforming several baseline algorithms.
of constraint satisfaction and
BACKGROUND
Constrained Markov Decision Process (CMDP) (Altman 1999) an extension of an MDP, for- mally defined by tuple M = (8, A, T, R, C = {Ck JC.K...K.
where S is the set { (.. po.7),
of states, A is the set of actions, T : 5 A + (SS) is a transition probability, : S x A + I is the reward function, C2 S xSxA A - R is the k-th cost function with its corresponding threshold E E R. Po E (SS) is the initial state distribution, and 7 E (0, 1] is the discount factor. A policy S + (AA is a mapping from state to distribution over actions. We will express solving CMDPs in terms of stationary distribution. For a given policy π, the stationary distribution d" is defined by:
(1
7)
Σ
7
Pr(st = 8, a- = a)
if <<< 1,
ds
=
(I)
lim
1
Σ Pr(s:= = =, = a) 7 = 1,
T-000
T+1
==
where so ~ Po. policy as VR(TT) == [d [R(s,a) E IR and Vc(T) := ((eaa)ar((Cie (C(s, E R Constrained
(Tt)), 8141 T(s az) for all timesteps t 2 0. We define the value of the
RL aims to learn an optimal policy that maximizes the reward while bounding the costs up to the thresholds by interactions with the environment:
max VR(T) s.t. Vo, (T) S é Vk
Lagrangian relaxation is typically employed to solve Eq. 2 leading to the unconstrained problem:
min max VR(T) A T (Vo(T)
X20
where À E K is the Lagrange multiplier and ê E K is the vector vector-valued cost threshold. The inner maximization in Eq. (3) corresponds to computing an optimal policy that maximizes scalarized rewards R(s, a) - X ((, a). due to the linearity of the value function with respect to the reward function. The outer minimization corresponds to balancing the cost penalty in the scalarized reward
2

---

Published as a conference paper at ICLR 2022
function: if the current policy is violating the k-th cost constraint, 2 increases so that the cost is penalized more in the scalarized reward, and vice versa.
In the offline RL setting, online interaction with the environment is not allowed, and the policy is optimized using the fixed offline dataset = {(5p, 8, a, r c, } collected with one or more (unknown) data-collection policies. The empirical distribution
the dataset denoted as dD, and we will abuse the notation dD for ~ d,, (s, a) dD, (s, "9,,,'' dD, We abuse (so, 8, a, s') d for su ~ Po: (s, a, a')") ~ dD. We denote the space of data samples (su. 8, G, s') as X.
A naive way to solve (3 in offline manner is to adopt an actor-critt based offline RL algorithm for max, C while jointly optimizing A. However, the intertwined training procedure of off- policy actor-critic algorithms often suffers from instability due the compounding error incurred by bootstrapping out-of distiiiiiitttion action values in an offline RL setting (Kumar et al. 2019). The instability would be exacerbated when the nested optimization for À is added.
3 OFFLINE CONSTRAINED RL VIA STATIONARY DISTRIBUTION CORRECTION ESTIMATION
In this section, we present our offline constrained RL. algorithm, Constrained Policy Optimization via stationary DIstribution Correction Estimation (COptiDICE). The derivation of our algorithm starts by augmenting the standard linear program for CMDP (Altman 1999) with an additional f- f-divergence regularization: m (aa) aDF(dlldd)
(4)
s.t. B ((e,))a((C(sss c dd(s'a''= - p)po + E d(ss.))((s\ss
4k: = 1, K
(5)
A9 (6)
8,0
d(s,a) 20
ss, a
where Dy(dlld)) = E (s [f( (x.a)) )] is the f-divergence between the distribution d and the dataset distribution dD, and a > is the hyperparameter that controls the degree of pessimism, i.e. how much penalize the distribution shift. commonly adopted principle for offline RL (Nachum et al. 2019b; Kidambi et al. 2020, Yu et al. 2020 Lee et al. 2021). We assume that d' > and f is strictly convex and continuously differentiable function with f(1) O. Note that when a = 0, the optimization (4 reduces to the standard linear program for CMDPs. The Bellman-flow constraints 6 7) ensure that d the stationary distribution of some policy, where d(s, a) can be interpreted as a normalized discounted occupancy measure of (s. a). Thus, we seek the stationary distribution of an optimal policy that maximizes the reward value 4 while bounding the cost values 5). Once the optimal solution d' has been estimated, its corresponding optimal policy is obtained by T (a|s)= Σ. (s.a7)
d(e.a)
d' d'(s.))
Now, consider the Lagrangian for the constrained optimization problem
K
min max E (s,,) a)) aDD(dd)d)) -E k=1
d (E (na))a 2x)
x20 d20
Ev(d) [ d(s'' - ))))ss)) y d a(sss))((s(sssaa)
(8)
s.a
where X E R. is the Lagrange multiplier for the cost constraints (5) and v(s) R is the Lagrange multiplier for the Bellman flow constraints (6). Solving (8) in its current form requires evaluation of T(s'|s, a) for (s,a) ~ d, which is not accessible in the offline RL setting. To make the optimization tractable, we rearrange the terms so that the direct dependence on dl is eliminated, while introducing new optimization variables w that represent stationary distribution corrections:
min max E (s.a)ad [R(s,a) a) > CC(ssa) + 7w((s') - v[s]] al((s,))do [ [ A20 d20 TT,a))
+ (1
B [v(0 T
min max E
(s.a)<d (w(s, af((w))))) + (1-7)1 7) (u(s0)] A (9)
A20,z #20
3

---

Published as a conference paper at ICLR 2022
where (s, == : - regarding y as a state value function, and w(s,a) a) :- (s.a] is the stationary distribution correction.
(T)((s ( tB [v()] v(s) is the advantage function by d(am)]
Every term in Eq. (9) can be estimated from samples in the offline dataset D):
min max E v,220 (co.s,a.s)) [6(s, a)xx, (8, a, af( a)) + (1 7))6sss) + + (10)
is the advantage estimate using single sample. As a consequence, (10) can be optimized in a fully offinn manner. Moreover, exploiting the strict convexity of
where (s, as, R(s, - (T) +yu(s)
further derive closed-form solution for the inner maximization in
D as follows. All the proofs can be found in Appendix B
Proposition For any v and the closed-form solution of the inner maximization proble ( is given by: W - (dexe(s,a)) where x4 = max(0.
(11)
Finally, by plugging the closed-form solution [1 into (9. we obtain the following convex mini- mization problem:
min T(A,V) = (s,a)aaa (8 a)exo(s, (x,,)( + (1 --) [(0))] + A Aā A20,0
(12)
To sum up, by operating in the space of stationary distributions, constrained (offline) RL can in principle be solved by solving a single convex minimization (12) problem. This is in contrast to existing constrained RI, algorithms that manipulate both Q-function and policy, and thus require solving triple optimization problems for the actor, the critic, and the cost Lagrange multiplier with three different objective functions. Note also that when À is fixed and treated as constant, (12) reduces to OptiDICE (Lee et al., 2021) for unconstrained RL with the scalarized rewards R(s, C(s, a), without considering the cost constraints. order to meet the constraints, À should also be optimized, and the procedure of
be understood as joint optimization of:
arg min L(X, v))
OOptiDICE for R CT)
(13)
A arg min XT (e- (a.a)ad (s, A20
(Cost Lagrange multiplier)
<Vo(()
Once the optimal solution of [12,, (*","",, is computed, w (s, 8" (a.0) is also derived by
(s.a)
which is the stationary distribution correction between the stationary distribution of the optimal policy for the CMDP and the dataset distribution.
3.1 COST-CONSERVATIVE CONSTRAINED POLICY OPTIMIZATION
Our first method based on [2) relies on off-policy evaluation (OPE) using DICE to ensure cost constraint satisfaction, i.e. "" (s, a)] ()) (C(s, a)] < é. However, as we will see later, constraining the cost value estimate naively can result in constraint violation when deployed the real environment. This is due the fact that off-policy value estimate based on a finite dataset inevitably has estimation error. Reward estimation error may be tolerated as long as the value estimates are useful as policy improvement signals: it may be sufficient to maintain the relative order of action values, while the absolute values matter less. For the cost value constraint, we instead rely on the estimated value directly.
To make policy robust against cost constraint violation in offline setting, we consider the con- strained policy optimization scheme that exploits the upper bound of the cost value estimate:
max VT(TT) s.t. UpperBounddVer (T)) S & Vk
(14)
Then, the key question is how to estimate the upper bound of the policy value. One natural way exploit bootstrap confidence interval (Efron & Tibshirani 1993. Hanna et al. 2017.. We can
4

---

Published as a conference paper at ICLR 2022
construct bootstrap datasets D, by resampling from D and run OPE algorithm each D, which yields population statistics for confidence interval estimation (Vc(=):) However, this procedure computationally very expensive since it requires solving m OPE tasks. Instead, we take a di different approach in a more computationally efficient way motivated by CoinDICE (Dai et al.] 20200)
recently proposed DICE-family algorithm for off-policy confidence interval estimation. Specifically,
given that our method estimates the stationary distribution corrections w(s, a) ~ [ of the
d=(s.o)
target policy v, we consider the following optimization problem for each cost function Cs:
Css(sa))))
(15)
BEA(X)
s.t. DKL (p((s su. s, a. s (Sa. s,), S 5E Ess' a a')w(s'' .d) (1 7)po(s) + +7 E) o
(16)
As'
(17)
where (80 8,0, 8) = (00))))(( (0))))(ss()(()))))) (s'|ss (s. a) is the distribution over data samples (sg, 8, a, E X which lies in the simplex A(X), and > 0 is the hyperparameter. In essence, want adversari- ally optimize the distribution over data samples p so that it overestimates the cost value by (15). At the same time, we enforce that the distribution ý should not be perturbed too much from the empirical data distribution dd by the KL constraint (16). Lastly, the perturbation of distribution should be done in way that maintains compatibility with the Bellman flow constraint. The constraint
(17) is anal-
ogous to the Bellman flow constraint (6) by noting that p(s, a)w((, a) f((s, a)
d" (s.a)
R (s, a).
(P
In this optimization, when O, the optimal solution is simply given by f dD, which yields the vanilla OPE result via DICE, i.e. E().))=a") [uo(s, a)C&(s,,))))) a)]. For e > 0, the cost value will be overestimated more as increases. Through derivation similar that for obtaining ([2), we can simplify the constrained optimization into a single unconstrained minimization problem as follows. We denote (su, 8, a, s') as for notational brevity.
Proposition 2. The constrained optimization problem 17 can be reduced to solving the follow- ing unconstrained minimization problem:
min (e(T, xi w) T log E Eg [exp (7 (w(s, ()(L(ss) + +x((s)) x(5)) + (1- (1--)x(30)))) + Te 20,x
(18)
where T E R.+ corresponds to the Lagrange multiplier for the constraint (T6), and x(s) E R
corresponds to the Lagrange multiplier for the constraint (17). In other words, min, 20,x ((T,)
Ey (uss, a)Cx& (s, a)) where 8- is the optimal perturbed distribution of the problem 15 Also, for the optimal solution (r" (x),,, f is given by:
pº (x) x d exp [ (w[s, (((ss,a)(C(sssoo (x(x + 7xxx
(19)
w (unnormalized weight for (80, 0, a,),
Note that every term in 18 can be estimated only using samples of the offline dataset D, thus it can be optimized in a fully offline manner. This procedure can be understood as computing the weights for each sample while adopting reweighting in the based OPE, i.c. Uppeeroond((Vo(T)) =
Exwnd [ōo (z) w(, (0,, as,(s, a]] where w" (z) (normalized w (x) of (19). The weights are given non non-uniformly so that the cost value is overestimated to the extent controlled by c.
Remark. CoinDICE DDa et al. 2020) solves the similar optimization problem to estimate upper cost value of the largei policy π as Jollows:
max min T log E. x #20 +20,0
exp (H(w(s))(CCuss + v(8 + (1 ())( 80, an))
+
ag-xxtan)
a'ox((s))
(20)
It is proven that 200 provides asymptotic (1 (1-00 caaaaappeeeccfffddee-eieeteeeeee of the policy value if So where &a is the (1 -l-aantuee of the x2 distribbutio with I degree of freedom (Dai al. 2020 Compared to our optimization problem (18). CoinDICE's (20) involves the additional outer ds.o),
maximization, which is for estimating w(s. a) ple.a) the stationary distribution corrections of the target policy n. In contrast, we consider the case when w is given, thus solving the inner minimization alone is enough.
5

---

Published as a conference paper at ICLR 2022
Finally, ready to present final algorithm COptiDICE, offline constrained RL algorithm that maximizes rewards while bounding the upper cost value, with the goal of computing a policy robust against cost violation. COptiDICE addresses (4) by solving the following joint optimization.
f arg min V(),V)
(OptiDICE for R - CT
(21)
X,X arg min =k=1 l (TRs Xxi
(Upper cost value estimation)
-20,x
+ arg min A T ((T,, (x
(Cost Lagrange multiplier)
A20
sUpper (Vc(x))
Compared to 113 the additional minimization for (T, x) is introduced estimate the upper bound of cost value.
3.2 POLICY EXTRACTION
Our algorithm estimates the stationary distribution corrections of the optimal policy, rather than directly obtaining the policy itself. Since the stationary distribution corrections do not provide
direct way to sample an action, we need to extract the optimal policy x= from w"(s = d= (s.a)
(s.
in order select actions when deployed. For finite CMDPs, it is straightforward to obtain "
d" (s.a) by x" (als) E. (x.a"] Eg (P((,)) (s.a] However, the same method cannot directly be
d (s.a))u (s.a)
applied to continuous CMDPs due the intractability of computing the normalization constant. For continuous CMDPs, we instead extract the policy using importance-weiighed behavioral cloning:
max (e.,))==-' [log (als)] ((sa))a [w" (s, a) log (Tals))
(22)
which maximizes the log- logeliicl of actions to be selected by the optimal policy x².
3.3 PRACTICAL ALGORITHM WITH FUNCTION APPROXIMATION
For continuous large CMDPs, we represent our optimization variables using neural networks. The Lagrange multipliers v and x are networks parameterized by 0 and & respectively: ve : + IR feedforward neural network that takes state as an input and outputs a scalar value, and X RK is defined similarly. the policy Ty, we use mixture density network (Bishop) 1994) where the parameters of a Gaussian mixture model are output by the neural network. The parameters of the Vy network are trained by minimizing the loss:
RK and IR are represented by K-dimensional vectors. For
min (0) EErnd [ a, -) CTs(s, + yuo(s') - -v(ss)
(23)
where wkss a.) (f') (2 af (1 ()00))(()0) + +
XT + (v0(s') vo(s))) While J can be
biased estimate L(), v) in 12 in general due to (f') (EI-)) * E((f') -(), we can show that JA is an upper bound of L(v, A) (i.e. we minimize the upper bound), and J L(v, A) holds if
transition dynamics are deterministic (c.g. Mujoco control tasks) (Lee et al. 202T). The parameters of the X network and can be trained by:
log E0 20,0 k=
min
exp
-xxo166)-xxx(9))
(24)
+1 (1- (U--))x.(soo))) THE
This involves a logarithm outside of the expectation, which implies that mini-batch approximations would introduce bias. Still, adopt the simple mini-batch approximation for computational efficiency, with a moderately large batch size (e.g. 1024), which worked well in practice. The empirical form of the loss use is given by:
min Jr.x (T, =Bnatch(D)-D) (C)(s, "Xo Xax(s)) (s0)) ThE
T log E, Exwhhtch(D) exp ( (10s,
(25)
+20,0

---

Published as a conference paper at ICLR 2022
Lastly, À and the policy parameter 1 are optimized by:
min JA(A) ( - (T,
(26)
A
min (+) Ez [ a, s') log (a]s) *
(27)
The complete pseudo-code is described in Appendix C
where every parameter is optimized jointly.
4 EXPERIMENTS
4.1 TABULAR CMDPs (RANDOMLY GENERATED CMDPs)
We first probe how COptiDICE can improve the reward performance beyond the data-collection policy while satisfying the given cost constraint via repeated experiments. We follow proto- col similar to the random MDP experiment in (Laroche al.) 2019: Lee et al.] 202I) but with cost as an additional consideration. We conduct repeated experiments for 10K runs, and for each run, a CMDP M is generated randomly with the cost threshold policy TD. constraint-satisfyingg policy (i.e. Vc(TD) = 0.09)
0.1. We test with
two types of data-collectio and a constraint violating policy (i.e. Vc(Tp) 0.11). Then, varying number of trajecto- ries N E (10, 20, 50, 100, 200, 500, 1000, 2000} are collected from the sampled CMDP using the constructed data-collection policy mp, which constitutes the offline dataset D. Finally, the of- fline dataset D is given to each offline constrained RL algorithm, and its reward and cost perfor- mance is evaluated. For the reward, we evaluate the normalized performance of the policy т by (V<()) (-o0, 1] to see the performance improvement of over TD intuitively, where π² is the optimal policy of the underlying CMDP. More details can be found in Appendix B
Va(=*)-VA(F))
Offline constrained RL has been mostly unexplored, thus lacks published baseline algorithms. We consider the following three baselines. First, BC denotes the simple behavior cloning algorithm to see whether the proposed method is just remembering the dataset. Second, Baseline denotes the algorithm that constructs a maximum-likelihood estimation (MLE) CMDP AM using 1) and then solves the MLE CMDP using a tabular CMDP solver (L. solver) (Altman) 1999). Third, C-SP IBB is the variant of SPIBB (an offline RL method for tabular MDPs) (Laroche et al. 2019). where we modified SPIBB deal with the cost constraint by Lagrange relaxation (Appendix
Figure Ta presents the result when the data-collection policy is constraint-satiifyingg. The perfor- mance of BC approaches the performance of the TD as the size of the dataset increases, as expected. When the size of the offline dataset is very small, Baseline severely violates the cost constraint when its computed policy is deployed fails to improve the reward performance over x (reward red curve in Figure [p). This result is ex- pected since Baseline overfits to the MLE CMDP, exploiting the highly uncertain ttate-actions, This can cause significant gap between the stationary distribution of the optimized policy computed in M and the one computed in M, leading to failure in both reward performance improvement and cost constraint satisfaction. To prevent such distributional shift, offline RL algorithms commonly adopt the pessimism principle, encouraging staying close the data support. We observe that such pessimism principle is also effective in offline constrained RI: both C-SPIBB (orange) and the naive version of COptiDICE (green) show consistent policy improvement over the data-collection policy while showing better constraint satisfaction. Still, the pessimism principle alone is not sufficient ensure constraint satisfaction, raising the need for additional treatment for cost-conservativeness. Finally, our COptiDICE (blue) shows much stricter cost satisfaction than other baseline algorithms while outperforming Baseline and C-SPIBB in terms of reward performance.
the real environment (cost red curve in Figure Th), and even
Figure μP presents the result when the data-collection policy is munstrint-viollatngg where all the baseline algorithms exhibit severe cost violation. This result shows that if the agent encouraged to stay close to the constraint-violating policy, it may negatively affect the constraint satisfaction, although the pessimism principle was beneficial in terms of reward maximization. In this situation, only COP1DICE (blue) could meet the given constraint in general, which demonstrates the effective- ness of our proposed method of constraining the upper bound of cost value. Although COptiDICE (blue) shows reward performance degradation when the size of dataset is very small, this is natural in that it sacrifices the reward performance to lower the cost value to meet the constraint conservatively, and it still outperforms the baseline in this low-data regime.
7

---

Published as a conference paper at ICLR 2022
Cost threshold 8
Data-collection policy
Oplimal policy COP (by (12))
BC COpiIDICE
Baseline
C-SPIBB
1.0 Reward
Cost
Reward
Cost
0.12
0.10
0.09
0.10
10
10" 10
10
10"
10"
10"
10"
10
10"
10"
number of trajectories
number trajectories
number trajectories
number of trajectories
when the data-collection policy cost-satisfying
(b) when the data-collection policy is cost-violating
Figure 1: Result of tabular COptiDICE and baseline algorithms in random tabular CMDPs for the varying number of trajectories and two types of data-collection policies. Plots of (a) correspond to the case when the data-collection policy is constraint-satisfynng, while the data-collection policy is constraint-violatnn in the last two plots of (b), The mean of normalized reward performance and the mean of cost value are reported for 10K runs, where the error bar denotes the standard error.
4.2 CONTINUOUS CONTROL TASKS (RWRL SUITE)
We also evaluate COptiDICE on domains from the Real-Works RL (RWRL) suite Amulac-Amold jet al., 2020), where the safety constraints are employed. The cost of given if the task specifcc safety constraint is violated at each time step, and the goal is to compute policy that maximizes re- wards while bounding the average cost up to è. Per-domain safety constraints and the cost constraint thresholds are given in Appendix D Due to lack of algorithm that addresses offline constrained RL in continuous action space, we compare COptiDICE with simple baseline algorithms, i.e. BC: A simple behavior-cloning agent, CRR (Wang et al., 2020):
of-te-of-the-at (unconstrained) offline RL algorithm, and C-CRR: the constrained variant of CRR with Lagrangian relaxation where a cost critic and a Lagrange multiplier for the cost constraint are introduced (Appendix F
Since there is no standard dataset for offline constrained RL, we collected data using online con- strained RL agents (C-DMPO; the constrained variant of Distributional MPO) (Abdolmaleki al., 2018 Mankowitz et al., 2021). We trained the online C-DMPO with various cost thresholds è and saved checkpoints at regular intervals, which constitutes the pool of policy checkpoints. Then, we generated datasets, where each of them consists of 3 x 100% constraint-satiste the rest constraint-vollating trajectories. The trajectories were sampled by policies in the pool of policy checkpoints. Since there is a trade-off between reward and cost in general, the dataset is a mixture of low-reward-low-coss trajectories and high-reward-highee high--osst trajectories.
trajectories and
Figure 2 presents our results in RWRL tasks, where the dataset contains mostly constraint constraint-satisfiing trajectories (3 0.8), with some cost-violating trajectories. This type of dataset representative of many practical scenarios: the data-collecting agent behaves safely in most cases, but sometimes exhibit exploratory behaviors which can be leveraged for potential performance improvement. Due to the characteristics of these datasets, BC (orange) generally yields monstraint-satiffyiii policy, but its reward performance is also very low. CRR (red) significantly improves reward performance over satisfied. C-CRR (brown) incurs relatively lower
BC, but it does not ensure that the constraints cost value than BC in Walker, Quadruped, and Humanoid, but its reward performance is clearly worse than BC. The naive COptiDICE algorithm (green) takes the cost constraint into account but nevertheless frequently violates the constraint due to limitations of OPE. Finally, COptiDICE (blue) computes policy that more robust to cost violation than other algorithms, while significantly outperforming BC in terms of reward performance. We observe that the hyperparameter e in (6) controls the degree of cost-conservativeness as expected: larger values of lead to overestimates of the cost value, yielding a more conservative policy.
To study the dependence on the characteristics of the dataset we experiment with different values of B. Figure 3 show the result for / = 0.8 wow-reeard-low-oo and Figure 3 for 3 = 0.2 (high-reward-high-ee more high-reward-high--co simple modification of the unconstrained CRR to the constrained one (brown) was not effective enough to satisfy the constraint. Our vanilla offline constrained RL algorithm (green), encouraged to stay close to the data, also suffers from severe constraint violation when most of the trajectories are given as the constraint-violating ones, which is similar to the result of Figure [t. Finally,
data), Figure 3 for / = 0.5, data). These results show the expected trend:
data leads to a joint increase in rewards and costs of all agents. A
8

---

Published as a conference paper at ICLR 2022
Cost threshold
BC
CRR
C-CRR
COpEDICE (by (12))
COpt (- = 0.00)
COPÍDDCE (+ 0.05)
COPiDICE (+ 0.1)
Reward
0.50
Average coet
Reward
Average cost
500
0.25
500
0
0.00
0
Heeraions
Itarations
Iterations
Reralions
1a5
(a) Carpole Swingup
(b) Walker
1000
Reward
Average cost
400
Reward
0.50
Average cost
300
0.25
500
0.0
200
0.00
Ierations
Itarations
Itorations
karations
1e8
(c) Quadruped
(d) Humanold
Figure 2: Result of RWRL control tasks. For each task, we report the reward return and the average cost. The results are averaged over 5 runs, and the shaded area represents the standard error.
Cost threshold è
BC
CRR
C-CRR
COPIIDICE (by (12))
COP (c 00.01)
COPEDICE (c- 0.05)
COPIDDIE 0.1)
1000
Reward
0.3
Average cost
1000
Reward
0.3
Average cost
1000
Rewerd
0.3
Average cost
0.2
0.2
0.2
500
0.1
0
0.0
0.0
0
0.0
10
10
10
5
10
10
iterations 1e5
iterations
kerations 165
iterations 1e5
iterations 1e5
Herations 1e5
(8) Walker (3== 0.8)
Walker (355
(c) Walker (3=0.2)
Figure 3: Result RWRL walker using three dataset configurations with different levels of con- straint satisfaction 3: for (a) the data is obtained with 3 = 0.8 (low-reward-low-coss data), (b)
with 3 = 0.5, and (c) with B 0.2 hiighreward-hiig--eee data).
COptiDICE (blue) demonstrates more robust behavior to avoid constraint violations across dataset configurations, highlighting the effectiveness of our method constraining the cost upper bound.
5
DISCUSSION AND CONCLUSION
The notion of safety in RL has been captured in various forms such as risk- (Chow et al.
2015 Urpi et al. 2021 Yang et al. 202T). Robust MDP <Tyengar 2005 Tamar et al.) 2014). and Constrained MDP (Altman 1999), among which we focus on CMDP as it provides a natural for- malism encode safety specifications (Ray et al. 2019). Most of the existing constrained RL algorithms (Achiam et al. 2017: Tessler et al.) 2019 Satija et al., 2020) are on-policy algorithms, which cannot be applied to the offline setting directly. 2019) that also aims to solve constrained RL in an offline setting, though its method is limited to discrete action spaces and relies on solving an MDP completely as an inner optimization, which is inefficient. It also relies the vanilla OPE estimate of the policy cost, which could result in severe constraint violation when deployed. Lastly, in a work done concurrently to ours, Xu et al. (202T) also exploits overestimated cost value to deal with the cost constraint, but their approach relies on an actor-critic algorithm, while ours relies on stationary distribution optimization.
recent exception is the work by Le et al.)
In this work, we have presented a DiCE-based offline constrained RL algorithm, COPtiDICE. DICE- family algorithms have been proposed for off-policy evaluation (Nachum et al. 2019a] Zhang et al., 2020a by Yang et al. 2020b: Dai et al., 2020). imitation learning (Kostrikov et al.; 2019). offline policy selection (Yang et al. 2020a). and RL (Nachum et al. (2096]] Lee et al. (021). but none of them for constrained RI. Our first contribution was derivation that constrained offfine RL can be tackled by solving single minimization problem. We demonstrated that such approach, in its sim- plest form, suffers from constraint violation in practice. To mitigate the issue, COPtDICE instead constrains the cost upper bound, which is estimated in way that exploits the distribution correc- tion w obtained by solving the RL problem. Such reuse of w eliminates the nested optimization in CoinDICE (Dai et al. 2020), and COptiDICE can be optimized efficiently results demonstrated that our algorithm achieved better trade-off between reward maximization and constraint satisfaction than several baselines, across domains and conditions.
result. Experimental

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like thank Rui Zhu for technical support and Sandy Huang for paper feed- back. Kee-Eung Kim was supported by the National Research Foundation (NRF) of Korea (NRF- 2019R1A2C1087634, IRF-202IM3II communication Technology (MSIT) of Korea (ITTP No.2019-00 20000000007,,, IITP No.2020-0-00940, IITP No.2021-0-02068),
and the Ministry of Science and Information
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learn- ing Representations (ICLR), 2018.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceed- ings of Machine Learning Research, pp. 22-31 PMLR, 06-11 Aug 2017.
Eitan Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999.
Christopher M. Bishop. Mixture density networks. Technical report, 1994.
V.S. Borkar, An actor- actor-critic algorithm for constrained markov decision processes. Systems & Control Letters, 54(3):207-213 2005.
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: evar optimization approach. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neurul Information Processing Systems, volume 28, 2015.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvari, and Dale Schuurmans. CoinDICE: Off-policy confidence interval estimation. In Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 33, pp.
2020.
Gabriel Dulac-Arold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learn- ing. 2020.
Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton, Florida, USA, 1993.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference Machine Learning (ICML), 2019.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML). 2018.
Josiah Hanna, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off- off-policy evaluation. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS, pp. 538-546. ACM, 2017.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie. Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint 2tw.::200000097790 2020. URL /eatttss//axxxv .tg.g abs/ 2006/206..00 00979
10

---

Published as a conference paper at ICLR 2022
Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 2005.
Natasha Jaques, Asma Ghandcharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribu- tion matching. In Proceedings of the 7th International Conference Learning Representations (ICLR), 2019.
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. Marina Meila and Tong Zhang (eds.), Proceed- ings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5774-5783. PMLR, 18-24 Jul 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-cpolccy Q- learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Reinforcement learning: State-of-the-ar Springer Berlin Heidelberg, 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference Machine Learning (ICML), 2019.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3703-3712. PMLR, 09-15 Jun 2019.
Byungjun Lee, Jongmin Lee, Peter Vranex, Dongho Kim, and Kee-Eung Kim. Batch reinforcement learning with hyperparameter gradients. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5725-5735. PMLR, 13-18 Jul 2020.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6120-6130. PMLR, 18-24 Jul 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems, 2020.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations. ICLR, 2016.
Daniel J. Mankowitz, Dan A. Calian, Rae Jeong, Cosmin Paduraru, Nicolas Heess, Sumanth Dathathri, Martin Riedmiller, and Timothy Mann. Robust constrained reinforcement learning for continuous control with model misspecification, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level. control through deep reinforcement learning. Nature, (18(555)))55)595555555 2015.
11

---

Published as a conference paper at ICLR 2022
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior aagaviit-agnottcc estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems (NeurIPS), 2019a.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. AlgaeDICE: 2019b.
Policy gradient from arbitrary experience. arXiv preprini
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019.
Harsh Satija, Philip Amortila, and Joelle Pineau. Constrained Markov decision processes via back- ward value functions. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 8502-8511, 13-18 Jul 2020.
Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmalcki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning, 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang. Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550:354-359, 2017.
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust mdps using function approximation. In Proceedings of the 31st International Conference on International Conference on Machine Learming (ICML), 2014.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations (ICLR), 2019.
Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learn- ing. In International Conference on Learning Representations (ICLR), 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 7768-7778, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, 2019.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning, 2021.
Mengjiao Yang. Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selec- tion under uncertainty, 2020a.
Mengjiao Yang. Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Qisong Yang, Thiago D. Simão, Simon H Tindemans, and Matthijs T. Spaan. Wesae: Worst- case soft actor critic for safety-constrained reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):10639-1064 May 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Infor- mation Processing Systems (NeurIPS), 2020.
Ruiyi Zhang, Bo Dai, Lihong Li. and Dale Schuurmans. GenDICE: Generalized offline estimation of stationary values. In Proceedings of the 8th International Conference on Learning Represen- fations (ICLR), 2020a.
12

---

Published as a conference paper at ICLR 2022
Shangtong Zhang. Bo Liu, and Shimon Whiteson. GradiemtDICE: Rethinking generalized offline estimation of stationary values. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2020b.
13

---

Published as a conference paper at ICLR 2022
A ALGORITHM FOR UNDISCOUNTED CMDP
Y 1, the optimization problem 4 constraint Σ,, d(s, a) =
should be modified by adding an additional normalization
max (ea) ([(s -DD
(28)
s.t. (( 2₁
4k = 1,
K
(29)
-d(s'a)) - (1)))) + (d(ss))
(30)
6,0
d(s,a)20 Σ dd(,,a)=1 =1
Ss.
(31)
(32)
8,8
Then, consider the Lagrangian:
K
min max (axx) (R [() - 220,0 d20
- A (( -44)
Σ ==1
Ed( -1
-1]
-1] (33)
where μ E R is the Lagrange multiplier for the normalization constraint (32 Then, we rearrange the terms that the direct dependence d is eliminated, while introducing optimization variables
w(s a) (s,a)
d(s.a) that represent stationary distribution corrections:
[R(s,a) - ( v(s) - 4] -BB(ea)ndo) [1( (( ]
min max E (s.a)dd X20,6 d20 (T.a()))
+ (1 )) E + +T
min max E (d (()x, pa) - af(w(s,a))
A20 w20
- (1-7) (300) + T
(34)
which yields the minmax optimization problem that can be optimized in fully offline manner. Due to the strict convexity of f, we can also derive the closed form solution of the inner maximization problem:
(f)
-1)
(35)
which simplify the the minmax problem
34 into the following single minimization problem:
min L(),U, 220.2.P
((s.a)dd (s (Ex, μ) af
(1 () [=)s ATē
(36)
Once the optimal solution x*",",,'" ,²)) are obtained, we can also compute the stationary distribution corrections of the optimal policy using (35). Still, naively using (36) would similarly result in cost violation when the resulting policy is deployed to the real environment. therefore adopt
constraining the upper bound of cost value:
V; μ f- arg min L(A, v, v) viu
(RL for ATC using (36)
(37)
T.X f- arg min Σ K Ek=1 l Xhs (..)
(Upper cost value estimation using (18)
-20,x
- arg min AT (é- -(T, w. w. X20
(Cost Lagrange multiplier)
Bupper ((
which completes the brief description of COptiDICE for 1.
14

---

Published as a conference paper at ICLR 2022
B PROOFS
Proposition
| For any and X, the closed-form solution of
max (W, = (o,a)=o ()ss.))exu(ssd)) a a((((s,))) + - 7) [M(50)] A w20
(38)
is given by:
-(') (teeoltt)))) where x4 max(0,
(39)
Proof. For a fixed and v, we write the dual of the maximization problem max 20 L(w, X, 2):
max min L(w, A, v) + Σ x(s, a)
M20
8,0
max L(w, + Σ
k(s, (s. (s) d: > 0).
220
8,a
By the strong duality, it sufficient to consider KKT condition for (w ,K²). Condition 1 (Primal feasibility) w" (s, a) 2 0
Condition 2 (Dual feasibility) n² n" a) 2 a
Condition 3 (Stationarity) d ds(s (s. af +x 0 Oss
(s = (s) + a)) (f"= (H(e (x
(40)
Condition 4 (Complementary slackness) w (s, a ()"" (s, Vs
Then, we will show that:
= (o)ss)))(()))
(2)x
(41)
satisfies the KKT conditions for (s, a). First, Primal feasibility always satisfied by definition of ww (s, a). Then, we consider either (s, > f'( or (s, < (0) for each (s, a).
(Case a a)) f(()):: In this case, (s, a) 0. where Dual feasibility and Complemen- tary slackness is satisfied. Stationarity also holds by:
(x, a) = ( ())))(s,)))
= (f (He))=( (eex(()
(by assumption)
= (
+ x (s,
40
Therefore, KKT conditions (Conditions 1-4) are satisfied.
(Case 2: (s, a) (0)): In this case, x² (s, (0) (s, a), where Dual feasibility holds by assumption. Also, w (s, a) = by assumption, which implies Complementary slackness also holds. Finally, Stationarity also holds by:
(s, a) = =(f') (#))
=0
(by assumption)
-((0)
= - + (s,
As a consequence, KKT conditions (Conditions 1-4) are always satisfied with w (8,a)
(f')- (ee,, a)) which concludes the proof.
15

---

Published as a conference paper at ICLR 2022
Proposition 2
The constrained optimization problem 17 can be reduced solving the follow- 17
ing unconstrained minimization problem:
min (e(T,xs log ED eep (+ (+(m(s, 7x((s) x(x)) + +1 (1-7)x((0)))
+
720x
18
where E R4 corresponds to the Lagrange multiplier for the constraint 16 and x(s)
IR
corresponds the Lagrange multiplier for the constraint (17). In other words, m {(T.
B(a,))=''' (wss, a)Ca(s (s, a)] where V is the optimal perturbed distribution of the problem 15 Also, for the optimal solution (r".x)), p is given by:
(x) d (x)) (+ (w(s, +x)
19
(x)
(unnormalized weight for (Su, 8, s'))
Proof. For the given constrained optimiizzionnnn problem:
E "eo.....'''"ppllss,. BEA(X)
(42)
s.t. DKr. 80, 8, s)
S Σ ((s((a')wwss d') - (1-7)0o(8') (o)n(s')++0) + Σ
(43)
AA
(44)
8.8
We consider the Lagrangian:
min F20.x.5
Σ D(50, a. (. - [ (55 8, log Blso.s.a.'))
-t)
7 (30,8,035) 57
82U
xxs') " 7))o('')))))) ( [
(45)
where r E 1 is the Lagrange multiplier for KL constraint 433 X(5) E R is the Lagrange mul- tiplier for (4.) and IR is the Lagrange multiplier for the normalization constraint that ensures Σ. p(x) Then, we rearrange (45) by:
min max ((( +xx(s) - x(s)) + +1 7)x(s0) - T rlo (z))) +20. $20
TE - < Σ - -1] =: (T.
(46)
Then, we can compute the non-parann non-paraettric closed form solution for each sample x (so, s, a, s') for the inner-maximization problem. Thanks convexity of KL-ddiergence, it is sufficient consider 0g(7,x.C.F) 0 for each æ. Then, 0p(z)
0g(T.x(C.)) w(s a) () + + x(s)) + (1 7)x(60)) 7 log 5(x) (x() -
0p(x)
+ p(x) x (x) exp ( ( a) + 7x(s') x(s)) + +1 (1-7)x(()))
(47)
with some normalization constant that ensures Σ, p(x) = 1, which is described with respect to Ç.
Then, by plugging (47) into (46), we obtain the result. Also, (19) is the direct result of (47).
16

---

Published as a conference paper at ICLR 2022
C PSEUDOCODE OF COPTIDICE
Algorithm COptiDICE
Input: An offline dataset D {(50, 8, a, c, s'): 1 learning rate n)
1: Initialize parameter vectors 0, 0, X, T, 1h
2: for each gradient step do
3: Sample mini-batches from D.
4: Compute gradients and perform SGD update:
5: 0 - 0 ())
(Eq. 23
Φ - Φ - 1 (
(Eq. (25)
6:
T T 7/ xx, (T.
(Eq. 25
A + [A mJ1(1)]+
(Eq. (26)
7:
+ V (16)
(Eq. 27
8: end for
D COMPARISON WITH COINDICE
CoinDICE Dai et al. 2020) is a Dimillamily algorithm for off-policy confidence interval estima-
tion. For a given policy CoinDICE essentially solves the following constrained optimization problem to estimate the upper confidence interval of the cost value:
max max maxE (1 PE PEA(X) w20
(48)
(plo, 8, a, s')l d (80, Se
(49)
pls', d'),''',, d') = (1 -
+ EpEss a)w(s, a)p(s'ls, a)a(d'ls') a))(dl)) Vs',a'
(50)
s.a
The constraint 500 is analogous to the ग dedependent Bellman flow constraint: d= (8', a') = - 70))))))p'')maaaaa + ((
Ao',a'
(51)
8,0
It is well known that the transposed Bellman equation (51) always has a unique solution dt, i.e. the stationary distribution of the given policy л. Note that for fixed p. the constrained optimization problem (48 50 is over-comstrained for w(s, a) by (50), and therefore the optimal solution will sim-
p(s.a) to satisfy the transposed Bellman equation (1 on the empirical MDP defined by P. Then, we want to adversarially optimize the distribution P so that it overestimates the cost value by (48). At the same time, we enforce that the distribution P should not be perturbed too much from the empirical dsta distribution dd by the KL constraint (49). Finally, following the
ply be given by w" (", a) = (a,()
similar derivation in Proposition 2 we can reduce the constrained optimization problem (48 50 to solving the following unconstrained max-min optimization.
max min T log E aud w20 +20,0 agrst(sg) (oTt(a)
Tep (+
twuls' v(s'((()) (1 (0(s,..a))) + +Te
where max (-) is estimate w(s, a) p(s.a)
(s.o)
In contrast, we consider the case when w is given and aim solve the following constrained opti- mization problem.
max Beonsaar)--losss))CCC REA(X)
15
s.t. DKL (5(so, s, a, (s), S E a')o'''' (oooss,aa)) a') = (1- (1-7))((6') + +7 *7s, ())")))))) As
16
3.a
This can be reduced to unconstrained minimization problem, without requiring nested optimization estimate wr
min T log E Exnap exp (2 (w(s, ()) xx(') - x(s)) + (1 -=)x((s))))] + re T20.X
17

---

Published as a conference paper at ICLR 2022
E EXPERIMENTAL SETTINGS
E.1 RANDOM CMDPS
For random CMDP experiments. We follow similar experimental protocol as (Laroche al. 2019
Lee et al. 2021 with additional consideration of cost constraint.
each run, we constructed a random CMDP with |S] = 50, A 4, 0.95, and a fixed initial state so. The transition probability is constructed randomly with con- nectivity 4, i.c. for each (s, a). we sample states uniformly, and then, the transition probabilities to those states are determined by Dirichlet 1, 1). The reward of 1 is given to single state that minimizes the optimal policy's ss reward value at so. and is given anywhere else. This reward design can be roughly understood as choosing a goal state that is most difficult to reach from 8o- The cost function is generated randomly, i.e. C(s, a) ~ Beta(0.2, 0.2) for Vs E S, a E {az, 03, 04) and C(s, a1) 0 to ensure existence of a feasible policy of the CMDP. Lastly, the cost threshold è = 0.1 is used.
Random CMDP generation
Data-collection policy construction The pseudo-code for the data-collection policy construction is presented in Algorithm where M is the underlying true CMDP, and ên E {0.09, 0.11} is the hyperparameter that determines the cost value TD. Starting from To = x², the policy softened
Algorithm Data Dala-collection policy construction
Input:
CMDP M (S, A, T, R. C, é, Po. 7), target cost value of the data-collection policy êp Compute the optimal policy π² and its reward value function Q7 (s, a) on the given CMDP M. Initialize Tsoft < x and TD f- π
Initialize temperature parameter T < 10-6
Compute 0.9-optimal behavior policy in terms of reward performance. while V
(5o) 0.9V (so) + 0.11 (so) do
Set sooft to soff (als) x exp (+QR (8,a) Vs, TD arg min Dy (d I|d s.t. (ssaaaa [C(s, a)] EDD
7/0.9
end while
Output: The data-collection policy TD
via Tooft (als) exp(QR a)/r) while projecting it into the cost-satisfying by solving Tp arg min, (d(d") (dd [|d s.t. [ssaa)=a= [C'() a)] S êp. This process is repeated until the reward performance of TD reaches 0.9-optimality while increasing the temperature т, i.c. V (5o) = 0.9V (8o) + 0.1V Tunit (so).
After the data-collection policy an is constructed, sample trajectories using wD, which stitutes the offline dataset D. We conducted experiments for a varying number of trajectories, i.e. (the number of trajectories) E {10, 20, 50, 100, 200, 500, 1000, 2000). Each episode is terminated either when 50 time steps have reached or the agent reached the goal state that yields a non-zero reward.
Hyperparameters For tabular COptiDICE, we used N and 0.1 N where N denotes the number of trajectories in D. We also used f(x) 1 (x - 1)2, which corresponds to diveergeee
C-SPIBB: Constrained variant of SPIBB C-SPIBB solves the offline RI. problem with respect to - XC via SPIBB while updating À in the direction of (Ve(TSPIBB) ê), where (c(TEPIB)) is evaluated using the MLE CMDP. For C-SPIBB, we used NA the hyperparameter of SPIBB.
18

---

Published as a conference paper at ICLR 2022
E.2 RWRL CONTROL TASKS
Network architecture and hyperparameters We used the ACME framework (Hoffman et al, 2020 For the P4 network and the X² network, used LayerNo rmMLP with hidden sizes of 152, 512, 256]. For the policy network Tv, we used the network architecture used in CRR (Wang al. 2020). Specifically, we the Te network consists of ResidualMLP blocks with hidden size of I024 and mixture of Gaussians policy head with 5 mixture components. We used the batch size of 1024. We use Adam optimizer with learning rate 3e-4. Similar ValueDICE (Kostrikov et al,) 2021), we additionally adopt gradient penalties (Gulrajani et al. 2017) for the network and the X network with coefficient 10e-5, We performed grid search for a {0.01, 0.05, 0.1} } for each domain. We used the following f as in OptiDICE (Lee et al. 2021):
x logg x 2 ++11 (x 1)2
f(x)
i
Task specification We conduct experiments on domains using the RWRL suite with safety-spec. The safety coefficient is flag in the RWRL suite with speety-spec, and its value can be between 0.0 and 1.0. Lowering the value of the flag incurs more cafett-constraing violation (cost of 1). Originally, each domain has multiple types of safety constraints, but hardest safety constraint to be satisfied by an online constrained RI agent. In summary, we used the following task specifications (safety coefficients, name of the used safety constraint, cost threshold a).
only of them, which was the
Cartpole (realworld-swingup)): safety-coeff-0.33 slider.pos, 0.1,
Walker frealworld-walk, safety-coeff-0.3, vo = 0.1.
Quadruped (realworld-walk): safety-coeff-0.5, joint. angle constraint é= 0.3.
Humanoid (realworld-walk, safety-coeff-0.5, a è = 0.3.
The offline dataset consists trajectories of 1000 episodes for Cartpole and Walker, and 5000 episodes for Quadruped and Humanoid.
C-CRR: Constrained variant of CRR C-CRR additionally introduces the cost critic Qe and the Lagrange multiplier À for the cost constraint. Then, the reward critic and the cost critic are trained by minimizing TD-error for reward and cost respectively. The actor is trained by weighted behavior- cloning: max, (s (xp(A((, a)/ log ([ass)] where A(s,a) (Qn(s,a) in f (QR(s, a²) AQo(s, d))). with ot T--ss). This corresponds to optimizing the policy with respect to the scalarized reward Rt - XC. The Lagrange multiplier is updated in the direction of (Bee (aaiaaall((ssss (s, a)] -)).
XQc(s, a))
19

---

Published as a conference paper at ICLR 2022
F DISCUSSION ON THE DATASET COVERAGE ASSUMPTION
Although we have made an assumption dD > in Section 3 this is not strictly required for the offline RL algorithm work in practice. We adopted this assumption same OptiDICE (Lee et al. 2021) for the simplicity of describing the algorithm derivation from Eq 7 to Eq (10: the assumption makes Eq. (IO) correspond to solving the underlying true CMDP Eq. 7 If we
take this assumption off, Eq. (TO) then becomes equivalent to solving the reduced CMDP where the state and action spaces are limited the support of d".
To see this, we consider the following constrained optimization problem, where the optimization variables d(s, a) are defined only for the (s, a) who are within the support of dP, We denote po and I as the empirical initial state distribution and the empirical transition function respectively.
max E (sa)ESupp(dP)
d(s, ((s.a)R(s,0) a) (dll()))
(52)
Σ
d(s,a)(C((s,d)) c
K
(s.a)ESupp(dP)
Σ
((d',')) (1 7))) +7
E (app))
(
As' E Supp(d
d'ESupp(a")
d(s, a) 20
Vs,a Supp(d
Then, by considering the Lagrangian for (52) and following the similar derivation of Eq.
we can finally arrive at Eq. (00) without any assumption on the coverage of dD, due to the fact that ((spa))Supn(a")) [d(s. (.)]
d(s.o) E SusppeSSpp() d' a) d (.) () [T') (.)]
always holds. In other words, our offline RL algorithms that rely on Eq. (T essentially solve the reduced CMDP that is limited to the support of d".
20

---

Published as a conference paper at ICLR 2022
G ABLATION EXPERIMENTS ON DIFFERENT COST THRESHOLDS
In order to see the sensitivity of the proposed method to different cost thresholds, we conduct abla- tion experiments on different cost thresholds using randomly generated CMDPs, The experimental setup is identical to the one described in Section 4.1
Cost threshold
Data-collection policy C-SPIBB
Optimal policy
BC COptiDICE
Baseline
COptIDICE (by (12))
Reward
Cost
Reward
Cost
1.0
0.14
0.5
0.12
0.10
0.0
0.10
00.5
0.08
10
10"
10
10
10"
10
10'
10
10"
number of trajectories in D
number of Iraeectories in D
number of trajectories in
number of trajectories in D
0.14
0.12
0.5
0.12
0.10
0.10
0.08
0.08
10
10"
10"
10"
10
10
10"
10"
number trajectories D
number of trajectories
number of trajectories in
number of trajectories
D
1.0
0.12
1.0
0.14
0.5
0.12
0.5
0.10
0.0
0.10
0.0
0.08
-0.5
0.08
10
10"
10
10"
10
10
10"
10"
10"
number trajectories
number traecctories
number trajectorics in
number trajectories
D
1.0
0.12
1.0
0.14
0.5
0.12
0.5
0.10
0.0
0.10
0.0
0.08
-0.5
0.08
10
10
10
10"
10"
10
10
10"
10'
10"
10'
number trajectortes
number trjecctores
number trajectories
number trajectoriis
1.0
0.12
1.0
0.14
0.5
0.5
0.12
0.10
0.10
0.0
0.08
0.08
10
10"
10
10
10
10
10
10"
10"
10"
10"
10"
number trajectortes in D
number of rrajectories
number of trajactories In D
number of trajectories D
(a) when Vo(ra) 0.09
(b) when Vo(Tp)
Figure 4: Result of tabular COptiDICE and baseline algorithms in random tabular CMDPs, Plots for the first two columns in (a) correspond to the case when the cost value of the data-collection policy 0.09 (corresponding to Figure [T). Plots for the last two columns (b) correspond to the case when the cost value of the data-collection policy is 0.11 (corresponding to Figure [p). The mean of normalized reward performance and the mean of cost value are reported for 10K runs, where the error bar denotes the standard error.
On the same data-collection policy (whose cost value is either 0.09 or 0.11) and the same of- fline dataset as in Figure [ we tested each algorithm against different target cost thresholds è {0.08, 0.09, 0.10, 0.11,0.122 Each row in Figure 4 presents the result for each target cost threshold, ranging from = 0.08 to = 0.12. Our COptiDCCE (blue) shows consistent robustness to constraint violation varying cost thresholds, while other algorithms fail to meet the constraints especially when the threshold is set to low values (e.g. rows and 2).
21

---

Published as a conference paper at ICLR 2022
H ADDITIONAL EXPERIMENTS USING MIXTURE DATASET
In Section we demonstrated the results when the offline dataset was collected by a single data- collection policy. However, real-world situations, would common that data-collecting agents act safely in most cases but have some unsafe attempts. To simulate this scenario, we conduct addi- tional experiments on the use of a mixture dataset, where the dataset is collected by both constraint- satisfying and constraint- constraint-violating policy.
Cost threshold C-SPIBB
Optimal policy
BC
Baseline
COptDDCE (by (12))
COptiDICE
Reward
Cost
Reward
Cost
0.12
0.50
0.50
0.10
0.10
10"
10"
10'
10"
10
10"
number of trajectories in
number of trajectoriis D
number of trajectories in D
number of trajactories in D
(a) 80% of trajectories are by cost-satisfying policy, 20% trajectories by cost-violating policy
(b) 20% of trajectories are by cost-satisfyin policy, 80% of trajectories are by cost-violating policy
Figure 5: Result of tabular COptiDICE and baseline algorithms in random tabular CMDPs, using mixture dataset. Plots (a) correspond to the case when the dataset is collected generally by the
cost-satisfynng policy, i.e. 80% of trajectories are by the cost-satisfying policy and 20% are by the cost-violating policy. Plots (b) correspond to the case when the dataset is collected generally by cost-violating policy, i.e. 20% of trajectories are by the cost-satisfying policy and 80% are by the cost-violating policy. The mean (unnormalized) reward value and the mean cost value are reported for 10K runs, where the error bar denotes the standard error.
Figure 5 presents the result, where the dataset collected by two policies, the cost-satisfying one (i.e. its cost value is 0.09) and the cost-violating one (i.e. its cost value is 0.11). The overall trend remains the same as in Figure (Figure 5h RJ Figure Ta. Figure 5 z Figure [p): when the dataset consists of mostly cost- satisfyiiii trajectories, baseline algorithms exhibits less constraint violation, while they show more constraint violation when the dataset consists of mostly cost-violating ones. Our COptiDICE (blue) still shows much stricter cost satisfaction than other baseline algorithms, demonstrating better trade-off between reward maximization and constraint satisfaction than base- lines.
Furthermore, solving the reduced CMDP is valid method in the offline RL setting since we may want to optimize the policy only within the dataset support in order to prevent unexpected perfor- mance degradation by -uitt--distribution actions.
22

---

Published as a conference paper at ICLR 2022
I ADDITIONAL EXPERIMENTS ON DATA-COLLECTION POLICY WITH LIMITED EXPLORATION POWER
For the random CMDP experiment, conducted additional experiments the data-collection policy FD does not cover the entire state-action space well. Specifically, we limit the support of D (-1s:: (a)ss) will have non-zero probabilities only for E Á where C A the subset of A {ai, d2, 03, aa). By doing so, ensure that ap covers only part of the entire state-action space. The detailed construction procedure of TD is described in Algorithm 3 We conduct experiments for = {ai}. A = {ai, a2). and A = {a1 Q2 0z), and the results are presented in F
the results when
Algorithm 3 Constructing a data-collection policy with limited action support
Input: CMDP M (S, A, T, R., C, è, po: 7). the subset of the entire actions, A A, to be consid- ered by the data-collection policy, ēp: the target threshold of the data-collection policy. 1: for each E S and E A do 2:
[(ss,a)
a E À a &
Ō(s,a)
Eliminate action a & Å by assigning large cost.
0
1/|A|
if E A if & À
Aumi((als)
0
3: end for
4: P f-1
5: while true
6: # Artificially lower the cost threshold (êD p) until TD meets the target constraint.
7: #* SolveCMDP(S, A, R, C, ?o)
8: TD + 0.9 #* 0.1 Funif # To make stochastic policy within Á.
9:
if (T)) > ēp then
p - p 0.99
11: else
break
13:
end if end while
Output: TD: the data-collection policy with limited action support.
23

---

Published as a conference paper at ICLR 2022
Cost threshold
Data-collection policy
Optimal policy
BC
Baseline
C-SPIBB
COptDICE (by (12))
COptiDICE
Reward
Cost
Reward
Cost
0.10
1.0
0.10
0.06
0.0
0.00
0.0
0.00
10
10
10"
10
10
10"
10
10"
10"
10"
number of trajectories in
number of trajectories in
number of trajectories in
number of traectofees in
0.11
1.0
0.10
0.10
0.55
0.09
0.5
0.09
0.08
0.08
0.07
0.0
0.07
10
10"
10
10
10"
10
10"
10"
10"
10"
trajecttoss
number trajectories
trajectories
number trajectoree
0.12
1.0-
0.12
0.5
0.0
0.08
0.08
10"
10
10 number trajectories in
10"
10
10
10"
10"
number trajectories in
number of trajectories in number of trajectories
when the data-collection policy cost-satisfying
(b) when the data-collection policy cost-violating
Figure 6: Result of tabular COptiDICE and baseline algorithms in random tabular CMDPs when the data-collection policy has limited exploration power. Plots for the first two columns in (a) correspond to the case when the data-collection policy is cost-satisfying. Plots for the last two columns (b) correspond the case when the data-collection policy is cost-violating (except for the first row). The first row denotes the case when Á {ai}. the second row denotes the case when Á {a1, 02)), and the third row denotes the case when A {a1, 02 az). The mean of normalized reward performance and the mean of cost value are reported for 10K runs, where the error bar denotes the standard
error. error.
The first row of Figure 6 shows the result when Ā = {az]. Since d1 the zero-cost action as described in Appendix Ej the cost value of TD is always 0. Besides, since the dataset contains only a single action a for each state due to the deterministic data-collection policy, no offline RI. algorithms can improve the performance beyond the data-collection policy: there must be more than one action in some states in the dataset, order for offline RL algorithms to have room to improve the performance in general.
The second row and the third row of Figure 6 shows the result when {a1, a2, d3} respectively. As the size of À increases from 2 to 3, offline RL algorithms further im- prove the performance over the collecoolection policy. This is natural since the larger Á implies that there is more room for offline RL algorithms to optimize. Finally, even when the exploration power of the data-collection policy limited, our COptiDICE (blue) shows a consistent advantage baseline algorithms, in terms of the better trade-off between reward maximization and constraint satisfaction.
{ai, 02) and Ā
24


Reviews:
[{"review_id": "q6j1bHcivWb", "reviewer": "Reviewer_jgQ7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provide a interesting and novel algorithm to address offline safe RL problems. Most of the theoretical development and empirical looks good to me, but I have the following question for the author.\n\n(1) In order to make sure COptiDICE works in practice, do we need any special requirements for the behavioral policy (sampling distribution)? Specifically, do we required the behavioral policy (or the sampling distribution) to be feasible?\n\n(2) In the experiment of continuous setting, we can still implement CRR as a safe RL method via combining it with primal-dual approach, i.e., you can simply replace the policy optimization step in primal-dual approach with CRR. If we implment such an algorithm, can COptiDICE still outperfrom this \"safe-CRR\" algorithm?\n\n(3) Has the author considered testing the algorithm in some standard safe RL environment such as Gather, Circle, Half-Cheetah Safe, etc?", "summary_of_the_paper": "This paper studied the policy optimization problem in the offline constrained MDP setting. Compare with previous works, in which policy gradient based approaches are widely adopted, this paper solves the problem via policy visitation distribution that rooted from the primal-dual formulation of Bellman operator, which is novel. In order to guarantee the constraints are always satisfied, this paper provides a novel approach based on CoinDICE to efficiently estimate an upper bound of constraint violation. The author also provide sufficient empirical verifications to support their proposed algorithm.", "summary_of_the_review": "The offline safe RL is a very challenging problem, but this paper provide a very promising approach to address several issues in this setting. Specifically, the approach proposed in this paper not only addresses the policy optimization issue in the behavioral agnostic setting but also guarantees the constraints satisfication, which are both significant contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lBBZiduD6RB", "reviewer": "Reviewer_NsmH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n\n--The paper's approach seems to be very mathematically well-founded. Although the end result is slightly biased, the minibatch-free version simply minimizes a convex optimization problem, so I would assume this bypasses many of the convergence problems common to popular TD-based RL methods.  A line or two clarifying the theoretical guarantees would be nice though. \n\n--The experimental results also seem to demonstrate that the proposed method is better at satisfying the problem constraints than other methods. On a number of environments, this constraint satisficing also comes with little to no cost to overall performance. The authors compare against a good number of baselines on a wide range of environments. \n\nWeaknesses\n--While the result is interesting, its novelty seems a bit limited. The transformation from constrained optimization problem to unconstrained optization is of course drawn from previous DICE methods, although it is adapted here to include the cost constraint. The idea of constraining an upper bound on the cost rather than the estimated cost itself is also well-established. Ultimately though, I think the extension of an existing class of methods to a new domain they have not been applied in yet is sufficient novelty for publication. \n\n", "summary_of_the_paper": "The authors propose a DICE-family method for solving constrained offline reinforcement learning problems. To do this, they adapt ideas from OptiDICE and find a reduction from a nested constrained optimization problem to a single unconstrained optimization problem that can be efficiently represented with a neural network. Additionally, they draw on ideas from CoinDICE to estimate a confidence interval over the cost, which makes their method better at obeying constraints. The  authors compare their method to a number of baselines on both random grid worlds and continuous environments, and find that COptiDICE achieves both good performance and better constraint satisfaction than alternative methods. ", "summary_of_the_review": "The paper is well-founded and feels like a \"correct\" solution to this problem, and the empirical results show the method performs well in practice. I", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S1nJoqBUDzE", "reviewer": "Reviewer_QeFP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) they propose to optimize the state-action stationary distribution directly, which avoids the instability caused by triple optimization problems for the actor, the critic and the cost Lagrange multiplier with three different objective functions in the existing work that manipulates both Q-function and policy.\n\n\nWeakness:\n1) important baselines (state-of-the-art offline RL with Lagrangian approach) are missing: \n- in CMDPs: besides MLE CMDP, only the variant of SPIBB, an offline RL method for tabular MDPs, is compared with the proposed method (applicable to continuous tasks). Behavior cloning is also missing in this setting to see whether the proposed method is just remembering the collected safe dataset.\n- in continuous control tasks: CRR, sota unconstrained offline RL without any modification to handle constraints, is compared with the proposed method\n2) It is always difficult to reach a confident conclusion that one method is better than others when a trade-off between rewards maximization and constraint satisfaction is needed. In both tabular CMDPs and continuous control tasks, although the proposed method shows better performance in satisfying the constraint, the constraint cost is always very close to the predefined single cost threshold. Ablation study on constraint cost threshold is required to both show the sensitivity of the proposed method to different cost threshold and consistent advantage over other baselines.\n\n\nSuggestions\n1) In tabular CMDPs, both constraint-satisfying policy and constraint-violating policy are used to collect datasets. In continuous control tasks, the online constrained RL agents are used to collect data. However, in the real world, agents normally act safely but have some unsafe attempts. It would be more interesting if a mixture dataset collected by both constraint-satisfying and constraint-violating policy is utilized for constrained offline policy learning.\n2) There is a concurrent work, constraints Penalized Q-Learning (CPQ) [1], similarly aiming at offline constrained RL. It would be great if some comparisons are studied between the proposed method and this work (either methodology aspect or empirical aspect).\n3) It would be great if there is some exploration describing how the proposed work can be adapted to hard-constraints scenarios since safety constraints in the real world are often hard constraints as well as constraints imposed by physical laws.\n\n[1] Xu, Haoran, Xianyuan Zhan, and Xiangyu Zhu. \"Constraints penalized q-learning for safe offline reinforcement learning.\" arXiv preprint arXiv:2107.09003 (2021).", "summary_of_the_paper": "This paper has presented a DICE-based offline constrained RL algorithm for constrained RL. Experimental results on tabular CMDPs and continuous control tasks show that the proposed method can achieve a better trade-off between reward maximization and constraint satisfaction.\n\n1st contribution: They firstly proposed to tackle constrained offline RL by solving a single minimization problem. \n\n2nd contribution: To mitigate constraint violation in practice, they exploit the distribution correction obtained by solving the RL problem for cost upper bound estimation and then constrain the upper bound. ", "summary_of_the_review": "The paper proposes to optimize the state-action stationary distribution directly without considering the unstable triple intertwined optimization porblems in actor-critic-based constrained RL algorithms. However, the effectiveness of the proposed method in obtaining a trade-off between reward maximization and constraint satisfaction will be more convincing if 1) more straightforward adaptation of state-of-the-art offline policy learning algorithms are considered as baeslines; 2) more ablation study to analyze the sensitivity of the performance against different cost thresholds.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ESuwaXXoHL3", "reviewer": "Reviewer_qywn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper is well-written and easy to follow. The motivation is well explained.\n- The paper considers applying DICE methods to constrained MDP problems. First, by imposing the closed-form solution of $w^*$, the paper reformulates the minimax optimization problem to a single minimization problem. Then,  the paper proposes a practical algorithm for continuous state-action space with neural networks as the function approximation. The paper contributes to the study of CMDP.\n- The paper provides an empirical study of the algorithm and shows it outperforms the baseline.\n\nWeakness:\n- The assumption that $d^D > 0$ is a strong assumption in offline RL, which assumes that the dataset uniformly covers the whole state-action space. Moreover, since $d^D$ is the empirical distribution of the dataset $D$, such assumption does not hold for continuous state-action space. The paper should provide more discussion on the restrictiveness of such an assumption.\n- The algorithm seems to be a combination of OptiDICE and CMDP. By considering the Lagrangian form of CMDP, the problem essentially reduces to an MDP problem. Thus, I expect less novelty of this paper compared with OptiDICE.\n- In the experiments, the dataset is generated in a way that enforces its exploration power, which makes the dataset covers the state-action space well. However, in real-world applications, the dataset may not be able to cover the whole state-action space, which is the main challenge of offline RL. In particular, in the case that the dataset does not cover the whole state-action space, I encourage the authors to show if the learned policy can outperform the data collecting policy.\n\n\n", "summary_of_the_paper": "The paper considers the offline constrained reinforcement learning problem and formulates the problem as a CMDP. First, the paper presents the algorithm COptiDICE, which directly estimates the stationary distribution corrections of the optimal policy. Then, the paper shows that COptiDICE outperforms the baseline algorithms in terms of constraint satisfaction and return-maximization. ", "summary_of_the_review": "Overall, the paper is well-written and motivated. However, due to the lack of discussion on the coverage of the dataset, I think the paper is below the borderline of acceptance. I would be happy to raise my evaluation if the paper could address my concerns in the main review part.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new technique for constrained offline RL. The proposed method is based on reducing a nested constrained optimization problem to a single unconstrained optimization problem that can be efficiently represented with a neural network. The proposed algorithm is tested against several baselines on both random grid-worlds and continuous environments. Results clearly show that the proposed algorithm outperforms baselines while keep the provided constraints satisfied. 

The reviewers agree that the paper is well-written, the proposed algorithm is novel and technically sound, and the empirical evaluation clearly supports the claims of the paper. There were some concerns regarding the novelty of this idea, but these concerns were properly addressed by the authors in the discussion.
Decision: Accept (Spotlight)