SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
Authors: Youngsoo Jang
Geon-Hyeong Kim
Jongmin Lee
Sungryull Sohn
Byoungjip Kim
Honglak Lee
Moontae Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Imitation learning
Preference-based learning
Safe imitation learning

Abstract: We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimics preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonstrations. This problem setting corresponds to various real-world scenarios, where satisfying safety constraints is more important than maximizing the expected return. However, it is very challenging to learn the policy to avoid constraint-violating (i.e. non-preferred) behavior, as opposed to standard imitation learning which learns the policy to mimic given demonstrations. In this paper, we present a hyperparameter-free offline safe IL algorithm, SafeDICE, that learns safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. Our algorithm directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non-preferred behavior. In the experiments, we demonstrate that our algorithm learns a more safe policy that satisfies the cost constraint without degrading the reward performance, compared to baseline algorithms.

SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
Youngsoo Jang', Geon-Hyeong Kim', Jongmin Lee",
Sungryull Sohn', Byoungjip Kim', Honglak Lee', Moontae Leel.3
I. Al Research
University of California, Berkeley 3 University of Illinois Chicago
Abstract
We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimies preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonstrations. This problem setting corresponds to various real-world scenarios, where satisfying safety con- straints is more important than maximizing the expected return. However, it is very challenging to learn the policy to avoid constraint-violating (i.e. non-preferred) behavior, opposed standard imitation learning which learns the policy mimic given demonstrations. In this paper, we present a hyperparameter free offline safe IL algorithm, SafeDICE, that learns safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. Our algo- rithm directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non non-pref non-preferred behavior. In the experi- ments, we demonstrate that algorithm learns more safe policy that satisfies the cost constraint without degrading the reward performance, compared to baseline algorithms.
Introduction
Reinforcement learning (RL) (28 aims to learn the optimal policy that maximizes the expected cumulative rewards through interacting with the environment. Recently, RL has achieved remarkable successes in various domains, but its application to real-world tasks is yet challenging mainly due to the following two reasons: First, since the interaction with the environment is costly in most real-world tasks, the policy should be learned solely from the pre-collected dataset (i.e. offline learning). Second, reward annotation, which is essential for applying the RL algorithm, is very strong requirement that is difficult to obtain real-world tasks.
Imitation learning (IL) 26 25 is a more realistic framework to deal with real-world tasks, where the expert's demonstrations are solely given without reward annotations. Standard IL aims to mimic the expert's 's demonstrations generated from the optimal policy in terms of expected return. However, in most real-world tasks, additional constraints should be considered in addition maximizing the expected return such the safe driving for autonomous driving and toxicity degree for the conversational agent. Fortunately, non-preferred demonstrations that violate constraints are often naturally collected. For example, black box data of an accident vehicle and toxic content reported by chatbot users are separately collected as non-preferred demonstrations, However, most cxisting studies have focused only imitating expert (i.c. preferred) behavior, and a safe imitation learning method that avoids non-prete preffereddd behavior has not yet been explored.
In this paper, are particularly interested in solving safe imitation learning in an offline setting: Given scarce but labeled non- ooeffrrerddd demonstrations, and abundant but unlabeled demonstrations
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

?
Preferred D
Non-Preferred D
Unlabeled D
DemoDiCE, DWBC
0
X
0
PPL, DExperts
0
0
0
SafeDICE (Ours)
Unlabeled D
Non-Prefeered D
Figure Illustrative example of our problem setting and the position of work among the prior methods and problem settings. First, prior offline imitation learning algorithms (DemoDICE: [I3] DWBC (31), which aim to imitate expert demonstrations, assume labeled preferred demonstrations and unlabeled demonstrations. Second, DExperts [99 require both labels of preferred and non-preferred demonstrations. Unlike the problem settings of prior works, our work focuses the problem of safe imitation learning assuming only the labels of non-preferred demonstration that can deal with safety more importantly.
iifference-baseee policy learning (PPL) algorithms and
of both preferred and non-preferred behavior, we aim to learn a safe policy that follows the preferred behavior while avoiding the non-preferred behavior (see Figure This problem setting has been mostly unexplored, and cannot be completely addressed by existing methods. One of the existing applicable approaches is the seeference-basee policy learning algorithm 4 35 177. which optimizes a policy based on preference rankings over demonstrations. This method can learn the policy to prefer unlabeled demonstrations non-preferred demonstrations, but it also prefers non-preferred demonstrations mixed the unlabeled demonstrations, which can lead to learning suboptimal policy. One of the approaches effectively using the unlabeled dataset is DWBC [11, weighted behavior
cloning based on a discriminator, which adopts the positive-umlabeled unlabeled learning (9] 36 30] for training
the discriminator. This method requires additional cost of hyperparameter search for training
the discriminator, and its performance is hyperparameter sensitive. Another notable approach in the natural language processing domains is DExperts [19], decoding-tine method, which combines trained models with preferred and non-preferred demonstrations. This method is also hyperparameter sensitive and prone the degeneration issue that leads to abnormal behavior. Both DWBC and DExperts, which require hyperparameter search, are unsuitable for real-world tasks where interaction with the real environment costly and risky.
We present an offline safe IL algorithm that learns safe policy by leveraging the non- preferred demon- strations in the space of stationary distributions. Our algorithm, offline Safe imitation learning by avoiding non- non-preferred demonstrations via Distribution Correction Estimation (SafeDICE), directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non-preferred behavior. SafeDICE essentially leverages the non-preferred demonstrations in the space of stationary distributions, unlike existing algorithms that leverage it in the space of policy [19] or the learning process of the discriminator [31]. We show that our algorithm can be reduced to single convex minimization problem without additional hyperparameter search, unlike existing algorithms that are highly sensitive to the hyperparameter. In the experiments, we demonstrate that our algorithm learns more preferred (i.e. constraint-satisfinng) policy compared baseline algorithms in constrained RL benchmarks including RWRL [ and Safety Gym 24
2 Preliminaries
2.1 Markoy Decision Process (MDP)
We consider an environment modeled as a Markov Decision Process (MDP), defined by M (S, A, R, Po: 7) 28). where S is the set of states, A is the set of actions, 7 S A A(S) is the transition probability, R S x A + R is the reward function, Po E A(S) is the distribution of the initial state, and 7 E [0, 1) is the discount factor. For the given policy n, its stationary distribution dl (s, a) is defined as follows:
d= (8, == :=
Σ xt Pr(si 8,0, t=0

---

where ~ Po, ~ T(-(s)), and 8141 ~ T(- st. ar) for all timesteps t 2 0. For brevity, d" will be used denote the stationary distribution d" (s, a).
We consider the offline imitation learning setting, where the policy should be optimized from solely pre-collected dataset without reward annotation and any additional online environment interaction. In our problem setting, preferred and non-preferred behaviors are defined by satisfying or violating constraints. We denote the preferred policy by n² the non-preferred policy by n², and the corresponding stationary distribution of T" and T by d and d' respectively. We assume a pre-collected dataset that consists of scarce but labeled dataset D generated by the non- con-preferred policy, and a abundant but unlabeled dataset D generated by both preferred and non non-preferred policies. We denote the corresponding stationary distribution of D by d' which can be regarded as convex combination of d" and dt with unknown ratio (i.e. d := (1 - a)d" + ad"). In this setting, we aim to learn safe policy from unlabeled demonstrations while avoiding non-preferred demonstrations that are labeled as non- non-preferred behavior.
2.2 Preference-based Offline Imitation Learning
Imitation learning (IL) aims imitate the behavior of expert policy solely from state-action demon- strations without reward annotations. The standard IL is naturally formulated as minimizing the KL divergence between stationary distributions of preferred policy (i.e. expert policy) and target policy
14
arg min DKL (dd (d")ld").
(1)
However, it requires large amount of annotated preferred demonstrations, which may be too strong requirement for various real-world scenarios. Recent offline IL method [13] relaxes this requirement for preferred demonstrations by leveraging the unlabeled dataset which may contain both preferred and non-preferred demonstrations:
== arg min DKL (d [[dP) aDKL (d (d" Idd))
where a 2 O is a hyperparameter that controls the balance between minimizing the KL divergence with d and d"". This offline imitation learning method can learn the preferred policy from large number unlabeled demonstrations with only a small amount of labeled preferred demonstrations.
In contrast to the mainstream of recent IL approaches focusing on mimicking expert (i.e. preferred) behavior, avoiding dangerous (i.e. non-preferee non-preferred) behavior is a more important problem in many real-world tasks, which has been unexplored yet. In the same manner as Eq. (2), we can naively consider the penalty for the non-preferred behavior by maximizing the KL divergence with dº=
mrg DKL (d (d" (d"lld") + aDKL (d=lla")
where a 2 is hyperparameter that controls the balance between minimizing the KL divergence with d and maximizing the KL divergence with dd However, since the KL divergence has no upper bound, optimizing Eq. (3) very unstable and challenging, where the (d= can blow up to infinity. Therefore, safe imitation learning while avoiding non- non-preferred behavior is not straightforward problem and has not yet been studied.
3 SafeDICE
In this section, we present offline Safe imitation learning by avoiding the non-preferred demonstrations via DIstribution Correction Estimation (SafeDICE), an offline IL algorithm that can learn the safe policy using labeled non- pre-rrferred demonstrations. SafeDICE optimizes the stationary distributions of the target policy match the stationary distribution of the preferred policy from the unlabeled dataset by excluding non- non-preferred demonstrations.
3.1 Offline Safe Imitation Learning with Mon-Preferred Demonstrations
Similar to standard II. approaches, the derivation of our algorithm starts from KI. divergence mini- mization between d= and d', which can be estimated by d and d follows:
d ad
arg min DRL.
d
(4)
3

---

where a is unknown ratio between the stationary distributions of preferred policy and non non-preferred policy (i.e. d" (1-a)d" + ad"), ). Optimizing the Eq. (4) sensitive to a, and selecting the with hyperparameter search is not suitable for real-world tasks where the interaction with the real environment very costly and risky. Therefore, is not straightforward to learn the preferred policy using non-preferred demonstrations with existing imitation learning algorithms. The main contribution of our paper is to present an algorithm that theoretically determines a and learns the preferred policy only with the labels of non-preferred demonstrations without any hyperparameter search. In the following description, we first consider the a as a known ratio in our derivation, then present the way to select the while providing the theoretical guarantees the error bound of optimized stationary distribution with selected a. All the proofs can be found in Appendix A
The main flow of our derivation follows prior DICE-based offline imitation learning methods 12 13 but it is clearly different from the existing methods that require hyperparameter search to deal with unknown a. First, we consider problem equivalent to Eq. (4) in terms of stationary distribution d:
max - -DLL (a d" ad d20
(5)
1 1-
S.L. (B. = (1 Mopss + (s) vs,
where (8) = E. d(s is the marginalization operator, and (s) Ex, T(s|s, a)d((, a) is the transposed Bellman operator. The Bellman flow constraint en- sures that d(s, a) is valid stationary distribution of some policy, where d(ss, a) can be interpreted as
a normalized occupancy measure of (s, a). Therefore, if we find optimal d' (s, a) that satisfies Eq. (5)
and (6). then its corresponding policy can be easily obtained by x² (als) = Σ. (s.a)
d" (s.a)
The Lagrangian dual formulation of the above constrained optimization problem is
max min DKL ( d -ad-V d20
=E'( u(s) ((s (7)())))+((())))))))) (
where v(s) are the Lagrange multipliers for the Bellman flow constraints 6 However, directly optimizing the Eq. ( is challenging in offline manner, where further environment interactions not available. To obtain a tractable objective, we introduce the following derivations:
-DKI [al]
d -ad=a
adN
+ Ev(s) (1-)) 7))
+ (T.d)(s) ()-)))))))
(s).)))))
1 a
(1 (1- a)
(1 Eno [m(s] Edd
7 (TT)) v(s) log d (8 ad (8,
d(s, a)
d (8. - ad (s,
= (1 - 7) Epu (v(s) Edd
((Tu) v(s) log j (s.aa
+ log
(1 (1- al-a)du(ss,0
=
= - =11-7)Bpu [M(s]] + Ed as( + (Tv)((()) - v(s) - llg wos
==A(aaaa
= - Ep [m(s]] + Eg (wss, a) (A, log wss a))]
=: ((w.v;rr.)
where (s, a) corresponds to the preference-ber reward function in the context of RI, which will be trained by the pre-collected dataset.
In summary, our goal is reduced to solve the following maximin optimization:
max min L(w, 1; ra),)
(8)
w20
where the optimal solution w of Eq. 8 represents the stationary distribution correction of an optimal policy n² (i.e. w² w" a) d=d (s.a)

---

3.2 Pretraining the Preference-basee Reward Function (s,
To optimize the Eq. S
we pretrain the discriminator to estimate the log ratio of d (s, (s,a) a) and d² (s,a), which is required to estimate for the ro (8,
arg max EaN [log c(s, a)] + Eg [log(1 - c(s,a))].
(s.a) With the optimal discriminator c* (s, a) (' (a,a) trained by Eq. (.. the preference-basee reward function To (8, a) can also be obtained as:
a1-a)(1- c² (s, )
1 - (1 a: (s.
(3, - log
(10)
However, according the value of a, the reward function (s, a) defined by Eq. (0 cannot guarantee that the value inside the log function always becomes a positive. Therefore, finding the optimal a through hyperparameter search is infeasible.
3.3 Reduction to the Single Min Optimization
Since the optimization Eq. convex optimization, we can change the order of operator from maximin optimization to minimax optimization (2]:
min max [(w, r)r).
(11)
w20
Then, we can replace the inner maximization of Eq. (TT) with a closed-form solution for w, which
can be easily obtained by exploiting the convexity.
Proposition 3.1. For any u, the closed-form solution to the inner maximization of Eq. 11 i.e.
" arg max [(w, v) given by:
(s, exp(AA((ss)) - 1)).
(12)
By using the closed-form solution w of Eq. tion follows:
12 our derivation can be reduced to a single minimiza-
min C(w wir) =(1 - Ep [v(s)] + Egu exp((A, - 1)] -)]
(13)
obtain the optimal stationary distribution correction w" which can casily extract the corresponding optimal policy π². To extract policy from the optimal stationary distribution correction w" (s, a), adopt weighted behavior cloning on the offline dataset as follows:
Finally, by optimizing the single minimization problem of Eq. 13
min [( [lo)) = - (a. (w" (s, a) log (als)],
(14)
where w" (8) = d= (s.a) du (s.o)
3.4 Hyperparameter a Selection
In the previous sections, have derived algorithm under the assumption that is known ratio. The remaining question is how we select the suitable a without additional hyperparameter search. Fortunately, in contrast the existing methods [19] R1 that require the additional cost of hyperparameter search, we can specify the with providing the theoretical guarantees on the error bound of the estimated stationary distribution.
(x.0)
Proposition 3.2. Suppose minl (s.a) we obtain the estimated stationary distribution by
optimizing the Eq. 7 with a set to â as follows:
d (((ss))
à = min (s,a]
(15)
dN (s, a)
5

---

then the total variation distance of true and estimated stationary distribution of preferred policy is bounded as follows:
Drv (s, (s, (s,)), I+E'
(16)
where Dry total variation distance, (s, is true stationary distribution of the preferred
policy and d (s.a) estimated stationary distribution which defined by â as å (s, d² (s.a)_dd" (s.a) (Proof in Appendix 1-a
Intuitively, since the behavior of the non-preferred and preferred policies are different, there must exist state-action spaces that are rarely visited by the preferred policy but are frequently visited by the non-preferred policy. Therefore e may be a very small positive value, then the total variation distance of the true and estimated stationary distribution is close to zero:
lim Drv (d (s, a), d (s) =. c0
(17)
We also provide proof in Appendix A.3 that (8, a) of Eq. (TO). defined by the a selected by Eq. 15 always well-defined with positive value inside the log-function in Eq. T
Finally, our algorithm optimizes the Eq. (7) with the a obtained by Eq. (15). where a is no longer an unknown hyperparameter. In short, by leveraging the non-preferred demonstrations in the space of stationary distributions, this problem can be resolved by solving a single convex minimization problem for specific a. This is in contrast to the existing methods, which directly leverage the non-preferred demonstrations for training the policy [19] or discriminator [31] that require the additional cost of hyperparameter search. The pseudocode for the whole process of SafeDICE can be found in Appendix B
4 Related Work
Stationary distribution correction estimation (DICE) Since the introduction of DICE for off- policy evaluation in DuaIDICE 221] various DICE-family algorithms have been developed for off- In addition to these studies, two algorithms, AlgaeDICE (22) and ValueDICE [14), have been proposed to address the challenges of off off-policy RL and II., respectively. However, they may encounter issues with numerical instability in an offline setting due to the nested min-max optimization and out-of-distribution action evaluation.
policy evaluation B3 34 32
To overcome the issues of numerical instability offline settings, recent DICE-based offline RL [5
16 and IL algorithms 13 20 12 have adopted strategy of directly optimizing stationary distribution
correction ratios, similar to the approach used in our work. Additionally, these offline II. algorithms utilize unlabeled demonstrations in effort to address the problem of distribution drift [25]. They share a similarity with our algorithm in that they all aim to recover the expert's 's behavior through direct optimization of the stationary distribution correction ratios. However, these algorithms rely on the availability of clean expert demonstrations, which can make them difficult to use in situations where expert demonstrations not available.
Policy Learning from Suboptimal Demonstrations Preference- policy learning algorithms use ranked suboptimal demonstrations to learn reward function. Many of these algorithms, such as T-REX 44, PrefPPO [8], and PEBBLE 17,, utilize Bradley-Terry model 3 to model the reward function. Once the reward function is learned, T-REX and PrefPPO use PPO 27 to train policy, while PEBBLE employs SAC [0]. While preference-basee policy learning algorithms allow for learning from unlabeled demonstrations and potentially disregarding the negative influence of non- preferred demonstrations, it is important to note that they may also give preference to non- non-preferred demonstrations mixed in the unlabeled demonstrations,
On the other hand, there are also many studies that utilize suboptimal demonstrations, even though they are not preference-bart based policy learning algorithms. D-REX [5] and SSRR [6] both learn policy by using given suboptimal demonstrations, and then generate trajectories with varying levels of optimality using this policy with injected noise. But, both algorithms require online sampling, which is not feasible in offline settings. Wu al. (29) proposed two algorithms that utilize positive- unlabeled learning [9 36 30 to train discriminator for constructing reward function. However,

---

both algorithms require ground-truth labels for uniformly sampled trajectories from unlabeled demonstrations. DWBC [3]] uses positive-unlabeld learning to train a discriminator, which is then used to construct weights for weighted BC. DExperts [19], a decoding-time method in the natural language processing domain, combines models for preferred and non-preferred demonstrations. Unfortunately, both DWBC and DExperts may not be suitable for offline safe imitation learning tasks due to their sensitivity to hyperparameter settings.
5 Experiments
In this section, we show the experimental results on various tasks from constrained RL benchmarks. First, we conduct the evaluation on domains from Real-Worid RL (RWRL) suite [8) which pro- vides challenges in real-world scenarios including safety constraints. Second, we also conduct the experiment on continuous control tasks from Safety Gym environment (24]. which provides practical scenarios of safety issues. In order to show the motivation of our work and the limitation of existing algorithms, we also provide qualitative results on tabular MDP environments. The illustrative examples and qualitative analysis on tabular MDP environments can be found in Appendix G
Baselines Since offlin safe II., which we consider in this paper, has not been explored yet much it lacks directly applicable baseline algorithms. Therefore, we consider the following four algorithms as baselines. First, BC denotes simple behavior cloning unlabeled demonstrations. Second, DWBC denotes a variant of the original discriminator-weij weighted behavior cloning algorithm [31 that adopts the uosibive-ullabeled learning for training the discriminator: To apply it to setting, we modified vositive-unlabele learning to begative-unlabele policy learning algorithm that adopts the preference- based reward learning algorithm based on the Bradley-Terry model 3 and weighted behavior cloning for policy learning. Lastly, we also compare with DExperts 19]. a decoding-tine method that combines trained models with unlabeled and non-preferred demonstrations. The details of baseline algorithms are provided in Appendix C
learning. Third, PPL denotes a preference- based
Evaluation metries We compare the performance of SafeDICE and baseline algorithms in terms of task performance (i.e. expected return) and safety (i.e. cost constraints). We report the results with the following metrics: 1) Normalized Return: evaluates the normalized task performance of the policy, 2) Average Cost: evaluates the mean cost the policy, 3) Cost- Violating Ratio: evaluates the ratio of cost-violating behavior of the policy, 4) Conditional Value at Risk performance (CVaR) 10% Cost: evaluates the mean cost of the worst 10% runs. We also provide CVaR 1% Cost for various values of k in Appendix E.I. Here, our main focus of the comparison is to evaluate the safety (Average Cost, Cost- Violating Ratio, and CVaR 10% Cost), and we use the Normalized Return to examine the degeneration issue, whether the agents fall into abnormal behaviors not. All metrics measured as the average value of 500 trajectories generated from the learned policy, and we report the results that are averaged over 5 runs.
5.1 Real-World Reinforcement Learning (RWRL) Suite
Now, we present the results of SafeDICE on set of continuous control benchmarks from Real-World RL (RWRL) suite 88, where the safety constraints are also provided by environments.
Task setup We conduct the experiments for three tasks from RWRL environment with safety constraints: Cartpole, Walker, and Quadruped. Since there is no standard dataset for offline IL considering the safety constraints, we collected data by training the online RL agents. To obtain an preferred and non-preferred policy, we trained the SAC [10] in different two ways that solve the RL problem with respect to reward penalized by the cost or not. Among these two policies which are both optimal in terms of return maximization, we define the preferred policy as policy that satisfies the constraint (i.e. high-return-lowwii the constraint (i.e. high-return-high-ooe behavior). This experimental setting representative of many real-world scenarios, where the data is collected by high-return behavior without consideration of the cost. Then, we generated scarce but labeled non-preferred demonstrations DN from the non- -oneprefered policy, and abundant but unlabeled demonstrations D from both preferred and non- preferred policies. We collected diverse trajectories by sampling with a stochastic policy from randomly initialized initial states. Our goal is to recover the low-cost behavior by avoiding the high-cost demonstrations while not hurting the high-return behavior of unlabeled demonstrations. The details for the experimental settings can be found in Appendix D
behavior) and the non-preferred policy as policy that violates
7

---

BC
DWBC Cartpole (Cost)
PPL
DExperts
SafeDICE (ours)
Cartpole (Retur))
Cartpole (Cost Violation)
Cartpole (CVaR Cost)
Oliit 10% Cooe
Normalzed
0.3M
3.3M
2.0M
0.6M
1.7M
5.0M
0.3M
0.6M
0.6M
1.2M
0.3M
0.7M
C
3.M
1:3M
0.0M
0.3M
CAM
0AWN
1.50
Walker (Return)
Walker (Co sit
Walker (Cost Viclation)
Walker (CVaR Cost)
118
Rote
CWAR 10% Core
Normazod
14
Cost
0.3M
0.9M
1:3M
0.0N
0.30
0.0M
0.8M
3.3M
0.9
2.3M
0.0N
0.3%
20N
0
1.20
Quadruped (Return)
Quadruped (Cost)
Quadruped (Cost Violation)
Quadruped (CVsR Cost)
CVAR 10% Coos
Nomral
12-
1.20
Training Darrations
Training ita Eurations
Training Datations
Training Hla atiinoo
Experimental results on RWRL control tasks. All results are averaged over runs, and the shaded area represents the standard error.
Figure 2:
Main results Figure summarizes the overall performance of SafeDICE and baseline algorithms in RWRL tasks. The results show that SafeDICE significantly outperforms baseline algorithms in all metries related to safety across all domains (except for CV&R 10% Cost in the Cartpole domain), while not hurting the performance of normalized return. In the case of the Cartpole domain, since there are cases where the cost is unavoidably violated depending on the initial position of cart that is initialized randomly, CVaR 10% Cost which measures the worst 10% performance shows similar performance to other algorithms. We have also evaluated the CVaR 1% Cost with various k and are provided in Appendix I PPL. outperforms BC on Walker and Quadruped domains but suffers from degeneration issues the Cartpole domain. DWBC outperforms matches the other baseline algorithms on all domains, but performs poorly compared to SafeDICE despite the best results obtained through the additional hyperparameter search. Similar to DWBC, DExperts shows results that are sensitive to hyperparameters, and are vulnerable to degeneration issues depending on hyperparameters. Additional experimental results according to various hyperparameters for DWBC and DExperts are provided in Appendix
Results with varying amount of N To study the dependence the amount of labeled non-preferred demonstrations, we experiment with different numbers of labeled non- non-preferred demonstrations |D*| on Walker domain. (The results performance of SafeDICE and baseline algorithms with varying the number of labeled non-preferred demonstrations ( (50, 20. 10, 5)). In this experiment, it is expected that performance in terms of safety gradually degrades as D decreases. As shown in Figure & as |DN] decreases, SafeDICE shows a greater difference from other baseline algorithms, significantly outperforming. PPL works similarly to SafeDICE when D is large but shows sensitive performance depending on |DN DWBC is not sensitive to | but it performs poorly compared to SafeDICE. DExperts easily suffers from degeneration issues when DV is small and performs poorly even when DV is large. These results represent that our algorithm SafeDICE can be applied more effectively real-world scenarios where labeled non- -oe-frrfr demonstrations are not provided in large quantities.
provided in Appendix E.33 Figure g summarizes the
5.2 Safety Gym Environment
To show that our algorithm robustly works in various environments, additionally conduct experi- ments on Safety G2m which is one of the representative benchmarks for safety in RL.
Task setup We conduct the experiments for Goal and Button tasks from the Safety Gym environment. For each task, the agent aims to maximize task performance while satisfying the constraints by avoiding the dangerous area (see Figure Since there is no standard dataset for offline IL considering
8

---

Goal
Button
Figure 3: Tasks of Safety Gym environment. For each task, the agent (red robot) aims achieve the objective while satisfying the cost constraints. The agents receive costs when entering hazards indicated by blue circles or when touching the obstacles. The objectives of each task are as follows: Goal: Move to series of green goal positions Button: Press a series of highlighted goal buttons.
BC
DWBC
PPL
DExperts
SafeDICE (ours)
Goal (Retur)
Goal (Cost)
Gcal (Cast Viclation)
Goal (CVaR Cost)
100-
03
Mormalzedd
Aveeng
CloR 10%
0.4
2.0H
5.2M
0.4M 6NNN
1.39
5.5M
0.4N Button (Cost)
DN
0.0 1.0M
0.0M
5.4M C 0.6N Button (Cost Violation)
0.2M DAN
LOv
Button (Retum)
Button (CVaR Cost)
100
Nomalzzed Nuturn
Meserng Cos
0.
0.4
2.3H 2.2M
5.4M
1.3M
5.5H
5.2N
0.4N
ONNN 0MM
1.0M
5.4M 5.6N 0.6N
0.0% 0.2N
CAN
1dv
Figure 4: Experimental results on Safety Gym control tasks. All results are averaged over 5 runs, and the shaded area represents the standard error.
the safety constraints, collected data by training the online RL agents. To obtain non-preferred and preferred policies, we trained PPO (2 and PPO-Lagrangian [24), which is combined with the Lagrangian approach to PPO, respectively. Here, we define the non-preferred policy policy that violates the constraint and the preferred policy as a policy that satisfies the constraint. Then, we generated scarce but labeled non-preferred demonstrations D from the PPO-Lagrangian policy, and abundant but unlabeled demonstrations D from both PPO and PPO-Lagrangian policies. We collected diverse trajectories by sampling with stochastic policy from randomly initialized initial states. In this setting, our goal is to learn safe policy (i.e. preferred policy) that satisfies the cost constraints by avoiding the violatilg (i.e. non- prefererrrddd demonstrations. The details for the experimental settings can be found in Appendix [
Results Figure summarizzs the overall performance of SafeDICE and baseline algorithms in Safety Gym tasks. The results show that SafeDICE significantly outperforms baseline algorithms in all metrics related to safety across all domains. In the result of the Button task, SafeDICE. shows low performance in Return, but it is desirable result because the non-preferred demonstrations were collected from safe policy with relatively low returns and very low costs (see Appendix D).
6 Conclusion
In this work, we presented SafeDICE, an offline safe IL algorithm, which can learn safe policy by leveraging the non-preferred demonstrations the space of stationary distributions. We formulated this problem as stationary distribution matching problem, which can be reduced to single convex minimization problem without additional hyperparameter search, unlike existing algorithms that are highly sensitive to the hyperparameter. In the experiments, we demonstrated that SaCE learns a more safe policy that satisfies the cost constraint constrained RL benchmarks including RWRL and Safety Gym. As the future work, we will consider an extension to the problem settings where preference information is given as a real value rather than binary value. There are no potential negative socictal impacts of our work.

---

References
[1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-483, 2009.
[2] S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[3] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[4] D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. CoRR, 0bs//994.0033) 2019.
06387
[5] D. Brown, W. Goo, and Niekum. Ranking-based reward extrapolation without rankings. CoRR, abs//1907.0397, 2019. URL.atp:///rxxxv.or/////aa//
03976
[6] L. Chen, R. R. Paleja, and M. C. Gombolay. Learning from suboptimal demonstration via self-supe reward regression. CoRR, abs/2010.11723, 2020. URL https::///// abs/2010. 11723
[7] B. Dai, O. Nachum, Y. Chow, L. Li, C. Szepesvari, and D. Schuurmans. CoinDICE: Off- policy confidence interval estimation. Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9398-944 9411, 2020.
[8] G. Dulac-Amold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning. CoRR, abs/2003. 1881, 2020. BtLphtts: wtwp/a////aa......////////////.e 2003. 11881
[9] C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 213-220, 2008
[10] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/180) 801.09290, 2018. URL.
http://axxii..or/bbb/////// wwwpg//g///../oo//o/og//o 1101110111111 ab00000...1190 01290
[11] I.. Ke, Choudhury, M. Barnes, W. Sun, Lee, and Srinivasa. Imitation learning as f-divergence mininization. In International Workshop the Algorithmic Foundations of Robotics, pages 313-329. Springer, 2020.
[12] G.-H. Kim J. Lee*, Y. Jang, H. Yang, and K.-E. Kim. Lobsdice: Offline learning from obser- vation via stationary distribution correction estimation. In Proceedings of Neural Information Processing Systems (NeurIPS), 12 2022.
[13] G.-H. Kim, Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022. URL httphippp.//ppeeveeee..eete net/forum? id=BrPdX1bDZkQ
[14] I. Kostrikov, O. Nachum, and Tomppsn. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations (ICLR), 2020.
[15] J. Lee, W. Jeon, B.-J. Lee, J. Pineau, and K.-E. Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning (ICML), 2021.
[16] Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations (ICLR), 2022. URI. https:ttp//. htttps://arxiy org/ htg//bs//00 2204 08957
10

---

[17] K. Lee, L. M. Smith, and P. Abbeel. Pebble: eeedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pages 6152-6163. PMLR, 2021.
[18] K. Lee, L. M. Smith, A. D. Dragan, and P. Abbeel. B-pref: Benchmarking preference-bbbeeeed reinforcement learning. CoRR, abs/2111.03026, 2021. URL http::///aax....og//bbbbsss/ 03026
[19] A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi. DExperts: Decoding-tine time controlled text generation with experts and anti anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 66916706, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/ 1/2022 lcl-- 522. /ttpss://///acaaannttt
wwwpoggg....rggeeeeee 20g/2021.200 acl -ong..ln..555. 522
[20] Y. Ma, A. Shen, D. Jayaraman, and O. Bastani. Versatile offline imitation from observations and examples via regularized state-oceupancy matching. In International Conference on Machine Learning, pages 14639-14660 14663. PMLR, 2022.
[21] O. Nachum, Y. Chow, B. Dai, and L. Li. DualDICE: Behavior-agnosto estimation of discounted stationary distribution corrections. In Advances Neural Information Processing Systems (NeurIPS), 2019.
[22] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. CoRR, abs/1912.02074, 2019. URL http://arxiv/ org// abs/1912.0200 02074
[23] A. B. Owen. Monte Carlo theory, methods and examples. 2013.
[24] A. Ray, J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01700 7(1):2, 2019.
[25] S. Ross, G. Gordon, and J. A. Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635, 2011.
[26] S. Schaal et al. Learning from demonstration. Advances in neural information processing systems (NeurIPS), pages 1040-1046, 1997.
[27] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1rxiv: 2017.
[28] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, 1998.
[29] Y.-H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning, pages 6818 0327. PMLR, 2019.
[30] D. Xu and M. Denil. Positive-umlabeled reward learning. CoRR, abs/1911.00453 2019. URL http://arxiv wwwp/rrg///aaoeeeeeeeeee http//911..000 00459
http///s/11111....
[31] H. Xu, X. Zhan, H. Yin, and H. Qin. Discriminator-weigheee offline imitation learning from suboptimal demonstrations. In K. Chaudhuri, S. Jegelka, I. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 24725-24742. PMLR, 17-23 haattps://proeett
Jul 2022.
mlr (itess/v122/u22... hhmll
[32] M. Yang, O. Nachum, B. Dai, L. Li, and D. Schuurmans. Off-policy evaluation via the regularized lagrangian. In Advances Neural Information Processing Systems (NeurIPS), 2020.
[33] R. Zhang, B. Dai, Li, and D. Schuurmans. GenDICE: Generalized offline estimation of sta- tionary values. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
11

---

[34] S. Zhang, B. Liu, and S. Whiteson. GradientDICE: Rethinking generalized offline estimation of stationary values. Proceedings of the 35th International Conference on Machine Learning (ICML), 2020.
[35] S. Zhang, Z. Cao, D. Sadigh, and Y. Sui. Confidence-aware imitation learning from demon- strations with varying optimality. CoRR, abs/2110.14754, 2021. httpss/:://arx org/ abs/2110 14754
[36] K. Zolna, C. Saharia, L. Boussioux, D. Y. Hui, M. Chevalier D. Bahdanau, and
Y. Bengio. Combating false negatives in adversarial imitation learning, CoRR, abs/2002.00412. 2020. URL URLhttps: //t///aavvv////////v///////aaav/ orgo arg.abs//0022.00 00412
12

---

A Theoretical Analysis
A.1 Closed-form solutions
Proposition 3.1. For any v, the closed-form solution the inner maximization of Eq. 11 i.e. wr arg wax L (w, r), is given by:
w((s cxp((A((s)) - 1).
(12)
Proof. For (s, a) with (s,a) > O, we can compute the derivative w(8 a) as follows:
al(w.rr) of ((w,rr) w.r.t. (s(s.a)
al(w,v:T)
Σ do Au(s, - log + w(s w(s,a)
@w(s,a)
(8,0)
Ap(s, a) log w(s, a) - 1 =
+ logw(s,0) 0) = (s( -1
+ w(s,a) exp (8, 1).
A.2 Theoretical analysis of error bound between dl and d'
d (a.a) Proposition 3.2. Suppose min 2 (s.a) S e. If we obtain the estimated stationary distribution by
optimizing the Eq. (7) with set to â as follows:
d (s,
à min (s.a) dt (s,
(15)
then the total variation distance of true and estimated stationary distribution of preferred policy is bounded as follows:
Drv (s, d'))) S I+E'
(16)
where Drv is total variation distance, ds(s,a) is true stationary distribution of the preferred
policy and d (s, is estimated stationary distribution which is defined by â d' d'(ss a)
d (s. (s,a) (Proof in Appendix 1-a
àd
Proof. Let, a is a true ratio between the stationary distribution of expert and anti-expert policy:
(s,a))= = a²) ad
"d (8. a² d (s)
(18)
Then, å can be represented by using a* as follows:
d (s.
min
(19)
(e.0) d' (8,
(1- a a a) o +
min (f.0)
(20)
d (s) d' (x,aa
= =a + (1 - a") min
(21)
(5.a)
d (s,a)
13

---

(s.a,) then the value inside the log function in The estimated stationary distribution de with å can be derived as follows: d' (8, d' d' a) - ānd (s.0)
(22)
1 1 du (s,aa= - (a* + (1 a²) (s.a) a (2,0)
df (s.a)
)d (s,
(23)
1- + (1 a*)) (..))
(s.a)
(x.a)
(1 - a')d"(ss (s, - (1 - a²) min (..))
d (a.a)
a)
d (s,a)
(24)
a-)+) + (1 ax)) (s,))
(f.a)
(s.0)
d (s,a) (s.o) d (e,a) d= (s.a)
2 (s,a)
(25)
(=,0) (s,a) [s,a)
1 + min(s.a)
(8.0) 2 min(s d' (x.a) (d (s, a) (s) + (s)
= d (s,
(26)
(P.a.)
1+ (i, (3.0)
(8) (i.)) d (x.0) (d (s, a) + (s,a)) (d + (s, (d) d (s,
as
(s,a)
d'(s.o)
1 1 min(s,a) (F.a
(27)
dل(aa) (d (s, a) + d (s, a)) (s.a) + d (s))))
(1+ + (s,a) (
d'((s (8,
min(s,a) (T.a
(..a
(28)
-s as + d +d (s) d' (s.a)
(29)
+ min(s,) as (8.a)
Finally, we can derive the total variation distance of the true and estimated stationary distribution of the expert policy:
1
Drv dP) = 2
Ild - -d8111
(30)
d ds + d ddd((ss (8, a)
Σ
d" (s) + (s,a)
(31)
2
1 + min (s,a) ग.. (,,00
(x,a)
d' (s + d (8,a) a)
dd (s( a) + d ())
(32)
2
1 min(),a) (3.0
dF.aa))
(s.a)
(a.a)
minl (s.a) d (s.0)
Σ (s, ((d(sa)))
(33)
1 + mi.()
dP [s.a]
a (s.a)
(9.0)
(a
min(
(3.0)
(34)
(s.a)
+ min
7 (x.a)
< 6
(35)
1+6
A.3 Proof that the reward function well defined with selected a
Proposition A.1. we select a as â min(s (..0) Eq. (T0) is always non-negative:
1 - (1+
-1a)(_c((ssi)) 0,
14

---

where (s) a d (s.0)
Proof. From the a selection rule, á S d (s.a) for all (s,a). We can reformulate this inequation (s,a) as d' (s, a) - ad (s, a) 2 O. Then the value inside the log function in Eq. (1O) can be derived as follows:
d (s.a) (d (s,a)
1 (1 (1+
1 1 (1 a
(1 - a1 (8)
(1- (s,a))aad (8,0)
d (8.a)
d (s.a)
1 1- ()) (s,a)td (8,0)
(1- a) (o(aa)) (s,a)
(U(aaa)
d (s) + dd (s, - (1 + aa+ (s
(1 a)dd
d (s. ad (s,a)
(1 a)d (s, a)
d² (s. ad (8,a)
a) a)
2
To avoid the case where this value is 0, in practical implementation, we select the a as & (e,o) where the is small positive value (e 10--).
(s.) (s.a]
15

---

B Pseudocode of SafeDICE
Algorithm SafeDICE
Input: A labeled non-pre peef-rrffeedd demonstrations D an unlabeled demonstrations D a discrimina- tor Cs with parameter 0, neural network Ve with parameter Ø. policy network T with parameter
1: Pretrain the discriminator by optimizing the Eq. (
fr arg max Ex [log (o a)] + E [log(1 ca a))l
2: Set the a as á from Eq. 15 on D
co- (s. -1)
min (s
3: for each iteration do
4: Update Us by optimizing the Eq. (13
min (1 Ep (s) + Eg
exp(Ar, a) -1)
5: Update policy Te by weighted behavior cloning of unlabeled demonstrations DV: min (s,)) [w (s, a) log (a]s)]
6: end for Output: Te. n²
C
Implementation Details of SafeDICE and Baseline Algorithms
In this section, we provide the implementation details SafeDICE and baseline algorithms. Our code is available on https://github. cc
C.1 SafeDICE
We implement SafeDICE based the Real-World RL (RWRL) suite [ and codebase of DemoDICE 13, which is one of recent DICE-based offline II. algorithms. Similar to DemoDICE [33, due to the instability issue of Eq. (33), implemented alternative objective that is more numerically stable while having the same optimal solution:
min L(w w)r) =(1 Em [v()) + logE," exp (A,(s,
(36)
Then, we estimate Eq. 14 using self-normalized importance sampling 23 as follows:
min B(s.a)=D= [Wpr (s, a) log (als)]
(37)
(( (ū0- (8, a)]
where 8* is optimal solution of Eq. 36 and uz- cxp(AA- (s, a)).
C.2 Diseriminator-Weighhee Behavior Cloning (DWBC)
We modified DWBC, which adopted positive-unlabeled learning, considered as baseline algorithm in our experimental settings. Similar to positive-umlabeled learning, we adopted negative- unlabeled learning the objective of discriminator learning as follows:
min [(e.a) log do (s, log m)]
+ +E +B(a)) (- log(1 - do log ]]]
(())) [- log(1 (s a, log l)g
(38)
16

---

where dg is discriminator with parameter 8, and is hyperparameter that corresponds to the proportion of negative samples unlabeled samples.
Based on the trained discriminator by Eq. 38). we adopted weighted behavior cloning on the unlabeled demonstrations for the policy learning as follows:
min - (( [[1 do(s, a)) log (a)s)(
(39)
C.3 Preference-based Policy Learning (PPL)
We trained the reward function with the following loss function follows from the Bradley-Terry model 3 which is commonly used in preference-based policy learning approaches [8.18]:
exp
E(s,a)er: Ro(s,a)
min
Σ log (.JE)
(40)
E(s.))en Rg( + exp E(saa)er, Ra(
exp
where 7r and Tj represent trajectories, and P= [(,,]) Tú > Ty). our problem setting, we adopt this model assuming that unlabeled demonstrations are better than anti-expert demonstrations (i.e. Ty > Tj for all E D and T E D
Based on the trained reward model by Eq. (40), we adopt weighted behavior cloning on the unlabeled demonstrations for the policy learning as follows:
min E (e.a) [Ro(s, log (a(ss))
(41)
C.4 DExperts
Tabular MDPs For the tabular MDPs, we separately trained unlabeled policy x and non- preferred policy π with behavior cloning model using unlabeled demonstrations and non-preferred demon- strations as follows:
min -B(r,)) [log " (a|s)),
(42)
min - -E [l (a|s)).
(43)
Then, as the final policy, the ensemble of Th and π at decoding- time is used:
T(a = - (als) (als) an (als),
(44)
where a is hyperparameter that controls the degree of leveraging the non-preferred policy.
Continuous MDPs For the continuous MDPs. it is not straightforward to apply DExperts decoding-time method. Therefore, we implemented DExperts with the following objective that is similarly considered training time:
min - (e [log o. (a]s)] + a (( [log (c]s)),
(45)
where a is hyperparameter that controls the degree leveraging the non-preferred demonstrations.
17

---

D Experimental Details
D.1 Experimental details for RWRL suite
Task specification We conduct experiments on the RWRL suite with safety constraints. Table shows the tasks, safety constraint settings, and the information of dataset for each domain that used in our experiments.
[
Task specification
Cartpole
Walker
Quadruped
name the used task
realworld_se swiinup
realworld walk
realworld_ walk wel.vvlootty_cossttaeee
name of the used safety constraint
slider_pos_ poo_ssnnttaintt
safety cocfficient
0.2
0.2
0.35
# of unlabeled demonstrations
5000
2000
2000
# of labeled non- oon-prefrred demonstrations
80
50
mean cost of preferred demonstrations
71.95
56.14
97.01
mean cost of non- pon-prrfrred demonstrations
316.75
377.58
323.60
mean return of preferred demonstrations
516.28
939.74
991.11
mean return of non prefferrdd demonstrations
519.82
940.06
989.59
Table Task specification of each domain used in our experimental results on RWRL environment.
peprraarreeeer configurations For fair comparison, ing rate to train the policy networks, discriminators, and reward models each algorithm. Ta- ble 2 summarizes the hyperparameter settings that used in experiments. The additional hyperparameters v) and of DWBC and DExperts were selected by the best performance among (0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.6, 0.83.
the same architecture and Iearn-
Hyperparameters
BC
DWBC
PPL
DExperts
SafeDICE
y (discount factor)
0.99
0.99
0.99
0.99
0.99
learning rate (actor)
3 x 10-4 [256, 256]
3 x 10-4 [256, 256]
3 x 10 [256, 256]
3 x 10-4 [256, 256]
3 x 10 [256, 256]
network size (actor)
learning rate (critic)
3 x 10-4
network size (critic)
[256. 256]
learning rate (diseriminator reward model)
3 x 10-1 [256,256]
x 10-4 [256,256]
3 10-1
network size (discriminator reward model) gradient penalty coefficient (discriminator)
[256, 256] 0.1
0.1
batch size # of training iterations
512 1,200,000
512 1,200,000
512 1,200,000
512 1,200,000
512 1,200,000
Table 2: Configurations of hyperparameters used in our experimental results on RWRL environment.
18

---

D.2 Experimental details for Safety Gym
Task specification We conduct experiments on the Safety Gym with safety constraints. Table shows the tasks, safety constraint settings, and the information of dataset for each domain that we used in our experiments.
Task specification
Goal
Button
type of robot
Point
Point
level of difficulty
1
# of unlabeled demonstrations
2000
2000
# of labeled non-preee non-preferred demonstrations
50
50
mean cost of preferred demonstrations
13.798
12.085
mean cost of non-preferred demonstrations
107.977
166.099
mean return of preferred demonstrations mean return of non-preferred demonstrations
19.911
8.286
20.018
21.933
Table 3: Task specification of each domain used in our experimental results on Safety Gym environ- ment.
Hyperparameter configurations For fair comparison, we use the same architecture and learn- ing rate to train the policy networks, discriminators, and reward models each algorithm. Ta- ble 4 summarizes the hyperparameter settings that we used in our experiments. The additional hyperparameters 7 and of DWBC and DExperts were selected by the best performance among {0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.8.
Hyperparameters
BC
DWBC
PPL
DExperts
SafeDICE
7 (discount factor)
0.99
0.99
0.99
0.99
0.99
learning rate (actor)
3 x 10 [256, 256]
3 x 10-4 [256, 256]
3 x 10 [256, 256]
3 x 10 [256, 256]
3 x 10 [256, 256]
network size (actor)
learning rate (critic)
3 x 10-4
network size (critic)
[256, 256]
learning rate (diseriminator/ reward model)
3 10-1 [256,256)
x 10-4 [256,256]
3 10-1
network size (discriminator reward model) gradient penalty coefficient (diseriminator)
[256, 256] 0.1
0.1
batch size # of training iterations
512
512 1,000,000
512 1,000,000
512 1,000,000
512
1,000,000
1,000,000
Table 4: Configurations of hyperparameters used our experimental results on Safety Gym environ- ment.
19

---

E Additional Results on RWRL suite
E.1 Results of CVaR %% Cost on RWRL suite
DExperts
(ours)
Ca cappll (CVaR 50% Cost)
Cartpolc (CVaR 20% Cost)
Cartpola (CVa 10% Cost)
Cartpolc (CVaR 5% Cast)
Cliit 10% Coll
20%
CVOR 5%
3.3M
6.6H
0.0M
0.6
3M
0.0M
0.30
0.3M
1.2M
0.0M
0.35
3:MM
0.90
1.20
Walker (CVoR 50% Cost)
Wal ker (CVaR 20% Cost)
Walker (CVaR 10% Cost)
Walleer (CVoR 5% Cost)
Cost.
Elen 10% Cost
CLoi 5% Cost
Quadruped
Quadruped 20%
Quadruped
CHAR 10% Cont
Cook
0.054
0.3H
5.0N
0.0N
1.20
0.3M
2.3M
0.6MM
1.2M
0.0N
0.34
0.8M
0.3M
2.2M
0.0%
0.354
5.0H
0.5N
120
Training Boradoos
Training Derations
Traiting iterations
Training Kerations
Figure 5: Additional experimental results of Figure 2 on metrics of CVaR 1% Cost.
20

---

E.2 Results of DWBC and DExperts with various hyperparamettrs on RWRL Walker domain
To study the hyperparameter sensitivity of DWBC and DExperts, we demonstrate the results of DWBC and DExperts with various hyperparameter settings. For DWBC, we compare the performance of
DWBC with different values of E {0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8] and DWBC without (i.e. naive discriminator training without Segative-unlaheles learning). As shown in Figure DWBC shows very sensitive performance to hyperparameter y and unstable learning that sometimes even suffers from degeneration issues.
DWBCOw/e f) DWBC 0.2)
DWBCI 0. 01) DWBC 0.02)
DWBCIn 0.05)
DWBCIn 0.11
DWBCI - 0.4)
DWBC 0.6)
DWBCIn - 0.8)
SafeDICE (ouns)
Violation
800
Rodo
Avergge Cost.
400
0.89
0.5M
0.6M
0.5M
2.3M
0.0M
0.34
0.8ht
0.1
1.20
0.09
0.3V
0.691
0.0M
3.3M
0.3M
0.0M 0.3M
1.24
Training Iberations
Tairing ltoraiion
Training Iterationd
Triniing Itcrations
Figure 6: Additional experimental results of DWBC with various hyperparameters Walker domain.
We also compare the performance of DExperts with different values of a {0, 0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8}. Here, DExperts(a = 0) is reduced to the naive behavior cloning model unlabeled demonstrations. As shown in Figure 7 DExperts shows very sensitive performance to hyperparameter a and is vulnerable to degeneration issues.
DExperts/a=01 DExpertsta 0.221 0.21
DExpertsla= 0.011 DExpertsta 0.4)
DExpertsia=
DExperts(a= DExpertsla 0.8)
DExpertsla
DExpertsig
SafeDICE (ours)
Return
Cost
Cost Viclation
CVaR Cost
800
800
Cops Villting Radio
Iveegge Coft
500
400
400
200
1.00
Training Derrations
Taining Ittrations
Triniin beratioon
Triniing itorations
Figure Additional experimental results DExperts with various hyperparameters on Walker
domain.
21

---

E.3 Results with varying amount of labeled non non-preferred demonstrations
DWBC
DExperts
SafeDICE (ours)
[|0"|=
(|0*)
Cost Vielation
Cost [|D*
Aveege C2P
600
Nommazzd
100
460
200
0.0M
0:30
6NNN
0.48
1.2M
0M0M
0.3M
0.6M
1.20
0.0M
0.30
0.40
1.2M
0.3M
0.84
0.8M
1.20
-|D"|=
20)
(I0*)
[|D*|=20
Avenge Cook
Molain
100
400-
Nora
200
0.0M
0.3M
0.00
3.3M
9..0
0.69
0.6
1.20
0.0M
0.30
2.6M
0.60
1.2M
5.NM
0.31
0.84
0.8M
1.2M
(I0*)=
10)
(I0*)
10)
(
CVaR Cost (D =00
Ieenge Cost
Wilating
660
Normaited
1
300
200
0.00M
0.0M
1.20
Retum (
(ID") =5)
(ID*)
(D")
Avergge Cost
Waazing
Normaized
200
200
0.00
0:31
0.0M
0.6
5.MM
1.20
Thsining Berratoons
Triiiin Iterttions
Thiining Ieerations
Triining Iterations
Figure 8: Experimental results RWRL Walker domain with different numbers of labeled non- preferred demonstrations (IDN) (50, 20, 10, 5)). All results are averaged over 5 runs, and the shaded area represents the standard error.
BC
DWBC
PPL
DExperts
SafeDICE (ours)
CVa.R 50% Cost (ID") - 50)
CVaR 20% Cost (I=) [101=-50 50)
CVaR 10% Cost (jp") -501
CVVR 5% Cost (ID*) 50)
Cost
Clin 10% Coe
CloR 50%
600
20%
600
830
5%
600
500
000
200
230
3.3M
0.2M
0.0M
3.3M
0.3M
3.5M
0.3M
0.2
2.2M
3.0
0.6M 0.N
CVVR 50% Cast
(ID*) - 20)
CVaR 20%
Cost (ID*) 20)
CVaR 10% Cost (ID -00
CVOR 5% Cost ((D)) 20)
CCRR Cost
800
Cla 20% Cost
800
CARR 1O% Cont
400
5% Cost
600
600
630
600
600
OBR
000
200
220
2.3M
0.3M
0.0M
0.5N
3.3M
0.34
3.5M
0.M
1.3M
1.2M
2.0M
0.6%
6.5M
CVaR 50%
Cost
(ID*) = 10)
CVaR 20%
Cost (I0)) 1000-1000 10)
OVaR 10% Cost (I0*)
101
CVaR, 5% Cost (jst 10)
800
800
CloR 10% Core
CORR 5% Coo
800
20%
800
$30
600
600
330
000
200
230
0.3M
0.6M
6.6N
3.5M
0.3M
3.MH
6.M
1.3M
0.3M
3.2M
3.M
0.60
6.6N
CVVR 50% Cost (ID*) 5)
CVaR 20% Cost (ID*) 5)
CVaR 10% Cost Cost 55)
CVaR 5% Cost (ID 5)
600
200
CieR 10% Coe
920
CBR 5% Cos
800
20%
500
600-
200
200
230
0.3N
0.5M
3.5M
0.3M
3.EM
0.MM
1.3M
0.31
3.3M
6.5M
1.30
Thining Ieertions
Trannnng heerations
Training Ineesatoon
Training Iterations
Figure 9: Additional experimental results of Figure g on metrics of CVaR 1.% Cost.
22

---

E.4 Results of SafeDICE with various a
To show the importance of theoretically determining a, we show additional results of SafeDICE with various values of a. According to the value of a. the reward function (s, a) defined by Eq. (T)) cannot guarantee that the value inside the log function always becomes a positive. Therefore, we simply performed learning with a reward function that artificially clipped the value inside the log function of Eq. (TO). As shown in Figure and T1 the results show various performance depending on the value of at and result with a value selected by Eq. (15) shows the best performance.
SafeDICE(s- 0)
Sa
SafeDICE(a
Safe 0.05)
SafeDICE(e 0.11
SafeDICElg 0.21
Safe 0.4)
SafeDiCElg -...DiEEl-..)))
SafeDICE 0.8)
SafeDICE (ours)
Return
Cest
Cost Violation
CVeR Cost
800
60--
Avernge Cost.
500
200
220
1.50
Training Itcrations
Itorations
Training Iberations
Triniin Itorations
Figure 10: Additional experimental results of SafeDICE with various hyperparameters on Walker domain.
SafeDICE(g- 0)
SafaDICE 0.01)
SafeDiCElg SafeDICEla
Ca 0.05)
SafeDiCE(a 00.1)
SafeDICEig
SafeDICE(z
SafeDiCE(g
(ours)
CVaR 50% Cost
CVaR 20% Cest
CVaR 10% Cost
CVoR 5% Cost
5:0
Niver Weergge Coit.
602
Contitteall mng
202
0.84
0,30
5.HH
3.3M
1.2M
0.0%
0.3%
2.M
6:N
1.2M
0.0N
3.3M
0.6M
0.M
1.29
5.0M
0,31
244
0.3M
1.20
Training
Training Beeaaooos
Tranning
Training
Figure 11: Additional experimental results of SafeDICE with various hyperparameters on Walker domain.
E.5 Impact of the stationary distribution constraint and labeled non-preferred demonstrations
We conducted additional experiments to evaluate the impact of the stationary distribution constraint and labeled non non-pre -preferrdd demonstrations, As can be from the Figure [2] the performance is degraded both in the absence of stationary distribution constraints and in the case of not using labeled non-preferred demonstrations.
SafeDICE (without laboled non preferred) Walker (Return)
SafeDICE (withaut staticnary distribution constraint)
SafeDICE Walker (CVaR Cost)
Walker (Cost)
Walker (Cost Violation)
Rade
CWAR 10% Cort
Training Barrations
Training Har ioms
Training Dataaions
Training Har al
Experimental results on RWRL Walker domain without stationary distribution constraint and labeled non- -re-rrrfrrred demonstrations. All results are averaged over 5 independent runs, and the shaded area represents the standard error.
Figure 12:
23

---

E.6 Results with various configurations of unlabeled demonstrations
In order to evaluate our algorithm with more various configurations of unlabeled demonstrations, we also conducted additional experiments for various configurations of unlabeled demonstra- tions consisting of different ratios of preferred and non-preferred demonstrations ((D), D
{(250, 1000), (500, 1000), (1000, 500), (1000, 250)}. where |D and | denote the number of preferred and non-preferred demonstrations in D respectively). As can be seen from the Figure 13 the results show that SafeDICE outperforms baselines in all settings with various configurations of unlabeled demonstrations. These results show that SafeDICE can be applied more effectively in various real-world scenarios regardless of the preferred and non non- non-preferred ratio of unlabeled demonstrations.
DWBC
PPL
DExperts
SafeDICE (ours)
Return
Gost
Cost Viclation
CVaR Cost [ 10 250,1000)
( (D9) 250.1000)
(IDE), =
250,1000)
(D)), (D 250,1000)
800
Avergge Cost.
600
920
400
200
3.4H 3.5M
2.0M
0.6N
0.0M 0.2M 0.4M 3.5M 2.8M
2.0M
0.2M
D.4N
Return
Cost
Cost Vielation
CVaR Cost
((DE), [D]] - 500.1000)
([DE). [0[] 500,1000)
((D)). (DX) = 500,1000
[ [02] - 500,
800
Avenge Cost
800
Normaltadd
400
200
200
tion 1000,500)
(108%) [D]] 100),500)
(PP6).10] 1000,500)
(1D)), (2)]
([DE') [0 1000,500)
Volaing Rado
800
Avcrige Cost.
400
200
200
0.2M 0.4H
3.0M
6.0M
0.6N
0.60
1.00
0.8M
3. 5:6M
3.0M
*5.M
6.7M
6:dN
0.iN
0.6N
1.
Viclation
CV3R
(1DP), [D - 2500
[D - 1000.250)
(1DE), (D
1500.250)
(JDE)) [0
Averag Cose.
500
Normazodd
12-
Virsinng
660
800-
200
0.0M 0.2M 3.4H 7.5H 2,8M
2.0M
0.8M
0.2M D.AN DAN 0.ON
1.N
0.39 0.2M 0.4M 3.5M 5,N 3.0M
0:3N 0.4M DAN 0.N 100
Training Derations
Trairing Rorations
Training Iterations
Trairing Itarations
Figure 13: Experimental results on RWRL Walker domain with different numbers of pre- ferred and non-preferred demonstrations in unlabeled demonstrations D D and [D denote the number of preferred and non-preferred demonstrations in D ((D), {2200, 1000), (500, 1000). (1000, 500), (1000, 250) }). All results are averaged over 5 independent runs, and the shaded area represents the standard error.
24

---

Comparison with constrained RL algorithm
To compare with the safe RL algorithm as oracle agent, we conducted additional experiments for an fifline-constrained RL method. We run COptiDICE [16], the state-of-the-a offline constrained RL algorithm, using the datasets that are augmented with additional (ground-truth reward and cost annotations. Table 5 summarize overall performance of SafeDICE and baseline algorithms including COptiDICE. The results show that SafeDICE is competitive with (or even outperforms in some domains) the constrained offline RL algorithm COptiDICE, even though SafeDICE uses only much smaller amount of annotation information.
F.1 Experimental results on RWRL control tasks
CARTPOLE
Normalized Return
Average Cost
Cost Violation
CVaR 10% Cost
BC
100.24 + 0.40
184.44 + 0.94
0.48 + 0.00
392.75 2.72
DWBC
99.78 - 0.55
189.97 = 3,21
0.47 = 0.00
408.71 + 4.01
PPL
83.70 + 0.87
229.80 + 1.64
0.48 + 0.01
440.68 3,49
DExperts
100.28 = 0.38
186.15 + 2.29
0.48 - 0.01
397.05 + 3.11
SafeDICE
99.91 = 0.60
154.88 + 1.79
0.34 + 0.00
404.22 1.56
COptiDICE
107.95 + 2.10
97.59 + 7.21
0.06 + 0.01
183.63 + 6.18
Table 5: Experimental results on Cartpole domain. All results indicate averages and standard errors
over 5 independent runs.
WALKER
Normalized Return
Average Cost
Cost Violation
CVaR 10% Cost
BC
98.41 + 0.11
152.25 + 5.33
0.22 + 0.01
531.07 +:2 7.53
DWBC
98.99 = 0.10
105.63 + 3.85
0.11 + 0.01
378.01 + 20.88
PPL
99.02 + 0.10
128.46
4.84
0.17 + 0.01
478.75 + 12.53
DExperts
95.21 = 0.75
173.20
11.22
0.27 + 0.03
518.38 + 5.25
SafeDICE
99.67 = 0.13
68.90 0.83
0.00 + 0.00
124.54 + 2.56
COptiDICE
98.07 = 0.34
77.86 2.14
0.01 + 0.00
159.91 + 8.90
Table 6: Experimental results on Walker domain. All results indicate averages and standard errors
over 5 independent runs.
QUADRUPED
Normalized Return
Average Cost
Cost Violation
CVaR 10% Cost
BC
100.19 = 0.08
178.10 = 5.20
0.38 + 0.02
328.10 + 1.83
DWBC
99.87 = 0.16
167.15 = 3.32
0.34 + 0.02
316.86 +/2 1.74
PPL
99.87 + 0.09
169.93 + 5.06
0.35 + 0.03
319.77 + 1.70
DExperts
97.05 = 2.62
178.75 = 4.78
0.38 + 0.03
317.55 + 4.40
SafeDICE
99.68 + 0.25
148.21 + 2.84
0.24 + 0.01
308.71 + 2.90
COptiDICE
87.37 = 3.19
130.21 = 5.20
0.16 + 0.03
263.41 + 17.00
Table 7: Experimental results Quadruped domain. All results indicate averages and standard emros over independent runs.
25

---

F.2
Experimental results on Safety Gym control tasks
GOAL
Normalized Return
Average Cost
Cost Violation
CVaR 10% Cost
BC
94.08 + 0.60
51.47 + 1.49
0.38 + 0.02
109.66 + 1.63
DWBC
93.93 + 0.53
47.96 = 1.77
0.31 + 0.02
108.79 = 2.12
PPL
94.43 + 0.66
52.31 = 1.52
0.37 + 0.02
111.41 + 2.37
DExperts
87.76 = 4.41
50.82 + 1.55
0.36 + 0.02
112.62 = 1.54
SafeDICE
92.04 + 0,44
39.49 + 0.94
0.21 + 0.02
93.35 = 1.45
COptiDICE
92.98 + 1.13
48.79 + 0.80
0.32 + 0.01
107.25 = 0.78
Table 8: Experimental results on Goal task. All results indicate averages and standard errors over independent runs.
BUTTON Normalized Return
Average Cost
Cost Violation
CVaR 10% Cost
BC 88.64 +: 1.08
98.18 + 2.19
0.49 + 0.01
220.87 +: 3.48
DWBC
85.53 + 0.80
96.31 + 2.38
0.47 + 0.02
225.49 + 3.52
PPL
86.76 + 1.06
99.97 + 2.19
0.50 + 0.02
229.46 E 3.30
DExperts
86.28 + 1.35
97.30 + 1.70
0.49 + 0.01
225.93 + 5.34
SafeDICE
71.48 + 1.33
70.67 + 1.61
0.30 + 0.01
185.64 + 1.54
COptiDICE
94.09 + 1.68
111.10 + 3.53
0.58 + 0.02
238.46 + 3.44
Table 9: Experimental results on Button task. All results indicate averages and standard errors over independent runs.
26

---

G Tabular MDPs Results
In order to show the motivation of the paper and the limitation of existing algorithms through illustrative examples, we conducted the experiment in tabular MDPs, which extends the Four Rooms environment 28 22
Non-preferred (D*)
Unlabeled (D")
Preferred (1 a)
Non-preferred (a)
Figure 14: Illustrative examples of experimental settings on Six Rooms environment. We assume
the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations D and abundant but unlabeled demonstrations D from both preferred and non- non-preferred policies.
PPL
SafeDICE
DWBC (v)=).1)
DWBC (m)=.2)
DWBC (7)=0.4)
DWBC (17=0.6)
DWBC (7==088
DExperts (x=0.1)
DExperts (a=0.2) DExperts (a=0..4) DExperts (a=0.6)
DExperts (a=0.8)
Figure 15: Experimental results on Six Room environment. We demonstrate the qualitative results of each algorithm learned using the dataset given Figure [4] For DWBC and DExperts that require additional hyperparameter search, we show the results with various hyperparameter settings. y) and represent hyperparameters that control degree of leveraging the non-preferred demonstrations of DWBC and DExperts, respectively.
Task setup We consider the Six Rooms environment, variant of the Four Rooms environment 28 22), in which an agent aims to navigate to a goal location within a gridworld. Figure [4 demonstrates experimental settings the Six Rooms environment. The orange and green squares denote the initial state and goal state, respectively. The arrows for each action at the square represent the policy, where the opacity of the arrow is determined by the probability <{ass). The opacity of each square represents the state marginals of each stationary distribution. In order to compose the task naturally, we assume that the preferred and non-preferred policies have both similar and different behaviors in some state-action spaces. We define the policy that reaches the goal state from the initial state through the downward direction as an non non-prrfereed policy, which represents constraint-violatte behavior, and the policy that reaches the goal state through the right direction as an preferred policy. In this task, the agent aims to learn the policy that recovers the preferred policy from unlabeled demonstrations while avoiding the constraint-violttinn behavior using labeled non- preferred demonstrations. scarce but labeled non non-preferred demonstrations N and abundant but unlabeled demonstrations
shows a given dataset to the agent for learning, which consists of
27

---

DU from both preferred and non- preferrefeedd policy. The details for the experimental settings can be found in Appendix D
Results Figure 15 summarizes the qualitative results of SafeDICE and baseline algorithms on Six Rooms environment. Naive BC on unlabeled demonstrations follows both preferred and non-preferred behavior since it does not consider avoiding the non-preferred demonstrations. On the other hand, PPL and SafeDICE show safe behaviors in which non-preferred demonstrations are completely avoided while following preferred demonstrations. For DWBC and DExperts that require additional hyperparameter search, we show the behaviors of learned policies with different hyperparameter settings. Among various hyperparameter settings, there are cases where an agent learns policy close an preferred behavior specific hyperparameter. However, the results show that both DWBC and DExperts are hyperparameter sensitive, which is unsuitable for real- world tasks where the deployment is costly or risky. addition, DExperts not only have cost- violatig behaviors but also cause degeneration issues that fall into abnormal behaviors that are completely different from given demonstrations. To show that SafeDICE robustly works various problem settings, we also provide additional results including cases where the distributions of labeled and unlabeled non- preferred demonstrations are dissimilar (see Appendix G.I. and more complex environments (see Appendix G.2). As shown in Appendix G.T SafeDICE learns a safe policy in which non-preferred behaviors are completely avoided while following preferred behavior, even when the distributions of labeled and unlabeled non non-preferred demonstrations dissimilar.
28

---

G.1 Experimental results with the dissimilar distribution of labeled and unlabeled non-preferred demonstrations
To show that SafeDICE works robustly, we conduct the experiment considering problem setting more similar real-world tasks, where the distributions of unlabeled and labeled non-preferred demonstrations are dissimilar. In this experimental setting, we define the non-preferred policy is multimodal policy that consists of mixture two different policies π and A We assume that non-preferred demonstrations of labeled and unlabeled are sampled from policies A and A with different ratios. As shown in Figure [6] baseline algorithms fail to completely recover the preferred behavior follow the non-preferred behavior. On the other hand, SafeDICE shows safe behavior which non-pre con-prefereed behaviors are completely avoided while following preferred behavior, even when the distributions of labeled and unlabeled non-preferred demonstrations are dissimilar.
True preferred (x²")
True non- preferred (1
non-preferred (zr
Non-preferred (D*)
Unlabeled (D")
Preferred (1 a)
Non-preferred (a)
Figure 16: Illustrative examples of additional experimental settings on the tabular environment. We assume the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations D and abundant but unlabeled demonstrations D from both preferred and non-preferred policies. this experiment, we consider more challenging problem setting where the distributions of labeled and unlabeled non- non-preferred demonstrations are dissimilar.
SafeDICE
DWBC (x)-..1)
DWBC (y=0.22
DWBC (7)0..4
DWBC (m=0066)
DWBC (7==.88
DExperts (a=0.1)
DExperts (a=0.22) DExperts (a=0.44)
DExperts (a=0.6)
DExperts (a=0.8)
Figure 17: Experimental results on the tabular environment, We demonstrate the qualitative results of
each algorithm learned using the dataset given in Figure 16
29

---

G.2 Additional results for various Tabular MDPs
Non-preferred (D*)
Unlabeled (D")
Preferred (1 ac)
Non-preferred (a)
Figure 18: Illustrative examples of additional experimental settings on the tabular environment. We
assume the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations D. and abundant but unlabeled demonstrations D from both preferred and non- con-rrferred policies.
BC
PPL
SafeDICE
DWBC (1)=..1)
DWBC (11=0.2)
DWBC (17=0.4)
DWBC (7=00.6
DWBC (7=0.88
DExperts (a=00.1)
DExperts (a=0.22)
DExperts (a=0.44
DExperts (ac=0.6) DExperts (a=0.8)
Figure 19: Experimental results on the tabular environment. We demonstrate the qualitative results of each algorithm learned using the dataset given in Figure [8] For DWBC and DExperts that require additional hyperparameter search, we show the results with various hyperparameter settings. r) and a represent hyperparameters that control the degree of leveraging the non-preferred demonstrations of DWBC and DExperts, respectively.
30

---

Non-preferred (D*)
Unlabeled (D")
Preferred (1 a))
Non-preferred (a)
Figure 20: Illustrative examples of additional experimental settings on the tabular environment. We
assume the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations D and abundant but unlabeled demonstrations D from both preferred and non-preferred policies.
SafeDICE
DWBC (m==..1
DWBC (1==022)
DWBC (r)=0.4)
DWBC (m==0.6)
DWBC (11=0.8)
DExperts (a=0.11 DExperts (a=0.2) DExperts (a=0.4)
DExperts (a<00.6)
DExperts (a=0.8)
Figure 21: Experimental results the tabular environment. We demonstrate the qualitative results of each algorithm learned using the dataset given in Figure 20 For DWBC and DExperts that require additional hyperparameter search, show the results with various hyperparameter settings. r and represent hyperparameters that control the degree of leveraging the non-preferred demonstrations of DWBC and DExperts, respectively.
31


Reviews:
[{"review_id": "DSkjVMkTtz", "reviewer": "Reviewer_kwD7", "summary": "This paper focuses on offline safe imitation learning (IL) setting. There are several unique properties of the problem setting: 1) there exists non-preferred (constraint violated) demonstrations, 2) there exists massive unlabelled data where you don't know whether they are preferred or non-preferred, 3) you don't have reward function and interactive environment.\n\nThe proposed SafeDICE method builds upon DIstribution Correction Estimation (DICE), but addresses several challenges. The first is how to takes non-preferred demonstrations into account? To do so, we assume an underlying unknown mix ratio alpha and the unlabelled dataset is the mixed dataset of non-preferred and preferred demonstrations. Doing such assumption allows us to follow DICE workflow. At the same time, we can use a preference-based reward function to estimate the stationary distribution d(s, a) of a state-action pair. The reward function is similar to those used in inverse RL and can be achieved by a classifier discriminating whether a datapoint is from non-preferred dataset or not. The authors derives a closed-form solution to avoid the training instability and provide an approximation to the unknown mix ratio to avoid hyperparameter search.\n\nExperiments show the proposed method can achieve good performance while greatly reduce the constraint violation in Real-World RL suite and SafetyGym.", "questions": "1. Some ablation studies should be added to support the claim. For example, what if we don't use the approximation of mix ratio alpha but really do a hyperparameter search? \n2. Can we have an experiment/discussion section discuss the impact of labelled non-preferred data? Like, if we have plenty of them and if we have very few those data, what would happen?\n3. Can we compare/discuss to offline RL method where you can have reward function? In offline RL the data is not always preferred as the author discussed for those offline IL method. So I wonder offline RL shares some settings in current problem. When comparing online Safe RL method, a competitive baseline is to use online RL with reward shaping (the constraint violation can be considered as a negative reward). Would this method also be competitive in offline Safe RL setting?", "limitations": "It seems that there is no much effort put in the paper to discuss the limitations. One I can think of is the collection of non-preferred data is in some sense still dangerous. The \"non-preferred data efficiency\" is not discussed in the paper yet. That is, how efficient the proposed method can learn from how much non-preferred data. This is also important since as the authors said collecting those data is costly.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed method is solid, clear and looks sound to me.\n3. The problem setting is novel and important.", "weaknesses": "1. The experiment showcases the superior performance of proposed method. But more can be added to study the particular behavior of the proposed method in different settings, such as when you have lot of / scare non-preferred demonstrations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and comments. Please feel free to ask any additional follow-up questions.\n\n**[Responses to Weaknesses]**\n\n**(1) (Different settings with non-preferred demonstrations)**  \nThank you for your suggestion for the additional experiment varying the amount of **labeled** non-preferred demonstrations. Our paper already provides the experimental results you suggested in Section 5.1 (lines 284-295) and Appendix E.3 (Figures 8 and 9). As can be seen from the results, as the number of labeled non-preferred demonstrations $|D^N|$ decreases, SafeDICE shows a greater difference from other baseline algorithms, significantly outperforming.  \nIn order to evaluate our algorithm with more various configurations of **unlabeled** demonstrations, we also conducted additional experiments for various configurations of unlabeled demonstrations $D^U$ consisting of different ratios of preferred and non-preferred demonstrations ($(|D^U_P|, |D^U_N|) \\in \\[ (1000, 1000), (500, 1000), (1000, 500), (250, 1000), (1000, 250)\\]$, where $|D^U_P|$ and $|D^U_N|$ denote the number of preferred and non-preferred demonstrations in $D^U$, respectively). As can be seen from the uploaded PDF (Figure 1), the results show that SafeDICE outperforms baselines in all settings with various configurations of **unlabeled** demonstrations. These results show that SafeDICE can be applied more effectively in various real-world scenarios regardless of the preferred/non-preferred ratio of unlabeled demonstrations. We will add these results and explanations to the final version of the paper.\n\n**[Responses to Questions]**\n\n**(1) (Experiment with hyperparameter search for $\\alpha$)**  \nOur paper already provides a comparison with the results of performing a hyperparameter search in Appendix E.4 (Figures 10 and 11). As shown in Figures 10 and 11, the results show various performances depending on the value of $\\alpha$, and the result with $\\alpha$ value selected by Eq. (15) shows the best performance.\n\n**(2) (Impact of labeled non-preferred data)**  \nAs mentioned in (Responses to Weaknesses 1), our paper already provides the experimental results you suggested in Section 5.1 (lines 284-295) and Appendix E.3 (Figures 8 and 9). We observed that, for all algorithms, the performance in terms of safety gradually degrades as the number of labeled non-preferred demonstrations $|D^N|$ decreases. However, as $|D^N|$ decreases, SafeDICE shows a greater difference from other baseline algorithms, significantly outperforming.\n\n**(3) (Comparison with offline Safe RL)**  \nPlease note that the offline imitation learning setting we consider in the paper does **not assume labels for both the reward and cost**. We assume only a small number of trajectory-level labeled preferred demonstrations and unlabeled demonstrations. Therefore, it is impossible to directly apply the offline safe RL algorithms. However, we can consider learning a reward function based on preference and then using it to perform offline RL. In the paper, PPL is an offline RL baseline using learned preference-based reward and weighted behavior cloning which is one of the representative offline RL algorithms."}, {"review_id": "E3voNiFNkI", "reviewer": "Reviewer_jY8W", "summary": "This paper proposes an offline safe imitation method to avoid non-preferred demonstrations. The mixture coefficient is alpha, and the paper provides an effective way to obtain hyperparameter instead of hyperparameter search techniques. The paper uses scarce but labeled non-preferred demonstrations from the non-preferred policy.", "questions": "Since the additional cost of hyperparameter search techniques have influence on the computational cost rather than final performance, it is not clear why SafeDICE performs better with the  obtained by Eq. 15. Can you give more explanations about the results between SafeDICE and DWBC?", "limitations": "As stated above, I would like to compare this methods with those robust imitation learning methods in the offline setting.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper models the stationary distributions as a mixture of preferred policy and non-preferred policy. The mixture coefficient is alpha, and the paper provides an effective way to obtain hyperparameter instead of hyperparameter search techniques.\n\nThe paper is easy to follow and understand.\n\nClaims made at the end of the introduction are supported by experimental results.", "weaknesses": "The problem setup is unclear to me. In the offline imitation learning setting, there's a line of research consider robust imitation learning (such as [1]) to discard non-preferred samples. The authors may need to discuss why labeled non-preferred demonstrations is nececssary in this situation.\n\nFurthermore, more discussion can be added to explain why SafeDICE is not sensitive to the number of labeled non-preferred demonstrations.\n\n[1] Robust Imitation Learning from Corrupted Demonstrations, https://arxiv.org/abs/2201.12594", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and comments. Please feel free to ask any additional follow-up questions.\n\n**[Responses to Weaknesses]**\n\n**(1.1) (Problem Setup & The need for labeled non-preferred demonstrations)**  \nIn our work, we focus on solving safe imitation learning in an offline setting. We assume scarce but **labeled** non-preferred demonstrations $D^N$, and abundant but **unlabeled** demonstrations $D^U$ of both preferred and non-preferred behavior. Our goal is to learn a safe policy that follows the preferred behavior while avoiding the non-preferred behavior.\n\nPlease note that our problem setup and the goal are clearly different from robust imitation learning (such as [1]). Robust imitation learning [1] assumes only unlabeled demonstrations and focuses on filtering out **outliers** from unlabeled demonstrations. However, in order to filter out outliers, robust imitation learning [2] assumes that the **majority of the unlabeled demonstrations are preferred demonstrations**.  In other words, if the non-preferred demonstration ratio of unlabeled demonstration is greater than 0.5, it cannot be applied. Therefore, cases in which preferred behavior can be learned only with unlabeled demonstrations without any labeled demonstrations are very limited.\n\nOn the other hand, we assume a small number of labeled non-preferred demonstrations, but **no assumptions are needed for unlabeled demonstrations** (ex. the ratios of preferred and non-preferred demonstrations in unlabeled demonstrations). To show that SafeDICE successfully works regardless of the configuration of unlabeled demonstration, we additional conducted experiments for various configurations of unlabeled demonstrations $D^U$ consisting of different ratios of preferred and non-preferred demonstrations  ($(|D^U_P|, |D^U_N|) \\in \\[(1000, 1000), (500, 1000), (1000, 500), (250, 1000), (1000, 250)\\]$, where $|D^U_P|$ and $|D^U_N|$ denote the number of preferred and non-preferred demonstrations in $D^U$, respectively). As can be seen from the uploaded PDF (Figure 1), the results show that SafeDICE outperforms baselines in all settings with various configurations of **unlabeled** demonstrations. These results show that SafeDICE can be applied more effectively in various real-world scenarios regardless of the preferred/non-preferred ratio of unlabeled demonstrations. We will add these results and explanations to the final version of the paper.\n\n**(1.2) (Robust Imitation Learning)**  \nAs the reviewer pointed out, BCND [2] and RBC [1] also share the aim of learning a safe policy, similar to SafeDICE. However, BCND and RBC focus on **filtering out outliers**, whereas SafeDICE imitates provided demonstrations while excluding non-preferred demonstrations.\n\nBoth BCND and RBC aim to learn an optimal policy using only the provided unlabeled demonstrations (unlabeled demonstrations are referred to as noisy demonstrations in [2] and as corrupted demonstrations in [1], respectively). **Assuming that the majority of the unlabeled demonstrations are preferred demonstrations**, BCND and RBC recover an optimal policy via iterative mode-seeking and Median-of-Means (MOM), respectively. While these approaches are effective for filtering out outliers, they might face challenges when preferred demonstrations are not the majority of the unlabeled demonstrations.\n\nIn contrast, SafeDICE aims to learn a safe policy that follows the preferred behavior using non-preferred demonstrations and unlabeled demonstrations. We want to emphasize that unlabeled demonstrations could include over half of the non-preferred demonstrations (see the results in the uploaded PDF, Figure 1). \n\nWe will add this discussion on robust imitation learning to the related work section of the final version of the paper.\n\n[1] Liu, Liu, et al. \"Robust imitation learning from corrupted demonstrations.\" arXiv 2022.  \n[2] Sasaki, Fumihiro, and Ryota Yamashina. \"Behavioral cloning from noisy demonstrations.\" ICLR 2021.\n\n**(2) (Sensitive to the number of labeled non-preferred demonstrations)**  \nIn the case of preference-based reward learning (i.e. reward learning in PPL), since trajectory-level samples are used for reward learning (see Eq (40) in Appendix C.3), it is relatively sensitive to the number of labeled non-preferred demonstrations $|D^N|$. On the other hand, DWBC and SafeDICE use state-action-pair-level samples for reward learning, so they learn relatively sample-efficiently and are not sensitive to $|D^N|$. For example, if $|D^N|=10$ and tiemstep_of_trajectory=1000, only 10 non-preferred samples are used in reward learning for PPL,  but in the case of DWBC and SafeDICE, 10000 non-preferred samples are used.\n\n**[Responses to Questions]**\n\n**(Related to hyperparameter search)**  \nIn the offline imitation learning setting where further environment interactions are not available, hyperparameter search through evaluation is not possible. Therefore, in offline imitation learning, it is very important to work without hyperparameter search. And as can be seen from the results in Appendix E.2 (Figures 6 and 7), baseline algorithms are hyperparameter sensitive, and finding the optimal hyperparameter is very challenging and expensive.\n\n**(Comparison with DWBC)**  \nAs shown in Appendix E.2 (Figures 6 and 7), since DWBC is hyperparameter sensitive, finding the optimal hyperparameter is difficult even if the hyperparameter search is allowed.\n\nIn addition to the hyperparameter-free or not, the main difference between SafeDICE and DWBC is that SafeDICE essentially leverages the non-preferred demonstrations **in the space of stationary distributions (i.e. directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non-preferred behavior)**, unlike DWBC that leverages it in the space of policy or the learning process of the discriminator. "}, {"review_id": "mG40p0CW6o", "reviewer": "Reviewer_ERpc", "summary": "Learning safe behaviors from a dataset of demonstrations is a challenging problem. The work presents SafeDICE, an algorithm which learns a safe policy using preference-based imitation learning. The method leverages non-preferred demonstrations in the space of stationary distributions in contrast to prior methods which operate in the policy and discriminator space. SafeDICE reduces the constrained optimization problem into a single convex minimization objective by constructing a Lagrangian and eliminating the maximization problem using a closed form solution. The new objective does not require additional hyperparameter search. Empirical performance demonstrates that SafeDICE is competitive to other methods in satisfying cost constraints.", "questions": "* Can you please provide a concrete explanation for comparing SafeDICE to generic Imitation Learning baselines? Can you compare SafeDICE to relevant constrained optimization baselines such as CPO, PPO-Lagrangian, TRPO and LAMBDA on small toy tasks?\n* What is the breakdown of the dataset? How many safe and unsafe actions does the data consist of? Do samples have cost variables as well? Can you please explain the reason for a lower number of non-preferred demonstrations? How does SafeDICE perform when number of non-preferred demonstrations are increased? Can the evaluation accomodate different kind of datasets?\n* How does SafeDICE perform in the absence of stationary distribution constraint? How does the performance vary in the absence of preferred/non-preferred demonstrations?\n* Can you please explain how $w(s,a)$ and $r(s,a)$ were obtained?\n\n## Minors\n\n* line 9: learns safe -> learns a safe\n* line 43: what does DWBC stand for?\n* line 49: what is the degeneration issue?\n* line 52: learns safe policy -> learns a safe policy\n* line 65: I believe the formulation is a Constrained MDP (CMDP) since the problem setting has cost constraints of the form $\\mathcal{C}: S \\times A \\rightarrow \\mathbb{C}$\n* line 89: Recent offline -> A recent offline\n* line 149: optimization of Eq. 7\n* Proposition 3.2: what is $\\epsilon$?\n* line 258: different two -> two different\n\n[1]. As et. al., Constrained Policy Optimization via Bayesian World Models, ICLR 2022.  \n[2]. Kumar et. al., Conservative Q-Learning for Offline Reinforcement Learning, NeurIPS 2021.  \n[3]. Bhardhwaj et. al., Conservative Safety Critics for Exploration, ICLR 2021.", "limitations": "The paper would benefit from a discussion on limitations. Authors could highlight the gap in reward-cost trade off and the increasing number of non-preferred demonstrations as potential limitations of the proposed approach.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is well motivated and within the scope of safe Imitation Learning.\n* The proposed method presents a novel stationary distribution formulation of the Lagrangian constraints which, to the best of my knowledge, has not been observed before.", "weaknesses": "* **Choice of Baselines:** My main concern is the choice of baselines used to compare SafeDICE. The paper only compares with naive BC, DWBC, PPL and DExperts. BC and DWBC are simple imitation learning algorithms which were not designed for constrained optimization problems. PPL utilizes a paired ranking scheme based on reward preferences which is tangential to imitation learning. This leaves DExperts as the only suitable baseline. There exist specific constrained optimization methods such as CPO, PPO-Lagrangian, TRPO and LAMBDA albeit for online and model-based settings [1]. Additionally, there exist offline RL methods [2, 3] which demonstrate safe behaviors as a result of their pessimistic nature. Authors should include these as relevant baselines on small tasks or provide a concrete explanation for why generic Imitation Learning baselines are suitable for constraint satisfaction.\n* **Dataset Comparison:** Due to the absence of safe offline IL datasets, the work constructs its own dataset based on preferred and non-preferred demonstrations. My concerns are around the design of the dataset. The paper does not provide a detailed breakdown of the dataset and the number of safe and unsafe actions. Tables 1 and 3 show the number of non-preferred demonstrations which are significantly less in comparison to the preferred demonstrations (50 non-preffered demonstrations against 1000 preferred demonstrations). While the authors make an attempt to compare algorithms on different dataset sizes in Figure 8, these comparisons are still well in the low data regime. In my opinion, the dataset is heavily biased and is not reflective of other benchmarks and real-world applications in safety. The paper could evaluate algorithms on different kinds of datasets with different behavior qualities. For instance, datasets could be constructed from replay buffers of trained lagrangian agents perturbed with gaussian noise or a balanced mix of safe and unsafe actions based on constraint satisfaction.\n* **Empirical Evaluation:** Experiments do not comprehensively evaluate the efficacy of the reduced convex minimization problem. The central contribution of the paper is estimating the stationary distribution corrections of the safe policy. This could be evaluated by ablating the Lagrangian with a constraint in which the stationary distributions are absent or non-preferred demonstrations are not utilized. In its current form, empirical evaluation does not throw light on the performance or utility of the proposed objective.\n* **Clarity of Derivation:** I am having trouble following the main derivation of Lagrangian $\\mathcal{L}(w, \\nu ; r_{\\alpha})$. Specifically, it would be helpful if authors could explain how they reached step 3 to obtain $w(s,a)$ and $r(s,a)$. It might be helpful to cover intermediate steps in the Appendix or provide a short note as footnote for the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and comments. Please feel free to ask any additional follow-up questions.\n\n**[Responses to Weaknesses]**\n\n**(1) (Choice of Baselines)**  \nPlease note that the setting we consider in the paper is **offline safe imitation learning, not constrained RL**, and **there is no reward and cost information** in the given offline dataset. Therefore, it is impossible to consider this problem as a constrained optimization problem, and also impossible to consider **online/offline constrained RL** algorithms (i.e. CPO, PPO-Lagrangian, TRPO, and LAMBDA) as our baseline methods. \n\nIn the problem setting we consider in the paper, we only assume a small number of **trajectory-level labeled** non-preferred demonstrations $D^N$ and unlabeled demonstrations $D^U$ which consist of both preferred and non-preferred demonstrations. Thus, we consider learning a reward function based on preference and using it to perform offline IL/RL. DWBC is one of the representative offline imitation learning algorithms that can leverage preference information from labeled non-preferred demonstrations and unlabeled demonstrations. Moreover, PPL is an offline RL baseline using learned preference-based reward and weighted behavior cloning which is one of the representative offline RL algorithms. The baseline algorithms we consider in the paper are not just simple IL algorithms, but offline IL/RL-based strong baseline algorithms that utilize preference information.\n\nHowever, if you are simply curious about the performance of the oracle agent (not the comparison as baseline under fair conditions), the information of the preferred demonstration shown in Tables 1 and 3 (Appendix D.1 and D.2) is the optimal performance of the online safe RL algorithm (i.e. oracle agent). To obtain the preferred demonstration of unlabeled demonstration, we trained the policy with online constrained RL, then generate the preferred demonstration. \n\n**(2) (Dataset Comparison)**  \nAs mentioned in the paper (lines 265-266), please note that our problem setting assumes a small amount of **labeled** preferred demonstrations and relatively large amounts of unlabeled demonstrations consist of both preferred and non-preferred demonstrations. The reason why the number of **labeled** non-preferred demonstrations in the paper is set to be significantly smaller than unlabeled demonstrations is **to consider the experimental setting similar to real-world problems**. In real-world scenarios, unlabeled demonstrations are relatively easy to obtain, but labeled demonstrations are expensive and difficult to obtain.\n\nAnd, for the **unlabeled** demonstrations used in all experiments, the ratio of preferred demonstrations to non-preferred demonstrations is 1:1 (ex. For the Walker domain, the number of **labeled** non-preferred demonstrations = 10, the number of **unlabeled** non-preferred demonstrations = 1000, and the number of **unlabeled** preferred demonstrations = 1000). Therefore, the datasets we used in the paper are not biased. We will add these details on the configuration of labeled and unlabeled demonstrations to the final version of the paper.\n\nTo avoid misunderstanding, the Figure 8 results mentioned in the review are experiments with varying amounts of **labeled** demonstrations to show the impact of **labeled** non-preferred demonstrations.\n\nIn order to evaluate in more various configurations of **unlabeled** demonstrations, we also conducted additional experiments for various configurations of unlabeled demonstrations $D^U$ consisting of different ratios of preferred and non-preferred demonstrations ($(|D^U_P|, |D^U_N|) \\in \\[(1000, 1000), (500, 1000), (1000, 500), (250, 1000), (1000, 250)\\]$, where $|D^U_P|$ and $|D^U_N|$ denote the number of preferred and non-preferred demonstrations in $D^U$, respectively). As can be seen from the uploaded PDF (Figure 1), the results show that SafeDICE outperforms baselines in all settings with various configurations of **unlabeled** demonstrations. These results show that SafeDICE can be applied more effectively in various real-world scenarios regardless of the preferred/non-preferred ratio of unlabeled demonstrations. We will add these results and explanations to the final version of the paper.\n\n**(3) (Empirical Evaluation)**  \nWe conducted additional experiments to evaluate the impact of the stationary distribution constraint and labeled non-preferred demonstrations. As can be seen from the uploaded PDF (Figure 2), the performance is degraded both in the absence of stationary distribution constraints and in the case of not using labeled non-preferred demonstrations. We will add this result and explanation to the final version of the paper.\n\n**(4) (Clarity of Derivation)**  \nWe added derivation details to the PDF and will add them to the final version of the paper.\n\n**[Responses to Questions]**\n\n**(1) (Baselines of imitation learning and constrained RL)**  \nSee the response of **(1) (Choice of Baselines)**\n\n**(2.1) (Breakdown of the dataset & number of safe/unsafe actions)**  \nSee the response of **(2) (Dataset Comparison)**\n\nIn the problem setting we consider in the paper, safe (preferred) and unsafe (non-preferred) are not defined at the action-level, but only defined at the trajectory-level. \n\n**(2.2) (Cost variables)**  \nPlease note that the setting we consider in the paper is **offline safe imitation learning, not constrained RL**, and **there are no reward and cost variables** in the given offline dataset.\n\n**(3) (Absence of stationary distribution constraint and preferred/non-preferred demonstrations)**  \nSee the response of **(3) (Empirical Evaluation)** and **(2) (Dataset Comparison)**\n\n**(4) (Derivation of $w(s,a)$ and $r(s,a)$)**  \nWe added derivation details to the PDF and will add them to the final version of the paper."}, {"review_id": "EhbB6lmLDY", "reviewer": "Reviewer_Ajjv", "summary": "This paper presents an offline safe imitation learning algorithm called SafeDICE. A unique point of SafeDICE is that a safe policy is learned by non-preferred and unlabeled demonstrations in the imitation learning framework. Based on the formulations studied in DICE family, this paper formulated the safe IL problem as a stationary distribution matching problem, and then solve a single convex minimization problem without additional hyper-parameter search. In the experiments using RWRL and Safety-Gym benchmarks, the authors demonstrate the effectiveness of the SafeDICE algorithms and show that a better policy can be learned.", "questions": "[Q1] What is the actual value of the return without normalization? Did the authors confirm that the tasks are successfully solved?\n\n[Q2] What is the threshold of the Safety Gym? Is it a default parameter (i.e., 20)?\n\n[Q3] I think it is impressive that SafeDICE does not require a hyper-parameter search. Though I agree that (15) is a beautiful and useful equation, do we have some chance to obtain a better $\\alpha$?", "limitations": "- Unless I miss something, limitations have not adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to understand. I think the presentation of the ideas is clear and concise.\n\n- The problem setting is well-motivated with a timely example (i.e., chatbot in lines 30-32). I agree that the problem setting is actually important and unexplored.\n\n- The empirical evaluation of this paper is great. I think the empirical evaluation is sufficient because the authors demonstrated the effectiveness of the proposed SafeDICE algorithm in two environments with comparison with reasonable baselines.", "weaknesses": "- It is a little bit unclear what is the true mathematical contributions. As the authors describes as the following, but it seems that the most of the mathematical formulations are based on the DICE family papers and the contributions of this paper is a little bit minor.\n> The main flow of our derivation follows prior DICE-based offline imitation learning methods [12, 13], but it is clearly different from the existing methods that require hyper-parameter search to deal with unknown $\\alpha$.\n\n- Though the authors demonstrated that the SafeDICE performs better than other baselines, the performance itself seems low if I understand correctly. And, since return is normalized, it is unclear even whether or not the tasks are solved. I think it would be better to add a typical Safe RL algorithm (e.g., CPO, TRPO-Lagrangian) as an oracle agent. Note that, I do *not* require the SafeDICE to perform better than Safe RL algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and comments. Please feel free to ask any additional follow-up questions.\n\n**[Responses to Weaknesses]**\n\n**(mathematical contributions)**  \nThe mathematical contribution of our paper is to present a well-defined mathematical formulation for a new problem setting that cannot be solved with existing DICE-based methods. \n\nFirst, our proposed algorithm deals with a new problem setting that aims to learn a safe policy using non-preferred demonstrations, and cannot be simply formulated and solved with existing methods including DICE-based algorithms and imitation learning algorithms, which assume that preferred demonstrations are explicitly given.\n\nSecondly, the most important mathematical contribution of this paper is that hyperparameter $\\alpha$ can be theoretically determined, unlike existing algorithms in which hyperparameter search is essential and hyperparameter sensitive. Moreover, it is impossible for our algorithm SafeDICE to be formulated and optimized the objective without hyperparameter $\\alpha$ selection. Considering the value of $r(s,a)$ in Eq (10) with arbitrary $\\alpha$ value, the value inside the log function can be negative for some $\\alpha$. However, our proposed $\\alpha$ selection rule not only theoretically bounds the difference between the stationary distributions of learned policy and preferred policy, but also guarantees that the value in the log function of equation (10) is always positive (see lines 179-180 and Appendix A.3).\n\n**(Normalized return & SafeRL algorithm)**  \nSafeDICE not only outperforms the other baseline algorithms but also the performance itself seems not low. We used a normalized return to improve readability, and the normalized return is a value normalized by optimal performance learned with online constrained RL as follows: $\\text{normalized return} = \\frac{\\text{return}}{\\text{maximum return of optimal policy}}$ (i.e. A normalized return value close to 100 means that the tasks are successfully solved). The policy used to generate the preferred demonstration is the policy learned with online constrained RL, and the information of the preferred demonstration shown in Tables 1 and 3 (Appendix D.1 and D.2) is the optimal performance of the online safe RL algorithm (i.e. oracle agent). We will add a clear explanation about normalized return, and the results of this oracle agent as dashed lines to the plots in the final version of the paper.\n\n**[Responses to Questions]**\n\n**(Q1) (Normalized return)**  \nAs previously mentioned, the normalized return is a value normalized by optimal performance learned with online constrained RL as follows: $\\text{normalized return} = \\frac{\\text{return}}{\\text{maximum return of optimal policy}}$ A normalized return value close to 100 means that the tasks are successfully solved and we confirm the tasks are successfully solved.\n\n**(Q2) (Threshold of the Safety Gym)**  \nIf it is correct that the threshold means the cost limit for each task, we used the default parameter ($d=25$) as in the original paper of the Safety Gym environment (page 16 in [1]). We will add these experimental details to Table 3 (Appendix D.2) in the final version of the paper.\n\n**(Q3) (Chance to obtain a better $\\alpha$)**  \nBased on Proposition 3.2 of our paper, we can theoretical guarantees on the error bound of the estimated stationary distribution with $\\alpha$ selected by Eq (15). As mentioned in the paper (lines 175-178 and Eq (15)), this error is close to zero in most of the real-world scenarios where the behavior of the non-preferred and preferred policies are different. However, since $\\epsilon$ is not exactly zero, theoretically there is a small possibility of finding a better $\\alpha$.\n\nPractically, the results of Figures 10 and 11 (Appendix E.4) show that it is very difficult to find better $\\alpha$ through hyperparameter search."}]

Author rebuttal:
We thank all the reviewers for their constructive feedback and comments. Below we restate the main clarification and experiments of our rebuttal. If you have any additional questions or concerns to our response, we are happy to provide additional responses during the rebuttal period.

**[Clarification of problem settings and baselines]**  
Please note that the setting we consider in the paper is **offline safe imitation learning, not constrained RL**, and **there is no reward and cost information in the given offline dataset**. We only assume a small number of *trajectory-level* **labeled** preferred demonstrations and abundant **unlabeled** demonstrations. Therefore, it is impossible to consider this problem as a constrained optimization problem, and also **impossible to consider online/offline constrained RL** algorithms as our baseline methods. However, the baseline algorithms we consider in the paper are not just simple IL algorithms, but offline IL/RL-based strong baseline algorithms that **can utilize preference information**.

**[Additional experiments]**  
1. In order to evaluate in more various configurations of unlabeled demonstrations, we also conducted additional experiments for various configurations of unlabeled demonstrations $D^U$ consisting of different ratios of preferred and non-preferred demonstrations ($(|D^U_P|, |D^U_N|) \in \[(1000, 1000), (500, 1000), (1000, 500), (250, 1000), (1000, 250)\]$, where $|D^U_P|$ and $|D^U_N|$ denote the number of preferred and non-preferred demonstrations in $D^U$, respectively). As can be seen from the uploaded PDF (Figure 1), the results show that SafeDICE outperforms baselines in all settings with various configurations of unlabeled demonstrations. These results show that SafeDICE can be applied more effectively in various real-world scenarios regardless of the preferred/non-preferred ratio of unlabeled demonstrations. We will add these results and explanations to the final version of the paper.

2. We conducted additional experiments to evaluate the impact of the stationary distribution constraint and labeled non-preferred demonstrations. As can be seen from the uploaded PDF (Figure 2), the performance is degraded both in the absence of stationary distribution constraints and in the case of not using labeled non-preferred demonstrations. We will add this result and explanation to the final version of the paper.

Meta-review: The authors address the problem of safe imitation learning in a problem formulation that assumes that, in addition to demonstrated target behavior, a small set of "non-preferred demonstrations" are provided that demonstrate the types of behaviors to avoid. A novel approach, SafeDICE, is introduced and empirical results demonstrate its effectiveness.

Initial reviews showed a large difference in assessments between reviewers. Noted strengths included the novel and relevant problem setting, soundness and novelty of the approach, strong empirical results and overall clarity of the presentation.

Noted weaknesses included the depth of empirical evaluation to go beyond purely proving strong performance of the approach and to provide deeper insights on e.g., the role of non-preferred demonstrations and other components of the algorithm as well as comparison to oracle and additional baseline performance. In addition, opportunities for improving the presentation were identified.

During the rebuttal period, reviewers and authors engaged in detailed conversations, clarified open questions, and agreed on additional oracle and other results to be included in the main paper. After this discussion, reviewers now unanimously recommend acceptance. The authors are strongly encouraged to consider all feedback when preparing the camera ready version.
Decision: Accept (poster)