Title: SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Imitation learning

Preference-based learning

Safe imitation learning
Abstract: We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimics preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonstrations. This problem setting corresponds to various real-world scenarios, where satisfying safety constraints is more important than maximizing the expected return. However, it is very challenging to learn the policy to avoid constraint-violating (i.e. non-preferred) behavior, as opposed to standard imitation learning which learns the policy to mimic given demonstrations. In this paper, we present a hyperparameter-free offline safe IL algorithm, SafeDICE, that learns safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. Our algorithm directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non-preferred behavior. In the experiments, we demonstrate that our algorithm learns a more safe policy that satisfies the cost constraint without degrading the reward performance, compared to baseline algorithms.

SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations

Abstract

We consider offline safe imitation learning (IL), where the agent aims to learn the safe policy that mimics preferred behavior while avoiding non-preferred behavior from non-preferred demonstrations and unlabeled demonstrations. This problem setting corresponds to various real-world scenarios, where satisfying safety constraints is more important than maximizing the expected return. However, it is very challenging to learn the policy to avoid constraint-violating (i.e. non-preferred) behavior, as opposed to standard imitation learning which learns the policy to mimic given demonstrations. In this paper, we present a hyperparameter-free offline safe IL algorithm, SafeDICE, that learns safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. Our algorithm directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non-preferred behavior. In the experiments, we demonstrate that our algorithm learns a more safe policy that satisfies the cost constraint without degrading the reward performance, compared to baseline algorithms.

1 Introduction

Reinforcement learning (RL) [28] aims to learn the optimal policy that maximizes the expected cumulative rewards through interacting with the environment. Recently, RL has achieved remarkable successes in various domains, but its application to real-world tasks is yet challenging mainly due to the following two reasons: First, since the interaction with the environment is costly in most real-world tasks, the policy should be learned solely from the pre-collected dataset (i.e. offline learning ). Second, reward annotation, which is essential for applying the RL algorithm, is a very strong requirement that is difficult to obtain in real-world tasks.

Imitation learning (IL) [26, 1, 25] is a more realistic framework to deal with real-world tasks, where the expert's demonstrations are solely given without reward annotations. Standard IL aims to mimic the expert's demonstrations generated from the optimal policy in terms of expected return. However, in most real-world tasks, additional constraints should be considered in addition to maximizing the expected return such as the safe driving for autonomous driving and toxicity degree for the conversational agent. Fortunately, non-preferred demonstrations that violate constraints are often naturally collected. For example, black box data of an accident vehicle and toxic content reported by chatbot users are separately collected as non-preferred demonstrations. However, most existing studies have focused only on imitating expert (i.e. preferred ) behavior, and a safe imitation learning method that avoids non-preferred behavior has not yet been explored.

In this paper, we are particularly interested in solving safe imitation learning in an offline setting: Given scarce but labeled non-preferred demonstrations, and abundant but unlabeled demonstrations

𝑫 𝑼

𝑫 𝑵

∪

DemoDICE, DWBC

PPL, DExperts

SafeDICE (Ours)

Preferred

𝑫 𝑷

Non-Preferred

𝑫 𝑵

Unlabeled

𝑫 𝑼

O

O

X

X

O

O

O

O

O

Unlabeled

Non-Preferred

of both preferred and non-preferred behavior, we aim to learn a safe policy that follows the preferred behavior while avoiding the non-preferred behavior (see Figure 1). This problem setting has been mostly unexplored, and cannot be completely addressed by existing methods. One of the existing applicable approaches is the preference-based policy learning algorithm [4, 35, 17], which optimizes a policy based on preference rankings over demonstrations. This method can learn the policy to prefer unlabeled demonstrations to non-preferred demonstrations, but it also prefers non-preferred demonstrations mixed in the unlabeled demonstrations, which can lead to learning suboptimal policy. One of the approaches effectively using the unlabeled dataset is DWBC [31], a weighted behavior cloning based on a discriminator, which adopts the positive-unlabeled learning [9, 36, 30] for training the discriminator. This method requires an additional cost of hyperparameter search for training the discriminator, and its performance is hyperparameter sensitive. Another notable approach in the natural language processing domains is DExperts [19], a decoding-time method, which combines trained models with preferred and non-preferred demonstrations. This method is also hyperparameter sensitive and prone to the degeneration issue that leads to abnormal behavior. Both DWBC and DExperts, which require hyperparameter search, are unsuitable for real-world tasks where interaction with the real environment is costly and risky.

We present an offline safe IL algorithm that learns safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. Our algorithm, offline Safe imitation learning by avoiding non-preferred demonstrations via DIstribution Correction Estimation (SafeDICE), directly estimates the stationary distribution corrections of the policy that imitate the demonstrations excluding the non-preferred behavior. SafeDICE essentially leverages the non-preferred demonstrations in the space of stationary distributions, unlike existing algorithms that leverage it in the space of policy [19] or the learning process of the discriminator [31]. We show that our algorithm can be reduced to a single convex minimization problem without additional hyperparameter search , unlike existing algorithms that are highly sensitive to the hyperparameter. In the experiments, we demonstrate that our algorithm learns a more preferred (i.e. constraint-satisfying) policy compared to baseline algorithms in constrained RL benchmarks including RWRL [8] and Safety Gym [24].

2 Preliminaries

2.1 Markov Decision Process (MDP)

We consider an environment modeled as a Markov Decision Process (MDP), defined by M = ⟨ S, A, T, R, p 0 , γ ⟩ [28], where S is the set of states, A is the set of actions, T : S × A → ∆( S ) is the transition probability, R : S × A → R is the reward function, p 0 ∈ ∆( S ) is the distribution of the initial state, and γ ∈ [0 , 1) is the discount factor. For the given policy π , its stationary distribution d π ( s, a ) is defined as follows:

where s 0 ∼ p 0 , a t ∼ π ( ·| s t ) , and s t +1 ∼ T ( ·| s t , a t ) for all timesteps t ≥ 0 . For brevity, d π will be used to denote the stationary distribution d π ( s, a ) .

We consider the offline imitation learning setting, where the policy should be optimized from solely a pre-collected dataset without reward annotation and any additional online environment interaction. In our problem setting, preferred and non-preferred behaviors are defined by satisfying or violating constraints. We denote the preferred policy by π P , the non-preferred policy by π N , and the corresponding stationary distribution of π P and π N by d P and d N , respectively. We assume a pre-collected dataset that consists of scarce but labeled dataset D N generated by the non-preferred policy, and a abundant but unlabeled dataset D U generated by both preferred and non-preferred policies. We denote the corresponding stationary distribution of D U by d U , which can be regarded as convex combination of d P and d N with unknown ratio α (i.e. d U := (1 -α ) d P + αd N ). In this setting, we aim to learn a safe policy from unlabeled demonstrations while avoiding non-preferred demonstrations that are labeled as non-preferred behavior.

2.2 Preference-based Offline Imitation Learning

Imitation learning (IL) aims to imitate the behavior of expert policy solely from state-action demonstrations without reward annotations. The standard IL is naturally formulated as minimizing the KL divergence between stationary distributions of preferred policy (i.e. expert policy) and target policy [14, 11]:

However, it requires a large amount of annotated preferred demonstrations, which may be a too strong requirement for various real-world scenarios. Recent offline IL method [13] relaxes this requirement for preferred demonstrations by leveraging the unlabeled dataset which may contain both preferred and non-preferred demonstrations:

where α ≥ 0 is a hyperparameter that controls the balance between minimizing the KL divergence with d U and d P . This offline imitation learning method can learn the preferred policy from a large number of unlabeled demonstrations with only a small amount of labeled preferred demonstrations.

In contrast to the mainstream of recent IL approaches focusing on mimicking expert (i.e. preferred) behavior, avoiding dangerous (i.e. non-preferred) behavior is a more important problem in many real-world tasks, which has been unexplored yet. In the same manner as Eq. (2), we can naively consider the penalty for the non-preferred behavior by maximizing the KL divergence with d N :

where α ≥ 0 is a hyperparameter that controls the balance between minimizing the KL divergence with d U and maximizing the KL divergence with d N . However, since the KL divergence has no upper bound, optimizing Eq. (3) is very unstable and challenging, where the D KL ( d π ∥ d N ) can blow up to infinity. Therefore, safe imitation learning while avoiding non-preferred behavior is not a straightforward problem and has not yet been studied.

3 SafeDICE

In this section, we present offline Safe imitation learning by avoiding the non-preferred demonstrations via DIstribution Correction Estimation (SafeDICE), an offline IL algorithm that can learn the safe policy using labeled non-preferred demonstrations. SafeDICE optimizes the stationary distributions of the target policy to match the stationary distribution of the preferred policy from the unlabeled dataset by excluding non-preferred demonstrations.

3.1 Offline Safe Imitation Learning with Non-Preferred Demonstrations

Similar to standard IL approaches, the derivation of our algorithm starts from KL divergence minimization between d π and d P , which can be estimated by d U and d N as follows:

where α is unknown ratio between the stationary distributions of preferred policy and non-preferred policy (i.e. d U := (1 -α ) d P + αd N ). Optimizing the Eq. (4) is sensitive to α , and selecting the α with hyperparameter search is not suitable for real-world tasks where the interaction with the real environment is very costly and risky. Therefore, it is not straightforward to learn the preferred policy using non-preferred demonstrations with existing imitation learning algorithms. The main contribution of our paper is to present an algorithm that theoretically determines α and learns the preferred policy only with the labels of non-preferred demonstrations without any hyperparameter search. In the following description, we first consider the α as a known ratio in our derivation, then present the way to select the α while providing the theoretical guarantees on the error bound of optimized stationary distribution with selected α . All the proofs can be found in Appendix A.

The main flow of our derivation follows prior DICE-based offline imitation learning methods [12, 13], but it is clearly different from the existing methods that require hyperparameter search to deal with unknown α . First, we consider a problem equivalent to Eq. (4) in terms of stationary distribution d :

where ( B ∗ d )( s ) := ∑ a d ( s, a ) is the marginalization operator, and ( T ∗ d )( s ) := ∑ ¯ s, ¯ a T ( s | ¯ s, ¯ a ) d (¯ s, ¯ a ) is the transposed Bellman operator. The Bellman flow constraint (6) ensures that d ( s, a ) is a valid stationary distribution of some policy, where d ( s, a ) can be interpreted as a normalized occupancy measure of ( s, a ) . Therefore, if we find optimal d ∗ ( s, a ) that satisfies Eq. (5) and (6), then its corresponding policy can be easily obtained by π ∗ ( a | s ) = d ∗ ( s,a ) ∑ a d ∗ ( s,a ) .

The Lagrangian dual formulation of the above constrained optimization problem is

where ν ( s ) are the Lagrange multipliers for the Bellman flow constraints (6). However, directly optimizing the Eq. (7) is challenging in an offline manner, where further environment interactions are not available. To obtain a tractable objective, we introduce the following derivations:

where r α ( s, a ) corresponds to the preference-based reward function in the context of RL, which will be trained by the pre-collected dataset.

In summary, our goal is reduced to solve the following maximin optimization:

where the optimal solution w ∗ of Eq. (8) represents the stationary distribution correction of an optimal policy π ∗ (i.e. w ∗ ( s, a ) = d π ∗ ( s,a ) d U ( s,a ) ).

3.2 Pretraining the Preference-based Reward Function r α ( s, a )

To optimize the Eq. (8), we pretrain the discriminator to estimate the log ratio of d N ( s, a ) and d U ( s, a ) , which is required to estimate for the r α ( s, a ) :

With the optimal discriminator c ∗ ( s, a ) = d N ( s,a ) d U ( s,a )+ d N ( s,a ) trained by Eq. (9), the preference-based reward function r α ( s, a ) can also be obtained as:

However, according to the value of α , the reward function r α ( s, a ) defined by Eq. (10) cannot guarantee that the value inside the log function always becomes a positive. Therefore, finding the optimal α through hyperparameter search is infeasible.

3.3 Reduction to the Single Min Optimization

Since the optimization Eq. (7) is a convex optimization, we can change the order of operator from maximin optimization to minimax optimization [2]:

Then, we can replace the inner maximization of Eq. (11) with a closed-form solution for w , which can be easily obtained by exploiting the convexity.

Proposition 3.1. For any ν , the closed-form solution to the inner maximization of Eq. (11) , i.e. w ∗ ν = arg max w ≥ 0 L ( w,ν ; r ) , is given by:

By using the closed-form solution w ∗ ν of Eq. (12), our derivation can be reduced to a single minimization as follows:

Finally, by optimizing the single minimization problem of Eq. (13), we can obtain the optimal stationary distribution correction w ∗ which can easily extract the corresponding optimal policy π ∗ . To extract a policy from the optimal stationary distribution correction w ∗ ( s, a ) , we adopt weighted behavior cloning on the offline dataset as follows:

where w ∗ ( s, a ) = d π ∗ ( s,a ) d U ( s,a ) .

3.4 Hyperparameter α Selection

In the previous sections, we have derived our algorithm under the assumption that α is a known ratio. The remaining question is how we select the suitable α without additional hyperparameter search. Fortunately, in contrast to the existing methods [19, 31] that require the additional cost of hyperparameter search, we can specify the α with providing the theoretical guarantees on the error bound of the estimated stationary distribution.

Proposition 3.2. Suppose min ( s,a ) d P ( s,a ) d N ( s,a ) ≤ ϵ . If we obtain the estimated stationary distribution by optimizing the Eq. (7) with α set to ˆ α as follows:

then the total variation distance of true and estimated stationary distribution of preferred policy is bounded as follows:

where D TV is total variation distance, d P ( s, a ) is true stationary distribution of the preferred policy and ˆ d P ( s, a ) is estimated stationary distribution which is defined by ˆ α as ˆ d P ( s, a ) = d U ( s,a ) -ˆ αd N ( s,a ) 1 -ˆ α . (Proof in Appendix A.)

Intuitively, since the behavior of the non-preferred and preferred policies are different, there must exist state-action spaces that are rarely visited by the preferred policy but are frequently visited by the non-preferred policy. Therefore ϵ may be a very small positive value, then the total variation distance of the true and estimated stationary distribution is close to zero:

We also provide proof in Appendix A.3 that r α ( s, a ) of Eq. (10), defined by the α selected by Eq. (15), is always well-defined with positive value inside the log-function in Eq. (10).

Finally, our algorithm optimizes the Eq. (7) with the α obtained by Eq. (15), where α is no longer an unknown hyperparameter. In short, by leveraging the non-preferred demonstrations in the space of stationary distributions, this problem can be resolved by solving a single convex minimization problem for a specific α . This is in contrast to the existing methods, which directly leverage the non-preferred demonstrations for training the policy [19] or discriminator [31] that require the additional cost of hyperparameter search. The pseudocode for the whole process of SafeDICE can be found in Appendix B.

4 Related Work

Stationary distribution correction estimation (DICE) Since the introduction of DICE for offpolicy evaluation in DualDICE [21], various DICE-family algorithms have been developed for offpolicy evaluation [33, 34, 32, 7]. In addition to these studies, two algorithms, AlgaeDICE [22] and ValueDICE [14], have been proposed to address the challenges of off-policy RL and IL, respectively. However, they may encounter issues with numerical instability in an offline setting due to the nested min-max optimization and out-of-distribution action evaluation.

To overcome the issues of numerical instability in offline settings, recent DICE-based offline RL [15, 16] and IL algorithms [13, 20, 12] have adopted a strategy of directly optimizing stationary distribution correction ratios, similar to the approach used in our work. Additionally, these offline IL algorithms utilize unlabeled demonstrations in an effort to address the problem of distribution drift [25]. They share a similarity with our algorithm in that they all aim to recover the expert's behavior through direct optimization of the stationary distribution correction ratios. However, these algorithms rely on the availability of clean expert demonstrations, which can make them difficult to use in situations where expert demonstrations are not available.

Policy Learning from Suboptimal Demonstrations Preference-based policy learning algorithms use ranked suboptimal demonstrations to learn a reward function. Many of these algorithms, such as T-REX [4], PrefPPO [18], and PEBBLE [17], utilize Bradley-Terry model [3] to model the reward function. Once the reward function is learned, T-REX and PrefPPO use PPO [27] to train a policy, while PEBBLE employs SAC [10]. While preference-based policy learning algorithms allow for learning from unlabeled demonstrations and potentially disregarding the negative influence of nonpreferred demonstrations, it is important to note that they may also give preference to non-preferred demonstrations mixed in the unlabeled demonstrations.

On the other hand, there are also many studies that utilize suboptimal demonstrations, even though they are not preference-based policy learning algorithms. D-REX [5] and SSRR [6] both learn a policy by using given suboptimal demonstrations, and then generate trajectories with varying levels of optimality using this policy with injected noise. But, both algorithms require online sampling, which is not feasible in offline settings. Wu et al. [29] proposed two algorithms that utilize positiveunlabeled learning [9, 36, 30] to train a discriminator for constructing a reward function. However,

both algorithms require ground-truth labels for uniformly sampled trajectories from unlabeled demonstrations. DWBC [31] uses positive-unlabeled learning to train a discriminator, which is then used to construct weights for weighted BC. DExperts [19], a decoding-time method in the natural language processing domain, combines models for preferred and non-preferred demonstrations. Unfortunately, both DWBC and DExperts may not be suitable for offline safe imitation learning tasks due to their sensitivity to hyperparameter settings.

5 Experiments

In this section, we show the experimental results on various tasks from constrained RL benchmarks. First, we conduct the evaluation on domains from Real-World RL (RWRL) suite [8], which provides challenges in real-world scenarios including safety constraints. Second, we also conduct the experiment on continuous control tasks from Safety Gym environment [24], which provides practical scenarios of safety issues. In order to show the motivation of our work and the limitation of existing algorithms, we also provide qualitative results on tabular MDP environments. The illustrative examples and qualitative analysis on tabular MDP environments can be found in Appendix G.

Baselines Since offline safe IL, which we consider in this paper, has not been explored yet much, it lacks directly applicable baseline algorithms. Therefore, we consider the following four algorithms as baselines. First, BC denotes simple behavior cloning on unlabeled demonstrations. Second, DWBC denotes a variant of the original discriminator-weighted behavior cloning algorithm [31] that adopts the positive-unlabeled learning for training the discriminator. To apply it to our setting, we modified positive-unlabeled learning to negative-unlabeled learning. Third, PPL denotes a preference-based policy learning algorithm that adopts the preference-based reward learning algorithm based on the Bradley-Terry model [3] and weighted behavior cloning for policy learning. Lastly, we also compare with DExperts [19], a decoding-time method that combines trained models with unlabeled and non-preferred demonstrations. The details of baseline algorithms are provided in Appendix C.

Evaluation metrics We compare the performance of SafeDICE and baseline algorithms in terms of task performance (i.e. expected return) and safety (i.e. cost constraints). We report the results with the following metrics: 1) Normalized Return : evaluates the normalized task performance of the policy, 2) Average Cost : evaluates the mean cost of the policy, 3) Cost-Violating Ratio : evaluates the ratio of cost-violating behavior of the policy, 4) Conditional Value at Risk performance (CVaR) 10% Cost : evaluates the mean cost of the worst 10% runs. We also provide CVaR k% Cost for various values of k in Appendix E.1. Here, our main focus of the comparison is to evaluate the safety ( Average Cost , Cost-Violating Ratio , and CVaR 10% Cost ), and we use the Normalized Return to examine the degeneration issue, whether the agents fall into abnormal behaviors or not. All metrics are measured as the average value of 500 trajectories generated from the learned policy, and we report the results that are averaged over 5 runs.

5.1 Real-World Reinforcement Learning (RWRL) Suite

Now, we present the results of SafeDICE on a set of continuous control benchmarks from Real-World RL (RWRL) suite [8], where the safety constraints are also provided by environments.

Task setup We conduct the experiments for three tasks from RWRL environment with safety constraints: Cartpole, Walker, and Quadruped. Since there is no standard dataset for offline IL considering the safety constraints, we collected data by training the online RL agents. To obtain an preferred and non-preferred policy, we trained the SAC [10] in different two ways that solve the RL problem with respect to reward penalized by the cost or not. Among these two policies which are both optimal in terms of return maximization, we define the preferred policy as a policy that satisfies the constraint (i.e. high-return-low-cost behavior) and the non-preferred policy as a policy that violates the constraint (i.e. high-return-high-cost behavior). This experimental setting is representative of many real-world scenarios, where the data is collected by high-return behavior without consideration of the cost. Then, we generated scarce but labeled non-preferred demonstrations D N from the non-preferred policy, and abundant but unlabeled demonstrations D U from both preferred and nonpreferred policies. We collected diverse trajectories by sampling with a stochastic policy from randomly initialized initial states. Our goal is to recover the low-cost behavior by avoiding the high-cost demonstrations while not hurting the high-return behavior of unlabeled demonstrations. The details for the experimental settings can be found in Appendix D.

0.0M

0.3M

0.6M

0.9M

1.2M

20

40

60

80

100

Normalized Return

Cartpole (Return)

0.0M

0.3M

0.6M

0.9M

1.2M

150

200

250

300

350

400

450

Average Cost

Cartpole (Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Cost-Violating Ratio

Cartpole (Cost Violation)

0.0M

0.3M

0.6M

0.9M

1.2M

390

400

410

420

430

440

450

460

470

CVaR 10% Cost

Cartpole (CVaR Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

0

20

40

60

80

100

Normalized Return

Walker (Return)

0.0M

0.3M

0.6M

0.9M

1.2M

100

200

300

400

500

600

700

Average Cost

Walker (Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Walker (Cost Violation)

0.0M

0.3M

0.6M

0.9M

1.2M

100

200

300

400

500

600

700

CVaR 10% Cost

Walker (CVaR Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

20

40

60

80

100

Normalized Return

Quadruped (Return)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

100

150

200

250

300

350

400

450

Average Cost

Quadruped (Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Quadruped (Cost Violation)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

150

200

250

300

350

400

450

CVaR 10% Cost

Quadruped (CVaR Cost)

BC

DWBC

PPL

DExperts

SafeDICE (ours)

Main results Figure 2 summarizes the overall performance of SafeDICE and baseline algorithms in RWRL tasks. The results show that SafeDICE significantly outperforms baseline algorithms in all metrics related to safety across all domains (except for CVaR 10% Cost in the Cartpole domain), while not hurting the performance of normalized return. In the case of the Cartpole domain, since there are cases where the cost is unavoidably violated depending on the initial position of cart that is initialized randomly, CVaR 10% Cost which measures the worst 10% performance shows similar performance to other algorithms. We have also evaluated the CVaR k %Cost with various k and are provided in Appendix E.1. PPL outperforms BC on Walker and Quadruped domains but suffers from degeneration issues on the Cartpole domain. DWBC outperforms or matches the other baseline algorithms on all domains, but performs poorly compared to SafeDICE despite the best results obtained through the additional hyperparameter search. Similar to DWBC, DExperts shows results that are sensitive to hyperparameters, and are vulnerable to degeneration issues depending on hyperparameters. Additional experimental results according to various hyperparameters for DWBC and DExperts are provided in Appendix E.2.

Results with varying amount of D N To study the dependence on the amount of labeled non-preferred demonstrations, we experiment with different numbers of labeled non-preferred demonstrations | D N | on Walker domain. (The results are provided in Appendix E.3.) Figure 8 summarizes the performance of SafeDICE and baseline algorithms with varying the number of labeled non-preferred demonstrations ( | D N | ∈ { 50 , 20 , 10 , 5 } ). In this experiment, it is expected that performance in terms of safety gradually degrades as | D N | decreases. As shown in Figure 8, as | D N | decreases, SafeDICE shows a greater difference from other baseline algorithms, significantly outperforming. PPL works similarly to SafeDICE when | D N | is large but shows sensitive performance depending on | D N | . DWBC is not sensitive to | D N | , but it performs poorly compared to SafeDICE. DExperts easily suffers from degeneration issues when | D N | is small and performs poorly even when | D N | is large. These results represent that our algorithm SafeDICE can be applied more effectively in real-world scenarios where labeled non-preferred demonstrations are not provided in large quantities.

5.2 Safety Gym Environment

To show that our algorithm robustly works in various environments, we additionally conduct experiments on Safety Gym[24], which is one of the representative benchmarks for safety in RL.

Task setup We conduct the experiments for Goal and Button tasks from the Safety Gym environment. For each task, the agent aims to maximize task performance while satisfying the constraints by avoiding the dangerous area (see Figure 3). Since there is no standard dataset for offline IL considering

Goal

Button

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

20

40

60

80

100

Normalized Return

Goal (Return)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

20

30

40

50

60

70

Average Cost

Goal (Cost)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Goal (Cost Violation)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

100

150

200

250

CVaR 10% Cost

Goal (CVaR Cost)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

0

20

40

60

80

100

Normalized Return

Button (Return)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

40

60

80

100

Average Cost

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Button (Cost Violation)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

120

140

160

180

200

220

240

CVaR 10% Cost

Button (CVaR Cost)

BC

DWBC

PPL

DExperts

SafeDICE (ours)

the safety constraints, we collected data by training the online RL agents. To obtain non-preferred and preferred policies, we trained PPO [27] and PPO-Lagrangian [24], which is combined with the Lagrangian approach to PPO, respectively. Here, we define the non-preferred policy as a policy that violates the constraint and the preferred policy as a policy that satisfies the constraint. Then, we generated scarce but labeled non-preferred demonstrations D N from the PPO-Lagrangian policy, and abundant but unlabeled demonstrations D U from both PPO and PPO-Lagrangian policies. We collected diverse trajectories by sampling with a stochastic policy from randomly initialized initial states. In this setting, our goal is to learn a safe policy (i.e. preferred policy) that satisfies the cost constraints by avoiding the cost-violating (i.e. non-preferred) demonstrations. The details for the experimental settings can be found in Appendix D.

Results Figure 4 summarizes the overall performance of SafeDICE and baseline algorithms in Safety Gym tasks. The results show that SafeDICE significantly outperforms baseline algorithms in all metrics related to safety across all domains. In the result of the Button task, SafeDICE shows low performance in Return , but it is a desirable result because the non-preferred demonstrations were collected from a safe policy with relatively low returns and very low costs (see Appendix D).

6 Conclusion

In this work, we presented SafeDICE, an offline safe IL algorithm, which can learn safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. We formulated this problem as a stationary distribution matching problem, which can be reduced to a single convex minimization problem without additional hyperparameter search, unlike existing algorithms that are highly sensitive to the hyperparameter. In the experiments, we demonstrated that SafeDICE learns a more safe policy that satisfies the cost constraint in constrained RL benchmarks including RWRL and Safety Gym. As the future work, we will consider an extension to the problem settings where preference information is given as a real value rather than binary value. There are no potential negative societal impacts of our work.

References

B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems , 57(5):469-483, 2009.

S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization . Cambridge university press, 2004.

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. CoRR , abs/1904.06387, 2019. URL http://arxiv.org/abs/1904.06387 .

D. S. Brown, W. Goo, and S. Niekum. Ranking-based reward extrapolation without rankings. CoRR , abs/1907.03976, 2019. URL http://arxiv.org/abs/1907.03976 .

L. Chen, R. R. Paleja, and M. C. Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. CoRR , abs/2010.11723, 2020. URL https://arxiv.org/ abs/2010.11723 .

B. Dai, O. Nachum, Y. Chow, L. Li, C. Szepesvari, and D. Schuurmans. CoinDICE: Offpolicy confidence interval estimation. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 9398-9411, 2020.

G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empirical investigation of the challenges of real-world reinforcement learning. CoRR , abs/2003.11881, 2020. URL https://arxiv.org/abs/2003.11881 .

C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 213-220, 2008.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR , abs/1801.01290, 2018. URL http://arxiv.org/abs/1801.01290 .

L. Ke, S. Choudhury, M. Barnes, W. Sun, G. Lee, and S. Srinivasa. Imitation learning as f-divergence minimization. In International Workshop on the Algorithmic Foundations of Robotics , pages 313-329. Springer, 2020.

G.-H. Kim*, J. Lee*, Y. Jang, H. Yang, and K.-E. Kim. Lobsdice: Offline learning from observation via stationary distribution correction estimation. In Proceedings of Neural Information Processing Systems (NeurIPS) , 12 2022.

G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum? id=BrPdX1bDZkQ .

I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations (ICLR) , 2020.

J. Lee, W. Jeon, B.-J. Lee, J. Pineau, and K.-E. Kim. OptiDICE: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning (ICML) , 2021.

J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations (ICLR) , 2022. URL https://arxiv. org/abs/2204.08957 .

K. Lee, L. M. Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning , pages 6152-6163. PMLR, 2021.

K. Lee, L. M. Smith, A. D. Dragan, and P. Abbeel. B-pref: Benchmarking preference-based reinforcement learning. CoRR , abs/2111.03026, 2021. URL https://arxiv.org/abs/2111. 03026 .

A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 6691-6706, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long. 522. URL https://aclanthology.org/2021.acl-long.522 .

Y. Ma, A. Shen, D. Jayaraman, and O. Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference on Machine Learning , pages 14639-14663. PMLR, 2022.

O. Nachum, Y. Chow, B. Dai, and L. Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. CoRR , abs/1912.02074, 2019. URL http://arxiv.org/ abs/1912.02074 .

A. B. Owen. Monte Carlo theory, methods and examples . 2013.

A. Ray, J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2, 2019.

S. Ross, G. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 627-635, 2011.

S. Schaal et al. Learning from demonstration. Advances in neural information processing systems (NeurIPS) , pages 1040-1046, 1997.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT Press, 1998.

Y.-H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning , pages 6818-6827. PMLR, 2019.

D. Xu and M. Denil. Positive-unlabeled reward learning. CoRR , abs/1911.00459, 2019. URL http://arxiv.org/abs/1911.00459 .

H. Xu, X. Zhan, H. Yin, and H. Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 24725-24742. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/xu22l.html .

M. Yang, O. Nachum, B. Dai, L. Li, and D. Schuurmans. Off-policy evaluation via the regularized lagrangian. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

R. Zhang, B. Dai, L. Li, and D. Schuurmans. GenDICE: Generalized offline estimation of stationary values. In Proceedings of the 8th International Conference on Learning Representations (ICLR) , 2020.

S. Zhang, B. Liu, and S. Whiteson. GradientDICE: Rethinking generalized offline estimation of stationary values. In Proceedings of the 35th International Conference on Machine Learning (ICML) , 2020.

S. Zhang, Z. Cao, D. Sadigh, and Y. Sui. Confidence-aware imitation learning from demonstrations with varying optimality. CoRR , abs/2110.14754, 2021. URL https://arxiv.org/ abs/2110.14754 .

K. Zolna, C. Saharia, L. Boussioux, D. Y. Hui, M. Chevalier-Boisvert, D. Bahdanau, and Y. Bengio. Combating false negatives in adversarial imitation learning. CoRR , abs/2002.00412, 2020. URL https://arxiv.org/abs/2002.00412 .

A Theoretical Analysis

A.1 Closed-form solutions

Proposition 3.1. For any ν , the closed-form solution to the inner maximization of Eq. (11) , i.e. w ∗ ν = arg max w ≥ 0 L ( w,ν ; r ) , is given by:

Proof. For ( s, a ) with d U ( s, a ) > 0 , we can compute the derivative ∂ L ( w,ν ; r ) ∂w ( s,a ) of L ( w,ν ; r ) w.r.t. w ( s, a ) as follows:

A.2 Theoretical analysis of error bound between d P and ˆ d P

Proposition 3.2. Suppose min ( s,a ) d P ( s,a ) d N ( s,a ) ≤ ϵ . If we obtain the estimated stationary distribution by optimizing the Eq. (7) with α set to ˆ α as follows:

then the total variation distance of true and estimated stationary distribution of preferred policy is bounded as follows:

where D TV is total variation distance, d P ( s, a ) is true stationary distribution of the preferred policy and ˆ d P ( s, a ) is estimated stationary distribution which is defined by ˆ α as ˆ d P ( s, a ) = d U ( s,a ) -ˆ αd N ( s,a ) 1 -ˆ α . (Proof in Appendix A.)

Proof. Let, α ∗ is a true ratio between the stationary distribution of expert and anti-expert policy:

Then, ˆ α can be represented by using α ∗ as follows:

The estimated stationary distribution ˆ d P with ˆ α can be derived as follows:

Finally, we can derive the total variation distance of the true and estimated stationary distribution of the expert policy:

A.3 Proof that the reward function is well defined with selected α

Proposition A.1. If we select α as ˆ α = min ( s,a ) d U ( s,a ) d N ( s,a ) , then the value inside the log function in Eq. (10) is always non-negative:

where c ∗ ( s, a ) = d N ( s,a ) d U ( s,a )+ d N ( s,a ) .

Proof. From the α selection rule, ˆ α ≤ d U ( s,a ) d N ( s,a ) for all ( s, a ) . We can reformulate this inequation as d U ( s, a ) -ˆ αd N ( s, a ) ≥ 0 . Then the value inside the log function in Eq. (10) can be derived as follows:

To avoid the case where this value is 0, in practical implementation, we select the α as ˆ α = min ( s,a ) d U ( s,a ) d N ( s,a ) -ϵ , where the ϵ is a small positive value ( ϵ = 10 -5 ) .

B Pseudocode of SafeDICE

Algorithm 1 SafeDICE

Input: A labeled non-preferred demonstrations D N , an unlabeled demonstrations D U , a discriminator c θ with parameter θ , a neural network ν ϕ with parameter ϕ , a policy network π ψ with parameter ψ

1: Pretrain the discriminator by optimizing the Eq. (9):

2: Set the α as ˆ α from Eq. (15) on D U :

3: for each iteration i do

4: Update ν ϕ by optimizing the Eq. (13):

5: Update policy π ψ by weighted behavior cloning of unlabeled demonstrations D U :

6: end for

Output: π ψ ≈ π ∗

C Implementation Details of SafeDICE and Baseline Algorithms

In this section, we provide the implementation details of SafeDICE and baseline algorithms. Our code is available on https://github.com/jys5609/SafeDICE .

C.1 SafeDICE

We implement SafeDICE based on the Real-World RL (RWRL) suite [8] and codebase of DemoDICE [13], which is one of recent DICE-based offline IL algorithms. Similar to DemoDICE [13], due to the instability issue of Eq. (13), we implemented an alternative objective that is more numerically stable while having the same optimal solution:

Then, we estimate Eq. (14) using self-normalized importance sampling [23] as follows:

where ˜ ν ∗ is optimal solution of Eq. (36) and ˜ w ˜ ν ∗ := exp( A ˜ ν ∗ ( s, a )) .

C.2 Discriminator-Weighted Behavior Cloning (DWBC)

We modified DWBC, which adopted positive-unlabeled learning, to be considered as a baseline algorithm in our experimental settings. Similar to positive-unlabeled learning, we adopted negativeunlabeled learning to the objective of discriminator learning as follows:

where d θ is a discriminator with parameter θ , and η is a hyperparameter that corresponds to the proportion of negative samples to unlabeled samples.

Based on the trained discriminator by Eq. (38), we adopted weighted behavior cloning on the unlabeled demonstrations for the policy learning as follows:

C.3 Preference-based Policy Learning (PPL)

We trained the reward function with the following loss function follows from the Bradley-Terry model [3] which is commonly used in preference-based policy learning approaches [4, 18]:

where τ i and τ j represent trajectories, and P = { ( i, j ) : τ i ≻ τ j } . In our problem setting, we adopt this model assuming that unlabeled demonstrations are better than anti-expert demonstrations (i.e. τ i ≻ τ j for all τ i ∈ D U and τ j ∈ D N ).

Based on the trained reward model by Eq. (40), we adopt weighted behavior cloning on the unlabeled demonstrations for the policy learning as follows:

C.4 DExperts

Tabular MDPs For the tabular MDPs, we separately trained unlabeled policy π U and non-preferred policy π N with a behavior cloning model using unlabeled demonstrations and non-preferred demonstrations as follows:

Then, as the final policy, the ensemble of π U and π N at decoding-time is used:

where α is a hyperparameter that controls the degree of leveraging the non-preferred policy.

Continuous MDPs For the continuous MDPs, it is not straightforward to apply DExperts as a decoding-time method. Therefore, we implemented DExperts with the following objective that is similarly considered at training time:

where α is a hyperparameter that controls the degree of leveraging the non-preferred demonstrations.

D Experimental Details

D.1 Experimental details for RWRL suite

Task specification We conduct experiments on the RWRL suite with safety constraints. Table 1 shows the tasks, safety constraint settings, and the information of dataset for each domain that we used in our experiments.

Hyperparameter configurations For a fair comparison, we use the same architecture and learning rate to train the policy networks, discriminators, and reward models of each algorithm. Table 2 summarizes the hyperparameter settings that we used in our experiments. The additional hyperparameters η and α of DWBC and DExperts were selected by the best performance among { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .

D.2 Experimental details for Safety Gym

Task specification We conduct experiments on the Safety Gym with safety constraints. Table 3 shows the tasks, safety constraint settings, and the information of dataset for each domain that we used in our experiments.

Hyperparameter configurations For a fair comparison, we use the same architecture and learning rate to train the policy networks, discriminators, and reward models of each algorithm. Table 4 summarizes the hyperparameter settings that we used in our experiments. The additional hyperparameters η and α of DWBC and DExperts were selected by the best performance among { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } .

Table 4: Configurations of hyperparameters used in our experimental results on Safety Gym environment.

E Additional Results on RWRL suite

E.1 Results of CVaR k %Cost on RWRL suite

0.0M

0.3M

0.6M

0.9M

1.2M

250

275

300

325

350

375

400

425

450

CVaR 50% Cost

Cartpole (CVaR 50% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

360

380

400

420

440

460

CVaR 20% Cost

Cartpole (CVaR 20% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

390

400

410

420

430

440

450

460

470

CVaR 10% Cost

Cartpole (CVaR 10% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

410

420

430

440

450

460

470

CVaR 5% Cost

Cartpole (CVaR 5% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

100

200

300

400

500

600

700

CVaR 50% Cost

Walker (CVaR 50% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

100

200

300

400

500

600

700

CVaR 20% Cost

Walker (CVaR 20% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

100

200

300

400

500

600

700

CVaR 10% Cost

Walker (CVaR 10% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

200

300

400

500

600

700

CVaR 5% Cost

Walker (CVaR 5% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

100

150

200

250

300

350

400

450

CVaR 50% Cost

Quadruped (CVaR 50% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

150

200

250

300

350

400

450

CVaR 20% Cost

Quadruped (CVaR 20% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

150

200

250

300

350

400

450

CVaR 10% Cost

Quadruped (CVaR 10% Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

150

200

250

300

350

400

450

500

CVaR 5% Cost

Quadruped (CVaR 5% Cost)

BC

DWBC

PPL

DExperts

SafeDICE (ours)

E.2 Results of DWBC and DExperts with various hyperparameters on RWRL Walker domain

To study the hyperparameter sensitivity of DWBC and DExperts, we demonstrate the results of DWBC and DExperts with various hyperparameter settings. For DWBC, we compare the performance of DWBCwith different values of η ∈ { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } and DWBC without η (i.e. naive discriminator training without negative-unlabeled learning). As shown in Figure 6, DWBC shows very sensitive performance to hyperparameter η and unstable learning that sometimes even suffers from degeneration issues.

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

20

40

60

80

100

Normalized Return

Return

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

Average Cost

Cost

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 10% Cost

CVaR Cost

DWBC(w/o

)

DWBC(

=0.2)

DWBC(

=0.01)

DWBC(

=0.4)

DWBC(

=0.02)

DWBC(

=0.6)

DWBC(

=0.05)

DWBC(

=0.8)

DWBC(

=0.1)

SafeDICE (ours)

We also compare the performance of DExperts with different values of α ∈ { 0 , 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } . Here, DExperts ( α = 0) is reduced to the naive behavior cloning model on unlabeled demonstrations. As shown in Figure 7, DExperts shows very sensitive performance to hyperparameter α and is vulnerable to degeneration issues.

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

20

40

60

80

100

Normalized Return

Return

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

Average Cost

Cost

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 10% Cost

CVaR Cost

DExperts(

=0)

DExperts(

=0.2)

DExperts(

=0.01)

DExperts(

=0.4)

DExperts(

=0.02)

DExperts(

=0.6)

DExperts(

=0.05)

DExperts(

=0.8)

DExperts(

=0.1)

SafeDICE (ours)

E.3 Results with varying amount of labeled non-preferred demonstrations

0.0M

0.3M

0.6M

0.9M

1.2M

0

20

40

60

80

100

Normalized Return

Return (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

Average Cost

Cost (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 10% Cost

CVaR Cost (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0

20

40

60

80

100

Normalized Return

Return (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

Average Cost

Cost (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 10% Cost

CVaR Cost (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0

20

40

60

80

100

Normalized Return

Return (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

Average Cost

Cost (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 10% Cost

CVaR Cost (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

20

40

60

80

100

Normalized Return

Return (|

D N

| = 5)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

Average Cost

Cost (|

D N

| = 5)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation (|

D N

| = 5)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 10% Cost

CVaR Cost (|

D N

| = 5)

BC

DWBC

PPL

DExperts

SafeDICE (ours)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 50% Cost

CVaR 50% Cost (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 20% Cost

CVaR 20% Cost (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 10% Cost

CVaR 10% Cost (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 5% Cost

CVaR 5% Cost (|

D N

| = 50)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 50% Cost

CVaR 50% Cost (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 20% Cost

CVaR 20% Cost (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 10% Cost

CVaR 10% Cost (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 5% Cost

CVaR 5% Cost (|

D N

| = 20)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 50% Cost

CVaR 50% Cost (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 20% Cost

CVaR 20% Cost (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 10% Cost

CVaR 10% Cost (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

0

200

400

600

800

CVaR 5% Cost

CVaR 5% Cost (|

D N

| = 10)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 50% Cost

CVaR 50% Cost (|

D N

| = 5)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 20% Cost

CVaR 20% Cost (|

D N

| = 5)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 10% Cost

CVaR 10% Cost (|

D N

| = 5)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 5% Cost

CVaR 5% Cost (|

D N

| = 5)

BC

DWBC

PPL

DExperts

SafeDICE (ours)

E.4 Results of SafeDICE with various α

To show the importance of theoretically determining α , we show additional results of SafeDICE with various values of α . According to the value of α , the reward function r α ( s, a ) defined by Eq. (10) cannot guarantee that the value inside the log function always becomes a positive. Therefore, we simply performed learning with a reward function that artificially clipped the value inside the log function of Eq. (10). As shown in Figure 10 and 11, the results show various performance depending on the value of α , and result with α value selected by Eq. (15) shows the best performance.

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

20

40

60

80

100

Normalized Return

Return

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

Average Cost

Cost

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 10% Cost

CVaR Cost

SafeDICE(

=0)

SafeDICE(

=0.2)

SafeDICE(

=0.01)

SafeDICE(

=0.4)

SafeDICE(

=0.02)

SafeDICE(

=0.6)

SafeDICE(

=0.05)

SafeDICE(

=0.8)

SafeDICE(

=0.1)

SafeDICE (ours)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

100

200

300

400

500

600

700

Normalized Return

CVaR 50% Cost

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

Average Cost

CVaR 20% Cost

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

100

200

300

400

500

600

700

Cost-Violating Ratio

CVaR 10% Cost

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

200

400

600

800

CVaR 10% Cost

CVaR 5% Cost

SafeDICE(

=0)

SafeDICE(

=0.2)

SafeDICE(

=0.01)

SafeDICE(

=0.4)

SafeDICE(

=0.02)

SafeDICE(

=0.6)

SafeDICE(

=0.05)

SafeDICE(

=0.8)

SafeDICE(

=0.1)

SafeDICE (ours)

E.5 Impact of the stationary distribution constraint and labeled non-preferred demonstrations

We conducted additional experiments to evaluate the impact of the stationary distribution constraint and labeled non-preferred demonstrations. As can be seen from the Figure 12, the performance is degraded both in the absence of stationary distribution constraints and in the case of not using labeled non-preferred demonstrations.

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0

20

40

60

80

100

Normalized Return

Walker (Return)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

100

200

300

400

500

600

700

Average Cost

Walker (Cost)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Walker (Cost Violation)

0.0M

0.3M

0.6M

0.9M

1.2M

Training Iterations

100

200

300

400

500

600

700

CVaR 10% Cost

Walker (CVaR Cost)

SafeDICE (without labeled non-preferred)

SafeDICE (without stationary distribution constraint)

SafeDICE

E.6 Results with various configurations of unlabeled demonstrations

In order to evaluate our algorithm with more various configurations of unlabeled demonstrations, we also conducted additional experiments for various configurations of unlabeled demonstrations consisting of different ratios of preferred and non-preferred demonstrations ( ( D U P , D U N ) ∈ { (250 , 1000) , (500 , 1000) , (1000 , 500) , (1000 , 250) } , where | D U P | and | D U N | denote the number of preferred and non-preferred demonstrations in D U , respectively). As can be seen from the Figure 13, the results show that SafeDICE outperforms baselines in all settings with various configurations of unlabeled demonstrations. These results show that SafeDICE can be applied more effectively in various real-world scenarios regardless of the preferred and non-preferred ratio of unlabeled demonstrations.

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

20

40

60

80

100

Normalized Return

Return

(|

D U

P

|, |

D U

N

| = 250,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

200

400

600

Average Cost

Cost

(|

D U

P

|, |

D U

N

| = 250,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation

(|

D U

P

|, |

D U

N

| = 250,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

200

400

600

800

CVaR 10% Cost

CVaR Cost

(|

D U

P

|, |

D U

N

| = 250,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

20

40

60

80

100

Normalized Return

Return

(|

D U

P

|, |

D U

N

| = 500,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

200

400

600

Average Cost

Cost

(|

D U

P

|, |

D U

N

| = 500,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation

(|

D U

P

|, |

D U

N

| = 500,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

200

400

600

800

CVaR 10% Cost

CVaR Cost

(|

D U

P

|, |

D U

N

| = 500,1000)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

20

40

60

80

100

Normalized Return

Return

(|

D U

P

|, |

D U

N

| = 1000,500)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

200

400

600

Average Cost

Cost

(|

D U

P

|, |

D U

N

| = 1000,500)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation

(|

D U

P

|, |

D U

N

| = 1000,500)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

0

200

400

600

800

CVaR 10% Cost

CVaR Cost

(|

D U

P

|, |

D U

N

| = 1000,500)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

0

20

40

60

80

100

Normalized Return

Return

(|

D U

P

|, |

D U

N

| = 1000,250)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

0

200

400

600

Average Cost

Cost

(|

D U

P

|, |

D U

N

| = 1000,250)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

0.0

0.2

0.4

0.6

0.8

1.0

Cost-Violating Ratio

Cost Violation

(|

D U

P

|, |

D U

N

| = 1000,250)

0.0M

0.2M

0.4M

0.6M

0.8M

1.0M

Training Iterations

0

200

400

600

800

CVaR 10% Cost

CVaR Cost

(|

D U

P

|, |

D U

N

| = 1000,250)

BC

DWBC

PPL

DExperts

SafeDICE (ours)

F Comparison with constrained RL algorithm

To compare with the safe RL algorithm as an oracle agent, we conducted additional experiments for an offline-constrained RL method. We run COptiDICE [16], the state-of-the-art offline constrained RL algorithm, using the datasets that are augmented with additional (ground-truth) reward and cost annotations. Table 5-9 summarize overall performance of SafeDICE and baseline algorithms including COptiDICE. The results show that SafeDICE is competitive with (or even outperforms in some domains) the constrained offline RL algorithm COptiDICE, even though SafeDICE uses only a much smaller amount of annotation information.

F.1 Experimental results on RWRL control tasks

F.2 Experimental results on Safety Gym control tasks

G Tabular MDPs Results

In order to show the motivation of the paper and the limitation of existing algorithms through illustrative examples, we conducted the experiment in tabular MDPs, which extends the Four Rooms environment [28, 22].

Unlabeled (

𝑫 𝑼

)

Preferred (

𝟏 - 𝜶

)

Non-preferred (

𝜶

)

=

+

Non-preferred (

𝑫 𝑵

)

SafeDICE

DWBC (

𝜼

=0.1)

DWBC (

𝜼

=0.2)

DWBC (

𝜼

=0.4)

DWBC (

𝜼

=0.6)

DWBC (

𝜼

=0.8)

BC

PPL

DExperts (

𝜶

=0.1)

DExperts (

𝜶

=0.2)

DExperts (

𝜶

=0.4)

DExperts (

𝜶

=0.6)

DExperts (

𝜶

=0.8)

++

F+

++

++

Task setup We consider the Six Rooms environment, a variant of the Four Rooms environment [28, 22], in which an agent aims to navigate to a goal location within a gridworld. Figure 14 demonstrates our experimental settings on the Six Rooms environment. The orange and green squares denote the initial state and goal state, respectively. The arrows for each action at the square represent the policy, where the opacity of the arrow is determined by the probability π ( a | s ) . The opacity of each square represents the state marginals of each stationary distribution. In order to compose the task naturally, we assume that the preferred and non-preferred policies have both similar and different behaviors in some state-action spaces. We define the policy that reaches the goal state from the initial state through the downward direction as an non-preferred policy, which represents constraint-violating behavior, and the policy that reaches the goal state through the right direction as an preferred policy. In this task, the agent aims to learn the policy that recovers the preferred policy from unlabeled demonstrations while avoiding the constraint-violating behavior using labeled nonpreferred demonstrations. Figure 14 shows a given dataset to the agent for learning, which consists of scarce but labeled non-preferred demonstrations D N and abundant but unlabeled demonstrations

D U from both preferred and non-preferred policy. The details for the experimental settings can be found in Appendix D.

Results Figure 15 summarizes the qualitative results of SafeDICE and baseline algorithms on Six Rooms environment. Naive BC on unlabeled demonstrations follows both preferred and non-preferred behavior since it does not consider avoiding the non-preferred demonstrations. On the other hand, PPL and SafeDICE show safe behaviors in which non-preferred demonstrations are completely avoided while following preferred demonstrations. For DWBC and DExperts that require additional hyperparameter search, we show the behaviors of learned policies with different hyperparameter settings. Among various hyperparameter settings, there are cases where an agent learns a policy close to an preferred behavior at a specific hyperparameter. However, the results show that both DWBC and DExperts are hyperparameter sensitive, which is unsuitable for real-world tasks where the deployment is costly or risky. In addition, DExperts not only have cost-violating behaviors but also cause degeneration issues that fall into abnormal behaviors that are completely different from given demonstrations. To show that SafeDICE robustly works in various problem settings, we also provide additional results including cases where the distributions of labeled and unlabeled nonpreferred demonstrations are dissimilar (see Appendix G.1), and more complex environments (see Appendix G.2). As shown in Appendix G.1, SafeDICE learns a safe policy in which non-preferred behaviors are completely avoided while following preferred behavior, even when the distributions of labeled and unlabeled non-preferred demonstrations are dissimilar.

G.1 Experimental results with the dissimilar distribution of labeled and unlabeled non-preferred demonstrations

To show that SafeDICE works robustly, we conduct the experiment considering a problem setting more similar to real-world tasks, where the distributions of unlabeled and labeled non-preferred demonstrations are dissimilar. In this experimental setting, we define the non-preferred policy is a multimodal policy that consists of a mixture of two different policies π A 1 and π A 2 . We assume that non-preferred demonstrations of labeled and unlabeled are sampled from policies π A 1 and π A 2 with different ratios. As shown in Figure 16, baseline algorithms fail to completely recover the preferred behavior or follow the non-preferred behavior. On the other hand, SafeDICE shows safe behavior in which non-preferred behaviors are completely avoided while following preferred behavior, even when the distributions of labeled and unlabeled non-preferred demonstrations are dissimilar.

Unlabeled (

𝑫 𝑼

)

Preferred (

𝟏 - 𝜶

)

Non-preferred (

𝜶

)

=

+

Non-preferred (

𝑫 𝑵

)

True preferred (

𝝅 𝑷

)

True non- preferred (

)

True non-preferred (

𝝅 𝑵𝟐

)

E

SafeDICE

DWBC (

𝜼

=0.1)

DWBC (

𝜼

=0.2)

DWBC (

𝜼

=0.4)

DWBC (

𝜼

=0.6)

DWBC (

𝜼

=0.8)

BC

PPL

DExperts (

𝜶

=0.1)

DExperts (

𝜶

=0.2)

DExperts (

𝜶

=0.4)

DExperts (

𝜶

=0.6)

DExperts (

𝜶

=0.8)

书

++

353

G.2 Additional results for various Tabular MDPs

Unlabeled (

𝑫 𝑼

)

Preferred (

𝟏 - 𝜶

)

Non-preferred (

𝜶

)

=

+

Non-preferred (

𝑫 𝑵

)

SafeDICE

DWBC (

𝜼

=0.1)

DWBC (

𝜼

=0.2)

DWBC (

𝜼

=0.4)

DWBC (

𝜼

=0.6)

DWBC (

𝜼

=0.8)

BC

PPL

DExperts (

𝜶

=0.1)

DExperts (

𝜶

=0.2)

DExperts (

𝜶

=0.4)

DExperts (

𝜶

=0.6)

DExperts (

𝜶

=0.8)

Unlabeled (

𝑫 𝑼

)

Preferred (

𝟏 - 𝜶

)

Non-preferred (

𝜶

)

=

+

Non-preferred (

𝑫 𝑵

)

SafeDICE

DWBC (

𝜼

=0.1)

DWBC (

𝜼

=0.2)

DWBC (

𝜼

=0.4)

DWBC (

𝜼

=0.6)

DWBC (

𝜼

=0.8)

BC

PPL

DExperts (

𝜶

=0.1)

DExperts (

𝜶

=0.2)

DExperts (

𝜶

=0.4)

DExperts (

𝜶

=0.6)

DExperts (

𝜶

=0.8)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an offline safe imitation learning algorithm called SafeDICE. A unique point of SafeDICE is that a safe policy is learned by non-preferred and unlabeled demonstrations in the imitation learning framework. Based on the formulations studied in DICE family, this paper formulated the safe IL problem as a stationary distribution matching problem, and then solve a single convex minimization problem without additional hyper-parameter search. In the experiments using RWRL and Safety-Gym benchmarks, the authors demonstrate the effectiveness of the SafeDICE algorithms and show that a better policy can be learned.\n\nSTRENGTHS:\n- This paper is well-written and easy to understand. I think the presentation of the ideas is clear and concise.\n\n- The problem setting is well-motivated with a timely example (i.e., chatbot in lines 30-32). I agree that the problem setting is actually important and unexplored.\n\n- The empirical evaluation of this paper is great. I think the empirical evaluation is sufficient because the authors demonstrated the effectiveness of the proposed SafeDICE algorithm in two environments with comparison with reasonable baselines.\n\nWEAKNESSES:\n- It is a little bit unclear what is the true mathematical contributions. As the authors describes as the following, but it seems that the most of the mathematical formulations are based on the DICE family papers and the contributions of this paper is a little bit minor.\n> The main flow of our derivation follows prior DICE-based offline imitation learning methods [12, 13], but it is clearly different from the existing methods that require hyper-parameter search to deal with unknown $\\alpha$.\n\n- Though the authors demonstrated that the SafeDICE performs better than other baselines, the performance itself seems low if I understand correctly. And, since return is normalized, it is unclear even whether or not the tasks are solved. I think it would be better to add a typical Safe RL algorithm (e.g., CPO, TRPO-Lagrangian) as an oracle agent. Note that, I do *not* require the SafeDICE to perform better than Safe RL algorithms.\n\nQUESTIONS:\n[Q1] What is the actual value of the return without normalization? Did the authors confirm that the tasks are successfully solved?\n\n[Q2] What is the threshold of the Safety Gym? Is it a default parameter (i.e., 20)?\n\n[Q3] I think it is impressive that SafeDICE does not require a hyper-parameter search. Though I agree that (15) is a beautiful and useful equation, do we have some chance to obtain a better $\\alpha$?\n\nLIMITATIONS:\n- Unless I miss something, limitations have not adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLearning safe behaviors from a dataset of demonstrations is a challenging problem. The work presents SafeDICE, an algorithm which learns a safe policy using preference-based imitation learning. The method leverages non-preferred demonstrations in the space of stationary distributions in contrast to prior methods which operate in the policy and discriminator space. SafeDICE reduces the constrained optimization problem into a single convex minimization objective by constructing a Lagrangian and eliminating the maximization problem using a closed form solution. The new objective does not require additional hyperparameter search. Empirical performance demonstrates that SafeDICE is competitive to other methods in satisfying cost constraints.\n\nSTRENGTHS:\n* The paper is well motivated and within the scope of safe Imitation Learning.\n* The proposed method presents a novel stationary distribution formulation of the Lagrangian constraints which, to the best of my knowledge, has not been observed before.\n\nWEAKNESSES:\n* **Choice of Baselines:** My main concern is the choice of baselines used to compare SafeDICE. The paper only compares with naive BC, DWBC, PPL and DExperts. BC and DWBC are simple imitation learning algorithms which were not designed for constrained optimization problems. PPL utilizes a paired ranking scheme based on reward preferences which is tangential to imitation learning. This leaves DExperts as the only suitable baseline. There exist specific constrained optimization methods such as CPO, PPO-Lagrangian, TRPO and LAMBDA albeit for online and model-based settings [1]. Additionally, there exist offline RL methods [2, 3] which demonstrate safe behaviors as a result of their pessimistic nature. Authors should include these as relevant baselines on small tasks or provide a concrete explanation for why generic Imitation Learning baselines are suitable for constraint satisfaction.\n* **Dataset Comparison:** Due to the absence of safe offline IL datasets, the work constructs its own dataset based on preferred and non-preferred demonstrations. My concerns are around the design of the dataset. The paper does not provide a detailed breakdown of the dataset and the number of safe and unsafe actions. Tables 1 and 3 show the number of non-preferred demonstrations which are significantly less in comparison to the preferred demonstrations (50 non-preffered demonstrations against 1000 preferred demonstrations). While the authors make an attempt to compare algorithms on different dataset sizes in Figure 8, these comparisons are still well in the low data regime. In my opinion, the dataset is heavily biased and is not reflective of other benchmarks and real-world applications in safety. The paper could evaluate algorithms on different kinds of datasets with different behavior qualities. For instance, datasets could be constructed from replay buffers of trained lagrangian agents perturbed with gaussian noise or a balanced mix of safe and unsafe actions based on constraint satisfaction.\n* **Empirical Evaluation:** Experiments do not comprehensively evaluate the efficacy of the reduced convex minimization problem. The central contribution of the paper is estimating the stationary distribution corrections of the safe policy. This could be evaluated by ablating the Lagrangian with a constraint in which the stationary distributions are absent or non-preferred demonstrations are not utilized. In its current form, empirical evaluation does not throw light on the performance or utility of the proposed objective.\n* **Clarity of Derivation:** I am having trouble following the main derivation of Lagrangian $\\mathcal{L}(w, \\nu ; r_{\\alpha})$. Specifically, it would be helpful if authors could explain how they reached step 3 to obtain $w(s,a)$ and $r(s,a)$. It might be helpful to cover intermediate steps in the Appendix or provide a short note as footnote for the reader.\n\nQUESTIONS:\n* Can you please provide a concrete explanation for comparing SafeDICE to generic Imitation Learning baselines? Can you compare SafeDICE to relevant constrained optimization baselines such as CPO, PPO-Lagrangian, TRPO and LAMBDA on small toy tasks?\n* What is the breakdown of the dataset? How many safe and unsafe actions does the data consist of? Do samples have cost variables as well? Can you please explain the reason for a lower number of non-preferred demonstrations? How does SafeDICE perform when number of non-preferred demonstrations are increased? Can the evaluation accomodate different kind of datasets?\n* How does SafeDICE perform in the absence of stationary distribution constraint? How does the performance vary in the absence of preferred/non-preferred demonstrations?\n* Can you please explain how $w(s,a)$ and $r(s,a)$ were obtained?\n\n## Minors\n\n* line 9: learns safe -> learns a safe\n* line 43: what does DWBC stand for?\n* line 49: what is the degeneration issue?\n* line 52: learns safe policy -> learns a safe policy\n* line 65: I believe the formulation is a Constrained MDP (CMDP) since the problem setting has cost constraints of the form $\\mathcal{C}: S \\times A \\rightarrow \\mathbb{C}$\n* line 89: Recent offline -> A recent offline\n* line 149: optimization of Eq. 7\n* Proposition 3.2: what is $\\epsilon$?\n* line 258: different two -> two different\n\n[1]. As et. al., Constrained Policy Optimization via Bayesian World Models, ICLR 2022.  \n[2]. Kumar et. al., Conservative Q-Learning for Offline Reinforcement Learning, NeurIPS 2021.  \n[3]. Bhardhwaj et. al., Conservative Safety Critics for Exploration, ICLR 2021.\n\nLIMITATIONS:\nThe paper would benefit from a discussion on limitations. Authors could highlight the gap in reward-cost trade off and the increasing number of non-preferred demonstrations as potential limitations of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes an offline safe imitation method to avoid non-preferred demonstrations. The mixture coefficient is alpha, and the paper provides an effective way to obtain hyperparameter instead of hyperparameter search techniques. The paper uses scarce but labeled non-preferred demonstrations from the non-preferred policy.

STRENGTHS:
The paper models the stationary distributions as a mixture of preferred policy and non-preferred policy. The mixture coefficient is alpha, and the paper provides an effective way to obtain hyperparameter instead of hyperparameter search techniques.

The paper is easy to follow and understand.

Claims made at the end of the introduction are supported by experimental results.

WEAKNESSES:
The problem setup is unclear to me. In the offline imitation learning setting, there's a line of research consider robust imitation learning (such as [1]) to discard non-preferred samples. The authors may need to discuss why labeled non-preferred demonstrations is nececssary in this situation.

Furthermore, more discussion can be added to explain why SafeDICE is not sensitive to the number of labeled non-preferred demonstrations.

[1] Robust Imitation Learning from Corrupted Demonstrations, https://arxiv.org/abs/2201.12594

QUESTIONS:
Since the additional cost of hyperparameter search techniques have influence on the computational cost rather than final performance, it is not clear why SafeDICE performs better with the  obtained by Eq. 15. Can you give more explanations about the results between SafeDICE and DWBC?

LIMITATIONS:
As stated above, I would like to compare this methods with those robust imitation learning methods in the offline setting.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on offline safe imitation learning (IL) setting. There are several unique properties of the problem setting: 1) there exists non-preferred (constraint violated) demonstrations, 2) there exists massive unlabelled data where you don't know whether they are preferred or non-preferred, 3) you don't have reward function and interactive environment.\n\nThe proposed SafeDICE method builds upon DIstribution Correction Estimation (DICE), but addresses several challenges. The first is how to takes non-preferred demonstrations into account? To do so, we assume an underlying unknown mix ratio alpha and the unlabelled dataset is the mixed dataset of non-preferred and preferred demonstrations. Doing such assumption allows us to follow DICE workflow. At the same time, we can use a preference-based reward function to estimate the stationary distribution d(s, a) of a state-action pair. The reward function is similar to those used in inverse RL and can be achieved by a classifier discriminating whether a datapoint is from non-preferred dataset or not. The authors derives a closed-form solution to avoid the training instability and provide an approximation to the unknown mix ratio to avoid hyperparameter search.\n\nExperiments show the proposed method can achieve good performance while greatly reduce the constraint violation in Real-World RL suite and SafetyGym.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. The proposed method is solid, clear and looks sound to me.\n3. The problem setting is novel and important.\n\nWEAKNESSES:\n1. The experiment showcases the superior performance of proposed method. But more can be added to study the particular behavior of the proposed method in different settings, such as when you have lot of / scare non-preferred demonstrations.\n\nQUESTIONS:\n1. Some ablation studies should be added to support the claim. For example, what if we don't use the approximation of mix ratio alpha but really do a hyperparameter search? \n2. Can we have an experiment/discussion section discuss the impact of labelled non-preferred data? Like, if we have plenty of them and if we have very few those data, what would happen?\n3. Can we compare/discuss to offline RL method where you can have reward function? In offline RL the data is not always preferred as the author discussed for those offline IL method. So I wonder offline RL shares some settings in current problem. When comparing online Safe RL method, a competitive baseline is to use online RL with reward shaping (the constraint violation can be considered as a negative reward). Would this method also be competitive in offline Safe RL setting?\n\nLIMITATIONS:\nIt seems that there is no much effort put in the paper to discuss the limitations. One I can think of is the collection of non-preferred data is in some sense still dangerous. The \"non-preferred data efficiency\" is not discussed in the paper yet. That is, how efficient the proposed method can learn from how much non-preferred data. This is also important since as the authors said collecting those data is costly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}