SlowGuess/NeurIPS_2023_OCR huggingface dataset:
both algorithms require ground-truth labels for uniformly sampled trajectories from unlabeled demonstrations. DWBC [31] uses positive-unlabeled learning to train a discriminator, which is then used to construct weights for weighted BC. DExperts [19], a decoding-time method in the natural language processing domain, combines models for preferred and non-preferred demonstrations. Unfortunately, both DWBC and DExperts may not be suitable for offline safe imitation learning tasks due to their sensitivity to hyperparameter settings.
In this section, we show the experimental results on various tasks from constrained RL benchmarks. First, we conduct the evaluation on domains from Real-World RL (RWRL) suite [8], which provides challenges in real-world scenarios including safety constraints. Second, we also conduct the experiment on continuous control tasks from Safety Gym environment [24], which provides practical scenarios of safety issues. In order to show the motivation of our work and the limitation of existing algorithms, we also provide qualitative results on tabular MDP environments. The illustrative examples and qualitative analysis on tabular MDP environments can be found in Appendix G.
Baselines Since offline safe IL, which we consider in this paper, has not been explored yet much, it lacks directly applicable baseline algorithms. Therefore, we consider the following four algorithms as baselines. First, BC denotes simple behavior cloning on unlabeled demonstrations. Second, DWBC denotes a variant of the original discriminator-weighted behavior cloning algorithm [31] that adopts the positive-unlabeled learning for training the discriminator. To apply it to our setting, we modified positive-unlabeled learning to negative-unlabeled learning. Third, PPL denotes a preference-based policy learning algorithm that adopts the preference-based reward learning algorithm based on the Bradley-Terry model [3] and weighted behavior cloning for policy learning. Lastly, we also compare with DExperts [19], a decoding-time method that combines trained models with unlabeled and non-preferred demonstrations. The details of baseline algorithms are provided in Appendix C.
Evaluation metrics We compare the performance of SafeDICE and baseline algorithms in terms of task performance (i.e. expected return) and safety (i.e. cost constraints). We report the results with the following metrics: 1) Normalized Return: evaluates the normalized task performance of the policy, 2) Average Cost: evaluates the mean cost of the policy, 3) Cost-Violating Ratio: evaluates the ratio of cost-violating behavior of the policy, 4) Conditional Value at Risk performance (CVaR)  $10\%$  Cost: evaluates the mean cost of the worst  $10\%$  runs. We also provide CVaR  $k\%$  Cost for various values of  $k$  in Appendix E.1. Here, our main focus of the comparison is to evaluate the safety (Average Cost, Cost-Violating Ratio, and CVaR  $10\%$  Cost), and we use the Normalized Return to examine the degeneration issue, whether the agents fall into abnormal behaviors or not. All metrics are measured as the average value of 500 trajectories generated from the learned policy, and we report the results that are averaged over 5 runs.
5.1 Real-World Reinforcement Learning (RWRL) Suite
Now, we present the results of SafeDICE on a set of continuous control benchmarks from Real-World RL (RWRL) suite [8], where the safety constraints are also provided by environments.
Task setup We conduct the experiments for three tasks from RWRL environment with safety constraints: Cartpole, Walker, and Quadruped. Since there is no standard dataset for offline IL considering the safety constraints, we collected data by training the online RL agents. To obtain an preferred and non-preferred policy, we trained the SAC [10] in different two ways that solve the RL problem with respect to reward penalized by the cost or not. Among these two policies which are both optimal in terms of return maximization, we define the preferred policy as a policy that satisfies the constraint (i.e. high-return-low-cost behavior) and the non-preferred policy as a policy that violates the constraint (i.e. high-return-high-cost behavior). This experimental setting is representative of many real-world scenarios, where the data is collected by high-return behavior without consideration of the cost. Then, we generated scarce but labeled non-preferred demonstrations  $D^{N}$  from the non-preferred policy, and abundant but unlabeled demonstrations  $D^{U}$  from both preferred and non-preferred policies. We collected diverse trajectories by sampling with a stochastic policy from randomly initialized initial states. Our goal is to recover the low-cost behavior by avoiding the high-cost demonstrations while not hurting the high-return behavior of unlabeled demonstrations. The details for the experimental settings can be found in Appendix D.
Figure 2: Experimental results on RWRL control tasks. All results are averaged over 5 runs, and the shaded area represents the standard error.
Main results Figure 2 summarizes the overall performance of SafeDICE and baseline algorithms in RWRL tasks. The results show that SafeDICE significantly outperforms baseline algorithms in all metrics related to safety across all domains (except for  $CVaR10\%$  Cost in the Cartpole domain), while not hurting the performance of normalized return. In the case of the Cartpole domain, since there are cases where the cost is unavoidably violated depending on the initial position of cart that is initialized randomly,  $CVaR10\%$  Cost which measures the worst  $10\%$  performance shows similar performance to other algorithms. We have also evaluated the  $CVaRk\%$  Cost with various  $k$  and are provided in Appendix E.1. PPL outperforms BC on Walker and Quadruped domains but suffers from degeneration issues on the Cartpole domain. DWBC outperforms or matches the other baseline algorithms on all domains, but performs poorly compared to SafeDICE despite the best results obtained through the additional hyperparameter search. Similar to DWBC, DExperts shows results that are sensitive to hyperparameters, and are vulnerable to degeneration issues depending on hyperparameters. Additional experimental results according to various hyperparameters for DWBC and DExperts are provided in Appendix E.2.
Results with varying amount of  $D^{N}$  To study the dependence on the amount of labeled non-preferred demonstrations, we experiment with different numbers of labeled non-preferred demonstrations  $|D^{N}|$  on Walker domain. (The results are provided in Appendix E.3.) Figure 8 summarizes the performance of SafeDICE and baseline algorithms with varying the number of labeled non-preferred demonstrations  $|D^{N}| \in \{50,20,10,5\}$ . In this experiment, it is expected that performance in terms of safety gradually degrades as  $|D^{N}|$  decreases. As shown in Figure 8, as  $|D^{N}|$  decreases, SafeDICE shows a greater difference from other baseline algorithms, significantly outperforming. PPL works similarly to SafeDICE when  $|D^{N}|$  is large but shows sensitive performance depending on  $|D^{N}|$ . DWBC is not sensitive to  $|D^{N}|$ , but it performs poorly compared to SafeDICE. DExperts easily suffers from degeneration issues when  $|D^{N}|$  is small and performs poorly even when  $|D^{N}|$  is large. These results represent that our algorithm SafeDICE can be applied more effectively in real-world scenarios where labeled non-preferred demonstrations are not provided in large quantities.
5.2 Safety Gym Environment
To show that our algorithm robustly works in various environments, we additionally conduct experiments on Safety Gym[24], which is one of the representative benchmarks for safety in RL.
Task setup We conduct the experiments for Goal and Button tasks from the Safety Gym environment. For each task, the agent aims to maximize task performance while satisfying the constraints by avoiding the dangerous area (see Figure 3). Since there is no standard dataset for offline IL considering
Figure 3: Tasks of Safety Gym environment. For each task, the agent (red robot) aims to achieve the objective while satisfying the cost constraints. The agents receive costs when entering hazards indicated by blue circles or when touching the obstacles. The objectives of each task are as follows: Goal: Move to a series of green goal positions Button: Press a series of highlighted goal buttons.
Figure 4: Experimental results on Safety Gym control tasks. All results are averaged over 5 runs, and the shaded area represents the standard error.
the safety constraints, we collected data by training the online RL agents. To obtain non-preferred and preferred policies, we trained PPO [27] and PPO-Lagrangian [24], which is combined with the Lagrangian approach to PPO, respectively. Here, we define the non-preferred policy as a policy that violates the constraint and the preferred policy as a policy that satisfies the constraint. Then, we generated scarce but labeled non-preferred demonstrations  $D^{N}$  from the PPO-Lagrangian policy, and abundant but unlabeled demonstrations  $D^{U}$  from both PPO and PPO-Lagrangian policies. We collected diverse trajectories by sampling with a stochastic policy from randomly initialized initial states. In this setting, our goal is to learn a safe policy (i.e. preferred policy) that satisfies the cost constraints by avoiding the cost-violating (i.e. non-preferred) demonstrations. The details for the experimental settings can be found in Appendix D.
Results Figure 4 summarizes the overall performance of SafeDICE and baseline algorithms in Safety Gym tasks. The results show that SafeDICE significantly outperforms baseline algorithms in all metrics related to safety across all domains. In the result of the Button task, SafeDICE shows low performance in Return, but it is a desirable result because the non-preferred demonstrations were collected from a safe policy with relatively low returns and very low costs (see Appendix D).
In this work, we presented SafeDICE, an offline safe IL algorithm, which can learn safe policy by leveraging the non-preferred demonstrations in the space of stationary distributions. We formulated this problem as a stationary distribution matching problem, which can be reduced to a single convex minimization problem without additional hyperparameter search, unlike existing algorithms that are highly sensitive to the hyperparameter. In the experiments, we demonstrated that SafeDICE learns a more safe policy that satisfies the cost constraint in constrained RL benchmarks including RWRL and Safety Gym. As the future work, we will consider an extension to the problem settings where preference information is given as a real value rather than binary value. There are no potential negative societal impacts of our work.
[34] S. Zhang, B. Liu, and S. Whiteson. GradientDICE: Rethinking generalized offline estimation of stationary values. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2020.
[35] S. Zhang, Z. Cao, D. Sadigh, and Y. Sui. Confidence-aware imitation learning from demonstrations with varying optimality. CoRR, abs/2110.14754, 2021. URL https://arxiv.org/abs/2110.14754.
[36] K. Zolna, C. Sahara, L. Boussioux, D. Y. Hui, M. Chevalier-Boisvert, D. Bahdanau, and Y. Bengio. Combating false negatives in adversarial imitation learning. CoRR, abs/2002.00412, 2020. URL https://arxiv.org/abs/2002.00412.
A.1 Closed-form solutions
Proof. For  $(s,a)$  with  $d^{U}(s,a) > 0$ , we can compute the derivative  $\frac{\partial\mathcal{L}(w,\nu;r)}{\partial w(s,a)}$  of  $\mathcal{L}(w,\nu;r)$  w.r.t.  $w(s,a)$  as follows:
$$
\begin{array}{l} \frac {\partial \mathcal {L} (w , \nu ; r)}{\partial w (s , a)} = \sum_ {(s, a)} d ^ {U} (s, a) \bigg (A _ {\nu} (s, a) - \log w (s, a) + w (s, a) \cdot - \frac {1}{w (s , a)} \bigg) = 0 \\ \Leftrightarrow A _ {\nu} (s, a) - \log w (s, a) - 1 = 0 \\ \Leftrightarrow \log w (s, a) = A _ {\nu} (s, a) - 1 \\ \Leftrightarrow w (s, a) = \exp \left(A _ {\nu} (s, a) - 1\right). \\ \end{array}
$$
A.2 Theoretical analysis of error bound between  $d^{P}$  and  $\hat{d}^P$
Proof. Let,  $\alpha^{*}$  is a true ratio between the stationary distribution of expert and anti-expert policy:
$$
d ^ {U} (s, a) = \left(1 - \alpha^ {*}\right) d ^ {P} (s, a) + \alpha^ {*} d ^ {N} (s, a). \tag {18}
$$
Then,  $\hat{\alpha}$  can be represented by using  $\alpha^{*}$  as follows:
$$
\begin{array}{l} \hat {\alpha} = \min  _ {(s, a)} \frac {d ^ {U} (s , a)}{d ^ {N} (s , a)} (19) \\ = \min  _ {(s, a)} \frac {\left(1 - \alpha^ {*}\right) d ^ {P} (s , a) + \alpha^ {*} d ^ {N} (s , a)}{d ^ {N} (s , a)} (20) \\ = \alpha^ {*} + (1 - \alpha^ {*}) \min  _ {(s, a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}. (21) \\ \end{array}
$$
The estimated stationary distribution  $\hat{d}^P$  with  $\hat{\alpha}$  can be derived as follows:
$$
\begin{array}{l} \hat {d} ^ {P} (s, a) = \frac {d ^ {U} (s , a) - \hat {\alpha} d ^ {N} (s , a)}{1 - \hat {\alpha}} (22) \\ = \frac {d ^ {U} (s , a) - \left(\alpha^ {*} + (1 - \alpha^ {*}) \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}\right) d ^ {N} (s , a)}{1 - \alpha^ {*} + (1 - \alpha^ {*}) \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (23) \\ = \frac {\left(1 - \alpha^ {*}\right) d ^ {P} (s , a) - \left(1 - \alpha^ {*}\right) \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)} d ^ {N} (s , a)}{\left(1 - \alpha^ {*}\right) + \left(1 - \alpha^ {*}\right) \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (24) \\ = \frac {d ^ {P} (s , a) - \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)} d ^ {N} (s , a)}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (25) \\ = d ^ {P} (s, a) - \frac {\min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)} \left(d ^ {P} (s , a) + d ^ {N} (s , a)\right)}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (26) \\ = d ^ {P} (s, a) - \frac {\min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)} \left(d ^ {P} (s , a) + d ^ {N} (s , a)\right) + \left(d ^ {P} (s , a) + d ^ {N} (s , a)\right) - \left(d ^ {P} (s , a) + d ^ {N} (s , a)\right)}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (27) \\ = d ^ {P} (s, a) - \frac {\left(1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}\right) \left(d ^ {P} (s , a) + d ^ {N} (s , a)\right) - \left(d ^ {P} (s , a) + d ^ {N} (s , a)\right)}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (28) \\ = - d ^ {N} (s, a) + \frac {d ^ {P} (s , a) + d ^ {N} (s , a)}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (29) \\ \end{array}
$$
Finally, we can derive the total variation distance of the true and estimated stationary distribution of the expert policy:
$$
\begin{array}{l} D _ {\mathrm {T V}} \left(d ^ {P}, \hat {d} ^ {P}\right) = \frac {1}{2} \left\| d ^ {P} - \hat {d} ^ {P} \right\| _ {1} (30) \\ = \frac {1}{2} \sum_ {(s, a)} \left| d ^ {P} (s, a) + d ^ {N} (s, a) - \frac {d ^ {P} (s , a) + d ^ {N} (s , a)}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} \right| (31) \\ = \frac {1}{2} \sum_ {(s, a)} \left(d ^ {P} (s, a) + d ^ {N} (s, a) - \frac {d ^ {P} (s , a) + d ^ {N} (s , a)}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}}\right) (32) \\ = \frac {1}{2} \left(\frac {\min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}}\right) \sum_ {(s, a)} \left(d ^ {P} (s, a) + d ^ {N} (s, a)\right) (33) \\ = \frac {\min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}}{1 + \min  _ {(s , a)} \frac {d ^ {P} (s , a)}{d ^ {N} (s , a)}} (34) \\ \leq \frac {\epsilon}{1 + \epsilon}. (35) \\ \end{array}
$$
A.3 Proof that the reward function is well defined with selected  $\alpha$
Proposition A.1. If we select  $\alpha$  as  $\hat{\alpha} = \min_{(s,a)}\frac{d^U(s,a)}{d^N(s,a)}$ , then the value inside the log function in Eq. (10) is always non-negative:
$$
\frac {1 - (1 + \alpha) c ^ {*} (s , a)}{(1 - \alpha) (1 - c ^ {*} (s , a))} \geq 0,
$$
$$
w h e r e c ^ {*} (s, a) = \frac {d ^ {N} (s , a)}{d ^ {U} (s , a) + d ^ {N} (s , a)}.
$$
Proof. From the  $\alpha$  selection rule,  $\hat{\alpha} \leq \frac{d^U(s,a)}{d^N(s,a)}$  for all  $(s,a)$ . We can reformulate this inequation as  $d^U(s,a) - \hat{\alpha} d^N(s,a) \geq 0$ . Then the value inside the log function in Eq. (10) can be derived as follows:
$$
\begin{array}{l} \frac {1 - (1 + \alpha) c ^ {*} (s , a)}{(1 - \alpha) (1 - c ^ {*} (s , a))} = \frac {1 - (1 + \alpha) \frac {d ^ {N} (s , a)}{d ^ {U} (s , a) + d ^ {N} (s , a)}}{(1 - \alpha) (1 - \frac {d ^ {N} (s , a)}{d ^ {U} (s , a) + d ^ {N} (s , a)})} \\ = \frac {1 - (1 + \alpha) \frac {d ^ {N} (s , a)}{d ^ {U} (s , a) + d ^ {N} (s , a)}}{(1 - \alpha) \frac {d ^ {U} (s , a)}{d ^ {U} (s , a) + d ^ {N} (s , a)}} \\ = \frac {d ^ {U} (s , a) + d ^ {N} (s , a) - (1 + \alpha) d ^ {N} (s , a)}{(1 - \alpha) d ^ {U} (s , a)} \\ = \frac {d ^ {U} (s , a) - \alpha d ^ {N} (s , a)}{(1 - \alpha) d ^ {U} (s , a)} \\ = \frac {d ^ {U} (s , a) - \alpha d ^ {N} (s , a)}{(1 - \alpha) d ^ {U} (s , a)} \\ \geq 0 \\ \end{array}
$$
To avoid the case where this value is 0, in practical implementation, we select the  $\alpha$  as  $\hat{\alpha} = \min_{(s,a)}\frac{d^U(s,a)}{d^N(s,a)} -\epsilon$ , where the  $\epsilon$  is a small positive value ( $\epsilon = 10^{-5}$ ).
Algorithm 1 SafeDICE
Input: A labeled non-preferred demonstrations  $D^{N}$ , an unlabeled demonstrations  $D^{U}$ , a discriminator  $c_{\theta}$  with parameter  $\theta$ , a neural network  $\nu_{\phi}$  with parameter  $\phi$ , a policy network  $\pi_{\psi}$  with parameter  $\psi$
1: Pretrain the discriminator by optimizing the Eq. (9):
$$
\theta^ {*} = \underset {\theta} {\arg \max} \mathbb {E} _ {d ^ {N}} [ \log c _ {\theta} (s, a) ] + \mathbb {E} _ {d ^ {U}} [ \log (1 - c _ {\theta} (s, a)) ]
$$
2: Set the  $\alpha$  as  $\hat{\alpha}$  from Eq. (15) on  $D^U$ :
$$
\hat {\alpha} = \min  _ {(s, a) \in D ^ {U}} \left(\frac {1}{c _ {\theta^ {*}} (s , a)} - 1\right)
$$
3: for each iteration  $i$  do
4: Update  $\nu_{\phi}$  by optimizing the Eq. (13):
$$
\min _ {\phi} (1 - \gamma) \mathbb {E} _ {p _ {0}} [ \nu_ {\phi} (s) ] + \mathbb {E} _ {d U} \bigg [ \exp (A _ {\nu_ {\phi}} (s, a) - 1) \bigg ]
$$
5: Update policy  $\pi_{\psi}$  by weighted behavior cloning of unlabeled demonstrations  $D^{U}$ :
$$
\min  _ {\psi} - \mathbb {E} _ {(s, a) \sim D ^ {U}} \left[ w ^ {*} (s, a) \log \pi_ {\psi} (a | s) \right]
$$
Output:  $\pi_{\psi} \approx \pi^{*}$
C Implementation Details of SafeDICE and Baseline Algorithms
In this section, we provide the implementation details of SafeDICE and baseline algorithms. Our code is available on https://github.com/jys5609/SafeDICE.
C.1 SafeDICE
We implement SafeDICE based on the Real-World RL (RWRL) suite [8] and codebase of DemoDICE [13], which is one of recent DICE-based offline IL algorithms. Similar to DemoDICE [13], due to the instability issue of Eq. (13), we implemented an alternative objective that is more numerically stable while having the same optimal solution:
$$
\min  _ {\nu} \mathcal {L} \left(w _ {\nu} ^ {*}, \nu ; r\right) = (1 - \gamma) \mathbb {E} _ {p _ {0}} [ \nu (s) ] + \log \mathbb {E} _ {d U} \left[ \exp \left(A _ {\nu} (s, a)\right) \right]. \tag {36}
$$
Then, we estimate Eq. (14) using self-normalized importance sampling [23] as follows:
$$
\min  _ {\pi} - \frac {\mathbb {E} _ {(s , a) \sim D ^ {U}} \left[ \tilde {w} _ {\tilde {\nu} ^ {*}} (s , a) \log \pi_ {\psi} (a | s) \right]}{\mathbb {E} _ {(s , a) \sim D ^ {U}} \left[ \tilde {w} _ {\tilde {\nu} ^ {*}} (s , a) \right]}, \tag {37}
$$
where  $\tilde{\nu}^*$  is optimal solution of Eq. (36) and  $\tilde{w}_{\tilde{\nu}^*} \coloneqq \exp(A_{\tilde{\nu}^*}(s, a))$ .
C.2 Discriminator-Weighted Behavior Cloning (DWBC)
We modified DWBC, which adopted positive-unlabeled learning, to be considered as a baseline algorithm in our experimental settings. Similar to positive-unlabeled learning, we adopted negative-unlabeled learning to the objective of discriminator learning as follows:
$$
\begin{array}{l} \min  _ {\theta} \eta \mathbb {E} _ {(s, a) \sim D ^ {N}} [ - \log d _ {\theta} (s, a, \log \pi) ] \\ + \mathbb {E} _ {(s, a) \sim D ^ {U}} [ - \log (1 - d _ {\theta} (s, a, \log \pi)) ] \\ - \eta \mathbb {E} _ {(s, a) \sim D ^ {N}} [ - \log (1 - d _ {\theta} (s, a, \log \pi)) ], \tag {38} \\ \end{array}
$$
B Pseudocode of SafeDICE
where  $d_{\theta}$  is a discriminator with parameter  $\theta$ , and  $\eta$  is a hyperparameter that corresponds to the proportion of negative samples to unlabeled samples.
Based on the trained discriminator by Eq. (38), we adopted weighted behavior cloning on the unlabeled demonstrations for the policy learning as follows:
$$
\min  _ {\psi} - \mathbb {E} _ {(s, a) \sim D ^ {U}} \left[ \left(1 - d _ {\theta} (s, a)\right) \log \pi_ {\psi} (a | s) \right]. \tag {39}
$$
C.3 Preference-based Policy Learning (PPL)
We trained the reward function with the following loss function follows from the Bradley-Terry model [3] which is commonly used in preference-based policy learning approaches [4, 18]:
$$
\min  _ {\theta} - \sum_ {(i, j) \in \mathcal {P}} \log \frac {\exp \left(\sum_ {(s , a) \in \tau_ {i}} R _ {\theta} (s , a)\right)}{\exp \left(\sum_ {(s , a) \in \tau_ {i}} R _ {\theta} (s , a)\right) + \exp \left(\sum_ {(s , a) \in \tau_ {j}} R _ {\theta} (s , a)\right)}, \tag {40}
$$
where  $\tau_{i}$  and  $\tau_{j}$  represent trajectories, and  $\mathcal{P} = \{(i,j):\tau_i\succ \tau_j\}$ . In our problem setting, we adopt this model assuming that unlabeled demonstrations are better than anti-expert demonstrations (i.e.  $\tau_{i}\succ \tau_{j}$  for all  $\tau_{i}\in D^{U}$  and  $\tau_{j}\in D^{N}$ ).
Based on the trained reward model by Eq. (40), we adopt weighted behavior cloning on the unlabeled demonstrations for the policy learning as follows:
$$
\min  _ {\psi} - \mathbb {E} _ {(s, a) \sim D ^ {U}} \left[ R _ {\theta} (s, a) \log \pi_ {\psi} (a | s) \right]. \tag {41}
$$
C.4 DExperts
Tabular MDPs For the tabular MDPs, we separately trained unlabeled policy  $\pi^U$  and non-preferred policy  $\pi^N$  with a behavior cloning model using unlabeled demonstrations and non-preferred demonstrations as follows:
$$
\min  _ {\pi^ {U}} - \mathbb {E} _ {(s, a) \sim D ^ {U}} \left[ \log \pi^ {U} (a | s) \right], \tag {42}
$$
$$
\min  _ {\pi^ {N}} - \mathbb {E} _ {(s, a) \sim D ^ {N}} [ \log \pi^ {N} (a | s) ]. \tag {43}
$$
Then, as the final policy, the ensemble of  $\pi^U$  and  $\pi^N$  at decoding-time is used:
$$
\pi (a | s) = \pi^ {U} (a | s) - \alpha \pi^ {N} (a | s), \tag {44}
$$
where  $\alpha$  is a hyperparameter that controls the degree of leveraging the non-preferred policy.
Continuous MDPs For the continuous MDPs, it is not straightforward to apply DExperts as a decoding-time method. Therefore, we implemented DExperts with the following objective that is similarly considered at training time:
$$
\min  _ {\psi} - \mathbb {E} _ {(s, a) \sim D ^ {U}} [ \log \pi_ {\psi} (a | s) ] + \alpha \mathbb {E} _ {(s, a) \sim D ^ {N}} [ \log \pi_ {\psi} (a | s) ], \tag {45}
$$
where  $\alpha$  is a hyperparameter that controls the degree of leveraging the non-preferred demonstrations.
D.1 Experimental details for RWRL suite
Task specification We conduct experiments on the RWRL suite with safety constraints. Table 1 shows the tasks, safety constraint settings, and the information of dataset for each domain that we used in our experiments.
Task specification	Cartpole	Walker	Quadruped
name of the used task	realworld_swingup	realworld.walk	realworld.walk
name of the used safety constraint	slider_pos Constraint	joint Velocity Constraint	joint Velocity Constraint
safety coefficient	0.2	0.2	0.35
# of unlabeled demonstrations	5000	2000	2000
# of labeled non-preferred demonstrations	80	10	50
mean cost of preferred demonstrations	71.95	56.14	97.01
mean cost of non-preferred demonstrations	316.75	377.58	323.60
mean return of preferred demonstrations	516.28	939.74	991.11
mean return of non-preferred demonstrations	519.82	940.06	989.59