SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyperparameter configurations For a fair comparison, we use the same architecture and learning rate to train the policy networks, discriminators, and reward models of each algorithm. Table 2 summarizes the hyperparameter settings that we used in our experiments. The additional hyperparameters  $\eta$  and  $\alpha$  of DWBC and DExperts were selected by the best performance among  $\{0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8\}$ .
Table 1: Task specification of each domain used in our experimental results on RWRL environment.
Hyperparameters	BC	DWBC	PPL	DExperts	SafeDICE
γ (discount factor)	0.99	0.99	0.99	0.99	0.99
learning rate (actor)	3 × 10-4	3 × 10-4	3 × 10-4	3 × 10-4	3 × 10-4
network size (actor)	[256, 256]	[256, 256]	[256, 256]	[256, 256]	[256, 256]
learning rate (critic)	-	-	-	-	3 × 10-4
network size (critic)	-	-	-	-	[256, 256]
learning rate (discriminator / reward model)	-	3 × 10-4	3 × 10-4	-	3 × 10-4
network size (discriminator / reward model)	-	[256,256]	[256,256]	-	[256, 256]
gradient penalty coefficient (discriminator)	-	0.1	-	-	0.1
batch size	512	512	512	512	512
# of training iterations	1,200,000	1,200,000	1,200,000	1,200,000	1,200,000
Table 2: Configurations of hyperparameters used in our experimental results on RWRL environment.
D.2 Experimental details for Safety Gym
Task specification We conduct experiments on the Safety Gym with safety constraints. Table 3 shows the tasks, safety constraint settings, and the information of dataset for each domain that we used in our experiments.
Task specification	Goal	Button
type of robot	Point	Point
level of difficulty	1	1
# of unlabeled demonstrations	2000	2000
# of labeled non-preferred demonstrations	50	50
mean cost of preferred demonstrations	13.798	12.085
mean cost of non-preferred demonstrations	107.977	166.099
mean return of preferred demonstrations	19.911	8.286
mean return of non-preferred demonstrations	20.018	21.933
Hyperparameter configurations For a fair comparison, we use the same architecture and learning rate to train the policy networks, discriminators, and reward models of each algorithm. Table 4 summarizes the hyperparameter settings that we used in our experiments. The additional hyperparameters  $\eta$  and  $\alpha$  of DWBC and DExperts were selected by the best performance among  $\{0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8\}$ .
Table 3: Task specification of each domain used in our experimental results on Safety Gym environment.
Hyperparameters	BC	DWBC	PPL	DExperts	SafeDICE
γ (discount factor)	0.99	0.99	0.99	0.99	0.99
learning rate (actor)	3 × 10-4	3 × 10-4	3 × 10-4	3 × 10-4	3 × 10-4
network size (actor)	[256, 256]	[256, 256]	[256, 256]	[256, 256]	[256, 256]
learning rate (critic)	-	-	-	-	3 × 10-4
network size (critic)	-	-	-	-	[256, 256]
learning rate (discriminator / reward model)	-	3 × 10-4	3 × 10-4	-	3 × 10-4
network size (discriminator / reward model)	-	[256,256]	[256,256]	-	[256, 256]
gradient penalty coefficient (discriminator)	-	0.1	-	-	0.1
batch size	512	512	512	512	512
# of training iterations	1,000,000	1,000,000	1,000,000	1,000,000	1,000,000
Table 4: Configurations of hyperparameters used in our experimental results on Safety Gym environment.
E Additional Results on RWRL suite
E.1 Results of CVaR  $k\%$  Cost on RWRL suite
Figure 5: Additional experimental results of Figure 2 on metrics of CVaR  $k\%$  Cost.
E.2 Results of DWBC and DExperts with various hyperparameters on RWRL Walker domain
To study the hyperparameter sensitivity of DWBC and DExperts, we demonstrate the results of DWBC and DExperts with various hyperparameter settings. For DWBC, we compare the performance of DWBC with different values of  $\eta \in \{0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8\}$  and DWBC without  $\eta$  (i.e. naive discriminator training without negative-unlabeled learning). As shown in Figure 6, DWBC shows very sensitive performance to hyperparameter  $\eta$  and unstable learning that sometimes even suffers from degeneration issues.
Figure 6: Additional experimental results of DWBC with various hyperparameters on Walker domain.
We also compare the performance of DExperts with different values of  $\alpha \in \{0, 0.01, 0.02, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8\}$ . Here,  $DExperts(\alpha = 0)$  is reduced to the naive behavior cloning model on unlabeled demonstrations. As shown in Figure 7, DExperts shows very sensitive performance to hyperparameter  $\alpha$  and is vulnerable to degeneration issues.
Figure 7: Additional experimental results of DExperts with various hyperparameters on Walker domain.
E.3 Results with varying amount of labeled non-preferred demonstrations
Figure 8: Experimental results on RWRL Walker domain with different numbers of labeled non-preferred demonstrations  $(|D^{N}| \in \{50, 20, 10, 5\})$ . All results are averaged over 5 runs, and the shaded area represents the standard error.
Figure 9: Additional experimental results of Figure 8 on metrics of CVaR  $k\%$  Cost.
E.4 Results of SafeDICE with various  $\alpha$
To show the importance of theoretically determining  $\alpha$ , we show additional results of SafeDICE with various values of  $\alpha$ . According to the value of  $\alpha$ , the reward function  $r_{\alpha}(s, a)$  defined by Eq. (10) cannot guarantee that the value inside the log function always becomes a positive. Therefore, we simply performed learning with a reward function that artificially clipped the value inside the log function of Eq. (10). As shown in Figure 10 and 11, the results show various performance depending on the value of  $\alpha$ , and result with  $\alpha$  value selected by Eq. (15) shows the best performance.
Figure 10: Additional experimental results of SafeDICE with various hyperparameters on Walker domain.
Figure 11: Additional experimental results of SafeDICE with various hyperparameters on Walker domain.
E.5 Impact of the stationary distribution constraint and labeled non-preferred demonstrations
We conducted additional experiments to evaluate the impact of the stationary distribution constraint and labeled non-preferred demonstrations. As can be seen from the Figure 12, the performance is degraded both in the absence of stationary distribution constraints and in the case of not using labeled non-preferred demonstrations.
Figure 12: Experimental results on RWRL Walker domain without stationary distribution constraint and labeled non-preferred demonstrations. All results are averaged over 5 independent runs, and the shaded area represents the standard error.
E.6 Results with various configurations of unlabeled demonstrations
In order to evaluate our algorithm with more various configurations of unlabeled demonstrations, we also conducted additional experiments for various configurations of unlabeled demonstrations consisting of different ratios of preferred and non-preferred demonstrations  $(D_P^U,D_N^U)\in$ $\{(250,1000),(500,1000),(1000,500),(1000,250)\}$ , where  $|D_P^U|$  and  $|D_N^U|$  denote the number of preferred and non-preferred demonstrations in  $D^{U}$ , respectively). As can be seen from the Figure 13, the results show that SafeDICE outperforms baselines in all settings with various configurations of unlabeled demonstrations. These results show that SafeDICE can be applied more effectively in various real-world scenarios regardless of the preferred and non-preferred ratio of unlabeled demonstrations.
Figure 13: Experimental results on RWRL Walker domain with different numbers of preferred and non-preferred demonstrations in unlabeled demonstrations  $D^{U}$ .  $|D_P^U|$  and  $|D_N^U|$  denote the number of preferred and non-preferred demonstrations in  $D^{U} \left( (D_{P}^{U}, D_{N}^{U}) \in \{(250, 1000), (500, 1000), (1000, 500), (1000, 250)\} \right)$ . All results are averaged over 5 independent runs, and the shaded area represents the standard error.
F Comparison with constrained RL algorithm
To compare with the safe RL algorithm as an oracle agent, we conducted additional experiments for an offline-constrained RL method. We run COptiDICE [16], the state-of-the-art offline constrained RL algorithm, using the datasets that are augmented with additional (ground-truth) reward and cost annotations. Table 5-9 summarize overall performance of SafeDICE and baseline algorithms including COptiDICE. The results show that SafeDICE is competitive with (or even outperforms in some domains) the constrained offline RL algorithm COptiDICE, even though SafeDICE uses only a much smaller amount of annotation information.
F.1 Experimental results on RWRL control tasks
CARTPOLE	Normalized Return	Average Cost	Cost Violation	CVaR 10% Cost
BC	100.24 ± 0.40	184.44 ± 0.94	0.48 ± 0.00	392.75 ± 2.72
DWBC	99.78 ± 0.55	189.97 ± 3.21	0.47 ± 0.00	408.71 ± 4.01
PPL	83.70 ± 0.87	229.80 ± 1.64	0.48 ± 0.01	440.68 ± 3.49
DExperts	100.28 ± 0.38	186.15 ± 2.29	0.48 ± 0.01	397.05 ± 3.11
SafeDICE	99.91 ± 0.60	154.88 ± 1.79	0.34 ± 0.00	404.22 ± 1.56
COptiDICE	107.95 ± 2.10	97.59 ± 7.21	0.06 ± 0.01	183.63 ± 6.18
Table 5: Experimental results on Cartpole domain. All results indicate averages and standard errors over 5 independent runs.
WALKER	Normalized Return	Average Cost	Cost Violation	CVaR 10% Cost
BC	98.41 ± 0.11	152.25 ± 5.33	0.22 ± 0.01	531.07 ± 7.53
DWBC	98.99 ± 0.10	105.63 ± 3.85	0.11 ± 0.01	378.01 ± 20.88
PPL	99.02 ± 0.10	128.46 ± 4.84	0.17 ± 0.01	478.75 ± 12.53
DExperts	95.21 ± 0.75	173.20 ± 11.22	0.27 ± 0.03	518.38 ± 5.25
SafeDICE	99.67 ± 0.13	68.90 ± 0.83	0.00 ± 0.00	124.54 ± 2.56
COptiDICE	98.07 ± 0.34	77.86 ± 2.14	0.01 ± 0.00	159.91 ± 8.90
Table 6: Experimental results on Walker domain. All results indicate averages and standard errors over 5 independent runs.
QUADRUPED	Normalized Return	Average Cost	Cost Violation	CVaR 10% Cost
BC	100.19 ± 0.08	178.10 ± 5.20	0.38 ± 0.02	328.10 ± 1.83
DWBC	99.87 ± 0.16	167.15 ± 3.32	0.34 ± 0.02	316.86 ± 1.74
PPL	99.87 ± 0.09	169.93 ± 5.06	0.35 ± 0.03	319.77 ± 1.70
DExperts	97.05 ± 2.62	178.75 ± 4.78	0.38 ± 0.03	317.55 ± 4.40
SafeDICE	99.68 ± 0.25	148.21 ± 2.84	0.24 ± 0.01	308.71 ± 2.90
COptiDICE	87.37 ± 3.19	130.21 ± 5.20	0.16 ± 0.03	263.41 ± 17.00
Table 7: Experimental results on Quadruped domain. All results indicate averages and standard errors over 5 independent runs.
F.2 Experimental results on Safety Gym control tasks
GOAL	Normalized Return	Average Cost	Cost Violation	CVaR 10% Cost
BC	94.08 ± 0.60	51.47 ± 1.49	0.38 ± 0.02	109.66 ± 1.63
DWBC	93.93 ± 0.53	47.96 ± 1.77	0.31 ± 0.02	108.79 ± 2.12
PPL	94.43 ± 0.66	52.31 ± 1.52	0.37 ± 0.02	111.41 ± 2.37
DExperts	87.76 ± 4.41	50.82 ± 1.55	0.36 ± 0.02	112.62 ± 1.54
SafeDICE	92.04 ± 0.44	39.49 ± 0.94	0.21 ± 0.02	93.35 ± 1.45
COptiDICE	92.98 ± 1.13	48.79 ± 0.80	0.32 ± 0.01	107.25 ± 0.78
Table 8: Experimental results on Goal task. All results indicate averages and standard errors over 5 independent runs.
BUTTON	Normalized Return	Average Cost	Cost Violation	CVaR 10% Cost
BC	88.64 ± 1.08	98.18 ± 2.19	0.49 ± 0.01	220.87 ± 3.48
DWBC	85.53 ± 0.80	96.31 ± 2.38	0.47 ± 0.02	225.49 ± 3.52
PPL	86.76 ± 1.06	99.97 ± 2.19	0.50 ± 0.02	229.46 ± 3.30
DExperts	86.28 ± 1.35	97.30 ± 1.70	0.49 ± 0.01	225.93 ± 5.34
SafeDICE	71.48 ± 1.33	70.67 ± 1.61	0.30 ± 0.01	185.64 ± 1.54
COptiDICE	94.09 ± 1.68	111.10 ± 3.53	0.58 ± 0.02	238.46 ± 3.44
Table 9: Experimental results on Button task. All results indicate averages and standard errors over 5 independent runs.
G Tabular MDPs Results
In order to show the motivation of the paper and the limitation of existing algorithms through illustrative examples, we conducted the experiment in tabular MDPs, which extends the Four Rooms environment [28, 22].
Non-preferred  $(D^{N})$
Unlabeled  $(D^U)$
Preferred  $(1 - \alpha)$
Non-preferred  $(\alpha)$
Figure 14: Illustrative examples of experimental settings on Six Rooms environment. We assume the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations  $D^{N}$ , and abundant but unlabeled demonstrations  $D^{U}$  from both preferred and non-preferred policies.
SafeDICE
DWBC  $(\eta = 0.2)$
DWBC  $(\eta = 0.4)$
DWBC  $(\eta = 0.6)$
DWBC  $(\eta = 0.8)$
DWBC  $(\eta = 0.1)$
DExperts  $(\alpha = 0.1)$
Figure 15: Experimental results on Six Room environment. We demonstrate the qualitative results of each algorithm learned using the dataset given in Figure 14. For DWBC and DExperts that require additional hyperparameter search, we show the results with various hyperparameter settings.  $\eta$  and  $\alpha$  represent hyperparameters that control the degree of leveraging the non-preferred demonstrations of DWBC and DExperts, respectively.
DExperts  $(\alpha = 0.2)$
DExperts  $(\alpha = 0.4)$
DExperts  $(\alpha = 0.6)$
DExperts  $(\alpha = 0.8)$
Task setup We consider the Six Rooms environment, a variant of the Four Rooms environment [28, 22], in which an agent aims to navigate to a goal location within a gridworld. Figure 14 demonstrates our experimental settings on the Six Rooms environment. The orange and green squares denote the initial state and goal state, respectively. The arrows for each action at the square represent the policy, where the opacity of the arrow is determined by the probability  $\pi(a|s)$ . The opacity of each square represents the state margins of each stationary distribution. In order to compose the task naturally, we assume that the preferred and non-preferred policies have both similar and different behaviors in some state-action spaces. We define the policy that reaches the goal state from the initial state through the downward direction as an non-preferred policy, which represents constraint-violating behavior, and the policy that reaches the goal state through the right direction as an preferred policy. In this task, the agent aims to learn the policy that recovers the preferred policy from unlabeled demonstrations while avoiding the constraint-violating behavior using labeled non-preferred demonstrations. Figure 14 shows a given dataset to the agent for learning, which consists of scarce but labeled non-preferred demonstrations  $D^{N}$  and abundant but unlabeled demonstrations
$D^{U}$  from both preferred and non-preferred policy. The details for the experimental settings can be found in Appendix D.
Results Figure 15 summarizes the qualitative results of SafeDICE and baseline algorithms on Six Rooms environment. Naive BC on unlabeled demonstrations follows both preferred and non-preferred behavior since it does not consider avoiding the non-preferred demonstrations. On the other hand, PPL and SafeDICE show safe behaviors in which non-preferred demonstrations are completely avoided while following preferred demonstrations. For DWBC and DExperts that require additional hyperparameter search, we show the behaviors of learned policies with different hyperparameter settings. Among various hyperparameter settings, there are cases where an agent learns a policy close to an preferred behavior at a specific hyperparameter. However, the results show that both DWBC and DExperts are hyperparameter sensitive, which is unsuitable for real-world tasks where the deployment is costly or risky. In addition, DExperts not only have cost-violating behaviors but also cause degeneration issues that fall into abnormal behaviors that are completely different from given demonstrations. To show that SafeDICE robustly works in various problem settings, we also provide additional results including cases where the distributions of labeled and unlabeled non-preferred demonstrations are dissimilar (see Appendix G.1), and more complex environments (see Appendix G.2). As shown in Appendix G.1, SafeDICE learns a safe policy in which non-preferred behaviors are completely avoided while following preferred behavior, even when the distributions of labeled and unlabeled non-preferred demonstrations are dissimilar.
G.1 Experimental results with the dissimilar distribution of labeled and unlabeled non-preferred demonstrations
To show that SafeDICE works robustly, we conduct the experiment considering a problem setting more similar to real-world tasks, where the distributions of unlabeled and labeled non-preferred demonstrations are dissimilar. In this experimental setting, we define the non-preferred policy is a multimodal policy that consists of a mixture of two different policies  $\pi^{A_1}$  and  $\pi^{A_2}$ . We assume that non-preferred demonstrations of labeled and unlabeled are sampled from policies  $\pi^{A_1}$  and  $\pi^{A_2}$  with different ratios. As shown in Figure 16, baseline algorithms fail to completely recover the preferred behavior or follow the non-preferred behavior. On the other hand, SafeDICE shows safe behavior in which non-preferred behaviors are completely avoided while following preferred behavior, even when the distributions of labeled and unlabeled non-preferred demonstrations are dissimilar.
True preferred  $(\pi^P)$
True non-preferred  $(\pi^{N_2})$
True non- preferred  $(\pi^{N_1})$
Unlabeled  $(D^U)$
Figure 16: Illustrative examples of additional experimental settings on the tabular environment. We assume the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations  $D^{N}$  and abundant but unlabeled demonstrations  $D^{U}$  from both preferred and non-preferred policies. In this experiment, we consider a more challenging problem setting where the distributions of labeled and unlabeled non-preferred demonstrations are dissimilar.
BC
DWBC  $(\eta = 0.1)$
DExperts  $(\alpha = 0.1)$
Figure 17: Experimental results on the tabular environment. We demonstrate the qualitative results of each algorithm learned using the dataset given in Figure 16.
G.2 Additional results for various Tabular MDPs
Unlabeled  $(D^U)$
#
Preferred  $(1 - \alpha)$
Figure 18: Illustrative examples of additional experimental settings on the tabular environment. We assume the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations  $D^{N}$  and abundant but unlabeled demonstrations  $D^{U}$  from both preferred and non-preferred policies.
+
Non-preferred  $(\alpha)$
DExperts  $(\alpha = 0.1)$
Figure 19: Experimental results on the tabular environment. We demonstrate the qualitative results of each algorithm learned using the dataset given in Figure 18. For DWBC and DExperts that require additional hyperparameter search, we show the results with various hyperparameter settings.  $\eta$  and  $\alpha$  represent hyperparameters that control the degree of leveraging the non-preferred demonstrations of DWBC and DExperts, respectively.
Figure 20: Illustrative examples of additional experimental settings on the tabular environment. We assume the pre-collected dataset which consists of scarce but labeled non-preferred demonstrations  $D^{N}$  and abundant but unlabeled demonstrations  $D^{U}$  from both preferred and non-preferred policies.
Figure 21: Experimental results on the tabular environment. We demonstrate the qualitative results of each algorithm learned using the dataset given in Figure 20. For DWBC and DExperts that require additional hyperparameter search, we show the results with various hyperparameter settings.  $\eta$  and  $\alpha$  represent hyperparameters that control the degree of leveraging the non-preferred demonstrations of DWBC and DExperts, respectively.