SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we additionally compare our method to three other baselines in offline IL: DWBC [68] that also trains a discriminator using Positive-Unlabeled (PU) learning, MARWIL [62] that is a naive adaptation of Reward-Weighted Regression [48] to offline scenarios with a similar actor objective as TAILO, and the recent Wasserstein-based method OTR [40] which computes Wasserstein distance between the task-specific and task-agnostic trajectories, assigns reward label based on optimization result and conducts offline RL. Among those methods, MARWIL and OTR can be directly applied to our scenario, while DWBC requires extra access to expert actions. Fig. 13 and Fig. 14 illustrate the result, respectively on the settings of Sec. F.5 and Sec. 4.3.
Figure 13: Performance comparison between our method and DWBC, MARWIL and OTR on settings of Sec. F.5. Our method outperforms all other methods, which all struggle in our settings.
Figure 14: Performance comparison between our method and DWBC, MARWIL and OTR on settings of Sec. 4.3. Our method outperforms all other methods, which all struggle in our settings.
F.3 Kitchen Environment Evaluated on "Kitchen-Complete-V0"
In Sec. 4.3, we mentioned that the SMODICE experiment uses expert trajectory that is only expert for the first 2 out of 4 subtasks. More specifically, SMODICE uses the expert trajectory in the "kitchen-complete-v0" environment in D4RL as the task-specific dataset, and uses data from the "kitchen-mixed-v0" environment as the task-agnostic dataset; they also evaluate on "kitchen-mixed-v0." However, the subtask list to complete in "kitchen-complete-v0" is {Microwave, Kettle, Light Switch, Slide Cabinet}, while the subtask list to complete in "kitchen-mixed-v0" is {Microwave, Kettle, Bottom Burner, Light Switch}.<sup>3</sup> Thus, the list of subtasks that is accomplished in expert
3See https://github.com/Farama-Foundation/D4RL/blob/master/d4rl/kitchen/_init__.py in D4RL and lines 29, 164 and 208 in https://github.com/JasonMa2016/SMODICE/blob/main/run_oil_observations.py for details.
trajectory (from "kitchen-complete-v0") and the list of subtasks evaluated by the environment (from "kitchen-mixed-v0") are only identical in the first two subtasks. We follow their setting in Sec. 4.3, and this is the reason why the normalized reward for any method is hard to reach 60 in Fig. 4, i.e., reach an average completion of 2.4 tasks.
We now evaluate on "kitchen-complete-v0," and Fig. 15 shows the result when evaluating on this environment. The other settings remain identical to the ones discussed in Sec. 4.3. Note, this new evaluation setting introduces another challenge - since "kitchen-complete-v0" has a different set of subtasks to accomplish, the dimensions masked to zero differ; to be more specific, the 41st and 42nd dimensions are zero-masked in "kitchen-complete-v0" but not in "kitchen-mixed-v0," and vice versa for the 49th dimension. Thus, the task-agnostic data are in a different state space from the environment for evaluation. This significantly increases the difficulty of the environment, because the states met by the agent in evaluation are out of distribution from the task-agnostic data. Even under such a setting, our method as well as BC remains robust, while all the other baselines struggle.
Figure 15: Reward curves for the kitchen environment evaluated on "kitchen-complete-v0"; only our method and BC are robust to the change of state space from task-agnostic data.
F.4 Hyperparameter Sensitivity Analysis
In this section, we conduct a sensitivity analysis for the hyperparameters in our method.
F.4.1 Effect of  $\alpha$
Fig. 16 illustrates the reward curves tested on the settings of Sec. 4.3 with different  $\alpha$  (for scaling of  $R(s)$ ), where the value we used throughout the paper is  $\alpha = 1.25$ . While an extreme selection of the hyperparameters leads to a significant decrease of performance (e.g.,  $\alpha = 2.0$  for antmaze), our method is generally robust to the selection of  $\alpha$ .
F.4.2 Effect of  $\beta_{1}$
Fig. 17 shows the reward curves tested on the settings of Sec. 4.3 with different  $\beta_{1}$  (ratio of "safe" negative samples), where the default setting is  $\beta_{1} = 0.8$ . Our method is again generally robust to the selection of  $\beta_{1}$ . Obviously the reward decreases with an extreme selection of  $\beta_{1}$ , such as  $\beta_{1} = 0.3$ . Intuitively, such robustness comes from two sources: 1) with small ratio of expert trajectories in the task-agnostic dataset ( $< 5\%$ ), thus there is little expert data erroneously classified as safe negatives; 2) the Lipschitz-smoothed discriminator yields a good classification margin, and thus the  $60\%+$  trajectories with lower average reward represent all non-expert data well.
F.4.3 Effect of  $\beta_{2}$
In this work, we use  $\beta_{2} = 1$  when the embodiment of the task-specific dataset  $D_{\mathrm{TS}}$  is different from that of the task-agnostic dataset  $D_{\mathrm{TA}}$ . We use  $\beta_{2} = 0$  otherwise. Ideally, after selecting safe negatives from the first step of the discriminator training, the obtained  $D_{\mathrm{TA safe}}$  should consist of (nearly)  $100\%$  non-expert data, and thus we use  $\beta_{2} = 0$ ; however, when  $D_{\mathrm{TS}}$  is collected from a different embodiment, the recognition of "safe negatives" will be much harder, because the states reachable by the task-specific expert could be different from the agent that was used to collect task-agnostic
Figure 16: Reward curves tested in the settings of Sec. 4.3 with different  $\alpha$ . The default  $\alpha = 1.25$  is plotted in red, lower  $\alpha$  in blue, and higher  $\alpha$  in green. The further  $\alpha$  deviates from the default, the deeper the color. Our method is generally robust to the selection of  $\alpha$ .
Figure 17: Reward curves tested in the settings of Sec. 4.3 with different  $\beta_{1}$ . The default  $\beta_{1}$  is plotted in red, lower in blue and higher in green, with deeper color indicating further deviation from the default. Our method is generally robust to the selection of  $\beta_{1}$ .
data. Thus, we use the debiasing objective in the formal training step, i.e.,  $\beta_{2} = 1$ . We compare the reward of using  $\beta_{2} = 0$  and  $\beta_{2} = 1$  in Fig. 18, where  $\beta_{2} = 1$  works significantly better than  $\beta_{2} = 0$  on halfcheetah with mismatched dynamics.
Figure 18: Reward curves for  $\beta_{2} = 1$  (i.e., debiasing objective) and  $\beta_{2} = 0$  (i.e., normal cross entropy loss) on environments with mismatching dynamics.  $\beta_{2} = 1$  is better than  $\beta_{2} = 0$  on the halfcheetah environment with mismatched expert dynamics.
F.4.4 Effect of  $\eta_{p}$
Fig. 19 shows the reward curves tested in the settings of Sec. 4.3 with different  $\eta_p$ , where the default setting is  $\eta_p = 0.2$ . The results show that our method is robust to the selection of  $\eta_p$ .
Figure 19: Reward curves tested on the settings of Sec. 4.3 with different  $\eta_p$ . The default  $\eta_p$  is plotted in red, lower in blue and higher in green, with deeper color indicating further deviation from default. Our method is robust to the selection of  $\eta_p$ .
F.4.5 Effect of  $\gamma$
Fig. 20 shows the reward curves tested in the settings of Sec. 4.3 with different  $\gamma$  (decaying factor for weight propagation along the trajectory); our default setting is  $\gamma = 0.98$  for the kitchen and pointmaze environment, and  $\gamma = 0.998$  otherwise. We found that when  $\gamma$  is too low (e.g., 0.95), weights from the future expert state cannot be properly propagated to initial states where most trajectories are similar, and thus the algorithm is more likely to fail; more extremely, when  $\gamma = 0$ , our method works much worse as shown in Fig. 21 in both Sec. F.5 and Sec. 4.3 settings, which illustrates the necessity of propagation of future returns. Generally however, our method is robust to the selection of  $\gamma$ .
Figure 20: Reward curves tested in the settings of Sec. 4.3 with different  $\gamma$ . The default  $\gamma$  is plotted in red, lower in blue and higher in green, with deeper color indicating further deviation from the default; note, kitchen uses a default  $\gamma$  of 0.98 and other environments use 0.998. Our method is robust to the selection of  $\gamma$ .
F.4.6 Effect of Batch Size
Since we use a different batch size (8192) than SMODICE and LobsDICE (512) for weighted behavior cloning, one possible concern is that the comparison might be unfair to the DICE methods, as we process more data. To address the concern, we test our algorithm with batch size 512 and SMODICE/LobsDICE with batch size 8192 in Sec. 4.1, Sec. 4.2, and Sec. 4.3. The results are illustrated in Fig. 22, Fig. 23, and Fig. 24. Generally, we found that on most testbeds, our method with a batch size of 512 is slightly less stable than the default batch size of 8192, but still on par or better than the baselines; meanwhile, SMODICE and LobsDICE with a batch size 8192 work
(a) SMODICE (Sec. F.5) setting
Figure 21: Ablation of our method with  $\gamma = 0$ .  $\gamma = 0$  struggles on both experiment settings of Sec. F.5 and Sec. 4.3, showing that  $\gamma > 0$  is crucial to the performance of our method.
(b) Sec. 4.3 setting
slightly better on Sec. 4.3 than batch size 512, but are less stable in other scenarios such as Sec. 4.1 and Sec. 4.2. In conclusion, there is no general performance gain by increasing the batch size for SMODICE and LobsDICE. Thus, our comparison is fair to SMODICE and LobsDICE.
Figure 22: Performance comparison of SMODICE, LobsDICE, and our method with batch size 512 and 8192 on mujoco testbeds of standard offline imitation from observation (Sec. 4.3). Non-default settings (SMODICE/LobsDICE with batch size 8192, ours with batch size 512) are single lines without standard deviations, while default settings (SMODICE/LobsDICE with batch size 512, ours with batch size 8192) are with standard deviations. Our method with batch size 512 works slightly worse, but is still better than baselines; baselines with batch size 8192 work slightly better than use of a batch size of 512.
F.4.7 Effect of Lipschitz Regularizer
A Lipschitz regularizer on the discriminator is used for many methods tested in this paper, such as our method, SMODICE, LobsDICE, and ORIL. Intuitively, the regularizer makes the classification margin of the discriminator  $c(s)$  smooth, and thus can give higher weights to expert trajectories in the task-agnostic dataset  $D_{\mathrm{TA}}$  instead of overfitting to the few states given in the task-specific dataset  $D_{\mathrm{TS}}$ . In practice, we found this regularizer to be crucial; Fig. 25 shows the change of average  $R(s) = \log \frac{c(s)}{1 - c(s)}$  of all expert trajectories under the setting of Sec. 4.3 with respect to gradient steps. The result shows that  $R(s)$  for the expert trajectories is extremely high when the Lipschitz regularizer is removed, which indicates severe overfitting and significantly worse performance.
Figure 23: Performance comparison of SMODICE, LobsDICE and our method with batch size 512 and 8192 on Mujoco testbeds of offline imitation with incomplete task-agnostic trajectories (Sec. 4.1). Note that SMODICE and LobsDICE with a larger batch size do not necessarily work better; in contrast, a larger batch size often leads to less stability on environments such as Hopper_1/2, Walker2d_1/2 and Ant_1/3. Our method with batch size 512 works slightly worse than 8192, but is still on par or better than the baselines.
F.5 Performance Comparison Under Identical SMODICE Settings
In Sec. 4.3, we use less expert trajectory in  $D_{\mathrm{TA}}$  than the SMODICE setting. Fig. 26 shows the result where we use identical dataset settings as SMODICE, where our method is still the best among all baselines. Such a result is consistent with that reported in SMODICE. For ReCOIL [56], as the code is not available, we plot their average reward as a straight line; our method significantly outperforms ReCOIL.
F.6 Ablation Comparing Our Method and ORIL
While our method and ORIL both utilize a discriminator-based  $R(s)$  with positive-unlabeled learning, there are three major differences:
1. Definition of  $R(s)$ . For a discriminator  $c(s)$  trained with non-expert samples as label 0 and expert sample as label 1, we use  $R(s) = \log \frac{d^{\mathrm{TS}(s)}}{d^{\mathrm{TA}}(s)} = \log \frac{c(s)}{1 - c(s)}$ , while ORIL uses  $R(s) = c(s)$ .
2. Positive-Unlabeled Learning Techniques. The training of discriminator  $C(s)$  in ORIL is only one step, and uses Eq. (9) as the training objective. In contrast, we first use Eq. (10) to find safe negative samples, and then use Eq. (10) or Eq. (7) (see Sec. A for details) to train the final discriminator  $c(s)$  and consequently obtain  $R(s)$ .
3. Policy Retrieval. ORIL conducts RL on the offline dataset with reward labeled by  $R(s)$ , while our method uses a non-parametric approach to calculate coefficients for weighted behavior cloning.
Hopper head90 tail10
Hopper head50 tail50
Hopper head1.tail100
Halfcheetah head90 tail10
Halfcheetah_head1.tail100
Ant head100 tail1
Ant head90 tail10
Walker2d head100 tail1
Walker2d head90 tail10
Walker2d_head1.tail100
Figure 24: Performance comparison of SMODICE, LobsDICE and our method with batch size 512 and 8192 on mujoco testbeds of offline imitation with incomplete task-specific trajectories (Sec. 4.2). Again, larger batch size sometimes is worse for SMODICE and LobsDICE, e.g., for Halfcheetah_head1TAIL100 and Hopper_head1TAIL100. Our method with batch size 512 works slightly worse than 8192, but remains on par or better than the baselines.
We compare eight variants of the methods, denoted as ours, ours-V1, ours-V2, ours-V3, ORIL-logR-V1, ORIL-logR-V2, ORIL-01-V1, and ORIL-01-V2. Tab. 5 summarizes the differences between the eight variants.
The results are illustrated in Fig. 27. Generally, all variants except "ours" work at least marginally worse than "ours," which shows that all three differences matter in the final performance. Among the three variants of "ours," ours-v2 has the closest performance to "ours," which indicates that the positive-unlabeled learning technique is the least important factor, though it still makes a difference on environments such as halfcheetah with mismatched dynamics as shown in Fig. 28. Moreover, while ours-v1 (different  $R(s)$ ) and ours-v3 (different  $R(s) + \mathrm{ORIL}$  positive unlabeled learning technique) are significantly worse than "ours," ORIL-01-V1, ORIL-01-V2, ORIL-logR-V1, and ORIL-logR-V2 perform even worse and are only marginally different from each other. This indicates that retrieval of policy (RL vs. non-parametric) is the most important factor for the performance gap.
Additionally, in order to illustrate the importance of Positive-Unlabeled (PU) learning in general, we test the performance of our method without PU learning and find a significant performance decrease on some scenarios; Fig. 29 illustrates two of the failure cases without PU learning.
G Computational Resource Usage
All our experiments are conducted on an Ubuntu 18.04 server with 72 Intel Xeon Gold 6254 CPUs @  $3.10\mathrm{GHz}$  and a single NVIDIA RTX 2080Ti GPU. Given these resources, our method requires about  $4 - 5$  hours to finish training in the standard offline imitation from observation scenario, while BC needs  $3 - 4$  hours, ORIL, SMODICE, and LobsDICE need about  $2.5 - 3$  hours. In our training process, training of  $R(s)$  (both steps included) requires only 15-20 minutes. The inference speed for all methods is similar as the actor network is the same, and is thus not a bottleneck.
Figure 25: The average  $R(s) = \log \frac{c(s)}{1 - c(s)}$  of the expert trajectories in the task-agnostic dataset with respect to the gradient steps of discriminator training on the halfcheetah environment. We use the settings discussed in Sec. 4.3. The run without Lipschitz regularizer (red curve) overfits to the expert state, and thus has much higher average  $R(s)$ . Such a run diverges to infinity in later training.
Figure 26: Reward curves for hopper, halfcheetah, ant, and walker2d evaluated under the identical settings as SMODICE; our method is the best among all methods including ReCOIL.
H Dataset and Algorithm Licenses
Our code is developed upon multiple algorithm repositories and environment testbeds.
Algorithm Repositories. We implement our method and behavior cloning from scratch. We test RCE, ORIL and SMODICE from the SMODICE repository, which has no license. We get LobsDICE code from their supplementary material of the publicized OpenReview submission, which also has no license.
Environment Testbeds. We utilize OpenAI gym [5], mujoco [59], and D4RL [16] as testbed, which have an MIT license, an Apache-2.0 license, and an Apache-2.0 license respectively.
- ours-V1 - ours-V3 ORIL-01-V2 - ours - ours-V2 ORIL-log-RV2 ORIL-01-V1 ORIL-log-RV1
Figure 27: Ablation between our method and ORIL; see Tab. 5 for reference to the legend. While ours-V2 with ORIL-style positive unlabeled learning has the closest performance to our final algorithm, it is still marginally worse. ORIL-01 and ORIL-logR fail, which shows that the retrieval of the policy is the most important factor for the performance gap.
Figure 28: Comparison of ours and ours-V2 on the halfcheetah environment with mismatched dynamics (Sec. 4.5). Ours-V2 fails on this environment.
16
ours no PU
Figure 29: Examples of Failure cases of our method without PU learning; on the left is the halfcheetah environment with mismatched dynamics 4.5, and on the right is the hopper environment with only first and last 50 steps in the task-specific dataset (Sec. 4.2). In both cases, our method without PU learning struggles while our method with PU learning succeeds.
Algorithm 1: Our Algorithm, TAULO
Input:state-only task-specific dataset  $D_{\mathrm{TS}}$  , state-action task-agnostic dataset  $D_{\mathrm{TA}}$  containing m trajectories   
Input:Discounted sum rate  $\gamma$  ratio of "safe" negative sample  $\beta_{1}$  positive prior  $\eta_{p}$    
Input:Number of gradient steps for pretraining  $n_1$  , formal training  $n_2$  , and number of epochs for weighted behavior cloning  $n_3$    
Input:Randomly initialized discriminator  $c(s)$  and  $c^{\prime}(s)$  , parameterized by  $w$  and  $w^{\prime}$    
Input:Learning rate a   
Output:Learned policy  $\pi_{\theta}(a|s)$  , parameterized by  $\theta$    
begin   
// Pretraining Discriminator   
for  $i\in \{1,2,\dots ,n_1\}$  do Sample  $s_1\sim D_{\mathrm{TS}},s_2\sim D_{\mathrm{TA}}$ $L_{1}\gets -[\eta_{p}\log c^{\prime}(s_{1}) + \max (0,\log (1 - c^{\prime}(s_{2})) - \eta_{p}\log (1 - c^{\prime}(s_{1})))]$ $w^{\prime}\leftarrow w^{\prime} - a\cdot \frac{\partial L_{1}}{\partial w^{\prime}}$    
// Labeling Safe Negative   
for  $i\in \{1,2,\ldots ,m\}$  do // Iterating through trajectories foreach  $s\in \tau_i$  do  $\begin{array}{r}\lfloor R^{\prime}(s)\leftarrow \log \frac{c^{\prime}(s)}{1 - c^{\prime}(s)} \end{array}$ $\bar{R} (\tau_i)\gets \frac{\sum_{s\in\tau_i}R(s)}{|\tau_i|}$  // average over trajectory   
 $q\gets \beta_{1}$  quantile of  $\bar{R}$ $D_{\mathrm{safetA}}\gets \{\tau |\bar{R} (\tau) <   q\}$    
// Formal training of Discriminator   
for  $i\in \{1,2,\dots ,n_2\}$  do Sample  $s_1\sim D_{\mathrm{TS}},s_2\sim D_{\mathrm{safetA}}$  if Expert embodiment is different then  $\begin{array}{rl} & {\big|L\gets -[\eta_p\log c(s_1) + \max (0,\log (1 - c'(s_2)) - \eta_p\log (1 - c'(s_1)))]}\\ & {\big|L\gets -[\log c(s_1) + \log (1 - c(s_2))]}\\ & {w\gets w - a\cdot \frac{\partial L}{\partial w}} \end{array}$    
// Assignment of Weights   
for  $i\in \{1,2,\dots ,m\}$  do // Iterating through trajectories foreach  $s\in \tau_i$  do  $\begin{array}{r}R(s)\leftarrow \log \frac{c(s)}{1 - c(s)}\\ W(s,a)\leftarrow \exp (\alpha R(s))\\ v\leftarrow \frac{R(s_{\mathrm{goal}})}{1 - \gamma}\\ W(s,a)\leftarrow v\\ v\leftarrow \gamma v + W(s,a) \end{array}$  //  $S_{\mathrm{goal}}$  is the last state in  $\tau_{i}$    
foreach  $s\in \tau_i$  do // This time in reverse   
// average over trajectory   
// Weighted Behavior Cloning   
for  $j\in \{1,2,\dots ,M_2\}$  do // for loop over epochs   
foreach  $(s,a)\sim D_{TA}$  do // for each data point
1   
2   
3   
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15   
16   
17   
18   
19   
20   
21   
22   
23   
24   
25
Hyperparameter	Value	Meaning
α	1.25	Scaling factor for calculation of weights
β1	0.8	Estimated ratio of safe negative samples
β2	1 (mismatch), 0 (others)	Whether to use debiasing objective in formal training
ηp	0.2	Positive prior
γ	0.98 (kitchen, pointmaze), 0.998 (others)	decay factor in weight propagation along the trajectory
Table 1: Hyperparameters specific to our method.
Type	Hyperparameter	ours	BC	LobsDICE	SMODICE	ORIL	RCE
Disc.	Network Size	[256, 256]	N/A	[256, 256]	[256, 256]	[256, 256]	[256, 256]
Activation Function	Tanh	N/A	ReLU	Tanh	Tanh	Tanh
Learning Rate	0.0003	N/A	0.0003	0.0003	0.0003	0.0003
Weight Decay	0	N/A	0	0	0	0
Training Length	(10+40)K steps	N/A	1M steps	1K steps	1K steps	1K steps
Batch Size	512	N/A	512	256	256	256
Optimizer	Adam	N/A	Adam	Adam	Adam	Adam
Actor	Network Size	[256, 256]	[256, 256]	[256, 256]	[256, 256]	[256, 256]	[256, 256]
Activation Function	ReLU	ReLU	ReLU	ReLU	ReLU	ReLU
Learning Rate	0.0001	0.0001	0.0003	0.0003	0.0003	0.0003
Weight Decay	10-5	10-5	0	0	0	0
Training length	1M steps	1M steps	1M steps	1M steps	1M steps	1M steps
Batch Size	8192	8192	512	512	512	512
Optimizer	Adam	Adam	Adam	Adam	Adam	Adam
Tanh-Squashed	Yes	Yes	Yes	Yes	Yes	Yes
Critic	Network Size	N/A	N/A	[256, 256]	[256, 256]	[256, 256]	[256, 256]
Activation Function	N/A	N/A	ReLU	ReLU	ReLU	ReLU
Learning Rate	N/A	N/A	0.0003	0.0003	0.0003	0.0003
Weight Decay	N/A	N/A	0.0001	0.0001	0	0
Training Length	N/A	N/A	1M steps	1M steps	1M steps	1M steps
Batch Size	N/A	N/A	512	512	512	512
Optimizer	N/A	N/A	Adam	Adam	Adam	Adam
Discount Factor	N/A	N/A	0.99	0.99	0.99	0.99
Table 2: Training paradigms for our method and baselines; Disc. is the abbreviation for discriminator. [256, 256] in network size means a network with two hidden layers and width 256. For our method,  $(10 + 40)\mathrm{K}$  means  $10\mathrm{K}$  gradient steps for the first step and  $40\mathrm{K}$  for the second step. Tanh-squashed means a Tanh applied at the end of the output to ensure the output action is legal.
Method	Hyperparameter	Value	Notation
LobsDICE	Regularization factor	0.1	α
RCE, ORIL	RL algorithm	TD3 [17]
Policy Update Frequency	2
Policy Noise	0.2
Noise Clip	[-0.5, 0.5]
Target Network Update Rate	0.005	τ
Table 3: Unique hyperparameters for other methods.
Environment	MOREL	MOPO	TAILO (Ours)
Halfcheetah-Medium	42.1	42.3	39.8
Hopper-Medium	95.4	28	56.2
Walker2d-Medium	77.8	17.8	71.7
Halfcheetah-Medium-Replay	40.2	53.1	42.8
Hopper-Medium-Replay	93.6	67.5	83.4
Walker2d-Medium-Replay	49.8	39.0	61.2
Halfcheetah-Medium-Expert	53.3	63.3	94.3
Hopper-Medium-Expert	108.7	23.7	111.5
Walker2d-Medium-Expert	95.6	44.6	108.2
Average	72.9	42.1	74.3
Table 4: The performance comparison between our method and model-based RL methods on D4RL Mujoco offline datasets. Our method works even better than the offline RL methods with extra access to the underlying reward label.
Name	R(s)	PU learning	Policy Retrieval
ours	log c(s)/1-c(s)	two-step with max(·,0)	non-parametric
ours-V1	10c(s)	two-step with max(·,0)	non-parametric
ours-V2	log c(s)/1-c(s)	one step without max(·,0)	non-parametric
ours-V3	10c(s)	one step without max(·,0)	non-parametric
ORIL-01-V1	c(s)	one step without max(·,0)	RL
ORIL-logR-V1	log c(s)/1-c(s)	one step without max(·,0)	RL
ORIL-01-V2	c(s)	two-step with max(·,0)	RL
ORIL-logR-V2	log c(s)/1-c(s)	two-step with max(·,0)	RL
Table 5: Configurations of the variants ablated in Fig. 27. Ours-V1 and ours-V3 use  ${10c}\left( s\right)$  instead of  $c\left( s\right)$  to scale the behavior cloning weight so as to be similar to "ours."