Improving Behavioural Cloning with Positive Unlabeled Learning
Authors: Qiang Wang
Robert McCarthy
David Cordova Bulens
Kevin McGuinness
Noel E. O’Connor
Francisco Roldan Sanchez
Nico Gürtler
Felix Widmaier
Stephen J. Redmond
Venue: CoRL 2023 Poster (2023)
Keywords: Offline policy learning
positive unlabeled learning
behavioural cloning

Abstract: Learning control policies offline from pre-recorded datasets is a promising avenue for solving challenging real-world problems. However, available datasets are typically of mixed quality, with a limited number of the trajectories that we would consider as positive examples; i.e., high-quality demonstrations. Therefore, we propose a novel iterative learning algorithm for identifying expert trajectories in unlabeled mixed-quality robotics datasets given a minimal set of positive examples, surpassing existing algorithms in terms of accuracy. We show that applying behavioral cloning to the resulting filtered dataset outperforms several competitive offline reinforcement learning and imitation learning baselines. We perform experiments on a range of simulated locomotion tasks and on two challenging manipulation tasks on a real robotic system; in these experiments, our method showcases state-of-the-art performance. Our website: \url{https://sites.google.com/view/offline-policy-learning-pubc}.

Improving Behavioural Cloning with Positive Unlabeled Learning
Qiang Wang', Robert McCarthy², David Cordova Bulens', Francisco Roldan Sanchez Kevin McGuinness Noel E. O'Connor94, Nico Gürtlers, Felix Widmaier', Stephen J. Redmond
University College Dublin, University College London, 3 Dublin City University *Iniight SFI Research Centre for Data Analytics, MMI for Intelligent Systems
Learning control policies offline from pre-recorded datasets is a promis- ing avenue for solving challenging real-world problems. However, available datasets are typically of mixed quality, with limited number of the trajecto- ries that we would consider as positive examples; i.e., high-quality demonstrations. Therefore, we propose a novel iterative learning algorithm for identifying ex- pert trajectories unlabeled mixed-quality robotics datasets given a minimal set of positive examples, surpassing existing algorithms in terms of accuracy. We show that applying behavioral cloning the resulting filtered dataset outper- forms several competitive offline reinforcement learning and imitation learning baselines. We perform experiments range of simulated locomotion tasks and on two challenging manipulation tasks on real robotic system; in these experiments, our method showcases state-of-the-art performance. Our website: https://sites. google. wwt//viee/offllneeoooooooo http:///oooo....ooooooeeeaaataa pubc.
Abstract:
Keywords: Offline policy learning, Positive unlabeled learning, Behavioural cloning
Introduction
Data-driven learning methods can discover sophisticated control strategies with minimal human in- volvement, and have demonstrated impressive performance in learning skills across many challenging domains [1, 2, 3, 4]. Nonetheless, data-driven methods are not often applied in real world applications due to the amount of interactions with the environment needed before an effective policy can be learned [5, 7J. Moreover, data acquisition can be costly and/or unsafe in physical environments. This inefficiency can potentially be improved by learning from previously-collected data; i.e., learning policy from historical dataset without needing additional data acquisition from the environment. This is termed offline policy learning.
Standard behavioral cloning (BC) is the simplest offline policy learning algorithm, it aims to find a policy that can mimic the behavior observed in a dataset capturing the performance of given task. The target behavior be cloned usually obtained from an expert; for instance, human [8, 9] or well-perfone pelforeeform scripted agent [10]. BC performs supervised regression, learning control policy that maps observations from a dataset to the corresponding actions taken by a behavior policy. When high-quality expert data is used for training, BC demonstrates high efficiency, and the resulting agent typically exhibits good performance [11].
However, major drawback of BC is its dependence on high-quality training dataset. Specifically, the data collected for BC should come from highly-skilled expert. Moreover, the actions conditioned the environment states must display a unimodal distribution to prevent regression ambiguity [12], as BC follows supervised machine learning paradigm. However, real-world/practinn training datasets are often of mixed-quality, containing examples of both high-quality and low low-quality
is the corresponding author
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

behaviors/data, which can be detrimental to the training process. The presence of low-quality data in these datasets can be attributed to factors such as the involvement of low-skilled agents, non-task- related policies, or environmental noise. If multiple experts are involved, the dataset may furthermore exhibit multi-modal behavior, leading to regression ambiguity BC. In this paper, we aim to study methods allowing the discrimination of a single target expert's data within mixed-quality dataset, and utilize this single-expert subset for BC learning.
1.1
Our work
We assume we are provided with a mixed-quality dataset, as discussed previously, along with a seed- positive dataset consisting of high-quality data generated by an expert. Our goal is to discriminate data in the mixed-quality dataset that shares the same behavioral patterns the seed deed-positive examples to obtain expert dataset suitable for BC. In practice. the seed- positive dataset can be quite small, typically constituting only 0.1% % to 0.4% of the mixed-quality dataset's size. This enables users to obtain the seed-positive examples using feasible methods, such as heuristically/manually sampling from the quality raw dataset or by requesting the target expert to collect a small amount of additional data. Our approach can considered an example of Positive Unlabeled (PU) learning, which constitutes an important subfield within conventional emii-supevisee learning. Concisely, datasets for PU learning comprise portion of data labeled positive, while the remains unlabeled. The objective of PU learning is to leverage the information from positive examples to classify the unlabeled data as either positive or negative.
We establish supervised learning signal by integrating munthtiicall-ggneeaate negative examples and seed-pos sedd-positive examples. Our training methodology adheres to a traditional semi-supervised learning paradigm; it begins the training with small, positive dataset and then iteratively discriminates positive examples from large, unlabeled dataset. These identified examples then added to the positively-labeled subset for the subsequent training cycle, until convergence is achieved. Once the dataset has been labeled using this iterative method, during the policy learning phase, we apply standard BC to this positively-labeled subset.
In general, we refer to our offline policy learning approach as Positive Unlabeled Behavioural Cloning (PUBC). PUBC stands out for its simplicity in implementation, quick training time, and ease of parameter tuning. In our experiment, PUBC achieves excellent performance across wide range of challenging robotic tasks, surpassing several state-of-the-art algorithms.
Related work
2.1 Offline policy learning
Research methods for offline policy learning can largely be categorized as offline imitation learning (IL) or offline reinforcement learning (RL); we refer readers to [13, 14] for comprehensive surveys, and to [15, 16, 17] for research benchmarks.
BC is the simplest form of offline IL. In addition to standard BC as mentioned earlier, researchers have recently proposed Implicit BC [18], which employs an energy-based model [19] to improve BC's performance. Furthermore, Inverse RL [20] is regarded as an alternative branch of IL, focusing on understanding the motivations behind an agent's 's actions by inferring the underlying reward structure, which guides the decision-maie making process in Inverse RL. Additionally, a Generative Adversarial Network (GAN) [21] has been integrated with IL. in [22], where it employs adversarial training with generative and discriminative models to learn the action distribution of the behavior policy.
Offline RL[14] aims to maximize the expectation of the sum of discounted rewards. However, unlike in online RL, no interactions with the environment are allowed. Most off off-policy RL [23] algorithms are applicable offline, but they typically suffer from the issue of outputting ouiti--distiibttiinn (OOD) actions due the distribution shift between the action distribution in the training dataset and that induced by the learned policy [10]. To mitigate this issue, several constraint methods, such policy regularization [24, 10, 25, 26] and conservative value estimates [27, 28] are proposed.

---

Recently, a novel offline RL approach, inspired by transformer models [29], has been introduced in [30, 31J. Unlike conventional RI. methods that depend on policy gradients or temporal differences, this approach takes a distinct route by utilising supervised sequence modelling to fit the policy data distribution. In the policy evaluation phase, the GPT model takes a target return and generates an action sequenee for that goal. This GPT-based method will be bascline in our paper.
Another branch of offline RL perfroms BC while focusing on learning transitions with higher significance in the datasets. It starts by learning advantage functions and subsequently utilizes them to downweight transitions with lower advantage [32, 33, 34, 35, 36, 37]. Similarly, in [38), the authors propose a GAN-like architecture [21] that combines BC and a discriminator. The discriminator selectively picks high-quality expert data from the dataset for BC learning.
2.2 Positive unlabeled learning
The main challenge in PU learning is to acquire negative examples from the unlabeled data to complement the available positive examples for training a supervised classifier. The two-step PU learning approach involves manually labeling subset of negative examples and using them along with positive examples to train the classifier [39]. The classifier can then label the remaining unlabeled data in the dataset. However, this method can be human labor-intensive, and it may not be effective if the underlying patterns of the negative examples are difficult to interpret.
Another solution is to naively treat the unlabeled data as negative examples during classifier training [40]. The classifier can then assign scores to the unlabeled examples, with positive examples typically receiving higher scores. This method has been improved in [41] by using the bagging technique to generate multiple subsets from the unlabeled dataset, which are combined with positive examples to train a series of weaker classifiers. Finally, the output from the classifier ensemble is used to produce a more accurate prediction. However, if an unsuitable loss function is used, biased errors may occur. This problem is addressed in [42] by introducing Unbiased PU learning. More recently, [43] proposed Non-negative PU learning to mitigate the overfitting problem associated with unbiased PU learning. We recommend [44] for an in-depth analysis of PU learning.
3 Positive Unlabeled Behavioural Cloning
3.1 Preliminaries
The offline policy learning problem is formulated in the context of a Markov decision process, M (S, A, R. P. 7) [45], where is the state space, A is the action space, R the reward function, P is environment dynamic and 7 is the discount factor. At each time step, f, the agent gets state 8, E S and outputs an action a₂ E AA according environment, the agent will get reward E R and the environment state transitions to 8141. We assume that we can obtain or are given a positive dataset, D+ = {($ at .r.. +t $141)t=1... with m time steps and a large mixed-quality offline dataset Dimix = {ss7iir amti rrmin s841)) ) with n time steps. We assume m << n, that D+ contains only data collected by the targeted expert, and D Pii includes proportion of data collected by the target expert. In our following description, we define a positive example as the data collected by the target expert, and negative example as any data not collected by the target expert.
policy <aa st); after applying the action to the
3.2 Generating the training examples
Our approach to generating negative examples for training the PU classifier similar to the two- step method outlined in Section 2.2. However, rather than depending on manual selection of negative examples from Dmir. we create them by randomly mixing the states and actions from different sources, including from D.,, from Dmizs and random examples from the state-action space (see Figure 1(a)). Therefore, the set of negative examples can be informally written as: {s+ mmmi))) U(st, ((++)))m, mir at) )n, mm ä)n, U(3, a a)n, U(š, amirr ) (3, 2) }.
D_ = D== where á and ä refer to the random states and actions that follow a uniform distribution within the range of minimum action maximum action; n1 ny correspond the number of state-action pairs generated for each combination of sources. We require that the artificially generated state-action pair be distinct from the state-action pair obtained from the raw dataset.
3

---

D
Update
Example
Clesdlier neural melwork
Example generator
Concatcnale
( at t=
Positive
Datasets
Train
Classifier
g
Label
'mii
R
Negative
(St. Ot )=I...
Encoder
MLL
co ppassed into classifier
(b)
Figure 1: PUBC learning block diagram. (a) Illustration of the iterative PU learning process. The red rounded rectangle shows how positive and negative training examples are generated, with the negative examples generated by intentionally mismatching actions and states from different time points and/or different data subsets to form action-state pairs that would most likely represent poorly performing behaviors; See section 3.2 for more comprehensive explanation. (b) The neural network structure of the classifier includes an encoder to reduce the usually high-dimensional state vector, followed by multilayer perceptron (MLP) classifier, taking input both state / and action vectors, and whose sigmoidal output is interpreted as the probability that the example was generated by an expert agent.
In the first training iteration, we treat the examples in the positive dataset as positive, similar to traditional PU learning. Once the classifier is trained, we use it to identify additional positive examples from the unlabeled dataset. These newly identified positive examples then replace the previous ones.
3.3 Classifier structure
Figure 1(b) shows the network structure of the classifier, which a combination MLPs; takes the state-action pair and outputs the probability that this pair was generated by the actions of the same expert agent that generated (we assume) most of the data in the training dataset, D4. Experimentally. we found that directly inputting a concatenation of raw state-action pairs into the neural network will cause the action to be effectively ignored, as the dimension of the observation space is typically much larger than the action space in practical task settings. Therefore, to reduce the dimension of the observation space, we encode the observation vector into a lower dimensional space before concatenating with the action vector and inputting it to the MLP to obtain a decision. The final layer of the MLP uses sigmoid activation function, sigmoid( 1/(1 + e"),, where the output F(s, a) is interpreted as the probability that the state-action pair generated by expert. Binary cross-entropy is chosen the training loss, L:
f =
E
log(1 F(st, (8))
E
| log F F(sg ax))
(1)
(s.,a))-))
3.4 Additional methods for optimizing PÚ learning performance
The policy used for data collection typically does not change within a trajectory (each interaction episode). The labels assigned all state-action pairs in a trajectory are therefore aggregated and the same label is applied to all transitions in the trajectory. This done by soft voting, taking the mean of the individually predicted probabilities of the state-action pairs in the trajectory, F(st, a,) for member of the trajectory time interval, create confidence score for the trajectory. Subsequently, a threshold, thcout, is applied to the confidence score to binarize it.
Classifying per trajectory:
Adaptive confidence threshold: It necessary to set confidence threshold theouf to discriminate positive trajectories the unlabeled dataset, D other words, all trajectories with confidence score exceeding the threshold classified subsets containing entirely positive examples. We discovered that employing adaptive threshold can enhance the classifier's performance. Briefly, it searches for local minimum in the confidence score histogram and uses this value decision

---

Algorithm 1: PUBC algorithm
Input: Mixed-quality dataset Diiz, positive dataset D++ while D+ not converged do
Randomly sample K subsets {D4 I==....K from D+:
Generate K corresponding negative subsets {D= k;=
Initialise K' classifiers )= with parameters 0,, .., 0k:
for k 1 to K do
Update 0g by minimizing the loss in Equation 1;
end
Get confidence threshold thcoat using the trained (Fer h==1..K;
Update the membership of D+ using Equation 2;
end
Initialise BC network my with parameters of 6;
for epoch f- 1 to epochs do Update by minimizing:
E (et.ar)~DD-
[- log TT (a |t)]
end
Output: 888
boundary, above which it is assumed the data is dominated by appert-generate trajectories (see Appendix for details).
Similar the approach in [41], we employ bagging to enhance the classifier's performance by concurrently learning multiple independent weaker classifiers and combining their individual decisions to determine final decision. The datasets used to train each elassifier subsets of D+, sampled with replacement.
Ensemble learning:
In Appendix D, conducted ablation study these three techniques individually.
3.5 Using the trained classifier and learning the policy
Considering the above techniques, the label decision for a trajectory can be determined using:
f=
Σ
K
Σ
Fe((se (r))
Wheonf
K
where [:] are indicator functions, T is the number of time steps in each trajectory; and K is the number of classifiers in the ensemble, set to an odd number to avoid ties.
The process iterates until trajectories in D+ converge. Once D+ has converged, we train a standard BC model mimic the behavior in D4. The entire semi-supervised training and filtering process for PU classifier as well as the training of BC can be succinetly summarized in Algorithm 1.
4
Experimental results
This section aims showcase the effectiveness of our proposed PUBC method by conducting experiments range of continuous control benchmark tasks, including challenging physical robotic manipulation tasks from the Real Robot Challenge (RRC) III competition and numerous MuJoCo locomotion tasks [47, 48] (see Figure 2).
4.1 Environments, tasks and datasets
4.1.1 Robotic manipulation tasks
Figure 2(a) illustrates the domains of RRC III, which focuses on two tasks: Push and Lift [49]. In the Push task, the cube must be moved to specified positions on the arena floor. The more challenging Lift task requires lifting the cube and maintaining at a target position and orientation.
'A robotic manipulation competition featured in the NeurIPS 2022 Competition Track, more details see https:// http://realrobbee challenge com/. We won the competition [46], with the filter based technique being of our key strategies, and the proposed PUBC method extends this filtering method.

---

In our experiment, we use the mixed datasets provided by the competition organizers for each task. Each mixed dataset collected by a mixture of different policies with varying levels of skill, and a significant portion of the data was collected by domain-specific experts, ensuring high quality. The subsequent discussion will denote these datasets by Lift/mixed and Push/mixed, respectively.
To obtain the seed-positive dataset, D+, we assume that the target expert has generated the trajectories with the highest returns. Therefore, we select the 0.4% episodes with the highest returns as our seed-positive dataset. The selection of this value is further investigated in Appendix D through an ablation study. Post-competition, we received trajectory labels from RRC III organizers, serving as ground truth to assess our PU learning method's accuracy.
4.1.2 MuJoCo locomotion tasks
As shown in Figure 2(b)-2(c), the bodies being controlled in the locomotion tasks comprise segments and joints. Actions are applied to maintain the balance of the body and to move forward. Here, We collected mixed--uality datasets comprising five different structures: 1. E+E: Expert+Expert dataset consisting of data from expert policies with similar performance but different habits/behaviors, with only one expert policy being of interest to us; 2. E+W: Expert+ dataset containing data from one expert policy and one poorly performing agent; 3. E+N: Expert+Noise dataset comprising expert data and an equal amount of domain noise; 4. E+E+W+N: combination of the above four types of data. In addition, we also included an expert dataset E, consisting only of expert data as a baseline. The configuration of each mixed dataset further detailed in Appendix C.
4.2 PU learning results
This section presents the accuracy our PU learning method, comparing traditional PU learning approaches, including Unbiased PU learning and Non-negative (NN) PU learning. The mathematical formulations for Unbiased PU and NN PU learning methodologies are provided in Appendix F. for reference. To maintain balanced comparison, also employed the techniques delineated in Section 3.4 to the baselines.
As illustrated in Table 1, there is no significant difference between the performances of the two baseline methods, while our approach shows a significant increase in accuracy compared these methods. The baseline methods. This could potentially be attributed to the relatively complex data distribution produced by the RRC III environment. In locomotion tasks, both the environment and the task are relatively stable, typically involving the operation of a simulated body complete the single task of moving forward. However, in the RRC III manipulation environment, there is a high degree of randomness; both the initial position of the object and the target are randomly initialized, meaning the tasks completed in each trajectory vary. This results a more complex data distribution. Furthermore, we observed that generally, in real-world environments, the performance of traditional algorithms tends to be inferior to their performance in simulators. This is due non-ideal hardware in physical environments introducing substantial environmental noise, making the data distribution complex.
In unbiased PU learning, introducing a reweighting operation to mitigate positive label data bias can shift the data distribution, resulting in poor model generalization, especially in complex data distributions like the RRC environment.
In the Non-negative PU learning, the weight of the positive samples enforced to non-negative, however, this may hinder the model from capturing essential nuances in the data. For some intricate data distributions, permitting the model to allocate negative weights to positive samples could be instrumental in uncovering the data's inherent structure and relationships more effectively.
In contrast this, the method we propose can effectively tackle these issues, demonstrating both high accuracy and strong robustness.
4.3 Offline policy learning
We present the evaluated policy performance in Table 2, where compare PUBC with other relevant baseline algorithms, including naive aoward-based filter before performing BC, CRR[32),

---

Table 1: Comparing our method's accuracy to baselines in classifying expert vs. non-expert trajec- tories. Accuracy formula: (TP + T (TP TN + FP + FN), where TP (True Positive) and TN (True Negative) denote the correct classification of expert trajectories and non-expert trajectories respectively. FP (False Positive) denotes that non-expert trajectories are incorrectly classified as expert, FN (False Negative) denotes that expert trajectories are incorrectly classified as non-expert.
Dataset
Unbiased PU
NN-PU
Ours
RRC-Sim-LLine Sim-Lift/mixed
82.5%
79.2%
99.7%
RRC-Sim- SiiSSSiimmmmmieee
94.3%
90.5%
100.0%
RRC-Sim Avg
88.4%
84.9%
99.9%
69.4%
64.8%
99.2%
88.7%
90.0%
100.0%
RRC-Real Avg
79.1%
77.4%
99.6% %
Ant E+E
98. 98%1%
95.3%
99.0%
Ant E+W
100.0%
99.2%
100.0%
Ant E+N
99.3%
100.0%
100.0%
Ant E+E+W+N
98.1%
96.9%
98.8%
Ant Avg
98.9%
97.9%
99.5% %
Humanoid E+E
93.4%
94.9%
99.7%
Humanoid E+W
93.7%
94.6%
100%
Humanoid E+N
92.1%
93.6%
100%
Humanoid E+E+W+N
91.0%
89.7%
99.0%
Humanoid Avg
92.6%
93.2%
99.7%
Overall Avg
89.7%
88.3%
99.6 %
DWBC[38] and IQL[24]. We provide a detailed description of these baselines in Appendix F.2 for reference. evident that our PUBC consistently outperforms the baseline approaches across all domains. From the overall scores, we can observe that our method demonstrates a performance advantage of over 12% compared the second-best performing approach.
Employing naive reward-based techniques like 10% BC and 50% BC can extract sufficient expert data from mixed-quality datasets some cases. This is evident from the performance on the Sim- and Real-Push/mixed datasets, and the E+W dataset for both locomotion tasks. However, naive filtering works well only there's ' a large performance difference between expert and non-expert policies, fraction of expert data is known, and reward noise low. Therefore, BC performs poorly on other datasets lacking these conditions.
The advantage-base CRR algorithm performs relatively poorly. This ineffectiveness stems from the challenging nature of estimating advantages in the offline RI. setting, particularly when the environment is stochastic and/or the given rewards are sparse or noisy. In contrast, by considering behaviors over an entire trajectory, our method more accurately identifies target expert trajectories.
While IQL is considered one of the pop-performin offline RL algorithms, our experiments have shown that its performance is not satisfactory on the selected problems. Although offline RI. theoretically has the ability to handle various types of data, including mixed-quality datasets, it is preferable to use high-quality datasets in any scenario. Previous research has demonstrated that offline RL algorithms generally struggle to handle suboptimal roboties data effectively [50]. Furthermore, the additional experiments in Appendix show that our PU method can improve the performance of offline RL. on range of D4RL benchmarks by enhancing the quality of the training data.
The GPT-based policy generation method DT, demonstrates strong performance in most RRC tasks, but it falls short in the Real Lift/Mixed tasks. We have noticed that DT requires about 6ms to generate an action at each time step. However, considering the demanding dexterity requirements of the Real Lift task, can only tolerate maximum delay of 2ms. This computational delay presents limitation that would hinder the practical extension of DT into real-world scenarios. Another constraint is its reliance high-quality data; its effectiveness reduces with noisy datasets, though it performs robustly on the expert's 's locomotion tasks.
7

---

Table 2: Averaged normalized scores of our method and the basclines. Each result is averaged over three training seeds, and training lasts 10 time steps. We evaluate each learned policy for 100 trajec- tories. The score is normalized by scorenorm (score
scoremin )/ //c/sssoeeaa scoremin)
Dataset
Data
50% BC
BC
DT
IQL
CRR
DWBC
PUBC (Ours)
Sif--miimmixedd
0.83
0.39
0.40
0.56
0.73
0.63
0.49
0.71
0.87
0.61
0.64
0.84
0.59
0.80
0.71
0.82
0.80
0.85
RRC-Sim total
1.44
1.02
1.24
1.15
1.53
1.34
1.32
1,51
1.72
M-C-Real-Lif/miiss
0.60
0.32
0.31
0.31
0.44
0.40
0.54
0.65
R--Real-ushh/ixeed
0.44
0.67
0.85
0.58
0.79
0.79
0.81
0.83
RRC-Real total
1.04
1.00
0.89
1.20
1.36
1.49
Ant
0.80
0.84
0.87
0.82
0.76
Ant- E+E
0.78
0.70
0.68
0.73
0.84
0.80
0.68
0.71
0.79
Ant E+W
0.53
0.82
0.80
0.53
0.73
0.79
0.41
0.80
0.79
Ant- -++N
0.42
0.62
0.54
0.47
0.52
0.74
0.73
0.76
0.79
Ant E+E+W+N
0.48
0.70
0.53
0.31
0.67
0.67
0.29
0.69
0.78
Ant total
2.22
2.84
2.56
2.03
2.76
3.00
2.12
2.95
3.16
Humanoid
0.92
0.87
0.91
0.78
0.23
Humanoid E+E
0.90
0.34
0.73
0.69
0.88
0.72
0,26
0.69
0.85
Humanoid E+W
0.58
0.87
0.92
0.29
0.83
0.45
0.46
0.78
0.86
Humanoid
E+N
0.63
0.45
0.37
0.25
0.59
0.11
0.60
0.81
0.86
Humanoid E+E+
0.65
0.57
0.45
0.25
0.72
0.29
0.53
0.60
0.82
Humanoid total
2.76
2.23
2.48
1,47
3.02
1.57
1.85
2.88
3.40
Overall
7.46
7.09
7.43
5.55
8.57
7.06
6.48
8.70
9.76
DWBC demonstrates the second-best overall performance in our experiment. Nonetheless, efficacy notably limited in challenging RRC Sim- and Real- Lift/mixed datasets. The filtering/weighting component of DWBC shares similarities with classical PU learning methods. However, these approaches have limitations when dealing with complex data distributions, mentioned previously.
5 Discussion
In summary, our PUBC method demonstrates superior accuracy and stability compared conven- tional approaches in filtering the expert trajectories from the mixed-quality datasets. Furthermore, PUBC can effectively learn policies from mixed-quality continuous control datasets, outperforming variety of sophisticated state-of-the-ar algorithms.
For certain applications, annotating rewards for all transitions can be costly endeavor, especially in complex, real-world scenarios. Therefore, an alternative approach is leverage our method to extract high-performing trajectories from mixed-quality dataset by incorporating a few demonstration samples instead of manually crafting reward function.
Of course, our methodology has certain limitations that should be acknowledged. Firstly, it is not applicable when the initial sced-positive dataset is unattainable. Secondly, when the data is not group in trajectoric trajectories but consists of disorganized transitions, the performance of PUBC will be harmed.
Lastly, our policy learning algorithm BC is inherently upper bounded by the performance of the expert behavior policy. Indeed, sophisticated RI. related algorithms often learn policies that generalize better and are able to directly learn from unknown datasets. In our future endeavors, we strive to enhance the performance of the trained policy by integrating the principles of our PU learning method with subsequent RL paradigms. Furthermore, we have plans to tackle more complex scenarios, including environments with partial observability and non- non-Markovian dynamics.
6 Conclusion
This paper introduces a new offline policy learning method termed PUBC, an effective approach to identifying transition behaviors generated by a specific expert policy in order to improve the quality of the dataset used for subsequent offline policy learning. This approach allows a learning algorithm to disregard low-skill behaviours, hence improving the performance of the learned policy. In our work, the PU learning method allows naive BC learning algorithm outperform other of of-t--o--hh-an offline RL algorithms in challenging physical problem domains.

---

Acknowledgments
This publication has emanated from research conducted with the financial support of China Scholar- ship Council under grant number CSC202006540003 and of Science Foundation Ireland under grant numbers 17 17/FRL/4833 and SFI/ 12/RI//RRRC//// P__. We are grateful about the invaluable suggestions and comments that reviewers given to help to imporve the quality of this paper. We extend our heartfelt gratitude to Dr. Kevin McGuinness for his invaluable contributions and expertise to this research. It is with deep sorrow that we note he did not live to see the completion of this work. His exceptional insights and unwavering dedication will forever be etched our memories.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 131.vv13/2.5602 2013.
[2] D. Silver, T. Hubert, Schrittwieser, Antonoglou, M. Lai, A. Guez, M. Lanctot, I.. Sifre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with general reinforce- ment learning algorithm. arXiv preprint 1rXiv:1712.018/1 2017.
[3] Q Wang, F. R. Sanchez, R. McCarthy, D. C. Bulens, K. McGuinness, N. O'Connor, M. Wüthrich, Widmaier, S. Bauer, and S. J. Redmond. Dexterous robotic manipulation using deep reinforcement learning and knowledge transfer for complex sparse reward-based tasks. arXiv preprint arXiv:2205.0966 096633 2022.
[4] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference Robot Learning, pages 1101-1112. PMLR, 2020.
[5] D. Yarats, A. Zhang, Kostrikov, B. Amos, J. Pineau, and R. Fergus. Improving sample efficiency model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10674-10681, 2021.
[6] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, 2018.
J. Merel, A. Lefrancq, et al. DeepMind control suite. arXiv preprint
[7] R. McCarthy, F. R. Sanchez, Q. Wang, D. C. Bulens, K. McGuinness, N. O'Connor, and J. Redmond. Solving the Real Robot Challenge using deep reinforcement learning. arXiv preprint arXiv:2109.15333 2021.
[8] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martn, What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint 20Xiv:2108.0322 2021.
[9] Sermanet, Lynch, J. Hsu, and Levine. Time-contrastive networks: Self-supervised learning from multi-view observation. In 2017 IEEE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pages 486-487. IEEE, 2017.
[10] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[11] Merel, L.. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint arXiv:1811.11711. 2018.
[12] S. Levine. Supervised Learning of Behaviors, 2022. URL http::/raill eecs. berkeley. 2. pdf. (Accessed 2022, Oct 10).
[13] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.

---

[14] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[15] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.0722 2020.
[16] C. Gulcehre, Z. Wang, A. Novikov, T. Paine, S. Gómez, K. Zolna, R. Agarwal, J. S. Merel, D. J. Mankowitz, C. Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:7248-7259, 2020.
[17] Fujimoto, E. Conti, M. Ghavamzadeh, and Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.011 2019.
[18] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168, PMLR, 2022.
[19] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial energy-based learning. Predicting structured data, 100), 2006.
[20] A. Y. Ng. S. Russell, et al. Algorithms for inverse reinforcement learning. In ICML, volume 1, page 2, 2000.
[21] I. Goodfellow, Pouget-Abadie Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11): (3(11):139-144, 2020.
[22] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[23] M. Uchara, C. Shi, and N. Kallus. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.063355, 2022.
[24] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2 /0.06616,,, 2021.
[25] W. Zhou, S. Bajracharya, and Held. Plas: Latent action space for offline reinforcement learning. In Conference Robot Learning, pages 1719-1735. PMLR, 2021.
[26] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
[27] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-Jeaning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[28] G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty-basee offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
[29] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[30] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances nearal information processing systems, 34:1 5084-1177,, 2021.
[31] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning big sequence modeling problem. Advances in neural information processing systems, 34:1273-1M 1286, 2021.
10

---

[32] Z. Wang, A. Novikov, K. Zolna. J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[33] Q. Wang, Xiong, L. Han, H. Liu, Zhang, et al. Exponentially weighted imitation learning for batched historical data, Advances in Neural Information Processing Systems, 31, 2018.
[34] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weiggted regresssoo Simple and scalable off-policy reinforcement learning. arXiv preprint
2019.
[35] X. Chen, Z. Zhou, Z. Wang, C. Wang, Y. Wu, and K. Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33: 18353-18363, 2020.
[36] N. Y. Siegel, T. Springenberg. Berkenkamp, A. Abdolmaleki, M. Neunert, Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint 2TXX::2002.088366 2020.
[37] G. Neumann and J. Peters. Fitted q-iteration by advantage weighted regression. Advances in neural information processing systems, 21, 2008.
[38] H. Xu, X. Zhan, H. Yin, and H. Qin. Dissrimminttor-eiighee offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pages 24725- 24742. PMLR, 2022.
[39] A. Kaboutari, J. Bagherzadeh, and F. Kheradmand. An evaluation of two-step techniques for positive-unlabeled learning in text classification. Int. J. Comput. Appl. Technol. Res, 3()): 592-594, 2014.
[40] C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 213-220, 2008.
[41] Mordelet and J.-P. Vert. bagging svm learn from positive and unlabeled examples. Pattern Recognition Letters, 37:201-209, 2014.
[42] M. Du Plessis, G. Niu, and M. Sugiyama. Convex formulation for learning from positive and unlabeled data. In International Conference on Machine Learning, pages 1386-1394 PMLR, 2015.
[43] R. Kiryo, G. Niu, M. C. Du Plessis, and M. Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in Neural Information Processing Systems, 30, 2017.
[44] Bekker and J. Davis. Learning from positive and unlabeled data: A survey. Machine Leurning, 109:719-760, 2020.
[45] M. L.. Puterman. Markov decision processes. Handbooks in operations research and manage- ment science, 2:331-434, 1990.
[46] Q. Wang, R. McCarthy, D. C. Bulens, and S. J. Redmond. Winning solution of real robot challenge iii. arXiv preprint arXiv: 13019, 2023.
[47] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
[48] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: (en:::166000444 2016.

---

[49] M. Wüthrich, F. Widmaier, F. Grimminger, J. Akpo, S. Joshi, V. Agrawal, B. Hammoud, M. Khadiv, M. Bogdanovic, V. Berenz, et al. Trifinger: An open-source robot for learning dexterity. arXiv preprint wxxiv20088.03590 2020.
[50] N. Gürtler, S. Blaes, P. Kolev, F. Widmaier, M. Wuthrich, S. Bauer, B. Schölkopf, and G. Mar- tius. Benchmarking offline reinforcement learning on real-robot hardware. In The Eleventh International Conference on Learning Representations, 2023.
[51] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[52] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[53] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernests,, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. URL pttt::/jjmrr org argpapprssvv22/000 1364. html.
[54] T. Seno and M. Imai. d3rlpy: An offline deep reinforcement learning library. arXiv preprint arXiv:2 100:2//1..378. 2021.
12

---

A Environments
The environments of the tasks considered in our work are illustrated in Figure
(a) TriFinger robot
(b) Ant
(c) Humanoid
Figure 2: (a) The physical TriFinger robot from the RRC III competition, where three identical robotic fingers are equally spaced 120" apart around the circular arena; the coloured cube is the object to be moved. (c)-(b) Illustrate the MuJoCo locomotion task environments.
B Description of how the adaptive confidence threshold is set
We use an adaptive mechanism to adjust the confidence threshold, (lcnnt which is used in Section 3.5 to convert the continuous classifier output probability to a discrete binary label. Figure shows an example of selecting the thoonf value over the PU training iterations for the RRC III competition Push/mixed and Lift/mixed datasets. Furthermore, we presented the filtering accuracy across iterations for these examples to better depict the convergence of accuracy.
We firstly apply the trained classifier to Dmiz to get confidence scores (a probability, as the final layer activation function is sigmoidal) that each trajectory was generated by an expert policy. Secondly, we calculate the histogram of these confidence scores across all trajectories. Finally, we use polynomial fit the confidence score histogram. The threshold determined by identifying the confidence score which the maximum local minimum the x-axis occurs. This point marked with a blue dot in the bottom row of subplots. As the iterative training process proceeds, the trajectories selected by the PU learning changes and eventually the filter output converges, In our work, we define convergence as the condition where the change in trajectory memberships between two consecutive iterations is within a threshold of 2%.
Table 3: Illustration of the convergence process of filtering the RRC Lift/mixed and Push/mixed dataset. Each iteration lasts for 20 epochs. The TP (True Positive) represents an expert-collected trajectories that are correctly classified as expert-collected. FP (False Positive) represents non- expert-collected trajectories that are incorrectly classified as expert-collected. FN (False Negative) represents expert-collected trajectories that are incorrectly classified as non-expert-collected, and TN (True Negative) represents non-expert-colleecee trajectories that are correctly classified as non-expert- collected.
Iteration
TP
FP
FN
TN
200
4
998
1193
2
1005
97
193
1100
Lift/mixed
3
1195
37
3
1160
4
1194
9
4
1188
1421
2
429
1918
Push/mixed
1915
0
5
1100
1920
0
0
1920
13

---

Iteration
Iteration
Iteration 2
500
400
Count
400
300
300
200
300
250
500-
600
400
Amount
400
150
500
1.0
100
100
Probability
Probability
Probability
Raw value
Polyfit curve
th th_conf
(a) Push/mixed
Iteration
Iteration
Iteration 2
Iteration
50-
160
40
80
140
150
Count
100
10-
20-
40-
50
25
100
175
175
150
150
80-
125
Amount
60-
100
20-
40-
50-
75
10-
20-
25
25
0:
0.0
0.2
0.4
0.8
0.2 0.4
0.6
0.8
1.0
0.6
0.7
3.0
0.6
0.9
1.0
Probability
Probability
Probability
Probability
Raw value
Polyfit curve
th_f
(b) Lift/mixed
Figure 3: A demonstration of selecting the adaptive confidence threshold, (hcoots for the Push/mixed and Lift/mixed datasets. The top row of each subplot shows the confidence score counts for all trajectories in the larger dataset, Dmiz- The bottom row displays polynomial curves fitted to these histograms. The position of the blue dot indicates the rightmost local minimum of the polynomial, which determines the confidence value, theoots used for the subsequent dataset filtering iteration.
C Configurations used to collect the locomotion datasets
This section describes the acquisition process of the datasets used in Section We selected two Mu- JoCo locomotion domains, namely Anr-v3 and Humanoid-v3, which have relatively high dimensional state and action spaces, well high levels of difficulty.
Initially, we trained three different agents: target expert policy (E.);; additional expert policy (E_); and weaker performing policy (W). We utilized online RL algorithms for each domain. We trained each agent using different random seeds to ensure that the resulting agents can exhibit diverse behaviors or habits[3]. These trained agents were then deployed interact with their corresponding environments. The interaction data was recorded in the form of {(se, az, Tt, Mr) Mr).......), where represents the ground truth label for each policy, indicating whether the data was collected by the target expert not.
14

---

Table 4: Configuration details on the collection of MuJoCo datasets. We utilized the Soft Actor Critic (SAC) [51] and Twin Delayed Deep Deterministic Policy Gradient Algorithm (TD3) [52], as implemented in mable-baselines: [53], using recommended hyperparameters. In the Humanoid v3
environment, the reset_noise___e was set to 10-2
Ant
Humanoid
Subject
E
E_
W
N
E,
E.
W
N
Algorithm
TD3
TD3
TD3
SAC
SAC
SAC
Train length
10c
10t
2 10
106
10"
3 x 102
Mean return
3034
2910
920
5725
5330
1576
Data amount
5 x 10S
5 x 10
5 x 109
x
x 10.
x 108
5 x 10
5 x 10
Positive example amount
2 x 10a
2 x 10
In addition to these datasets, we also included noise data (N), in which the state, action, and reward were sampled using a uniform distribution within the range of minimum reward to maximum reward. Further details about the online training process, the performance of the trained agents, and the size of the collected datasets are provided in Table 4.
D Ablation study
In this section, we conduct an ablation study to examine the impact of various factors on the performance of our PU learning method. We have structured our study into several groups of ablations. Firstly, we aim to examine the techniques presented Section 3.4. For each group in the study, we remove one technique to assess its effect on the overall performance. Additionally, we conduct an investigation into the size of the dataset seed seed-positive dataset D+. In this part of the study, we gradually increase the size of the D+ starting from 0. 1.1% to 1% of the unlabeled dataset Dini-
From the results shown in Table 5, it is easy to see that adaptive thresholds play a significant role in these relatively challenging physical robotic manipulation domains. Additionally, performing classification at the trajectory level can boost overall accuracy. The bagging not the primary determinant of accuracy, but can serve as beneficial complement. Looking the impact of the size of the positive dataset PU learning, there were several direct failures when the dataset was extremely small. However, once the size increases to 0.4%, it is sufficient for PU learning to achieve optimal performance.
D.1 Guidance on tuning the parameters
Based on our experience, an appropriately chosen of hyperparameters can result in distinct dual peaks in the output histogram within as few as three iterations, as depicted in Figure.3, Among these peaks, the peak representing higher probability corresponds to the expert data. Otherwise, it might indicate a set of poorly chosen hyperparameters; in which case, the following suggestions based on our experience may be helpful:
Setting an adaptive threshold is a critical technique that we highly recommend enabling, especially in scenarios where the expert policy and supposedly non-expert policy (or policies) used to generate the dataset exhibit similar behaviours. In adaptive thresholding. one crucial hyperparameter to consider is the order of the polynomial used to fit the histogram. If the order is set too high, it may lead to overfitting the histogram, causing the adaptive threshold to select thresholds at extremely high probabilities, which could fail to effectively distinguish a sufficient amount of expert data. On the other hand, the order is set too low, may miss the optimal threshold by not fitting the shape of the histogram well enough. We have found that a 10th-order polynomial yields the best results for our case. Empirically, when dealing with complex mixed datasets, such our real-life-mixt lixe- dataset, suggest slightly increasing the order range between 10 and 20. Conversely, in cases of simpler composite datasets, we recommend reducing the order to range between 5 and 10.
2. If you observe that the histogram generated during training consistently exhibits unimodal peak, this might be because you have not classified the trajectory by aggregating the state-actions within
15

---

Table 5: Results of the ablation study. Group represents the removal of trajectory-basee classifica- tion, instead opting for individual state-action pair classification. Group employs a non-adaptive confidence threshold, using a fixed threshold set at 0.5 (considering the sigmoid interval of [0,1], the midpoint of 0.5 is arbitrarily chosen). Group involves using a single classifier model instead of an ensemble (i.e., no bagging). Lastly, the 0.1 % 1% range refers to varying sizes of the D.., where the Dmir size a fixed value. Examples of failure classification are indicated by X, which signifies that the neural network's s learning process has completely collapsed, rendering it unable to learn any meaningful information.
D+ size / Dinu
min size
Datasets
Ours
Group 1
Group 2
Group 3
0.
0.4%
1%
RRC-Real Lift/mxxdd
99.2%
X
92.8%
X
99.2%
99.2%
100.0%
77.8%
X
100.0%
89.8%
100.0%
100.0%
Humanoid E+E
99.7%
89.3%
99.7%
94.8%
X
99.7%
99.7%
Humanoid E+W
100.0%
94.8%
100.0%
100.0%
69.7%
100.0%
100.0%
Humanoid E+N 100.0%
98.7%
100.0%
100.0%
100.0% X
100.0%
100.0%
Humanoid E+E+W+N
99.0%
83.2%
99.0%
91.0%
99.0%
99.0%
it. Classifying single state-action pairs may not effectively capture the inherent policy behavior. On the other hand, classification based on trajectories can take into account the correlation within the data to create a more precise classification relationship. This approach not only improves the model's s accuracy but also accelerates the iterative process. Therefore, we recommend enabling this feature when applicable, such in datasets that follow the D4RL protocol. Another potential reason might be that the initial amount of seed data you collected/separated could prevent the neural network from effectively capturing the behavioral characteristics of the target expert data.
insufficient. This
you notice a converging trend the number of expert trajectories being filtered out, but also observe substantial fluctuations (instability lack of convergence) over subsequent iterations, you might consider increasing the number of classifier models in the ensemble. The quantity of models in the ensemble is relatively straightforward hyperparameter that can be adjusted ascertain the optimal value. This procedure is similar to the typical strategy of adjusting the learning rate.
E Benefits of PU learning in offline RL
This section aims to demonstrate the advantages of employing PU learning in stochastic offline RL algorithms. We utilize three medium-exper datasets available from the D4RL benchmark, specifically, sancheetthh-mediiu---eeeeer
These datasets have previously served as benchmarks in numerous studies [24, 26, 25] and have been effectively addressed by range algorithms. Our objective is to show that the application of PU learning can further enhance the performance of offline RL on these datasets.
poopper-ediumm-xxprr-ia and ceaerr2ddmmmiuuxeeee---e-r-
The aforementioned datasets are all of mixed-quality, each of which was collected by two agents exhibiting different skill levels, specifically, medium and expert. Given that our approach requires a very small-scale seed-positive dataset, we extract the top 0.2% of trajectories based on cumulative reward from each mixed dataset constitute the posiiiosi data subset.
Our results, as displayed in Table 6, include comparisons with various fate-of-theear offline RL algorithms such as IQL [24], TD3+BC [26], and PLAS [25], with BC acting as the baseline. It is evident that the implementation of PU learning enhances agent performance. Examining the learning curves (shown in Figure 4), we can see that PU learning not only accelerates the learning process but also enhances the stability of all the investigated offline RL algorithms.
F Descriptions of compared algorithms
F11 PU learning
Unbiased PU learning [42]: Traditional PU methods train classifiers by minimizing empirical risk, wherein unlabeled examples are directly treated negative examples. This approach, however,
16

---

Table 6: Averaged normalized scores [15] with PU vs without PU for three D4RL benchmark tasks. Each result includes three random seed values and each model training session lasts for 106 time steps. We evaluate each learned policy for 100 environmental trajectories.
PU
BC
PLAS
IQL
TD3+BC
Total
/
0.93
0.92
0.93
0.94
3.72
Halichhetal-mmoddii---eper----
x
0.56
0.74
0.69
0.93
2.92
/
1.11
0.66
0.64
1.10
3.51
Hopper-mediuu--xperrtt00
X
0.51
0.32
0.35
0.89
2.07
/
1.08
1.09
1.10
1.10
4.37
Valker22-mediiun-xxppttv00
X
0.76
0.99
1.06
1.11
3.92
With PU
Without PU
Halichestah LIQ.
Halfchestah PLAS
Halfchentah TDD3+BC
Halfcheetah / BC
Nomaal
40
40
Hogper IQL
Hopper PLAS
Hopger / TD3+BC
Hoaper
Normalse
60
Walker:2D
Walker:2D
Walker-2D
Walkerr:2D
Normalised
40
60
80
100
20
40
60
80
100
0
20
40
60
80
100
20
40
60 80
100
Time step (1e6)
Time step (1e6)
Time step (1e6)
Time step (1e6)
Figure 4: Training curves of Table 6; curves are averaged over three random seeds, with the shaded areas representing the minimun/maximum values across these three seeds. Each data point refers to the average normalised score [15] of 10 environmental episodes.
may lead high empirical risk for the negative class. To address this issue, Du Plessis al. [42] re- weighted the losses for positive and unlabeled examples. Hence the training objective of Unbiased PU learning becomes minimizing the following, where 8 is the proportion of positive examples to unlabeled examples in the unlabeled dataset (same our approach, we set it to 0.4% here):
Lunbiased = (+.,0e)D++
[-log(F(ss ac)) +
E (s
|o - F(st, at))
8
(olo - -(st,
(M,
Non-negative PU learning[43]: While the complexity of the model increases, the risk (Equa- tion 3) on the training may approach become negative, while the corresponding risk on the test set increases. This suggests that the model is overfitting. To tackle this problem, Kiryo et al.
17

---

[43] introduced non negative PU learning, in which the training objective is modified as follows, where again 5 the proportion positive examples to unlaheled examples in the unlabeled dataset (same as our approach, we set to 0.4% here):
Lnon-ney 8 E |- log((F(s a:)]] max(0, (DL,
E
| [-log(1 [ 4x))
8
E (S,0))DD))
log(11 F(st, a))))
Compared to Unbiased PU learning and Non-negative PU learning, the PU method introduced in this paper offers a more decent approach for generating negative examples, with improved diversity and logical correctness. Additionally, our method employs a simpler loss function.
F.2 Policy learning algorithms
Naive reward-base -ased filter BC: Reward may be used for filtering expert data for policy training, experts are more likely to achieve higher rewards. Our results show two variations of this approach: 10% BC and 50%BC, which involve selecting trajectories with the top 10% and 50% highest cumulative (over the trajectory) returns, respectively, then using the filtered subsets train BC.
Critic Regularized Regression (CRR) [32]: A state-of-the-art filter/weight-based BC algorithm. It utilizes reward-based ad advantage function to weight the significance of training examples in the dataset. In fact, the CRR method can be imagined as soft-weighting process, while our method is a heig-wegg process.
Discriminator Weighted Behaviour Cloning (DWBC) [38]: This is similar to our approch. DWBC employs discriminator to distinguish high-quality data from mixed datasets. Additionally, it use a small geed-positive dataset to initiate the training process. The discriminator used here is similar to Unbiased PU learning, but they introduce the additional policy function as input. This approach combines the training process of BC with the discriminator create GAN-like architecture to improve performance. DWBC can be BC, whereas our approach introduces a novel PU learning method to guide BC.
fusion of Unbiased PU learning and
Implicit Q-learning (IQL) [24]: IQL is offline reinforcement learning method that solves the OOD problem by setting implicit constraints. It has previously been used to address the problem of learning strategies in mixed-quality data.
Decision Transformer (DT) [30]: In DT, a causal transformer is employed to model policy trajectories, taking as input the sequence {Ro, S0. Op, R1, $1, 01 ..., Rx-1, SK 4 where represents the desired return-to-go in a trajectory; we specifically set this value to the maximum attainable reward for each domain. 8 denotes the state and represents the action taken each time step. At each time-step the first 3 tokens are fed into the transformer predict the action time f, denoted as p(ay Ru, So, an. .... R2-1, $1-1,
G Additional classification function
Here, we introduce an alternative option for the classification function discussed in Section3.5. In this approach, we employ logistic operations to calculate the product of F::
T Σ
K
log(F (o
2 Uhront
(5)
2
This would introduces trade-off between disregarding positive data by being overly strict and incorporating non-positive data by being too lenient.
H Implementations and training
The neural network architecture for PU learning of the PUBC is illustrated in Figure 5. The PU training for the RRC Sim- and Real- Lif/mie datasets lasted for iterations; for the Sim- and
18

---

Table 7: Hyperparameters of algorithms used in our experiments
Algorithm
Hyperparameter
Barming_rate=00001:: batch sizc= 1024; optimizer = adam;
PU
epochs_per_iierraioonn models in_ sates_neeenneeeee....___ polynomial_ 10derr-00
BC
leariing_rate=00001; bawcbbaaitccch00000 optimizer = adam; epochs=200
actor_learinng = critic_ laal__leanninngartec = 0.0003; batch_size=256;
CRR
optimizer adam; beta=1.0
actor hhtio__eamnnnngrrrrerrriieeeeeen
IQL
optimizer adam; expectile= weeghh
DWBC
Bauning_ratc=0.00011. batch sinecc__see225;;::00 alpha= eta=0.5;
ct==0: no no__u=Falsee
TD3+BC
actor actor_leaming crwttcctitreaaee crittii_eeaamnnnnnrrat = 0.0003; batchs022bacchhssie=25556
alpha-22
actor 1tr___/ammnn__ttt0000000000
rate=0.0001;; critic learing__rac=-0.000 ratc==000t optimizer adam;
PLAS
warmup_stees==5000 beta=0.5
Poal-Push/mixed datasets, iterations; and for each MoJoCo dataset, 3 iterations. each iteration, the subset negative examples, D_, is set to have the same size as the subset of positive examples, D+. Furthermore, the number of each type of negative example, (n1 n7), remains consistent throughout the subset of negative examples, D
Once our work is accepted, we plan to open spercsource our full implementations of PUBC on GitHub. The implementations of the baseline PU learning algorithms from
com/ cwceccimiitae isterss///// pu-learning. All the offline RL and BC algorithms used in our study are sourced from d3rlpy [54]. The implementation DWBC [38] is based on the original authors' work at https: / het///gttttbbbb Com/ryanxhr/DWBC
The key hyperparameters for training each algorithm involved in our experiment are displayed in Table 7. Our experiments ran on PC with an Intel 19-12900F CPU (2.40 GHz x 16, 32 GB RAM) and an NVIDIA 3090 GPU.
FC(Action dim 2) BatchNorm ReLu
Classifier neural network
FC(512) BatchNorm ReLu
FC(256) BatchNorm ReLu
FC(128) Narchhorm ReLu
FC(Action dim
State
FC(128)) BatchNorm ReLa
FC(I) Sigmoid
Output
BatchNorm ReL.
Action
Encoder
Figure 5: The neural network structure of classifier, illustrating the details of Figure 1(b).
19


Reviews:
[{"review_id": "p3RVM1t4mXA", "reviewer": "Reviewer_KCwR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a positive-unlabeled learning scheme for implementing filtered Behavior Cloning. Given a handful of successful demonstrations and a large replay buffer of mixed quality (e.g. some optimal transitions, many sub-optimal), this algorithm first learns an ensemble of classifiers (w/ positive unlabeled learning) to separate out the good transitions in the buffer. Then an agent is trained on these optimal transitions (original optimal demos + filtered transitions from buffer) using behavior cloning. \n\nThe proposed method shows a significant improvement over simple/obvious reward filtered BC baseline, and also improves over competing offline RL algorithms (IQL[1] /DWBC [2] /CRR [3]).\n\n[1] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning.\n\n[2] H. Xu, X. Zhan, H. Yin, and H. Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. \n\n[3] Z. Wang, et al. Critic regularized regression. ", "summary_of_recommendation": "This is a good, well-executed paper that will (at the very least) serve as a good \"sanity check\" baseline for future offline-RL work. The only important issue is missing discussion of reward + sequence modelling [5,6] in related work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the weaknesses I listed above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n* This is a simple and well executed idea and will be a helpful baseline for future offline RL papers.\n* The methods are convincingly tested in real world using the real robot challenge [4]\n\n**Weaknesses**\n* The most obvious missing baseline here seems to be other (potentially more scalable?) alternatives for reward filtered imitation learning via sequence modelling -- like Decision Transformer [5] and Trajectory Transformer [6]. The authors should make a paragraph for this in their related work section, and (potentially) add these methods as baselines.\n* The PUBC method seems to have a lot of moving parts (e.g. ensemble, thresholding, etc.) to generate a score for each transition. An ablation study would be helpful to understand what parts of the proposed classifier scheme are most important.\n\n\n[4] R. McCarthy, F. R. Sanchez, Q. Wang, D. C. Bulens, K. McGuinness, N. O’Connor, and S. J Redmond. Solving the Real Robot Challenge using deep reinforcement learning.\n\n[5] Chen, L., et al. Decision transformer: Reinforcement learning via sequence modeling. \n\n[6] Janner, M., Li, Q., & Levine, S. Offline reinforcement learning as one big sequence modeling problem. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "lBlG1OBlYLg", "reviewer": "Reviewer_aoto", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors proposed a method to improve the imitation performance of offline imitation learning with a mixed-quality dataset in addition to a positive dataset by classifying and extracting trajectories similar to ones in the positive dataset, which were actively used for training BC.\nFor this purpose, the authors developed an algorithm that learns the discriminator whether a given state-action pair should be in the positive dataset or not by generating negative examples heuristically.\nAs a result, the authors contributed to a solid improvement, including a first place in a competition.\n", "summary_of_recommendation": "The effective use of the mixed-quality dataset proposed in this paper is well done and is worthy of publication, as it shows a definite improvement in performance.\nHowever, the criteria for deciding whether or not to add data from D_mix to D_+ are still open to debate.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "While the proposed discriminator can evaluate whether a state-action pair belongs to D_+ or not at each time step, the proposed method classifies whether a trajectory should be moved from D_mix to D_+.\nIt is understandable that the judgment at each time step is prone to misjudgment and does not contribute to performance improvement as shown in the Ablation study, but there must be some trajectories in D_mix that are partially optimal but fail in the rest.\nIf this is the case, then it would be considerable to classify whether adding at each segment of trajectories with appropriate horizon since it might avoid misjudgments enough and ignore non-positive data contained in the trajectories.\n\nAlso, the classification per trajectory is calculated by the summation (average) of F, but the product of F would be also valid.\nI think this is a tradeoff between ignoring positive data by judging too strictly and adding non-positive data by judging too loosely, so I would like to see the discussion which is more effective.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- The mixed-quality dataset was mostly utilized while maintaining imitation in an offline manner.\n- The proposed method achieved the solid performance improvement through extensive validation, including on real robot.\n\nWeaknesses\n- The convergence of the process of extracting data from D_mix to D_+ is not given, so the worst-case computation time for learning is unclear.\n- It is difficult to distinguish between trajectories included in D_+ and trajectories that yield large returns (should we define some meta objectives?), so how to construct D_+ in practice is unclear.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "jZrnor4mK8L", "reviewer": "Reviewer_oaHQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Proposes a semi-supervised method (”Positive unlabeled”) which is able to use a seed dataset of expert trajectories to determine what trajectories are best for training on for BC from a large set of mixed-quality datasets, without the need for reward annotations.", "summary_of_recommendation": "The paper has sufficiently comprehensive experiments including on real robots and the method is intuitive and performant across a variety of metrics. \n\nI would recommend more environments and perhaps a better-presented motivation, but to me this is still at least a weak accept at the moment.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see weaknesses above for details.\n\nI like the method but I think the motivation could be better presented throughout the paper and especially in the introduction. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths and Weaknesses\n\n## Strengths\n\n******************Writing:****************** Overall the paper is pretty well-written an easy to understand. Section C in the appendix does a good job demonstrating how the confidence threshold selection process actually makes quite a bit of sense.\n\n****************Method:**************** The PUBC method is quite simple and demonstrates good performance across a moderately comprehensive set of environments. The authors make some very reasonable design choices, such as thresholding entire trajectories rather than individual (s, a) tuples.\n\n**************************Experiments:************************** Experiments are pretty thorough if including the appendix experiments (the one demonstrating PUBC helps with Offline RL is nice). \n\n## Weaknesses\n\n************************Motivation:************************ The motivation for this paper as described in the intro isn’t that convincing (L 40-42). The paper focuses on the setting where a mixed-quality dataset is provided from which we are trying to extract a single expert’s data from. But why would the expert not label their own data? That is the simplest thing to label when collecting data. A slightly better motivation is presented in L46-48, stating that the authors are attempting to extract data that shares the same patterns as the *************seed-positive************* provided dataset. Perhaps the best way to frame this paper is with respect to *********reward annotations:********* It can be very expensive to annotate rewards for all transitions, especially in complex, real-world settings, so therefore we can use PUBC to extract high-performing trajectories from a mixed quality dataset *************************by providing a few demos************************* instead of manually creating a reward function.\n\n************************Experiments:************************\n\n- Section 4.2 has no *context* for *why* the authors’ method works better than the unbiased and non-negative PU learning baselines. The authors should put a short description of each and their differences for context. (on the other hand 4.3 does a good job)\n\n- I do think more simulated environments should be included in the main paper for comparison with offline RL (why not the entire D4RL suite?), given that many offline learning paper do compare on many mujoco environments.\n\n**************************Minor issues:**************************\n\n- L268: “Is not group” → “is not grouped”", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "EtO1EtKduN", "reviewer": "Reviewer_MudH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a pipeline for improving the behavior cloning from mixed quality datasets. The main idea is to adopt the idea of semi-supervised learning that iteratively labels the samples as positive when the classifier is certain about its prediction. Specifically, it learns the classifier on a small set of expert demonstrations and then iteratively labels the remaining dataset. And the paper introduces several techniques that can further enhance the pseudo-labelling procedure. The method is evaluated on both real-world and simulated benchmarks.", "summary_of_recommendation": "This paper presents a pipeline with a combination of well-executed techniques, exhibiting strong experimental results on both real-world and simulation benchmarks. But it's missing some qualitative analysis and detailed discussion on several components crucial for making the method indeed work, hence I recommend the score to be weak accept at the current status of the draft.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the concerns and questions in Weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- A well-executed pipeline with a combination of simple or already existing methods, exhibiting the strong experimental results. Not every paper should have fancy novel components and I appreciate the main contribution of this paper.\n- Clear writing with illustrative figures and pseudocode\n\nWeaknesses\n- Qualitative analysis that illustrates how the illustrative labelling procedurue continues can be helpful for understanding the method, like providing the learning dynamics with accuracy or some visualization can be helpful.\n- It seems like the proposed pipeline quite heavily depends on the additional methods proposed in Section 3.4, as the ablation study shows that the method collapses without them. It might be worth moving the ablation study results to the main section and put a more discussion and practical guides for future researches, which could be helpful for fully understanding how the method works and useful for researchers who are willing to adopt the pipeline.\n- 'Accuracy' in the caption of Table 1 is a bit confusing. Please specify what 'accuracy' means here.\n- Please fix minor typos in the draft (e.g., Perfroms in line 85, Section ?? in line 410)", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have agreed that the paper presents a conceptually simple yet effective method to an important problem (offline RL) with extensive experimental evaluation. The AC agrees with the reviewer's assessment and recommend the paper to be **accepted** and presented as a **poster**.
Decision: Accept (Poster)