Robust Offline Imitation Learning from Diverse Auxiliary Data
Authors: Udita Ghosh
Dripta S. Raychaudhuri
Jiachen Li
Konstantinos Karydis
Amit Roy-Chowdhury
Venue: Accepted by TMLR ()

Abstract: Offline imitation learning enables learning a policy solely from a set of expert demonstrations, without any environment interaction. To alleviate the issue of distribution shift arising due to the small amount of expert data, recent works incorporate large numbers of auxiliary demonstrations alongside the expert data. However, the performance of these approaches rely on assumptions about the quality and composition of the auxiliary data, and they are rarely successful when those assumptions do not hold. To address this limitation, we propose Robust Offline Imitation from Diverse Auxiliary Data (ROIDA). ROIDA first identifies high-quality transitions from the entire auxiliary dataset using a learned reward function. These high-reward samples are combined with the expert demonstrations for weighted behavioral cloning. For lower-quality samples, ROIDA applies temporal difference learning to steer the policy towards high-reward states, improving long-term returns. This two-pronged approach enables our framework to effectively leverage both high and low-quality data without any assumptions. Extensive experiments validate that ROIDA achieves robust and consistent performance across multiple auxiliary datasets with diverse ratios of expert and non-expert demonstrations. ROIDA effectively leverages unlabeled auxiliary data, outperforming prior methods reliant on specific data assumptions.

Published in Transactions on Machine Learning Research (04/2025)
Robust Offline Imitation Learning from Diverse Auxiliary Data
Udita Ghosh University of Colifornia, Riverside
ughos0023wur..ele edu
Dripta S. Raychaudhuri AWS AT Labs
drayc001 @uer edu
Jiachen Li
jaachen.li@uce edu
University of California, Riverside
Konstantinos Karydis
kkarydis Bece. ucr edu
University of California, Riverside
Amit K. Roy-Chowdhury University of California, Riverside
amitro@Bece.urr.co
Reviewed on OpenReview: https: openreview. net/ forum? id= (d-HyXKAldqa
Abstract
Offiine imitation learning enables learning policy solely from set of expert demonstrations, without any environment interaction. To alleviate the issue of distribution shift arising due to the small amount of expert data, recent works incorporate large numbers of auxiliary demonstrations alongside the export data. However, the performance of these approaches rely on assumptions about the quality and composition of the auxiliary data, and they are rarely successful when those assumptions do not hold. To address this limitation, we propose Robust Offiine Imitation from Diverse Auriliary Data (ROIDA). ROIDA first identifies high-quality transitions from the entire auxiliary dataset using learned reward function. These high- reward samples are combined with the expert demonstral insistratiinsss tions for weighted behavioral cloning, For lower-quality samples, ROIDA applies temporal difference learning to steer the policy lowards high-reward states, improving long-tern berm-erm returns. This two-pronged approach enables our framework to effectively leverage both high and low-quality data without any assumptions. Extensive experiments validate that ROIDA achieves robust and consistent performance across multiple auxiliary datasets with diverse ratios of expert and non-expert demonstrations. ROIDA effectively leverages unlabeled auxiliary data, outperforming prior methods reliant specific data assumptions. Our code is available at https://github.eo cactuuu..oo/uute htw.t//ooooo//uooo/ll.o wudittghoso/oooooooooooooooooooe roida.
1 Introduction
Integration of deep neural networks in reinforcement learning (RL), coupled with the development of efficient training algorithms, has yielded remarkable performance across a wide variety of sequential decision-making tasks, such as playing games (Mnih al., 2015; Silver et al., 2017; 2018) and solving complex robotics tasks (Duan al., 2016; Levine al., 2016; Kaufmann et al., 2023). Despite this progress, two challenges still remain: the need for extensive enviroument interactions (Levine al., 2020), and the inherent difficulty in designing reward functions for complex real-world tasks (Abbeel & Ng, 2004).
Imitation lcarning (IL), where an agent learns directly from task demonstrations, has been employed as one way to lackle the aforementioned challenges (Abbeel & Ng, 2004; Ross & Bagnell, 2010; Ho & Ermon, 2016).
Work done outside A Amaonn

---

Published in Transactions on Machine Learning Research (04/2025)
IL methods can be categorized online or offline. Online IL algorithms rely experiences gathered from the environment by executing intermediate policies during training (Ho & Ermon, 2016; Kostrikov et al., 2019). However, online interaction may be infeasible, unsafe, or expensive in many real-world settings. Offline IL provides a safer altornative, where agents learn solely from pre-collected expert demonstrations without environmental interaction. Offline IL methods like behavioral clouing (BC) (Bojarski al., 2016) remove the need for online experience. However, offline imitation remains vulnerable distribution shifts as a result of error accumulation over time (Ross et al., 2011).
To address the challenge of distribution shift, recent offline IL methods incorporate substantial number of auxiliary imperfect demonstrations alongside expert demonstrations. These auxiliary demonstrations are not expected to meet any optimality criteria, encompassing a mix of expert, near-expert, and non-expert trajectories. A recent work, DWBC (Xu et al., 2022), treats this auxiliary data as a mixture of expert and sub- optimal data, and trains discriminator for weighted behavioral cloning. On the other hand, DemoDICE (Kim et al., 2022) performs state-action distribution matching on the auxiliary data as a regularization term, in addition to matching the distribution over the expert set by solving convex optimization problem in the dual space. Both methods share the assumption that some high-reward behavioral data are present in the auxiliary datasct, and consequently, utilize only these expert transitions for policy learning by filtering out the non-expert trajectories. However, as shown in Fig. 1, their performance fluctuates as the proportion of expert data in the auxiliary set varies, since they fail to leverage the information available in the non-expert data. Although non-expert data may not explicitly provide knowledge of the optimal policy, it contains substantial dynamics information for the agent. In practical scenarios, is highly unlikely that the quality of the demonstration data in the auxiliary dataset will be known priori. Thus, we design an offline II. algorithm that is more robust to the demonstration quality in the auxiliary data. Our approach does not make any assumptions about the quality of the auxiliary data and achieves reasonably consistent performance as the proportion of high quality to low quality data varies, as shown in Fig. 1.
In this paper, we present Robust Offline Imitation from Diverse Auxiliary Data (ROIDA), an algorithum that combines the simplicity of BC with the capabil- ity of offline RL to leverage transition data of varying quality in the auxiliary dataset. ROIDA does not impose any assumptions on the composition of the auxiliary dataset, and can effectively utilize diverse auxiliary datasets encompassing different ratios of ex- pert and non-expert demonstrat ions. To achieve this, we identify potential expert state-action pairs the auxiliary set and assign large weights to these sam-
Composition of auxilary dataset Low-quality
RODDA
High-ooward Urajec tories in muallary dataset
High-qualty
Proportion High-g pihhtpalty
ples in the subsequent weighted BC objective. This involves two key steps: 1) training discriminator to distinguish between expert and non-expert transi- tions using PositiveUUnlabelie (PU) learning (Elkan & Noto, 2008), and 2) applying weighted BC to all state-action pairs in the auxiliary data, with weights derived from importance sampling ratios based on thresholded scores provided by the discrim- inator. However, as previously mentioned, the aux- iliary dataset may lack expert state-action pairs. To address sub-optimal transitions, we perform tempo- ral difference learning using the importance sampling ratios from the discriminator as rewards. This ap- proach aims to guide the policy toward the expert states, thereby improving long-term returns (as measured by the discriminator) on states not observed by experts. This guidance allows ROIDA to extract value from low-quality transitions, in addition to expert behaviors, in contrast to previous works.
70
80 90
Nomalzzdd
Figure 1: Robustness to composition of auxiliary data. Performance of existing offline IL algorithmss, such as DWBC (Xu et al., 2022), varies significantly depending on the amount of high-quality transitions present in the auxiliary data (given expert set is kept fixed). In contrast, ROIDA more robust, highlighting its ability to extract information even from low-quality transitions. The setup shown here is on the Hopper environment; refer to Sec. 5 for details.
2

---

Published in Transactions on Machine Learning Research (04/2025)
Experiments on the D4RL benchmark (Fu et al., 2020) show that ROIDA consistently achieves high performance across seven environments, using auxiliary datasets with varying proportions of expert data. This consistent success highlights ROIDA's 's ability to leverage diverse unlabeled data without assumptions on data quality. In contrast, cxisting offline IL methods perform well only in selective scenarios that match their specific assumptions about. the data composition. Our approach is the first to relax the dala quality assumptions of the auriliary dataset, utilizing it gain comprehensive knowledge about the expert policy and the environment. No other method fully utilizes both the expert and sub-optimal demonstrations in the auxiliary data for policy learning, leading to suboptimal performance.
To summarize, our primary contributions are as follows:
1. We analyze state-of-the-art (SOTA) offline IL methods that utilize auxiliary data along with a small expert set. Our empirical analysis highlights the unrealistic assumptions of these methods, particularly regarding the composition of the auxiliary set. With this in mind, we design an offline IT. algorithm, ROIDA, that addresses the limitations of these different methods and remains robust. to the quality of demonstrations in the auxiliary dataset.
2. ROIDA incorporates PU learning alongside temporal difference learning to effectively utilize both expert and sub-optimal transitions in the auxiliary data.
3. We empirically validate that ROIDA is robust to the quality of the auxiliary data and consistently achieves high performance across different environments.
2 Related Works
2.1 Imitation learning
Imitation learning (Schaal, 1999) leverages expert demonstrations to train a policy that successfully mimics the expert's behavior. A common approach is behavioral cloning (BC) (Pomerleau, 1989; Bojarski et al., 2016), which frames policy learning a supervised learning problem. However, BC exhibits sub-optimal performance in states distant from the training data (Ross et al., 2011). Alternatively, inverse reinforcement learning (IRL) first. learns a reward function to explain the demonstrated actions before using it to train a policy through any RL algorithm. While popular IRL algorithms (Ho & Ermon, 2016; Fu et al., 2017; Abbeel & Ng,, 2004; Ziebart et al., 2008) can outperform BC, the majority are online methods requiring a substantial number of environment interactions during training, resulting poor sample efficiency. To circumvent the need for environment. interactions, several offline IRL methods (Kostrikov et al., 2019; Swamy et al., 2021; Garg et al., 2021) based adversarial training have been proposed. However, these approaches assume that all demonstrrtionns are equally good, resulting in performance degradation when demonstrations contain sub-optimal data, as in our case.
2.2 Learning from noisy demonstrations
Various approaches have been introduced to address the challenge of imitation learning from sub-optimal or noisy experts (Wu et al., 2019; Tangkaratt et al., 2020; Brown et al., 2019; Wang et al., 2021; Sasaki & Yamashina, 2021). However, many of these works rely on strong assumptions about the dataset, such as the expert data dominating the majority of the offline dataset (Sasaki & Yamashina, 2021) or defining sub-optimality as additive Gaussian noise to the action (Tangkaratt et al., 2020). Other works assume that trajectories are provided with labels indicating degree of optimality (Wu et al., 2019; Wang al., 2021) or preference rankings between trajectory pairs (Brown et al., 2019). Furthermore, these approaches require environment interactions during learning while we focus on the offline setting.
The ofline IL setup with an auxiliary dataset was first explored in DemoDICE (Kim et al., 2022). DemoDICE conducts state-action distribution matching over the expert set and introduces a regularization constraint to ensure the learned policy remains close to the behavior policy of the auxiliary dataset. DWBC (Xu et al., 2022) treats the auxiliary data as a mixture of expert and sub-optimal data, and utilizes positive-umlabeld learning to train discriminator for weighted behavioral cloning. Both approaches encounter challenges
3

---

Published in Transactions on Machine Learning Research (04/2025)
when the auxiliary data is highly sub-optimal and might even exhibit inferior performance compared to counterparts utilizing only expert data. In recent work (Shao et al., 2023), the authors propose an offline IL algorithm specifically for cases where no expert data is present in the auxiliary dataset. This approach assigns a reward of 1 to the expert transitions and 0 to all auxiliary transitions, employing an offlnee RL approach alongside BC. While this design can enhance performance when there no expert data in the auxiliary dataset, the assignment of zero reward to all auxiliary transitions can lead to poor performance when the proportion of experts in the auxiliary dataset is increased. In practical settings, is highly unlikely that the data quality in the auxiliary dataset will be known beforehand. With this in mind, design offline IL algorithm that addresses the limitations of these different approaches and remains robust to the quality of demonstrations in the auxiliary dataset.
2.3 Offline reinforcement learning
Offiine RL (Levine et al., 2020) aims to learn policies by utilizing static offline datasets without requiring additional interactions with the environment. Notably, in offline RT, the training dataset is permitted to contain non-optimal trajectories, and the reward for each state-action-next state transition triplet is known. Our algorithm takes inspiration from subset of methods within the oflline RL literature, specifically those employing filtered advantage weighted regression (Peng et al., 2019; Wang et al., 2020; Nair et al., 2020) and behavioral cloning augmented off-policy learning (Fujimoto & Gu, 2021) In another recent work, UDS (Yu et al., 2022) utilizes an auxiliary set without reward labels in addition to the usual reward labeled offline RL dataset. The approach applies zero rewards uniformly to any unlabeled data and can be effective in highly specific offline RL scenarios. Different from this related work, we do not have access to any reward-labeled dataset in the offiin IL sctting.
3 Problem Setting
We formulate our problem using the standard fully-observable Markov Decision Process (MDP) frame- work (Sutton & Barto, 2018). An MDP M is characterized by the tuple AS, TT 7, P.Po,, where S denotes the state space, Po denotes the initial state distribution, and A represents the action space. At each time step t, given a state s, E S, the agent selects an action a, E A according to its policy (a,,s;) E A(A), where ((A) denotes the probability simplex over A. Following the execution of action a,, the MDP transi- tions to new state 8141 E S based on the transition probability T(s441|$1, az), while the agent reccives a reward r(st, 0z) E IR. The primary objective for the agent is to maximize the expected discounted reward E [Σ, 7"r(s, at]] with discount factor 7 E [0, 1]. The state-action distribution of this policy æ under the transition function T is defined as d= = =1 (1- 101-27 7) d'df, where df is the distribution of (s,, ae) under π at step
In the offline IL setup we do not have access to the reward function Instead, we utilize set of demonstrations Ng provided by an expert policy TE in the form of dataset of expert tuples Dr [(s, 0i, == where (s, is sampled from s= and s' is sampled from T(s'ls, amount of pre-collected demonstrations Do = {8x, ai, s;) No gathered by some unknown behavior policy from the distribution no (No >> Ng). It is important to note that these tuples are not presumed to satisfy any ipaimal criteria for the specific task at hand. Given this expert set DE and the auriliory sel Do, our objective is to learn a policy x² capable of maximizing the unknown reward r, without the need for direct interaction with the environment.
[s, a). Additionally, we assume access to a substantial
4 Method
4.1
Overview
As discussed in Sec. 3, the auxiliary demonstrations are not expected to adhere to any optimality criteria, and can contain a mix of expert, near-expert, and non-expert trajectorics. Existing methods rely on certain assumptions about the quality of this data to effectively utilize them. DWBC (Xu et al., 2022) and DeCODICE. (Kim et al., 2022) assume the presence of high-reward expert data in the auxiliary dataset. Tn
4

---

Published in Transactions on Machine Learning Research (04/2025)
practice, knowing the data quality in the auxiliary dataset beforehand is highly unlikely. Thus, propose ROIDA to effectively leverage transition data of varying quality in the auxiliary dataset. ROIDA uses two key ideas to achieve this.
First, ROIDA aims to emulate expert behavior by considering both expert demonstrations and any task-optimal state-action pairs present in the auxiliary dataset. To accomplish this, employ discriminator d(ss, a) trained using PU learning to approximate reward f(s, a) for each state-action tuple in the auxiliary set (Sec. 4.2). This reward assesses the optimality of each data point in the auxiliary set compared to the expert set. When the reward surpasses a designated threshold, ROIDA includes the corresponding instance as an approximate expert data point, applying weighted BC on this sample with the discriminator oulput acting the weight (Sec. 4.3).
Stage 1: Reward model training
D
PU learning
d
F(s,a)
Do
Add reward labels auxiliary data
However, the auxiliary dataset may contain significant number of state-action pairs that do not exceed this reward threshold. Rather than completely excluding these samples from optimization, ROIDA incorporates its second critical element: leveraging the transition information in the data via temporal difference learning, using the estimated rewards (Sec. 4.4). This approach aims to steer the policy toward high-reward states, thereby enhancing long-term returns on states for which optimal actions are not readily available. This two-pronged guidance enables ROIDA to extract value from low-quality transitions addition expert behavior, without imposing any assumptions on auxiliary dataset composition. Fig. 2 presents overview of our approach.
Stage 2: Policy training
TD learning
Do
D
Q
Weighted BC Reward threshold
BC
Maximize Q value
4.2 Learning reward model
In order to perform both weighted BC and temporal difference learning the auxiliary dataset, construct reward model by training discriminator d(s,a) to discern between expert and sub-optimal transitions. Unlike prior approaches (Ho & Ermon, 2016; Kim et al., 2022) that treat this as a standard binary classification task, designating all samples from Do as negative, we opt for PU learning the discriminator training process. This decision is driven by the potential presence of expert transitions within the auxiliary dataset. As a result, consider the auxiliary dataset as unlabeled set, encom- passing both positive samples (expert statc-action transitions) and varied negative samples (non-experl transitions), with the expert dataset serving as the labeled positive dataset.
Figure Framework overview. ROIDA first learns reward function using PU learn- ing. It then identifies high-reward expert-like transitions and combines them with the expert data for weighted BC (Sec. 4.2,4.3). To extract value from lower quality samples, ROIDA ap- plies TD learning, steering the policy towards high reward states (Sec. 4.4). By combining weighted BC and TD learning, ROIDA effec- tively leverages uncurated offline data.
The core idca in PU learning is to ro-weight the different losses for the positive and the unlabeled data in effort to derive an estimate of the model loss on negative samples, which is not directly accessible. Due to the limited amount of expert data, we use non-negative risk estimator described in (Kiryo et al., 2017) in order to make the discriminator more robust:
min E [- logd(ss))) max (o, E (s.0)~DB
[- log((1 d(s,a))) -1
E. |- log(1 d(s, ())] (1)
((..epDD (s.a)~Dz
Here, y is hyperparameter that represents the positive class prior.
5

---

Published in Transactions on Machine Learning Research (04/2025)
Given the trained discriminator, we calculate the reward for each state-action tuple in the auxiliary dataset as follows,
d(s,
F(s, a) = log 1- d(s,a)
(2)
where d(s, a) is clipped to the range of (0.10.9) to prevent unbounded rewards. All samples in Dg are assigned value of d(s, a) = 0.9 and the reward is calculated accordingly.
The specific form of the reward is inspired by DICE (Distribution Correction Estimation) approaches (Kim d= (s,a) This ratio serves as an indicator for the et al., 2022; Ma et al., 2022) which try to estimate log d (s. importance of state-action tuple; higher values mean that the expert often takes the action at state While DICE methods perform this estimation by training simple binary classifier, we use PU learning to train a more robust. discriminator to prevent treating expert samples from Do as negatives.
4.3
Neward-weighted behavioral cloning
Using the rewards obtained from the discriminator, we can identify the expert transitions in the auxiliary dataset for policy training. Instead of directly employing the rewards as weights, we employ direct thresholding scheme to exclude highly sub-optimal state-action tuples. Finally, we use the filtered samples, alongside those from the expert dataset, to perform weighted behavioral cloning:
min E |- log (a|s)] + γ (B.0)-DE
E [olgs=(als) f(s.a) 1 17>77 7]
(s,a)~Do
Here, is a hyperparramete that governs the strength of thresholding and helps balance between excluding sub-optimal transitions and incorporating expert transitions from the auxiliary dataset. Hyperparameter a is used to weigh in the overall BC loss from the auxiliary dataset. The auxiliary dataset may include state-action pairs which fall below the reward threshold. Instead of discarding them, we integrate these samples into the training process via temporal difference (TD) learning, as elaborated below.
4.4 TD learning using learned rewards
Despite not moeting the reward threshold, sub-optimal state-action tuples possess the potential to enhance the learned policy, augmenting its robustness against distribution shifts during deployment. This is due to the broader coverage of the state space within the auxiliary dataset, surpassing the limited span of the small expert dataset. In the absence of access to expert behavior in these states for direct learning, we employ shortest path strategy to leverage these samples. More precisely, objective these states is to steer the policy toward states observed by the expert and subsequently imitate the expert's behavior accordingly. To accomplish this, we use a TD3-stylc (Fujimoto & Gu, 2021) algorithm to learn Q-function using the approximated rewards and then direct the policy to maximize the long-term return on these expert-unobserved states. The Q-function is learned as follows,
argmin Q
Σ
(B=Q(s.))-((s...
(4)
(
where D denotes the Bellman operator, that is
Bd ()((s,a)=((s,0 = + Σ (T(a))s))((s'' d')l
(5)
a'eA
Using this Q-function we can formulate our refined policy learning objective as
min E log r (als)] + + If (s.a)>DK
E |- loggs((ass) f(s, a) 1|F > 7 + B (s.a)>Do
E DDDUDDO
(6)
Here, hyperparameter which controls the contribution of the Q-loss. By maximizing this Q-function alongside the weighted BC objective, incentivize the policy to: 1) act optimally in states where have

---

Published in Transactions on Machine Learning Research (04/2025)
expert actions, and 2) guide the agent efficiently from expert-unobserve states to expert-observed states and act optimally subsequently.
The pseudo-code for the overall framework is presented in Algorithm 1.
Algorithm Robust Offline Imitation from Diverse Auxiliary Data (ROIDA) )
Require:
Dataset DE and Dg, hyperparameter 77, a, B,
1: Initialize the imitation policy n, the discriminator and Q-function approximator Q
Train discriminator with non-negative PU learning following Eqn. 1
for t=1 to T do
Sample (8e, ae) ~ DE and (so, Do to form training batch B
5: Compute log rtlss) values for samples in B using the learned policy
6: Compute Q-function output values Q(s, a) using sampled (s, a) and (*,(s, a) using Equ.
7: Update Q by minimizing the learning objective given in Equ. 4
if mod frey then
Update ı by minimizing the learning objective in Eqn. 6
10: end if
11: end for
5
Experiments
In this scction, analyze the effectiveness of ROIDA for offline IL by utilizing an unlabeled auxiliary dataset. We begin by explaining our experimental setup, including the datasets used and the baseline methods for comparison. Next, we evaluate ROIDA against these baselines across multiple imitation learning scenarios. Specifically, our experiments aim to answer two key questions:
How robust is ROIDA when the quality of the auxiliary data varies? (Sec. 5.3.1)
How does ROIDA compare to other methods as the size of the expert dataset changes? (Sec. 5.3.2)
In addition to benchmarking against other methods, we perform ablation studies to analyze the contribution of each component of our framework and its scalability.
5.1 Experimental setup
We conduct experiments on locomotion and manipulation tasks from the D4RL benchmark (Fu et al., 2020).
Locomotion We use MuJoCo environments for the locomotion lasks: hopper, halfcheelah, walker2d, and ant. For expert demonstrations in each environment, we use the corresponding dataset: hopper-expert-u2 eappeeetta,-e,er--ee,, walker@ddexperrt-d2 and ant-eapert-v22 For the sub-optimal demonstrations, we source trajectories from the respective random-v2 datasets. To create Dg, we randomly sample 5, 7 trajectories per environment. depending the chosen setting. For Do, we create 3 settings per environment: 1000 randomly sampled sub-optimal trajectories plus another 0, 3, or 5 expert trajectories. This allows us to test our method's ability to identify and leverage expert demonstrations within different mixes of sub-optimal and expert data.
Manipulation We evaluate our method on 4 ADROIT manipulation lasks from using simulated 24 DoF hand: pen twirling, hammering nail, opening door, and relocating ball. For expert demonstrations, we sample 50 trajectories from pen-ezpert-vl, hammer-ezpert-vl,, door-erpert-vl, and relocate-ezpert-v respectively. For the sub-optimal demonstrations, we use 1000 trajectories from the datasets pen-cloned-vl, hamamer-cloned-vll, door-cloned-v1l, and relocule-cloned-r1, plus 0, 30, or 50 expert trajectories from the corresponding expert-ul datasets.
Note that for all the evaluations, we report the average of the mean normalized score for the last 10 evaluations of training over 5 random seeds. Additional implementation details for method can be
7

---

Published in Transactions on Machine Learning Research (04/2025)
found in Appendix A.2. We also present evaluations using the rliable (Agarwal et al., 2021) framework in Appendix A.3.
5.2 Baselines
We compare ROIDA against the following algorithmss
BC-exp: BC-exp denotes behavioral cloning solely on the expert dataset Dg. Since Dg contains only a small number of expert demonstrations, training only on this data can lead degraded performance test. time due to compounding errors caused by distribution shift.
BC-all: In this casc, both Dg and Do are used to learn the policy. Despite the large number of demonstrations, significant portion of them are random or of low qualily. As result, the learned policy tends to be sub-optimal because of the inclusion of poor demonstrations.
DWBC: DWBC (Xu et al., 2022) treats the auxiliary data as a mixture of expert and sub-optimal data, and utilizes PU learning to train discriminator for weighted behavioral cloning. They perform dual learning strategy where they alternately train the discriminator and the policy by taking the output of each model as an imput to the other. Due to relying solely on BC, DWBC performs poorly when the number of expert transitions is low.
DemoDICE: DemoDICE (Kim et al., 2022) conducts state-action distribution matching over DD and introduces a regularization constraint to ensure the learned policy remains close to the behavior policy of Do. It shares the same drawbacks as DWBC due to the second term, resulting in a suboptimal policy especially when Do contains a large proportion of noisy data.
ORIL: ORIL (Zolna et al., 2020) first learns a reward function and then performs Critic- Aattic-Reuularieed Regression (Wang et al., 2020) to learn the policy by enriching the data using different augmentation strategics.
5.3
Results
5.3.1
Varying the quality of auxiliary data
Table demonstrates how imitation performance changes with auxiliary data of different quality levels. The Setting column denotes the specific quality level; here, z/y indicates using x expert trajectories in Dø and y expert trajectories Do. We evaluate three auxiliary datasets of inercasing quality (higher number of expert demonstrations for each of the environments. To evaluate each method's capacity to exct maximal information from the auxiliary data, regardless of its quality, we present the average performance across all these settings (shown in gray).
Our results show that ROIDA significantly outperforms the baselines, achieving the best performance on 21 out of 24 tested scenarios. Most notably, ROIDA attains the highest average performance across all auxiliary datasets for every environment. As expected, the performance generally increases with higher quality auxiliary data. However, ROIDA able to extract substantially more information from the auxiliary datasets even when those contain lower quality trajectories. This demonstrates the robustness of our approach across diverse settings, without relying on assumptions about the data. In summary, ROIDA consistently outperforms baselines, especially with lower-quality auxiliary data. This highlights ROIDA's effectiveness at leveraging unlabeled data for offline imitation learning.
The poor BC-exp results highlight the challenge of imitation learning with scarce expert data. BC-all uses all available offline data for cloning, without accounting for polentially low-quality policies in the unlabeled data. This often leads to weaker performance than BC-exp.
Although DWBC achieves the second highest average performance, it degrades substantially compared ROIDA when auxiliary data quality is poor (5/0 and 5/3). This stems from DWBC's inability extract.

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Imitation learning performance on locomotion (first rows) and manipulation (next rows) tasks from the D4RL benchmark. Results are shown as the number of expert demonstrations in Do increased. The best performing method on each task is highlighted in red and the second best in blue.
Method
Env.
Setting
BC-exp
BC-all
DemoDICE
ORT.
DWBC
ROIDA
5
0
2.12 + 0.26
38.56
8.65
3.83 = 1.24
72.04 + 36.82
84.63 + 16.01
5 / 3 5
2.38 + 0.76
51.00 + 17.65
15.69 t 18.78
74.62 + 11.58
86.66 +: 21.94
Hopper
2.67 + 1.05
68.82
15.36
17.83
1: 21.72
80.85 + 23.56
88.45 +: 8.46
Aeg.
67.15 1 16.03
2.39 1. 0.76
52.80
14.40
12.45
16.59
75.84 1. 26.11
86.58
16.42
/
2.25 +
2.25
0.00
2.25
0.00
9.02
2.88
15.89
9.60
3
2.25 +
3.20
0.21
2.25
0.00
16.04 + 6.40
18.73
3.67
HalfCheetah
5
2.25 + 0.00
4.70 +
0.13
2.25
0.00
21.19 = 7.53
24.70 +
4.95
Aug.
8.70 2.83
2.25 + 0.00
3.38
0.14
2.25
0.00
15.41
5.94
19.78
6.58
5 / 0
1.42 1 2.28
105.13
3.35
0.65 1 0.06
106.50 1 4.09
108.73 73 1
0.28
5
0.31 1 0.06
107.99
3.52
0.58 1 0.07
108.09 1 0.37
108.52 52
0.21
Walker2D
0.34 +
0.16
106.32
2.44
10.02
21.00
108.04
0.43
108.79
0.09
Ang.
103.12 11.48
0.69 + 1.32
106.48 + 3.14
3.75 + 12.13
107.54 + 2.39
108.68 + 0.21
/ 0
31.48 + 0.07
49.85
6.12
38.31
7.36
61.33 + 11.26
65.73
25.19
5 / 3
31.49 + 0.04
51.64 + 6.86
38.82 = 21.29
73.03 + 6,33
77.52 + 4.98
Ant
5
31.46 +: 0.07
46.97 +: 11.43
37.30 1: 13.98
72.92 1 22.64
76.68 +: 9.44
Aug.
58.78 1 2.46
31.48
49.49
8.47
38. 15.31
73.31
15.79
50
10.12 + 16.08
58.68
14.95
35.79 + 18.71
88.52 + 16.14
102.55 + 8.92
50 /
17.32 + 17.43
68.03
7.26
48.47 17.88
101.45 + 7.85
97.39
6.91
Pen
50 / 30
12.32 + 16.40
77.46
30.08
33.95
12.94
100.00 + 16.66
96.41
8.64
Ang.
73.92 + 10.76
13.25 + 16.65
68.06
19.84
39.40 + 16.70
96.66 + 14.14
98.78
8.21
50/ / 0
0.11 T. 0.05
0.00
0.00
0.02 1 0.01
6.03 1 8.21
9.79
14.66
50 / 30
0.12 1. 0.07
0.03 1 0.03
0.01 1 0.01
9.01 1 8.85
7.29 1 9.47
Door
50/ / 50
-0.08 + 0.05
0.24 + 0.48
0.01 = 0.01
14.89 + 18.51
17.70 + 16.42
Aeg.
5.50 + 12.37
0.10 + 0.06
0.09 + 0.28
0.01 + 0.01
9.98 + 12.76
11.50 + 13.84
50
0.27 + 0.01
7.55
9.78
0.30
0.01
81.82 + 18.14
118.33
18.71
50 / 30
0.25 + 0.01
8.28
9.09
0.29
0.01
102.75 = 29 23.83
124.71
4.31
Hammer
50 / 50
0.26 1 0.01
5.52
4.80
0.29 1
0.01
110.45 1 20.49
120.91
5.47
Aug.
73.26 + 14.89
0.26 + 0.01
7.12
8.19
0.29
0.01
98.34
23.37
121.32 11.53
50/ /
0.04 1: 0.04
2.25 1: 0.77
8.44 1 9.72
35.12 1 13.59
59.74 1: 19.47
50 / 30
0.06 + 0.07
3.24 + 0.96
11.11 + 9.47
51.68 + 11.05
62.45 + 18.54
Relocate
50/ / 30
0.04 + 0.05
2.12
1.81
13.67
9.66
62.94 + 17.92
71.89 10.50
Ang.
32.97 + 19.28
0.05 t 0.05
2.54 + 1.26
11.07 + 9.62
49.91 + 14.47
64.69 + 16.67
useful information from poor quality unlabcled demos. In contrast, ROIDA can utilize the auxiliary data through TD learning, despite poor quality. Both DemoDICE and ORIL perform worse in all settings.
5.3.2 Varying the size of the expert dataset
In Table 2, we examine how varying the size of the expert dataset affects policy performance. For locomotion lasks, we fix the auxiliary dataset and set the number of expert trajectories in Do to 5, while varying the number in Dg between 3, 5, and 7. For the Adroit tasks, we hold the number of expert trajectories in Do at and vary DE from 50 70. ROIDA achieves comparable performance DWBC, slightly outperforming in out of environments. As the amount of expert data in Dg increases, all the methods improve, expected. In summary, ROIDA maintains competitive performance across varying expert dataset sizes.

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Imitation learning performance on locomotion tasks as the number of expert demonstrations in De is increased. The best performing method each task is highlighted in red and the second best in bluc.
Method
Environment
Setting
BC-exp
BC-all
DemoDICE
ORIL
DWBC
ROIDA
3 5
74.84 1 18, 88
2.44 1 1.36
75,75 28.31
17.11 1 25.33
87.00 1 14.70
81.53 1 10.77
5 5
67.15 = 16.03
2.67 = 1.05
68.82 + 15.36
17.83 + 21.72
80.85 + 23.56
88.45 + 8.46
Hopper
7
78.83 = 16.52
2.12 = 0.34
72.57 + 10.67
23.10 29.66
87.74 + 3.93
90.88 + 11.32
Ang.
73.61 + 17.19
2.41 + 1.01
72.38
19.59
19.35
25.77
85.20 + 16.19
86.95 + 10.26
/ 5
5.40
3.16
2.25 = 0.00
2.25
0.00
2.25
0.00
7.87 = 1.12
8.59 + 0.62
5 5
14.50 = 9.88
2.25 = 0.00
4.70
0.13
2.25
0.00
21.19 + 7.53
24.70 = 4.95
HalfCheetah
7 5
25.75 1 9.37
2.25
0.00
7.84
4.28
2.25 1: 0.00
31.79 1: 5.76
39.19 1 2.10
Avg.
15.21 1 8.07
2.25
0.00
4.93
2.47
2.25
0.00
20.28 1 5.51
24.16 1 3.12
3 / 5
94.23 + 15.91
0.21 + 0.05
106.84 + 2.31
6.89 + 25.02
106.49 + 3.65
106.95 + 3.82
5 5
103. + 11.48
0.34 + 0.16
106.32 + 2.44
10.02 + 21.00
108.04 + 0.43
108.79 + 0.09
Walker2D)
7/
5
108.14 0.55
0.31 = 0.06
107.64
6.51
21.59 17.22
106.76 + 3.65
107.71 + 0.17
Aug.
101.83 = 11.33
0.29 = 0.10
106.93 + 3.72
12.83 + 21.32
107.10 + 2.99
107.8 + 2.21
/ 5
43.83 26.82
31.51
0.04
42.09
15.02
26.88 13.50
57.52 12.40
61.43
5 / 5
58.78 1 2.46
31.46 1 0.07
46.97 11.43
37.30 1 13.98
76.27 1 26.05
76.68 1 9.44
Aut
7 5
80.19 = 6.22
31.38 = 0.16
69.10 + 27.95
52.59 + 32.42
89.14 + 10.18
95.19 + 6.00
Avg.
60.93
15.96
31.45 = 0.10
52.72 19.47
38.93 21.82
74.31 + 17.66
77.77
8.37
50/ / 50
73.92 = 10.76
12.32 = 16. 40
77.46 + 30,08
33.95 + 12.94
100.00 = 16.66
96.41 + 8.64
70 / 50
84.62
22.74
25.29 20.72
68.65 16.88
55.96 18.43
105.25 17.86
100.19 + 11.55
Pen
Ang.
79.27 = 17.79
18.80 = 18.69
73.06 + 24.39
44.96 + 15.92
102.62 + 17.27
98.30 + 10.20
50/ / 50
5.59
12.37
0.08 + 0.05
0.24
0.48
0.01
14.89 + 18.51
17.70
16.42
70/ / 50
8.23 + 13.56
0.12 = 0.03
0.03 + 0.03
0.02 1 0.06
8.27 + 8.56
8.13 + 7.71
Door
Avg.
6.91
12.98
0.10
0.04
0.13
0.34
0.02
0.04
11.58 + 14.42
12.92 + 12.83
50 / 50
73.26 1 14.89
0.26 1 0.01
5.52 1: 4.80
0.29 1 0.01
110.45 1: 20.49
120.91 1 5.47
70 / 50
96.44 = 13.43
0.28 = 0.01
9.43 + 16.42
1.19
2.92
115.41 = 9.47
119.53 +
5.58
Hammer
Avg.
84.85 1 14.18
0.27 T 0.03
7.47 T. 12.10
0.74 1 2.06
112.93 1 15.96
120.22 1 5.52
50 / 50
32.97
19.28
-0.04
0.03
2.12 + 1.81
13.67
9.66
62.94 + 17.92
71.89
10.50
70 50
60.78
15.07
0.00 = 0.05
4,02
2.87
18.39
7.81
73.65 + 6.58
74.29 - 3.33
Relocate
Aug.
46.88 = 17.30
0.02 = 0.05
3.07 + 2.40
16.03 + 8.78
68.30 = 13.50
73.09 + 7.79
5.4 Ablation study
In this section, we conduct experiments to analyze each component ROIDA, and also benchmark the performance as the scale of the auxiliary dataset variod.
First, systematically remove each part of ROIDA and evaluate the resulting performance impact. The ablation results in Table 3 present the performance across three auxiliary dataset settings: 5/0, 5/3, 5/5 (denoting different auxiliary data qualities).
We begin by examining the impact of ablating the reward weighted BC component by setting a = 0. Performance drops indicate the importance filtering the data using a learned reward model. The precise reward assignment enables ROIDA to discern the varying quality of transitions and utilize them accordingly.
Next, we study the impact of the reward function formulation by directly using the discriminator output. the reward. The results demonstrate that our reward formulation based on the DICE methods improves policy performance compared to directly using the discriminator rewards.
We also examine the choice of PU learning (Eqn. 1) over binary classification for reward learning. PU lcarning accounts for the possibility of optimal transitions within the auxiliary dataset, whereas binary classification risks misclassifying all auxiliary samples as suboptimal, which could degrade performance.
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 3: Ablation study on each module's contribution to final policy performance on locomotion tasks.
Method
Enviromment
Setting
ROIDA
w/o reward-weighted BC
w/o modified reward
w/o PU learning
5
0
84.63 + 16.01
41.68 + 39.67
78.07 = 11.00
84.55=9.01
Hopper
5 3
86,66 = 21.94
78,06 + 16. 16.17
92.06 7.10
82.78=11.49
/ 5
88.45 = 8.46
81.51 15.70
84.73 11.26
82. 82.23:111.25
Aug.
86.58 = 16.42
67.08 + 26.34
84.96 = 9.97
83.19:110.44
/
15.89 9.60
1.48
13.93 3.31
15.74 18.63
HalfCheetah
/ 3
18.73 = 3.67
14.93
8.61
16.35 = 4.44
14.53 #3.02
5 / 5
24.70 + 4.95
18.22 + 4.61
22.53
9.91
16.68 7726
Ang.
19.78 + 6.58
14.19
5.70
17.61
6.55
15.65 +6.74
/ 0
108.73 = 0.28
87.72 + 28.91
104.65
7.79
108.08:00
Walker2D
5 / 3
108.52 1: 0.21
108.11
0.66
104.77
6.83
104.78113.57
/
108.794 0.090
107.97
0.35
4.81
105.05 10-.05:1-4.79
Aeg.
108.68 1 0.21
101.27 1. 16.70
104.99 6.60
105.971.3 9.113.46
/
0
65.73 + 25.19
57.42 20.89
49.37 = 11.64
70.05 18.71
Ant
5 / 3
77.52 + 4.98
64.40 + 11.32
62.94 = 10.27
69.77 10.31
5
76,68 = 9.44
58.95 + 17.68
76.14 18.52
10..0=111000
Aug.
73.31 15.79
60.25 17.14
62.81 13.95
69.63:13..8
Table 4: Performance on the Hopper task as the reward threshold varied.
Reward threshold
Setting
T 0
== 1 (Ours)
== 2
5 / 0 /
83.20 1 24.58
84.63 16.01
70.24 + 27.75
86.82 1 6.85
86.66 21.94
81.91 + 18.33
/ 5
88.23 + 11.42
88.45 1 8.46
90.70 + 6.88
When the auxiliary dataset contains only suboptimal transitions (5/0), both meth- ods perform similarly. However, as the diversity of the auxiliary dataset increases (5/3 and 5/5), the performance of binary classification decreases. This drop occurs because binary classification incorrectly labels some optimal transitions subop- timal, negatively impacting overall perfor- mance.
5/0
Scale of suxiliary data (In %6)
50
Second, we study the impact of the re- ward threshold т. Our specific reward formulation leads to rewards the range [-2.19, 2.19). Based on this, we decide set the threshold between utilizing the "good" transitions in the auxiliary data and neglecting the poor quality transitions. To evaluate the impact this threshold, we perform a study where we vary the threshold's value using the Hopper environment and report the results in Table 4.
90
52
Normalized return
to strike balance
Figure 3: Scalability to the size of the auxiliary dataset. We visualize the performance of ROIDA on the Hopper environment the number of random transitions is varied. Here, we show 3 scenarios corresponding to different proportions of the DARL random set. This highlights ROIDA's 's ability to learn policies even when the expert and noisy data ratio is quite imbalanced.
11

---

Published in Transactions on Machine Learning Research (04/2025)
We observe that setting 0 does not impact the results too much since it is situated at the midpoint of the reward range. When we increase the threshold to = 2 we see a drop in performance in the 5/0 setting; in this setting, no experts are present and thus, the majority of the performance increase should come from the "good" transitions which are rejected due to the high value of the threshold. The performance increase between 5/0 and 5/3 also sheds light on the threshold parameter. When a higher threshold is sel, the increase between the settings is quite high, indicating that the performance improvements are coming mostly from the expert data in the auxiliary set and not from the "good" transitions.
Third, we change the scale of the auxiliary dataset and analyze the impact the results. We achieve this by including 25%, 50% or 100% of the entire random set from D4RL (instead of the fixed 1000 trajectories in the previous experiments). As shown in Fig. 3, even with an increased number of suboptimal demonstrations, performance improves as more expert trajectories are added. This clearly highlights ROIDA's ability to distinguish expert data even when the ratio of expert to noisy data is highly skewed. We also observe that adding more auxiliary data while keeping the number of expert trajectorics fixed slightly improves performance, indicating ROIDA's i ability to extract information from suboptimal data.
6 Conclusion
We propose ROIDA, a simple yet effective framework for oflline imitation that can maximize utilization of an unlabeled auxiliary dataset of unknown quality alongside a small set of expert demonstrations. Unlike previous methods that make assumptions about auxiliary datasct quality, ROIDA can seamlessly leverage uncurated, unlabeled offinn datasets without relying on any quality assumptions. We demonstrate ROIDA's efficacy on multiple manipulation and locomotion tasks, encompassing wide variety of auxiliary dataset. quality settings. The consistent performance gains over baselines validate ROIDA's ability to unlock the full potential of heterogeneous offline datasets without relying on quality assumptions.
Limitations
While ROIDA demonstrates strong performance across various environments, we believe there is still room for improvement in the reward estimation process. To investigate this, we conduct an experiment shown in Table 5, where we substitute the estimated reward with the ground-truth reward from the D4RL benchmark. The results indicate a performance gap between the estimated and ground-truth rewards. This finding suggests that our method could potentially achieve higher performance if the reward estimation process is further refined and improved.
Table Performance on the Hopper task with ground-trut truth rewards.
Method
Setting
ROIDA
ROIDA w/ GT rewards
5 / 0
84.63 + 16.01
94.63 = 20.76
/ 3
86.66 +:22 21.94
98.53 = 11.70
/ 5 88.45 + 8.46
104.46 = 5.42
Aug. 86.58 1 16.42
99.21 14.11
In our particular framework, the reward estimation can be improved by an accurate choice of the hyperpa- rameter 7 by performing mixture proportion estimation (Zhu et al., 2023; Ramaswamy et al., 2016). However, this is beyond the scope of our work. In order to avoid any assumption about the auxiliary data in our work, we have chosen 17 0.5 which is an unbiased estimate. We also provide additional results with 17 0.3 and y m 0.7 in Table 6. Here, we obtain better results when is closer (1 0.3 is closer than r y= 0.7) to the true ratio between expert and suboptimal demonstration in the auxiliary dataset (0.01 for setting 5/0, 0.12 for setting 5/3 and 0.19 for setting 5/5). Since this true ratio is unknown, estimating it would be a problem in its own right, which could then be combined with our method.
Additionally, the current implementation of ROIDA is designed for a single-task setting. An interesting avenue for future work is to extend our framework to multi-task or goal-conditioned settings, where the model
12

---

Published in Transactions on Machine Learning Research (04/2025)
Table 6: Performance on the Hopper task with varying 7..
Method
Setting
1) 0.3
0.5
1) 0.7
5
0
88.12 + 14.93
84.63 = 16.01
82.40 = 20.76
5
90.42 + 18.84
86.66 21.94
84.85 = 7.13
5
91.02 7.97
88.45 = 8.46
86.45 18.61
Aug. 89.85 + 14.62
86.58 16.42
84.57 = 14.39
is trained on datasets from different tasks or goals. such a scenario, ROIDA could be used to learn a new, unscon, but related task based on these multi-task datasots. This extension would open up possibilitics for more generalizable and flexible reinforcement learning applications across various tasks.
Broader impact statement
Training robots for various tasks using human demonstrations is well-establisher However, obtaining high-quality demonstrations in large numbers extremely challenging and impractical in many cases. Our work provides method where robot can learn from mixture of limited number of expert high-quality demonstrations and large number of lower-quality demonstrations. This is more practically feasible setting and offers promise for developing more efficient approaches to train robots. One possible risk is that the robot can learn unsafe behaviors since the training set may have large numbers non-expert demonstrations. However, since this is an offine training procedure, the risk is very minimal and can be mitigated through evaluations in lab settings.
Acknowledgment
This work was partially supported by The National Science Foundation Award No. 2326309, National Institute for Food and Agriculture Award No. 2021-67022-33453 and the UC Multi-campus Rescarch Programs Initiative.
References
Picter Abbecl and Andrew Y Ng. Apprenticeship learming via inverse reinforcement learning. In Proceedings of the twenty-fistt first international conference on Machine learning, pp. 2004.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathow Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprinl ar Xiv: 060460736, 2016.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783 792. PMLR, 2019.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement lcarning for continuous control. In International conference on machine learning, pp. 1329-1338. PMLR, 2016.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. pp. 213-220, 08 2008. doi: 041145/14018901140999
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. Xi preprint arXiv:1700. 11248, 2017.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. Xrviv preprint or Xim:2004.07217 2020.
Scott Fujimoto and Shixiang (Shane) Gu. A minimalist approach offine reinforcement learn-
ing. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan
(eds.), Advances in Newral Information Processing Systems, volume 34, pp. 20132 20145. Cur- ran Associates, Inc., 2021. URL https: /procee////.c//nggg... wtt.:ss.eerrrr..ccccoooo 88166ia0605004447770744444488888 Paapaaepeeeeeeepeppp.......
ftless/pappr//22///ii///
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Proc. of NeuIPS, 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advonces in neural information processing systems, 29, 2016.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-lovel drone racing using deep reinforcement learning. Nature, 620(7976):982 2023.
Gcon-Hyeong Kim, Scokin Sco, Jongmin Lec, Wonscok Jeon, HycongJoo Hwang, Hongscok Yang, and Kee-Eung Kim. DemoDIGE: Offline imitation learning with supplementary imperfect demonsttration.. International Conference on Learning Representations, 2022. URL //tperr//ppnrrevie http://ppprrreeeeeeeeeeerrrreee net/forum?id= BrPdX1bDZkQ.
Ryuichi Kiryo, Gang Niu, Marthinus Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in neural information processing systems, 30, 2017.
llya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. arXiv preprinl orxiv:11122 05032, 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-13 1373, 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXav:2005.0166 2020.
Yecheng Jason Ma, Audrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. 2022. URL https://arxive org/ ab2/ 02433.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540))522 533, 2015.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv preprint, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintagg-weighteed regression: Simple and scalable off- off-policy reinforeement learning. ArXiv preprint, 2019.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Proc. of NeulPS, 1989.
Harish G. Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via. kernel embeddings of distributions. Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Mochine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2052 2060. JMLR.org, 2016. URL http://jmlr...
http://cceennnspppprrseee/saeese / html.
Stéphanc Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thir- teenth international conference on artificial intelligence and statistics, pp. 661-668. JMLR Workshop and Conference Proceedings, 2010.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning- In Proceedings of the fourleenth international conference on arlificial intelligence and statistics, pp. 627 635. JMLR Workshop and Conference Proceedings, 2011.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Proc. of ICLR, 2021.
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognelive sciences, 3(6):233-244) 1999.
Jie-Jing Shao, Hao-Sen Shi, Tian Xu, Lan-Zhe Guo, Yang Yu, and Yu-Feng Li. Offline imitation learning without auxiliary high-quality behavior data. 2023.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-3555 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Gracpel, ct al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(64199:11400 1144, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Gokul Swamy, Sanjiban Choudhury, Zhiwei Steven Wu, and J Andrew Bagnell. Of moments and matching: Trade-offs and treatments in imitation learning. ArXiv preprint, 2021.
Voot. Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation learning with quverse-quality demonstrations. In International Conference on Machine Leurning, pp. 9407 9417. PMLR, 2020.
Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. International Conference on Machine Learning, pp. 10961-10970. PMLR, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768 7778, 2020.
Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning, pp. 6818 6827. PMLR, 2019.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weiigheed offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp. 24725 24742.. PMLR, 2022.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforeement learning. In International Conference on Machine Learning, pp. 25611 25635. PMLR, 2022.
Yilu Zhu, Aaron Fjeldsted, Darren Holland, George Landon, Azaree Lintereur, and Clayton Scott. Mixture proportion estimation beyond irreducibility. Proceedings of Machine Learning Research, 202:42962 42982, 2023. ISSN 2640-3498. Publisher Copyright: c 2023 Proceedings Machine Learning Research. All rights reserved.; 40th International Conference on Machine Learning, 1CML 2023 Conference date: 23-07-2023 Through 29-07-2023.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcchre, Ziyun Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott E. Reed. Offline learning from demonstrations and unlabeled experience. Ar Xiv, abs/2011 13885, 2020. URL https://api. semanticscholaar I22//299.
15

---

Published in Transactions on Machine Learning Research (04/2025)
A Appendix
A.1 Additional comparisons
We present additional comparisons in Table 7 with another recent unpublished algorithm, BCDP (Shao et al., 2023), which tailors offline IL algorithm specifically for scenarios where the auxiliary offline dataset contains no expert data. While this design can enhance performance when the auxiliary dataset lacks expert demonstrations, ions, it can lead to suboptimal performance as the proportion of expert data in the auxiliary dataset. increases. The results indicate the ROIDA significantly outperforms BCDP on all the seven environments.
Table 7: Imitation learning performance on locomotion (first 3 columns) and manipulation (next 4 columns) lasks from the D4RL benchmark. For each locomotion environment. the performance is averaged across scenarios containing different number of expert trajectories in Do (5/0, 5/3, 5/5). For each manipulation environment the performance is avcragod across scenarios containing different number of expert trajectories in Do (50/0, 50/30, 50/50).
Environment
Method
Hopper
Walker2D
Ant
Pen
Door
Hammer
Relorate
BCDP ROIDA
65.45 i 14.81
103.16 = 8.62
47.38 1: 17.55
11.95 1 14.90
3.23 1 9.52
0.27 0.02
0.44 1.14
86.58 1 16.42
108.68 0.21
73.31 1: 15.79
98.78 1 8.21
11.59 1 13.84
121.32 1 11.53
64.69 16.67
A.2
Experimental details
A.2.1
Implementation
We largely follow the architecture and hyperparameters from DWBC (Xu et al., 2022) for fair comparison. The policy network is a 3-layer MLP with 256 hidden units and tanh outputs. The discriminator is a 4-layer MLP with 128 hidden units, with sigmoid outputs clipped to [0.1, 0.9). In the PU learning objective 1, we replace the non-differentiable maz with the softplus function to make the loss function differentiable. The Q-function network is an MLP of 3 layers with 256 units. All networks use ReLU activations and the Adam optimizer.
The discriminator learning rate is set to le and a cosine annealing scheduler is added. The policy and Q-function learning rate is set to 34-4, with a policy weight decay of 0.005. The balancing factors a and B are set dynamically based on the loss ratios. Considering the batch-wise BC loss on expert data to be Å the batch-wise weighted BC loss auxiliary data to be X2, and batch wise Q-function loss to be Λg, then A 7.5 and B 0.01 X * 7.5 The additional factor of 7.5 emphasizes the BC loss on expert data, and is adopted from previous work. The discount factor is 0.5. The frequency of actor model update, lfreg is set to 3 for all the environments.
= 0.01* A1
The DICE reward function bounds F(s,a) between [-2.2, 2.2]. For filtering high quality data, we use 7 = 1.
All experiments are conducted using PyTorch on single RTX 3090 GPU.
A.2.2 Dataset
All datascts are from D4RL (Fu ct al., 2020), an offline IL benchmark. Expert trajectorics for locomotion tasks are from <environment>-eng -eppert--2 Expert trajectories for manipulation tasks are from < <lask>- expert-el. Sub-optimal transitions are from (manipulation). Table 8 details the number of trajectories and transitions in each datasct. The column Transitions refers to the full D4RL datasets for the corresponding random transitions with total number of trajectories given in brackets.
(locomotion) and <task> -task--cloned-ul
16

---

Published in Transactions on Machine Learning Research (04/2025)
Table 8: Details on total number of trajectories and transitions in the expert and auxiliary datasets for each of the settings in locomotion and manipulation tasks
Expert data DE
Auxiliary Data Do
Environment
Setting
Trajectories
Transitions
Trajectories
Transitions
Transitions?
/ 0
5000
1000
21723
999996 (45239)
/
3
5000
1003
24723
5 /
5
5000
1005
26723
Hopper
/ 5
3000
1005
26723
7/ 5
7000
1005
26723
/
5000
999
999999
999999 (999)
5/ 3
5000
1002
1002999
Halfcheetah
/
1004
1004999
5
3000
1004
1004999
7/3
7000
1004
1004999
/ 5 / 3
5000
1000
19877
999997 (48907)
5000
1003
22877
Walker2d
/
5000
1003
24877
/ 5
3000
1005
24877
7/ 5
7000
1005
24877
/ 0
4465
1000
180912
999930 (5821)
/ 3
4465
1003
183912
Aut
/ 5
4465
1005
185912
/ 5
3000
1005
185377
5
6165
1005
1 185912
50
0
50
5000
1000
99881
499886 (3751)
50 / 30
5000
1030
102862
Pen
50 / 50
50
5000
1050
104862
/ 50
7000
1050
104862
50 0
10000
1000
200000
999939 (4337)
50 / 30
50
10000
1030
206000
Door
50 / 50
10000
1050
210000
70 / 50
70
14000
1050
210000
50 0
50
10000
1000
200000
999872 (3605)
50 / 30
50
10000
1030
206000
Hammer
50 / 50
50
10000
1050
210000
70 / 50
70
14000
1050
210000
50 0
50
10000
1000
200000
999724 (3747)
50/ / 30
10000
1030
206000
Relocate
50 / 50
50
10000
1050
210000
70 / 50
14000
1050
210000
A.3 RLiable evaluation
In this section, we present evaluations using the rliable (Agarwal al., 2021) framework for our algo- rithun ROIDA and DWBC, which the closest contender. The rhiable framework aims reliably evaluate performance with limited number of runs by employing rigorous evaluation methodology that accounts for uncertainty in results. It presents more robust and efficient aggregate metries, such as interquartile mean (IQM) scores, to achieve small uncertainties in the evaluation outcomes.
We group the environments into two benchmarks, locomotion and adroit. Locomotion contains the Hopper, Walker2D and Ant environments and consists of 9 tasks environments [5/0, 5/3, 5/5)). Adroit contains the Pen, Door, Hammer and Relocate environments and consists of 12 tasks (4 environments [50/0, 50/30, 50/50)). We divide the scores by 100 to obtain values in the [0, range, as used in the rliable paper. Using
17

---

Published in Transactions on Machine Learning Research (04/2025)
Median
IQM
Mean
Optimality Gap
DWBC ROIDA
0.72
0.80
0.88
0.84
0.88
0.92
0.960.80
0.84
0.88
0.92
0.12
0.16
0.20
Human Noraaizedd Score
Figure 4: rliable evaluation on the Locomotion benchmark.
Median
IQM
Mean
Optimality Gap
DWBC
ROIDA
0.64
0.72
0.80
0.88 0.64
0.72
0.60 0.65 0.70 0.75
0.32
0.36
0.40
Human Nomalizedd Score
Figure 5: vliable evaluation on the Adroit benchmark.
this setup, we report the following performance metrics with 95% confidence intervals: 1) Median performance (higher better), 2) Mean performance (higher better), 3) IQM (higher better), and 4) Optimality gap (lower better).
A.4 Analysis on hyperparameters
In the main paper, we study the impact of 7) and т. In this section, we analyze the effect of varying the weighting factors a and 3, which control the contributions of the weighted BC loss and the Q-learning loss, respectively. The results are presented Figure 6. We vary both hyperparameters across a range from 0.001 1.0. Based on the ablation study for the Hopper-5/5 scenario, we select 0.01 for both and / across all sctups. These values are then dynamically scaled according to the loss balancing scheme outlined in Sention A.2.
120
80
82
Mean
Meatn
(a) Varying a (3 0.01)
(b) Varying 3 (a 0.01)
Figure 6: Performance on Hopper-5/5 and B are varied.
A.5 Reward visualization
To understand the accuracy the proposed reward learning and filtering strategy, we visualize the overlap between the distribution of high-reward transitions as estimated by ROTDA and the overall set of transitions for the Hopper environment. As shown in Figure 7, ROIDA correctly identifies the ground-truth high-reward transitions as expert transitions.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Distribution of Grourd Truth Rewards Class
Preckite Export Cluss Pres SuB Class
Ground Truch Rewards
Figure 7: Reward estimation accuracy on Hopper 5/5. By combining PU Bearning-based reward
estimation with the filtering strategy, ROIDA effectively identifies the high-reward transitions.
19


Reviews:
[{"review_id": "RYhoa2x0hg", "reviewer": "Reviewer_KxMB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. **Novel Framework:** ROIDA introduces a robust approach to offline imitation learning by leveraging both high- and low-quality auxiliary data without requiring prior knowledge about data quality. This is a significant improvement over existing methods.\n2. **Empirical Performance:** The method consistently outperforms baselines on diverse environments (locomotion and manipulation tasks from D4RL), demonstrating its robustness and general applicability.\n## Weaknesses\n**Limited Ablation studies:** The ablation studies in the paper claim to validate the contribution of each component of ROIDA. However, some key components in the method still lack clear explanations. For exmaple, does TD learning with sub-optimal state-action pairs really improve the performance? Furthermore, I would be more interesting in the training objective of the reward model, like whether the improvement from the binary classifier objective to the PU learning is significant. The statement \"We also study the impact of the reward function formulation by directly using the discriminator output as the reward\" is somewhat unclear to me. If the third row in Table 3 addresses this point, please feel free to clarify or correct me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces ROIDA, a novel offline imitation learning framework designed to handle auxiliary datasets of mixed-quality demonstrations without making assumptions about their composition. Unlike existing methods which know the quality of the auxiliary demonstrations as a priori, ROIDA identifies expert-like transitions from auxiliary datasets using a discriminator trained with Positive-Unlabeled (PU) learning and performs weighted behavioral cloning. For lower-quality transitions, temporal difference learning is used to guide the policy toward high-reward states.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could you please add some experiments to verify the function of TD learning with sub-optimal state-action pairs? It is unclear that the performance benefits from this additional term if these state-action pairs deviate from the expert data.\n2. If the paper lacks a comparison of different reward model training objectives, such as the binary classifier objective versus PU learning, it would be valuable to include this.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hs7ut6dFTt", "reviewer": "Reviewer_vPy3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths: \nWell-presented and clear, strong empirical results.\n\n## Weaknesses: \nIncremental, lack of theoretical analysis for the proposed method, reproducibility (no code available).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work introduces an approach to effectively learn policies from a substantial amount of pre-collected demonstrations not necessarily optimal. The algorithm presents to main stages: (1) a reward model training via Positive-Unlabeled learning and (2) a policy training step which combines together BC, W-BC and TD3. This approach outperforms the presented baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In the problem setting it is claimed that $N_0 >> N_E$, this assumption does not seem to hold true in the proposed experiments or at least I find the notation confusing: what do 0, 3, and 5 stem for when referred to datasets. \n\n2. There is a substantial number of hyperparameters that require at least an ablation study ($\\tau$, $\\eta$, $\\alpha$, $\\beta$). I see most of this analysis done in the appendix. I would consider adding it to the main text. \n\n3. Limitations not discussed in the main text. Is the method robust to mismatches between the learning agent and the behavioral agent (the agent from which the data are collected)? \n\n4. Release the code.\n\n5. I find the setting quite interesting since it is an attempt to close the gap between what is assumed in OfflineRL and IL and what happens when dealing with real-world scenarios. I would find interesting to further test this approach in a multi-task or goal-conditioned setting where datasets are collected for different tasks/goals and the approach is used to learn an unseen, but related, task.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ddXH5ku4rS", "reviewer": "Reviewer_PAuM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper tackles a practical challenge of limited expert data in offline imitation learning.  The proposed method leverages readily available, albeit often noisy, auxiliary data, which is a realistic scenario in many robotics applications.\n\n- ROIDA presents a novel combination of PU learning and temporal difference learning within the offline IL framework. This approach allows the algorithm to effectively utilize both high-quality and low-quality transitions in the auxiliary dataset.\n\n- A core strength of ROIDA is its robustness to the quality of the auxiliary data. The algorithm performs consistently well across a spectrum of auxiliary dataset compositions, unlike existing methods that are sensitive to the proportion of expert demonstrations. This robustness is crucial for real-world applications where the quality of auxiliary data may be unpredictable.\n\n\n## Weaknesses\n\n- This paper claims that ROIDA does not impose any assumptions on the composition of the auxiliary dataset. However, it appears to still implicitly rely on certain assumptions about the auxiliary dataset. As described in Section 4.4, low-quality data is utilized to learn a Q function based on the learned reward, which subsequently directs the policy to maximize the long-term return. However, this process implicitly assumes that the low-quality data has relatively good state coverage. If the state coverage is too narrow, the resulting Q function is likely to be highly unreliable, thereby failing to effectively direct the policy.\n\n\n- As noted by the authors in the limitations section, the reward estimation process based on PU learning has room for improvement. The performance gap observed when using ground truth rewards indicates that further refinement of the reward estimation could enhance overall performance. Additionally, it would be great to explore methods for illustrating the accuracy of the learned reward. For instance, comparing high-reward transitions with the overall set of transitions and analyzing the overlap between the two distributions could provide meaningful insights.\n\n- ROIDA introduces four additional hyperparameters: $\\eta$ in PU, $\\alpha$ and $\\tau$ in reward-weighted BC, and $\\beta$ in TD learning. While the paper provides the chosen values, a more comprehensive analysis of the impact of these hyperparameters and practical guidelines for tuning them would be valuable. Although these hyperparameters may enhance performance, they also introduce an additional layer of complexity.\n\n- This paper primarily compares against a few imitation learning baselines. However, offline RL with sparse rewards could also serve as a potential baseline. Specifically, sparse rewards of 1 could be assigned to expert trajectories and 0 to the auxiliary datasets, followed by running offline RL on this setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces ROIDA (Robust Offline Imitation from Diverse Auxiliary Data), a novel offline imitation learning algorithm designed to learn effectively from a combination of limited expert demonstrations and a larger auxiliary dataset of potentially varying quality. Existing methods often struggle with performance fluctuations depending on the quality of the auxiliary data, as they rely on assumptions about the presence of high-reward transitions.  ROIDA addresses this limitation by employing a two-pronged approach: it first identifies high-quality transitions from the entire auxiliary dataset using a learned reward function. These high-reward samples are combined with expert demonstrations for weighted behavioral cloning. For lower-quality samples, it applies temporal difference learning to steer the policy toward high-reward states, improving long-term returns.\n\nExperiments on the D4RL benchmark demonstrate that ROIDA consistently outperforms existing offline imitation learning methods across various environments and auxiliary dataset compositions.  Its robustness to varying data quality stems from its ability to extract information from both high-quality and low-quality transitions, making it a promising approach for practical scenarios where the quality of auxiliary data is unknown or unreliable.  Ablation studies further confirm the contribution of each component of ROIDA to its overall performance.\n\nThe key contributions of this paper are:\n- Point out the unrealistic assumptions of previous offline IL methods, particularly regarding the composition of the auxiliary set. \n- Design an offline IL algorithm, ROIDA, that addresses the limitations of these different methods and remains robust to the quality of demonstrations in the auxiliary dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.\n\nAdditional questions:\n\n- In Figure 1, the caption states \"given expert set is kept fixed,\" yet the y-axis indicates varying numbers of expert trajectories. Could you clarify this inconsistency?  \n- Section 4.3 mentions, \"Instead of directly employing the rewards as weights, we employ a direct thresholding scheme to exclude highly sub-optimal state-action tuples.\" What advantages does this approach have over directly using rewards as weights?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper meets both acceptance criteria and all reviewers recommended acceptance. The paper was already revised during the review process.
Decision: Accept (poster)