Title: SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: imitation learning

offline imitation learning

reference reward

supplementary data

ranked dataset
Abstract: We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. 

In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL .

SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

Abstract

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency.

In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL.

1 Introduction

Reinforcement learning (RL) has established itself as a strong and reliable framework for sequential decision-making with applications in diverse domains: robotics [19, 18], healthcare [34, 28, 27], and environment generation [9, 24]. Unfortunately, RL requires an underlying simulator that can provide rewards for different experiences, which is usually not available.

Imitation Learning (IL) [16, 29, 13, 17] handles the lack of reward function by utilizing expert demonstrations to guide the learning scheme to compute a good policy. However, IL approaches still require the presence of a simulator that allows for online interactions. Initial works in Offline IL [35, 23, 40, 2] tackle the absence of simulator by considering an offline dataset of expert

demonstrations. These approaches extend upon Behavioral Cloning (BC), where we aim to maximize the likelihood of the expert's decisions from the provided dataset. The key advantage with BC is the theoretical justification on converging to expert behaviors given sufficient trajectories. However, when there are not enough expert trajectories, it often suffers from distributional shift issues [30]. Thus, a key drawback of these initial IL approaches is the need for a large number of expert demonstration datasets.

To deal with limited expert demonstrations, recent works utilize non-expert demonstration datasets to reduce the reliance on only expert demonstrations. These additional non-expert demonstrations are referred to as supplementary data. Directly applying BC to these larger supplementary datasets will lead to sub-optimal policies, so most prior work in utilizing supplementary data attempts to extract expert-like demonstrations from the supplementary dataset in order to expand the expert demonstrations [31, 21, 20, 37, 39]. These works assume expert-like demonstrations are present in the supplementary dataset and focus on identifying and utilizing those, while eliminating the non-expert demonstrations. Eliminating non-expert trajectories can result in loss of key information (e.g., transition dynamics) about the environment. Additionally, these works primarily rely on BC, which is known to overlook the sequential nature of decision-making problems - a small error can quickly accumulate when the learned policy deviates from the states experienced by the expert.

Wedevelop our algorithm based on an inverse Q-learning framework that better captures the sequential nature of the decision-making [13] and can operate under the more realistic assumption that the data is collected from people/policies with lower expertise levels 1 (not experts). To illustrate, consider a scenario in robotic manipulation where the goal is to teach a robot to assemble parts. Expert demonstrations might show precise and efficient methods to assemble parts, but are limited in number due to the high cost and time associated with expert involvement. On the other hand, sub-optimal demonstrations from novice users are easier to obtain and more abundant. Our SPRINQL approach effectively integrates these sub-optimal demonstrations, giving appropriate weight to the expert demonstrations to ensure the robot learns the optimal assembly method without overfitting to the limited expert data or the inaccuracies in the sub-optimal data. We utilize these non-expert trajectories to learn a Q function that contributes to our understanding of the environment and the ground truth reward function.

Contributions: Overall, we make the following key contributions in this paper:

(i) We propose SPRINQL, a novel algorithm based on Q-learning for offline imitation learning with expert and multiple levels of sub-optimal demonstrations .

(ii) We provide key theoretical properties of the SPRINQL objective function, which enable the development of a scalable and efficient approach. In particular, we leverage distribution matching and reward regularization to develop an objective function for SPRINQL that not only help address the issue of limited expert samples but also utilizes non-expert data to enhance learning. Our objective function is not only convex within the space of Q functions but also guarantees the return of a Q function that lower-bounds its true value.

(iii) We provide an extensive empirical evaluation of our approach in comparison to existing best algorithms for offline IL with sup-optimal demonstrations. Our algorithms provide state-of-the-art (SOTA) performance on all the benchmark problems. Moreover, SPRINQL is able to recover a reward function that shows a high positive correlation with the ground-truth rewards, highlighting a unique advantage of our approach compared to other IL algorithms in this context.

1.1 Related Work

Imitation Learning. Imitation learning is recognized as a significant technique for learning from demonstrations. It begins with BC, which aims to maximize the likelihood of expert demonstrations. However, BC often under-performs in practice due to unforeseen scenarios [30]. To overcome this limitation, Generative Adversarial Imitation Learning (GAIL) [16] and Adversarial Inverse Reinforcement Learning (AIRL) [10] have been developed. These methods align the occupancy distributions of the policy and the expert within the Generative Adversarial Network (GAN) framework [14]. Alternatively, Soft Q Imitation Learning (SQIL) [29] bypasses the complexities of adversarial training by assigning a reward of +1 to expert demonstrations and 0 to the others, subsequently learning a value function based on these rewards.While the aforementioned imitation learning algorithms show

promise, they require interaction with the environment to obtain the policy distribution, which is often impractical.

Offline Imitation Learning. ValueDICE [22] introduces a novel approach for off-policy training, suitable for offline training, using Stationary Distribution Corrections [25, 26]. However, ValueDICE necessitates adversarial training between the policy network and the Q network, which can make the training slow and unstable. Recently, algorithms like PWIL [8] and IQ-learn [13] have optimized distribution distance, offering an alternative to adversarial training schemes. Since such approaches rely on occupancy distribution matching, a large expert dataset is often required to achieve the desired performance. Our approach, SPRINQL is able to bypass this requirement of a large set of expert demonstrations through the use of non-expert demonstrations (which are typically more available) in conjunction with a small set of expert demonstrations.

Imitation Learning with Imperfect Demonstrations. T-REX [5] and D-REX [6] have shown that utilizing noise-ranked demonstrations as a reference-based approach can return a better policy without requiring expert demonstrations in online settings. Moreover, there are also several works [36, 33] that utilize the GAN framework [14] for sub-optimal datasets and have achieved several successes. Meanwhile, in the offline imitation learning context, TRAIL [38] utilizes sup-optimal demonstrations to learn the environment's dynamics. It employs a feature encoder to map the high-dimensional state-action space into a lower dimension, thereby allowing for a scalable way of learning of dynamics. This approach may face challenges in complex environments where predicting dynamics accurately is difficult, as shown in our experimental results. Other works assume that they can extract expert-like state-action pairs from the sub-optimal demonstration set and use them for BC with importance sampling [31, 37, 39, 21, 20]. However, expert-like state-actions might be difficult to accurately identify, as true reward information is not available. In contrast, our approach is more general, as we do not assume that the sub-optimal set contains expert-like demonstrations. We also allow for the inclusion of demonstrations of various qualities. Moreover, while prior works only recover policies, our approach enables the recovery of both expert policies and rewards, justifying the use of our method for Inverse Reinforcement Learning (IRL)[1].

2 Background

Preliminaries. We consider a MDP defined by the following tuple M = ⟨ S, A, r, P, γ, s 0 ⟩ , where S denotes the set of states, s 0 represents the initial state set, A is the set of actions, r : S × A → R defines the reward function for each state-action pair, and P : S × A → S is the transition function, i.e., P ( s ′ | s, a ) is the probability of reaching state s ′ ∈ S when action a ∈ A is made at state s ∈ S , and γ is the discount factor. In reinforcement learning (RL), the aim is to find a policy that maximizes the expected long-term accumulated reward max π { E ( s,a ) ∼ ρ π [ r ( s, a )] } , where ρ π is the occupancy measure of policy π : ρ π ( s, a ) = (1 -γ ) π ( a | s ) ∑ ∞ t =1 γ t P ( s t = s | π ) .

MaxEnt IRL The objective in MaxEnt IRL is to recover a reward function r ( s, a ) from a set of expert demonstrations, D E . Let ρ E be the occupancy measure of the expert policy. The MaxEnt IRL framework [41] proposes to recover the expert reward function by solving

Intuitively, the aim is to find a reward function that achieves the highest difference between the expected value of the expert policy and the highest expected value among all other policies (computed through the min loop).

IQ-Learn Given a reward function r and a policy π , the soft Bellman equation is defined as B π r [ Q ]( s, a ) = r ( s, a ) + γ E s ′ [ V π ( s ′ )] , where V π ( s ) = E a ∼ π ( a | s ) [ Q ( s, a ) -log π ( a | s )] . The Bellman equation B π r [ Q ] = Q is contractive and always yields a unique Q solution [13]. In IQ-learn, they further define an inverse soft-Q Bellman operator T π [ Q ] = Q ( s, a ) -γ E s ′ [ V π ( s ′ )] . [13] show that for any reward function r ( a, s ) , there is a unique Q ∗ function such that B π r [ Q ∗ ] = Q ∗ , and for a Q ∗ function in the Q -space, there is a unique reward function r such that r = T π [ Q ∗ ] . This result suggests that one can safely transform the objective function of the MaxEnt IRL from r -space to the

Q-space as follows:

which has several advantages; [13] show that Φ( π, Q ) is convex in π and linear in Q , implying that (2) always yields a unique saddle point solution. In particular, (2) can be converted into a maximization over the Q-space, making the training problem no longer adversarial.

3 SPRINQL

We now describe our inverse soft-Q learning approach, referred to as SPRINQL ( S ub-o P timal demonstrations driven R eward regularized IN verse soft QL earning). We first describe the three key components in the SPRINQL formulation:

We formulate the objective function that enables matching the occupancy distribution of not just expert demonstrations, but also sub-optimal demonstrations.

To mitigate the effect of limited expert samples (and larger sets of sub-optimal samples) that can bias the distribution matching of the first step to sub-optimal demonstrations, we introduce a reward regularization term within the objective. This regularization term is to ensure reward function allocates higher values to state-action pairs that appear in higher expertise demonstrations.

We show that while this new objective does not have the same advantageous properties as the one in inverse Q-learning [13], with some minor (yet significant) changes it is possible to restore all the important properties.

3.1 Distribution Matching with Expert and Suboptimal Demonstrations

We consider a setting where there are demonstrations classified into several sets of different expertise levels D 1 , D 2 , ...., D N , where D 1 consists of expert demonstrations and all the other sets contains suboptimal ones. This setting is general than existing work in IL with sup-optimal demonstrations, which typically assumes that there are only two quality levels: expert and sub-optimal. Let D = ⋃ i ∈ [ N ] D i be the union of all the demonstration sets and ρ 1 , ..., ρ N be the occupancy measures of the respective expert policies. The ordering of expected values across different levels of expert policies would then be given by:

where r ∗ ( ., . ) are the ground-truth rewards. Typically, the number of demonstrations in first level, D 1 is significantly lower than those from other expert levels, i.e., |D 1 | ≪|D i | , for i = 2 , ..., N The MaxEnt IRL objective from Equation 1 can thus be adapted as follows:

where w i ≥ 0 is the weight associated with the expert level i ∈ [ N ] and we have w 1 > w 2 > ... > w N and ∑ i ∈ [ N ] w i = 1 . There are two key intuitions in the above optimization: (a) Expert level i accumulates higher expected values than expert levels greater than i ; and (b) Difference in values accumulated by expert policies and the maximum of all other policies is maximized. The optimization term can be rewritten as:

where ρ U = ∑ i ∈ [ N ] w i ρ i . Here we note that the expected reward ∑ i ∈ [ N ] w i E ρ i [ r ( s, a )] is empirically approximated by samples from the demonstration sets D 1 , D 2 , ...., D N . The number of demonstrations in the best demonstration set D 1 (i.e. the set of expert demonstrations) is significantly lower when compared to other demonstration sets. So, an empirical approximation of E ρ 1 [ r ( s, a )] using samples from D 1 would be inaccurate.

3.2 Regularization with Reference Reward

We create a reference reward based on the provided expert and sub-optimal demonstrations and utilize the reference function to compute a regularization term that is added to the objective of Equation 3.

Concretely, we define a reference reward function r ( s, a ) such that:

The aim here is to assign higher rewards to demonstrations from higher expertise levels, and zero rewards to those that do not belong to provided demonstrations. We will discuss how to concretely estimate such reference reward values later.

We utilize this reference reward as part of the reward regularization term, which is added into the MaxEntIRL objective in (3) as follows:

where α > 0 is a weight parameter for the reward regularizer term. With (4), the goal is to find a policy with an occupancy distribution that matches with the occupancy distribution of different expertise levels appropriately (characterized by the weights, w i ). Simultaneously, it ensures that the learning rewards are close to the pre-assigned rewards, aiming to guide the learning policy towards replicating expert demonstrations, while also learning from sub-optimal demonstrations.

3.3 Concave Lower-bound on Inverse Soft-Q with Reward Regularizer

Even though (4) can be directly solved to recover rewards, prior research suggests that transforming (4) into the Q-space will enhance efficiency. We delve into this transformation approach in this section. As discussed in Section 2, there is a one-to-one mapping between any reward function r and a function Q in the Q-space. Thus, the maximin problem in (4) can be equivalently transformed as:

where r ( s, a ) is replaced by T π [ Q ]( s, a ) and

In the context of single-expert-level, [13] demonstrated that the objective function in the Q -space as given in Equation 2 is concave in Q and convex in π , implying that the maximin problem always has a unique saddle point solution. Unfortunately, this property does not hold in our case.

Proposition 3.1. H ( Q,π ) (as defined in Equation 5) is concave in Q but is not convex in π .

In general, we can see that the first and second term of (6) are convex in Q , but the reward regularizer term, which can be written as

is not concave in π (details are shown in the Appendix). The property indicated in Proposition 3.1 implies that the maximin problem within the Q -space max Q min π J ( Q,π ) may not have a unique saddle point solution and would be more challenging to solve, compared to the original inverse IQ-learn problem.

Another key property of Equation 2 is with regards to the inner minimization problem over π , which yields a unique closed-form solution, enabling the transformation of the max-min problem into a non-adversarial concave maximization problem within the Q-space. The closed-form solution was given by π Q = argmax π V π ( s ) for all s ∈ S . Unfortunately, this result also does not hold with the new objective function in (6), as formally stated below:

Proposition 3.2. H ( Q,π ) may not necessarily be minimized at π ∗ such that π ∗ = argmax π V π ( s ) , for all s ∈ S .

To overcome the above challenges, our approach involves constructing a more tractable objective function that is a lower bound on the objective of (6). Let us first define Γ( Q ) = min π H ( Q,π ) . We then look at the regularization term, which causes all the aforementioned challenges, and write:

We then take out the negative part of ( r ( s, a ) -Q ( s, a )) using ReLU, and consider a slightly new objective function as follows:

Let ̂ Γ( Q ) = min π ̂ H ( Q,π )) . The proposition below shows that ̂ Γ( Q ) always lower-bounds Γ( Q ) . Proposition 3.3. For any Q ≥ 0 , we have ̂ Γ( Q ) ≤ Γ( Q ) and max Q ̂ Γ( Q ) ≤ max Q Γ( Q ) . Moreover, Γ( Q ) = ̂ Γ( Q ) if Q ( s, a ) ≤ r ( s, a ) for all ( s, a ) .

We note that assuming Q ≥ 0 is not restrictive, as if the expert's rewards r ∗ ( s, a ) are non-negative (typically the case), then the true soft-Q function, defined as Q ∗ ( s, a ) = E [ ∑ s t ,a t γ t ( r ∗ ( s, a ) -log π ( s, a )) | ( s 0 , a 0 ) = ( s, a )] , should also be non-negative, for any π . As ̂ Γ( Q ) provides a lowerbound approximation of Γ( Q ) , maximizing ̂ Γ( Q ) over the Q -space would drive Γ( Q ) towards its maximum value. It is important to note that, given that the inner problem involves minimization, obtaining an upper-bound approximation function is easier. However, since the outer problem is a maximization one, an upper bound would not be helpful in guiding the resulting solution towards optimal ones. The following theorem indicates that ̂ Γ( Q ) is more tractable to use.

Theorem 3.4. For any Q ≥ 0 , the following results hold: (i) The inner minimization problem min π ̂ H ( Q,π ) has a unique optimal solution π Q such that π Q = argmin π V π ( s ) for all s ∈ S and π Q ( a | s ) = exp( Q ( s,a )) ∑ a exp( Q ( s,a )) , (ii) max π V π ( s ) = log( ∑ a exp( Q ( s, a ))) def = V Q ( s ) , and (iii) ̂ Γ( Q ) is concave for Q ≥ 0 .

The above theorem tells us that new objective ̂ Γ( Q ) has a closed form where V π ( s ) is replaced by V Q ( s ) . Moreover ̂ Γ( Q ) is concave for all Q ≥ 0 . The concavity is particularly advantageous, as it guarantees that the optimization objective is well-behaved and has a unique solution Q ∗ such that ( Q ∗ , π Q ∗ ) form a unique saddle point of max Q min π ̂ H ( Q,π ) . Thus, our tractable objective has all the nice properties that the original IQ-Learn objective had, while being able to work for the offline case with multiple levels of expert trajectories and our reward regularizer.

3.4 SPRINQL Algorithm

Algorithm 1 provides the overall SPRINQL algorithm. We first estimate the reference rewards in lines 2-6 of Algorithm 1 and the overall process is described in Section 3.4.1. Before we proceed to the overall training, we have to estimate the weights, w i (associated with the ranked demonstration sets) employed in ˆ H ( Q,π ) . We provide a description of this estimation procedure in Section 3.4.2. Finally, to enhance stability and mitigate over-estimation issues commonly encountered in offline Q-learning, we employ a conservative version of ˆ H ( Q,π ) in lines 8-15 of the algorithm and is described in Section 3.4.3. Some other practical considerations are discussed in the appendix.

Algorithm 1 SPRINQL : Inverse soft-Q Learning with Sub-optimal Demonstrations

3.4.1 Estimating the Reference Reward

We outline our approach to automatically infer the reference rewards r ( s, a ) from the ranked demonstrations. The general idea is to learn a function that assigns higher values to higher expert-level demonstrations. To achieve this, let us define R ( τ ) = ∑ ( s,a ) ∈ τ r ( s, a ) (i.e., accumulated reward of trajectory τ ). For two trajectories τ i , τ j , let τ i ≺ τ j denote that τ i is lower in quality compared to τ j (i.e., τ i belongs to demonstrations from lower-expert policies, compared to τ j ). We follow the Bradley-Terry model of preferences [4, 5] to model the probability P ( τ i ≺ τ j ) as

where the first term of L ( r ) serves to guarantee that the reward reference values for ( s, a ) pairs within the same demonstration group are similar, and the second term aims to increase the likelihood that the accumulated rewards of trajectories adhere to the expert-level order. Importantly, it can be shown below that L ( r ) is convex in r (Proposition 3.5), making the learning well-behaved. In practice, one can model r ( s, a ) by a neural network of parameters θ and optimize L ( θ ) over θ -space.

Proposition 3.5. L ( r ) is strictly convex in r .

3.4.2 Preference-based Weight Learning for w i

Each weight parameter w i used in (4) should reflect the quality of the corresponding demonstration set D i , which can be evaluated by estimating the average expert rewards of these sets. Although this information is not directly available in our setting, the reward reference values discussed earlier provide a convenient and natural way to estimate them. This leads to the following formulation for

3.4.3 Conservative soft-Q learning

Over-estimation is a common issue in offline Q-learning due to out-of-distribution actions and function approximation errors [11]. We also observe this in our IL context. To overcome this issue and enhance stability, we leverage the approach in [23] to enhance our inverse soft-Q learning. The aim is to learn a conservative soft-Q function that lower-bounds its true value. We formulate the conservative inverse soft-Q objective as:

where µ ( a | s ) is a particular state-action distribution. We note that in (8), the conservative term is added to the objective function, while in the conservative Q-learning algorithm [23], this term is added to the Bellman error objective of each Q-function update. This difference makes the theory developed in [23] not applicable. In Proposition 3.6 below, we show that solving max Q ̂ H C ( Q ) will always yield a Q-function that is a lower bound to the Q function obtained by solving max Q ̂ H ( Q,π ) .

We can adjust the scale parameter β in Equation 8 to regulate the conservativeness of the objective. Intuitively, if we optimize Q over a lower-bounded Q-space, increasing the scale parameter β will force each Q ( s, a ) towards its lower bound. Consequently, when β is sufficiently large, ̂ Q C will point-wise lower-bound ̂ Q , i.e., ̂ Q C ( s, a ) ≤ ̂ Q ( s, a ) for all ( s, a ) .

4 Experiments

4.1 Experiment Setup

Baselines. We compare our SPRINQL with SOTA algorithms in offline IL with sub-optimal demonstrations: TRAIL [38], DemoDICE [21], and DWBC [37]. Moreover, we also compare with

other straightforward baselines: BC with only sub-optimal datasets (BC-O), BC with only expert data (BC-E), BC with all datasets (BC-both), and BC with fixed weight for each dataset (W-BC), SQIL [29], and IQ-learn [13] with only expert data. In particular, since TRAIL, DemoDICE, and DWBC were designed to work with only two datasets (one expert and one supplementary), in problems with multiple sub-optimal datasets, we combine all the sub-optimal datasets into one single supplementary dataset. Meanwhile, BC-E, BC-O, and BC-both combine all available data into a large dataset for learning, while W-BC optimizes ∑ i E s,a ∼D i -w i ln( π ( a | s )) , where w i are our weight parameters. T-REX [5] is a PPO-based IL algorithm that can work with ranked demonstrations. It is, however, not suitable for our offline setting, so we do not include it in the main comparisons. Nevertheless, we will later conduct an ablation study to compare SPRINQL with an adapted version of T-REX. Moreover, [15] developed IPL for learning from offline preference data. This approach requires comparisons between every pair of trajectories, thus is not suitable for our context.

Environments and Data Generation: Wetest on five Mujoco tasks [32] and four arm manipulation tasks from Panda-gym [12]. The maximum number transition of ( s, a ) per trajectory is 1000 (or 1k for short) for the Mujoco and is 50 for Panda-gym tasks (descriptions in Appendix B.3). The sub-optimal demonstrations have been generated by randomly adding noise to expert actions and interacting with the environments. We generated large datasets of expert and non-expert demonstrations. For each seed, we randomly sample subsets of demonstrations for testing. This approach allows us to test with different datasets across seeds, rather than using fixed datasets for all seeds as in previous works. More details of these generated databases can be found in the Appendix B.4.

Metric: The return is normalized by score = return -R.return E.return -R.return × 100 where R.return is the mean return of random policy and E.return is the mean return of the expert policy. The scores are calculated by taking the last ten evaluation scores of each seed, with five seeds per report.

Experimental Concerns. Throughout the experiments, we aim to answer the following questions: ( Q1 ) How does SPRINQL perform compared to other baselines? ( Q2 ) How do the distribution matching and reward regularization terms impact the performance of SPRINQL? ( Q3 ) What happens if we augment (or reduce) the expert data while maintaining the sub-optimal datasets? ( Q4 ) What happens if we augment (or reduce) the sub-optimal data while maintaining the expert dataset? ( Q5 ) How does the conservative term help in our approach? ( Q6 ) How does increasing N (the number of expertise levels) affect the performance of SPRINQL? ( Q7 ) Does the preference-based weight learning approach provide good values for the weights w i ? ( Q8 ) How does SPRINQL perform in recovering the ground-truth reward function?

4.2 Main Comparison Results

In this section, we provide comparison results to answer (Q1) with three datasets (i.e., N = 3 ). Additional comparison results for N = 2 can be found in the appendix. From lowest to highest

Table 1: Comparison results for three Mujoco and three Panda-gym tasks.

expertise levels, we randomly sample (25k-10k-1k) transitions for Mujoco tasks and (10k-5k-100) transitions for Panda-gym tasks for every seed (details of these three-level dataset are provided in Appendix B.4). Table 1 shows comparison results across 3 Mujoco tasks and 3 Panda-gym tasks (the full results for all the nine environments are provided in Appendix C.1). In general, SPRINQL significantly outperforms other baselines on all the tasks.

4.3 Ablation Study - No Distribution Matching and No Reward Regularizer

We aim to assess the importance of the distribution matching and reward regularizer terms in our objective ( Q2 ). To this end, we conduct an ablation study comparing SPRINQL with two variants: (i) noReg-SPRINQL , derived by removing the reward regularizer term from (6), and (ii) noDMSPRINQL , obtained by removing the distribution matching term from (6). Here, we note that the noDM-SPRINQL performs Q-learning using the reward reference function, this can viewed as an adaption of the T-REX algorithm [5] to our offline setting. The conservative Q-learning term is employed in the SPRINQL and the two variants to enhance stability. The comparisons for N = 2 and N = 3 on five Mujoco tasks are shown in Figure 1 (the full comparison results for all tasks are provided in the appendix). These results clearly show that SPRINQL outperforms the other variants, indicating the value of both terms in our objective function.

Cheetah

2

3

number of levels

0

50

100

Ant

2

3

number of levels

0

50

100

Walker

2

3

number of levels

0

50

100

Hopper

2

3

number of levels

0

50

100

Humanoid

2

3

number of levels

0

50

100

noReg-SPRINQL

noDM-SPRINQL

expert

4.4 Other Experiments

Experiments addressing the other questions are provided in the appendix. Specifically, Sections C.1 and C.2 provide full comparison results for all the Mujoco and Panda-gym tasks for three and two datasets (i.e., N = 3 and N = 2 ), complementing the answer to Q1 . Section C.3 provides the learning curves of the three variants considered in Section 4.3 above (answering Q2 ). Section C.4 provides experiments to answer Q3 ( what would happen if we augment the expert dataset? ) and Section C.5 addresses Q4 ( what would happen if we augment the sub-optimal dataset? ). Section C.6 experimentally shows the impact of the conservative term in our approaches (i.e., Q5 ). Section C.7 reports the performance of SPRINQL with varying numbers of expertise levels N (i.e., Q6 ). Section C.8 addresses Q7 , and Section C.10 shows how SPRINQL performs in terms of reward recovering (i.e. Q8 ). In addition, Section C.11 reports the distributions of the reference rewards and Section C.12 provides α choosing range.

Concretely, our extensive experiments reveal the following: (i) SPRINQL outperforms other baselines with two, three, or even larger numbers of datasets; (ii) the conservative term, distribution matching, and reward regularizer terms are essential to our objective-all three significantly contribute to the success of SPRINQL; (iii) the preference-based weight learning provides good estimates for the weights w i ; and (iv) SPRINQL performs well in recovering rewards, showing a high positive correlation with the ground-truth rewards, justifying the use of our method for IRL.

5 Conclusion and Limitations

(Conclusion) We have developed SPRINQL, a novel non-adversarial inverse soft-Q learning algorithm for offline imitation learning from expert and sub-optimal demonstrations. We have demonstrated that our algorithm possesses several favorable properties, contributing to its well-behaved, stable, and scalable nature. Additionally, we have devised a preference-based loss function to automate the estimation of reward reference values. We have provided extensive experiments based on several benchmark tasks, demonstrating the ability of our SPRINQL algorithm to leverage both expert and non-expert data to achieve superior performance compared to state-of-the-art algorithms.

(Limitations) Some limitations of this work include: (i) SPRINQL (and other baselines) still requires a large amount of sub-optimal datasets with well-identified expertise levels to learn effectively, (ii) there is a lack of theoretical investigation on how the sizes of the expert and non-expert datasets affect the performance of Q-learning, which we find challenging to address, and (iii) it lacks a theoretical exploration of how the reward regularizer term enhances the distribution matching term when expert samples are low-this question is relevant and interesting but also challenging to address. These limitations will pave the way for our future work.

References

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations , 2022.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning , pages 783-792. PMLR, 2019.

Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning , pages 330-359. PMLR, 2020.

Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning , pages 1262-1277. PMLR, 2021.

Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. In ICLR , 2021.

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems , 33:13049-13061, 2020.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. pandagym: Open-source goal-conditioned environments for robotic learning. arXiv preprint arXiv:2106.13687 , 2021.

Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems , 34:4028-4039, 2021.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems , 36, 2023.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Huy Hoang, Tien Mai, and Pradeep Varakantham. Imitate the good and avoid the bad: An incremental approach to safe reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 12439-12447, 2024.

Minh-Huy Hoang, Long Dinh, and Hai Nguyen. Learning from pixels with expert observations. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1200-1206, 2023.

Ozsel Kilinc and Giovanni Montana. Reinforcement learning for robotic manipulation using simulated locomotion demonstrations. Machine Learning , pages 1-22, 2022.

Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Kee-Eung Kim. Lobsdice: Offline learning from observation via stationary distribution correction estimation. Advances in Neural Information Processing Systems , 35:8252-8264, 2022.

Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations , 2021.

Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations , 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Wenjun Li and Pradeep Varakantham. Generalization through diversity: Improving unsupervised environment design. International Joint Conference on Artificial Intelligence , 2023.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems , 32, 2019.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 , 2019.

Mila Nambiar, Supriyo Ghosh, Priscilla Ong, Yu En Chan, Yong Mong Bee, and Pavitra Krishnaswamy. Deep offline reinforcement learning for real-world treatment optimization applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4673-4684, 2023.

Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09602 , 2017.

Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.11108 , 2019.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations , 2020.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In International Conference on Machine Learning , pages 10961-10970. PMLR, 2021.

Wei-Hung Weng, Mingwu Gao, Ze He, Susu Yan, and Peter Szolovits. Representation and reinforcement learning for personalized glycemic control in septic patients. arXiv preprint arXiv:1712.00654 , 2017.

Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv , abs/1911.11361, 2019.

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning , pages 6818-6827. PMLR, 2019.

Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In Proceedings of the 39th International Conference on Machine Learning , pages 24725-24742, 2022.

Mengjiao Yang, Sergey Levine, and Ofir Nachum. Trail: Near-optimal imitation learning with suboptimal data. In International Conference on Learning Representations , 2021.

Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offline imitation learning with suboptimal demonstrations via relaxed distribution matching. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11016-11024, 2023.

Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor K. Prasanna. Brac+: Improved behavior regularized actor critic for offline reinforcement learning. ArXiv , abs/2110.00894, 2021.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

A Missing Proofs

We provide proofs for the theoritical results claimed in the main paper.

Proposition 3.1 J ( Q,π ) is concave in Q but is not convex in π .

Proof. We recall that

where T π [ Q ]( s, a )) = Q ( s, a ) -γ E s ′ ∼ P ( s ′ | s,a ) [ V π ( s ′ )] and V π ( s ) = E a ∼ π ( a | s ) [ Q ( s, a ) -log π ( a | s )] . We see that T π [ Q ]( s, a ) is linear in Q for any ( s, a ) . Thus, the first and second terms of J ( Q,π ) in (9) are linear in Q . The last term of (9) involves a sum of squares of linear functions of Q , which are convex. So, J ( Q,π ) is concave in Q .

To see that J ( Q,π ) is generally not convex in π , we will consider a quadratic component of the reward regularization term ( T π [ Q ]( s, a )) -r ( s, a )) 2 and show that there is an instance of Q and r values that makes this term convex. We first write:

For simplification, let us choose Q ( s ′ , a ′′ ) = 0 for all s ′ such that P ( s ′ | s, a ) > 0 . This allows us to simplify T π [ Q ]( s, a )) -r ( s, a ) as

We further see that, for any s ∈ S , ∑ a ∈ A π ( a | s ) log π ( a | s ) achieves its minimum value at π ( a | s ) = 1 / | A | for all a ∈ | A | , and ∑ a ∈ A π ( a | s ) log π ( a | s ) ≥ log 1 / | A | for any policy π . As a result we have:

So if we select Q ( s, a ) such that Q ( s, a ) -r ( s, a ) -γ log | A | ≥ 0 , then T π [ Q ]( s, a )) -r ( s, a ) ≥ 0 for any π . Now we consider the quadratic function Γ( π ) = ( λ ( π )) 2 where λ ( π ) = T π [ Q ]( s, a )) -r ( s, a ) . Since each term π ( a ′′ ) log π ( a ′′ | s ′ ) is convex in π , λ ( π ) is convex in π . To show Γ( π ) is convex in π , we will show that for any two policies π 1 , π 2 and α ∈ [0 , 1] , Γ( απ 1 +(1 -α ) π 2 ) ≤ α Γ( π 1 ) + (1 -α )Γ( π ) . To this end, we write

where ( a ) is because the function h ( t ) = t 2 is convex in t , and ( b ) is because

(i) αλ ( π 1 ) + (1 -α ) λ ( π 2 ) ≥ λ ( απ 1 +(1 -α ) π 2 ) (as λ ( π ) is convex in π )

(ii) αλ ( π 1 ) + (1 -α ) λ ( π 2 ) and λ ( απ 1 + (1 -α ) π 2 ) are both non-negative, and function h ( t ) = t 2 is increasing for all t ≥ 0 .

So, we see that with the Q values chosen above, function ( T π [ Q ]( s, a )) -r ( s, a )) 2 ) is convex and -α ( T π [ Q ]( s, a ) -r ( s, a )) 2 is concave. So, intuitively, when α is sufficiently large, J ( Q,π ) would be almost concave (so not convex), which is the desired result.

Proposition 3.2 J ( Q,π ) may not necessarily be minimized at π Q such that π Q = argmax π V π ( s ) , for all s ∈ S .

Proof. We first write J ( Q,π ) as

̸

We then see that the terms E ρ i [ Q ( s, a ) -γ E s ′ V π ( s ′ )] and -γ E s 0 V π ( s 0 ) are minimized (over π ) when V π ( s ) , for all s , are maximized. We will prove that it would not be the case for the last term. Let us choose Q and r such that Q ( s, a ) -r ( s, a ) > γ E s ′ V π Q ( s ′ ) . We see that for any policy π = π Q , we have

which implies that

So the last term of (10) would not be minimized at π = π Q . In fact, in the above scenario, this last term will be maximized at π = π Q . As a result, there is always α sufficiently large such that the last term significantly dominates the other terms and J ( Q,π ) is not minimized at π = π Q .

Proposition 3.3 For any Q ≥ 0 , we have ̂ Γ( Q ) ≤ Γ( Q ) and max Q ≥ 0 ̂ Γ( Q ) ≤ max Q ≥ 0 Γ( Q ) . Mover, Γ( Q ) = ̂ Γ( Q ) if Q ( s, a ) ≤ r ( s, a ) for all s, a .

Proof. We first write H ( Q,π ) as

Since Q ≥ 0 , V π ( s ) = E a ∼ π ( . | s ) [ Q ( s, a ) -log π ( a | s )] ≥ 0 . Thus 2 ReLU ( r ( s, a ) -Q ( s, a )) E s ′ V π ( s ) ≥ 2( r ( s, a ) -Q ( s, a )) E s ′ V π ( s ) . As a result, the last term of ̂ H ( Q,π ) is bounded as

It then follows that ̂ H ( Q,π ) ≤ J ( Q,π ) . Thus, min π ̂ H ( Q,π ) ≤ min π J ( Q,π ) or ̂ Γ( Q ) ≤ Γ( Q ) . Mover, we see that if r ( s, a ) ≥ Q ( s, a ) for all ( s, a ) , then 2 ReLU ( r ( s, a ) -Q ( s, a )) E s ′ V π ( s ) = 2( r ( s, a ) -Q ( s, a )) E s ′ V π ( s ) , implying that ̂ H ( Q,π ) = J ( Q,π ) and ̂ Γ( Q ) = Γ( Q ) . This completes the proof.

Theorem 3.4 For any Q ≥ 0 , the following results hold

(i) The inner minimization problem min π ̂ H ( Q,π ) has a unique optimal solution π ∗ such that π Q = argmin π V π ( s ) for all s ∈ S and

(iii) ̂ Γ( Q ) is concave for Q ≥ 0

Proof. We first rewrite the formulation of ̂ H ( Q,π ) as

We then see that the first and second term of ̂ H ( Q,π ) are minimized when V π ( s ) are minimized, i.e., at π = π Q . For the last term, since V π ( s ) ≥ 0 (because Q ≥ 0 ), -( E s ′ V π ( s ′ )) 2 and -2 ReLU ( r ( s, a ) -Q ( s, a )) E s ′ V π ( s ) are also minimized at π = π Q . So, ̂ H ( Q,π ) is minimized at π = π Q as desired.

( ii ) is already proved in [13].

For ( iii ) , we rewrite ̂ H ( Q,π ) as

Then, the first and second terms of (12) is linear in Q . The fourth term is concave. For third term, let Φ( Q ) = | r ( s, a ) -Q ( s, a ) | + E s ′ V π ( s ) . We see that Φ( Q ) ≥ 0 for any Q ≥ 0 and Φ( Q ) is convex in Q (because V π ( s ) is linear in Q ). It then follows that, for any η ∈ [0 , 1] and Q 1 , Q 2 ≥ 0 , we have

where ( a ) is due to the fact function h ( t ) = t 2 is convex, and ( b ) is because h ( t ) = t 2 is nondecreasing for all t ≥ 0 , and Φ( Q ) is convex and always takes non-negative values. The last inequality in(13) implies that (Φ( Q )) 2 is convex in Q . So the last term of (12) is concave in Q . Putting all together we conclude that ̂ H ( Q,π ) is concave in Q as desired.

Proposition 3.5 L ( r ) is strictly convex in r .

Proof. We first write L ( r ) as

We then see that the first term is a sum of squares of linear functions of r , thus is strictly convex. Moreover, since R ( τ i ) is linear in r for any τ i , the term ln(exp( R ( τ i )) + exp( R ( τ j ))) has a logsum-exp form. So this term is convex as well [3]. Putting all together we see that L ( r ) is strictly convex in r as desired.

Proof. We write as desired.

B Additional Details

B.1 Practical Training Objective

We discuss a practical implementation of the training objective in (4) using samples from both the expert and sub-optimal demonstration sets. We first note that the term E ρ π [ T π [ Q ]( s, a ))] -E ρ π [log π ( s, a )] can be written as [13]:

for any valid occupancy measure ρ ∗ . So, for any policy occupancy measure, we can write this term though the union of expert policies ρ U as

where ( a ) is because ̂ Q C = argmax Q ̂ H C ( Q,π ) and ( b ) is because ̂ H ( ̂ Q,π ) ≥ ̂ H ( ̂ Q C , π ) . Combine (16) and (17) we get

As a result, by replacing V π ( s ) with V Q ( s ) , we can write ̂ Γ( Q ) in a compact form as

In an empirical offline implementation, to maximize ̂ Γ( Q ) , samples ( s, a, s ′ ) from demonstrations can be used to approximate the expectations over ρ i : ∑ ( s,a,s ′ ) ∈D i [ Q ( s, a ) -γV Q ( s ′ )] ,

We note that in continuous-action controls, the computation of V Q ( s ) involves a sum over infinitely many actions, which is impractical. In this case, we can update both Q and π in a soft actor-critic (SAC) manner. That is, for each π , we update Q towards max Q { ̂ H ( Q,π ) } and for each Q , we update π to bring it towards π Q by solving max π { V π ( s ) } , for all s . As shown above, ̂ H ( Q,π ) in concave in Q and V π ( s ) is convex in π , so we can expect this SAC will exhibit good behavior and stability.

B.2 Algorithm Overview

The Figure 2 provide the overview of our algorithm.

Reference Reward Training

Policy Training

(8,a,)~ D

8 ~ D

D1

D2

DV

i ∈ [N]

[n(8，a)-Fn（8,a²)]²

Tn

（8,a),(s’，a’）~Di

Tn

Tg

L(F)

α ~ Tg(- |8)

h,k ∈ [N]

W1, W2,.·., WN

Q

Tn

Q

exp(R(Tk))

h<k

exp(R(Th)+exp(R(Tk))

h~Dh

Tk~Dk

α[△(s,a)]

Eamo(als[@v(s a) - 1(π(a1s)]

i∈[N]

∑wi[Q(s,a)-VQ(s)]

∑vQ(s)-β ∑ Q(s,a)]

i∈[N]

C(Q)

(8,a)=[(Q(8,a)-r(8,a)²+V(s)²

i∈[N]

α~μ(a|s)

s~D

+2ReLU(r(s,a) -Q(s,a))V"(s’)

B.3 Environments

This section provide a detailed descrition of the enviroments used in our experiments.

B.3.1 Mujoco

MuJoCo gym environments [32] like HalfCheetah, Ant, Walker2d, Hopper, and Humanoid are integral to the field of reinforcement learning (RL), particularly in the domain of continuous control and robotics:

HalfCheetah: This environment simulates a two-dimensional cheetah-like robot. The objective is to make the cheetah run as fast as possible, which involves learning complex, coordinated movements across its body.

Ant : This environment features a four-legged robot resembling an ant. The challenge is to control the robot to move effectively, balancing stability and speed.

Walker2d : This environment simulates a two-dimensional bipedal robot. The goal is to make the robot walk forward as fast as possible without falling over.

Hopper : The Hopper environment involves a single-legged robot. The primary challenge is to balance and hop forward continuously, which requires maintaining stability while in motion.

Humanoid : The Humanoid environment is among the most complex, featuring a bipedal robot with a human-like structure. The task involves mastering various movements, from walking to more complex maneuvers, while maintaining balance.

An expert is trying to maximize the reward function by moving with a trajectory length limit of 1000. All five environments are shown in Figure 3.

Cheetah

Ant

Walker

Hopper

Humanoid

B.3.2 Panda-gym

The Panda-gym environments [12], designed for the Franka Emika Panda robot in reinforcement learning, include PandaReach , PandaPush , PandaPickandPlace , and PandaSlide . Here's a short description of each:

PandaReach : The task is to move the robot's gripper to a randomly generated target position within a specific volume. It focuses on precise control of the gripper's movement.

PandaPush : In this environment, a cube placed on a table must be pushed to a target position. The gripper remains closed, emphasizing the robot's ability to manipulate objects through pushing actions.

PandaPickandPlace : This more complex task involves picking up a cube and placing it at a target position above the table. It requires coordinated control of the robot's gripper for both lifting and accurate placement.

PandaSlide : Here, the robot must slide a flat object (like a hockey puck) to a target position on a table. The gripper is fixed in a closed position, and the task demands imparting the right amount of force to slide the object to the target.

In these environments, the reward function is -1 for every time step it has not finished the task. Moreover, the maximum horizon is extremely short, with a maximum of 50, while an expert can complete the task after several steps. All four different environments are shown in Figure 4.

Reach

Push

PnP

Slide

B.4 Qualities of the Generated Expert and Non-Expert Datasets

In this paper, we provide a new setting utilizing ranked sub-optimal datasets to perform imitation learning in the offline setting (illustration in Figure 5).

Ranked sub-optimal datasets

Increasing optimality

1 (expert)

2

3

N

We create sub-optimal datasets by adding noise to actions of the expert policy and let them interact with the environments to collect the sub-optimal trajectories. Each task has its own difficulty and sensitivity to the noise of the actions. The averaged returns of the generated datasets, computed as percentages w.r.t. the maximum returns, are reported in Figure 6.

Cheetah (11415)

3

2

1

0

25

50

75

100

Ant (5187)

3

2

1

0

25

50

75

100

Walker (4477)

3

2

1

0

25

50

75

100

Hopper (3748)

3

2

1

0

25

50

75

100

Humanoid (8038)

3

2

1

0

25

50

75

100

Reach (-1.83)

3

2

1

0

25

50

75

100

Push (-3.61)

3

2

1

0

25

50

75

100

PnP (-6.61)

3

2

1

25

50

75

100

Slide (-8.99)

3

2

1

0

25

50

75

100

B.5 Hyper Parameters and Experimental Implementations

In our experiments, for every algorithm, we run five different seeds corresponding to five different datasets sampled from our databases in Appendix B.4.

We use double Q critic network for our implementation to increase the stability in the offline training scheme.

For IQ-learn [13],SQIL [29], we conduct experiment from its official implement with double Q critic network.

For DemoDICE [21] we conduct experiment from its official implementation .

For DWBC [37] we conduct experiment from its official implementation .

Inspired by double Q-learning, to avoid overfitting in the offline setting, some experiments apply a training trick using KL-divergence between the target actor and the training actor to prevent rapid policy changes.

For SAC-based algorithms, we use a fixed exploration parameter which is commonly used in previous work.

We conducted all experiments on a total of 8 NVIDIA RTX A5000 GPUs and 64 core CPUs. We use 1 GPUs and 8 core CPUs per task with approximately one day per 5 seeds. The detailed hyper-parameters are reported in Table 2.

C Supplementary Experiments

In this section, we present additional experiments that complement those reported in the main paper, along with ablation studies to address the experimental questions stated therein.

C.1 Full Experiment Results for Mujoco and Panda-gym

We report the full results for the 5 different Mujoco environments and 4 different Panda-gym environments, with 3 datasets (i.e., N = 3 ), supplementing the results reported in Table 1 in the main paper. The detailed results for Mujoco in Table 3 and Panda-gym in Table 4

C.2 Comparison Results for N = 2 , i.e., One Expert and One Sub-optimal Datasets - Q1

We provide additional comparison results for N = 2 , complementing to the answer of ( Q1 ), i.e., how does SPRINQL perform compared to other baselines? In this experiment, we want to test the ability of our algorithm in the same scenario of DemoDICE, DWBC which include expert dataset and only one supplementary dataset. Number of expert transitions in Mujoco domains is 1000 and 100 for Panda-gym. Meanwhile, for the supplementary dataset, it is 25000 transitions for Mujoco and 5000 for Panda-gym. In general, although we experience a downgrade in performance due to lack of ranking (only one supplementary dataset), leading to misunderstanding the true reward function, our method is still able to leverage the sub-optimal dataset for understanding the expert demonstrations and provide the highest average score. Reported results are shown in Table 5 6 and Figure 7.

Cheetah

0.0

0.5

1.0

1e6

0

50

100

Ant

0.0

0.5

1.0

1e6

0

50

100

Walker

0.0

0.5

1.0

1e6

0

50

100

Hopper

0.0

0.5

1.0

1e6

0

50

100

Humanoid

0.0

0.5

1.0

1e6

0

50

100

Reach

0.0

0.5

1.0

1e6

0

50

100

Push

0.0

0.5

1.0

1e6

50

100

PnP

0.0

0.5

1.0

1e6

0

50

100

Slide

0.0

0.5

1.0

1e6

0

50

100

DWBC

Weighted-BC

BC

DemoDICE

Expert

C.3 Learning Curves of noReg-SPRINQL and noDM-SPRINQL

This section reports additional results for the comparison between SPRINQL and the two variants noReg-SPRINQL and noDM-SPRINQL (supplementing the answer of Q2 in the main paper). The comparison results for Panda-gym environments are reported in Figure 8 and the learning curves are plotted in Figure 9 and Figure 10.

Reach

2

3

number of levels

0

50

100

Push

2

3

number of levels

0

50

100

PnP

2

3

number of levels

0

50

100

Slide

2

3

number of levels

0

50

100

noReg-SPRINQL

noDM-SPRINQL

expert

Cheetah

0.0

0.5

1.0

1e6

0

50

100

Ant

0.0

0.5

1.0

1e6

25

50

75

100

Walker

0.0

0.5

1.0

1e6

0

50

100

Hopper

0.0

0.5

1.0

1e6

0

50

100

Humanoid

0.0

0.5

1.0

1e6

0

50

100

Reach

0.0

0.5

1.0

1e6

0

50

100

Push

0.0

0.5

1.0

1e6

50

100

PnP

0.0

0.5

1.0

1e6

0

50

100

Slide

0.0

0.5

1.0

1e6

0

50

100

noReg-SPRINQL

noDM-SPRINQL

expert

Cheetah

0.0

0.5

1.0

1e6

0

50

100

Ant

0.0

0.5

1.0

1e6

25

50

75

100

Walker

0.0

0.5

1.0

1e6

0

50

100

Hopper

0.0

0.5

1.0

1e6

0

50

100

Humanoid

0.0

0.5

1.0

1e6

0

50

100

Reach

0.0

0.5

1.0

1e6

0

50

100

Push

0.0

0.5

1.0

1e6

0

50

100

PnP

0.0

0.5

1.0

1e6

0

50

100

Slide

0.0

0.5

1.0

1e6

0

50

100

noReg-SPRINQL

noDM-SPRINQL

expert

C.4 Augmented Expert Demonstrations

We provide experiments to address ( Q3 ) -What happens if we augment the expert data while maintaining the sub-optimal datasets ? To this end, we use two Mujoco and two Panda-gym tasks, keeping the same sup-optimal datasets and add more expert demonstrations to the training sets. The comparison results are reported in Figure 11. For the Mujoco tasks, which are more difficult, adding more expert trajectories significantly enhances the performance of all the algorithms. However, for the two Panda-gym tasks, the influence of adding more expert data appears to be less significant in Push and completely absent in PnP. This would be because expert trajectories in these tasks are typically short, consisting of only 2-7 transitions. Hence, a larger quantity of additional expert data may be required to enhance performance.

Cheetah

0.0

0.5

1.0

1e6

0

50

100

Ant

0.0

0.5

1.0

1e6

0

50

100

Push

0.0

0.5

1.0

1e6

0

50

100

PnP

0.0

0.5

1.0

1e6

0

50

100

DWBC

DWBC (+E)

DemoDICE (+E)

BC (+E)

Expert

C.5 Augmented Sup-optimal Demonstrations

Cheetah

10k-5k

25k-10k

25k-25k

0

50

100

Ant

10k-5k

25k-10k

25k-25k

0

50

100

Push

5k-1k

10k-5k

10k-10k

0

50

100

PnP

5k-1k

10k-5k

10k-10k

0

50

100

DWBC

BC

DemoDICE

Expert

In this experiment, we want to answer the question ( Q4 ) -What happens if we augment (or reduce) the sub-optimal data while maintaining the expert dataset? . We present numerical results to assess

the impact of sub-optimal data on the performance of our SPRINQL and other baselines. To this end, we keep the same expert dataset and adjust the non-expert data used in Table 1. The performance is reported in Fig.12. It is evident that reducing the amount of data in sub-optimal datasets can result in a significant degradation in the performance of our SPRINQL and the other baselines. Conversely, adding more sub-optimal data can enhance overall stability and lead to improved performance.

In the Figure 13 we show the learning curves with varied sizes for sub-optimal datasets. It can be seem that the overall performance tends to improve with more sup-optimal datasets. In particular, for the Ant task, our SPRINQL even fails to maintain stability when the sizes of sup-optimal datasets are low (10k-5k-1k). Moreover, while BC seems to show improvement with more sub-optimal data, the performance of DWBC and DemoDICE remains unchanged. This may be because these approaches rely on the assumption that expert demonstrations can be extracted from the sub-optimal data, which is not the case in our context.

Cheetah

10k-5k-1k

0.0

0.5

1.0

1e6

0

50

100

25k-10k-1k

0.0

0.5

1.0

1e6

0

50

100

25k-25k-1k

0.0

0.5

1.0

1e6

0

50

100

Ant

10k-5k-1k

0.0

0.5

1.0

1e6

0

100

25k-10k-1k

0.0

0.5

1.0

1e6

0

50

100

25k-25k-1k

0.0

0.5

1.0

1e6

0

50

100

Push

5k-1k-100

0.0

0.5

1.0

1e6

50

100

10k-5k-100

0.0

0.5

1.0

1e6

50

100

10k-10k-100

0.0

0.5

1.0

1e6

50

100

PnP

5k-1k-100

0.0

0.5

1.0

1e6

0

50

100

10k-5k-100

0.0

0.5

1.0

1e6

0

50

100

10k-10k-100

0.0

0.5

1.0

1e6

0

50

100

DWBC

BC

DemoDICE

Expert

C.6 Impact of the Conservative Term

In this experiment, we aim to answer ( Q5 ) -How does the conservative term help in our approach ? The Equation 8 introduce the conservative Q learning (CQL) term into our work. Here we also test three variants of our method from Section 4.3 and show the impact of CQL to the final performance. The experimental results are shown in Figure 14.

Cheetah

0.0

0.5

1.0

1e6

0

50

100

Ant

0.0

0.5

1.0

1e6

0

100

Push

0.0

0.5

1.0

1e6

0

50

100

PnP

0.0

0.5

1.0

1e6

0

50

100

noReg-SPRINQL (no CQL)

noReg-SPRINQL

noDM-SPRINQL (no CQL)

noDM-SPRINQL

expert

C.7 Performance with Varying Number of Expertise Levels

5

4

3

2

1

0

25

50

75

100

1 sub-optimal dataset

0.0

0.5

1.0

1e6

0

50

100

2 sub-optimal datasets

0.0

0.5

1.0

1e6

0

50

100

3 sub-optimal datasets

0.0

0.5

1.0

1e6

0

50

100

4 sub-optimal datasets

0.0

0.5

1.0

1e6

0

50

100

DemoDICE

BC

DWBC

Expert

We provide an experiment to answer ( Q6 ) -How does increasing N (the number of expertise levels) affect the performance of SPRINQL? We assess our algorithm with different numbers of expertise levels to investigate how adding or removing sub-optimal expertise levels influences performance. Specifically, we keep the same expert dataset comprising 1 expert trajectory and conduct tests with 1 to 4 sub-optimal datasets from the Cheetah task. Details of the average returns of the five datasets are reported in Fig. 15. In this context, SPRINQL outperforms other algorithms in utilizing non-expert

demonstrations. Furthermore, BC successfully learns and achieves performance comparable to the performance of SPRINQL with 2 dataset, while DemoDICE and DWBC struggle to learn. The detailed results are plotted in Figure 16. In Section (C.2) below, We particularly delve into the situation of having two datasets (one expert and one sub-optimal), which is a typical setting in prior work.

C.8 Ablation Study for the Preference-based Weight Learning

We provide this experiment to answer ( Q7 ) -Does the preference-based weight learning approach provide good values for the weights w i ? To this end, we compare the performance of SPRINQL based on the weights determined by the preference-based methods described in the main paper (denoted as auto W ), and the following weighting scenarios:

Uniform W: We chose the weights w i uniformly over [0 , 1] as w = { 0 . 55 , 0 . 35 , 0 . 15 } with a ratio of approximately 10 : 7 : 4 .

Reduced W: Starting from Uniform W , we reduced the weights of the non-expert data and tested the weights w = { 0 . 65 , 0 . 2 , 0 . 15 } with a ratio of approximately 10 : 3 : 2 .

Increased W: Starting from Uniform W , we increased the weights of the non-expert data and chose w = { 0 . 4 , 0 . 32 , 0 . 28 } with a ratio of approximately 10 : 8 : 7 .

These weight vectors w are normalized to follow w 1 > w 2 > . . . > w N and ∑ i ∈ [ N ] w i = 1 . The comparison results are shown in Figure 17.

Cheetah

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0.0

0.2

0.4

0.6

0.8

1.0

Uniform W

Reduced W

Increased W

auto W

Expert

Ant

0.0

0.2

0.4

0.6

0.8

1.0

1e6

0.2

0.4

0.6

0.8

1.0

Uniform W

Reduced W

Increased W

auto W

Expert

C.9 D4RL mujoco dataset

In this section, we present additional experiments using the official D4RL dataset. Unfortunately, since our algorithm requires meaningful demonstrations, we exclude the Random dataset and are only able to test with N = 2 (Medium and Expert datasets). The detailed results are shown in Figure 18.

Cheetah

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

-20

0

20

40

60

80

100

Score

DemoDICE

DWBC

BC-all

Expert

Ant

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

-20

0

20

40

60

80

100

Score

DemoDICE

DWBC

BC-all

Expert

Walker

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

-20

0

20

40

60

80

100

Score

DemoDICE

DWBC

BC-all

Expert

Hopper

0.0

0.2

0.4

0.6

0.8

1.0

Steps

1e6

-20

0

20

40

60

80

100

Score

DemoDICE

DWBC

BC-all

Expert

C.10 Reward Recovering

In this experiment, we want to answer ( Q8 ) -How does SPRINQL perform in recovering the groundtruth reward function? Compared to BC-based algorithms, one notable advantage of Q-learning based algorithms is their ability to recover the reward function. Here, we present experiments demonstrating reward function recovery across five MuJoCo tasks, comparing recovered rewards to the actual reward function. To achieve this, we introduce increasing levels of random noise to the actions of a trained agent and observe its interactions with the environment. We collect the state, action, and next state for each trajectory, then predict the recovered reward and compare it to the true reward from the environment. For the sake of comparison, we include noReg-SPRINQL , which can be considered an an adaption of IQ-learn [13] to our setting, and noDM-SPRINQL , which is in fact an adaption of T-REX to our offline setting.

Comparison results are presented in Figure 19. We observe a linear relationship between the true and predicted rewards for SPRINQL across all testing tasks, whereas the other approaches fail to return correct relationships for some tasks.

Cheetah

noReg-SPRINQL

5000

10000

True Return

-150000

-100000

-50000

0

Predicted Return

noDM-SPRINQL

2500

5000

7500

10000

True Return

200

300

400

Predicted Return

2500

5000

7500

10000

True Return

300

400

500

600

700

Predicted Return

Ant

noReg-SPRINQL

0

2000

4000

True Return

0

100

200

300

400

Predicted Return

noDM-SPRINQL

0

2000

4000

True Return

0

100

200

300

400

500

Predicted Return

0

2000

4000

True Return

0

100

200

300

400

500

Predicted Return

Walker

noReg-SPRINQL

0

1000

2000

3000

4000

True Return

0

100

200

300

400

Predicted Return

noDM-SPRINQL

0

1000

2000

3000

4000

True Return

0

200

400

Predicted Return

0

1000

2000

3000

4000

True Return

0

200

400

600

Predicted Return

Hopper

noReg-SPRINQL

0

1000

2000

3000

True Return

50

100

150

200

250

Predicted Return

noDM-SPRINQL

0

1000

2000

3000

True Return

50

100

150

200

250

Predicted Return

0

1000

2000

3000

True Return

0

200

400

600

Predicted Return

Humanoid

noReg-SPRINQL

0

2500

5000

7500

True Return

-300000

-200000

-100000

Predicted Return

noDM-SPRINQL

0

2500

5000

7500

True Return

-100000

-80000

-60000

-40000

-20000

Predicted Return

0

2000

4000

6000

8000

True Return

0

200

400

600

Predicted Return

C.11 Reference Reward Distribution

From the experiment reported in Table 1, we plot the distributions of the reward reference values in Figure 20, where the x -axis shows the level indexes and the y -axis shows reward values. The rewards seem to follow desired distributions, with larger rewards assigned to higher expertise levels. Moreover, rewards learned for expert demonstrations are consistently and significantly higher and exhibit smaller variances compared to those learned for sub-optimal transitions.

Cheetah

3

2

1

0.00

0.25

0.50

0.75

1.00

Ant

3

2

1

0.00

0.25

0.50

0.75

1.00

Walker

3

2

1

0.00

0.25

0.50

0.75

1.00

Hopper

3

2

1

0.00

0.25

0.50

0.75

1.00

Humanoid

3

2

1

0.00

0.25

0.50

0.75

1.00

Reach

3

2

1

0.0

0.2

0.4

0.6

0.8

1.0

Push

3

2

1

0.0

0.2

0.4

0.6

0.8

1.0

PnP

3

2

1

0.0

0.2

0.4

0.6

0.8

1.0

Slide

3

2

1

0.0

0.2

0.4

0.6

0.8

1.0

C.12 Different alpha ablation study

As our objective is a combination of two terms with a balancing parameter α , we conduct additional experiments to evaluate the performance of our method across a range of α values. The detailed results are presented in Figure 21. Overall, the results indicate that α can be selected within the range of 0.1 to 10 to achieve good performance.

Cheetah (N=3)

0

0.01

0.1

1

10

100

+oo

Alpha

0.0

0.2

0.4

0.6

0.8

1.0

Score

Expert

Ant (N=3)

0

0.01

0.1

1

10

100

+oo

Alpha

0.0

0.2

0.4

0.6

0.8

1.0

Score

Expert

Cheetah (N=2)

0

0.01

0.1

1

10

100

+oo

0.0

0.2

0.4

0.6

0.8

1.0

Score

Expert

Ant (N=2)

0

0.01

0.1

1

10

100

+oo

0.0

0.2

0.4

0.6

0.8

1.0

Score

Expert

Alpha

Alpha

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract includes our main claims reflecting our main contributions and finding.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a discussion on the limitations of our work in the conclusion section. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the proofs of the theorems and propositions stated in the main paper are provided in the appendix with clear references.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details on the environments and hyper-parameter settings in the appendix. We also uploaded our source code for re-productivity purposes.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have describe how to generate our data as well as provide it along with our submitted source code with sufficient instructions for their use.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have detailed these information in the main paper and the appendix of our paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have reported the mean scores and standard deviations for the result tables. We have also shown training curves constructed from mean scores and shaded by standard error. All the experiments are reported with multiple training seeds as well as different datasets.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided these information in the 'Hyper parameter and Experimental Implementations' section in our appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper provides a general offline imitation learning with multiple expert levels and only testing on the simulated environments. As such, we do not foresee any direct societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our training data are generated from open source simulated environments which have no risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have provided clear citations to the source code and data we used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibility or benchmarking purposes.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We have no crowdsourcing experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

We do not have study participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of offline imitation learning from sub-optimal demonstrations. The paper proposes the SPRINQL algorithm, which automatically weights the demonstrations, regularizes the loss with a reference reward function, and then performs inverse soft-Q learning. Experimental results show that the proposed method achieves better performance compared with baselines. Extensive ablations are also provided.\n\nSTRENGTHS:\nThe paper is overall well written. The experimental results are strong, and the ablation studies are extensive. It is impressive to see the proposed method improves its performance with more sub-optimal demonstrations provided.\n\nWEAKNESSES:\n1. Some parts of the proposed approach are unclear to me. Please see questions 1-4. \n\n2. The influence of the reward regularization term is not completely studied. Please see question 5.\n\nQUESTIONS:\n1. In Equation (3), a modified objective is presented. However, it seems that this objective depends largely on the quality of weights, and it does not necessarily recover the ground truth reward function nor the optimal policy. Given that the weight is chosen with a heuristic, what kind of guarantee can the authors give about the converged policy?\n\n2. Following the first question, in the experiments, seems that the proposed algorithm always outperforms BC-E because of the lack of expert demonstrations. However, I am curious to see what will happen given sufficient expert demonstrations. \n\n3. In equations above line 162, what if $(s, a) = (s', a')$, given that it is possible for two stochastic policies to choose the same action in the same state?\n\n4. Line 173, regarding \"prior research\", could the authors add some citations?\n\n5. It seems that the weight for reward regularization $\\\\alpha$ is an important hyperparameter. Will the performance of the proposed method change a lot with different $\\\\alpha$? In the extreme case, what if we learn a policy with RL using reward function $\\\\bar r$?\n\nLIMITATIONS:\nThe authors discuss the limitations adequately in the last Section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of imitation learning on suboptimal demonstrations given coarse trajectory expertise rankings. It introduces SPRINQL, a method combining inverse soft-Q learning, conservative Q learning, and reward learning from ranked trajectory preferences. SPRINQL estimates a reference reward function given trajectory expertise rankings, and extends inverse soft-Q learning to the suboptimal demonstration setting by adding reward regularization and weights based on the estimated reference reward. The paper shows that the SPRINQL objective is tractable with a unique solution over $Q, \\pi$. Experiments show that SPRINQL outperforms prior methods on suboptimal trajectories in simulated environments. Ablations justify each component in SPRINQL and demonstrate robustness across a range of dataset sizes and number of suboptimal datasets.\n\nSTRENGTHS:\n- SPRINQL works with a number of suboptimal demonstrations, with reasonable assumptions such as access to trajectory expertise rankings.\n- The main objective is shown to admit a unique saddle point solution over $Q, \\pi$ and is tractable for optimization.\n- Experiments show that SPRINQL outperforms relevant offline imitation learning baselines on suboptimal trajectories in multiple simulated locomotion and manipulation environments.\n- Extensive ablations demonstrate that the method works with a range of numbers/sizes of suboptimal datasets, and justify each component in SPRINQL.\n\nWEAKNESSES:\n- Suboptimal trajectories are sourced from adding random noise to expert actions. While this is a reasonable proxy, it would be more convincing to see them generated from under-trained RL policies instead.\n- It would be great to see SPRINQL work on image-based environments.\n\nQUESTIONS:\n- How does SPRINQL perform on suboptimal trajectories generated from under-trained RL policies, instead of rollouts from noisy expert actions?\n\nLIMITATIONS:\n- SPRINQL assumes access to trajectory expertise rankings.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper presents SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning), a novel algorithm for offline imitation learning with multiple levels of expert demonstrations. SPRINQL leverages distribution matching and reward regularization to effectively leverage the mixture of non-expert data and expert data to enhance learning. An extensive evaluation is conducted to assess SPRINQL, answering eight well-defined questions regarding the capability and performance of SPRINQL.

STRENGTHS:
+ The proposed technique presents an improvement for learning from mixed-expert demonstrations and is backed by theory. 
+ The paper is very well-written and contains sufficient detail to understand the key proposed novelties. 
+ The evaluation (including those in the appendix) is extensive and presents strong support for the proposed algorithm.

WEAKNESSES:
- The authors mention that this setting (having several sets of data labeled with relative quality scores) is more general than two quality levels: expert and sub-optimal. Could the authors provide further support for this claim? I would argue that having additional quality annotations makes this setting more niche.
- Table 1 contains a very large evaluation but there isn't much commentary on the different trends between baselines and SPRINQL.
Comment: The example in the introduction does not seem complete.

QUESTIONS:
Please address the weaknesses above.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents an offline imitation learning (IL) method that leverages both expert and suboptimal demonstrations. Distinct from previous methods, this approach assumes access to suboptimal demonstrations across a spectrum of performance levels, with a known ranking to the learners.\nTo utilizes suboptimal demonstations with various levels, this work suggest SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse Q-Learning), a reward reguarized inverse soft Q-learning method for the offline IL problem.\nThe evaluation of this method is conducted using the Mujoco and Panda-gym environments, employing a small dataset of expert demonstrations and a larger collection of demonstrations generated by introducing varying levels of noise to expert actions.\n\nSTRENGTHS:\nThis work aims to extend the domain of offline imitation learning by addressing the challenge of leveraging imperfect demonstrations with multiple suboptimality levels, presenting a novel and intriguing approach. Drawing on foundational principles from prior research, notably IQ-Learn, this study demonstrates theoretically that the proposed extension, SPRINQL, retains the advantageous properties of IQ-Learn.\nFurthermore, SPRINQL adeptly incorporates sophisticated concepts such as the Bradley-Terry model and Conservative Q-Learning (CQL) into its training framework. The experimental outcomes are notable, with SPRINQL achieving surprisingly effective results that surpass previous baselines, which were unable to successfully imitate expert performance using the datasets used in this work.\n\nWEAKNESSES:\nSignificant concerns with this submission are the questionable reporting on the performance of baseline methods and issues related to reproducibility. \nThe experimental results presented are not entirely convincing, as the performance of the baselines substantially deviates from those reported in their original papers. \nSpecifically, even in the case with N=2 suboptimality levels, suitable baselines such as TRAIL, DemoDICE, and DWBC fail to outperform even basic BC-all in nearly all the Mujoco tasks listed in Table 3 (Appendix). \nThe authors need to clarify the reasons behind this anomaly.\nTo enhance the trustfulness of experiments, it would be beneficial for the authors to release the datasets utilized in their experiments or to use standardized datasets such as D4RL or RL-unplugged. This would allow for a more reliable validation of the results and facilitate comparisons with other methods.\n\nQUESTIONS:\n- Could the authors clarify the motivation of employing reference rewards?\n- Moreover, could you elucidate why reference rewards are utilized as a regularization mechanism rather than directly applying RL using reference rewards?\n- The problem setting appears closely aligned with preference-based learning, where optimality (or preference) levels are explicitly known. Could you delineate the distinctions between these two problem settings?\n- Can you provide detailed justifications for why comparing the proposed method with IPL is deemed unsuitable? \nIt appears feasible to compare by segmenting each trajectory by suboptimality levels, then pairing segments from differing levels to facilitate IPL. Could this approach not be applied effectively?\n\n[minor comments] reference [15] in line 398 from \"NeurIPS 2024\" to \"NeurIPS 2023.\"\n\nLIMITATIONS:\nThe authors addressed their limitations in Conclusion section in the main manuscript. They described that this work does not have any direct societal impacts in the Checklist #10, Broader Impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-27"}