SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

Authors: huy  hoang
tien  mai
pradeep  varakantham
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:16:26

Abstract: We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. 

In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL .

Main paper: 
 introduction: 
 Reinforcement learning (RL) has established itself as a strong and reliable framework for sequential decision-making with applications in diverse domains: robotics [19, 18] , healthcare [34, 28, 27] , and environment generation [9, 24] . Unfortunately, RL requires an underlying simulator that can provide rewards for different experiences, which is usually not available.Imitation Learning (IL) [16, 29, 13, 17] handles the lack of reward function by utilizing expert demonstrations to guide the learning scheme to compute a good policy. However, IL approaches still require the presence of a simulator that allows for online interactions. Initial works in Offline IL [35, 23, 40, 2] tackle the absence of simulator by considering an offline dataset of expert demonstrations. These approaches extend upon Behavioral Cloning (BC), where we aim to maximize the likelihood of the expert's decisions from the provided dataset. The key advantage with BC is the theoretical justification on converging to expert behaviors given sufficient trajectories. However, when there are not enough expert trajectories, it often suffers from distributional shift issues [30] . Thus, a key drawback of these initial IL approaches is the need for a large number of expert demonstration datasets.To deal with limited expert demonstrations, recent works utilize non-expert demonstration datasets to reduce the reliance on only expert demonstrations. These additional non-expert demonstrations are referred to as supplementary data. Directly applying BC to these larger supplementary datasets will lead to sub-optimal policies, so most prior work in utilizing supplementary data attempts to extract expert-like demonstrations from the supplementary dataset in order to expand the expert demonstrations [31, 21, 20, 37, 39] . These works assume expert-like demonstrations are present in the supplementary dataset and focus on identifying and utilizing those, while eliminating the non-expert demonstrations. Eliminating non-expert trajectories can result in loss of key information (e.g., transition dynamics) about the environment. Additionally, these works primarily rely on BC, which is known to overlook the sequential nature of decision-making problems -a small error can quickly accumulate when the learned policy deviates from the states experienced by the expert.We develop our algorithm based on an inverse Q-learning framework that better captures the sequential nature of the decision-making [13] and can operate under the more realistic assumption that the data is collected from people/policies with lower expertise levels 1 (not experts). To illustrate, consider a scenario in robotic manipulation where the goal is to teach a robot to assemble parts. Expert demonstrations might show precise and efficient methods to assemble parts, but are limited in number due to the high cost and time associated with expert involvement. On the other hand, sub-optimal demonstrations from novice users are easier to obtain and more abundant. Our SPRINQL approach effectively integrates these sub-optimal demonstrations, giving appropriate weight to the expert demonstrations to ensure the robot learns the optimal assembly method without overfitting to the limited expert data or the inaccuracies in the sub-optimal data. We utilize these non-expert trajectories to learn a Q function that contributes to our understanding of the environment and the ground truth reward function.Contributions: Overall, we make the following key contributions in this paper: (i) We propose SPRINQL, a novel algorithm based on Q-learning for offline imitation learning with expert and multiple levels of sub-optimal demonstrations. (ii) We provide key theoretical properties of the SPRINQL objective function, which enable the development of a scalable and efficient approach. In particular, we leverage distribution matching and reward regularization to develop an objective function for SPRINQL that not only help address the issue of limited expert samples but also utilizes non-expert data to enhance learning. Our objective function is not only convex within the space of Q functions but also guarantees the return of a Q function that lower-bounds its true value. (iii) We provide an extensive empirical evaluation of our approach in comparison to existing best algorithms for offline IL with sup-optimal demonstrations. Our algorithms provide state-of-the-art (SOTA) performance on all the benchmark problems. Moreover, SPRINQL is able to recover a reward function that shows a high positive correlation with the ground-truth rewards, highlighting a unique advantage of our approach compared to other IL algorithms in this context.Imitation Learning. Imitation learning is recognized as a significant technique for learning from demonstrations. It begins with BC, which aims to maximize the likelihood of expert demonstrations. However, BC often under-performs in practice due to unforeseen scenarios [30] . To overcome this limitation, Generative Adversarial Imitation Learning (GAIL) [16] and Adversarial Inverse Reinforcement Learning (AIRL) [10] have been developed. These methods align the occupancy distributions of the policy and the expert within the Generative Adversarial Network (GAN) framework [14] . Alternatively, Soft Q Imitation Learning (SQIL) [29] bypasses the complexities of adversarial training by assigning a reward of +1 to expert demonstrations and 0 to the others, subsequently learning a value function based on these rewards.While the aforementioned imitation learning algorithms show promise, they require interaction with the environment to obtain the policy distribution, which is often impractical.Offline Imitation Learning. ValueDICE [22] introduces a novel approach for off-policy training, suitable for offline training, using Stationary Distribution Corrections [25, 26] . However, ValueDICE necessitates adversarial training between the policy network and the Q network, which can make the training slow and unstable. Recently, algorithms like PWIL [8] and IQ-learn [13] have optimized distribution distance, offering an alternative to adversarial training schemes. Since such approaches rely on occupancy distribution matching, a large expert dataset is often required to achieve the desired performance. Our approach, SPRINQL is able to bypass this requirement of a large set of expert demonstrations through the use of non-expert demonstrations (which are typically more available) in conjunction with a small set of expert demonstrations.Imitation Learning with Imperfect Demonstrations. T-REX [5] and D-REX [6] have shown that utilizing noise-ranked demonstrations as a reference-based approach can return a better policy without requiring expert demonstrations in online settings. Moreover, there are also several works [36, 33] that utilize the GAN framework [14] for sub-optimal datasets and have achieved several successes. Meanwhile, in the offline imitation learning context, TRAIL [38] utilizes sup-optimal demonstrations to learn the environment's dynamics. It employs a feature encoder to map the high-dimensional state-action space into a lower dimension, thereby allowing for a scalable way of learning of dynamics. This approach may face challenges in complex environments where predicting dynamics accurately is difficult, as shown in our experimental results. Other works assume that they can extract expert-like state-action pairs from the sub-optimal demonstration set and use them for BC with importance sampling [31, 37, 39, 21, 20] . However, expert-like state-actions might be difficult to accurately identify, as true reward information is not available. In contrast, our approach is more general, as we do not assume that the sub-optimal set contains expert-like demonstrations. We also allow for the inclusion of demonstrations of various qualities. Moreover, while prior works only recover policies, our approach enables the recovery of both expert policies and rewards, justifying the use of our method for Inverse Reinforcement Learning (IRL) [1] . 
background: 
 Preliminaries. We consider a MDP defined by the following tuple M = ⟨S, A, r, P, γ, s 0 ⟩, where S denotes the set of states, s 0 represents the initial state set, A is the set of actions, r : S × A → R defines the reward function for each state-action pair, and P : S × A → S is the transition function, i.e., P (s ′ |s, a) is the probability of reaching state s ′ ∈ S when action a ∈ A is made at state s ∈ S, and γ is the discount factor. In reinforcement learning (RL), the aim is to find a policy that maximizes the expected long-term accumulated reward max π E (s,a)∼ρπ [r(s, a)] , where ρ π is the occupancy measure of policy π:ρ π (s, a) = (1 -γ)π(a|s) ∞ t=1 γ t P (s t = s|π).MaxEnt IRL The objective in MaxEnt IRL is to recover a reward function r(s, a) from a set of expert demonstrations, D E . Let ρ E be the occupancy measure of the expert policy. The MaxEnt IRL framework [41] proposes to recover the expert reward function by solvingEQUATIONIntuitively, the aim is to find a reward function that achieves the highest difference between the expected value of the expert policy and the highest expected value among all other policies (computed through the min loop).IQ-Learn Given a reward function r and a policy π, the soft Bellman equation is defined asB π r [Q](s, a) = r(s, a) + γE s ′ [V π (s ′ )], where V π (s) = E a∼π(a|s) [Q(s, a) -log π(a|s)]. The Bellman equation B π r [Q] = Q is contractiveand always yields a unique Q solution [13] . In IQ-learn, they further define an inverse soft-Q Bellman operator [13] show that for any reward function r(a, s), there is a uniqueT π [Q] = Q(s, a) -γE s ′ [V π (s ′ )].Q * function such that B π r [Q * ] = Q * ,and for a Q * function in the Q-space, there is a unique reward function r such that r = T π [Q * ]. This result suggests that one can safely transform the objective function of the MaxEnt IRL from r-space to the Q-space as follows:EQUATIONwhich has several advantages; [13] show that Φ(π, Q) is convex in π and linear in Q, implying that (2) always yields a unique saddle point solution. In particular, (2) can be converted into a maximization over the Q-space, making the training problem no longer adversarial. 
sprinql: 
 We now describe our inverse soft-Q learning approach, referred to as SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning). We first describe the three key components in the SPRINQL formulation:(1) We formulate the objective function that enables matching the occupancy distribution of not just expert demonstrations, but also sub-optimal demonstrations.(2) To mitigate the effect of limited expert samples (and larger sets of sub-optimal samples) that can bias the distribution matching of the first step to sub-optimal demonstrations, we introduce a reward regularization term within the objective. This regularization term is to ensure reward function allocates higher values to state-action pairs that appear in higher expertise demonstrations.(3) We show that while this new objective does not have the same advantageous properties as the one in inverse Q-learning [13] , with some minor (yet significant) changes it is possible to restore all the important properties.We consider a setting where there are demonstrations classified into several sets of different expertise levels D 1 , D 2 , ...., D N , where D 1 consists of expert demonstrations and all the other sets contains suboptimal ones. This setting is general than existing work in IL with sup-optimal demonstrations, which typically assumes that there are only two quality levels: expert and sub-optimal. Let D = i∈[N ] D i be the union of all the demonstration sets and ρ 1 , ..., ρ N be the occupancy measures of the respective expert policies. The ordering of expected values across different levels of expert policies would then be given by:E ρ 1 [r * (s, a)] > E ρ 2 [r * (s, a)] > ... > E ρ N [r * (s, a)],where r * (., .) are the ground-truth rewards. Typically, the number of demonstrations in first level, D 1 is significantly lower than those from other expert levels, i.e., |D 1 | ≪ |D i |, for i = 2, ..., N The MaxEnt IRL objective from Equation 1 can thus be adapted as follows:EQUATIONwhere w i ≥ 0 is the weight associated with the expert level i ∈ [N ] and we have w 1 > w 2 > ... > w N and i∈[N ] w i = 1. There are two key intuitions in the above optimization: (a) Expert level i accumulates higher expected values than expert levels greater than i; and (b) Difference in values accumulated by expert policies and the maximum of all other policies is maximized. The optimization term can be rewritten as:E ρ U [r(s, a)] -E ρπ [r(s, a)] -E ρπ [log π(s, a)],where ρ U = i∈[N ] w i ρ i . Here we note that the expected reward i∈[N ] w i E ρ i [r(s, a)] is empirically approximated by samples from the demonstration sets D 1 , D 2 , ...., D N . The number of demonstrations in the best demonstration set D 1 (i.e. the set of expert demonstrations) is significantly lower when compared to other demonstration sets. So, an empirical approximation of E ρ 1 [r(s, a)] using samples from D 1 would be inaccurate.We create a reference reward based on the provided expert and sub-optimal demonstrations and utilize the reference function to compute a regularization term that is added to the objective of Equation 3.Concretely, we define a reference reward function r(s, a) such that: r(s, a) > r(s ′ , a ′ ), ∀(s, a) ∈ D 1 and (s ′ , a ′ ) / ∈ D 1 and r(s, a) > r(s ′ , a ′ ), ∀(s, a) ∈ D 2 and ∀(s ′ , a ′ ) / ∈ D 2 ∪ D 1 and so onThe aim here is to assign higher rewards to demonstrations from higher expertise levels, and zero rewards to those that do not belong to provided demonstrations. We will discuss how to concretely estimate such reference reward values later.We utilize this reference reward as part of the reward regularization term, which is added into the MaxEntIRL objective in (3) as follows:max r min π E ρ U [r(s, a)] -E ρπ [r(s, a)] + E ρπ [log π(s, a)]Occupancy matching-αE ρ U [(r(s, a) -r(s, a)) 2 ]Reward regularizer (4) where α > 0 is a weight parameter for the reward regularizer term. With (4), the goal is to find a policy with an occupancy distribution that matches with the occupancy distribution of different expertise levels appropriately (characterized by the weights, w i ). Simultaneously, it ensures that the learning rewards are close to the pre-assigned rewards, aiming to guide the learning policy towards replicating expert demonstrations, while also learning from sub-optimal demonstrations.Even though (4) can be directly solved to recover rewards, prior research suggests that transforming (4) into the Q-space will enhance efficiency. We delve into this transformation approach in this section. As discussed in Section 2, there is a one-to-one mapping between any reward function r and a function Q in the Q-space. Thus, the maximin problem in (4) can be equivalently transformed as:EQUATIONwhere r(s, a) is replaced by T π [Q](s, a) andT π [Q](s, a) = Q(s, a) -γE s ′ [V π (s ′ )], V π (s) = E a∼π(a|s) [Q(s, a) -log π(a|s)]In the context of single-expert-level, [13] demonstrated that the objective function in the Q-space as given in Equation 2 is concave in Q and convex in π, implying that the maximin problem always has a unique saddle point solution. Unfortunately, this property does not hold in our case.Proposition 3.1. H(Q, π) (as defined in Equation 5) is concave in Q but is not convex in π.In general, we can see that the first and second term of (6) are convex in Q, but the reward regularizer term, which can be written asαE ρ U (Q(s, a) -r(s, a) -E s ′ ∼P (.|s,a) E a ′ ∼π(.|s ′ ) (Q(s ′ , a ′ ) -log π(s ′ , a ′ ))) 2 ,is not concave in π (details are shown in the Appendix). The property indicated in Proposition 3.1 implies that the maximin problem within the Q-space max Q min π J(Q, π) may not have a unique saddle point solution and would be more challenging to solve, compared to the original inverse IQ-learn problem.Another key property of Equation 2 is with regards to the inner minimization problem over π, which yields a unique closed-form solution, enabling the transformation of the max-min problem into a non-adversarial concave maximization problem within the Q-space. The closed-form solution was given by π Q = argmax π V π (s) for all s ∈ S. Unfortunately, this result also does not hold with the new objective function in (6) , as formally stated below: Proposition 3.2. H(Q, π) may not necessarily be minimized at π * such that π * = argmax π V π (s), for all s ∈ S.To overcome the above challenges, our approach involves constructing a more tractable objective function that is a lower bound on the objective of (6) . Let us first define Γ(Q) = min π H(Q, π). We then look at the regularization term, which causes all the aforementioned challenges, and write:(T π [Q](s, a))-r(s, a)) 2 = (Q(s, a) -r(s, a) -E s ′ [V π (s ′ )]) 2 = (Q(s, a) -r(s, a)) 2 + (E s ′ [V π (s ′ )]) 2 + 2(r(s, a) -Q(s, a))E s ′ [V π (s ′ )]We then take out the negative part of (r(s, a) -Q(s, a)) using ReLU, and consider a slightly new objective function as follows:EQUATIONLetΓ(Q) = min π H(Q, π)).The proposition below shows that Γ(Q) always lower-bounds Γ(Q).Proposition 3.3. For any Q ≥ 0, we have Γ(Q) ≤ Γ(Q) and max Q Γ(Q) ≤ max Q Γ(Q). Moreover, Γ(Q) = Γ(Q) if Q(s, a) ≤ r(s, a) for all (s, a).We note that assuming Q ≥ 0 is not restrictive, as if the expert's rewards r * (s, a) are non-negative (typically the case), then the true soft-Q function, defined as Q * (s, a) = E[ st,at γ t (r * (s, a)log π(s, a))|(s 0 , a 0 ) = (s, a)], should also be non-negative, for any π. As Γ(Q) provides a lowerbound approximation of Γ(Q), maximizing Γ(Q) over the Q-space would drive Γ(Q) towards its maximum value. It is important to note that, given that the inner problem involves minimization, obtaining an upper-bound approximation function is easier. However, since the outer problem is a maximization one, an upper bound would not be helpful in guiding the resulting solution towards optimal ones. The following theorem indicates that Γ(Q) is more tractable to use.Theorem 3.4. For any Q ≥ 0, the following results hold: (i) The inner minimization problem min π H(Q, π) has a unique optimal solution π Q such that π Q = argmin π V π (s) for all s ∈ S andπ Q (a|s) = exp(Q(s,a)) a exp(Q(s,a)) , (ii) max π V π (s) = log( a exp(Q(s, a))) def = V Q (s), and (iii) Γ(Q) is concave for Q ≥ 0.The above theorem tells us that new objective Γ(Q) has a closed form where V π (s) is replaced byV Q (s). Moreover Γ(Q) is concave for all Q ≥ 0.The concavity is particularly advantageous, as it guarantees that the optimization objective is well-behaved and has a unique solution Q * such that (Q * , π Q * ) form a unique saddle point of max Q min π H(Q, π). Thus, our tractable objective has all the nice properties that the original IQ-Learn objective had, while being able to work for the offline case with multiple levels of expert trajectories and our reward regularizer.Require: (8) 12:(D 1 , D 2 , ..., D N ), (w1, w2, ..., w N ), rη, Q ψ , π θ 1: # estimate reward reference function 2: for iteration i...Ne do 3: d ← (d 1 , d 1 , ..., d N ) ∼ (D 1 , D 2 , ..., D N ) 4: from dataset d, calculate L(rη) by (7) 5: η ← η -∇ηL(rη) 6: end for 7: # train SPRINQL 8: for iteration i...N do 9: d ← (d 1 , d 1 , ..., d N ) ∼ (D 1 , D 2 , ..., D N ) 10: # Update Q function 11: from dataset d, calculate H C (Q ψ , π θ ) byψ ← ψ + ∇ ψ H C (Q ψ , π θ )# Update policy for actor-critic 14:θ ← θ + ∇ E s∼d a∼π θ (a|s) [Q ψ (s, a) -ln(π θ (a|s))]Algorithm 1 provides the overall SPRINQL algorithm. We first estimate the reference rewards in lines 2-6 of Algorithm 1 and the overall process is described in Section 3.4.1. Before we proceed to the overall training, we have to estimate the weights, w i (associated with the ranked demonstration sets) employed in Ĥ(Q, π). We provide a description of this estimation procedure in Section 3.4.2. Finally, to enhance stability and mitigate over-estimation issues commonly encountered in offline Q-learning, we employ a conservative version of Ĥ(Q, π) in lines 8-15 of the algorithm and is described in Section 3.4.3. Some other practical considerations are discussed in the appendix.We outline our approach to automatically infer the reference rewards r(s, a) from the ranked demonstrations. The general idea is to learn a function that assigns higher values to higher expert-level demonstrations. To achieve this, let us define R(τ ) = (s,a)∈τ r(s, a) (i.e., accumulated reward of trajectory τ ). For two trajectories τ i , τ j , let τ i ≺ τ j denote that τ i is lower in quality compared to τ j (i.e., τ i belongs to demonstrations from lower-expert policies, compared to τ j ). We follow the Bradley-Terry model of preferences [4, 5] to model the probability P (τ i ≺ τ j ) asP (τ i ≺ τ j ) = exp(R(τj ))exp(R(τi))+exp(R(τj )) and use the following loss function:min r {L(r) = i∈[N ] (s,a),(s ′ ,a ′ )∈D i (r(s, a) -r(s ′ , a ′ )) 2 - h,k∈[N ],h>k,τi∈D h ,τj ∈D k ln P (τ i ≺ τ j )} (7)where the first term of L(r) serves to guarantee that the reward reference values for (s, a) pairs within the same demonstration group are similar, and the second term aims to increase the likelihood that the accumulated rewards of trajectories adhere to the expert-level order. Importantly, it can be shown below that L(r) is convex in r (Proposition 3.5), making the learning well-behaved. In practice, one can model r(s, a) by a neural network of parameters θ and optimize L(θ) over θ-space. Proposition 3.5. L(r) is strictly convex in r.Each weight parameter w i used in (4) should reflect the quality of the corresponding demonstration set D i , which can be evaluated by estimating the average expert rewards of these sets. Although this information is not directly available in our setting, the reward reference values discussed earlier provide a convenient and natural way to estimate them. This leads to the following formulation for inferring the weights w i from the ranked data:w i = E (s,a)∼D i [r(s,a)] j∈[N ] E (s,a)∼D j [r(s,a)] .Over-estimation is a common issue in offline Q-learning due to out-of-distribution actions and function approximation errors [11] . We also observe this in our IL context. To overcome this issue and enhance stability, we leverage the approach in [23] to enhance our inverse soft-Q learning. The aim is to learn a conservative soft-Q function that lower-bounds its true value. We formulate the conservative inverse soft-Q objective as:EQUATIONwhere µ(a|s) is a particular state-action distribution. We note that in (8) , the conservative term is added to the objective function, while in the conservative Q-learning algorithm [23] , this term is added to the Bellman error objective of each Q-function update. This difference makes the theory developed in [23] not applicable. In Proposition 3.6 below, we show that solving max Q H C (Q) will always yield a Q-function that is a lower bound to the Q function obtained by solving max Q H(Q, π).Proposition 3.6. Let Q = argmax Q H(Q, π) and Q C = argmax Q H C (Q, π), we have s∼D a∼µ(a|s) Q C (s, a) ≤ s∼D a∼µ(a|s) Q(s, a).We can adjust the scale parameter β in Equation 8 to regulate the conservativeness of the objective. Intuitively, if we optimize Q over a lower-bounded Q-space, increasing the scale parameter β will force each Q(s, a) towards its lower bound. Consequently, when β is sufficiently large, Q C will point-wise lower-bound Q, i.e., Q C (s, a) ≤ Q(s, a) for all (s, a). 
experiment setup: 
 Baselines. We compare our SPRINQL with SOTA algorithms in offline IL with sub-optimal demonstrations: TRAIL [38] , DemoDICE [21] , and DWBC [37] . Moreover, we also compare with other straightforward baselines: BC with only sub-optimal datasets (BC-O), BC with only expert data (BC-E), BC with all datasets (BC-both), and BC with fixed weight for each dataset (W-BC), SQIL [29] , and IQ-learn [13] with only expert data. In particular, since TRAIL, DemoDICE, and DWBC were designed to work with only two datasets (one expert and one supplementary), in problems with multiple sub-optimal datasets, we combine all the sub-optimal datasets into one single supplementary dataset. Meanwhile, BC-E, BC-O, and BC-both combine all available data into a large dataset for learning, while W-BC optimizes i Es,a∼D i -w i ln(π(a|s)), where w i are our weight parameters. T-REX [5] is a PPO-based IL algorithm that can work with ranked demonstrations. It is, however, not suitable for our offline setting, so we do not include it in the main comparisons. Nevertheless, we will later conduct an ablation study to compare SPRINQL with an adapted version of T-REX. Moreover, [15] developed IPL for learning from offline preference data. This approach requires comparisons between every pair of trajectories, thus is not suitable for our context.Environments and Data Generation: We test on five Mujoco tasks [32] and four arm manipulation tasks from Panda-gym [12] . The maximum number transition of (s, a) per trajectory is 1000 (or 1k for short) for the Mujoco and is 50 for Panda-gym tasks (descriptions in Appendix B.3). The sub-optimal demonstrations have been generated by randomly adding noise to expert actions and interacting with the environments. We generated large datasets of expert and non-expert demonstrations. For each seed, we randomly sample subsets of demonstrations for testing. This approach allows us to test with different datasets across seeds, rather than using fixed datasets for all seeds as in previous works. More details of these generated databases can be found in the Appendix B.4.The return is normalized by score = return-R.return E.return-R.return × 100 where R.return is the mean return of random policy and E.return is the mean return of the expert policy. The scores are calculated by taking the last ten evaluation scores of each seed, with five seeds per report.Experimental Concerns. Throughout the experiments, we aim to answer the following questions: (Q1) How does SPRINQL perform compared to other baselines? (Q2) How do the distribution matching and reward regularization terms impact the performance of SPRINQL? (Q3) What happens if we augment (or reduce) the expert data while maintaining the sub-optimal datasets? (Q4) What happens if we augment (or reduce) the sub-optimal data while maintaining the expert dataset? (Q5) How does the conservative term help in our approach? (Q6) How does increasing N (the number of expertise levels) affect the performance of SPRINQL? (Q7) Does the preference-based weight learning approach provide good values for the weights w i ? (Q8) How does SPRINQL perform in recovering the ground-truth reward function?In this section, we provide comparison results to answer (Q1) with three datasets (i.e., N = 3). Additional comparison results for N = 2 can be found in the appendix. From expertise levels, we randomly sample (25k-10k-1k) transitions for Mujoco tasks and (10k-5k-100) transitions for Panda-gym tasks for every seed (details of these three-level dataset are provided in Appendix B.4). Table 1 shows comparison results across 3 Mujoco tasks and 3 Panda-gym tasks (the full results for all the nine environments are provided in Appendix C.1). In general, SPRINQL significantly outperforms other baselines on all the tasks.We aim to assess the importance of the distribution matching and reward regularizer terms in our objective (Q2). To this end, we conduct an ablation study comparing SPRINQL with two variants: (i) noReg-SPRINQL, derived by removing the reward regularizer term from (6) , and (ii) noDM-SPRINQL, obtained by removing the distribution matching term from (6) . Here, we note that the noDM-SPRINQL performs Q-learning using the reward reference function, this can viewed as an adaption of the T-REX algorithm [5] to our offline setting. The conservative Q-learning term is employed in the SPRINQL and the two variants to enhance stability. The comparisons for N = 2 and N = 3 on five Mujoco tasks are shown in Figure 1 (the full comparison results for all tasks are provided in the appendix). These results clearly show that SPRINQL outperforms the other variants, indicating the value of both terms in our objective function. (what would happen if we augment the sub-optimal dataset?). Section C.6 experimentally shows the impact of the conservative term in our approaches (i.e., Q5). Section C.7 reports the performance of SPRINQL with varying numbers of expertise levels N (i.e., Q6). Section C.8 addresses Q7, and Section C.10 shows how SPRINQL performs in terms of reward recovering (i.e. Q8). In addition, Section C.11 reports the distributions of the reference rewards and Section C.12 provides α choosing range.Concretely, our extensive experiments reveal the following: (i) SPRINQL outperforms other baselines with two, three, or even larger numbers of datasets; (ii) the conservative term, distribution matching, and reward regularizer terms are essential to our objective-all three significantly contribute to the success of SPRINQL; (iii) the preference-based weight learning provides good estimates for the weights w i ; and (iv) SPRINQL performs well in recovering rewards, showing a high positive correlation with the ground-truth rewards, justifying the use of our method for IRL. 
conclusion and limitations (conclusion): 
 We have developed SPRINQL, a novel non-adversarial inverse soft-Q learning algorithm for offline imitation learning from expert and sub-optimal demonstrations. We have demonstrated that our algorithm possesses several favorable properties, contributing to its well-behaved, stable, and scalable nature. Additionally, we have devised a preference-based loss function to automate the estimation of reward reference values. We have provided extensive experiments based on several benchmark tasks, demonstrating the ability of our SPRINQL algorithm to leverage both expert and non-expert data to achieve superior performance compared to state-of-the-art algorithms.(Limitations) Some limitations of this work include: (i) SPRINQL (and other baselines) still requires a large amount of sub-optimal datasets with well-identified expertise levels to learn effectively, (ii) there is a lack of theoretical investigation on how the sizes of the expert and non-expert datasets affect the performance of Q-learning, which we find challenging to address, and (iii) it lacks a theoretical exploration of how the reward regularizer term enhances the distribution matching term when expert samples are low-this question is relevant and interesting but also challenging to address. These limitations will pave the way for our future work. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The paper provides a general offline imitation learning with multiple expert levels and only testing on the simulated environments. As such, we do not foresee any direct societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our training data are generated from open source simulated environments which have no risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have provided clear citations to the source code and data we used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibility or benchmarking purposes.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: We have no crowdsourcing experiments.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: We do not have study participants.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide proofs for the theoritical results claimed in the main paper.Proposition 3.1 J(Q, π) is concave in Q but is not convex in π.Proof. We recall thatEQUATIONwhereT π [Q](s, a)) = Q(s, a) -γE s ′ ∼P (s ′ |s,a) [V π (s ′ )] and V π (s) = E a∼π(a|s) [Q(s, a) - log π(a|s)]. We see that T π [Q](s, a)is linear in Q for any (s, a). Thus, the first and second terms of J(Q, π) in ( 9) are linear in Q. The last term of ( 9) involves a sum of squares of linear functions of Q, which are convex. So, J(Q, π) is concave in Q.To see that J(Q, π) is generally not convex in π, we will consider a quadratic component of the reward regularization term (T π [Q](s, a)) -r(s, a)) 2 and show that there is an instance of Q and r values that makes this term convex. We first write:T π [Q](s, a)) -r(s, a) = Q(s, a) -γ s ′ P (s ′ |s, a)V π (s ′ ) -r(s, a) = Q(s, a) -γ s ′ P (s ′ |s, a) a ′′ π(a ′′ |s ′ )(Q(s ′ , a ′′ ) -log π(a ′′ |s ′ )) -r(s, a)For simplification, let us choose Q(s ′ , a ′′ ) = 0 for all s ′ such that P (s ′ |s, a) > 0. This allows us to simplify T π [Q](s, a)) -r(s, a) asT π [Q](s, a)) -r(s, a) = γ s ′ P (s ′ |s, a) a ′′ π(a ′′ |s ′ ) log π(a ′′ |s ′ ) + Q(s, a) -r(s, a)We further see that, for any s ∈ S, a∈A π(a|s) log π(a|s) achieves its minimum value at π(a|s) = 1/|A| for all a ∈ |A|, and a∈A π(a|s) log π(a|s) ≥ log 1/|A| for any policy π. As a result we have: γs ′ P (s ′ |s, a) a ′′ π(a ′′ |s ′ ) log π(a ′′ |s ′ ) ≥ γ log 1 |A| So if we select Q(s, a) such that Q(s, a) -r(s, a) -γ log |A| ≥ 0, then T π [Q](s, a)) -r(s, a) ≥ 0 for any π. Now we consider the quadratic function Γ(π) = (λ(π)) 2 where λ(π) = T π [Q](s, a)) - r(s, a). Since each term π(a ′′ ) log π(a ′′ |s ′ ) is convex in π, λ(π) is convex in π.To show Γ(π) is convex in π, we will show that for any two policies π 1 , π 2 and α ∈ [0, 1], Γ(απ 1 + (1 -α)π 2 ) ≤ αΓ(π 1 ) + (1 -α)Γ(π). To this end, we writeαΓ(π 1 ) + (1 -α)Γ(π) (a) ≥ (αλ(π 1 ) + (1 -α)λ(π 2 )) 2 (b) ≥ (λ(απ 1 + (1 -α)π 2 )) 2 = Γ(απ 1 + (1 -α)π 2 )where (a) is because the function h(t) = t 2 is convex in t, and (b) is because(i) αλ(π 1 ) + (1 -α)λ(π 2 ) ≥ λ(απ 1 + (1 -α)π 2 ) (as λ(π) is convex in π) (ii) αλ(π 1 ) + (1 -α)λ(π 2) and λ(απ 1 + (1 -α)π 2 ) are both non-negative, and function h(t) = t 2 is increasing for all t ≥ 0.So, we see that with the Q values chosen above, function(T π [Q](s, a)) -r(s, a)) 2 ) is convex and -α(T π [Q](s, a) -r(s, a)) 2 is concave. So, intuitively, when α is sufficiently large, J(Q, π) would be almost concave (so not convex), which is the desired result.Proposition 3.2 J(Q, π) may not necessarily be minimized at π Q such that π Q = argmax π V π (s), for all s ∈ S.Proof. We first write J(Q, π) asEQUATIONWe then see that the termsE ρ i [Q(s, a) -γE s ′ V π (s ′ )]and -γE s0 V π (s 0 ) are minimized (over π) when V π (s), for all s, are maximized. We will prove that it would not be the case for the last term.Let us choose Q and r such that Q(s, a) -r(s, a) > γE s ′ V π Q (s ′ ). We see that for any policy π ̸ = π Q , we haveQ(s, a) -r(s, a) > γE s ′ V π Q (s ′ ) ≥ E s ′ V π (s ′ ) Thus, Q(s, a) -r(s, a) -V π (s ′ ) ≥ Q(s, a) -r(s, a) -V π Q (s ′ ) > 0 which implies that -αE ρ U [(Q(s, a) -r(s, a) -γE s ′ V π (s ′ )) 2 ] ≤ -αE ρ U [(Q(s, a) -r(s, a) -γE s ′ V π Q (s ′ )) 2 ]So the last term of (10) would not be minimized at π = π Q . In fact, in the above scenario, this last term will be maximized at π = π Q . As a result, there is always α sufficiently large such that the last term significantly dominates the other terms andJ(Q, π) is not minimized at π = π Q . Proposition 3.3 For any Q ≥ 0, we have Γ(Q) ≤ Γ(Q) and max Q≥0 Γ(Q) ≤ max Q≥0 Γ(Q). Mover, Γ(Q) = Γ(Q) if Q(s, a) ≤ r(s, a)for all s, a.Proof. We first write H(Q, π) asEQUATIONSinceQ ≥ 0, V π (s) = E a∼π(.|s) [Q(s, a) -log π(a|s)] ≥ 0. Thus 2ReLU(r(s, a) - Q(s, a))E s ′ V π (s) ≥ 2(r(s, a) -Q(s, a))E s ′ V π (s).As a result, the last term of H(Q, π) is bounded as-αE ρ U (Q(s, a) -r(s, a)) 2 + (E s ′ V π (s ′ )) 2 + 2ReLU(r(s, a) -Q(s, a))E s ′ V π (s) ≤ -αE ρ U (Q(s, a) -r(s, a)) 2 + (E s ′ V π (s ′ )) 2 -2(Q(s, a) -r(s, a))E s ′ V π (s) = -αE ρ U [(T π [Q](s, a) -r(s, a)) 2 ]It then follows thatH(Q, π) ≤ J(Q, π). Thus, min π H(Q, π) ≤ min π J(Q, π) or Γ(Q) ≤ Γ(Q).Mover, we see that if r(s, a) ≥ Q(s, a) for all (s, a), then 2ReLU(r(s, a)-Q(s, a))E s ′ V π (s) = 2(r(s, a)-Q(s, a))E s ′ V π (s), implying that H(Q, π) = J(Q, π) and Γ(Q) = Γ(Q).This completes the proof.Theorem 3.4 For any Q ≥ 0, the following results hold (i) The inner minimization problem min π H(Q, π) has a unique optimal solution π * such that π Q = argmin π V π (s) for all s ∈ S andπ Q (a|s) = exp(Q(s, a)) a exp(Q(s, a)) . (ii) max π V π (s) = log( a exp(Q(s, a))) def = V Q (s). (iii) Γ(Q) is concave for Q ≥ 0Proof. We first rewrite the formulation of H(Q, π) asH(Q, π) = i∈[N ] w i E ρ i [Q(s, a) -γE s ′ V π (s ′ )] -(1 -γ)E s0 V π (s 0 ) -αE ρ U (Q(s, a) -r(s, a)) 2 + (E s ′ V π (s ′ )) 2 + 2ReLU(r(s, a) -Q(s, a))E s ′ V π (s)We then see that the first and second term of H(Q, π) are minimized when V π (s) are minimized, i.e., at π = π Q . For the last term, sinceV π (s) ≥ 0 (because Q ≥ 0), -(E s ′ V π (s ′ )) 2 and -2ReLU(r(s, a) -Q(s, a))E s ′ V π (s) are also minimized at π = π Q . So, H(Q, π) is minimized at π = π Q as desired.(ii) is already proved in [13] .For (iii), we rewrite H(Q, π) asEQUATIONThen, the first and second terms of ( 12) is linear in Q. The fourth term is concave. For third term, letEQUATIONwhere (a) is due to the fact function h(t) = t 2 is convex, and (b) is because h(t) = t 2 is nondecreasing for all t ≥ 0, and Φ(Q) is convex and always takes non-negative values. The last inequality in (13) implies that (Φ(Q)) 2 is convex in Q. So the last term of ( 12) is concave in Q.Putting all together we conclude that H(Q, π) is concave in Q as desired.Proposition 3.5 L(r) is strictly convex in r.Proof. We first write L(r) asEQUATIONEQUATIONWe then see that the first term is a sum of squares of linear functions of r, thus is strictly convex. Moreover, since R(τ i ) is linear in r for any τ i , the term ln(exp(R(τ i )) + exp(R(τ j ))) has a logsum-exp form. So this term is convex as well [3] . Putting all together we see that L(r) is strictly convex in r as desired.Proposition 3.6 Let Q = argmax Q H(Q, π) and Q C = argmax Q H C (Q, π), we have s∼D a∼µ(a|s) Q C (s, a) ≤ s∼D a∼µ(a|s) Q(s, a)Proof. We writeEQUATION≥ -β s∼D a∼µ(a|s)EQUATIONwhere (a) is because EQUATIONWe discuss a practical implementation of the training objective in (4) using samples from both the expert and sub-optimal demonstration sets. We first note that the term E ρπ [T π [Q](s, a))] -E ρπ [log π(s, a)] can be written as [13] :E ρπ [T π [Q](s, a))]-E ρπ [log π(s, a)] = E (s,s ′ )∼ρ * [V π (s) -γE s ′ V π (s ′ )]for any valid occupancy measure ρ * . So, for any policy occupancy measure, we can write this term though the union of expert policies ρ U asEQUATIONAs a result, by replacing V π (s) with V Q (s), we can write Γ(Q) in a compact form asEQUATIONwhere∆ Q r (s, a) = (Q(s, a) -r(s, a)) 2 + (E s ′ V Q (s ′ )) 2 + 2ReLU(r(s, a) -Q(s, a))E s ′ V Q (s ′ ).In an empirical offline implementation, to maximize Γ(Q), samples (s, a, s ′ ) from demonstrations can be used to approximate the expectations over ρ i :(s,a,s′ )∈D i [Q(s, a) -γV Q (s ′ )], (s,a,s ′ )∈D i [V Q (s) -γV Q (s ′ )], and (s,a)∈D i [∆ Q r (s, a)].We note that in continuous-action controls, the computation of V Q (s) involves a sum over infinitely many actions, which is impractical. In this case, we can update both Q and π in a soft actor-critic (SAC) manner. That is, for each π, we update Q towards max Q { H(Q, π)} and for each Q, we update π to bring it towards π Q by solving max π {V π (s)}, for all s. As shown above, H(Q, π) in concave in Q and V π (s) is convex in π, so we can expect this SAC will exhibit good behavior and stability.The Figure 2 provide the overview of our algorithm. This section provide a detailed descrition of the enviroments used in our experiments.MuJoCo gym environments [32] like HalfCheetah, Ant, Walker2d, Hopper, and Humanoid are integral to the field of reinforcement learning (RL), particularly in the domain of continuous control and robotics:• HalfCheetah: This environment simulates a two-dimensional cheetah-like robot. The objective is to make the cheetah run as fast as possible, which involves learning complex, coordinated movements across its body.• Ant: This environment features a four-legged robot resembling an ant. The challenge is to control the robot to move effectively, balancing stability and speed.• Walker2d: This environment simulates a two-dimensional bipedal robot. The goal is to make the robot walk forward as fast as possible without falling over.• Hopper: The Hopper environment involves a single-legged robot. The primary challenge is to balance and hop forward continuously, which requires maintaining stability while in motion.• Humanoid: The Humanoid environment is among the most complex, featuring a bipedal robot with a human-like structure. The task involves mastering various movements, from walking to more complex maneuvers, while maintaining balance.An expert is trying to maximize the reward function by moving with a trajectory length limit of 1000. All five environments are shown in Figure 3 . Walker Hopper HumanoidThe Panda-gym environments [12] , designed for the Franka Emika Panda robot in reinforcement learning, include PandaReach, PandaPush, PandaPickandPlace, and PandaSlide. Here's a short description of each:• PandaReach: The task is to move the robot's gripper to a randomly generated target position within a specific volume. It focuses on precise control of the gripper's movement.• PandaPush: In this environment, a cube placed on a table must be pushed to a target position. The gripper remains closed, emphasizing the robot's ability to manipulate objects through pushing actions.• PandaPickandPlace: This more complex task involves picking up a cube and placing it at a target position above the table. It requires coordinated control of the robot's gripper for both lifting and accurate placement.• PandaSlide: Here, the robot must slide a flat object (like a hockey puck) to a target position on a table. The gripper is fixed in a closed position, and the task demands imparting the right amount of force to slide the object to the target.In these environments, the reward function is -1 for every time step it has not finished the task. Moreover, the maximum horizon is extremely short, with a maximum of 50, while an expert can complete the task after several steps. All four different environments are shown in Figure 4 . PnP SlideIn this paper, we provide a new setting utilizing ranked sub-optimal datasets to perform imitation learning in the offline setting (illustration in Figure 5 ).Ranked sub-optimal datasets 1 (expert) 2 3 N We create sub-optimal datasets by adding noise to actions of the expert policy and let them interact with the environments to collect the sub-optimal trajectories. Each task has its own difficulty and sensitivity to the noise of the actions. The averaged returns of the generated datasets, computed as percentages w.r.t. the maximum returns, are reported in Figure 6 . • In our experiments, for every algorithm, we run five different seeds corresponding to five different datasets sampled from our databases in Appendix B.4. • We use double Q critic network for our implementation to increase the stability in the offline training scheme. • For IQ-learn [13] ,SQIL [29] , we conduct experiment from its official implement with double Q critic network. • For DemoDICE [21] we conduct experiment from its official implementation.• For DWBC [37] we conduct experiment from its official implementation.• Inspired by double Q-learning, to avoid overfitting in the offline setting, some experiments apply a training trick using KL-divergence between the target actor and the training actor to prevent rapid policy changes. In this section, we present additional experiments that complement those reported in the main paper, along with ablation studies to address the experimental questions stated therein.We report the full results for the 5 different Mujoco environments and 4 different Panda-gym environments, with 3 datasets (i.e., N = 3), supplementing the results reported in Table 1 in the main paper. The detailed results for Mujoco in Table 3 and Panda-gym in We provide additional comparison results for N = 2, complementing to the answer of (Q1), i.e., how does SPRINQL perform compared to other baselines? In this experiment, we want to test the ability of our algorithm in the same scenario of DemoDICE, DWBC which include expert dataset and only one supplementary dataset. Number of expert transitions in Mujoco domains is 1000 and 100 for Panda-gym. Meanwhile, for the supplementary dataset, it is 25000 transitions for Mujoco and 5000 for Panda-gym. In general, although we experience a downgrade in performance due to lack of ranking (only one supplementary dataset), leading to misunderstanding the true reward function, our method is still able to leverage the sub-optimal dataset for understanding the expert demonstrations and provide the highest average score. Reported results are shown in Table 5 This section reports additional results for the comparison between SPRINQL and the two variants noReg-SPRINQL and noDM-SPRINQL (supplementing the answer of Q2 in the main paper). The comparison results for Panda-gym environments are reported in Figure 8 and the learning curves are plotted in Figure 9 and Figure 10 . We provide experiments to address (Q3) -What happens if we augment the expert data while maintaining the sub-optimal datasets? To this end, we use two Mujoco and two Panda-gym tasks, keeping the same sup-optimal datasets and add more expert demonstrations to the training sets. The comparison results are reported in Figure 11 . For the Mujoco tasks, which are more difficult, adding more expert trajectories significantly enhances the performance of all the algorithms. However, for the two Panda-gym tasks, the influence of adding more expert data appears to be less significant in Push and completely absent in PnP. This would be because expert trajectories in these tasks are typically short, consisting of only 2-7 transitions. Hence, a larger quantity of additional expert data may be required to enhance performance. Figure 12 : Performance comparisons with varied sub-optimal data sizes. The x-axis shows the size of Level 2 and 3 sup-optimal datasets, and y-axis shows the scores.In this experiment, we want to answer the question (Q4) -What happens if we augment (or reduce) the sub-optimal data while maintaining the expert dataset?. We present numerical results to assess the impact of sub-optimal data on the performance of our SPRINQL and other baselines. To this end, we keep the same expert dataset and adjust the non-expert data used in Table 1 . The performance is reported in Fig. 12 . It is evident that reducing the amount of data in sub-optimal datasets can result in a significant degradation in the performance of our SPRINQL and the other baselines. Conversely, adding more sub-optimal data can enhance overall stability and lead to improved performance.In the Figure 13 we show the learning curves with varied sizes for sub-optimal datasets. It can be seem that the overall performance tends to improve with more sup-optimal datasets. In particular, for the Ant task, our SPRINQL even fails to maintain stability when the sizes of sup-optimal datasets are low (10k-5k-1k). Moreover, while BC seems to show improvement with more sub-optimal data, the performance of DWBC and DemoDICE remains unchanged. This may be because these approaches rely on the assumption that expert demonstrations can be extracted from the sub-optimal data, which is not the case in our context. In this experiment, we aim to answer (Q5) -How does the conservative term help in our approach?The Equation 8 introduce the conservative Q learning (CQL) term into our work. Here we also test three variants of our method from Section 4. We provide an experiment to answer (Q6) -How does increasing N (the number of expertise levels) affect the performance of SPRINQL? We assess our algorithm with different numbers of expertise levels to investigate how adding or removing sub-optimal expertise levels influences performance. Specifically, we keep the same expert dataset comprising 1 expert trajectory and conduct tests with 1 to 4 sub-optimal datasets from the Cheetah task. Details of the average returns of the five datasets are reported in Fig. 15 . In this context, SPRINQL outperforms other algorithms in utilizing non-expertIn this experiment, we want to answer (Q8) -How does SPRINQL perform in recovering the groundtruth reward function? Compared to BC-based algorithms, one notable advantage of Q-learning based algorithms is their ability to recover the reward function. Here, we present experiments demonstrating reward function recovery across five MuJoCo tasks, comparing recovered rewards to the actual reward function. To achieve this, we introduce increasing levels of random noise to the actions of a trained agent and observe its interactions with the environment. We collect the state, action, and next state for each trajectory, then predict the recovered reward and compare it to the true reward from the environment. For the sake of comparison, we include noReg-SPRINQL, which can be considered an an adaption of IQ-learn [13] to our setting, and noDM-SPRINQL, which is in fact an adaption of T-REX to our offline setting.Comparison results are presented in Figure 19 . We observe a linear relationship between the true and predicted rewards for SPRINQL across all testing tasks, whereas the other approaches fail to return correct relationships for some tasks. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We have describe how to generate our data as well as provide it along with our submitted source code with sufficient instructions for their use.• The answer NA means that paper does not include experiments requiring code. Justification: We have provided these information in the "Hyper parameter and Experimental Implementations" section in our appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).38th Conference on Neural Information Processing Systems (NeurIPS 2024).While there have been works[5,6,7] that have attempted to minimize the required demonstrations using ranked datasets by preference-based RL, their algorithms are only applicable to online settings. 


Decision: This paper proposes an offline imitation learning (IL) method that leverages both expert and suboptimal demonstrations. This method requires access to suboptimal demonstrations across a spectrum of performance levels, with a known ranking to the learners. Experiments  reported in the paper show that the proposed solution outperforms prior methods on suboptimal trajectories in simulated environments. Ablations justify each component in the proposed solution.

Reviews: review: paper_summary: paper_summary: This work presents an offline imitation learning (IL) method that leverages both expert and suboptimal demonstrations. Distinct from previous methods, this approach assumes access to suboptimal demonstrations across a spectrum of performance levels, with a known ranking to the learners.
To utilizes suboptimal demonstations with various levels, this work suggest SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse Q-Learning), a reward reguarized inverse soft Q-learning method for the offline IL problem.
The evaluation of this method is conducted using the Mujoco and Panda-gym environments, employing a small dataset of expert demonstrations and a larger collection of demonstrations generated by introducing varying levels of noise to expert actions.
strength_weakness: strength_weakness: Significant concerns with this submission are the questionable reporting on the performance of baseline methods and issues related to reproducibility. 
The experimental results presented are not entirely convincing, as the performance of the baselines substantially deviates from those reported in their original papers. 
Specifically, even in the case with N=2 suboptimality levels, suitable baselines such as TRAIL, DemoDICE, and DWBC fail to outperform even basic BC-all in nearly all the Mujoco tasks listed in Table 3 (Appendix). 
The authors need to clarify the reasons behind this anomaly.
To enhance the trustfulness of experiments, it would be beneficial for the authors to release the datasets utilized in their experiments or to use standardized datasets such as D4RL or RL-unplugged. This would allow for a more reliable validation of the results and facilitate comparisons with other methods.
questions: questions: - Could the authors clarify the motivation of employing reference rewards?
- Moreover, could you elucidate why reference rewards are utilized as a regularization mechanism rather than directly applying RL using reference rewards?
- The problem setting appears closely aligned with preference-based learning, where optimality (or preference) levels are explicitly known. Could you delineate the distinctions between these two problem settings?
- Can you provide detailed justifications for why comparing the proposed method with IPL is deemed unsuitable? 
It appears feasible to compare by segmenting each trajectory by suboptimality levels, then pairing segments from differing levels to facilitate IPL. Could this approach not be applied effectively?

[minor comments] reference [15] in line 398 from "NeurIPS 2024" to "NeurIPS 2023."
limitations: limitations: The authors addressed their limitations in Conclusion section in the main manuscript. They described that this work does not have any direct societal impacts in the Checklist #10, Broader Impacts.
paper_summary: paper_summary: This paper presents SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning), a novel algorithm for offline imitation learning with multiple levels of expert demonstrations. SPRINQL leverages distribution matching and reward regularization to effectively leverage the mixture of non-expert data and expert data to enhance learning. An extensive evaluation is conducted to assess SPRINQL, answering eight well-defined questions regarding the capability and performance of SPRINQL.
strength_weakness: strength_weakness: - The authors mention that this setting (having several sets of data labeled with relative quality scores) is more general than two quality levels: expert and sub-optimal. Could the authors provide further support for this claim? I would argue that having additional quality annotations makes this setting more niche.
- Table 1 contains a very large evaluation but there isn't much commentary on the different trends between baselines and SPRINQL.
Comment: The example in the introduction does not seem complete.
questions: questions: Please address the weaknesses above.
limitations: limitations: Yes
paper_summary: paper_summary: This paper considers the problem of imitation learning on suboptimal demonstrations given coarse trajectory expertise rankings. It introduces SPRINQL, a method combining inverse soft-Q learning, conservative Q learning, and reward learning from ranked trajectory preferences. SPRINQL estimates a reference reward function given trajectory expertise rankings, and extends inverse soft-Q learning to the suboptimal demonstration setting by adding reward regularization and weights based on the estimated reference reward. The paper shows that the SPRINQL objective is tractable with a unique solution over $Q, \pi$. Experiments show that SPRINQL outperforms prior methods on suboptimal trajectories in simulated environments. Ablations justify each component in SPRINQL and demonstrate robustness across a range of dataset sizes and number of suboptimal datasets.
strength_weakness: strength_weakness: - Suboptimal trajectories are sourced from adding random noise to expert actions. While this is a reasonable proxy, it would be more convincing to see them generated from under-trained RL policies instead.
- It would be great to see SPRINQL work on image-based environments.
questions: questions: - How does SPRINQL perform on suboptimal trajectories generated from under-trained RL policies, instead of rollouts from noisy expert actions?
limitations: limitations: - SPRINQL assumes access to trajectory expertise rankings.
paper_summary: paper_summary: This paper considers the problem of offline imitation learning from sub-optimal demonstrations. The paper proposes the SPRINQL algorithm, which automatically weights the demonstrations, regularizes the loss with a reference reward function, and then performs inverse soft-Q learning. Experimental results show that the proposed method achieves better performance compared with baselines. Extensive ablations are also provided.
strength_weakness: strength_weakness: 1. Some parts of the proposed approach are unclear to me. Please see questions 1-4. 

2. The influence of the reward regularization term is not completely studied. Please see question 5.
questions: questions: 1. In Equation (3), a modified objective is presented. However, it seems that this objective depends largely on the quality of weights, and it does not necessarily recover the ground truth reward function nor the optimal policy. Given that the weight is chosen with a heuristic, what kind of guarantee can the authors give about the converged policy?

2. Following the first question, in the experiments, seems that the proposed algorithm always outperforms BC-E because of the lack of expert demonstrations. However, I am curious to see what will happen given sufficient expert demonstrations. 

3. In equations above line 162, what if $(s, a) = (s', a')$, given that it is possible for two stochastic policies to choose the same action in the same state?

4. Line 173, regarding "prior research", could the authors add some citations?

5. It seems that the weight for reward regularization $\\alpha$ is an important hyperparameter. Will the performance of the proposed method change a lot with different $\\alpha$? In the extreme case, what if we learn a policy with RL using reward function $\\bar r$?
limitations: limitations: The authors discuss the limitations adequately in the last Section.

Comments: comment: Thank you for your feedback. We would like to clarify that when there is sufficient expert data available: 

- Achieving expert-level performance consistently with our approach may not always be possible, as part of our objective involves learning from sub-optimal data. 
- Methods such as DWBC, Demo-DICE, or our proposed approach are unnecessary. Pure offline imitation learning techniques, like BC or IQ-Learn, should be preferred.  

Below, we provide a quick set of results on the Cheetah and Ant environments, including BC-E for comparison. 

| |  | Cheetah |  | | | Ant | | |
|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
|   Expert Transitions   |   1k   |   5k   |   10k   |   25k   |   1k   |   5k   |   10k   |   25k   |
|   BC-E   |   -3.2 $\pm$ 0.9    |   1.9 $\pm$ 3.0    |   44.5 $\pm$ 10.6    |   79.8 $\pm$ 6.2    |   6.4 $\pm$ 19.1    |   9.0 $\pm$ 6.6  |  50.5 $\pm$ 7.2   |   86.7 $\pm$ 3.9    |
| DemoDICE    |   0.4 $\pm$ 2.0  | 12.2 $\pm$ 5.2  | 40.0 $\pm$ 10.5  | 71.5 $\pm$ 10.8  | 31.7 $\pm$ 8.9  | 79.4 $\pm$ 6.6  | 81.7 $\pm$ 5.2  | 84.0 $\pm$ 4.9 |
| SPRINQL    | 73.6 $\pm$ 4.3  | 86.7 $\pm$ 3.5  | 85.9 $\pm$ 4.1  | 88.6 $\pm$ 3.4  | 77.0 $\pm$ 5.6  | 88.6 $\pm$ 4.3  | 96.6 $\pm$ 8.3  | 96.7 $\pm$ 9.5  |


*We hope the above response resolves your remaining concerns.*
Thank the authors for their reply! However, these experimental results do not reply directly to my question. My question was whether, with sufficient expert demonstrations, could the proposed algorithm recover the expert policy or not. Could the authors please clarify this, or provide the comparison with BC-E with a different number of expert transitions?
Thank you to the reviewer for carefully reading our responses, offering further feedback, and maintaining a positive view of our paper.

We would like to clarify that, in our context, "sufficient" expert data refers to the quantity required by offline RL methods (e.g., BC, IQ Learn) that rely exclusively on expert data to achieve near-expert performance. Conversely, "insufficient" data would mean that these offline RL methods are unable to effectively learn using only expert demonstrations. These thresholds can vary depending on the environment.

In our main experiments, we intentionally kept the amount of expert demonstrations "insufficient" (otherwise, standard imitation methods like IQ-Learn or BC could perform well with just expert data) to demonstrate how our method can leverage suboptimal data to support learning when expert data alone is insufficient.

To further support our response, we conducted a few additional experiments (see table below) where the number of expert demonstrations was increased from "insufficient" to "sufficient". The results generally show that our method performs well with a significantly smaller number of expert demonstrations, remains competitive when the number of demonstrations is sufficient, and can achieve expert-level performance in the Ant environment.


|||Cheetah||||Ant|||
|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|
| Expert Transitions    | 1k   | 5k   | 10k   | 25k   | 1k   | 5k   | 10k   | 25k   |
| DemoDICE    |   0.4 $\pm$ 2.0  | 12.2 $\pm$ 5.2  | 40.0 $\pm$ 10.5  | 71.5 $\pm$ 10.8  | 31.7 $\pm$ 8.9  | 79.4 $\pm$ 6.6  | 81.7 $\pm$ 5.2  | 84.0 $\pm$ 4.9 |
| SPRINQL    | 73.6 $\pm$ 4.3  | 86.7 $\pm$ 3.5  | 85.9 $\pm$ 4.1  | 88.6 $\pm$ 3.4  | 77.0 $\pm$ 5.6  | 88.6 $\pm$ 4.3  | 96.6 $\pm$ 8.3  | 96.7 $\pm$ 9.5  |

*We hope the responses above help address the remaining concerns you may have!*
Thank you to the reviewer for reading our responses and maintaining a positive view on our paper!
Thank you to the reviewer for reading our responses and maintaining a positive view of our work!
We are grateful to the reviewer for reading our responses and for the positive evaluation of our work. We will update our paper, incorporating your constructive feedback and suggestions
Thank you for the additional experiments. Overall I think this is a good paper and I would like to keep my score of Accept.
Dear Authors,

Thank you for your response. After reading all the reviews and their respective replies, I have decided to maintain my score of Accept.
Thank you for addressing my concerns and providing thoughtful comments. 

After reviewing your responses and the additional experimental results on D4RL, I find the results more convincing, even though the improvement over BC-all is slightly marginal in this context. Despite this, I believe the initial experimental setup proposed by the authors more effectively demonstrates the advantages of the suggested method. 

Hence, I have raised my assessment and am now leaning slightly towards accepting this paper.
I want to thank the authors for their reply. The reply addresses my concerns, and the additional experiments provided are informative. However, one of my remaining concerns is that even with sufficient expert demonstrations, seems the proposed algorithm still cannot recover the expert policy. The authors claim that the proposed algorithm works better when "the amount of expert data is limited", but it is difficult to quantify how much data is "limited" and how much is "sufficient". Overall, I am positive about the paper, and I am keeping my original score.