SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning
Authors: Huy Hoang
Tien Anh Mai
Pradeep Varakantham
Venue: NeurIPS 2024 poster (2024)
Keywords: imitation learning
offline imitation learning
reference reward
supplementary data
ranked dataset
Primary area: reinforcement_learning
TL;DR: We develop a novel inverse soft-Q learning for offline imitation learning with expert and non-expert demonstrations.

Abstract: We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. 

In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL .

SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning
Huy Hoang
Tien Mai
Singapore Management University hoang. 2024@phdcs smu, sg
Singapore Management University atmai0smu edu.
Pradeep Varakantham Singapore Management University pradeepv@smu. edu.sg
Abstract
We focus offline imitation learning (IL), which aims mimic expert's behav- ior using demonstrations without any interaction with the environment. One of the main challenges in offline is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible obtain numerous expert demonstrations, it is often possible gather a larger set of sub-optimal demonstrations. For example, in treatment optimiza- tion problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency.
this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance aligning with sub-optimal ones. A key contribution of our approach, called SPRINQI, is transforming the offline IL problem into convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https//giitubbcomm/mmmyySSSS
Introduction
Reinforcement learning (RL) has established itself as a strong and reliable framework for sequential
decision-making with applications in diverse domains: robotics [19] 18) healthcare (34 28 27,, and
environment generation 9 224). Unfortunately, RL requires an underlying simulator that can provide rewards for different experiences, which is usually not available.
Imitation Learning (IL)
handles the lack of reward function by utilizing expert demonstrations to guide the learning scheme to compute a good policy. However, IL approaches still require the presence of simulator that allows for ouline interactions. Initial works in Offline IL
[6 29
23 40 2 tackle the absence of simulator by considering an offline dataset of expert
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

demonstrations. These approaches extend upon Behavioral Cloning (BC), where we aim to maximize the likelihood of the expert's 's decisions from the provided dataset. The key advantage with BC the theoretical justification on converging to expert behaviors given sufficient trajectories. However, when there are not enough expert trajectories, it often suffers from distributional shift issues [30- Thus, a key drawback of these initial II. approaches is the need for a large number of expert demonstration datasets.
To deal with limited expert demonstrations, recent works utilize non-expert demonstration datasets to reduce the reliance on only expert demonstrations. These additional non-expert demonstrations are referred to as supplementary data. Directly applying BC to these larger supplementary datasets will lead to sub-optimal policies, so most prior work in utilizing supplementary data attempts to extract expert-like demonstrations from the supplementary dataset in order to expand the expert
37 39). These works assume expert-like demonstrations are present in the supplementary dataset and focus on identifying and utilizing those, while eliminating the non-expert demonstrations. Eliminating non-expert trajectories can result loss of key information (e.g., transition dynamics) about the environment. Additionally, these works primarily rely on BC, which is known to overlook the sequential nature decision-makinn problems - a small error can quickly accumulate when the learned policy deviates from the states experienced by the expert.
demonstrations 31 21
20
We develop our algorithm based on an inverse Q-learning framework that better captures the sequential nature of the decision decision-making [13] and can operate under the more realistic assumption that the data collected from people/policies with lower expertise level." (not experts). To illustrate, consider scenario in robotic manipulation where the goal is to teach a robot to assemble parts. Expert demonstrations might show precise and efficient methods to assemble parts, but are limited in number due to the high cost and time associated with expert involvement. On the other hand, sub-optimal demonstrations from novice users are easier to obtain and more abundant. Our SPRINQL approach effectively integrates these sub-optimal demonstrations, giving appropriate weight to the expert demonstrations to ensure the robot learns the optimal assembly method without overfitting to the limited expert data or the inaccuracies in the sub-optimal data. We utilize these non-expert trajectories to learn a Q function that contributes to our understanding of the environment and the ground truth reward function.
Contributions:
Overall, we make the following key contributions in this paper: (i We propose SPRINQL,, novel algorithm based on Q-learning for offline imitation learning with expert and multiple levels of sub-optimal demonstrations.
(ii) We provide key theoretical properties of the SPRINQL objective function, which enable the development of scalable and efficient approach. In particular, we leverage distribution matching and reward regularization to develop an objective function for SPRINQL that not only help address the issue of limited expert samples but also utilizes non-expert data to enhance learning. Our objective function is not only convex within the space of Q functions but also guarantees the return of Q function that lower-bounds its true value.
(iii We provide an extensive empirical evaluation of our approach in comparison to existing best algorithms for offline IL. with sup-optimal demonstrations. Our algorithms provide state-of-the-ar (SOTA) performance on all the benchmark problems. Moreover, SPRINQL is able to recover reward function that shows a high positive correlation with the ground-truth rewards, highlighting a unique advantage of our approach compared to other IL algorithms in this context.
1.1 Related Work
Imitation Learning. Imitation learning is recognized as a significant technique for learning from demonstrations. It begins with BC, which aims to maximize the likelihood of expert demonstrations. However, BC often under-performs in practice due to unforeseen scenarios [30] To overcome this
limitation, Generative Adversarial Imitation Learning (GAIL) [66] and Adversarial Inverse Reinforce- ment Learning (AIRL) [10 have been developed. These methods align the occupancy distributions of the policy and the expert within the Generative Adversarial Network (GAN) framework [14). Alternatively, Soft Q Imitation Learning (SQIL) [29] bypasses the complexities of adversarial training by assigning a reward of +1 to expert demonstrations and 0 to the others, subsequently learning a value function based on these rewards. While the aforementioned imitation learning algorithms show
While there have been works 5 6 7 that have attempted to minimize the required demonstrations using ranked datasets by seseeenceebaseed RL, their algorithms are only applicable to online settings.

---

promise, they require interaction with the environment to obtain the policy distribution, which is often impractical.
Offiine Imitation Learning. ValueDICE [22 introduces a novel approach for off-policy training,
suitable for offline training, using Stationary Distribution Corrections [25 26]. However, VaIu necessitates adversarial training between the policy network and the Q network, which can make the
training slow and unstable. Recently, algorithms like PWIL [8] and IQ-learn [13 have optimized distribution distance, offering an alternative to adversarial training schemes. Since such approaches rely occupancy distribution matching, large expert dataset is often required achieve the desired performance. Our approach, SPRINQL is able to bypass this requirement of a large set of expert demonstrations through the use of non-expert demonstrations (which are typically more available) in conjunction with a small set of expert demonstrations.
Imitation Learning with Imperfect Demonstrations. T-REX S and D-REX 6 have shown that utilizing noise-ranked demonstrations as a reference-based approach can return a better policy without requiring expert demonstrations in online settings. Moreover, there are also several works [36 33
that utilize the GAN framework [14] for sub-optimal datasets and have achieved several successes. Meanwhile, in the offline imitation learning context, TRAIL 38 utilizes sup-optimal demonstrations to learn the environment's dynamics. employs a feature encoder to map the high-dimensional state-action space into lower dimension, thereby allowing for scalable way of learning of dynamics. This approach may face challenges in complex environments where predicting dynamics accurately is difficult, as shown in experimental results. Other works assume that they can extract expert-like state-actions pairs from the sub-optimal demonstration set and use them for BC with importance
21 20) However, expert-like state-actions might be difficult to accurately identify, as true reward information is not available. In contrast, our approach is more general, as we do not assume that the sub-optimal set contains expert-like demonstrations. We also allow for the inclusion of demonstrations of various qualities. Moreover, while prior works only recover policies, our approach enables the recovery of both expert policies and rewards, justifying the use of our method for Inverse Reinforcement Learning (I.
sampling B1 37 39
Background
Preliminaries. We consider a MDP defined by the following tuple M (S, A,r, 8,0,, where S denotes the set of states, 80 represents the initial state set, A is the set of actions, r S x A + I defines the reward function for each state-action pair, and P
- S is the transition function, ie.. P(s'|s, a) is the probability of reaching state s' E S when action and y is the discount factor. In reinforcement learning (RL), the aim is to find a policy that maximizes the expected long-term accumulated reward max, (s.a))) ((s,a)]], where Px is the occupancy measure of policy æ: (x
A is made at state S,
(a)ss))))))))))))))))) Σ Et=1 PPss s/
MaxEnt IRL
The objective in MaxEnt IRL recover a reward function r(s, a) from set of expert demonstrations, DE_ Let of be the occupancy measure of the expert policy. The MaxEnt IRL framework 41 proposes to recover the expert reward function by solving
max min [Epss((s - (E,,, [r(s, a)] - E,, [logg(ss (ss, a)))}
Intuitively, the aim is find reward function that achieves the highest difference between the expected value of the expert policy and the highest expected value among all other policies (computed through the min loop).
IQ-Learn Given a reward function r and a policy म, the soft Bellman equation is defined as
BT (s) r(s, a) + [V*(s')), where V (T(s) Ea-r((a)s) ((s a) log T(a ss]. The
Bellman equation B [Q] = Q is contractive and always yields unique Q solution [3 In IQ-leam,
they further define an inverse soft-Q Bellman operator T*(Q) = Q(+, a) -E, | ]. [3] show
that for any reward function r(a, s), there is a unique Q* function such that B Q*) Q", and for
Q' function in the Q-space, there is a unique reward function such that 7 (Q") This result
suggests that one safely transform the objective function of the MaxEnt IRL from r-space to the
3

---

Q Q-space as follows:
max (T = = (Q] a)]] Bp. [T" (s, a)] + Ep. log (T.
Q
which has several advantages; 13 show that Φ(T,, Q) is convex and linear in Q. implying that always yields a unique saddle point solution. In particular, 22) can be converted into maximization over the Q-space, making the training problem no longer adversarial.
SPRINQL
We now describe our inverse soft-Q learning approach, referred to SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning). We first describe the three key components in the SPRINQL formulation:
(I) We formulate the objective function that enables matching the occupancy distribution of not just expert demonstrations, but also sub-optimal demonstrations.
(2) To mitigate the effect of limited expert samples (and larger sets of sub-optimal samples) that can bias the distribution matching of the first step to sub-optimal demonstrations, we introduce a reward regularization term within the objective. This regularization term is ensure reward function allocates higher values to state-action pairs that appear in higher expertise demonstrations.
(3) We show that while this new objective does not have the same advantageous properties as the one in inverse Q-learning [3], with some minor (yet significant) changes is possible to restore all the important properties.
3.1 Distribution Matching with Expert and Suboptimal Demonstrations
We consider setting where there are demonstrations classified into several sets of different expertise levels 2 D2 ....., DN, where D consists of expert demonstrations and all the other sets contains sub- optimal ones. This setting is general than existing work in with sup-optimal demonstrations, which typically assumes that there are only two quality levels: expert and sub sub-optimal. Let D = UVE[N] D be the union of all the demonstration sets and p² ..., PN be the occupancy measures of the respective expert policies. The ordering of expected values across different levels of expert policies would then be given by:
Ept (r" a)l (s, > ....>Eex (r=((r((s (s)a)),
where r" (...) are the ground-trutt rewards. Typically, the number of demonstrations in first level, D is significantly lower than those from other expert levels, i.e., [D=| < [D', for i 2,... N The MaxEnt IRL objective from Equation Caan thus be adapted as follows:
max min
(rE,,(((ssa)) - Ep, [r(8, a)) + Ep, llgg(ss
(3)
RE[M]
where 2 is the weight associated with the expert level i E [N] and we have
WN
and ΣiEN w; = There are two key intuitions in above optimization: (a) Expert level
accumulates higher expected values than expert levels greater than i; and (b) Difference in values accumulated by expert policies and the maximum of all other policies is maximized. The optimization term can be rewritten as:
E Ir(s, a)] - Ir(s, a)] Ep, [log (s, a)).
where ,² = = wap'. Here we note that the expected reward ΣiENN] Wo (r(s, a)] em- pirically approximated by samples from the demonstration sets D D2 ...., D.V The number of demonstrations in the best demonstration set D (i.e. the set of expert demonstrations) is significantly lower when compared other demonstration sets. So, an empirical approximation of E [r(s, a)] using samples from D would be inaccurate.
3.2 Regularization with Reference Reward
We create a reference reward based on the provided expert and sub-optimal demonstrations and utilize the reference function to compute regularization term that is added to the objective of Equation

---

Concretely, we define a reference reward function F(s, a) such that:
T(s T(s ), V() D! and (s', $ DD and F(s, F(s' ,'), V(s, D2 and v(s,,a) , & D2 D and so on The aim here to assign higher rewards demonstrations from higher expertise levels, and zero rewards to those that do not belong to provided demonstrations. We will discuss how to concretely estimate such reference reward values later.
We utilize this reference reward as part of the reward regularization term, which is added into the MaxEntRL objective in (3) follows:
max min { Epe (r(s, a)] pp (r(s, a)] + Ep, [log ((,, a)] - -alEE ((r(s,))-((ss a)) a)
Occupancy matching
Reward regularizer
where a > O is a weight parameter for the reward regularizer term. With (4), the goal is to find policy with occupancy distribution that matches with the occupancy distribution of different expertise levels appropriately (characterized by the weights, wy). Simultaneously, it ensures that the learning rewards are close the pre-assigned rewards, aiming to guide the learning policy towards replicating expert demonstrations, while also learning from sub-optimal demonstrations.
3.3 Concave Lower-bound on Inverse Soft-Q with Reward Regularizer
Even though can be directly solved to recover rewards, prior research suggests that transforming 44) into the Q-space will enhance efficiency. We delve into this transformation approach in this section. As discussed in Section there is a one-to-one mapping between any reward function r and a function Q in the Q-space. Thus, the maximin problem in (4) can be equivalently transformed as:
max min H(Q,T) m def [T"[7 a))] - Ep, [T"
[2] (s, a))] + Ep, [log
(s, a)]
alE, ( a)) ((s.a))))
(5)
where r(s, a) is replaced by [=[ a) and
T" T"[Q = Q(s, (E,
(s) = (e)) log
In the context of angle-experr-levell [13 demonstrated that the objective function the Q-space as given in Equation is concave in Q and convex in implying that the maximin problem always has a unique saddle point solution. Unfortunately. this property does not hold in our case.
Proposition 3.1. H(Q, σ) (as defined in Equation 5 is concave in Q but is not convex in æ.
In general, we can see that the first and second term of (6 are convex in Q, but the reward regularizer term, which can be written as
a'E (((ss,) -og (1 (Q(s', a')
log (5', a')))²
not concave in æ (details are shown in the Appendix). The property indicated Proposition 3.1] implies that the maximin problem within the Q-space maxę min, J[Q, T) may not have a unique saddle point solution and would be more challenging to solve, compared to the original inverse IQ-learn problem.
Another key property of Equation is with regards the inner minimization problem over x, which yields a unique closed-form solution, enabling the transformation of the max-min problem into non-adversarial concave maximization problem within the Q-space. The closed-form solution was given by л argmax,, V*(s) for all s E S. Unfortunately, this result also does not hold with the new objective function in (6), as formally stated below:
Proposition 3.2. H(Q, π²) may not necessarily be minimized n" such that n. -argmax, V V*()). for all 8 S.
5

---

To overcome the above challenges, our approach involves constructing a more tractable objective function that is a lower bound on the objective of ( Let first define [(Q) min, HiQ, x). We then look the regularization term, which causes all the aforementioned challenges, and write:
(T (Q),,,),, - a))² = 7 - - (s")(=
(Q(s. -(s.,))r(s.aa))) ((
2(r(s, a) Q(s, a))
We then take out the negative part of (T(s, a) - Qs, a)) using ReLU, and consider a slightly new objective function as follows:
x) def Σ
R. (TF - (Ep (T [Q] a)) p, Dlog (x)
iE|M
- aEE [Q(ss.)) - ((s. (E. 2ReLU(7(s, ((s,s),,))(((sa))))))) ((s.a))E, (6)
Let [(Q) = min, #)(2,))) The proposition below shows that ((Q) always lower-bounds (Q)).
Proposition 3.3. For any Q 2 0, we have (()) S (Q) and maxq f(Q) S maxq I()). Moreover r(Q) = (Q) if (s, a) I F(s, a) for all (s.
We note that assuming Q 2 is not restrictive, as if the expert's ' rewards r"( a) are non-negative (typically the case), then the true soft-Q function, defined Q*(s, a) E|E.,, ('rr"ss, log (1),))))(s)))))))))) = (s, a)), should also be non-negative, for any As I()) provides a lower- bound approximation of [()), maximizing f(Q) over the Q-space would drive I(Q) towards its maximum value. It is important to note that, given that the inner problem involves minimization, obtaining an upper-bound approximation function is easier. However, since the outer problem is maximization one, an upper bound would not be helpful in guiding the resulting solution towards optimal ones. The following theorem indicates that [(Q) is more tractable to use.
Theorem 3.4. For any Q 2 O, the following results hold: (i) The inner minimization problem min, H(Q, x) has unique optimal solution 7 such that π = argmin, V*(s) for all S and expiQ(s.a))
x (als) Σ. expp(((ssa] (ii) max, (*(s) =log(E, exp(Q(s, a))) def ( and (iii) ((Q) is
concave for Q 0. 0.
The above theorem tells us that new objective T()) has a closed form where (8) is replaced by (s). Moreover f(Q) is concave for all Q 2 0. The concavity is particularly advantageous, as it guarantees that the optimization objective is well-behaved and has a unique solution Q" such that (Q",=²") form a unique saddle point of maxQ min, H((2, x). Thus, our tractable objective has all the nice properties that the original IQ-Learn objective had, while being able to work for the offline case with multiple levels of expert trajectories and our reward regularizer.
3.4 SPRINQI. Algorithm
Algorithm| I provides the overall SPRINQL al- gorithm. We first estimate the reference rewards in lines 2-6 of Algorithm and the overall pro- cess is described in Section 3.4.1 Before proceed to the overall training, we have to esti- mate the weights, w, (associated with the ranked demonstration sets) employed in Hi(Q, TT) We provide a description of this estimation proce- dure in Section| 3.4.2 Finally, to enhance sta- bility and mitigate over-estimation issues com- monly encountered offline Q-learning, we em- ploy a conservative version of (Q, π) in lines
Algorithm 1 SPRINQL: Inverse soft-Q Learning with Sub-optimal Demonstrations
Require: (D), D2, D..... D, (wi, mas myy Yy, Qg.
1: estimate reward reference function
2:
for iteration i...N, No
dN ~ (D*, D2 ..., D 4: from datnset d, caleulate ((Fr))
3:
4 (d* d',
5:
1 7 (,,(TT))
6: end for
7: train SPRINQL
8:
for iteration i...N
9:
d e- (d' dd' ........ .t. (D) D2 DD,,..., D
10:
# +Update Q function
11:
by 13: Update policy actor critic
from dataset d, calculate (Qu. by
8-15 of the algorithm and is described in Sec-
12:
+- + V. (Qu, To)
tion 3.4.3 Some other practical considerations discussed in the appendix.
14:
0 + + v [an In(ra(ale))]
anod
15: end

---

3.4.1 Estimating the Reference Reward
We outline our approach to automatically infer the reference rewards F(s, a) from the ranked demon- strations. The general idea is to learn function that assigns higher values higher expert-level demonstrations. To achieve this, let us define R(T) ((sajer F(s, (i.e., accumulated reward of trajectory 7). For two trajectories Ti: Tj. let
тj denote that т; is lower in quality com- pared to Tj (i.c., Tj belongs to demonstrations from lower-expert policies, compared to Ty). We follow the Bradley-Terry model of preferences [4 5 to model the probability P(r. Tz) as
P(T, < Ty) exp(R(rr)) (R(()) and use the following loss function:
(xp(R(T,))
m (C()) E E
((s.a))F(ss'.co
Σ
In P(T 3T))) (7)
iE[N] (s',),
A.ME TTED"*
where the first term of (T)) serves guarantee that the reward reference values for (s, a) pairs within the same demonstration group are similar, and the second term aims to increase the likelihood that the accumulated rewards of trajectories adhere the expert-level order. Importanll, can be shown below that [(T) is convex in F (Proposition 3.5 making the learning well-behaved. In practice, one can model T(s a) by neural network of parameters 0 and optimize [()) over O-space. Proposition 3.5. [(r) is strictly convex in r.
3.4.2 Preference-based Weight Learning for wt;
Each weight parameter used in 4 should reflect the quality of the corresponding demonstration set D', which can evaluated by estimating the average expert rewards of these sets. Although this information is not directly available in our setting, the reward reference values discussed earlier provide a convenient and natural way to estimate them. This leads to the following formulation for inferring the weights w; from the ranked data: w;
Bloa) (P(s.a))
ESe(n) Di [F(s.)]]
3.4.3 Conservative soft-Q learning
Over-estimation is a common issue in offline Q-learning due out-of-distribttion actions and function approximation errors (L]. We also observe this in our IL context. To overcome this issue and enhance stability, we leverage the approach in [23] to enhance our inverse soft-Q learning. The aim to learn a conservative soft-Q function that lower-bounds its true value. We formulate the conservative inverse soft-Q Q objective
H. (Q, n. = --3 Σ
[Q(s, a)s (((a))))+(((Q))
(8)
BND, amup(als)
where (aass) is particular state-action distribution. We note that in the conservative term is added to the objective function, while in the conservative Q-learning algorithm (23], this term is added the Bellman error objective of each Q-function update. This difference makes the theory developed
(33 not applicable. In Proposition 3.6 below, we show that solving maxq H. (Q) will always yield Q-function that is a lower bound to the Q function obtained by solving maxQ H(Q, T). Proposition 3.6. Let Q a ((( and Q (s,a).
argmax AC (Q, have
Σ BND a~m(a]s)
a)) amp(ap (a]s)
We can adjust the scale parameter 33 in Equation to regulate the conservativeness of the objective. Intuitively, we optimize Q over lower-bounded Q-space, increasing the scale parameter will foree each Q(s, a) towards its lower bound. Consequently, when sufficiently large, QC will point-wise lower-bound Q, i.e., @ a) < @(s, a) for all (s, a).
Experiments
4.1 Experiment Setup
Baselines. We compare our SPRINQL with SOTA algorithms in offline II. with sub-optimal demonstrations: TRAIL (38), DemoDICE (21. and DWBC [37]. Morcover, we also compare with
7

---

other straightforward baselines: BC with only sub-optimal datasets (BC-O), BC with only expert data (BC-E). BC with all datasets (BC-both), and BC with fixed weight for each dataset (W-BC), SQIL 29 and IQ-learn [13 with only expert data. In particular, since TRAIL,, DemoDICE, and DWBC were designed to work with only two datasets (one expert and one supplementary), in problems with multiple sub-optimal datasets, we combine all the sub-optimal datasets into one single supplementary dataset. Meanwhile, BC-E, BC-O, and BC-both combine all available data into a large dataset for learning, while W-BC optimizes E; Le.o-D'-w;; (n(s(a)))) where w; are weight parameters. T-REX [5] a PPO-based IL algorithm that can work with ranked demonstrations. is, however, not suitable for offline setting, Nevertheless, we will later conduct an ablation study to compare SPRINQL with an adapted version of T-REX. Moreover, [[5] developed IPL for learning from offline preference data. This approach requires comparisons between every pair of trajectories, thus not suitable for our context.
do not include it in the main comparisons.
Environments and Data Generation: We test on five Mujoco tasks 32 and four arm manipulation tasks from Panda-gyn gymm [1]]. The maximum number transition of (s, a) per trajectory is 1000 (or 1k for short) for the Mujoco and 50 for Panda-gym tasks (descriptions in Appendix [B.3). The sub-optimal demonstrations have been generated by randomly adding noise expert actions and interacting with the environments. We generated large datasets expert and non-expert demonstrations, For each seed, randomly sample subsets of demonstrations for testing. This approach allows test with different datasets across seeds, rather than using fixed datasets for all seeds as in previous works. More details of these generated databases can be found in the Appendix B.4
Metric:
return The return is normalized by score E.return cesturn-R.rres return mean return of random policy and Eerurrn is the mean return of the expert policy. The scores calculated by taking the last ten evaluation scores of each seed, with five seeds per report.
100 where R. R.return is the
Experimental Concerns. Throughout the experiments, we aim to answer the following questions: (Q1) How does SPRINQL perform compared to other baselines? (Q2) How do the distribution matching and reward regularization terms impact the performance of SPRINQL? (Q3) What happens if we augment (or reduce) the expert data while maintaining the sub-optimal datasets? (Q4) What happens if we augment (or reduce) the sub-optimal data while maintaining the expert dataset? (Q5) How does the conservative term help in approach? (Q6) How does increasing N (the number of expertise levels) affect the performance of SPRINQL? (Q7) Does the preference-based weight learning approach provide good values for the weights w;? (Q8) How does SPRINQL perform in recovering the ground-truth reward function?
4.2 Main Comparison Results
In this section, we provide comparison results to answer (Q1) with three datasets (i.e., N 3). Additional comparison results for 2 can be found in the appendix. From lowest highest
Mujoco
Panda-gym
Cheetah
Ant
Humanoid
Push
PnP
Slide
Avg
BC-E
-3.210.9
6.41119.1
1.3.0.2
8.2.33.8
3.7.22.7
0.0.10.0
2.7
BC-O
14.2.1.9
35.21120.1
10.616.3
8.8.14.5
3.942.7
0.1:0.33
12.1
BC-both
13.2.13.6
47.0.15.9
9.013.55
9.0.14.3
4.4.1..0
0.1.0.4
13.8
W-BC
12.9+2288
47.3+6.4
19.6+19.0
8.8+4.3
3.7+2.8
0.0+0.0
15.4
TRAIL
-..1+0..3
.4.7
2.640.66
11.7+:.00
7.8+3.7
1.71188
3,9
IQ-E
-3.4:0.6
1..41..3
2.4+0.6
26.3+110.9
18.1=12.5
0.1
6.7
IQ-both
-6.14.144
-58.210.0
0.8.10.0
8.313.9
3.8--3.33
0.0:10.2
-8.6
SQIL-E
-5.0.1077
-33.817.4
0.940.11
9.6.1..33
3.241-99
0.1:00.3
-4.
SQIL-both
-5.6+0.55
-58.040.44
0.8+0.0
8.24-3.8
3.3--2.3
0.1+0.33
-12.6
DemoDICE
0.422.0
31.718.9
2.610.8
8.1
4.3:2.44
0.1+0..5
7.9
DWBC
-0.242.5
10.425.0
3.710.3
36.917.4
25.0=6.3
11.614.
14.6
SPRINQI-
73.6+4..
77.0
82.9+11.2
72.0
63.2
37.7+6..6
67.7
Table 1: Comparison results for three Mujoco and three Panda gamd tasks.

---

expertise levels, we randomly sample (25k-10k-18) transitions for Mujoco tasks and (10k-5k- 100) transitions for Panda-gaa gym tasks for every seed (details of these three-level dataset are provided in Appendix BB.). Table D shows comparison results across 3 Mujoco tasks and 3 Panda-gya gym tasks In general, SPRINQL.
(the full results for all the nine environments are provided in Appendix significantly outperforms other baselines on all the tasks.
4.3 Ablation Study No Distribution Matching and No Reward Regularizer
We aim assess the importance of the distribution matching and reward regularizer terms in our objective (Q2). To this end, we conduct an ablation study comparing SPRINQL with two variants: (i) noReg-SPRINQL. derived by removing the reward regularizer term from (6). and (ii) noDM- SPRINQL. obtained by removing the distribution matching term from (6. Here, we note that the noDM-SPRINQL performs Q-learning using the reward reference function, this can viewed as an adaption of the T-REX algorithm [ to our offline setting. The conservative Q-learning term is employed the SPRINQL and the two variants enhance stability. The comparisons for N and N = 3 on five Mujoco tasks are shown in Figure [ (the full comparison results for all tasks are provided in the appendix). These results clearly show that SPRINQL outperforms the other variants, indicating the value of both terms in our objective function.
Cheetah
Ant
Walker
Hopper
Humanoid
100
100
100
50
50-
50
50-
levels
number
of levels
noReg-SPRINQL
noDM-SPRING
SPRINOL
expert
Figure I: Comparison of three variants of SPRINQL across five Mujoco environments.
4.4 Other Experiments
Experiments addressing the other questions are provided in the appendix. Specifically, Sections C. and provide full comparison results for all the Mujoco and Panda-gym tasks for three and two datasets (i.e., N 3 and N = 2), complementing the answer Q1. Section learning curves of the three variants considered in Section a.3vv (answering Q2). Section C.4
C..3 provides the
provides experiments to answer Q3 (what would happen we augment the expert dataset?) and
Section C.5 addresses Q4 (what would happen if we augment the sub-optimal dataset?). Section C.6
experimentally shows the impact of the conservative term in our approaches (i.e., Q5). Section C.7
re-
ports the performance of SPRINQL with varying numbers of expertise levels N (i.e., Q6). Section C.8
addresses Q7, and Section C.100shows how SPRINQL performs in terms of reward recovering (i.e.
Q8). In addition, Section C.11 reports the distributions of the reference rewards and Section C.12 provides a choosing range.
Concretely, our extensive experiments reveal the following: (i) SPRINQL outperforms other baselines with two, three, or even larger numbers of datasets; (ii) the conservative term, distribution matching. and reward regularizer terms are essential to our objective-all all three significantly contribute to the success of SPRINQL; (ii) the preference-based weight learning provides good estimates for the weights wi; and (iv) SPRINQL performs well in recovering rewards, showing a high positive correlation with the ground-true -truth rewards, justifying the use of our method for IRL.
5 Conclusion and Limitations
(Conclusion) We have developed SPRINQL, a novel non-adversarial inverse soft- Q learning algo- rithm for offline imitation learning from expert and sub- optimal demonstrations. We have demon- strated that our algorithm possesses several favorable properties. contributing to its well-behaved, stable, and scalable nature. Additionally, we have devised preference- loss function to au- tomate the estimation of reward reference values. We have provided extensive experiments based on several benchmark tasks, demonstrating the ability of our SPRINQI. algorithm to leverage both expert and non-expert data to achieve superior performance compared bate-of-the-aa algorithms.

---

(Limitations) Some limitations of this work include: (i) SPRINQL (and other baselines) still requires a large amount of sub-optimal datasets with well-identified expertise levels to learn effectively, (ii) there is a lack of theoretical investigation on how the sizes of the expert and non-expert datasets affect the performance of Q-learning, which we find challenging to address, and (iii) lacks a theoretical exploration of how the reward regularizer term enhances the distribution matching term when expert samples are low-this question is relevant and interesting but also challenging to address. These limitations will pave the way for our future work.
Acknowledgement
This research is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2-RP-2020-016 and Lee Kuan Yew Fellowship awarded Pradeep Varakantham.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-firsi international conference on Machine learning, page 1, 2004.
[2] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Reprexentations, 2022.
[3] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[4] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-355, 1952.
[5] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
[6] Daniel Brown, Wonjoon Goo, and Scott Niekum. Better-hhan-demonstaaoo imitation learning via automatically-ranked demonstrations. In Conference on robot learning, pages 330-359, PMLR, 2020.
[7] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning, pages 1262-1277 PMLR, 2021.
[8] Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. ICLR, 2021.
[9] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 2020.
[10] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 2017.
[11] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018.
[12] Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. panda- gym: Open-source goal coaliconditioned environments for robotic learning. arXiv preprint arXiv:2106. 13687, 2021.
[13] Divyansh Garg. Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-4033 2021.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
10

---

[15] Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-bace rl without reward function. Advances Neural Information Processing Systems, 36, 2023.
[16] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[17] Huy Hoang, Tien Mai, and Pradeep Varakantham. Imitate the good and avoid the bad: An incremental approach to safe reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 12439-12447, 2024.
[18] Minh-Huy Hoang, Long Dinh, and Hai Nguyen. Learning from pixels with expert observations. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1200-1206, 2023.
[19] Ozsel Kilinc and Giovanni Montana. Reinforcement learning for robotic manipulation using simulated locomotion demonstrations. Machine Leurning, pages 1-22, 2022.
[20] Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Kee-Eung Kim. Lobs- dice: Offline learning from observation via stationary distribution correction estimation. Ad- vances in Neural Information Processing Systems, 35:8252-8266 2022.
[21] Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang. and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference Leurning Reprexentations, 2021.
[22] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribu- tion matching. In International Conference on Learning Representations, 2019.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 17:1
1191, 2020.
[24] Wenjun Li and Pradeep Varakantham. Generalization through diversity: Improving unsupervised environment design. International Joint Conference on Artificial Intelligence, 2023.
[25] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
[26] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074. 2019.
[27] Mila Nambiar, Supriyo Ghosh, Priscilla Ong, Yu En Chan, Yong Mong Bee, and Pavitra Krishnaswamy. Deep offline reinforcement learning for real-world treatment optimization applications. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 4673-4684, 2023.
[28] Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint Sttv:1711.09600 2017.
[29] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv: /905. 108, 2019.
[30] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMIR Workshop and Conference Proceedings, 2011.
[31] Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations, 2020.
[32] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEFF/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEFF, 2012.
[33] Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In International Conference on Machine Learning, pages 10961-10970. PMLR, 2021.
[34] Wei-Hung Weng, Mingwu Gao, Ze He, Susu Yan, and Peter Szolovits. Representation and reinforcement learning for personalized glyeemic control in septic patients. arXiv preprint arXiv:1712.00654. 2017.
11

---

[35] Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911.11361, 2019.
[36] Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning, pages 8-18-6827 PMIR, 2019.
[37] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator- offline imitation learning from suboptimal demonstrations. In Proceedings of the 39th International Conference Machine Learning, pages 24725-24744 2022.
[38] Mengjiao Yang, Sergey Levine, and Ofir Nachum. Trail: Near-optimal imitation learning with suboptimal data. In International Conference on Learning Representations, 2021.
[39] Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offlin imitation learning with suboptimal demonstrations via relaxed distribution matching. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11016-11024, 2023.
[40] Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor K. Prasanna. Brac+: Improved behavior regularized actor critic for offline reinforcement learning. ArXiv, 10t//2110.00944 2021.
[41] Brian Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, II., USA, 2008.
12

---

A Missing Proofs
We provide proofs for the theoritical results claimed in the main paper.
Proposition 3.1
J TQ, is concave in Q but is not convex in n.
Proof. We recall that
=(Q = Σ
WiBp pt (T ([]( E,p, (7" - Ep [og (8,a))
AE[N]
- aB (T [Q] (s) a(s,
(9)
where (s, a)) ((s,a) Pll'' [V*(s') and V*(s) [
log s)]. We that TT] a) is linear in Q for any (s. Thus, the first and second terms of (Q, γ) in 4 are linear in Q. The last term of (9) involves a sum of squares of linear functions of Q, which convex. So, J(Q, x) concave in Q.
To see that J(Q, r is generally not convex in т, we will consider a quadratic component of the reward regularization term (T* a)) - 7(s, a))² and show that there is an instance of Q and F values that makes this term convex. We first write:
T=))(s,a)) F(s,a) Q(),a) Σ P(6
= Q(s, - ->E P(s'ss, a) Σ
- =og=<a")s))) r(8,a)
a"
For simplification, let us choose Q(s', a") = 0 for all s' such that P(s'|s, a) > 0. This allows us to simplify T ["] a)) 7(s,a) as
T=(((s,a)) F(s, a) - Σ P(dlss,)) Σ T(a"ls)) log < + Q( F(s,a)
We further see that, for any E S, EeEn 7 (als) log o (als) achieves its minimum value at (als)) = 1/|A| for all E [A], and Ens (Tas)) ( log (als) 2 log 1/|A| for any policy æ. As result we have:
P(s a) Σ
T(a"la logm((a') (sa 2 7 log A
So if we select Q(s, a) such that Q(s, (ls,a))r(s,0)) - - 2 0, then T [(((s,,))--((ss)))
- r(s, 2 0
for any Now consider the quadratic function [(T) (A(=))² where A(T) = (s) 7(8, a). Since each term ((a") log (aals') is convex in ,, A(T) is convex in ग. To show [(m]] is
convex in æ, will show that for any two policies nI, mo and [0, 1]. N(am (1 a)m) m) S
aT(m)) + (1 - (m)) To this end, we write
af(m)) - (1 aT((T)
(a\(T1)) + + (1-a)>(Tz)²
(A(mm: + (1 - 2 P(am + - (1=
where (a) is because the function l<(t) = 2 is convex in f, and (b) is because
(i) aA(TT)) + (1 a))a()) Xaam + (1 - m2) (as X(T) is convex in m)
(ii) (A()))) + (1 a)2(2)) and X(am + (1 a2 are both non-negative, and function
h(t)= f² is increasing for all 2
So, we see that with the Q values chosen above, function (T [Q] r(s, a)) is convex and -a(7" (s, r(s,a)) is concave. So, intuitively, when is sufficiently large, J(Q, T) would be almost concave (so not convex), which is the desired result.
13

---

Propasition 3.2 for all s S.
J(Q. T) may not necessarily be minimized at ι such that nq = argmax, (*(s).
Proof. We first write J(Q, π)
as
J(Q, = Σ
BpB.p [T (s) - Ey, (7*(Q[(s. a))] - Eyp log (s, a)]
NE[N]
of pe [T* a)) - F(s, a))²]
Σ E
yB, ( -
(1-7) (50)
=
NE[N]
- - [(((s,)) - - -B V* (=
(10)
We then see that the terms Rp [Q(s - V..V when (s), for all s, are maximized. We will prove that it would not be the case for the last term. Let us choose Q and 7 such that (s) - 7(s,a) > >B.Ve (*))s') We see that for any policy + qQ. we have
and - V (so) are minimized (over x)
Q(s.a) F(s,a) FE, 2 (s)) Thus, ((s,a) r(s,a) (*( Q(+,a) r(8,a) V >0
which implies that
-a ( - (*)))( -((s - Vo
So the last term of (0)) would not be minimized at क = =q. fact, the above scenario, this last term will be maximized at =Q- As a result, there is always a sufficiently large such that the last term significantly dominates the other terms and J(Q, π) is not minimized = TQQ
For any Q 2 0, we have Π(Q) s [(Q) and maxq2o T(Q) I maxq20 I(Q). Mover, T(Q) = T(Q) (s) F(s, a) for all s.a.
Proposition 3.3
Proof. We first write AQ TT as
A)) Σ E. (7)())))))) - E,, (T=(Q((s,))) - Ep, (og
NE[N]
aEE
((s,,)) (E.
+ 2ReLU(7(s,a) Q(s, o) (s)
(11)
Since Q 2 O. (s) ((= logs(a|s)] 2 O. Thus 2ReLU(F(s,a)
a)s, (s)s) 2 ((s, (s). As result, the last term of Hi, ④) is bounded
as
aB ((s,,))F(s,a))) -1B1:1166)0---RR00r1a-00-00--000
<-aEE, (Q(s,a) (F, -(QQ)))))) -F
= --EE [(T a) ((s.
then follows that A(( m) < J(Q,z). Thus, min, f(, x) I min, (). or f()) < (Q)). Mover, we see that if r(s.a) 2 a(s,a) for all (s, a). then 2ReLU a) - Q(s, a) 2(F(s,a) (f(ssa))Q(ss0e /a (s),, implying that H(QQ π) [Q, T) and ((Q) r(Q). This completes the proof.
14

---

Theorem 3.4
For any Q 2 O, the following results hold
(i) The inner minimization problem min, HQQ, x) has unique optimal solution x² such that x Q argmin
V (*) for all 8 E S and
exp(Q(s,
π (als))
E. exp(Q(s,a)
(ii) max, (*(() log Ea exp(Q(ss a))) def (9)s).
(iii) ((Q) is concave for Q 20
Proof. We first rewrite the formulation of ()) a) as
+)( Σ B, (Q(s,)) ,E, V (s')] - (1 7) V V*'(so)
iE|N
-E (Q(s. -Qss.a))F((aa)) +
2ReLU(T(s, - ((s,
We then see that the first and second term of 7Q(Q, T) are minimized when V (s) are minimized,
i.e., at π TQ- For the last term, since V*(s) 2 0 (because Q 2 0), - (E V (8'))² and
-2ReLU(r(s.a) ((s,, a.) are also minimized at Tq. So, H(Q, TT) is minimized at π TQQ as desired.
(it) is already proved in 13
For (iiz), we rewrite H(Q2, π) as
()) T) Σ
wrlBpp pt [T [Q](s a)] - (1 - 7) aw (Q(s0, a) - log Tals()))))]
iE(N)
aEp" - Q(s, a)) + E (s))² aB,
Q(s,
Σ
wrRpp (T (0)) -
(9)s00) - logo(also))
aBEu Q(s, ((s BB + aR, (min - -((,,a)))
(12)
Then, the first and second terms of 12 is linear in Q. The fourth term is concave. For third term, let Φ ()) = |r(s, a) ((s.a) + Eg (s).
(s). We see that ((Q) 2 0 for any Q 2 and ((Q) is convex in Q (because axx (s) is linear in Q). It then follows that, for any 1 [0, and Q1, Q2 0, we have
( + (1 (1-1)(((1))) ()())) (T) (1-1))((1))) (Q)))
N ( + +11 (1=))
(13)
where (a) is due to the fact function (t(t) = is convex, and (b) is because h(t) = 1 is non- decreasing for all & 2 O, and ((Q) is convex and always takes non-negative values. The last inequality 13 implies that (o()))² is convex in Q. So the last term of [22 is concave in Q
Putting all together we conclude that H(Q x) is concave in Q as desired.
D
Proposition 3.5
((T) is strictly convex F.
15

---

Proof. We first write (F)
Σ In exp([((T)))
[(r) E
Σ
-
exp(R(Ty)) + exp(T:)() ((7))
N (a))
A. TED" EDE
(14)
Σ
Σ
-((,)))
E M RED* EEE
R((T))
N (f"
In (cxp(R(T))) exp()) Q(r))
(15)
We then see that the first term is a sum of squares of linear functions of 7, thus is strictly convex. Moreover, since R(T:) is linear in for any T the term In(exp [((::) exp([((T)) has a log- sum-exp form. So this term is convex as well 3 Putting all together we see that (T) is strictly convex desired.
Proposition 3.6
Lel Q argmax F(Q, and @ argmax (Q, R), we have
(s) S
(ss
KND a>a((a(s)
AND a>p((ass)
Proof. We write
-)) =B Σ + m)
(16)
aND ampp(als)
Σ
((s.a)+A((...)) + +((
am<x(a]s)
(6)
(s. a) + Ā((
(17)
assp(ass)
where (a) is because
QC argmax
Q FC (Q, T) and (b) because f(( 2 il( R²) Combine
(16) and 17 we get
3 Σ (s, Σ ((s,a)
BND a>u((oss)
SND am=p(ala)
as desired.
B Additional Details
B.1 Practical Training Objective
We discuss practical implementation of the training objective in 4 using samples from both the expert and sub-optimal demonstration sets. We first note that the term E,, [T (s) E,,, [log n (s, a)) can be written as IK
Ep. 1 (s, a))] Ep, ((, a)] = Eo. (V) 7E,
for any valid occupancy measure p² So, for any policy occupancy measure, we can write this term though the union of expert policies p as
Epp 1T (Q(s E., log (8, a))
Σ
w;E.p (V) "E.
(18)
NE[N]
16

---

As result, by replacing V (s) with (s). we can write [(Q) in compact form as
T(Q)= = Σ
wi [Q(s, (B,.
sE[1]
Σ
-2 W. - - a Σ wele (A (s, a)]
(19)
kE|M]
NE
where
= (Q(s,, = -(s.a)) (E (Q 2RcLU(7(s,a) ( (Q('..
In an empirical offline implementation, to maximize I()), samples (s, from demonstra- tions can be used to approximate the expectations over pt: E(sa (Q(e)) (Q(,,a) (8)) (V))
and (4..)))=:44(ss.a))
We note that in continuous-action controls, the computation of (Q(() involves sum infinitely many actions, which is impractical. In this case, we can update both Q and x in a soft actor-critic (SAC) manner. That is, for each we update Q towards max; (*((2.=)) x)} and for each Q, we update bring it towards 7 by solving max, {V* (s)}, for all As shown above, H(Q, π) in concave in Q and ("(s) is convex in IT, so we can expect this SAC will exhibit good behavior and stability.
B.2 Algorithm Overview
The Figure
2 provide the overview of our algorithm.
Reforanca Reward Training
Funcilon Training
MP
A
Rinul
cap Rlmml Rxpl
-n Y (4 (aaa))) KGM
3. Ri of (ncmooll))
(% ((uu))) Flaol YII
Σ qisa
F(a
Figure 2: Overview of SPRINQL.
B.3 Environments
This section provide a detailed descrition of the enviroments used in our experiments.
B.3.1 Mujoco
like HalfCheetah, Ant, Walker2d, Hopper, and Humanoid are integral to the field of reinforcement learning (RL), particularly in the domain of continuous control and roboties:
MuJoCo gym environments 332
HalfCheetah: This environment simulates a two-dimensional cheetah-like robot. The objective is to make the cheetah as fast as possible, which involves learning complex, coordinated movements across its body.
Ant: This environment features four-legged robot resembling an ant. The challenge is to control the robot move effectively, balancing stability and speed.
Walker2d: This environment simulates a woo-dimensional bipedal robot. The goal is to make the robot walk forward fast possible without falling over.
17

---

The Hopper environment involves a single-legged robot. The primary challenge is balance and hop forward continuously, which requires maintaining stability while in motion.
Hopper:
Humanoid:
The Humanoid environment is among the most complex, featuring bipedal robot with a human-like structure. The task involves mastering various movements, from walking more complex maneuvers, while maintaining balance.
An expert is trying to maximize the reward function by moving with trajectory length limit of 1000. All five environments are shown in Figure
Cheetah
Ant
Walker
Hopper
Humanoid
Figure 3: Five different Mujoco environments.
B.3.2 Panda-gym
The Punda-gym environments 12], designed the Franka Emika Panda robot in reinforcement learning, include PandaReach, PandaPush, PandaPickandPlle and PandaSlide. Here's a short description of each:
The task is to move the robot's gripper randomly generated target position within specific volume. It focuses on precise control of the gripper's 's movement.
PandaReach:
PandaPush: In this environment, cube placed a table must be pushed to target position. The gripper remains closed, emphasizing the robot's ability to manipulate objects through pushing actions.
This complex task involves picking up cube and placing a target position above the table. requires coordinated control of the robot's gripper for both lifting and accurate placement.
PandaSlide: Here, the robot must slide flat object (like a hockey puck) a target position a table. The gripper is fixed in closed position, and the task demands imparting the right amount of force to slide the object to the target.
In these environments, the reward function is for every time step has not finished the task. Moreover, the maximum horizon is extremely short, with a maximum of 50, while an expert can complete the task after several steps. All four different environments are shown Figure
Reach
Push
PnP
Slide
Figure 4: Five different Panda environments.
B.4 Qualities of the Generated Expert and Non-Expert Datasets
In this paper, we provide a new setting utilizing ranked sub-optimal datasets to perform imitation learning in the offline setting (illustration in Figure 5)
18

---

Ranked sub-optimal datasets
N
3
(expert)
Increasing optimality
Figure 5: Training datasets illustration.
We create sub-optimal datasets by adding noise to actions of the expert policy and let them interact with the environments to collect the sub-optimal trajectories. Each task has its own difficulty and sensitivity to the noise of the actions. The averaged returns the generated datasets, computed as percentages w.r.t. the maximum returns, are reported in Figure 6
Cheetah (11415)
Ant (5187)
Walker (4477)
Hopper (3748)
Humanoid (8038)
100-
100
Reach (-1.83)
Push (-3.61)
PnP (-6.61)
Slide (-8.99)
100-
50
50
50
15
Figure 6: Whisker plots illustrate the average returns of both expert and non-expert datasets nine distinct environments in Mujoco and Panda-gym. The numerical values following the task names represent the actual mean return of the expert policy.
B.5 Hyper Parameters and Experimental Implementations
In our experiments, for every algorithm, we run five different seeds corresponding five different datasets sampled from our databases in Appendix [B.]]
We use double Q critic network for our implementation to increase the stability the offline training scheme.
For IQ-learn 13 I.QII.
29
we conduct experiment from its official implement with
double Q critic network.
For DemoDICE
21 conduct experiment from its official implementation
For DWBC
37
conduct experiment from its official implementation
Inspired by double Q-learning, to avoid overfitting in the offline setting, some experiments apply training trick using KL-divergence between the target actor and training actor to prevent rapid policy changes.
For SAC-based algorithms, we use fixed exploration parameter which is commonly used in previous work.
We conducted all experiments total of 8 NVIDIA RTX A5000 GPUs and 64 core CPUs. We use GPUs and 8 core CPUs per task with approximately one day per 5 seeds. The detailed hyper-parameters reported in Table 2
19

---

HYPER PARAMETER
BC-BASED
SPRINQL
ACTOR NETWORK
[256.256]
[256,256]
CRITIC NETWORK
[256.256]
[256,256]
TRAINING STEP
1,000,000
1,000,000
GAMMA
0.99
0.99
LR ACTOR
0,0001
0.00003
LR CRITIC
0,0003
0.0003
LR REWARD REFERENCE
0.0003
0.0003
BATCH SIZE
256
256
CRITIC FACTOR
0.005
SOFT UPDATE ACTOR FACTOR
0.00003
EXPLORATION TEMPERATURE
0.01
REWARD REGULARIZE TEMPERATURE (a)
1.0
CQL TEMPERATURE (8)
1.0
Table 2: Hyper parameters.
C
Supplementary Experiments
In this section, present additional experiments that complement those reported in the main paper, along with ablation studies to address the experimental questions stated therein.
C.1 Full Experiment Results for Mujoco and Panda-gym
We report the full results for the 5 different Mujoco environments and 4 different Panda- Panda-gym environments, with 3 datasets (i.e., = 3), supplementing the results reported in Table [ in the main paper. The detailed results for Mujoco in Table 3 and Panda-gym in Table
Cheetah
Ant
Walker
Hopper
Humanoid
Avg
BC-E
-3.240.9
6.4
0.3+0..
8.3+t-.1
1.3+-0.2
2.6
BC-O
14.242.9
35.2+22.11
66.1+10.8
16.7+4.0
10.6+6..3
29.0
BC-both
13.243..
47.01:5.9
63.9117.3
22.6112888
9.0.13.5
31.1
W-BC
12.91228
47.3
58.619..1
20.916.6
19.6.11.00
31.9
TRAIL
4.1.1..3
-4.7.11.9
0.2.1.33
1.1.0.5
2.6.1..6
-1.0
IQ-E
-3.440.66
-3.411.3
0.1
0.3+0.2
2.400.6
-0.8
IQ-both
-6.141.4
-58.2+0.0
-0.2.0.1
0.0:0.00
0.8+0.0
-12.7
SQIL-E
-5.0+0.7
-33.817.4
0,2+0.2
0.2+0.0
0.9+0.1
-7.5
SQIL-both
-5.6.10.5
-58.04.0.4
-0.24:..0
0.04-0.0
0.8+1.00
-12.6
DemoDICE
0.44:2.0
31.7+8.9
7.243.1
18.747.5
2.6.10.8
12.1
DWBC
-0.2-42.5
10.4+5.0
25.1
66.0+120.9
3.74-0.3
21.0
SPRINQL (ours)
73.6+14.3
7..0+5.6
87.9
70.0
82.9+11.2
78.3
Table 3: Comparison results for Mujoco tasks.
20

---

Reach
Push
PnP
Slide
Avg
BC-E
13
8.2.13.8
3.712.77
0.010.00
6.5
BC-O
16.215.5
8.8.14.5
3.9.2.7
0.140.3
7.3
BC-both
16.315.0
9.0.14.3
4.4.13.0
0.110.4
7.3
W-BC
15.7=5.1
8.8+4.3
3.7
0.0+0.0
7.0
TRAIL
18.3#5.1
11.7+4.0
7.8+3.7
1.7
9,9
IQ-E
97.7+2.4
26.3+10.9
18.1
0.1
35.6
IQ-both
5.7433.4
8.3:13.9
3.8.13.3
0.0:0.2
4.5
SQIL-E
22.1=15.1
9.643..3
3.242.9
0.140.3
8.8
SQIL-botth
8.0:1:422
8.2443.8
3.342.33
0.1
4,9
DemoDICE
14.025..
8.113.7
4.312.4
0.1
DWBC
93.41:.33
36,947.4
25.0116.3
11.61:44
41.7
SPRINQL (ours)
99.8+0099
72.0+5.3
63.2+66.4
67.7++6.6
68.2
Table 4: Comparison results for Punda-gym tasks.
C.2 Comparison Results for N = 2, i.e., One Expert and One Sub-optimal Datasets Q1
We provide additional comparison results for 2, complementing to the answer of (Q1), i.e., how does SPRINQL perform compared to other baselines? In this experiment, we want to test the ability of our algorithm in the same scenario of DemoDICE, DWBC which include expert dataset and only one supplementary dataset. Number of expert transitions in Mujoco domains is 1000 and 100 for Panda-gym. Meanwhile, for the supplementary dataset, 25000 transitions for Mujoco and 5000 for Panda-gym. In general, although experience downgrade in performance due lack of ranking (only one supplementary dataset), leading to misunderstanding the true reward function, our method is still able to leverage the sub-optimal dataset for understanding the expert demonstrations and provide the highest average score. Reported results are shown Table 5 and Figure
Method
Cheetah
Ant
Walker
Hopper
Humanoid
Avg
BC
1.3
39.2.16.6
14.5
28.4:19.0
24.41:25.2
28.2
W-BC
3.1+1.0
39.617..3
46.1=12.3
29.5+9.0
28.6
29.4
DemoDICE
-1.6:0.66
31.611.55
6.5
31.9:12.3
2.710.66
DWBC
-1.2+1..3
9,513.4
13.1:4.7
87.0
4.2+0..4
22.5
SPRINQL (ours)
49.2:118.5
56,918.2
77.8=17.6
39.1121.0
32.416.4
51.1
Table 5: Comparison results for Mujoco tasks in expert datasets.
Method
Reach
Push
PnP
Slide
Avg
BC
17.1
8.1.3388
3.6.1..4
0.3.1..6
7.3
W-BC
18.515.2
9.8.14.5
3.4.13.1
0.1:0..4
8.0
DemoDICE
14.8+55.7
8.8:14.4
3.8+2.9
0.2::0.6
6.9
DWBC
98,2+22.4
38.447.
27.748.1
14.5=4.3
44.7
SPRINQL (ours)
93.9.13.3
61.3.17.2
64.2-16.77
26.6.1..5
61.5
Table 6: Comparison results for Panda-gym tasks in two expert datasets.
21

---

Cheetah
Ant
Walker
Hopper
Humanoid
100
100
100
103
50
0.0
10
0.5
Reach
Push
PnP
Slide
50
50
100
106
105
DWBC
Weighted-BC
BC
DemoDICE
SPRINQL
Expert
Figure 7: Learning curves of two dataset experiment.
C.3
Learning Curves of noReg-SPRINQL and noDM-SPRINQL
This section reports additional results for the comparison between SPRINQL and the two variants noReg-SPRINQL, and noDM-SPRINQL (supplementing the answer of Q2 in the main paper). The comparison results for Panda-gym environments are reported Figure and the learning curves are plotted in Figure and Figure 10
Reach
Push
PnP
Slide
100
100
100
100
0-
levels
noReg-SPRINAL
noDM-SPRINOL
SPRINQL
expert
Figure 8: Ablation study show the performance of three variants SPRINQL across four Panda-gym environments.
Cheetah
Ant
Walker
Hopper
Humanoid
75
25
0.0
10
0.0
0.
05
Reach
Push
PnP
Slide
1001
1001
noRea-SPRINOL
noDM-SPRINOL
SPRINQL
expert
Figure 9: Two term ablation study for two level dataset scenario.
22

---

Cheetah
Ant
Walker
Hopper
Humanoid
100
100
100
103
100
50
60
0.0
10
0.5
165
Reach
Push
PnP
Slide
100
100
50
50
160
166
105
noReg-SPRINNL
noDM-SPRINQL
SPRINQL
expert
Figure 10: Two term ablation study for three level dataset scenario.
C.4 Augmented Expert Demonstrations
We provide experiments to address (Q3) What happens if we augment the expert data while maintaining the sub-optimal datasets? To this end, we use two Mujoco and Panda-gym tasks, keeping the same sup-optimal datasets and add more expert demonstrations to the training sets. The comparison results are reported Figure IT] For the Mujoco tasks, which are more difficult, adding more expert trajectories significantly enhances the performance of all the algorithms. However, for the two Panda-ga -nmda-gymma gym tasks, the influence of adding more expert data appears be less significant in Push and completely absent in PnP. This would be because expert trajectories in these tasks are typically short, consisting of only 2-7 transitions. Hence, larger quantity of additional expert data may be required to enhance performance.
Cheetah
Ant
Push
PnP
100
100
100
100
50
50
50
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
166
1e6
1e6
DWBC
DemoDICE (+E)
SPRINOL
SPRINQL (noE)
DWBC (+E)
BC (+E)
SPRINOL (+E)
Expert
Figure 11: Comparison results with additional expert demonstrations: (+E) signifies that the expert dataset increased from Ik to 5k for Mujoco and from 100 to 500 for Panda-gym, while (noE) indicates expert data.
C,5
Augmented Sup-optimal Demonstrations
Cheetah
Ant
Push
PnP
100
100
100
100
50
60
50
50
0
10k-5k
25k-10k
25k-25k
10k-5k
25k-10k
25k-25k
Sk-1k
10k-5k
10K-10k
Sk-1k
10k-5k
10k-10k
DWBC
BC
DemoDICE
SPRINQL
Expert
Figure 12: Performance comparisons with varied sub-optimal data sizes. The z-axis shows the size of Level 2 and sup-optimal datasets, and y-axis shows the scores.
In this experiment, we want to answer the question (Q4) What happens if we augment (or reduce) the sub-optimal data while maintaining the expert dataset?. We present numerical results to assess
23

---

the impact of sub-optimal data on the performance of our SPRINQL and other baselines. To this end, we keep the same expert dataset and adjust the non-expert data used in Table The performance is reported in 12 It is evident that reducing the amount of data in sub-optimal datasets can result in a significant degradation the performance of our SPRINQL and the other baselines. Conversely, adding more sub-optimal data can enhance overall stability and lead to improved performance.
In the Figure 13 we show the learning curves with varied sizes for sub-optimal datasets. It can be seem that the overall performance tends to improve with more sup-optimal datasets. In particular, for the Ant task, our SPRINQL even fails to maintain stability when the sizes of sup-optimal datasets are low (10k-5k-1k) Moreover, while BC seems to show improvement with more sub-optimal data, the performance of DWBC and DemoDICE remains unchanged. This may be because these approaches rely on the assumption that expert demonstrations can be extracted from the sub-optimal data, which not the case in our context.
10k-5k-1k
25k-10k-1k
25k-25k- 1k
100
100
100
Cheetah
50
50
50
0
0.
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
10k-5k-1k 1k
15k-10k-1k
25k-25k-11 1k
100
100
100
Ant
50
50
0
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
5k- 1k-100
10k-5k-100
10k-10k-100
100
Push
50
50
50
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
5k- 100
10k-5k-100
10k-10k-100
100
100
100
PnP
50
50
50
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
DWBC
BC
DemoDICE
SPRINQL
Expert
Figure 13: Evaluation curves with different sub-optimal-dataset size.
24

---

C.6 Impact of the Conservative Term
In this experiment, we aim to answer (Q5) How does the conservative term help in our approach? The Equation @ introduce the conservative Q learning (CQL) term into our work. Here we also test three variants of our method from Section 4.3 and show the impact of CQL to the final performance.
The experimental results are shown in Figure 14
Cheetah
Ant
Push
PnP
100
100
50-
50
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
SeReg-SPRINOLL. (no CQL)
noDM-SPRINOL (no CQL)
SPRINOL (no CQL) SPRINQL
expert
noReg-SPRINOL
noDM-SPRINQL.
Figure 14: Performance of variant with and without the CQL term in four different environments accross two domains.
C.7 Performance with Varying Number of Expertise Levels
100
75-
50
25
0
5
4
3
2
1
Figure 15: Average returns of the 5 HalfCheetah datasets (one expert and four sub-optimals).
sub-optimal dataset
sub-optimal datasets
3 sub-optimal datasets
4 sub-optimal datasets
100
100
50
50
50
50
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
166
1e6
DemoDICE
DWBC
SPRINQL
Expert
Figure 16: Experiment results for different numbers of sub-optimal datasets. The learning curves are calculated by mean with shaded by the standard error of data seeds.
We provide an experiment to answer (Q6) - How does increasing N (the number of expertise levels) affect the performance of SPRINQL? We assess our algorithm with different numbers of expertise levels to investigate how adding or removing sub-optimal expertise levels influences performance. Specifically, we keep the same expert dataset comprising expert trajectory and conduct tests with sub-optimal datasets from the Cheetah task. Details the average returns of the five datasets are reported in Fig. 15 In this context, SPRINQL outperforms other algorithms in utilizing non-expert
25

---

demonstrations. Furthermore, BC successfully learns and achieves performance comparable to the performance SPRINQL with 2 dataset, while DemoDICE and DWBC struggle to learn. The detailed results are plotted in Figure T6 In Section (C.2) below, We particularly delve into the situation of having two datasets (one expert and one sub-optimal), which is typical setting in prior work.
C.8 Ablation Study for the Preference-based Weight Learning
We provide this experiment to answer (Q7) - Does the preference-based weight learning approach provide good values for the weights w.? To this end, we compare the performance SPRINQL based on the weights determined by the preference-basee methods described in the main paper (denoted as auto W), and the following weighting scenarios:
Uniform W: We chose the weights w; uniformly over [0, 1] as w {0.55, 0.35, 0.15] with ratio of approximately 10: ::
Reduced W: Starting from Uniform W, we reduced the weights of the non-expert data and (0.65, 0.2, 0.2,,,0.2,000 0.15) with ratio of approximately 10 30:3: :
tested the weights
Increased W: Starting from Uniform W, we increased the weights of the non-expert data and chose w {0.4, 0.32, 0.28) with a ratio of approximately 10 7.
These weight vectors w are normalized to follow wi >> u2 > > WN and E Ws 1.
The comparison results are shown in Figure 17
Cheetah
Ant
1.0
10
Reduced Increased W
Expert
Reduced W Incressed W
Expert
0.0-
0.5
0.2
0.2
0.0
0.0
0.8
1.0
0.2
0.4
0.8
1e5
1el
Figure 17: Experiment results for SPRINQL with different manual selection of weight W.
C.9 D4RL mujoco dataset
In this section, we present additional experiments using the official D4RL dataset. Unfortunately, since our algorithm requires meaningful demonstrations, exclude the Random dataset and are only able to test with N = 2 (Medium and Expert datasets). The detailed results are shown in Figure 18
Cheetah
Ant
Walker
Hopper
Figure 18: Performance in D4RL dataset with Medium (25,000 transitions) and Expert(1,000 transitions. The results are reported from 5 seeds per method.
26

---

C.10 Reward Recovering
In this experiment, we want to answer (Q8) How does SPRINQL perform recovering the ground- truth function? Compared to BC- algorithms, one notable advantage of Q-learning based algorithms their ability to recover the reward function. Here, we present experiments demonstrating reward function recovery across five MuloCo tasks, comparing recovered rewards to the actual reward function. To achieve this, we introduce increasing levels of random noise to the actions of a trained agent and observe its interactions with the environment. We collect the state, action, and next state for each trajectory, then predict the recovered reward and compare to the true reward from the environment. For the sake of comparison, we include SPReg- which can be considered an an adaption of IQ-learn [13] to our setting, and noDM-SPRINQL, which in fact an adaption of T-REX to our offline setting.
Comparison results are presented Figure 19 predicted rewards for SPRINQL across all testing tasks, whereas the other approaches fail to return correct relationships for some tasks.
We observe linear relationship between the and
27

---

MoReg-SPRINOL
noDM-SPRINQL
SPRINQL
Predicted Return
Cheetah
400
100000
Prodicted
Predicted
500
300
400
150000
200
5000 True Retum
10000
2500
5000
7500
10000
2500
5000
7500
16000
True Return
Return
noDM- DORM-SP
SPRINQL
500
500
400
Predicted Return
Predicted Retum
400
Ant
300
Predicted
300
200
100-
2000
4000
2000
4000
2000
4000
Tree Return
True Return
True Return
WLDM--PRINQL
SPRINQL
400
Predicted Retum
Predicted Retum
Predicted Retum
Walker
300
200
200
200
100-
1000
2000
3000
4000
10000
2000
3000
4000
1000
2000
3000
4000
The Return
True Return
True Return
noReg- PoRee-SPPINNLL
noDM-SPRINQL
SPRINQL
2504
600
Hopper
200
800
Predicted
150
Predicted
150
Predicted
100.
100
50
True
noDM--PPE -PRIILL
SPRINQL
Humanoid
Predicted Retum
-100000-
-20000
Predicted Return
600
-200000
Predicted
-60000
400
300000
-80000
200
-100000
7500
2500
5000
7500
2000
4000
6000
8000
True Return
Figure 19: Recovered return and the true return of five Mujoco environments.
28

---

C.11 Reference Reward Distribution
From the experiment reported in Table I we plot the distributions of the reward reference values in Figure 20 where the z-axis shows the level indexes and the y-axis shows reward values. The rewards seem to follow desired distributions, with larger rewards assigned higher expertise levels. Moreover, rewards learned for expert demonstrations are consistently and significantly higher and exhibit smaller variances compared to those learned for sub-optimal transitions.
Cheetah
Ant
Walker
Hopper
Humanoid
1.50
1.00
1.00
1000
1.c0
1.15-
0.75
0.3
0.0
0.00
0.00
0.00
0.c0
Reach
Push
PnP
Slide
1.0
1.0
10-
100
0.5
0.8
0.8
08
9.
0.4
0.0
Figure 20: Whisker plots illustrate the reward reference distribution of three datasets of each environ- ment for one seed.
C.12 Different alpha ablation study
As our objective is combination of two terms with balancing parameter o, we conduct additional experiments to evaluate the performance of our method across a range of a values. The detailed results are presented in Figure 21 Overall, the results indicate that can be selected within the range of 0.1 to 10 to achieve good performance.
Cheetah (N=3)
Ant (N=3)
1.0
SPRINCL
Export
SPRINGL
Export
0.8
08
0.8
0.4
0.7
0.0
Cheetah (N=2)
Ant (N=2)
1.0
1.0-
SPHIINCI
Expert
SPYINGI
Expert
0.8
0.6
0.6
Scors
0.4
02
0.2
0.0
0.01
0.5
10
100
0.01
0.1
10
100
700
Alpho
Mohha
Figure 21: Performance of SPRINQL in different a. The results are reported from seeds per a value.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our abstract includes main claims reflecting main contributions and finding.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussion on the limitations of our work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

All the proofs of the theorems and propositions stated in the main paper are provided in the appendix with clear references.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details on the environments and hyper-parameter settings in the appendix. We also uploaded our source code for re-productivity purposes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We have describe how to generale our data as well as provide it along with our submitted source code with sufficient instructions for their use.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctaaps.cc/publcc/guee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have detailed these information in the main paper and the appendix of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reported the mean scores and standard deviations for the result tables. We have also shown training curves constructed from scores and shaded by standard error. All the experiments are reported with multiple training seeds as well as different datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided these information the "Hyper parameter and Experimental Implementations" section in our appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https/////eurpppsccc////b///biie
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper provides general offtine imitation learning with multiple expert levels and only testing on the simulated environments. As such, we do not foresee any direct societal impact.
Guidelines:
33

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our training data are generated from open source simulated environments which have no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We have provided clear citations to the source code and data we used in the paper.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
34

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibilityy benchmarking purposes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We have crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "gMhf099a8e", "reviewer": "Reviewer_g6FW", "summary": "This work presents an offline imitation learning (IL) method that leverages both expert and suboptimal demonstrations. Distinct from previous methods, this approach assumes access to suboptimal demonstrations across a spectrum of performance levels, with a known ranking to the learners.\nTo utilizes suboptimal demonstations with various levels, this work suggest SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse Q-Learning), a reward reguarized inverse soft Q-learning method for the offline IL problem.\nThe evaluation of this method is conducted using the Mujoco and Panda-gym environments, employing a small dataset of expert demonstrations and a larger collection of demonstrations generated by introducing varying levels of noise to expert actions.", "questions": "- Could the authors clarify the motivation of employing reference rewards?\n- Moreover, could you elucidate why reference rewards are utilized as a regularization mechanism rather than directly applying RL using reference rewards?\n- The problem setting appears closely aligned with preference-based learning, where optimality (or preference) levels are explicitly known. Could you delineate the distinctions between these two problem settings?\n- Can you provide detailed justifications for why comparing the proposed method with IPL is deemed unsuitable? \nIt appears feasible to compare by segmenting each trajectory by suboptimality levels, then pairing segments from differing levels to facilitate IPL. Could this approach not be applied effectively?\n\n[minor comments] reference [15] in line 398 from \"NeurIPS 2024\" to \"NeurIPS 2023.\"", "limitations": "The authors addressed their limitations in Conclusion section in the main manuscript. They described that this work does not have any direct societal impacts in the Checklist #10, Broader Impacts.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work aims to extend the domain of offline imitation learning by addressing the challenge of leveraging imperfect demonstrations with multiple suboptimality levels, presenting a novel and intriguing approach. Drawing on foundational principles from prior research, notably IQ-Learn, this study demonstrates theoretically that the proposed extension, SPRINQL, retains the advantageous properties of IQ-Learn.\nFurthermore, SPRINQL adeptly incorporates sophisticated concepts such as the Bradley-Terry model and Conservative Q-Learning (CQL) into its training framework. The experimental outcomes are notable, with SPRINQL achieving surprisingly effective results that surpass previous baselines, which were unable to successfully imitate expert performance using the datasets used in this work.", "weaknesses": "Significant concerns with this submission are the questionable reporting on the performance of baseline methods and issues related to reproducibility. \nThe experimental results presented are not entirely convincing, as the performance of the baselines substantially deviates from those reported in their original papers. \nSpecifically, even in the case with N=2 suboptimality levels, suitable baselines such as TRAIL, DemoDICE, and DWBC fail to outperform even basic BC-all in nearly all the Mujoco tasks listed in Table 3 (Appendix). \nThe authors need to clarify the reasons behind this anomaly.\nTo enhance the trustfulness of experiments, it would be beneficial for the authors to release the datasets utilized in their experiments or to use standardized datasets such as D4RL or RL-unplugged. This would allow for a more reliable validation of the results and facilitate comparisons with other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading our paper and providing us with constructive feedback. \n\n---\n\n> Significant concerns with this submission are the questionable reporting on the performance of baseline methods and issues related to reproducibility. The experimental results presented are not entirely convincing, as the performance of the baselines substantially deviates from those reported in their original papers...\n\nOur dataset, as well as all the code to reproduce the experiments, is already uploaded to an anonymized Google Drive folder and is ready to be made publicly available. \n\n**Poor performance of DemoDICE and DWBC:**\n\nIn DemoDICE and DWBC papers, their dataset for N=2 contains:\n- Set 1: a small set of expert trajectories; and  \n- Set 2: an unlabelled set of trajectories (which also contains expert trajectories).  \n\nIn our case, for N=2, our dataset contains: \n- Set 1: a small set of expert trajectories \n- Set 2: a large set of sub-optimal trajectories (no expert trajectories are present). \n\nThe key difference is the presence of expert trajectories in Set 2. DemoDICE and DWBC learn only from expert trajectories in Set 1 and Set 2, and ignore the sub-optimal trajectories. Their performance is reliant on expert trajectories being present in Set 2. Since, in our case, we do not have any expert trajectories in the set of sub-optimal trajectories, their performance suffers as they must learn only from the set of labeled expert trajectories.  \n\nOur approach on the other hand learns from the sub-optimal trajectories in Set 2 as well and hence can provide better performance.  \n \nIn summary, the two reasons for the poor performance of DemoDICE and DWBC are: \n- Our dataset does not contain expert trajectories in Set 2.   \n- DemoDICE and DWBC learn from expert trajectories in Set 1 and Set 2 and ignore sub-optimal trajectories. \n\nTo support the above explanation, we have conducted additional experiments using a standard dataset (i.e., D4RL - see Figure 1 in the attached 1-page PDF), where sub-optimal data does not contain expert demonstrations. The results show that our algorithm and BC on average perform better than DemoDICE and DWBC. \n\n**Poor performance of TRAIL:**\n\nFor TRAIL, they do not assume to have expert demonstrations in the sub-optimal data but require a very large set of sub-optimal data to learn the environmental dynamics. For instance, in the TRAIL paper, they use 1 million sub-optimal transitions, compared to (25000+10000) in our approach.\n\n**Comparison to BC:**\n\nTRAIL, DemoDICE, and DWBC cannot outperform BC if their strong assumptions -- expert trajectories in Set 2 for DemoDICE and DWBC, a large number of sub-optimal trajectories in Set 2 --  do not hold. Please refer to our additional experiments reported in Figure 1 of the 1-page PDF. \n\n---\n\n> Could the authors clarify the motivation of employing reference rewards?\n\nDistribution matching (DM) term commonly used in offline imitation learning leads to overfitting when there is only a small amount of expert data. Therefore, we utilized the reference rewards in a regularization form to ensure that expert demonstrations are assigned higher rewards than non-expert ones, enhancing the effectiveness of the DM. Our experiments in Section 4.3 clearly demonstrate this aspect. \n\n> Moreover, could you elucidate why reference rewards are utilized as a regularization mechanism rather than directly applying RL using reference rewards?\n\nWe utilize it as a regularization mechanism, as Distribution Matching (DM) is essential for good performance in imitation learning settings.  This is observed in our ablation study of Section 4.3, where the noDM-SPRINQL (directly utilizing RL with the reference rewards) performs worse than our main algorithm, especially for challenging tasks such as  Humanoid. \n\n> The problem setting appears closely aligned with preference-based learning, where optimality (or preference) levels are explicitly known. Could you delineate the distinctions between these two problem settings?\n\nThe distinction here is that, in preference-based learning, trajectories are ranked by experts, so given a pair of trajectories, we know which one is more preferred. In our work, information comes from non-experts as well. The data come from experts of different levels, so we can only ensure that the expected rewards from higher-level experts are better. Unlike in preference-based learning, in our case, a specific trajectory from a higher-level expert is not necessarily better or preferred than one from a lower-level expert. Therefore, our setting is different. \n\n> Can you provide detailed justifications for why comparing the proposed method with IPL is deemed unsuitable? It appears feasible to compare by segmenting each trajectory by suboptimality levels, then pairing segments from differing levels to facilitate IPL. Could this approach not be applied effectively?\n\nAs discussed above, IPL is a preference-based approach, and due to differences in assumptions about how the data is obtained, we did not find the comparison suitable. However, reviewers' point is well noted and we address this in two ways: \n- IPL is quite similar to our noDM-SPRINQL (which learns Q functions from ranked data using the Bradley-Terry model). The key difference is that IPL learns the reward and Q function simultaneously, while noDM-SPRINQL learns them sequentially. Section 4.3 shows that noDM-SPRINQL is not effective in our setting.  \n- We have now provided additional experiments (Figure 2 in the 1-page PDF) comparing our method with IPL, which generally show that our algorithm performs significantly better.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G6weSbuSuO", "reviewer": "Reviewer_78nP", "summary": "This paper presents SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning), a novel algorithm for offline imitation learning with multiple levels of expert demonstrations. SPRINQL leverages distribution matching and reward regularization to effectively leverage the mixture of non-expert data and expert data to enhance learning. An extensive evaluation is conducted to assess SPRINQL, answering eight well-defined questions regarding the capability and performance of SPRINQL.", "questions": "Please address the weaknesses above.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed technique presents an improvement for learning from mixed-expert demonstrations and is backed by theory. \n+ The paper is very well-written and contains sufficient detail to understand the key proposed novelties. \n+ The evaluation (including those in the appendix) is extensive and presents strong support for the proposed algorithm.", "weaknesses": "- The authors mention that this setting (having several sets of data labeled with relative quality scores) is more general than two quality levels: expert and sub-optimal. Could the authors provide further support for this claim? I would argue that having additional quality annotations makes this setting more niche.\n- Table 1 contains a very large evaluation but there isn't much commentary on the different trends between baselines and SPRINQL.\nComment: The example in the introduction does not seem complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and the insightful comments.\n\n---\n\n> The authors mention that this setting (having several sets of data labeled with relative quality scores) is more general than two quality levels: expert and sub-optimal. Could the authors provide further support for this claim? I would argue that having additional quality annotations makes this setting more niche.\n\nFirst, we would like to restate that in our setting, there are datasets labeled with different levels of expertise, but relative scores are not available. If there are only two expertise levels, our setting corresponds to the prior work's two-quality level setting. Additionally, our setting, which includes several datasets of different expertise levels, stems from practical observations, such as from projects on treatment optimization we are running, where we have demonstrations from doctors with varying levels of experience. One can also think of applications in autonomous driving, where demonstrations come from drivers of different experience levels, or demonstrations from large language models of different qualities. \n\n> Table 1 contains a very large evaluation but there isn't much commentary on the different trends between baselines and SPRINQL. Comment: The example in the introduction does not seem complete. \n\nWe did not include detailed commentary in our paper due to space constraints. The reviewer’s points are well noted. We will add more discussion to analyze the results reported in Table 1. Additionally, we will elaborate on the robotic manipulation example in the introduction and provide another example on treatment optimization to further support our motivation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlieN1ACXP", "reviewer": "Reviewer_vP7z", "summary": "This paper considers the problem of imitation learning on suboptimal demonstrations given coarse trajectory expertise rankings. It introduces SPRINQL, a method combining inverse soft-Q learning, conservative Q learning, and reward learning from ranked trajectory preferences. SPRINQL estimates a reference reward function given trajectory expertise rankings, and extends inverse soft-Q learning to the suboptimal demonstration setting by adding reward regularization and weights based on the estimated reference reward. The paper shows that the SPRINQL objective is tractable with a unique solution over $Q, \\pi$. Experiments show that SPRINQL outperforms prior methods on suboptimal trajectories in simulated environments. Ablations justify each component in SPRINQL and demonstrate robustness across a range of dataset sizes and number of suboptimal datasets.", "questions": "- How does SPRINQL perform on suboptimal trajectories generated from under-trained RL policies, instead of rollouts from noisy expert actions?", "limitations": "- SPRINQL assumes access to trajectory expertise rankings.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- SPRINQL works with a number of suboptimal demonstrations, with reasonable assumptions such as access to trajectory expertise rankings.\n- The main objective is shown to admit a unique saddle point solution over $Q, \\pi$ and is tractable for optimization.\n- Experiments show that SPRINQL outperforms relevant offline imitation learning baselines on suboptimal trajectories in multiple simulated locomotion and manipulation environments.\n- Extensive ablations demonstrate that the method works with a range of numbers/sizes of suboptimal datasets, and justify each component in SPRINQL.", "weaknesses": "- Suboptimal trajectories are sourced from adding random noise to expert actions. While this is a reasonable proxy, it would be more convincing to see them generated from under-trained RL policies instead.\n- It would be great to see SPRINQL work on image-based environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and the positive feedback. \n\n---\n\n> Suboptimal trajectories are sourced from adding random noise to expert actions. While this is a reasonable proxy, it would be more convincing to see them generated from under-trained RL policies instead. How does SPRINQL perform on suboptimal trajectories generated from under-trained RL policies, instead of rollouts from noisy expert actions?\n\nWe thank the reviewer for the comment. While we believe adding noise is a practical setting, we agree that evaluating undertrained policies is also reasonable. In response, we have conducted additional experiments in this setting using the standard D4RL dataset (see Figure 1 in the attached 1-page PDF). These results generally show that our approach remains competitive or superior compared to other baselines. \n\n> It would be great to see SPRINQL work on image-based environments.\n\nThe reviewer's point is well noted. Unfortunately, to the best of our knowledge, there are no image-based datasets available for offline imitation learning with sub-optimal data. We have utilized the accepted benchmarks from the literature in offline imitation learning for our evaluation and benchmarking. We hope to consider this for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1xKo7KgER", "reviewer": "Reviewer_qfkX", "summary": "This paper considers the problem of offline imitation learning from sub-optimal demonstrations. The paper proposes the SPRINQL algorithm, which automatically weights the demonstrations, regularizes the loss with a reference reward function, and then performs inverse soft-Q learning. Experimental results show that the proposed method achieves better performance compared with baselines. Extensive ablations are also provided.", "questions": "1. In Equation (3), a modified objective is presented. However, it seems that this objective depends largely on the quality of weights, and it does not necessarily recover the ground truth reward function nor the optimal policy. Given that the weight is chosen with a heuristic, what kind of guarantee can the authors give about the converged policy?\n\n2. Following the first question, in the experiments, seems that the proposed algorithm always outperforms BC-E because of the lack of expert demonstrations. However, I am curious to see what will happen given sufficient expert demonstrations. \n\n3. In equations above line 162, what if $(s, a) = (s', a')$, given that it is possible for two stochastic policies to choose the same action in the same state?\n\n4. Line 173, regarding \"prior research\", could the authors add some citations?\n\n5. It seems that the weight for reward regularization $\\\\alpha$ is an important hyperparameter. Will the performance of the proposed method change a lot with different $\\\\alpha$? In the extreme case, what if we learn a policy with RL using reward function $\\\\bar r$?", "limitations": "The authors discuss the limitations adequately in the last Section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well written. The experimental results are strong, and the ablation studies are extensive. It is impressive to see the proposed method improves its performance with more sub-optimal demonstrations provided.", "weaknesses": "1. Some parts of the proposed approach are unclear to me. Please see questions 1-4. \n\n2. The influence of the reward regularization term is not completely studied. Please see question 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading our paper and for the constructive feedback. \n\n---\n\n> In Equation (3), a modified objective is presented. However, it seems that this objective depends largely on the quality of weights, and it does not necessarily recover the ground truth reward function nor the optimal policy. Given that the weight is chosen with a heuristic, what kind of guarantee can the authors give about the converged policy?\n\nThank you for the comment. In our setting, we have datasets with different levels of sub-optimality, and the amount of expert data is limited, making it generally impossible to recover the ground truth rewards or the optimal policy. What we can infer from equation (4) is that if $\\alpha = 0$, then the algorithm will converge to $\\rho^U$, which is a union of occupancy measures of different expert levels. If $\\alpha$ is much larger than 1, the algorithm will return a policy optimal for the reference rewards (obtained by a preference-based approach). Our experiments show that neither extreme case works well (Section 4.3). In our experiments, we simply choose $\\alpha = 1$, which we find reasonable to balance between the two extremes. In theory, the objective in equation (4) will generally return a policy that lies between the one given by matching the unioned occupancy measures and the one given by the reference rewards. \n\nTo support our response above, we have conducted additional experiments (shown in Figure 3 of the attached 1-page PDF) testing our algorithm with varying $\\alpha$ and computed the normalized score. These experiments generally show that $\\alpha$ can be chosen between $0.1$ to $10$ to achieve good performance. \n\nWe hope this addresses some of your concerns about the role of $\\alpha$. If you have any other comments or questions regarding this, we would be happy to address them. \n\n> Following the first question, in the experiments, seems that the proposed algorithm always outperforms BC-E because of the lack of expert demonstrations. However, I am curious to see what will happen given sufficient expert demonstrations.\n\nIt is well known that as we increase the number of expert demonstrations, the performance of BC-E will gradually improve and eventually surpass all other approaches (prior studies have already shown that BC performs very well with sufficient data). However, it is not feasible to have many expert trajectories and if we do have enough trajectories, there is no need for optimized offline imitation learning approaches. In Section C.4 and C.5 (appendix) of the experiments, we increased the amount of expert data and sub-optimal data to observe the effect. However, the focus was **not** on providing “sufficient” data to align with our initial problem setting and motivation.\n\n> In equations above line 162, what if $(s,a) = (s',a')$, given that it is possible for two stochastic policies to choose the same action in the same state?\n\nThank you for the observation. In our datasets,  if a (state, action) pair appears in an expert dataset, it will be excluded from any lower-level expert dataset. We will clarify this point. \n\n> Line 173, regarding \"prior research\", could the authors add some citations?\n\nThis has been investigated in the prior IQ-learn paper and some subsequent works. We will add relevant citations for this. Thank you for the comments. \n\n> It seems that the weight for reward regularization $\\alpha$ is an important hyperparameter. Will the performance of the proposed method change a lot with different $\\alpha$? In the extreme case, what if we learn a policy with RL using a reward function \n$\\bar{r}$?\n\nThe two extreme cases $(\\alpha = 0$ and $\\alpha \\gg 1$) have already been considered and tested; they correspond to noDM-SPRINQL and NoReg-SPRINQL in the ablation study in Section 4.3. These two extreme cases are generally outperformed by our main algorithm. In Figure 3 of the attached 1-page PDF, we provided additional experiments showing the performance of our algorithm with varying $\\alpha$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for carefully reading our paper and providing constructive feedback. We have made efforts to address all the points raised. Please find here, some key major points we wish to emphasize:

**Comparison to DWBC, DemoDICE, TRAIL:** Our key contribution is an offline imitation learning approach that learns not only from:
- Expert trajectories; but also, from  
- sub-optimal trajectories.

On the other hand, 
- DWBC and DemoDICE focus only on learning from expert trajectories.  
- DWBC and DemoDICE have an unlabelled set of trajectories, but they identify expert trajectories from the unlabelled set and ignore the sub-optimal trajectories. 
- TRAIL learns from sub-optimal trajectories but requires a significant number (1 million) of transitions compared to our approach (35000). 

We have provided additional experiments using D4RL (Figure 1 in the 1-page pdf), that demonstrate the superiority of our approach. We also provide experiments to compare against IPL (Preference-based learning) in Figure 2 of the 1-page pdf.  

**Motivation for Multiple experience levels:** Additionally, our setting, which includes several datasets of different expertise levels, stems from practical observations, such as from projects on treatment optimization we are running, where we have demonstrations from doctors with varying levels of experience. One can also think of applications in autonomous driving, where demonstrations come from drivers of different experience levels, or demonstrations from large language models of different qualities. 


**Utilizing under-trained policies instead of noise to create sub-optimal datasets:**  We have conducted these additional experiments in this setting using the standard D4RL dataset (see Figure 1 in the 1-page PDF). These results generally show that our approach remains competitive or superior compared to other baselines.  

**Impact of the reward regularization weights on the performance of the proposed algorithm:** We have conducted additional experiments (shown in Figure 3 of the attached 1-page PDF) testing our algorithm with varying $\alpha$.  

___

We have provided specific responses to individual reviewer questions below and happy to answer any other questions reviewers/AC may have.

Meta-review: This paper proposes an offline imitation learning (IL) method that leverages both expert and suboptimal demonstrations. This method requires access to suboptimal demonstrations across a spectrum of performance levels, with a known ranking to the learners. Experiments  reported in the paper show that the proposed solution outperforms prior methods on suboptimal trajectories in simulated environments. Ablations justify each component in the proposed solution.
Decision: Accept (poster)