No Experts, No Problem: Avoidance Learning from Bad Demonstrations
Authors: Huy Hoang
Tien Anh Mai
Pradeep Varakantham
Venue: NeurIPS 2025 poster ()
Keywords: Offline Imitation Learning
Imitation Learning
Avoidance Learning
Undesirable Demonstrations
Q Learning
Primary area: reinforcement_learning
TL;DR: We develop an inverse reinforcement learning approach that avoid undesirable demonstrations

Abstract: This paper addresses the problem of learning avoidance behavior within the context of offline imitation learning. In contrast to conventional methodologies that prioritize the replication of expert or near-expert demonstrations, our work investigates a setting where expert (or desirable) data is absent, and the objective is to learn to eschew undesirable actions by leveraging demonstrations of such behavior (i.e., learning from negative examples).

To address this challenge, we propose a novel training objective grounded in the maximum entropy principle. We further characterize the fundamental properties of this objective function, reformulating the learning process as a cooperative inverse Q-learning task. Moreover, we introduce an efficient strategy for the integration of unlabeled data (i.e., data of indeterminate quality) to facilitate unbiased and practical offline training. The efficacy of our method is evaluated across standard benchmark environments, where it consistently outperforms state-of-the-art baselines.

No Experts, No Problem: Avoidance Learning from Bad Demonstrations
Huy Hoang
Tien Mai
Singapore Management University Institute for Infocomm Research, A A*STAR, Singapore mh. hoang. 2024@phdcs smu wdw.eddu.g...
Singapore Management University atmai @smu. edu. sg
Pradeep Varakantham Singapore Management University pradeepv0smu. edu. agg
Abstract
This paper addresses the problem of learning avoidance behavior within the context of offline imitation learning. contrast to conventional methodologies that priori- tize the replication of expert or near-expert demonstrations, our work investigates a setting where expert (or desirable) data is absent, and the objective learn eschew undesirable actions by leveraging demonstrations of such behavior (i.e., learning from negative examples).
To address this challenge, we propose a novel training objective grounded in the maximum entropy principle. We further characterize the fundamental properties of this objective function, reformulating the learning process as a cooperative inverse Q-learning task. Moreover, of unlabeled data (i.e., data of indeterminate quality) to facilitate unbiased and practical offline training. The efficacy of method is evaluated across stan- dard benchmark environments, where it consistently outperforms state-of-the-art basclines.
introduce an efficient strategy for the integration
Introduction
Imitation learning [ 47 offers feasible alterna- tive to Reinforcement Learning (RL), enabling agents learn directly from expert demonstrations without the need for explicit reward signals. It has proven effective in several tasks, even with limited expert data, and is particularly useful in capturing human preferences.
Large
Unlabeled
Most existing imitation learning approaches prior- itize maximizing task performance (i.e., expected return) by closely mimicking expert demonstra- tions [8 28 [2]. However, in practice, expert or near-expert demonstrations may be unavailable insufficient. First, acquiring expert demonstrations can be prohibitively expensive or time-consuming as it often requires specialized skills resources [40]
Figure 1: Datasets for offline imitation learn ing. In setting, we only Undesirable labeled dataset (DUN) and Unlabeled dataset (DMix).
Second, in many domains, such healthcare [1S] or autonomous driving (6) expert-level perfor-
mance may be rare or difficult to define, leading to a scarcity of high-quality demonstrations [30]
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Third, expert demonstrations may not cover the full range of scenarios or edge cases, limiting their generalizability 43].
Instead of expert demonstrations, there may be collections of undesirable demonstrations that should be avoided and sub-optimal behaviors that may be partially imitated. In the development of self- driving cars [5]. while companies may collect user driving data to train their models, the system must ensure it does not replicate faulty behavior, such as traffic violations or unsafe driving practices [6]. In the field of treatment optimization, data may include actions that led to bad patient outcomes, and the system must learn avoid such behaviors
Although this is an important and interesting problem setting, there has been limited research address- ing it effectively. Formally, given dataset containing undesired demonstrations we wish to avoid, alongside a much larger unlabeled dataset consisting of both desired and undesired demonstrations, the goal is learn desirable policy that avoids undesired trajectories. To the best of our knowledge, only SafeDICE (20] directly addresses this setting by mixing the undesirable and unlabeled datasets, assigning negative weights the former, and then mimicking the mixed policy. This approach, how- ever, may suffer from the fact that the mixed policy is not necessarily desirable one to follow, which often the case in practice. There are existing methods, such preference-based RL 8 32 and Discriminator- Weighted Behavioral Cloning (DWBC) 45 that, while not specifically designed to address this problem, can be adapted to handle it.
Towards addressing this problem of avoiding undesirable trajectories, our key contributions are:
First, we formulate the new learning problem a cooperative training task ensure stable training. in contrast to prior imitation learning approaches where the objective is adversarial [18 28).
To efficiently solve the training problem using limited undesirable demonstrations, we introduce our algorithm, UNIQ. which employs an occupancy correction mechanism to recast the training objective, allowing the expectation over the undesirable policy to be unbiasedly approximated using unlabeled trajectories.
We evaluate our method on two popular benchmark environments, Mujoco and Safety-Gym using public datasets [38 [T]. Our experiments demonstrate superior performance compared to several state-of-the baselines.
2 Related Work
Imitation learning. Imitation learning is a key technique for learning from demonstrations. Be- havioral Cloning (BC) maximizes the likelihood of expert demonstrations but often struggles due to distributional shift [40]. To improve, Generative Adversarial Imitation Learning 18 aligns the learner's policy with the expert's using GANs 14). while SQIL [39 assigns simple rewards to expert and non-expert demonstrations to learn a value function. PWIL [9] uses the Wasserstein
distance [44] to compute rewards. While these methods show promise, they rely on online interaction,
which can be impractical. For offline learning, AICE (34) and ValueDICE (29) use Station- ary Distribution Correction Estimation (DICE) but face stability issues. Inspired by ValueDICE,
O-NAIL 4 introduced an offline method without adversarial training. IQ-learn [12]. popular approach, supports both online and offline learning and offers a state-of-the- framework with several variants developed based on it [2] 19 Unlike the above mentioned works, our focus in this paper on offline reverse imitation learning that aims to avoid undesirable trajectories, as opposed to imitating expert trajectories. As indicated earlier, this requires fundamentally different methods due to the nature of the problem.
Imitation Learning from Sub-optimal Demonstrations. There are two main research directions in this area. The first focuses on online and offline preference-basee imitation learning methods. Online approaches, such as T-REX [8) PrefPPO [31]. and PEBBLE [32). leverage ranked sub-optimal demonstrations to learn a preference- based reward function using the Bradley-Terry model [ While these methods achieve strong performance, they rely on interactions with the environment. In contrast, offline methods, such as those proposed by [24 22 17,, rely heavily on an extensive dataset of pairwise trajectory comparisons. SPRINQL. [19 addressed this reliance on large offline dataset, by utilizing demonstrations categorized into different levels of expertise, resulting in better performance with fewer comparison data points. While these methods concentrate on imitating
2

---

preferred (or expert) trajectories, our focus is on avoiding non non-preferred (or undesired) trajectories. This important distinction necessitates solving a different optimization problem and consequently change in methodology.
The second direction focuses the use of additional unlabeled datasets enhance learning from expert data. Beginning with DemoDICE [26). several DICE-based methods [33 25 46) have been developed utilize small of expert demonstrations, supplemented by larger unlabeled datasets, In addition to these DICE-based methods, DWBC 45 proposes a simple and efficient method based on training a classifier using positive-unlabeled learning 277. SafeDICE (0 presents DICE-based framework capable of learning from undesirable demonstrations, This method combines an undesirable policy (represented by an undesirable dataset) with a random policy (represented by a larger unlabeled dataset), assigns negative weights to the undesirable policy, and then applies standard DICE-base -ased approach (26 25 to mimic the combined policy by minimizing the KI. divergence between the learning policy and the mixed policy. The primary limitation of this approach is that, with the quality of the unlabeled dataset being unknown, imitating the mixed policy may not lead the desired learning outcome.
3 Background
Preliminaries. We consider a MDP defined by the following tuple . = P,, so)) S denotes the set of states, so represents the initial state set, is the of actions, r 8 A I defines the reward function for each state-action pair, and P: P x A + S is the transition function, i.e., P(s'|s, a) is the probability of reaching state s' S when action and is the discount factor. In reinforcement learning (RL), the aim is to find policy that maximizes the expected long-term accumulated reward: max, (s)p)) [r(s, a[]. where pr the occupancy measure of policy w: p=(s, a) = (1 - ( Σ P(s = s/T).
where
A is made state 8 E S.
Offline MaxEnt IRL The goal of offline MaxEnt IRL is to derive a reward function r(s, a) based on set of expert demonstrations, DE without interacting with the environment. Let 6 denote the occupancy measure of the expert policy. The MaxEnt IRL framework (4]16) aims to recover the expert's 's reward function by maximizing the gap between the expected reward under the expert's policy and the maximum expected reward across all other policies (as determined by the inner minimization).:
max min { (r(s, a)l [r., H(T, -
(1)
where (T,, Ep- log
is the discounted causal entropy of the policy given behavior policy μr, ,8 is the regularization strength, and (6(r) RSXA + R is convex reward regularizer. When is uniform, the objective reduces to the original MaxEnt objective [47]. In the offline RL setting, y is objective imposes a conservative KI. constraint on the learned policy, ensuring it stays close to the behavior policy. This helps mitigate the out-of-distribution issues that commonly arise in offline RI. 36 16).
[ssaa) ptals]
be the behavior policy that generated the offline dataset. As result, this
Inverse Q-learning (IQ-Learn) from expert demonstrations. Given a reward function r and policy म, the soft Bellman equation is defined as
x (a [s)
B (T r(s, a) + Y. (VT(ss) where V = E 8) (((s.a) a) - -3g
[(aas)) s)
The Bellman equation [ = Q is contractive and always yields a unique Q solution [2]]6]. In IQ-
Learn [12), they further define an inverse soft-Q Bellman operator T [Q] Q(s, a) - (s')
[12] show that for any reward function r(a, s), there is a unique Q" function such that B (²= = Q"
and for a Q" function in the Q-space, there unique reward function such that r T= This
result suggests that one can safely transform the objective function of Equation] I from r-space to the Q-space as follows:
max min (<TT, Q) E. [T) a)) - Ep, (T=(Q((ss)) a)) H( p)) ((T*(Q((s,))) (2)
Q
3

---

which has several advantages, namely, the objective function (TT, Q) is concave in π and convex in Q. Moreover. the inner problem min, क Q) has a closed form solution as exp(Q(s. (3/(/) E. exp(Q(x, (a,((((s)))/))))) /3)/8) As result, the maximin problem can be converted to non-adversarial problem the Q-space as:
max pE [T[Q] a))1 - (1 B)s (V) - - (TTQQ((s)))
(3)
where (I))(,,a) E. ((aas) exp((((sa)/3)) which is a softmax of the Q function. The reward
Q(s,a) - PPe (VQ and VQ
(8 Glog
function can then be recovered as ( a) = [[Q)ss Thus, in (3). the objective can be interpreted as training reward function (via a Q-function) that maximizes the expected reward under the expert policy while minimizing the overall expected reward. An important and appealing characteristic of the inverse Q-learning framework described above is that the learning objective is concave in Q, theoretically guaranteeing convergence to a unique solution in the Q-space. This property also ensures stable and robust training outcomes in practice. Later, we will show that this property does not hold in our learning- from-mistakes setting, prompting us to adopt more recent framework, Extreme Q-learning 113 to reformulate the training process using updates based on convex loss functions.
4 Undesired Demonstrations driven Inverse Q-learning UNIQ
We now describe our approach for avoidance (and not imitation) learning of undesirable demonstra- tions in the presence of an unlabeled (unknown quality) demonstrations dataset. First, we explain our novel MaxEnt objective that facilitates this avoidance learning problem (Section 4.1). However, directly solving this novel objective in reward space is inefficient, we next provide reformulation in the the Q-space (Section 4.2). Third, we leverage the extreme Q-learning method (Section 4.3) to solve this reformulated objective. Finally, we make our approach sample efficient by utilizing an unlabeled dataset of trajectories (Section 4.4).
4.1 Novel MaxEnt Objective
In our setting, we have a set of undesired demonstrations, denoted as DUN, along with a supplementary set of unlabeled demonstrations, denoted as DMXX The unlabeled dataset DMIX may contain mix of random, undesired, and expert demonstrations, and it will be used to support offline learning. Let J be the occupancy measure (or stationary distribution) of the undesired policy (represented by the undesired dataset).
Adapting the MaxEnt RL framework, the goal here is to find reward function and policy that minimizes: (a) the expected reward obtained by undesirable state, action pairs (part of undesirable trajectories); and (b) negative of the expected reward obtained by stale, action pairs not in undesirable trajectories. To this end, we introduce the following new learning objective:
min min [(TTT)) BAN [r(s, a)] Epp [r(s, - H(T, u) + +((r)
(4)
In our context, where the objective contrasts with the standard
scheme, the learning problem is longer adversarial in prior im- itation learning approaches [18] 28). Instead, it can be framed as a cooperative learning problem, where the ob- jective is to jointly identify policy and reward function that minimize the objective function L(F, r).. Solving { directly encourages the learning policy to deviate as much as possible from the undesired policy, which is derived from undesirable demonstrations (we give a detailed dis- cussion appendix). This approach contrasts with that of SafeDICE 200.. which minimizes the KI divergence between the learning policy and the mixed policy, which could suffer from low quality mixing datasets.
45
S
So
S₁
4:
S
Figure 2: An illustrative MDP.

---

Illustrative Example We provide an illustrative example to demonstrate how the proposed objec- tive guides the learning of good policy from undesirable demonstrations. Consider the sim- ple MDP shown in Figure 2 There are three trajectories from So to Ss: P1 = [So. S2, Ss], P2 = [So St, S4, Ss]. and Ps [So, S1. S3, Ss]. We treat P1 as an undesirable trajectory, while P2 and P3 are considered optimal. The undesirable dataset is Dund {pi}. and the unlabeled dataset includes all trajectories: Dunlabeted = (p..P2.ps) p3}. Under Equation (4), the training objective becomes:
min (R(p)) - (P" (pp)) + PM (pp2) + Pa (ps) + H(m)},
where R(p) denotes the total accumulated reward along trajectory p. e.g., R(p1) = r(Su. a1) 01)= r(S1, (g) r(5,, 07) (ignoring discount factor y for simplicity). PM (p) is the probability of trajectory p under policy x, and H(xx) ) is the entropy regularization term. Simplifying the objective yields:
-PT (p2))) P (pa) + (1 - 10-P"pp (p (pI)))((p)))
To minimize this expression, the algorithm is incentivized to assign higher rewards to Pa and Ps. and a lower reward to p1- Consequently, to minimize the expected reward (note the negative coefficients), the optimal policy should assign high probabilities to P2 and Ps. and low probability to P1- This encourages the policy to avoid undesirable behaviors and prefer optimal ones even in the absence of direct expert demonstrations.
4.2 Learning in the Q-space
Directly solving Eq. is not efficient. Instead, prior research indicates that transforming the reward learning problem into the Q-space and simplifying it can improve efficiency 12 17]. As discussed in Section there is one-to-one mapping between any reward function r and corresponding function Q in the Q-space. Therefore, the minimization problem in Eq. A can equivalently be transformed as:
min min { Q *
[1 Q) = pUN (Q)(s) a))] - p. [T (s, a)] H(x) + ve (T (Q),aa)))) (5)
where (=)))ss)a) = (V and V=(() = (a)s) (s, a) 3 log m(els)
Compared to the primary objective of the standard MaxEnt framework 18] 12 our objective function in Eq. is no longer adversarial with respect to Q and π.
Before we describe ways of simplifying objective in (5) we wish to highlight the properties that make IQ-leam and its variants efficient and effective: Property I: Closed-form optimization over π.
Property 2: Concavity of the simplified objective in the Q-space.
In comparison to imitation learning problem in the MaxEnt framework, since we negate the objective function, it alters its nature, turning concave function into a convex one. We show that despite this change, property is still preserved in the new objective. However, the simplified objective obtained by using property 1 is neither convex nor concave.
We first focus on Property 1, where we provide a closed form expression for policy, x.. If the regularizer function 4-(-) is non-decreasing. then the objective function L(T, Q) is convex in т. Furthermore, the minimization problem mint L(T, Q) retains a closed-form solution, thereby simplifying the learning objective as shown in Proposition 4.1
Proposition 4.1. The function L(T, Q) is convex in π and the problem ming L(T, Q) has a unique expl Q(a.a)/8)
Moreover, the learning objective function can be optimal solution at 9 a) = Σ. exp(((s,a'1703) simplified as: ming min, [L(T,, Q)) ming F(Q), where
F(Q) = E, 9 a)) - (1 - Bo ( + ()) ( 33 log (E. (alss) ((x, B)) and 9 (s, a) = Q(+ a) PPeee) (9
Proposition 4.1 shows that, similar to standard inverse Q-learning algorithms, our training objective can be framed as optimization problem over the Q-space, where the optimal policy be computed as the soft-max of the Q-function. Thereby preserving Property 1.
However with regards to Property 2, while in inverse Q-learning with expert demonstrations, the training objective is convex within the Q-space, this not the case in our context, where the objective F(Q) is neither convex nor concave in Q. We state this observation as follows:
5

---

Proposition 4.2. The learning objective finction F(Q) is not convex in Q.
This non-convexity arises from the presence of the term - VQ in the objective function, where VQ is a log-sum-exp of Q. which is itself convex in Q. This directly implies that F(Q) is not convex in Q. Such con-convexity is a significant disadvantage compared prior inverse Q-learning approaches that learn from expert demonstrations. In the following. we leverage the Extreme Q-learning method [3] to reformulate the training as updates based on convex loss functions, effectively mitigating the challenges associated with non-convexity.
4.3 Extreme Q-learning and Convexity
In the above formulations, the optimal value function V can be expressed as log-sum-exp of the Q- function, i.e., (2()) = 3 log(E.. pl(as) ()) exp(Q(ss, a)/$)). This log-sum-ext computation is often impractical as ((aas) is not explicitly known and the action space would be large. Furthermore, as discussed earlier, it is the primary cause of the non-convexity in our learning objective function. To ad- dress this issue, we leverage the Extreme Q-Learning (XQL) method [33 to update V simultaneously with Q-function. Specifically, we define the Extreme-V V function as: J(V E(s.))=e' - (t) -1
where @(s.a)-Vi V(s) where can be shown that minimizing J(VIQ) over yields an optimal
solution V (s) such that: (s) /3 log (Ea ((e)))))(e.))))) We then define the Q-learning
function conditional on V as F)QIV), which is defined as
V(QV)= E, Q(s a) - >E, [V('']] (1 -1--10 -0 (V()s (Qap)+++(Q(ssssa)
The minimization of our learning objective F(Q) can be achieved through the following alternating updates: (i) Update Q: Minimize the objective V)Q and (ii) Update V: Minimize the Extreme- function J(V)Q). This procedure is particularly convenient as both updates can be shown to have favorable convexity properties. Specifically, demonstrate that the loss functions are convex in their respective spaces:
Proposition 4.3. Under any convex regularizer v both F(QIV) and J(VIQ) are convex in Q and V. respectively.
4.4 Unbiased Q-Learning with Unlabeled Data
We now discuss the optimization in the Q-space. Directly solving ming F(QVV) using only the undesirable dataset would be inefficient due to its small size (as shown in Appendix E.S) Furthermore, directly combining the undesirable and unlabeled datasets for training is risky, as the unknown quality of the unlabeled dataset could lead to learning in the wrong direction. For instance, the model could unintentionally mimic low-quality demonstrations, might occur in methods like SafeDICE. To overcome this challenge, we address the following critical question: How can we unbiasedly estimate the objective minq F(QIV) while effectively utilizing the unlabeled dataset?
To address this question, we leverage the larger set of unlabeled data DMIX to enhance the offline training. To achieve this, we first let MIX be the occupancy measure (or stationary distribution) of the policy represented by unlabeled dataset. We rewrite the expectation over UN as: EE Eppx ( a)] =
[psx[T(s, 9 (s, a)). where 7(s, a) = (s,a), represents the occupancy ratio between UN and
UN (s.a)
Mix we then rewrite the learning objective as follows:
min ((Q)I)) = E, [T, a) a)l - (1- B), (aa) ((19)
(6)
where (s, = Q(s, (V In this approach, the expectation E, ()) a)l can be empirically approximated using the unlabeled samples from MIX where T(s, a) acts as an occupancy correction. This correction allows to leverage samples from the unlabeled dataset to unbiasedly estimate the expectation over the undesirable policy. A key challenge here that the occupancy ratio T(s, a) is unknown. To address this, we propose estimating the ratio by solving the following implicit maximization problem:
max
{g(0)}
1.5xA-100111

---

where g M, [log(1 - ((s, a)] + ppx (s,, a. The above formulation is similar to the discriminator- based formulations widely used in prior adversarial imitation learning work [ 23]. In fact, the objective function g(0) is strictly concave in d. Let d denote the unique optimal solution ('(a,))
then the oceupancy ratio can be estimated as T T(s, o) (s.a
So, our learning process can be broken down into steps. In the first step, learn the occupancy ratios by solving the maximization problems presented in Following this, we optimize the following problem learn Q and V functions.
min ((QIV) = Epsi
0- (' [
(
O (8, a) - -1) +
(8)
- 0*(ssa)
is important to note that we utilize the stationary distribution M (represented by trajectories in the unlabeled dataset) in the objective function in (8) However, thanks to the occupancy correction
('(o.)) T=d"(s,a) the outcome of the training theoretically independent of the quality of the unlabeled
policy p M This distinguishes our approach from SafeDICE [20]. where the performance heavily
relies on the quality of the unlabeled data.
4.5 Practical Implementation
Our algorithm consists of two main steps. In the first step, we construct network, 12 where á are learnable parameters. We then use samples from DUN and DM to estimate the objective function g(V). the second step, after obtaining e from the first step, we calculate the occupancy correction 5* and use it to update the Q and V functions as described in Section 4.3 For the Q-updates, we utilize the following empirical objective: F(QIV) (ssa,,")=>>a [7" a)r (s, 8') + (8 a, s'))
(1-7) E (s), where the reward function computed as r (s, a, 8') = Q(s, -V We choose the divderegee for the reward regularizer (1) = 0. - L, a popular choice inverse Q-learning algorithm. For policy extraction we utilize the following practical weighted behavior
cloning (WBC) with the objective: max {Ess.)) exp(A(s, a)) (ogs(a)))) }, where A(s,a) a) is
the advantage function defined as A(s, = (Q(s,a) - V(5))/ 37 35
Experiments
We evaluate algorithm in the context of unconstrained RL and safe RL (i.e., constrained RL) using the Mujoco (42) and Safery-Gym (38 21] domains, respectively. In the unconstrained task, we aim to test whether our algorithm can identify undesirable behaviors (from random or low-reward demonstrations) and avoid them. In the safety learning task, we focus on assessing whether our algorithm can satisfy safety constraints while maintaining reasonable rewards. Safe RL is also the main focus of SafeDICE in their experiments.
5.1 Experiment Setting
Baselines. We compare our algorithm against several baseline methods: (i) BC, which learns
from the entire unlabeled dataset (BC-mix); (ii) IPL I7. state-of-the-art algorithm preference based RL; (iii) DWBC [45] and (iv) LS-IQ 3 two leading IL algorithms for learning from expert demonstrations, adapted to avoid undesirable demonstrations; and (v) SafeDICE (00, specifically designed for learning from undesirable demonstrations We also include comparisons with modified baselines (similar to DWBC); however, due to space constraints, these are presented in Appendix E.7 Detailed implementation of these baselines is provided in Appendix D.3
Environments and Dataset Generation. For our experiments, we define the components of the undesirable dataset DUN and the unlabeled dataset D Mix In the Mujoco experiments, we use the official D4RL dataset [10), which consists of three performance levels: random, medium, and expert. We combine the random and medium levels to create the undesirable dataset UNN and use all three performance levels to construct the unlabeled dataset DMIX In the safety learning experiments, we train two policies: a constrained policy (using PPO-Lagrangian (38 to achieve low cost performance) and an unconstrained policy (using PPO 41 maximizes the return while ignoring the cost signals). Data is collected from both policies, with the unconstrained policy is used to generate high-cost
7

---

demonstrations for undesirable dataset DUIN and both datasets are combined to form the unlabeled dataset DMIX, More details about the datasets can be found in Appendix D.2
Evaluation Metrics. For the Mujoco domain, a higher return indicates better performance. In the Safety-Gym domain, the ideal outcome is to achieve the lowest cost possible while not significantly sacrificing the return. All experiments are run with at least 5 training seeds.
Experiment Concerns. Throughout the experiments, we aim to address several key questions: (Q1) How does UNIQ perform compared to other baselines in the context of unconstrained and safe RL? (Q2) How does the presence of an undesirable dataset contribute to the performance of UNIQ and other baselines?
(Q3) How does the quality of unlabeled dataset affect the performance of all the algorithmms? (Q4) What happens if we do not use the unlabeled dataset?
(Q5) We use WBC for the policy extraction; what if we directly extract the policy from the Q-function? In the main paper, provide experiments for (Q1), (Q2), and (Q3), while the other questions and some additional experiments are addressed in the appendix.
5.2
Main Comparison
BC-mix
IPI.
LS-IQ
DWBC
SafeDICE
UNIQ
Expert
Halfcheetah
Return
17.926.
3.2 0.7
-2.8
2.240.0
3.120.
75.7
84.6g7.6
Ant
Return
86.3+11s
84.3+1a.1
1.845.
84.6+124
4.442.
104.4:10.5
117.5+11.0
Hopper
Return
3.7
11.3gu4
10010
14811157
51.01a
73.5
110.8
Walker2d
Return
10.7
11.8
-0.140.
38.8
41.7444
105.9:40
107.9.00
Point-Goal
Return
27.1
26.920.1
-6.8 4.4
26.920.1
27 27.0.0.1
23.420.4
25.920.2
Cost 4
48.8229
52.7234
18.0 29.3
45.8ga.
46.81a
27.1
26.022.
Car-Goal
Return
31.1
31.7=0.
-0.6 2.5
32.8+0.7
33.5g0.
27.920.
26.2+0.7
Cost 4
52.014.2
54.41a.
58.5 41.7
47.413.
50.514
310222
23.6ga.
Return
17 17.620.7
16.920.9
-13.3
17.2+0.9
15.
12.6a14
14.21.
Point-Button
Cost J.
120.21100
124.81m1
11.0 10.2
123.52144
91.0104
23.0
80.6133
Return
17.6.0.7
17.220.
-8.6.t
17.1
17.4
12.7.11
14.0.0.s
Car-Button
Cost 4
241.6g153
257
28.0 + 27.0
249.21m0.s
201.3110.8
148.6
107.6gsa
Table 1: Comparison results for Mujoco (i.e. unconstrained RL) and Safety-gym (i.e. constrained RL) tasks.
In this section, we aim to answer question (Q1), comparing UNIQ with the mentioned baselines across four Mujoco tasks and four Safety-Gym tasks. For the Mujoco tasks, we use 5 trajectories each from the random and medium datasets to construct the undesirable dataset DUN, and 500, 500, and 100 trajectories from the random, medium, and expert datasets, respectively, to construct the unlabeled dataset MIX, For the Safety-Gym tasks, we use 100 high-cost trajectories for the undesirable dataset DUN and combine 1600 high-cost and 400 low-cost trajectories for the unlabeled dataset MiX To facilitate comparison, we provide the performance of policies learned from desirable data (the expert dataset in Mujoco and the constrained dataset in Safety-Gyme Gymm, which serve as the baseline for the highest performance. We note that our experiments require more samples than standard dearning-from-expe
experiments. This is intuitive, as learning from mistakes typically requires more examples than learning from good demonstrations.
The experimental results are shown in Table I Overall, BC-mix, LS-IQ, and IPL fail to meet our objectives (except in the Ant environment, where BC using all datasets seems achieve good performance). These experiment results also show that the state-of-the-an algorithm in imitation learning, LS-IQ, does not perform well in the "avoid bad"" scenario. DWBC outperforms BC-mix in Mujoco by using a discriminator to understand the undesirable dataset but struggles in the Safety- Gym tasks. Meanwhile, SafeDICE slightly outperforms DWBC in the Safety-Gym context. In general, UNIQ is the highest-performing algorithm in the Mujoco tasks and achieves closest (both in terms of return and costs) performance to policies learned from only the expert demonstrations (reported in "Expert" Column).
8

---

Halfcheetah
Ant
Hopper
Walker2d
20
50
50
50
expert
DWBC
SafeDICE
UNIQ
Figure 3: Comparison of results (MuloCo tasks) for different sizes of expert demonstrations included in the unlabeled dataset, shaded by standard deviation.
5.3 Ablation Studies
Impact of the Quality of the Unlabeled Dataset. To address Question (Q3). we modify the quality of the unlabeled dataset DM by adjusting the number of desirable demonstrations while keeping the same DUN as Section 5.2 The detailed results are shown in Figure Overall, UNIQ con- sistently achieves the highest performance and adapts effectively to variations in dataset quality. The corresponding training curves for Mujoco tasks and Safety-gym tasks can be found in Appendix
Impact of the Size of the Undesirable Dataset. We address (Q2): the impact of the undesir- able datasel on the final performance. Detailed results are presented in Figure 4 where the x-- axis represents the number of undesirable tra- jectories in DNN and keeping the same MIX as Section 5.2 Overall, the size of the dataset sig- nificantly influences the final cost across all al- gorithms. As the size of the undesirable dataset increases, UNIQ achieves the lowest cost, al- beit with a trade-off in return. Detailed learning curves for Mujoco and Safety-Gym can be found in the Appendix EE33 Furthermore, we discuss methods control the conservativeness (trade- off between return and cost) of our approach in the Appendix E.10
Point-Gov
Car-Goal
Return
10
10
50
100
350
306
072
100
200
597
Cost 30
30
702
200
Desirable
DWBC
SeAeDICE
UNIO
Figure 4: Comparison results for different sizes of the undesirable dataset in Safety gym, shaded by standard deviation.
Conclusion, Future Work
Conclusion. We have developed UNIQ. a principled framework designed to facilitate learning from undesirable demonstrations. UNIQ formulates the learning process as updates on Q- and V-functions based on convex loss functions, enabling unbiased training using unlabeled datasets of unknown quality. Additionally, UNIQ requires minimal hyper-parameter tuning, as it does not introduce any additional hyperparameters beyond those typically used in inverse Q-learning algorithms. Furthermore, UNIQ demonstrates superior performance in generating safe policies across several safe reinforcement learning experiments, outperforming other baseline methods.
Limitations and Future work. There are some aspects that have not been addressed in this paper, as they are too significant to be fully explored here. For instance, we assume the presence of only one set of undesirable demonstrations, whereas multiple datasets of varying quality could be leveraged to enhance the training. Additionally, each undesirable trajectory may not be undesirable in its entirety, as it could contain some good actions. Extracting the good parts from undesirable demonstrations could improve sample efficiency but introduces new challenges that warrant further investigation.
Acknowledgments and Disclosure of Funding
This research is supported by the Singapore International Graduate Award (SINGA) Scholarship from the Agency for Science, Technology and Research, Singapore.

---

References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page 1, 2004.
[2] Firas Al-Hafez. Davide Tateo, Oleg Arenz. Guoping Zhao, and Jan Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. In Eleventh International Conference on Learning Representations (ICIR), 2023.
[3] Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. In ICLR, 2023.
[4] Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods. arXiv preprint 2xXiv:2008.03525 2020.
[5] Claudine Badue, Rånik Guidolini, Raphael Vivaequa Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al. Self-driving cars: A survey. Expert systems with applications, 165:1 13816, 2021.
[6] Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv: /8/2.03079, 2018.
[7] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[8] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
[9] Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. In International Conference Learning Representations, 2021.
[10] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[11] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[12] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song. and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 14:402840334 2021.
[13] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learnin: Maxent rl without entropy. In International Conference Learning Representations (ICLR), 2023.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[15] Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature medicine, 25(1):16-18) 2019.
[16] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXi::1812.05905 2018.
[17] Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference based without reward function. Advances in Neural Information Processing Systems, 36, 2024.
[18] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[19] Huy Hoang. Tien Mai, and Pradeep Varakantham. Sprinql: Sub-optimal demonstrations driven offline imitation learning, 2024.
10

---

[20] Youngsoo Jang, Geon- Kim, Jongmin Lee. Sungryull Sohn, Byoungjip Kim, Honglak Lee, and Moontae Lee. Safedice: offline safe imitation learning with non- preferree demonstra- tions. Advances in Neural Information Processing Systems, 36, 2024.
[21] Jiaming Ji, Borong Zhang. Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 2023.
[22] Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. In International Conference on Machine Learning, pages 15753-15768. PMLR, 2023.
[23] Michael Kelly, Chelsea Sidrane, Katherine Driggs-Camphel and Mykel Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 1077-8083 IEEE, 2019.
[24] Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. In The Eleventh International Conference on Learning Representations, 2023.
[25] Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Kee-Fung Kim. Lobs- dice: Offline learning from observation via stationary distribution correction estimation. Ad- vances in Neural Information Processing Systems, 35:8252-8264, 2022.
[26] Geon- HronnHy Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HycongJoo Hwang. Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021.
[27] Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in neural information processing systems, 30, 2017.
[28] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribu- tion matching. In International Conference on Learning Representations, 2019.
[29] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribu- tion matching. International Conference on Learning Representations, 2020.
[30] Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hierarchical imitation and reinforcement learning. In International conference machine learning, pages 2917-2926. PMLR, 2018.
[31] K Lec, L Smith, A Dragan, and P Abbeel. B-pref: Benchmarking preference- based reinforce- ment learning. Neural Information Processing Systems (NeurIPS). 2021.
[32] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: fredback-efficieen interactive rein- forcement learning via relabeling experience and unsupervised pre- pre-training. In International Conference on Machine Learning, pages 6152-6163. PMLR, 2021.
[33] Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference on Machine Learning, pages 14639-14663. PML.R, 2022.
[34] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 2019.
[35] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09339, 2020.
[36] Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of markov decision processes. arXiv preprint a
2017.
[37] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off poolyy reinforcement learning. arXiv preprint arXiv:1910.00117 2019.
11

---

[38] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv:1910.01708, 7:1, 2019.
[39] Siddharth Reddy, Anca Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprini
2019.
[40] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[42] Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEF/RSJ international conference on intelligent robots and systems, pages 4026-5033 IEFE, 2012.
[43] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.015 2018.
[44] Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, de- scribing large systems of automata. Problemy Peredachi Informatsii 5(3):64-72, 1969.
[45] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighee offline imitation learning from suboptimal demonstrations. In Proceedings of the 39th International Conference on Machine Learning, pages
2022.
[46] Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offine imitation learning with suboptimal demonstrations via relaxed distribution matching. In Proceedings of the AAAI conference artificial intelligence, volume 37, pages 1016-11024, 2023.
[47] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, II. USA, 2008.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction include our main claims reflecting main contributions and finding.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussion on the limitations of our work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

All the proofs of the theorems and propositions stated in the main paper are provided in the appendix with clear references.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details on the environments and hyper-parameter settings in the appendix. We also uploaded our source code for re-productivity purposes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We have describe how to generale our data as well as provide it along with our submitted source code with sufficient instructions for their use.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctaaps.cc/publcc/guee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have detailed these information in the main paper and the appendix of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reported the mean scores and standard deviations for the result tables. We have also shown training curves constructed from scores and shaded by standard error. All the experiments are reported with multiple training seeds as well as different datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided these information in the "Computational Resource" section in our appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper provides a general offline imitation learning algorithm that only testing on the simulated environments. As such, we not foresee any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our training data are from official datasets and official simulator which have no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have provided clear citations to the source code and data we used in the
paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for re-producibilityy benchmarking purposes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We have crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper is not research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https://neurl)) hect:s///earrrsccc/ooofeeeesssse cc///oofeeeeeeese/222///... for what should or should not be described.
19

---

Appendix
The appendix contains the following details:
Missing Proofs: Refer to Appendix for proofs omitted from the main paper.
Experimental Details: We provide information on:
Baseline implementation (Appendix (D.3)
Hyper-parameter selection for each task domain (Appendix D.4
Task descriptions (Appendix) D.1
Generation of undesirable and unlabeled datasets (Appendix D.2p
Additional Experiments: We address the following remaining questions:
(Q2) How does the presence of undesirable dataset contribute to the performance of
UNIQ and other baselines? (Appendix E.1
(Q3) How does the quality of unlabeled dataset affect the performance of all the algorithms? (Appendix F.2 and Appendix F.3
(Q4) What happens if we do not use the unlabeled dataset (the training is solely based on the undesirable dataset)? (Appendix (E.5)
(Q5) We use WBC for the policy extraction; what if we directly extract the policy from the Q-function? (Appendix E.4)
Additionally, present experiments demonstrating the performance of UNIQ and other baseline methods on the dataset from the safeDICE paper (see Appendix E.12), performance in the Mujoco- velocity benchmark(see Appendix E.III comparison results using CVaR costs (see Appendix E.9). How does effect to the overall results? (Appendix [.6). What if we want to avoid all possible undesirable performance in Safetygym (low-reward low-cost, low-reward high-cost, high-reward high-cost)? (Appendix E.8). and how to control the conservativeness in the Safety-gym tasks(see Appendix E.
20

---

A
Pseudo Code of UNIQ
Below we present a pseudo code of our UNIQ algorithm.
Algorithm 1 UNIQ: UNdesired Demonstrations driven Inverse Q Q-Learning
Require: DUN DM Ss, me, N,,,N, Qu, and Ve, networks.
1: # Estimating the occupancy correction 7*
2: for certain number of iterations: i 1...Nr do
3: Update (Ø) to maximize g(0a).
4: end for
5: # Train Q and V, and policy functions
6: for certain number of iterations I...N do
7: Update Wy to minimize (QQ,,, IV,,)
8: Update Wa to minimize the Extreme-V function: ((V...Qu.:
9: Update 0 via the WBC: max,
[E(aa))DMi exp(A(s, a)) log x (als)
10: end for
B Missing Proofs
We provide proofs that are omitted in the main paper.
B.1 Proof of Proposition 4.1
Proposition.
The following statements hold:
(i) The function L(T, Q) is convex æ and the problem ming L(x, Q) has a unique optimal solution at n (s, a) E. exp(Q(e.a)77)))
exp(Q(s.a)/
(ii) The learning objective function can be simplified as:
min min {L(T, Q)} = -min (F(Q) = E (11 (so)) +
Q T
where (2 log (E. p(ass) exp(((s,.a)88)) and (s Q(s,a)
BEgw 20) (a) (s))..
Proof. We first express the second and third terms of the objective function L(T, Q) as:
BEpp [log π (s. a)
Ep [1](ss]) ((T) Ep- -1
m(als)
(ss.a) E.p10((s, - -Blog u(aass 7E, [V Ep, (V(s) - - (V*
= (1 - B P [V' (so)).
Thus, the objective function becomes:
L(m, 1 110 (sto) Σ (Q((s.0))EE(((>
We observe that (s) Ba- (Q(s log pla is concave in m. Therefore, both terms E, [Q(s, a) yE,/ [V (s]]) and -(1 E V (so) are convex in æ. Additionally, since vb(t) is convex and non- con-increasing in t, and Q EE (V*()) is convex in π, each
r(a [s]ss)
function (Q((s, a) yE,, [V* x (s')]) is convex in æ. Thus, combining all terms, we conclude that L(T, Q) is convex in
Furthermore, each term Q(s, -R (V*(' ((1 -),n- (so)), and (Q(ss, a) >yF..) strictly decreases in V* implying that the minimization of L(TT, Q) over is achieved when V*
21

---

is maximized for all s. Since V* is strictly concave in π, maximizing (*(s) over has a unique optimal solution:
exp(Q(s.a)/3)
π (als)
En exp(Q(s,a)/3)
This validates part (i) of the theorem.
For part (ii,, we observe that the problem max, and the optimal value is:
(T(s) has the optimal solution T as shown above,
T(als)Q(s,a) BT(als) s) log <(ass)
max (s)
(8) = max
p(a
= 8 log
Σ plals) pldds)exp(((ss))// (9()).
This directly leads
min min [[T, L Q)) min (F(Q) EAIN (r (s, -(1 - -7) [2(00)) + ())
Q
m
Q
as required.
B.2 Proof of Proposition 4.2
Proposition: The learning objective function F(Q) not convex in Q, even without any regularizer.
Proof. We rewrite the function F(Q) (without the regularizer (5())):
F(Q) = RNNU [Q(s, a) - 7B, (v - ( - (m, (VQ
where (s) = 3 log {E. ((saa)) is log-summexx function, which is convex in Q. Since (s) is convex Q, all the components associated with ()) in F(Q) are concave in Q. This directly implies the non-convexity of F(Q).
When the regularizer included, it can observed that if (-) is convex in Q, then (s a)) is also convex in Q, As a result, the objective function F(Q) takes the form of a difference--f-comme program.
B.3 Proof of Proposition 4.3
Proposition:
Under any convex regularizer v., both F(QIV) and (V(QQ convex in Q and V, respectively.
Proof.
We first write the loss function for the Quupdates
F(QIV) = E,us [Q(s.a) 7B. [V (1 Br + 7E. (VE.))
where:
The first term is linear in Q,
The second term is constant (since the V V-fuuction fixed when updating the Q-function), and
The last term involves convex function of Q, making it convex.
Thus, F(Q[V) is convex in Q.
For the convexity of J(V(Q), first recall its formulation:
(oaa) Via)
(Q(s.a) V(()
-1]
J(V(Q) [s,a)=
3
The convexity of (VV))) can be observed as follows:
22

---

The first term is an exponential function of V, which is convex in V,
The remaining terms either linear in V or constants.
Therefore, J(VIQ) is convex in V.
C Connection between UNIQ and Statistical Distance Maximization
In this section, we connect the new MaxEnt objective function of UNIQ with the concept statistical distance. Specifically, we show that solving the new MaxEnt objective min, min, L(x, r)) in (4) is equivalent maximizing a statistical distance between the occupancy measures of the learning policy and the undesirable policy. In particular, if the feasible set of the reward function is unrestricted, the following equality holds:
min min [L(T, r)} = max (d (0=, P'N) (T))
where (0=, = 10 (p" PUN), and 2b" the convex conjugate of the convex function 1) i.e.,
v (1) sup (lz v=))).
To verify 99, we write the objective function as:
L(T,T)) Σ r(s, a) (s -()) (T)).
Thus, the minimization of L( r) over can be expressed as:
min = (T) + min
-s Σ r((s (p - (s, + v(())
8,a
(T) max
Σ ass -app-puug (U) 061)
Since the feasible set of the reward function is unrestricted, we have:
max (rr pap=-pan ,UN) - w()) = (0= UN
This allows us to rewrite the training problem as:
min min (L(T,r)} = min {11(T) 60" - |N
max { (v=" (v'(0" - PUN) [[))
max (d ,NUN - H(m)
We thus obtain the desired equation in (.. proving the equivalence.
D
Experimental settings
D.I
Environmental Details
D.1.1 Mujoco
The MuloCo locomotion benchmark focuses on a set of environments designed to evaluate the ability of reinforcement learning algorithms to control agents with complex, multi-joint dynamics. These tasks involve training agents to move efficiently and stably in simulated environments. The illustrations are shown in Figure
23

---

Halfcheetah
Ant
Hopper
Walker2d
Figure 5: Mujoco environments.
D.1.2 Safe-Gym
Safe-Gym is collection of reinforcement learning environments designed with a focus on safety, built on top of the OpenAI Gym framework. It introduces constraints that simulate safety-critical scenarios commonly encountered in real-world applications. In Safe-Gym, agents are rewarded for completing task-specific objectives but face penalties for violating safety constraints, such as surpassing speed limits, colliding with obstacles, or entering restricted zones. These constraints enable Safe-Gym to replicate environments where safety is paramount, including robotic navigation in congested areas, autonomous vehicle control, and industrial automation. The environment features two types of agents: Point (an easy agent) and Car more challenging agent), as well as two types of tasks: Goal (easy) and Button (hard), Additionally, the environment dynamics change with each new episode, introducing variability and increasing the complexity tasks. Illustrations of these tasks are shown in Figure
Point-Goal
Car-Goal
MhittBuutonn
Car-Button
Figure 6: Safety-gy gym environments.
D.1.3 Mujoco-velocity
From original Mujoco, Mujoco- Velocity is a specialized environment within the Mujoco physics simulation suite, focusing controlling the velocity of two specific agents: Cheetah and Ant. These agents must complete locomotion tasks while adhering to safety constraints on their speed. The goal to balance task performance with maintaining safe velocity limits. For instance, Cheetah must run as fast as possible while staying within predefined speed bounds to avoid penalties, mimicking real-world scenarios where exceeding speed limits cause system failure unsafe operations. Similarly, Ant must navigate through its environment without violating velocity constraints, ensuring stability and safety.
D.2 Dataset generation details
In this paper, tackle the problem of Offfine Learning with Undesirable Demonstrations by utilizing two datasets:
Unlabeled dataset DMIx A large dataset comprising both desired and undesired demon- strations, reflecting real-world data (e.g., chat conversations, driving behaviors, treatment decisions, etc.).
24

---

Undesired dataset DUN; A smaller, accurate dataset containing demonstrations that exhibit behaviors we aim avoid.
D.2.1 Mujoco dataset
We use the offical D4RL, dataset with three different performance: random, medium (sub-optimal), and expert. We then combine them to collect the dataset as follow:
Unlabeled dataset DiXx We mix all three dataset into large unlabeled dataset with specific ratio.
Undesired dataset DUN, We mix random and medium dataset with the same ratio (1 1).
D.2.2 Safety-gym and Mujoco-velocity dataset
We simulate this scenario using the Safety-gym and Mujoco-velocity environments. First, we train
both unconstrained and constrained policies using PPO [41 and PPO-lagrangian 38 We then collect the training datasets as follows:
Unlabeled dataset Mix, We roll out the constrained and unconstrained policies, mixing them at ratios of (1 : 4).
Undesired dataset DUN; We roll out the unconstrained policy, gathering trajectories that violate the constraint.
The quality of Safety-gym dataset are shown in Table while details information Mujoco-velocity datasets are shown in Table 3
Point-Goal
Car-Goal
Point-Button
Car-Button
Mean Unconstrained return
26.8 + 1.1
35.2 + 2.1
21.49 + 4.87
17.82 +: 5.57
Mean Constrained return
2..3
25.7 6.6
13.71 6.44
14.99 +: 8.45
Mean Unconstrained cost
57.8 + 38.9
61.9 = 48.1
138.9 77.3
236.5 115.9
Mean Constrained cost
22.3 28.0
21.0 = 31.0
27.4 34.1
113.5 33.2
Cost Threshold
25.0
25.0
25.0
100.0
Table 2: Safety-Gym expert policies performance.
Cheetah
Ant
Mean Unconstrained return
3027.4 = 400.6 2751.2 + 11.6
2972.2 + 1020.0
Mean Constrained return
2830.0 = 145.5
Mean Unconstrained cost
626.7 95.0
624.0 231.0
Mean Constrained cost
14.1 + 4.5
18.7 + 4.4
Cost Threshold
25.0
25.0
Table 3: Mujoco-velocity expert policies performance.
D.3
Baseline implementation details
D.3.1
BC
We use the orginal BC objective:
min E, Eg log (als)
(10)
25

---

D.3.2 LS-IQ
We use the official implementation of LS-IQ in this link We modify the objective to fit with our scenario when undesirable dataset available:
min E BAN [(Q(s, a) Qmin) + aD [(Q(s, -maaz >
where V(s) = Q(, Hz)); Qmin Tmin 7); and Tmán and Imaz predefined.
D.3.3 IPL
We use the official implementation of IPL 17 from this link The only difference between our setting and IPL is the pairwise dataset. We create pairwise comparisons from the unlabeled dataset and the undesired dataset and train the Q-function with the following new loss function:
exp (T) ax11x)
Po- > UN
exp (T=Q) MIX _.x1xx exp
where:
(VT,
D.3.4 DWBC
We use the official implementation of DWBC discriminator that assigns to the undesired dataset DUN and the unlabeled dataset DMIX while keeping the Positive Unlabeled learning technique from the paper:
45
from this link We modify the algorithm to train
min (sa) |- log do (s, a, log r]]
+ E (s,a)-DiN [- log (1 dø (s, a, log 7T)]
()B))s.)) [- log (1 dg (8, a, log r))]
D.3.5
SafeDICE
We use the official implementation of SafeDICE (20 from this link, As the algorithm has been designed to solve this problem, we do not make any further modifications.
D.4
Hyper-parameter selection
For fair comparison, we keep the same basic hyper-pann parameters across all the baselines which are detailed as follow for Mujoco, Safety-gym gym and Mujoco-velocity tasks Table
HYPER PARAMETER
MUJOCO
SAFETY-GYM
MUJOCO-VELOCITY VELOCITY
ACTOR NETWORK
256x3
256x3
256x3
CRITIC NETWORK
256X3
256X3
256X3
VALUE NETWORK
256X3
256X3
256X3
TRAINING STEP
1,000,000
1,000,000
1,000,000
GAMMA
0.99
0.99
0.99
LR ACTOR
0.0003
0.0001
0.0001
LR CRITIC
(0.0001 .0.0003)
0.0003
0.0003
LR VALUE
0.0003
0.0003
0.0003
DISCRIMINATOR
0.0001
0.0001
0.0001
BATCH SIZE
256
256
256
SOFT UPDATE CRITIC FACTOR
[0.001,0.01]
0.005
0.005
Table 4: Hyper parameters.
In UNIQ, the and parameters selected in range [0.5, 10.0j. Lastly, apply state normalization for Mujoco and Mujoco-velocity datasets as follow:
Snormalized
26

---

Where:
P
S'ED
(8' 1u)2
D
SEED
D.5
Computational Resource
Our experiments were carried out on a GPU cluster equipped with 8 NVIDIA RTX 3090 GPUs. Each experimental setup run with five distinct training seeds in parallel, sharing single GPU, eight CPU cores, and 64 GB of RAM. Within this shared configuration, completing one million training steps across all five seeds took approximately 30 minutes. The software environment was built using JAX version 0.4.28, with support for CUDA 12- --ecciiiic.lll. CUDA version 12.3.2 and cuDNN version 8.9.7.29.
important to note that in the Safety-Gym domain, evaluation relatively slow due to the use of 50 different environment random seeds for each (we have 100 evaluations in total). While the training phase takes around 30 minutes, the extensive evaluation increases the total runtime per experiment to approximately 2 hours.
27

---

E
Additional Experiments
E.1 Performance with Different Sizes of undesirable Dataset
In this section, we provide a complete experiment to evaluate the contribution of the undesirable dataset to our algorithm. We test our method with increasing size of the undesirable dataset, DUN = {5, 10, 20, 50, 100, ]00 for the Mujoco tasks and DUN 225, 50, 100, 200, 300, 500}. The full experiment results are shown in Figure (Mujoco) and Figure (Safety-Gymi
HalfCheetah
Ant
Hopper
Walker2d
UN=5
0.0
0.0
1.
UN=10
100
000
0.0
3.0
1w5
UN=20
100
100
100
100
000
0.0
0.0
1.0
1e0
1el
1eb
1e5
UN=50
100
100
100
100
80
50
50
50
000
0.5
1.
0.0
0.5
1.0
0.0
1.0
9.0
2.5
1.0
UN=100
100
100
60
50
00
60-
960
0.5
0.5
0.0
1.0
0.0
05
1.0
UN=200
100
100
0.0
1.5
SafeDICE
DWBC
UNIO
Expert
Figure 7: Comparison with different size of Undesirable dataset in Mujoco environments.
28

---

Point-Good (Return)
Point-Goal (Cost)
Car-Goal (Retur))
Car-Goal (Cost)
UN=25
20
60
40-
20
0.0
168
105
165
168
UN=50
60
20
01
0.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
UN=100
60-
40
20
0.0
0.6
1.6
0.0
0.5
1.0
0.0
0.6
1.0
0.0
0.5
1.8
1wA
Twek
168
60
UN=200
20
20
20
-20
9eh
UN=300
20
20
-20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
UN=500
20
20
40-
20
-20
1.0
0.0
196
166
166
166
SafeDICE
DWBC
UNIQ
Figure 8: Comparison with different size of Undesirable dataset in Safety- gym environments.
29

---

E.2
Performance with Different Number of Undesirable Demonstrations in the Unlabeled Dataset
We further change the number of undesirable demonstrations in the unlabeled dataset D to examine how it impacts the performance of the algorithm. The experimental results are shown in Figure 9 The results show that when the quality of the dataset decreases significantly, the performance of all algorithms worsens. However, UNIQ still achieves the highest performance across all environments.
R(250)-M(250)
R(500)-M(500)
R(1000)-M(1000)
HalfCheetah
100
100
100
50
50
50
0
0.5
0.0
0.5
0.0
1.0
1e8
1e8
1e6
100
100
100
Ant
50
50
50
166
1e6
1e8
100
100
100
Hopper
50
50
50
0.0
0.0
0.0
1e6
1e6
1e6
Walker2d
100
100
100
50
50
60
0.0
0.5
0.0
0
1e8
1e8
1e6
SafeDICE
DWBC
UNIQ
Expert
Figure 9: Comparison with different size Undesirable dataset in Mujoco environments.
30

---

E.3
Performance with Different Number of Desirable Demonstrations in the Unlabeled Dataset
We also test how changing the amount of desirable data in the unlabeled dataset affects the perfor- mance. The experimental results are shown in Figure I0 (Mujoco) and Figure T (Safety-Gyym We observe an increasing trend in performance as the size of the desirable data increases (higher returns Mujoco and lower costs Safery-Gymi
HalfCheetah
Ant
Hopper
Walker2d
Expert = 50
0.0
0.5
0.5
0.0
0.5
0.0
0.5
1.0
Expert = 100
100
100
100
100
30
60
60
0.0
0.5
1.0
0.0
0.5
10
0.0
05
100
0.0
0.5
1.0
1e6
feb
165
Expert 200
0.0
0.0
0.0
106
160
168
Expert = 300
100
50-
50-
60
0.3
0.5
1.0
6.6
0.5
1.0
0.0
05
1.0
0.0
0.5
1.0
SafoDICE
DWBC
UNIQ
Export
Figure 10: Comparison with different size of Undesirable dataset in Mujoco environments.
31

---

Constrained = 1600 Constrained = 400 Constrained = 100
Point-Goal (Return)
Point- (Cost)
Car-Goal (Return)
Car-Goal (Cost)
20
40-
20-
-20
1,0
0.0
10
166
166
166
166
20
40
20
1:3
1.0
0.0
0.5
100
166
166
166
166
20
40-
60-
40-
20
-20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
10
1e6
1e6
1e6
SafeDICE
DWBC
UNIQ
Figure 11: Comparison with different size of Undesirable dataset in Safety-gy environments.
E.4 Direct Policy Extraction from Q-functions
As actor update objective uses Weighted Behavioral Cloning, question arises: what if directly extract the actor from the Q-function [123]] We conduct experiments to evaluate the performance of UNIQ when the actor is learned directly from the Q-function. The results are presented in Figure 12 and Figure 13
Halfcheetah
Ant
Hopper
Walker2d
100
100
100
100
50
50
60
50
9.0
1.0
-50
0.0
1.0
0.0
0.5
1.0
0.0
16E
165
1ed
166
UNIQ extraction)
UNIQ (weighted BC)
Figure 12: compare UNIQ BC update with Q extraction update in D4RL
32

---

Return
Cost
CVaR cost
30
Point-Goo Goal
20
40
400
10
20
200
0.0
0.5
0.5
1.0
0.5
1.0
166
166
1e8
40
300
60
Car-Goal
200
20
100
0.0
1.0
1e6
1e6
1e6
UNIQ (Q extraction)
UNIQ (weighted BC)
Figure 13: compare UNIQ BC update with Q extraction update in Safetygym
E,5
BC without Unlabeled Dataset
As UNIQ uses an unlabeled dataset learn by avoiding undesirable behaviors and learning from the remaining parts of the dataset, an interesting question arises: what if we only avoid the undesirable dataset (no unlabeled dataset is given)? To explore this, we train a Behavioral Cloning (BC) agent that, instead following the dataset, explicitly avoids the actions from the undesirable dataset by minimizing the probability of those assigned actions, denoted BC-UN. The detailed results are shown in Figure 14 (Mujoco) and Figure 15 (Safety-Gym).
Halfcheetah
Ant
Hopper
Walker2d
75.
100
100
50-
25
100
070
1.0
0.0
0.5
1e0
1e5
fed
1e6
BC-UN
UNIQ
Figure 14: BC only avoid undesirable data D4RL
33

---

Return
Cost
CVaR cost
30
60
Point-Gon Goal
200
20
40
10
100
20
0
0
0.0
0.5
0.5
1.0
0.5
1.0
166
166
1e8
40
150
60
Car-Goal
100
20
0
0.0
1.0 1e6
1.0
1e6
1e6
BC-UN
UNIQ
Figure 15: BC only avoid undesirable data in Safetygym
E.6 ablation study with different
Task
1
Non-converged
Converged
HalfCheetah
18.5 + 5,6
64.9 = 5.2
75.7 + 6.8
Ant
88.5 + 8.2
95.2 + 12.4
104.4 = 10.5
Hopper
57.4 + 12.4
58.0 + 14.6
73.5 + 20.6
Walker2d
12.8 = 7.7
72.0 6.1
105.9 = 4.0
Table Ablation study with different 5: We maintain the same dataset construction in the main experiment.
To assess the effect of occupancy ratio estimation algorithm, conducted additional ablation studies using three configurations: (i) uniform occupancy ratio (r = 1), (ii) non-converged esti- mation of 7, and (iii) properly converged т. The results of this ablation study (for the unconstrained RL setting) are shown in Table
E.7 Additional comparison with modified baselines for avoiding undesirable
Task
BC-remove-bat
IS-WBC
ILID
UNIQ
HalfCheetah
2.3 = 0.0
2.2 0.0
2.1 0.0
75.7 + 6.8
Ant
36.2 = 19.6
73.4 12.3
106.44 4.3
104.4 + 10.5
Hopper
19.7 18.9
7.6 + 5.0
5.9 + 6.5
73.5 + 20.6
Walker2d
37.8 33.3
18.8 + 19.3
69.6 + 13.0
105.9 4.0
Table 6: Additional comparison with BC-remove-bart IS- WBC, and ILID: while keeping the dataset construction consistent with the main experiment.
Since our method is designed to learn from undesirable demonstrations, we evaluate it alongside several baseline algorithms that were originally developed for expert demonstrations but have been adapted for our setting:
BC-remove-bad: We first train classifier to detect and remove undesirable demonstrations from the unlabeled dataset. Behavior Cloning (BC) is then applied the filtered data to learn the policy.
ISW-BC: We modify the original discriminator objective to better suit the learning- ffom- undesirable-ddmoonsttaiie framework. Specifically, train the discriminator differ-
34

---

entiate between undesirable and unlabeled demonstrations, rather than between expert and non-expert data as in the original method. This adaptation helps guide the policy away from undesirable behaviors without requiring expert data.
LLI: We adapt the original ILID approach by removing all states in the unlabeled dataset that resemble those in the undesirable dataset. However, the original ILII also includes regularization component that prevents the learned policy from deviating significantly from BC-trained expert policy. Since expert demonstrations are unavailable in our setting, this regularization is excluded in our adaptation.
The comparative performance results are reported Table Overall, UNIQ yell the best performance in Mujoco tasks.
E.8 Performance with 4 category types (low-cost low-reward, low-cost high-reward, high-cost low-reward, high-cost high-reward) dataset in Safety-gym
DWBC
SafeDICE
UNIQ
Expert
Return
20.1 + 1.1
19.9 0.4
24.3 + 0.8
25.9 + 0.2
Point-Goal
Cost
47.0 +: 4.5
41.7 5.0
26.0 + 4.4
26.0 + 2.6
Return 20.9 = 1.5
22.0
22.5 +
26.2 1 0.7
Car-Goal
Cost 30.3 3.3
30.5 + 7.7
19.9 + 2.8
23.6 + 2.8
Table 7: Performance of 4 different category of performance types on Point-Goal and Car-Goal Tasks.
The objective of this experiment is to evaluate whether the algorithm can successfully avoid all types of undesirable demonstrations in the Safety- environment. While our main experiments (Table T primarily assess safety performance with minimal return degradation, this experiment focuses on the algorithm's 's ability to comprehensively reject undesirable trajectories.
To achieve this, we construct dataset based on four categories of Safety-Gym performance: high- return low-cost (H-L), high-return high-cost (H-H), low-return low-cost (L-L), and low-return high- cost (L-H). The dataset is divided into two subsets: the undesirable dataset D which consists of 50 trajectories each from the H-H, L-L, and L-H categories (totaling 150 trajectories); and the unlabeled dataset DIXX which contains 500 trajectories from each category, resulting in 2,000 trajectories in total.
The experimental results are reported in Table may initially appear and UNIQ exhibit reduced cost despite the more challenging data. However, this outcome is likely due to trade-off inherent the policy behavior: lower rewards are often correlated with lower costs. In particular, policies that prioritize minimizing cost may opt for conservative behaviors, such as remaining stationary, which avoids hazards but also leads to diminished task progress and, consequently, lower returns.
35

---

E.9
Comparisons with CVaR 10% Cost for Syf Tasks
also report the CVaR 10% cost for Safety-gym tasks, supporting the result of the Table | with CVaR is the mean of 10% highest in cost trajectories during the evaluation process. The full results are shown Table g
BC-mix
LS-IQ
IPL
DWBC
SafeDICE
UNIQ
Return
27.1110.1
-6.8 4.4
26.910.1
26.911.11
27.0.10.1
23.411..4
Point-Goal
Cost
48.8:12.9
18.0 +: 29.3
52.713.4
45.8.13.4
46.8.1..1
27.1113.0
CVaR
115.4.17.7
133.3 =: 240.7
117.918.2
110.4.7.8
111.011.66
85.9 18.9
Return
34,1
-0.6 2.5
34.7+0.33
32.8+0.7
33,5
27.9110,8
Car-Goal
Cost
52.0=4.2
58.5 41.7
54.413.7
47.4=3.8
50,5
31.0=2.8
CVaR
13..8++00.6
311.1 = 208.9
134.948.7
123.2+10..6
128.5+
93.3 + 8.4
Return
17.6
6.7
16.9
0.9
15.1 = 0.5
12.6 1.4
Point-Button
Cost
1 10.0
11.0 10.2
124.8411.3
123.5 14.4
910=
23.0
CVaR
311.0 47.8
58.2 89.1
307.2 47.3
309.6 + 63.7
21.2
98.7 33.1
Return
17.6 77.6+0.7 0.7
-8.6
17.2
17.1 1.0
17.4 0.6
12.7 + 1.1
Car-Button
Cost
241.6 + 15.3
28.0 27.0
257
249.2 + 20.9
201.3 10.8
148.6 + 18.7
CVaR
545.3 + 45.6
195.9 = 204.1
553.2 + 35.7
550.5 1 66.3
410.5 + 34.3
449.8 + 63.4
Table 8: Full comparison results in Return, Cost, and CVaR 10%.
E.10
Controlling Conservativeness in UNIQ
Point-Goal
Car-Goal
31
25.0
Return
24.5
30
24.0
23.5
23.0
0.1
0.3
3.0
10.0
0.01
0.1
0.3
3.0
10.0
Beta
Beta
50
Cost
30-
30
0.01
0.1
0.3
1.0
3.0
10.0
0.01
0.1
0.3
1.0
3.0
10.0
Figure 16: Comparison results of UNIQ with different selections.
The main paper demonstrates the policy returned by UNIQ achieves significantly lower costs (indicating safety) but, in some cases, also lower rewards compared to other imitation learning baselines. While this aligns well with our objective of learning safe policies by avoiding unsafe demonstrations, it also raises concerns about the algorithm's conservativeness.
In this section, we show that the conservativeness of UNIQ can be effectively controlled by introducing parameter to the Weighted BC formulation. Specifically, we adjust the conservativeness of the algorithm by adding a parameter 3 the Weighted BC update:
Σ
expl(Qu, (s, - Via, (8)) * 3) log (als))
(M.0)~DMix
36

---

When 3 = 1, the Weighted BC theoretically returns the exact policy derived from Q-learning, as reported in the main paper. In contrast:
As B - 0, the Weighted BC returns random policy.
As - oo, the resulting policy becomes deterministic, always selecting the best action with probability
Thus, by varying 3, we can deviate the outcome of the Weighted BC from the policy given by Q-learning, reducing the conservativeness the learned policy.
To experimentally demonstrate this, we vary 3 and report the corresponding returns and costs on four MuloCo environments. The results presented in Figure [6] Table 9 showing how different values of / impact the trade-off between safety and performance.
Figure T6 demonstrates that UNIQ achieves its safest (and most conservative) performance when 3=11 1. At this value, the policy prioritizes minimizing costs, making it the most risk-averse option. However, as 3 deviates from 1, both the cost and return increase. This indicates that the Weighted BC formulation produces less conservative policies that are less safe but capable of achieving higher rewards.
Table provides more detailed breakdown the costs and returns for different values B. The results show that UNIQ can effectively balance safety and performance: by adjusting 3, it is possible to achieve a safer policy (i.c., lower cost) while maintaining competitive returns (compared to other baselines). This adaptability highlights the flexibility of UNIQ.
When safety is critical, setting 3 ensures the conservative policy, aligning with the objective of avoiding unsafe demonstrations. On the other hand, by varying 8, one can tune the trade-off to achieve policies that are less safe but yield higher rewards, making UNIQ suitable for range of scenarios depending on the desired practicality across different applications with varying safety requirements.
balance. This versatility demonstrates its
DWBC
SafeDICE
UNIQ (0.01)
UNIQ (0.1)
UNIQ (0.3)
UNIQ (1.0)
UNIQ (3.0)
Return 26.910..
27.010.1
25.1 0.1
25.0
24.5 0.2
23.4.00.4
24.3 1 0.2
Point-Gool
Cost 43.8
46.813.1
44.0 3.6
40.5 #32
32.5
27.11300
30.8 4: 2,9
Return
32.8.10.7
33.510.7
30.6 0.5
30.1 1 0.5
0.6 4..2
27.9.10.8
28.9 1 0.6
Car-Goal
Cost
47.41.3.8
50.514.0
46.4 1.7.
45,9 4.2
31.011.88
40.4 1 3,4
Table 9; Comparison with different
37

---

E.11 Full Numerical Experiment for Mujoco Velocity Tasks
We evaluate our method on two MuJoCo velocity tasks: Cheetah and Ant. we test the method with varying sizes of the undesired dataset, annotated as "env-UN= {1. 10)" while the unlabeled dataset DMix is combined from 1600 high-cost and 400 low-cost trajectories. The detailed results are summarized in Table T0 and learning curves are shown in Figure 17 and Figure 18 Overall, increasing the size of the undesired dataset helps SafeDICE and DWBC achieve higher performance, while UNIQ reaches its peak performance with just a single undesired trajectory.
DWBC
SafeDICE
UNIQ
Return
3135.61127..4
2841.9:56..1
2662.01133.1
Cheetah-UN=10 Cost
311.0=116.0
550.2:13..5
0.0.1.00
CVaR
89.07110.0
682.2:1:4. 14.4
0.0.1..0
Return
3430.9+107.5
2860.8
2661.2+29.7
Cheetah-UN=5
Cost
578.8119.6
$53.91125.3
0.0.1..0
CVaR
309.2-16.3
686.71117.3
0.0.10.0
Return
3720.7133.2
2910.0:61.8
2755.3=23.8
Eheetah-UNas
Cost
823.0=17.5
575.5#23.0
0.0:0.00
CVaR
916.4=5.0
702.2+20.00
0.0:
Return
2225.0-1759.3
2713.0+:56.2
2850.5
Ant-UN=10
Cost
170.5=162.8
439.7::57.7
15.2
CVaR
795.0=103.7
668.7114.6
24.6:1:3.7
Return
2210.0++655.7
2727.4+-99.8
2838.2-
Ant-UN=5
Cost
196.5+146.8
364.4435.3
13.147.5
CVaR
805.7116.4
671.2116.0
22.11:00.5
Return
2259.41153..8
2724.4190.7
2841.41214.9
Ant-UN=1
Cost
507.5+147.8 506.5+40.3
16,9+7.1
CVaR
789.3499..3
685.8+16.4
27.2+t9.6
Table 10: Full comparison between UNIQ and other baselines in Mujoco- domain. With decreasing of undesirable dataset size, the performance of DWBC and SafeDICE become worse. In contrast, UNIQ able achieve highest performance with just a single undesirable trajectory.
38

---

Return
Cost
CVaR cost
HalfChheetal--NNee -aaaahh---100
3000
750
400
2000
500
200
1000
250
0
0.0
0.5
0.0
0.5
1.0
0.5
1.0
166
166
1e8
HalfCheetah-UNNa
600
3000
400
2000
500
200
1000
0
0.
0.0
0.5
0.5
1.0
1e6
1e6
1e6
MirChheahh--NNe
750
3000
2000
500
500
250
1000
0
0-
0.0
0.5
0.5
1.0
0.5
1.0
1e6
1e6
1e6
DWBC
SafeDICE
UNIQ
Figure 17: Cheetah task with unlabeled Cataset(400-1600) and different undesired dataset.
E.12 Performance with the Dataset Employed in the SafeDICE Paper
As we are using a different dataset from the SafeDICE dataset, we also provide comparison with the dataset from SafeDICE paper. The detailed performance of the expert dataset is shown in Table []
Point-Goal Goal
Point Button
Mean non-preferred demonstrations cost
20.018
21.933
Mean preferred demonstrations return
19.911
8,286
Mean non-preferred demonstrations cost
107.977 13.798
166.099
Mean preferred demonstrations return
12.085
Table 11: SafeDICE dataset performance.
We mix 300 preferred demonstrations and 1200 non non-preferred demonstrations for the unlabeled 100 non-preferred demonstrations for the undesired dataset. The performance is shown in Figure 19 It is clearly that our method can achieve lower cost than SafeDICE.
dataset and
39

---

3000
750
Ant-UN=10
2000
500
200
1000
250
0
0
0:
0.0
0.5
1.0
1e6
1e6
1e6
3000
750
Ant-UN=5
2000
400
500
1000
200
250
0
0
0.0
0.0
0.5
1.0
1e6
1e6
1e6
3000
600
750
Ant-UN=]
2000
400
500
1000
200
250
0
0.
1.0
0.0
0.5
1.0
0.5
1.0
1e6
1e6
1e6
DWBC
SafeDICE
UNIQ
Figure 18: Ant task with unlabeled dataset(400-160M) and different undesired dataset.
Return
Cost
CVaR
20
60
Point- Point-Goal Goal
15
150
10
40
100
20
0.0
0.5
0.5
1.0
0.5
1.0
1e6
1e6
1e6
30-
Point-Bi Button
100
20
10
100
0.00
0.5
0.0
0.5
1.0
0.5
1.0
1e6
1e6
1e6
SafeDICE
UNIQ
Figure 19: Comparison between UNIQ compared to SafeDICE in their dataset.
40

---

E.13
Occupancy Ratio Estimation Analysis
To further address the reviewer's insightful comments regarding the accuracy of diseriminator- based occupancy ratio estimation, we conduct an additional analysis to empirically examine how well our discriminator approximates the true occupancy ratio under controlled setting.
Motivation. As noted by the reviewer, density ratio estimation using a discriminator can be sensitive when the underlying distributions of undesirable and unlabeled data differ significantly. However, it is also important to understand the behavior of the estimator when both datasets originate from the same distribution, where the ground-trutt -ound-truth occupancy ratio 7 (s(s, a) 1 for all (s.a). This experiment allows us to isolate the intrinsic estimation accuracy the discriminator, independent of distributional shift effects.
Experimental setup. We construct synthetic datasets by randomly sampling both undesirable and unlabeled trajectories from shared data pool, ensuring identical underlying state-action distributions. We then train our discriminator following the same procedure described in our method, and compute the estimated occupancy ratio 7 7(5,a) a) for varying dataset sizes |D] {10, 20, 50, 100, 300, 500, } trajectories. For each configuration, we report the mean and stan- dard deviation of the estimated ratio across multiple random seeds.
Results. The results in Table 12 demonstrate that the estimated occupancy ratio converges rapidly toward the true value of as the number of trajectories increases. When the dataset size exceeds roughly 300 trajectories, both the mean and standard deviation stabilize around 1.0 = 0.0, indicating that the discriminator can accurately approximate the true occupancy ratio given moderate amount of data. This suggests that the density ratio estimation subproblem our framework not limiting factor for policy learning in practice.
Sample size
10
20
50
100
300
500
1000
Estimated 0.26 + 0.46
0.34
1.33
0.77 0.30
0.85 + 0.57
1.00 0.00
1.00 = 0.00
1.00 1 0.00
Table 12: Empirical evaluation of discriminator- based occupancy ratio estimation when the undesir- able and unlabeled datasets are drawn from the same distribution. The true ratio is expected to be for all (s, a) pairs. The estimated ratio converges toward 1 as the number of trajectories increases.
E.14 Ablation Study: Extreme Q-Learning
To evaluate the empirical impact of the Extreme Q-learning (XQL) update that contribute to the success of our method, we conducted an ablation study comparing XQL with alternative Q-function update strategies: Implicit Q-Learning (IQL) and Sparse Q-Learning (SQL).
Table 13 reports the results across four continuous control tasks. We observe that XQL consistently outperforms both IQL and SQL,, indicating that the extreme Q-update is key contributor to the performance gains reported in Section 5.2
Table 13: Ablation study comparing different Q-function update strategies. Results are reported as mean + standard deviation over 5 seeds.
Cheetah
Ant
Hopper
Walker2d
IQL
3.8 + 2.7
21.0 + 27.0
57.2 = 12.8
85.1 + 19.2
SQL
54.2 - 5.1
38.3 + 23.3
53.9 17.3
85.9 21.1
XQL
75.7 = 6.8
104.4 + 10.5
73.5 = 20.6
105.9 + 4.0
These results demonstrate that the extreme Q-update substantially improves performance across all evaluated tasks, confirming its effectiveness as a critical component of our approach.
E.15 significantly larger number of undesirable demonstrations
we conducted additional experiments using a significantly larger number of undesirable demon- strations to enable further analysis and comparison. The table 14 report the performance of UNIQ
41

---

and baseline methods on the Car-Goal and Point-Goal tasks, with to 1000 undesirable demon- strations. Overall, the results show that UNIQ consistently achieves lower costs indicating safer policies-wihhe maintaining comparable returns to the baselines. This supports the robustness and effectiveness of UNIQ even in heavily imbalanced settings dominated by suboptimal data.
Task
Method
|DUN=
100
300
500
700
1000
Return
26.9:10.1
26.61:.2
26.4=0.3
26.5+10.4
25.6:10.3
DWBC
Cost
45.8:13.4
39.1+2.66
35.9+3.1
36.1+:2.7
33.2+:2.1
Return
27.0:10.11
26.9:1011
26.9-10.1
25.6:10.3
25.8:10.1
SafeDICE
Cost
46.8+t3.1
44.8
42.3=3.7
38.0+
38.4+0.8
Point-Goal
Return
23.4+00.4
21.110.6
21.0=0.7
20.7+10.4
20.9
UNIQ
Cost
27.1+3.0
22.4+3.6
20.2+33.7
19.4:4.2
18.7+3.2
Return
32.8+10.7
28.
25.6=2.2
24.1
23.942.3
DWBC
Cost
47.4+:3.8
37.8+3.1
30.6+3.4
23.543.66
19.9+449
Return
33.5+10.7
32.7=0.8
31,9
31.5+10.7
31.1+0.00
SafeDICE
Cost
50.5+:4.0
47.2:1.22
43.2+1..0
44.0:15.5
42.7+1.7
Car-Goal
Return
27.9:10.0
24.6110.
23.3:2077 19.6#2.2
23.1+0.8 20.0+3.7
22.1 18.9+2.4
UNIQ
Cost
31.0++2.8
22.8+2.9
Table 14: Results for Point-Goal and Car-Goa tasks. Each cell reports mean std.
E.16 Impact of Unlabeled Data Quality
To further analyze the robustness of our method, we conducted an additional experiment to investigate how the quality of the unlabeled dataset impacts performance. A key question is how our approach fares when the proportion of undesirable (random and medium) trajectories in the unlabeled dataset is substantially increased.
For this study, we evaluated method the same undesirable dataset, which consists of random and 5 medium trajectories. The unlabeled dataset was constructed by combining a varying number of random and medium trajectories (X) with the same 100 expert trajectories used in our main experiments. The total size of the unlabeled dataset denoted as X:100, where represents the combined count of random and medium trajectories, split equally between the two.
MuJoCo tasks: Cheetah and Ant. We maintained
The results, presented in Table [5| demonstrate the performance of SafeDICE, DWBC, and UNIQ as we increase the number non expeexpert trajectories in the unlabeled dataset from 1000 to 3000.
Cheetah
1000:100
1500:100
2000:100
2500:100
3000:100
SafeDICE
3.1:0.9
5.1
5.244.0
4.316.1
2.4
DWBC
2.2.0.00
2.2+0.00
2.2+0.00
2.2:0..0
2.2+:0.0
UNIQ
75.746.8
76.0
71.2+4.9
67.543.11
64.8+15.3
Ant 1000:100
1000:100
2000:100
2500:100
3000:100
SafeDICE
4.4+2.6
5.8+7..8
5.6
4.7+E4.1
5,3=5.8
DWBC 84.6+12.4
84.9+1.9
84,4=3.8
83,4+4.3
84.3+6.88
UNIQ
10..4:1.0.5
105.2:19.0
100.0
100.747.3
99.5-18.8
Table 15: Performance evaluation on Cheetah and Ant tasks with varying sizes of unlabeled random and medium trajectories.
E.17 Ablation Study: Sensitivity to Hyperparameters
While our method aims for minimal hyperparameter tuning, it is important to understand its sensi- tivity to key parameters. To address this, we conducted an ablation study focusing on the entropy regularization coefficient, 3, which is standard hyperparameter in maximum entropy frameworks.
42

---

The results of this study are presented in Table 16 Our findings are consistent with prior work on maximum entropy methods, showing that the performance of UNIQ remains stable across reasonable range of / values. For other standard hyperparameters, such as the regularization strength and the discriminator architecture, we adopted settings from prior work. These choices have been shown to be effective and well-validated, representing strong defaults for our setting.
3
0.5
5
10
Cheetah
13.714.88
16.2.33.0
74,413.9
4.3
6.3:15.7
Ant 90.8+200.0
104.3+1.6
17.2+19.3
26.5+4.22
24.6+8.2
Hopper 7.3:
4.0=1.8
43.4+20.9
73.4=16.8
75.1=1110
Walker 20.2
34.5=20.0
80.6
105.914.0
56.814.6
Table 16: Performance of UNIQ across different values of the entropy regularization coefficient 8.
The results show that performance is stable within reasonable range of values. The best performance for each task is highlighted in bold.
As shown in the table, the optimal value for 3 varies depending the task. For instance, the Ant environment performs best with smaller 3 of 1, while the Hopper environment benefits from a larger B of 10. Despite this variation, the performance remains high for range of values around the optimum for each task (e.g., B E [3, 5] for Cheetah and Walker). This indicates that while is an important parameter, method is not overly sensitive to its precise value and does not require exhaustive tuning to achieve strong results.
43


Reviews:
[{"review_id": "cegSjT6PSW", "reviewer": "Reviewer_tHAr", "summary": "The paper introduces UNIQ, an offline imitation learning framework for avoidance learning where the goal is to avoid undesirable behaviors based solely on negative demonstrations and unlabeled data, without relying on expert data. In contrast to traditional imitation learning, which learns by mimicking expert behavior, this method learns to avoid behaviors exhibited in low-quality demonstrations.\n\nUNIQ formulates the problem using a MaxEnt inverse reinforcement learning objective, reformulated in the Q-space to enable stable optimization. The method combines Extreme Q-learning (to address non-convexity) with a novel occupancy correction mechanism that leverages unlabeled data in a statistically unbiased way. The authors demonstrate the approach on Mujoco and Safety-Gym benchmarks, where UNIQ outperforms several baselines, including SafeDICE and DWBC, in both return and constraint satisfaction.", "questions": "The discriminator estimating occupancy ratios plays a central role. How robust is this estimation under class imbalance or covariate shift between DUN and DMIX? Would the method degrade gracefully or fail catastrophically?\n\nThe authors note partial undesirability in demonstrations as a limitation. Could the method be extended to segment trajectories and learn from sub-trajectory labeling, possibly via importance weighting or causal inference?\n\nWhile the paper claims minimal hyperparameter tuning, can the authors provide more details on how sensitive UNIQ is to choices like β, the regularization coefficient, or the discriminator architecture?\n\nHave the authors considered applying this to high-dimensional observation spaces (e.g., images) or domains with long-horizon credit assignment challenges?\n\nAlthough occupancy correction is intended to remove bias, is there any empirical observation on how performance changes with extreme noise or very low quality unlabeled data?\n\nIf the robustness of occupancy ratio estimation is shown to fail under modest noise, that would weaken the empirical claims.\n\nA demonstration of some adaptation to partially undesirable trajectories would significantly increase the paper’s practical impact.", "limitations": "The paper includes a discussion of limitations in Section 6. \nThe authors acknowledge:\n_ the assumption of a single, fully undesirable dataset.\n_ that some undesirable trajectories may contain good segments.\n_ that more sophisticated use of sub-trajectory analysis could improve sample efficiency.\n_ that learning from mistakes typically requires more data than learning from expert behavior.\n\nOne additional point that could be added: the method assumes availability of clearly labeled undesirable data, which may not always be available or well-defined in real-world applications. This could be mitigated by future work that incorporates human feedback or automatic anomaly detection.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\n+ The paper presents a rigorous and well-motivated formulation of the \"learning from mistakes\" problem.\n+ Theoretical grounding is strong, with multiple propositions and a solid use of recent developments in inverse Q-learning and occupancy ratio estimation.\n+ Experimental results are extensive and convincingly demonstrate the superiority of the proposed approach over strong baselines across diverse tasks.\n- The performance relies on estimating occupancy ratios from a classifier trained on undesirable vs. unlabeled data. While standard, this method may be brittle in real-world scenarios, but this is not deeply analyzed.\n- The practical aspects of scaling to more complex domains (e.g., larger action/state spaces, partial observability) are not explored.\n\nClarity\n+ The writing is generally clear, and most derivations and concepts are well explained.\n+ Illustrative examples (e.g., Figure 2) are helpful in clarifying the objective and behavior of the learning algorithm.\n- Some sections (e.g., Section 4.3 on Extreme Q-learning) may be hard to parse without prior familiarity. The presentation could benefit from more intuitive explanations or graphical summaries.\n- The distinction between SafeDICE and UNIQ in terms of practical behavior could be better emphasized.\n\nSignificance\n+ The paper addresses an under-explored but practically important problem: how to learn safe or desirable behavior in the absence of expert data. The problem is relevant in real-world domains like autonomous driving or healthcare, where mistakes are more abundant than expert behavior.\n+ The method achieves strong empirical results across multiple environments.\n- While the use of unlabeled data is novel in this context, the assumptions about being able to isolate undesirable examples may limit applicability in domains without clear labels.\n\nOriginality\n+ The approach is original in shifting the focus of imitation learning from expert-mimicking to anti-imitation of undesirable behavior.\n+ The use of Extreme Q-learning to ensure convexity and the derivation of a novel cooperative training objective are significant innovations.\n+ UNIQ’s unbiased use of unlabeled data via occupancy correction distinguishes it from prior work.\nSome components (e.g., occupancy correction, WBC) are adapted from existing literature. The novelty is more in the integration and framing than in entirely new algorithms.", "was_revised": true, "final_justification": "I believe my questions have been addressed, I confirm my score, and I have completed the mandatory acknowledgement.", "rebuttal": "*We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.*\n\n---\n\n> The discriminator estimating occupancy ratios plays a central role. How robust is this estimation under class imbalance or covariate shift between DUN and DMIX? Would the method degrade gracefully or fail catastrophically?\n> If the robustness of occupancy ratio estimation is shown to fail under modest noise, that would weaken the empirical claims.\n\nWe thank the reviewer for the insightful comment. It is indeed true that the estimation of occupancy ratios can be inaccurate when datasets are small or imbalanced. However, as shown in our experiments (especially those reported in appendix), despite the UN (undesirable) dataset typically having a low sample size, our UNIQ algorithm remains robust and effective.\n\nThe key reason is that precise estimation of the occupancy ratio is not strictly necessary for learning a good policy. Specifically, in our main training objective (Equation 6), the discriminator is designed to assign a higher value to τ(s,a) when the state-action pair (s,a) appears more frequently in the UN dataset, and lower values otherwise. As a result, the objective encourages the learned Q-function and policy to assign lower rewards r^Q(s,a) to undesirable state-action pairs, thereby guiding the policy away from undesirable behaviors and toward more desirable ones.\n\nThis phenomenon is consistent with prior findings in related works, such as GAIL and DICE-based algorithms, where learning remains effective even when occupancy ratios are estimated from low-sample datasets.\n\n> The authors note partial undesirability in demonstrations as a limitation. Could the method be extended to segment trajectories and learn from sub-trajectory labeling, possibly via importance weighting or causal inference?\n\nWe thank the reviewer for the insightful comments. As mentioned in our paper, extending our approach to handle partially undesirable trajectories would introduce several new challenges:\n- **Violation of policy-level assumptions:** Our method relies on estimating occupancy measures for undesirable and mixed policies, assuming that each dataset is generated by a well-defined policy—either desirable or undesirable. When trajectories are only partially undesirable, this assumption no longer holds, as the generating policy cannot be clearly classified as good or bad. This ambiguity makes our current formulation unsuitable for such cases.\n- **Lack of segment-level labels:** Identifying which parts of a trajectory are desirable or undesirable is itself a challenging task. Existing benchmark datasets such as D4RL do not provide this level of granularity—they typically consist of full trajectories generated by either expert or non-expert policies, without annotations indicating which segments are good or bad. Addressing this would require additional supervision or expert feedback, as well as new methodological tools and thorough empirical validation.\n\nGiven the complexity and scope of these challenges, we believe this extension falls outside the focus of our current work, but we agree it is a promising direction for future research.\n\n> While the paper claims minimal hyperparameter tuning, can the authors provide more details on how sensitive UNIQ is to choices like β, the regularization coefficient, or the discriminator architecture?\n\nWe thank the reviewer for the comment. To address this, we conducted additional ablation studies to analyze the impact of the entropy regularization parameter β on the performance of UNIQ. The results are presented below. We note that β is a standard hyperparameter in maximum entropy frameworks, and its effects have been extensively studied in prior work. Our findings align with those results, showing that performance remains stable across a reasonable range of  values.\n\nFor other standard hyperparameters—such as the regularization strength and the discriminator architecture—we adopted settings from prior work. These choices have been shown to be effective and well-validated,  representing strong defaults for our setting.\n\n| $\\beta$ | 0.5 | 1 | 3 | 5 | 10 |\n|---|---|---|---|---|---|\n| Cheetah | 13.7±4.8 | 16.2±3.0 | 74.4±3.9 | **75.3±4.3** | 6.3±5.7 |\n| Ant | 90.8±20.0 | **104.3±1.6** | 57.2±19.3 | 26.5±4.2 | 24.6±8.2 |\n| Hopper | 7.3±3.9 | 4.0±1.8 | 43.4±20.9 | 73.4±16.8 | **75.1±11.0** |\n| Walker | 20.2±14.7 | 34.5±20.0 | 80.6±15.2 | **105.9±4.0** | 56.8±4.6 |\n\n> Have the authors considered applying this to high-dimensional observation spaces (e.g., images) or domains with long-horizon credit assignment challenges?\n\n We thank the reviewer for the thoughtful comments. In this work, we have focused our experiments on widely used benchmark tasks from the D4RL suite, which are standard in the RL and IL communities. These tasks provide a solid foundation for evaluating and comparing the performance of our method against state-of-the-art baselines.\n\nWe acknowledge that applying our approach to more complex settings—such as high-dimensional observation spaces (e.g., vision-based tasks) or domains with long-horizon credit assignment challenges—would be a valuable extension. However, these settings often require significant additional engineering effort (e.g., integrating convolutional architectures or temporal abstraction mechanisms) and computational resources for robust experimentation and validation. We consider this an important and promising direction for future work.\n\n> Although occupancy correction is intended to remove bias, is there any empirical observation on how performance changes with extreme noise or very low quality unlabeled data?\n\nWe thank the reviewer for the insightful comment. To further address this point, we conducted additional experiments to evaluate the performance of our algorithm under conditions where the unlabeled dataset is of extremely low quality. Specifically, we varied the number of suboptimal (bad) demonstrations within the unlabeled set, thereby progressively degrading its overall quality. As expected, we observed a noticeable performance degradation as more bad demonstrations were added to the unlabeled dataset. When the overall quality of the unlabeled data became extremely poor, the algorithm's performance dropped significantly. This outcome is reasonable, as the learning signal becomes increasingly noisy and less informative in such scenarios.\n\n**Experimental description:** In this experiment, we evaluate two Mujoco tasks. We keep the same undesirable dataset $\\mathcal{D}^{UN}$ as used in the main comparison in Section 5.2, consisting of 5 random and 5 medium trajectories. For the unlabeled dataset $\\mathcal{D}^{MIX}$, we also retain the same 100 expert trajectories from Section 5.2, but vary the number of random and medium trajectories. Specifically, the unlabeled dataset has the format X:100, where X is the total number of random and medium trajectories combined (with X/2 random and X/2 medium), together with the 100 expert trajectories.\n\n\n\n**Cheetah**\n\n| | 1000:100 | 1500:100 | 2000:100 | 2500:100 | 3000:100 |\n|---|---|---|---|---|---|\n| SafeDICE | 3.1±0.9 | 5.1±5.2 | 5.2±4.0 | 4.3±6.1 | 2.4±3.2 |\n| DWBC | 2.2±0.0 | 2.2±0.0 | 2.2±0.0 | 2.2±0.0 | 2.2±0.0 |\n| UNIQ | 75.7±6.8 | 76.0±1.1 | 71.2±4.9 | 67.5±3.1 | 64.8±5.3 |\n\n**Ant**\n\n| | 1000:100 | 1500:100 | 2000:100 | 2500:100 | 3000:100 |\n|---|---|---|---|---|---|\n| SafeDICE | 4.4±2.6 | 5.8±7.8 | 5.6±3.7 | 4.7±4.1 | 5.3±5.8 |\n| DWBC | 84.6±12.4 | 84.9±1.9 | 84.4±3.8 | 83.4±4.3 | 84.3±6.8 |\n| UNIQ | 104.4±10.5 | 105.2±9.0 | 100.0±7.2 | 100.7±7.3 | 99.5±8.8 |\n\n\n--- \n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments  and discussion described above. If you have any further comments or questions, we would be happy to address them*"}, {"review_id": "TKIhfR2CQl", "reviewer": "Reviewer_nyTV", "summary": "This paper addresses the challenge of learning avoidant behaviors in offline imitation learning without expert demonstration. It proposes UNIQ, a novel framework that leverages undesirable demonstrations and unlabeled data to learn strategies to avoid undesirable behavior. Starting from the training goal based on maximum entropy, UNIQ reformulates the problem as a cooperative inverse Q-learning task and integrates unlabeled data for an effective strategy for offline training. Experiments on Mujoco and Safety-Gym have shown that UNIQ outperforms other baselines when given a set of undesirable demonstrations.", "questions": "- If the probability of the demonstration appearing in undesirable demonstrations is less than the probability of appearing in unlabeled data, then is the optimization direction of Equation 4 reversed, and have you done any relevant experiments? For example, you can simply duplicate two copies of the undesirable demonstrations or add more to the unlabeled data.\n- At present, the number of experts in the setup is relatively small. Is there an experiment with a larger number of experts, and will it be better than other methods?\n- Have you tried adding optimal demonstrations and undesirable demonstrations to train together, and will the results of UNIQ be better?\n- The cost metric of the Car-Button in table 1 is much lower than the UNIQ on the LS-IQ method, but it shows that the UNIQ is the best, is it a recording error?\n- What are the advantages of the method proposed in this paper compared to confidence based imitation learning methods, such as 2IWIL, CAIL, WGAIL, PN-GAIL [1-4].\n- Has the time spent in each method been recorded?\n- It seems that the model is effective by avoiding undesirable demonstrations, how to ensure the comprehensiveness of undesirable demonstrations?\n\n[1] Wu Y H, Charoenphakdee N, Bao H, et al. Imitation learning from imperfect demonstration[C]//International Conference on Machine Learning. PMLR, 2019: 6818-6827.\n\n[2] Wang Y, Xu C, Du B, et al. Learning to weight imperfect demonstrations[C]//International Conference on Machine Learning. PMLR, 2021: 10961-10970.\n\n[3] Zhang S, Cao Z, Sadigh D, et al. Confidence-aware imitation learning from demonstrations with varying optimality[J]. Advances in Neural Information Processing Systems, 2021, 34: 12340-12350.\n\n[4] Liu Q, Fu H, Tang K, et al. PN-GAIL: Leveraging Non-optimal Information from Imperfect Demonstrations[C]//The Thirteenth International Conference on Learning Representations.", "limitations": "- The limitations of the method have been outlined in the conclusion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- Well-written paper with clear objectives. The authors have the intention to share the code.\n- The question raised in the paper is reasonable and practical.\n- Theoretical analysis is complete.\n- The proposed method demonstrates good performance compared to the baselines.\n\n### Weaknesses\n- This approach seems to simply add the setting of undesirable demonstrations to other methods.\n- As mentioned in the paper in Limitations, 'each undesirable trajectory may not be undesirable in its entirety', which may affect the final effect when used in practice.", "was_revised": true, "final_justification": "My concerns have been resolved, and I will raise my score to 4.", "rebuttal": "*We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.*\n\n---\n\n> This approach seems to simply add the setting of undesirable demonstrations to other methods.\n\nWe thank the reviewer for the feedback. We would like to emphasize that our work introduces a novel perspective by explicitly focusing on learning from undesirable behaviors—a setting that diverges significantly from conventional IL, which typically relies on expert or near-expert demonstrations. This shift presents a fundamentally new problem, requiring new training objectives and theoretical insights. Notably, one of our key and novel findings is that imitation learning can still succeed even when no expert demonstrations are explicitly identified in the dataset. We believe this opens an important and underexplored direction in the field, and represents a meaningful contribution to advancing the boundaries of IL research.\n\n> As mentioned in the paper in Limitations, 'each undesirable trajectory may not be undesirable in its entirety', which may affect the final effect when used in practice.\n\nWe thank the reviewer for the insightful comment. Extending our approach to handle partially undesirable trajectories is indeed valuable, but it introduces significant challenges. Our method assumes each trajectory comes from a clearly defined policy—either desirable or undesirable—an assumption that breaks down when trajectories are mixed. Moreover, current benchmarks like D4RL lack segment-level labels needed to distinguish good and bad parts within a trajectory. Handling this would require additional supervision and methodological changes, which we consider a promising direction for future work beyond the scope of this paper.\n\n> If the probability of the demonstration appearing in undesirable demonstrations is less than the probability of appearing in unlabeled data, then is the optimization direction of Equation 4 reversed, and have you done any relevant experiments? For example, you can simply duplicate two copies of the undesirable demonstrations or add more to the unlabeled data.\n\nWe thank the reviewer for the helpful comment. To clarify, our learning objective in Equation 4 does not involve the unlabeled dataset directly, so the optimization direction is theoretically unaffected even if the probability of a demonstration appearing in the undesirable dataset is lower than in the unlabeled dataset. \n\nYour point regarding adding more undesirable samples to the unlabeled data is well taken. To further investigate this, we conducted an additional experiment (see below) where we substantially increased the proportion of bad demonstrations in the unlabeled dataset to analyze how performance changes as its quality degrades.\n\n**Experimental description:** In this experiment, we evaluate two Mujoco tasks. We keep the same undesirable dataset $\\mathcal{D}^{UN}$ as used in the main comparison in Section 5.2, consisting of 5 random and 5 medium trajectories. For the unlabeled dataset $\\mathcal{D}^{MIX}$, we also retain the same 100 expert trajectories from Section 5.2, but vary the number of random and medium trajectories. Specifically, the unlabeled dataset has the format X:100, where X is the total number of random and medium trajectories combined (with X/2 random and X/2 medium), together with the 100 expert trajectories.\n\n\n\n**Cheetah**\n\n| | 1000:100 | 1500:100 | 2000:100 | 2500:100 | 3000:100 |\n|---|---|---|---|---|---|\n| SafeDICE | 3.1±0.9 | 5.1±5.2 | 5.2±4.0 | 4.3±6.1 | 2.4±3.2 |\n| DWBC | 2.2±0.0 | 2.2±0.0 | 2.2±0.0 | 2.2±0.0 | 2.2±0.0 |\n| UNIQ | 75.7±6.8 | 76.0±1.1 | 71.2±4.9 | 67.5±3.1 | 64.8±5.3 |\n\n**Ant**\n\n| | 1000:100 | 1500:100 | 2000:100 | 2500:100 | 3000:100 |\n|---|---|---|---|---|---|\n| SafeDICE | 4.4±2.6 | 5.8±7.8 | 5.6±3.7 | 4.7±4.1 | 5.3±5.8 |\n| DWBC | 84.6±12.4 | 84.9±1.9 | 84.4±3.8 | 83.4±4.3 | 84.3±6.8 |\n| UNIQ | 104.4±10.5 | 105.2±9.0 | 100.0±7.2 | 100.7±7.3 | 99.5±8.8 |\n\n> At present, the number of experts in the setup is relatively small. Is there an experiment with a larger number of experts, and will it be better than other methods?\n\nWe thank the reviewer for the comment. To clarify, expert demonstrations are included only in the unlabeled dataset. As detailed in Section E.3 of the Appendix, we conducted an ablation study evaluating the performance of UNIQ and key baselines under varying amounts of expert data (from 50 to 300 demonstrations, with 300 considered a relatively large amount). The results show that UNIQ consistently outperforms the baselines across all settings.\n\n> Have you tried adding optimal demonstrations and undesirable demonstrations to train together, and will the results of UNIQ be better\n\nWe thank the reviewer for the insightful comments. Our work is primarily focused on the setting where expert (or optimal) demonstrations are not available, and the algorithm is explicitly designed with this assumption in mind. Incorporating expert demonstrations directly into our learning objective (Equation 4) would substantially alter its structure and potentially lead to instability, as the learning signals from expert and undesirable data may conflict and drive the optimization in opposite directions. As such, we believe the current form of our algorithm is not well-suited for simultaneously learning from both expert and undesirable demonstrations. Addressing this scenario would likely require a fundamentally new algorithmic framework.\n\nThat said, in settings where high-quality expert demonstrations are available, existing methods designed for learning from both expert and sub-optimal data—such as DemoDICE or DWBC—may be more appropriate.\n\n> The cost metric of the Car-Button in table 1 is much lower than the UNIQ on the LS-IQ method, but it shows that the UNIQ is the best, is it a recording error?\n\nThank you for the comment. In this experiment, the LS-IQ policy achieves low cost but at the expense of an unacceptably low reward. In fact,  LS-IQ policy in fact does nothing, so the  cost is low, but it also fail to gain meaningful rewards. Therefore,  it is reasonable not to consider LS-IQ’s performance as best in this case. We will clarify this point in the updated version of the paper.\n\n> What are the advantages of the method proposed in this paper compared to confidence based imitation learning methods, such as 2IWIL, CAIL, WGAIL, PN-GAIL [1-4].\n\nWe thank the reviewer for the comment. The key difference between our method and confidence-based approaches like 2IWIL, CAIL, WGAIL, and PN-GAIL is that those methods rely on per-sample confidence estimation and typically assume access to expert or near-expert demonstrations. In contrast, UNIQ is specifically designed for the more challenging setting where expert data is unavailable and only undesirable and unlabeled demonstrations are provided. This makes UNIQ more practical in scenarios where collecting high-quality expert data is infeasible.\n\n> Has the time spent in each method been recorded?\n\nWe thank the reviewer for the comment. To clarify this, we compare the training speed of UNIQ to DWBC, SafeDICE, IPL, and LSIQ on a single task (Cheetah), using the same RTX 3090 GPU. The training speeds have been recorded as below:\n\n| algo | DWBC | SafeDICE | IPL | LSIQ | UNIQ |\n|---|---|---|---|---|---|\n| Walltime | ~150 mins | ~250 mins | ~90 mins | ~140 mins | ~30 mins |\n\n> It seems that the model is effective by avoiding undesirable demonstrations, how to ensure the comprehensiveness of undesirable demonstrations?\n\nWe thank the reviewer for the thoughtful comment. While achieving full coverage of undesirable demonstrations is indeed challenging, our work is motivated by the practical observation that collecting undesirable data is often easier and less costly than collecting expert demonstrations. In real-world scenarios, such data can be obtained from logged failures or suboptimal policies, which typically span a sufficiently diverse set of poor actions. Based on this, we assume that the undesirable dataset provides adequate coverage of common failure modes. Moreover, our experiments—including those presented in the appendix—demonstrate that UNIQ remains robust across a variety of settings. That said, we also observe a decline in performance when the number of undesirable demonstrations is very low, which is expected and consistent with our assumptions.\n\n---\n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments  and discussion described above. If you have any further comments or questions, we would be happy to address them.*"}, {"review_id": "3hK8ttltpW", "reviewer": "Reviewer_ffqG", "summary": "The paper introduces a new imitation learning setting with the goal of both avoiding bad demonstrations and learning from unlabeled demonstrations. They present a non-adversarial method (UNIQ) to do this by combining 3 ideas from past work: IQ-Learn, Extreme Q-Learning, and occupancy ratio estimation. They demonstrate that UNIQ outperforms baselines in constrained and unconstrained continuous control settings.", "questions": "Minor \n1. I'm not sure what point the toy MDP setting is supposed to make: wouldn't minimizing KL against a mixing baseline like SafeDICE (which down-weights p1) still upweight p2 and p3? \n2. Most of the baselines seem incapable of learning in the unconstrained setting. Do you have an explanation for why this happens and what their limiting factor is? \n3. Just to clarify: for Proposition 4.1 and Section 4.3, is the optimal V being relative to the behavior policy a property of the offline setting? \n4. Do you have a sense of whether the cooperative nature of the setting allow you to do better than the expert? \n5. Figure 3 seems to demonstrate that UNIQ scales better with expert demos vs other methods. Do you have an explanation for this behavior? \n6. Are there results for the data scaling experiments (expert and undesirable demos) with larger amounts of data? Right now, the ablation only spans 1 OOM. \n7. This is out of scope of the paper, but do you have any thoughts on how UNIQ scales in the online setting? \n\nMajor \n1. Re. weakness 1: I think the setting is novel, but the scope is pretty narrow to the offline, data-constrained setting, and I have some concerns of broader significance. I'm willing to change my mind on this, given discussion and supporting experiments. For example, one such argument could be: there are settings where  it's easier to classify bad behavior than good behavior and UNIQ gives significant gains. Showing this would require 1. Evidence of such a setting where this gap exists 2. Empirical evidence that UNIQ scales better with undesirable demos than baselines in this setting. For example, Figure 4 (Point-Goal) doesn't seem to really show this as the baselines seem to maintain better return with decreasing cost (and they might cross if the x-axis was extended). I'm open to other arguments and happy to discuss.   \n2. Re. weakness 2: I have some concerns about estimating the occupancy measure using a discriminator. Although the idea is pretty classic, density ratio estimation in this way can struggle to accurately model the scale of the ratio when the distributions are far apart. I wonder if this is an issue if the undesirable and unlabeled distributions grow further apart. I'd be very interested in seeing an ablation doing some preliminary science for this. I believe this would be a valuable contribution for understanding the empirical properties of algorithmic primitives for estimating occupancy ratios. I think you could do this in the safety constrained setting where the oracle occupancy ratio is known since you have likelihoods of the PPO-Lagrangian and PPO policies. The main experiment would then show that NCE is sufficient to learn a \"good estimator\" (as measured by some divergence against the oracle) and goodness of estimation scales favorably with the amount of undesirable data. Note: I view this as a valuable contribution for understanding, even if the experiment shows that the discriminator doesn't scale well with more undesirable data. \n3. I'm willing to increase my rating to 5 if both points are addressed adequately.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The overall presentation of the work is clear. The authors do a good job of summarizing major related work, motivating their setting, and iteratively building up the core parts of their method. \n2. The core set of results (Table 1) are convincingly better across both unconstrained and constrained RL settings. UNIQ is the only method that matches (or comes close) to expert performance across all settings. The authors do a reasonable job of doing the science of ablating data scaling, see Questions for some more comments on this point.\n3. The method mostly combines well-known ideas (IQ-Learn, Extreme Q-learning, occupancy ratio estimation), but it's well-motivated and applies these ideas to a fresh setting. This combination distinguishes the method and setting from past work.\n   \nWeaknesses\n1. My main concern is that the scope is pretty narrow to the offline, data-constrained setting, and I have some concerns of broader significance.\n2. I have a secondary concern about the method for estimating the occupancy measure ratio using a discriminator and would like to see rigorous experiments characterizing this. \n3. I'm willing to update my score based on rebuttal, see Questions for specifics.", "was_revised": true, "final_justification": "I updated my score from a 4 to a 5 after rebuttal. I had 2 main concerns: 1. scope of the work and scaling behavior of UNIQ with respect to undesirable demos 2. estimation of occupancy ratios. The authors did a reasonable job of addressing both concerns. For 1, they argue that the proposed setting is relevant to several real-world safety-critical applications and show additional results for positive behavior when scaling the number of undesirable demos. For 2, they provided 2 different sanity check experiments that the estimated occupancy ratio is reasonable. Both of my main concerns were adequately addressed, so I raised my score.", "rebuttal": "*We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.*\n\n---\n\n> I'm not sure what point the toy MDP setting is supposed to make: wouldn't minimizing KL against a mixing baseline like SafeDICE (which down-weights p1) still upweight p2 and p3?\n\nThe purpose of the toy MDP is not to compare against other baselines, but to illustrate that our learning objective can return near-optimal policies when learning from undesirable demonstrations. We believe that methods like SafeDICE could also do the same in this toy example.\n\n> Most of the baselines seem incapable of learning in the unconstrained setting. Do you have an explanation for why this happens and what their limiting factor is?\n\nAmong all the baselines, only SafeDICE is explicitly designed to learn from undesirable demonstrations, and it primarily focuses on constrained settings. As discussed in our paper, SafeDICE ‘s performance is highly sensitive to the quality of the unlabeled dataset. In contrast, our approach is theoretically designed to be less dependent on the quality of the unlabeled data, offering greater robustness in challenging settings.\n\n> Just to clarify: for Proposition 4.1 and Section 4.3, is the optimal V being relative to the behavior policy a property of the offline setting?\n\nYes, this is a common property of the offline setting, where data is assumed to be sampled from a behavior policy. This assumption is widely adopted in both offline RL and IL.\n\n> Do you have a sense of whether the cooperative nature of the setting allow you to do better than the expert?\n\nIn our setting, outperforming the expert cannot be guaranteed and is inherently challenging, as we do not have access to expert demonstrations for guidance. Instead, the cooperative nature of our approach contributes to a more stable training process and helps steer learning toward a good policy.\n\n> Figure 3 seems to demonstrate that UNIQ scales better with expert demos vs other methods. Do you have an explanation for this behavior?\n\nWe thank the reviewer for the comment. Compared to other baselines, our method can better leverage expert demonstrations within the unlabeled data. As shown in our learning objective (Eq. 6), when more expert demonstrations are present, the algorithm assigns lower τ(s,a) values to those expert-like pairs, encouraging higher rewards for them and ultimately leading to a better policy.\n\n> Are there results for the data scaling experiments (expert and undesirable demos) with larger amounts of data? Right now, the ablation only spans 1 OOM.\n\nWe thank the reviewer for the comment. In our current ablation study (Section E.3), we vary the number of expert demonstrations from 50 to 300, which we believe reflects a realistic and practical range in scenarios where expert data is typically limited. \n\n> This is out of scope of the paper, but do you have any thoughts on how UNIQ scales in the online setting?\n\nWe thank the reviewer for the question. While the current framework could be extended to the online setting, making it effective would require substantial effort, including adapting the learning objective and regularization, as well as benchmarking against a new set of baselines. We consider this a valuable and promising direction for future work.\n\n> Re. weakness 1: I think the setting is novel, but the scope is pretty narrow to the offline, data-constrained setting,...\n\nWe thank the reviewer for the constructive feedback and the helpful suggestion. Our work indeed focuses on the offline, data-constrained setting, which we agree is narrower in scope compared to more general-purpose IL frameworks. However, we believe this setting is both practically relevant and novel, particularly in safety-critical applications (e.g., robotics, autonomous driving, or healthcare), where undesirable behaviors—such as collisions or failures—are easier to collect or identify than expert demonstrations. In such cases, it is often significantly more feasible to label or log what not to do, rather than gather consistent expert behavior.\n\n> Empirical evidence that UNIQ scales better with undesirable demos than baselines in this setting. For example, Figure 4 (Point-Goal) ...\n\nWe thank the reviewer for raising this valuable point. To address it, we conducted additional experiments using a significantly larger number of undesirable demonstrations to enable further analysis and comparison. The tables below report the performance of UNIQ and baseline methods on the Car-Goal and Point-Goal tasks, with up to 1000 undesirable demonstrations. Due to NeurIPS rebuttal format restrictions, we are unable to include figures, so we report only the mean and std of the final scores. Overall, the results show that UNIQ consistently achieves lower costs—indicating safer policies—while maintaining comparable returns to the baselines. This supports the robustness and effectiveness of UNIQ even in heavily imbalanced settings dominated by suboptimal data.\n\n\n**Point-Goal**\n\n| $\\|\\mathcal{D}^{UN}\\|$ | 100 | 300 | 500 | 700 | 1000 |\n|---|---|---|---|---|---|\n| DWBC | Return = 26.9±0.1; Cost = 45.8±3.4 | Return = 26.6±0.2; Cost = 39.1±2.6 | Return = 26.4±0.3; Cost = 35.9±3.1 | Return = 26.5±0.4; Cost = 36.1±2.7 | Return = 25.6±0.3; Cost = 33.2±2.1 |\n| SafeDICE | Return = 27.0±0.1; Cost = 46.8±3.1 | Return = 26.9±0.1; Cost = 44.8±3.6 | Return = 26.9±0.1; Cost = 42.3±3.7 | Return = 25.6±0.3; Cost = 38.0±1.8 | Return = 25.8±0.1; Cost = 38.4±0.8 |\n| UNIQ | Return = 23.4±0.4; Cost = 27.1±3.0 | Return = 21.1±0.6; Cost = 22.4±3.6 | Return = 21.0±0.7; Cost = 20.2±3.7 | Return = 20.7±0.4; Cost = 19.4±4.2 | Return = 20.9±0.3; Cost = 18.7±3.2 |\n\n**Car-Goal**\n\n| $\\|\\mathcal{D}^{UN}\\|$  | 100 | 300 | 500 | 700 | 1000 |\n|---|---|---|---|---|---|\n| DWBC | Return = 32.8±0.7; Cost = 47.4±3.8 | Return = 28.8±1.5; Cost = 37.8±3.1 | Return = 25.6±2.2; Cost = 30.6±3.4 | Return = 24.1±1.8; Cost = 23.5±3.6 | Return = 23.9±2.3; Cost = 19.9±4.9 |\n| SafeDICE | Return = 33.5±0.7; Cost = 50.5±4.0 | Return = 32.7±0.8; Cost = 47.2±4.2 | Return = 31.9±0.8; Cost = 43.2±4.0 | Return = 31.5±0.7; Cost = 44.0±5.5 | Return = 31.1±0.0; Cost = 42.7±1.7 |\n| UNIQ | Return = 27.9±0.0; Cost = 31.0±2.8 | Return = 24.6±0.9; Cost = 22.8±2.9 | Return = 23.3±0.7; Cost = 19.6±2.2 | Return = 23.1±0.8; Cost = 20.0±3.7 | Return = 22.1±1.1; Cost = 18.9±2.4 |\n\n> Re. weakness 2: I have some concerns about estimating the occupancy measure using a discriminator. Although the idea is pretty classic, ... I believe this would be a valuable contribution for understanding the empirical properties of algorithmic primitives for estimating occupancy ratios. \n\nWe thank the reviewer for the valuable and insightful comment. Regarding the accuracy of occupancy ratio estimation, we fully agree that it can be inaccurate when datasets are small in size—an issue common in practice. However, as shown in our experiments, our algorithm remains robust and effective even with limited undesirable data. The key reason is that precise occupancy ratio estimation is not strictly required for good policy learning. In our main training objective (Equation 6), the discriminator encourages higher τ(s,a) values for state-action pairs that appear more frequently in the undesirable dataset, and lower values otherwise. This naturally drives the learned Q-function and policy to assign lower rewards r^Q(s,a) to undesirable behaviors, guiding the agent toward better actions. This behavior aligns with findings from prior work such as GAIL and DICE-based methods, which also demonstrate robustness under noisy ratio estimates.\n\n> I think you could do this in the safety constrained setting where the oracle occupancy ratio is known since you have likelihoods of the PPO-Lagrangian and PPO policies. \n\nWe thank the reviewer for the insightful  suggestion. However, we are currently unsure how to compute the oracle occupancy ratio using PPO or PPO-Lagrangian policies. Estimating this ratio would typically require access to both undesirable and unlabeled datasets, along with accurate sampling from the target policy's occupancy distribution. In our case, we are not sure how to use PPO policies to get  a reliable estimate of the ratio. If the reviewer could clarify the intended procedure, we would be happy to explore it further and include the results.\n\n> The main experiment would then show that NCE is sufficient to learn a \"good estimator\"...\n\nTo further address your concern, we conducted an additional experiment to evaluate the estimation of the occupancy ratio τ(s,a) across varying dataset sizes. Due to NeurIPS rebuttal policy restrictions, we are unable to include figures, so we report only the mean and standard deviation of the estimated ratios across the (s,a) space. The results generally indicate that the estimation of the occupancy ratio becomes stable as the dataset size grows. In the revised version of the paper, we will include detailed distribution plots and provide further analysis to strengthen the discussion around this estimation.\n\n| $\\|\\mathcal{D}^{MIX}$\\| | 50 | 100 | 200 | 500 | 1000 | 2000 | 5000 |\n|---|---|---|---|---|---|---|---|\n| $\\tau$ | 0.60±0.20 | 0.61±0.19 | 0.60±0.19 | 0.61±0.20 | 0.61±0.20 | 0.62±0.20 | 0.60±0.21 |\n\n| $\\|\\mathcal{D}^{UN}\\|$| 1 | 5 | 10 | 25 | 50 | 100 | 500 | 1000 |\n|---|---|---|---|---|---|---|---|---|\n| $\\tau$ | 0.68±0.19 | 0.61±0.19 | 0.62±0.23 | 0.60±0.20 | 0.59±0.21 | 0.61±0.20 | 0.61±0.18 | 0.61±0.19 |\n\n---\n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments  and discussion described above. If you have any further comments or questions, we would be happy to address them.*"}, {"review_id": "JeTVbYvy4s", "reviewer": "Reviewer_v5LQ", "summary": "The paper introduces UNIQ (Undesired Demonstrations driven Inverse Q-learning), a method for offline imitation learning in settings where two types of data are available: a small set of undesirable demonstrations, and a large pool of unlabeled demonstrations that may include both good and bad behaviors. The objective is to learn a policy that avoids learning the undesirable behaviors while still acquiring task-relevant skills from the unlabeled data. \n\nUNIQ builds on the IQ-Learn framework by inverting its objective: rather than maximizing reward on expert trajectories as in standard imitation learning, UNIQ instead minimizes reward on the undesirable trajectories. To leverage the much larger unlabeled dataset, UNIQ introduces an occupancy correction mechanism where it reweighs the learning objective using the likelihood ratio of how likely a state-action pair in the unlabeled set originates from the undesirable distribution. \n\nExperiments are conducted on 8 environments from D4RL and Safety-Gym. Results show the UNIQ achieves lower safety costs or constraint violations and high returns compared to several baselines.", "questions": "- Have you tried more diverse data compositions for experiments? For example, some medium behaviors in undesirable set, and random+expert in unlabeled set. \n- What is the empirical impact of the Extreme Q-learning in Section 4.3? Do you have ablations?", "limitations": "The authors do provide a limitations section. I would also strongly encourage them to think how difficult it is to capture undesirable demonstrations. People always talk of the difficulty of curating an expert set, but I think it is also very challenging to capture really \"bad\" demonstrations. Some clarity on practical curation would be welcome.\n\nI don't believe the method has any distinctive negative societal impact beyond those that exist for ML methods.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper addresses a practically important problem, as demonstrated by the examples provided for autonomous driving.\n- Flipping the IQ-Learn objective leads to a loss of convexity with respect to the Q-function, which could make optimization unstable. The authors address this with a separately learned value function, which I found to be interesting.\n- The paper is well-written and clearly organized.\n\n**Weaknesses**\n\n1. Experiments:\n- Dataset creation: For the D4RL environments, the undesirable dataset is constructed by combining trajectories from the random and medium levels. The unlabeled dataset is formed by combining trajectories from all three levels: random, medium, and expert. For the Safety-Gym environments, two policies are first trained: a constrained policy that respects safety constraints, and an unconstrained policy that optimizes for return while ignoring safety. Data collected from the unconstrained policy is used to create the undesirable dataset, while data from both policies is combined to form the unlabeled dataset. This results in the unlabeled data having a clear bimodal distribution. Furthermore, for the D4RL setup, the space of undesired behavior is extremely large and essentially encompasses everything except for expert behavior. \n- Baselines: Building on the above point, DWBC should solve this problem quite easily. DWBC originally learns a discriminator to distinguish between expert and non-expert data using PU learning. Thus, one can easily adapt it by treating unsafe data as the \"positive\" data in PU learning and then performing BC using inverse weighting instead of direct weighting. However, the authors choose the unlabeled dataset as the \"positive\" class which is confusing and a much weaker baseline.\n- Ablation: The impact of adding the extra value function is not analyzed.\n\n2. The likelihood estimation is done by training a binary classifier. This will lead to issues since the the negative class (unlabeled set) contains the positive class (undesirable set).\n\n3. The paper is slightly incremental, since it is basically flips the objective of IQ-Learn.", "was_revised": false, "final_justification": "", "rebuttal": "*We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.*\n\n---\n\n> Baselines: Building on the above point, DWBC should solve this problem quite easily. DWBC originally learns a discriminator...\n\nWe thank the reviewer for the insightful comment. While we had previously adapted DWBC to better fit our setting, the approach you suggested provides a valuable alternative perspective. To address this, we conducted an additional experiment using the version you suggested, and we report the results below. Overall, the results indicate that our adapted version of DWBC generally achieves better performance. \n| | Cheetah | Ant | Hopper | Walker2d |\n|---|---|---|---|---|\n| Suggested DWBC (return) | 4.7±1.4 | 38.7±2.5 | 12.2±3.9 | 5.9±7.8 |\n| Our adapted DWBC (return)| 2.2±0.0 | 84.6±12.4 | 14.8±15.7 | 38.8±24.7 |\n\n| | Point-Goal | Car-Goal | Point-Button | Car-Button |\n|---|---|---|---|---|\n| Suggested DWBC | Return = 27.1±0.1; Cost = 47.9±0.3 | Return = 35.4±0.5; Cost = 54.8±2.3 | Return = 24.5±0.8; Cost = 142.4±5.0 | Return = 18.2±0.5; Cost = 271.4±31.2 |\n| Our adapted DWBC | Return = 26.9±0.1; Cost = 45.8±3.4 | Return = 32.8±0.7; Cost = 47.4±3.8 | Return = 17.2±0.9; Cost = 123.5±14.4 | Return = 17.1±1.0; Cost = 249.2±20.9 |\n\n> The likelihood estimation is done by training a binary classifier. This will lead to issues since the the negative class (unlabeled set) contains the positive class (undesirable set).\n\nWe thank the reviewer for the insightful comments. In theory, the discriminator is trained to estimate an occupancy ratio that allows us to leverage unlabeled samples while remaining consistent with the original learning objective. In practice, the discriminator tends to assign higher values to τ(s,a) when the state-action pair appears more frequently in the undesirable dataset than in the unlabeled one, and lower values otherwise. This estimated ratio is directly used in our learning objective (Eq. 6) to guide the policy away from undesirable behaviors and toward better policies. Therefore, the presence of undesirable samples within the unlabeled dataset does not pose a problem for our method.\n\n> The paper is slightly incremental, since it is basically flips the objective of IQ-Learn.\n\nWe thank the reviewer for the comment. While our approach builds on the structure of inverse Q-learning and, more broadly, the maximum entropy (MaxEnt) framework, it introduces a novel perspective by explicitly focusing on learning from undesirable behaviors—a setting that differs significantly from traditional imitation learning methods, which rely on expert or near-expert demonstrations. Although our objective mirrors the MaxEnt framework in form, it is reconstructed in the opposite direction, and many theoretical properties from MaxEnt no longer apply. This shift requires new analysis and empirical validation. For these reasons, we believe our work offers a meaningful contribution that advances the field in a new and underexplored direction.\n\n> Have you tried more diverse data compositions for experiments? For example, some medium behaviors in undesirable set, and random+expert in unlabeled set.\n\nWe thank the reviewer for the question. To clarify, our datasets are already diverse: the undesirable data includes both medium and random demonstrations, while the unlabeled dataset contains a mix of random, medium and expert demonstrations. In the experiments presented in both the main paper and the appendix, we include several ablation studies—specifically, Section E.1 shows results when more random and medium demonstrations are added to the undesirable set, and Sections E.2 and E.3 explore the effect of adding more random and expert demonstrations to the unlabeled dataset. We hope these experiments address your concern.\n\n> What is the empirical impact of the Extreme Q-learning in Section 4.3? Do you have ablations?\n\nWe thank the reviewer for the constructive comment. To evaluate the impact of our extreme Q-update, we conducted additional experiments comparing different Q-function update strategies, including Implicit Q-Learning (IQL) and Sparse Q-Learning (SQL). The results are reported below, showing that Extreme Q-learning (XQL)  achieves the best performance as.\n\n| | Cheetah | Ant | Hopper | Walker2d |\n|---|---|---|---|---|\n| IQL | 3.8±2.7 | 21.0±27.0 | 57.2±12.8 | 85.1±19.2 |\n| SQL | 54.2±5.1 | 38.3±23.3 | 53.9±17.3 | 85.9±21.1 |\n| XQL | 75.7±6.8 | 104.4±10.5 | 73.5±20.6 | 105.9±4.0 |\n\n---\n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments  and discussion described above. If you have any further comments or questions, we would be happy to address them.*"}]

Author rebuttal:

Meta-review: The submitted paper explores learning avoidance behavior in offline imitation learning without access to expert demonstrations, i.e., avoiding undesirable behavior learned from negative examples. The authors propose a novel training objective (UNIQ) based on the maximum entropy principle and reformulate the problem as a cooperative inverse Q-learning task. They also introduce a method to incorporate unlabeled data for unbiased training. Experimental results on benchmark environments demonstrate that their approach outperforms state-of-the-art baselines in several settings.

**Strengths of the Paper**
* Problem setting: The paper addresses a practically relevant and under-explored problem, i.e., learning from undesirable demonstrations without expert data. This can be particularly relevant in safety-critical domains like autonomous driving and healthcare.
* Theoretical framework: The paper provides a sensible reformulation of the MaxEnt inverse reinforcement learning objective in Q-space, along with a novel occupancy correction mechanism.
* Empirical evaluation: Experiments on various benchmarks (Mujoco, Safety-Gym) demonstrate that UNIQ can achieve superior performance compared to baselines, particularly in safety-constrained settings. The authors also provide detailed ablation studies and additional experiments in the rebuttal, which when added to the paper, make the experimental evaluation solid and “complete”. 

**Weaknesses of the Paper**
* Novelty: Some reviewers noted that the method primarily builds on existing ideas (e.g., IQ-Learn, Extreme Q-learning, occupancy ratio estimation) and flips the objective of IQ-Learn. While the integration is novel, the individual components are not entirely new.
* Dataset assumptions: The method assumes that undesirable demonstrations are clearly labeled and that the unlabeled dataset contains a mix of good and bad behaviors. This assumption may not hold in real-world scenarios where such clear distinctions are unavailable.
* Robustness concerns: The reliance on occupancy ratio estimation via a binary classifier could result in limited robustness. The authors provided additional experiments to address emphasize robustness (and that exact estimation might not be necessary), further analysis would strengthen the paper.


** Discussion** 
Both reviewers and authors actively engaged in the discussion phase. In particular, the following central points were discussed:
* Realism of assumptions: One reviewer raised concerns about the assumption of clearly labeled undesirable data and the conflation of suboptimal and undesirable behaviors. Another reviewer argued that the assumption of negative samples is realistic in safety-critical settings, where undesirable behaviors (e.g., collisions) are easier to classify than expert behaviors. The authors clarified in response that their method is designed for settings where undesirable data is easier to collect than expert data. They acknowledged the limitations of their assumptions and suggested future work on sub-trajectory labeling and human feedback (which introduces additional challenges).
* Occupancy ratio estimation: One reviewer expressed concerns about the accuracy of occupancy ratio estimation and its impact on policy learning. The authors conducted additional experiments to evaluate the robustness of the discriminator and showed that precise ratio estimation is not strictly necessary for effective learning.
* Baselines and comparisons: One reviewer suggested an alternative adaptation of DWBC for the proposed problem setting. The authors implemented this suggestion and compared it to their adapted version of DWBC. However, both versions underperform compared to UNIQ.
* Data composition and scaling: One reviewer suggested experiments with more diverse data compositions. The authors conducted these experiments and found that their proposed UNIQ (as well as DWBC-based methods) struggles.
* Impact of extreme Q-learning: One reviewer suggested an ablation study to evaluate the impact of extreme Q-learning. The authors conducted this ablation and provided results showing that extreme Q-learning achieves better performance compared to other Q-function update strategies (e.g., IQL, SQL).

**Recommendation**
The paper addresses a novel and practically important problem, provides a well-motivated and theoretically grounded solution, and demonstrates strong empirical results across diverse benchmarks. While some concerns about robustness and dataset assumptions remain, the authors have adequately addressed these during the rebuttal period through additional experiments and clarifications. UNIQ’s focus on learning from undesirable behaviors without expert data represents a meaningful contribution to the field of imitation learning. Hence, in line with the recommendation of all reviewers, I recommend the acceptance of the paper.
Decision: Accept (poster)