UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations

## Abstract

Abstract We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting involves avoiding undesirable behavior (specified using undesirable demonstrations). To tackle this problem, unlike standard imitation learning where the aim is to minimize the distance between learning policy and expert demonstrations, we formulate the learning task as maximizing a statistical distance, in the space of state-action stationary distributions, between the learning policy and the undesirable policy. This significantly different approach results in a novel training objective that necessitates a new algorithm to address it. Our algorithm, UNIQ, tackles these challenges by building on the inverse Q-learning framework, framing the learning problem as a cooperative (non-adversarial) task. We then demonstrate how to efficiently leverage unlabeled data for practical training. Our method is evaluated on standard benchmark environments, where it consistently outperforms state-of-the-art baselines.
The code implementation can be accessed at: https://github.com/hmhuy0/UNIQ .

## 1 Introduction

Reinforcement learning (RL) is a powerful framework for learning to maximize expected returns and has achieved remarkable success across various domains. However, applying reinforcement learning to real-world problems is challenging due to difficulties in designing reward functions and the requirement for extensive online interactions with the environment. While some approaches have addressed these challenges, they often rely on costly datasets, requiring either accurate labeling or clean, consistent data, which is often impractical. Imitation learning (Abbeel & Ng, ; Ziebart et al., ; Kelly et al., ) offers a more feasible alternative, enabling agents to learn directly from expert demonstrations without the need for explicit reward signals. It has proven effective in several tasks, even with limited expert data, and is particularly useful in capturing human preferences.

Most existing imitation learning approaches prioritize maximizing task performance (i.e., expected return) by closely mimicking expert demonstrations (Ho & Ermon, ; Fu et al., ; Kostrikov et al., ; Garg et al., ). However, in practice, expert or near-expert demonstrations may be unavailable or insufficient. In many scenarios, instead of high-quality examples, there may be collections of undesirable (or suboptimal) demonstrations that should be avoided. For example, in a large dataset of user conversations used to train a chatbot, the system must learn to avoid inappropriate or sensitive content that is present in the data (Duan et al., ). Similarly, in the development of self-driving cars, while companies may collect user driving data to train their models, the system must ensure it does not replicate faulty behavior, such as traffic violations or unsafe driving practices (Bansal et al., ). In the field of treatment optimization, data may include actions that led to bad patient outcomes and the system must learn to avoid such behaviors. In these cases, avoiding undesirable behaviors is essential to training a safe and effective model. These types of undesirable demonstrations are common in real-world applications and are crucial for shaping the desired policy.

Although this is an important and interesting problem setting, there has been limited research addressing it effectively. Here, we formulate this challenge as an Offline Reverse Imitation Learning problem: given a dataset containing undesired demonstrations we wish to avoid, alongside a much larger unlabelled dataset consisting of both desired and undesired demonstrations, the goal is to learn a desired policy. The key question is whether we can leverage the undesired demonstrations to learn a policy that avoids undesirable behaviors.

Figure: Figure 1: A general overview of offline imitation learning approaches: while prior approaches mostly aim at minimizing the distance between the learning policy and expert data (or a combination of non-expert and unlabeled data), our goal is to maximize the distance between the learning policy and an undesired policy. In UNIQ, the unlabeled dataset ($\rho^{\textsc{Mix}}$) is used in the empirical approximation of the main objective function.
Refer to caption: x1.png

To the best of our knowledge, only SafeDICE (Jang et al., ) directly addresses this setting by mixing the undesirable and unlabeled datasets, assigning negative weights to the former, and then mimicking the mixed policy. This approach, however, may suffer from the fact that the mixed policy is not necessarily a desirable one to follow, which is often the case in practice. There are existing methods that, while not specifically designed to address this problem, can be adapted to handle it. For instance, preference-based RL methods (Brown et al., ; Lee et al., ; Hejna & Sadigh, ) can be modified by creating pairwise comparisons between the undesirable and unlabeled demonstrations, assuming the unlabeled demonstrations are more preferred. However, this approach is indirect and may not be effective. Similar to SafeDICE, this approach will also suffer as the unlabeled dataset may contain undesirable demonstrations that are not necessarily preferable to those in the undesirable dataset. Another approach, Discriminator-Weighted Behavioral Cloning (DWBC) (Xu et al., ), can be adapted to train a discriminator to distinguish between desirable and undesirable demonstrations, allowing the agent to avoid learning from undesirable samples. Our experiments later will show that our method significantly outperforms the above baselines on benchmark problems.

In this paper, we develop a principled framework for learning from undesirable demonstrations, based on the well-known MaxEnt RL framework (Ziebart et al., ) and inverse Q-learning (Garg et al., ) — a state-of-the-art imitation learning method. In our work, unlike traditional imitation learning approaches where the main goal is to minimize the distance between the learning policy and expert (or near-optimal) demonstrations, we instead aim to maximize a statistical distance between the learning policy and the undesirable policy (represented by undesirable demonstrations) in the state-action stationary distribution space. To address this novel learning objective, we adopt the inverse Q-learning framework. In our context, since this objective requires optimization in the opposite direction as compared to the standard IQ-learn method (Garg et al., ), it demands significantly novel investigations. Specifically, our contributions can be summarized as follows:

- •
We demonstrate that maximizing the statistical distance between the learning policy and the undesirable policy can be formulated as a cooperative training task, in contrast to previous imitation learning approaches where the objective is adversarial Ho & Ermon (); Fu et al. (); Kostrikov et al. (). This new cooperative training objective poses new challenges that existing algorithms cannot readily solve. To tackle this, we show that certain beneficial characteristics of the standard IQ-learn algorithm can be brought over in our context – the training problem can be equivalently reformulated as a minimization problem over the Q-space, where the optimal policy is recovered as a soft-max of the Q-function.
- •
To efficiently solve the training problem in the Q-space using limited undesirable demonstrations, we introduce an occupancy correction to recast the training objective so that the expectation over the undesirable policy can be empirically approximated by unlabeled trajectories. This approach offers three main advantages: (i) the unlabeled data contains many more samples than the undesired data set, so using unlabeled demonstrations would greatly improve the accuracy of the empirical approximations, (ii) the training outcome is theoretically independent of the quality of the unknown random policy (represented by the unlabeled data), and (iii) it does not require an additional hyper-parameter since we do not combine the two datasets (undesired and unlabeled) as in most prior approaches. Moreover, we show that the occupancy correction can be estimated by solving a convex optimization problem using samples from both undesirable and unlabeled data.
- •
For policy extraction, we employ a weighted behavior cloning (WBC) formulation.
This WBC approach theoretically ensures to recover the optimal policy derived from the Q-learning process while providing more stable offline training outcomes, compared to directly extracting the policy from the Q-function. While our algorithm, UNIQ, tackles a significantly different and more challenging setting compared to the original IQ-learn, it only requires a minimal adaptation from the IQ-learn algorithm, with two additional steps — computing the occupancy correction and extracting the policy via WBC — which can be carried out quickly and efficiently.
- •
We evaluate our method on the Safety-Gym and Mujoco-velocity benchmarks (Ray et al., ; Ji et al., ), two of the most challenging environments in constrained reinforcement learning, and demonstrate superior performance compared to several state-of-the-art baselines.

## 2 Related Work

Imitation learning. Imitation learning is a key technique for learning from demonstrations. Behavioral Cloning (BC) maximizes the likelihood of expert demonstrations but often struggles due to distributional shift (Ross et al., ). To improve, Generative Adversarial Imitation Learning (Ho & Ermon, ; Fu et al., ) aligns the learner’s policy with the expert’s using GANs (Goodfellow et al., ), while SQIL (Reddy et al., ) assigns simple rewards to expert and non-expert demonstrations to learn a value function. PWIL (Dadashi et al., ) uses the Wasserstein distance (Vaserstein, ) to compute rewards. While these methods show promise, they rely on online interaction, which can be impractical. For offline learning, AlgaeDICE (Nachum et al., ) and ValueDICE (Kostrikov et al., ) use Stationary Distribution Correction Estimation (DICE) but face stability issues. Inspired by ValueDICE, O-NAIL (Arenz & Neumann, ) introduced an offline method without adversarial training. IQ-learn (Garg et al., ), a popular approach, supports both online and offline learning and offers a state-of-the-art framework with several variants developed based on it (Al-Hafez et al., ; Hoang et al., ).
Unlike the above mentioned works, our focus in this paper is on offline reverse imitation learning that aims to avoid undesirable trajectories, as opposed to imitating expert trajectories. As indicated earlier, this requires fundamentally different methods due to the nature of the problem.

Imitation Learning from Sub-optimal Demonstrations.
There are two main research directions in this area. The first focuses on online and offline preference-based imitation learning methods. Online approaches, such as T-REX (Brown et al., ), PrefPPO (Lee et al., ), and PEBBLE (Lee et al., ), leverage ranked sub-optimal demonstrations to learn a preference-based reward function using the Bradley-Terry model (Bradley & Terry, ). While these methods achieve strong performance, they rely on interaction with the environment. In contrast, offline methods, such as those proposed by (Kim et al., ; Kang et al., ; Hejna & Sadigh, ), rely heavily on extensive pairwise trajectory comparisons. SPRINQL (Hoang et al., ) addressed this limitation by utilizing demonstrations categorized into different levels of expertise, resulting in better performance with fewer comparison demands. While these methods concentrate on imitating preferred (or expert) trajectories, our focus is on avoiding non-preferred (or undesired) trajectories. This important distinction necessitates significant changes in methodology.

The second direction focuses the use of additional unlabeled datasets to enhance learning from expert data. Beginning with DemoDICE (Kim et al., ), several DICE-based methods (Ma et al., ; Kim et al., ; Yu et al., ) have been developed to utilize small sets of expert demonstrations, supplemented by larger unlabeled datasets. In addition to these DICE-based methods, DWBC (Xu et al., ) propose a simple and efficient method based on training a classifier using positive-unlabeled learning (Kiryo et al., ). SafeDICE (Jang et al., ) presents a DICE-based framework capable of learning from undesirable demonstrations. This method combines an undesirable policy (represented by an undesirable dataset) with a random policy (represented by a larger unlabeled dataset), assigns negative weights to the undesirable policy, and then applies a standard DICE-based approach (Kim et al., ; ) to mimic the combined policy. In our paper, we advance this line of work by developing an new inverse Q-learning algorithm (Garg et al., ) where the primary goal is to maximize the statistical distance between the learning policy and the undesirable policy. Our method offers several advantages over prior approaches, including minimal hyper-parameter tuning and reduced sensitivity to the quality of the unlabeled data.

## 3 Background

##### Preliminaries.

We consider a MDP defined by the following tuple $\mathcal{M}=\left\langle S,A,r,P,\gamma,s_{0}\right\rangle$, where $S$ denotes the set of states, $s_{0}$ represents the initial state set, $A$ is the set of actions, $r:S\times A\rightarrow\mathbb{R}$ defines the reward function for each state-action pair, and $P:S\times A\rightarrow S$ is the transition function, i.e., $P(s^{\prime}|s,a)$ is the probability of reaching state $s^{\prime}\in S$ when action $a\in A$ is made at state $s\in S$, and $\gamma$ is the discount factor. In reinforcement learning (RL), the aim is to find a policy that maximizes the expected long-term accumulated reward
$\max_{\pi} \left\{\mathbb{E}_{(s,a)\sim\rho_{\pi}}[r(s,a)]\right\}$, where $\rho_{\pi}$ is the occupancy measure of policy $\pi$:
$\rho_{\pi}(s,a)=(1-\gamma)\pi(a|s)\sum_{t=1}^{\infty}\gamma^{t}P(s_{t}=s|\pi).$

##### MaxEnt IRL

The goal of MaxEnt IRL is to derive a reward function $r(s,a)$ based on a set of expert demonstrations, ${\mathcal{D}}^{E}$. Let $\rho^{E}$ denote the occupancy measure of the expert policy. The MaxEnt IRL framework, as introduced by (Ziebart et al., ), aims to recover the expert’s reward function by optimizing the following expression:
$\max_{r}\min_{\pi}\left\{\mathbb{E}_{\rho^{E}}[r(s,a)]-\mathbb{E}_{\rho_{\pi}}
[r(s,a)]-H(\pi)-\psi(r)\right\}$,
where $H(\pi)=\mathbb{E}_{\rho^{\pi}}[-\log\pi(s,a)]$ is the discounted causal entropy of the policy $\pi$ and $\psi(r):\mathbb{R}^{{\mathcal{S}}\times{\mathcal{A}}}\rightarrow\mathbb{R}$ is a convex reward regularizer.
In essence, the objective is to identify a reward function that maximizes the gap between the expected reward under the expert’s policy and the maximum expected reward across all other policies (as determined by the inner minimization).

##### Inverse Q-learning (IQ-Learn) from expert demonstrations.

Given a reward function $r$ and a policy $\pi$, the soft Bellman equation is defined as ${\mathcal{B}}^{\pi}_{r}[Q](s,a)=r(s,a)+\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s
^{\prime})],\text{ where }V^{\pi}(s)=\mathbb{E}_{a\sim\pi(a|s)}[Q(s,a)-\log\pi
(a|s)].$ The Bellman equation ${\mathcal{B}}^{\pi}_{r}[Q]=Q$ is contractive and always yields a unique Q solution (Garg et al., ). In IQ-Learn, they further define an inverse soft-Q Bellman operator
${\mathcal{T}}^{\pi}[Q]=Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{\prime})]$. (Garg et al., ) show that
for any reward function $r(a,s)$, there is a unique $Q^{*}$ function such that ${\mathcal{B}}^{\pi}_{r}[Q^{*}]=Q^{*}$, and for a $Q^{*}$ function in the $Q$-space, there is a unique reward function $r$ such that $r={\mathcal{T}}^{\pi}[Q^{*}]$.
This result suggests that one can safely transform the objective function of the MaxEnt IRL from $r$-space to the Q-space as follows:

$$ $\displaystyle\max_{Q}\min_{\pi}~{}~{}$ $\displaystyle\Phi(\pi,Q)=\mathbb{E}_{\rho^{E}}[{\mathcal{T}}^{\pi}[Q](s,a))]- \mathbb{E}_{\rho_{\pi}}[{\mathcal{T}}^{\pi}[Q](s,a)]-H(\pi)-\psi({\mathcal{T}} ^{\pi}[Q](s,a)))$ (1) $$

which has several advantages, namely, the objective function $\phi(\pi,Q)$ is concave in $\pi$ and convex in Q. Moreover, the inner problem $\min_{\pi}\phi(\pi,Q)$ has a closed form solution as $\pi^{*}(a|s)=\exp(Q(s,a))/\sum_{a}\exp(Q(s,a^{\prime}))$. As a result, the maximin problem can be converted to a non-adversarial problem in the Q-space as:

$$ $\max_{Q}~{}~{}\mathbb{E}_{\rho^{E}}[{\mathcal{T}}[Q](s,a))]-(1-\gamma)\mathbb{ E}_{s_{0}}[V^{Q}(s)]-\psi({\mathcal{T}}[Q](s,a)))$ (2) $$

where ${\mathcal{T}}[Q](s,a))=Q(s,a)-\gamma\mathbb{E}_{s^{\prime}\sim P(s^{\prime}|s,
a)}[V^{Q}(s^{\prime})]$ and $V^{Q}(s)=\log\Big{(}\sum_{a}\exp(Q(s,a))\Big{)}$ , which is a softmax of the Q function. The reward function can then be recovered as $r^{Q}(s,a)=\mathcal{T}[Q](s,a)$. Thus, in 2, the objective can be interpreted as training a reward function (via a Q-function) that maximizes the expected reward under the expert policy while minimizing the overall expected reward.

## 4 UNIQ: Inverse Q-learning from Undesired Demonstrations

We now introduce UNIQ, our framework for inverting a Q function based on undesired demonstrations. This approach can be seen as a reverse version of the standard Inverse Q-learning algorithm (Garg et al., ), where the goal is not to imitate but rather to avoid undesired behaviors.

### 4.1 Inverse Q-learning from Undesired Demonstrations

In our setting, we have a set of undesired demonstrations, denoted as ${\mathcal{D}}^{\textsc{UN}}$, along with a supplementary set of unlabeled demonstrations, denoted as ${\mathcal{D}}^{\textsc{Mix}}$. The unlabeled dataset ${\mathcal{D}}^{\textsc{Mix}}$ may contain a mix of random, undesired, and expert demonstrations, and it will be used to support offline learning. Let $\rho^{\textsc{UN}}$ be the occupancy measure (or stationary distribution) of the undesired policy (represented by the undesired dataset). Adapting the MaxEnt RL framework, we consider the following learning objective:

$$ $\min_{\pi}\min_{r}\left\{L(\pi,r)=\mathbb{E}_{\rho^{\textsc{UN}}}[r(s,a)]- \mathbb{E}_{\rho_{\pi}}[r(s,a)]-H(\pi)+\psi(r)\right\}$ (3) $$

The objective can be interpreted as finding a reward function that assigns low rewards to the undesired demonstrations and high rewards to others, while also identifying a policy function that maximizes the expected long-term reward. It is important to note that, in our context, where the objective contrasts with the standard learning-from-demonstration scheme, the learning problem is no longer adversarial as in prior imitation learning approaches (Ho & Ermon, ; Kostrikov et al., ). Instead, it can be framed as a cooperative learning problem, where the objective is to jointly identify a policy and reward function that minimize the objective function $L(\pi,r)$.

To gain a deeper understanding of the objective function in Eq. 3, the following proposition demonstrates that solving Eq. 3 is indeed equivalent to maximizing the statistical distance, parameterized by $\psi$, between the undesired policy and the learning policy.

###### Proposition 4.1 .

For a non-restricted feasible set of the reward function:

$$ $\min_{r}\min_{\pi}\left\{L(\pi,r)\right\}=-\max_{\pi}\{d_{\psi}(\rho^{\pi}, \rho^{\textsc{UN}})-H(\pi)\}$ (4) $$

where $d_{\psi}(\rho^{\pi},\rho^{\textsc{UN}})=\psi^{*}(\rho^{\pi}-\rho^{\textsc{UN}})$, and $\psi^{*}$ is the convex conjugate of the convex function $\psi$, i.e., $\psi^{*}(t)=\sup_{z}\{\langle t,z\rangle-\psi(z)\}$.

It can be observed that solving 3 directly encourages the learning policy to deviate as much as possible from the undesired policy, which is derived from undesirable demonstrations. This approach contrasts with that of SafeDICE (Jang et al., ), which minimizes the KL divergence between the learning policy and the mixed policy, enabling the use of standard imitation learning methods. The primary limitation of this approach is that, with the quality of the unlabeled dataset being unknown, imitating the mixed policy may not lead to the desired learning outcome. In contrast, our approach does not rely on such a combination. Instead, we focus on maximizing the statistical gap in 4, leveraging the unlabeled dataset to support the practical training of our primary objective in 3.

Even though the minimization problem Eq. 3 can be directly solved using standard optimization algorithms to recover a reward and policy function, prior research indicates that transforming Eq. 3 into the Q-space can improve efficiency. As discussed in Section 3, there is a one-to-one mapping between any reward function $r$ and a corresponding function $Q$ in the Q-space. Therefore, the minimization problem in Eq. 3 can equivalently be transformed as:

$$ $\displaystyle\min_{Q}\min_{\pi}\left\{L(\pi,Q)=\mathbb{E}_{\rho^{\textsc{UN}}} [{\mathcal{T}}^{\pi}[Q](s,a))]-\mathbb{E}_{\rho_{\pi}}[{\mathcal{T}}^{\pi}[Q]( s,a)]-H(\pi)+\psi\left({\mathcal{T}}^{\pi}[Q](s,a)\right)\right\}$ (5) $$

where ${\mathcal{T}}^{\pi}[Q](s,a))=Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{
\prime})]$ and $V^{\pi}(s)=\mathbb{E}_{a\sim\pi(a|s)}[Q(s,a)-\log\pi(a|s)]$

Compared to the primary objective of the standard IQ-learn algorithm in Eq. 1, our objective function in Eq. 5 is no longer adversarial with respect to $Q$ and $\pi$. As a result, there are questions regarding whether the key advantages of the IQ-learn algorithm—such as the closed-form for the optimization over $\pi$ and the concavity of the objective in the Q-space— would still hold with the new objective. To address this, we introduce the following proposition, which states that if the regularizer function $\psi(\cdot)$ is non-decreasing, then the objective function $L(\pi,Q)$ is convex in $\pi$. Furthermore, the minimization problem $\min_{\pi}L(\pi,Q)$ retains a closed-form solution, thereby simplifying the learning objective.

###### Proposition 4.2 .

The following statements hold:

- (i)
The function $L(\pi,Q)$ is convex in $\pi$ and the problem $\min_{\pi}L(\pi,Q)$ has a unique optimal solution at $\pi^{Q}(s,a)=\frac{\exp(Q(s,a))}{\sum_{a^{\prime}}\exp(Q(s,a^{\prime}))}$.
- (ii)
The learning objective function can be simplified as:
$\min_{Q}\min_{\pi}\left\{L(\pi,Q)\right\}=\min_{Q}\left\{{\mathcal{F}}(Q)=
\mathbb{E}_{\rho^{\text{UN}}}[r^{Q}(s,a)]-(1-\gamma)\mathbb{E}_{s_{0}}[V^{Q}(s
_{0})]+\psi(r^{Q})\right\}$
where $V^{Q}(s)=\log\left(\sum_{a}\exp(Q(s,a))\right)$ and $r^{Q}(s,a)=Q(s,a)-\gamma\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}V^{Q}(s^{
\prime})$.

Proposition 4.2 shows that, similar to IQ-learn, our training objective can be framed as an optimization problem over the Q-space, where the optimal policy can be computed as the soft-max of the Q-function. However, there is a key difference lying in the nature of the objective: while in IQ-learn, the training objective is convex within the Q-space, this is not the case in our context, where the objective ${\mathcal{F}}(Q)$ is neither convex nor concave in $Q$.

We now discuss the learning within the Q-space:

$$ $\displaystyle\min_{Q}\left\{{\mathcal{F}}(Q)=\mathbb{E}_{\rho^{\text{UN}}}[r^{ Q}(s,a)]-(1-\gamma)\mathbb{E}_{s_{0}}[V^{Q}(s_{0})]+\psi(r^{Q})\right\}$ (6) $$

The learning objective can be interpreted as finding a reward function (expressed as a function of Q) that assigns the lowest possible rewards to the undesired demonstrations while maximizing the overall expected rewards, $\mathbb{E}_{s_{0}}[V^{Q}(S_{0})]$.

###### Proposition 4.1 .

###### Proposition 4.2 .

### 4.2 Learning with Unlabeled Data

To solve Eq. 6, the expectation $\mathbb{E}_{\rho^{\text{UN}}}[r^{Q}(s,a)]$ can be empirically approximated using samples from the set of undesired demonstrations ${\mathcal{D}}^{\textsc{UN}}$. However, in our setting, this set is limited. Additionally, since the learning process must be conducted offline, without interaction with the environment, directly using the limited samples in ${\mathcal{D}}^{\textsc{UN}}$ is not effective. Therefore, we leverage the larger set of unlabeled data ${\mathcal{D}}^{\textsc{Mix}}$ to enhance the offline training. To achieve this,
we first let $\rho^{\textsc{Mix}}$ be the occupancy measure (or stationary distribution) of the policy represented by unlabeled dataset. We rewrite the expectation over $\rho^{\textsc{UN}}$ as:

$$ $\displaystyle\mathbb{E}_{\rho^{\textsc{UN}}}[r^{Q}(s,a)]$ $\displaystyle=\sum_{s,a}\rho^{\textsc{UN}}(s,a)r^{Q}(s,a)=\sum_{s,a}\rho^{ \textsc{Mix}}(s,a)\tau(s,a)r^{Q}(s,a)=\mathbb{E}_{\rho^{\textsc{Mix}}}[\tau(s, a)r^{Q}(s,a)]$ $$

Where $\tau(s,a)=\frac{\rho^{\textsc{UN}}(s,a)}{\rho^{\textsc{Mix}}(s,a)}$ represents the occupancy ratio between $\rho^{\textsc{UN}}$ and $\rho^{\textsc{Mix}}$, we then rewrite the learning objective as follows:

$$ $\min_{Q}\left\{{\mathcal{F}}(Q)=\mathbb{E}_{\rho^{\text{{Mix}}}}[\tau(s,a)r^{Q }(s,a)]-(1-\gamma)\mathbb{E}_{s_{0}}[V^{Q}(s_{0})]+\psi(r^{Q})\right\}$ $$

In this approach, the expectation $\mathbb{E}_{\rho^{\text{{Mix}}}}[\tau(s,a)r^{Q}(s,a)]$ can be empirically approximated using the unlabeled samples from ${\mathcal{D}}^{\textsc{Mix}}$, where $\tau(s,a)$ acts as an occupancy correction. This correction allows us to leverage samples from the unlabeled dataset to estimate the expectation over the undesirable policy. A key challenge here is that the occupancy ratio $\tau(s,a)$ is unknown. To address this, we propose estimating the ratio by solving the following implicit maximization problem:

$$ $\displaystyle\max_{\mu_{1},\mu_{2}:\mathbb{R}^{{\mathcal{S}}\times{\mathcal{A} }}\rightarrow[0,1]}\quad g(\mu_{1},\mu_{2})$ $\displaystyle=\mathbb{E}_{\rho^{\textsc{Mix}}}[\log(\mu_{2}(s,a)-\mu_{1}(s,a) \mu_{2}(s,a))]$ $\displaystyle+\mathbb{E}_{\rho^{\textsc{UN}}}[\log(\mu_{1}(s,a)-\mu_{1}(s,a) \mu_{2}(s,a))]$ (7) $$

The above formulation is an extension of the discriminator formulation widely used in prior work (Ho & Ermon, ; Kelly et al., ). The following proposition theoretically shows that solving 7 will exactly return the occupancy ratio $\tau$.

###### Proposition 4.3 .

$g(\mu_{1},\mu_{2})$ is strictly concave in $\mu_{1},\mu_{2}$. Furthermore, let $\mu^{*}_{1}$ and $\mu^{*}_{2}$ be the unique optimal solutions to 7, then we have:
$\tau(s,a)=\frac{\mu^{*}_{1}(s,a)}{\mu^{*}_{2}(s,a)}.$

So, our learning process can be broken down into two steps. In the first step, we learn the occupancy ratios by solving the maximization problems presented in 7. Following this, we optimize the following problem to learn a Q function.

$$ $\min_{Q}\left\{{\mathcal{F}}(Q)=\mathbb{E}_{\rho^{\text{{Mix}}}}\left[\frac{ \mu^{*}_{1}(s,a)}{\mu_{2}^{*}(s,a)}r^{Q}(s,a)\right]-(1-\gamma)\mathbb{E}_{s_{ 0}}[V^{Q}(s_{0})]+\psi(r^{Q})\right\}$ (8) $$

It is important to note that we utilize the stationary distribution $\rho^{\textsc{Mix}}$ (represented by trajectories in the unlabeled dataset) in the objective function in 8. However, thanks to the occupancy correction $\frac{\mu^{*}_{1}(s,a)}{\mu_{2}^{*}(s,a)}$, the outcome of the training is independent of the quality of the unlabeled policy $\rho^{\textsc{Mix}}$. This distinguishes our approach from SafeDICE (Jang et al., ), where the performance heavily relies on the quality of the unlabeled data.

###### Proposition 4.3 .

### 4.3 Policy Extraction

The Q function obtained from solving the minimization problem in 8 can be used to recover a soft policy $\pi^{Q}(s,a)=\exp(Q(s,a))/\sum_{a^{\prime}}\exp(Q(s,a))$, as stated in Proposition 4.2 above. However, this approach may suffer from overestimation, a common issue in offline Q-learning caused by out-of-distribution actions and function approximation errors (Ross et al., ). To address this problem and improve policy extraction, we instead propose using weighted behavior cloning (WBC) with the objective:
$\max_{\pi}\left\{\sum_{s,a}\exp(A(s,a))\log\pi(a|s)\right\},$
where $A(s,a)$ is the advantage function defined as $A(s,a)=Q(s,a)-V(s)$. It can be shown that solving the WBC problem yields the exact desired soft policy, i.e.,
$\pi^{Q}(a|s)=\frac{\exp(Q(s,a))}{\sum_{a^{\prime}}\exp(Q(s,a))},~{}~{}\forall a
,s,$ is optimal for the WBC.

## 5 Practical Implementation

Our algorithm consists of two main steps. The first step involves solving 7 to estimate the occupancy ratio $\tau(s,a)=\frac{\rho^{\textsc{UN}}(s,a)}{\rho^{\textsc{Mix}}(s,a)}$. In the second step, we use these ratios to train the Q-function and extract a policy by solving a weighted behavior cloning problem.

In the first step, we construct two networks, $\mu_{\phi_{1}}(s,a)$ and $\mu_{\phi_{2}}(s,a)\in[0,1]$, where $\phi_{1}$ and $\phi_{2}$ are learnable parameters. We then use samples from ${\mathcal{D}}^{\textsc{UN}}$ and ${\mathcal{D}}^{\textsc{Mix}}$ to estimate the expectations, leading to the following practical objective:

$$ $\displaystyle\max_{\phi_{1},\phi_{2}}\quad\widetilde{g}(\phi_{1},\phi_{2})$ $\displaystyle=\sum_{(s,a)\sim{\mathcal{D}}^{\textsc{Mix}}}\left[\log(\mu_{\phi _{2}}(s,a)-\mu_{\phi_{1}}(s,a)\mu_{\phi_{2}}(s,a))\right]$ $\displaystyle+\sum_{(s,a)\sim{\mathcal{D}}^{\textsc{UN}}}\left[\log(\mu_{\phi_ {1}}(s,a)-\mu_{\phi_{1}}(s,a)\mu_{\phi_{2}}(s,a))\right]$ (9) $$

In the second step, after obtaining $\phi_{1}^{*}$ and $\phi_{2}^{*}$ from the first step, we utilize the following empirical training objective:

$$ $\displaystyle\min_{w}\Big{\{}\widetilde{{\mathcal{F}}}(w)$ $\displaystyle=\sum_{(s,a,s^{\prime})\sim{\mathcal{D}}^{\textsc{Mix}}}\left[ \frac{\mu_{\phi_{1}^{*}}(s,a)}{\mu_{\phi_{2}^{*}}(s,a)}r^{Q_{w}}(s,a,s^{\prime })+\psi(r^{Q_{w}}(s,a,s^{\prime}))\right]$ $\displaystyle-(1-\gamma)\sum_{s^{0}\sim{\mathcal{D}}^{\textsc{Mix}}}V^{Q_{w}}( s)\Big{\}}$ (10) $$

where $w$ are the learnable parameters for the Q-network, and the reward function is computed as $r^{Q_{w}}(s,a,s^{\prime})=Q_{w}(s,a)-\gamma V^{Q_{w}}(s^{\prime})$, with $V^{Q_{w}}(s)=\log\left(\sum_{a\sim{\mathcal{D}}^{\textsc{Mix}}}\exp(Q_{w}(s,a)
)\right)$. Following (Garg et al., ), we choose the reward regularizer function $\psi(t)=t-t^{2}$, i.e., the $\chi^{2}$-divergence. To extract a policy, we simply solve the weighted BC problem: $\max_{\theta}\sum_{(s,a)\in{\mathcal{D}}^{\textsc{Mix}}}\exp(Q_{w}(s,a)-V^{Q_{
w}}(s))\log\pi_{\theta}(a|s)$, where $\theta$ are learnable parameters of the policy network. Our main algorithm, UNIQ, is summarized as follows, noting that the training of the Q-function $Q_{w}$ and the updating of the policy network $\pi_{\theta}$ are performed simultaneously to enhance efficiency.

Figure: Algorithm 1 UNIQ: UNdesired Demonstrations driven Inverse Q Learning

## 6 Experiments

We assess our algorithm in the context of safe RL (or constrained RL) problems, where each state-action pair is associated with a cost value, and the objective is to learn a policy that satisfies certain cost constraints. We define an undesirable trajectory as one where the accumulated cost exceeds a specified threshold. This setting is similar to the one used in SafeDICE (Jang et al., ), one of the SOTA algorithms in the context of learning from undesirable demonstrations.

### 6.1 Experiment setup

Baselines. We compare our algorithm against several baseline methods. First, we benchmark UNIQ against standard imitation learning algorithms that utilize the entire unlabelled demonstration dataset, specifically BC and IQ-learn, which we refer to as BC-mix and IQ-mix, respectively. Additionally, we benchmark our approach against the SOTA preference-based reinforcement learning algorithm, IPL (Hejna & Sadigh, ). Lastly, we include comparisons with DWBC (Xu et al., ) and SafeDICE (Jang et al., ), both of which are recognized for their ability to learn from undesired demonstrations. More details are provided in the Appendix B.3.

Environments and Data Generation. We evaluate our method in four Safety-Gym and two Mujoco environments (Ray et al., ; Ji et al., ). Following the setup from (Jang et al., ), the undesired policy(^1^11In the context of safe RL, a desired policy is also referred to as a safe policy, while an undesired policy is termed an unsafe policy.) is trained using unconstrained PPO, while the safe policy is trained with SIM (Hoang et al., ). The unlabeled dataset is constructed by combining trajectories from both safe and undesired policies in a specified ratio. The undesired data consists of trajectories that violate the constraint. Detailed information about the policies and datasets is provided in the Appendix B.2.

Metrics. The main goal is to train a policy that is safe in the context of constrained RL. Therefore, an ideal outcome is achieving the lowest possible cost without significantly sacrificing return. We report the accumulated return and cost for the trained policies, computed based on the last 20 evaluations, with all results summarized across at least 5 training seeds.

Experimental Concerns. Throughout the experiments, we aim to address several key questions: (Q1) How does UNIQ perform compared to other baseline methods? (Q2) How does the presence of undesired demonstrations impact the performance of UNIQ and other baselines? (Q3) What happens if the policy is directly extracted from the Q-function, rather than by solving the WBC? (Q4) How does the policy learned by UNIQ compare to a policy trained purely from expert demonstrations? (Q5) How do UNIQ and other baselines perform when being trained without access to the unlabeled dataset? While (Q1) and (Q2) are addressed in the main paper, the experiments for the remaining questions are provided in the appendix. The appendix also includes proofs of the theoretical claims made in the main paper, additional details about our experimental setup, and further results such as CVaR cost comparisons, the complete set of experiments for the MuJoCo tasks, comparisons using datasets from the SafeDICE paper, and detailed learning curves.

### 6.2 Main comparison on Safety-Gym Tasks

In this section, we aim to evaluate the performance of our method in comparison with the mentioned baselines. We test across three difficulty levels for each environment by varying the amount of safe data ($100$, $400$, and $1600$ trajectories) while keeping the number of undesired data fixed at $1600$ trajectories. These settings are referred to as env-1, 2, 3, respectively. This allows us to examine how the proportion of the desired and undesired demonstrations in the unlabeled dataset impacts the performance of each baseline. The experiment results are shown in Tab. 1.

**Table 1: Comparison results for Safety-Gym tasks.**
|  |  | BC-mix | IQ-Mix | IPL | DWBC | SafeDICE | UNIQ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Point-Goal-1 | Return | 26.8$\pm$0.1 | 26.7$\pm$0.6 | 26.9$\pm$0.1 | 26.5$\pm$0.2 | 26.6$\pm$0.2 | 20.7$\pm$0.7 |
| Cost | 42.7$\pm$3.5 | 43.9$\pm$11.9 | 53.7$\pm$3.7 | 33.0$\pm$3.2 | 36.3$\pm$2.9 | 23.5$\pm$4.5 |  |
| Point-Goal-2 | Return | 27.1$\pm$0.1 | 27.0$\pm$0.4 | 26.9$\pm$0.1 | 26.9$\pm$0.1 | 27.0$\pm$0.1 | 23.4$\pm$0.4 |
| Cost | 48.8$\pm$2.9 | 46.7$\pm$10.4 | 52.7$\pm$3.4 | 45.8$\pm$3.4 | 46.8$\pm$3.1 | 27.1$\pm$3.0 |  |
| Point-Goal-3 | Return | 27.1$\pm$0.1 | 43.8$\pm$29.1 | 26.9$\pm$0.1 | 27.1$\pm$0.1 | 27.1$\pm$0.1 | 26.4$\pm$0.2 |
| Cost | 51.0$\pm$3.4 | 55.6$\pm$27.6 | 53.6$\pm$3.7 | 50.2$\pm$3.6 | 50.7$\pm$3.6 | 40.6$\pm$3.1 |  |
| Car-Goal-1 | Return | 32.0$\pm$0.6 | 31.7$\pm$1.6 | 33.6$\pm$0.5 | 28.1$\pm$1.2 | 29.8$\pm$0.8 | 21.0$\pm$0.8 |
| Cost | 43.4$\pm$4.1 | 42.0$\pm$12.5 | 51.1$\pm$3.9 | 30.5$\pm$3.3 | 36.4$\pm$2.9 | 15.4$\pm$2.1 |  |
| Car-Goal-2 | Return | 34.1$\pm$0.5 | 34.2$\pm$1.5 | 34.7$\pm$0.3 | 32.8$\pm$0.7 | 33.5$\pm$0.7 | 27.9$\pm$0.8 |
| Cost | 52.0$\pm$4.2 | 49.7$\pm$13.2 | 54.4$\pm$3.7 | 47.4$\pm$3.8 | 50.5$\pm$4.0 | 31.0$\pm$2.8 |  |
| Car-Goal-3 | Return | 35.2$\pm$0.3 | 35.3$\pm$0.7 | 35.2$\pm$0.2 | 35.0$\pm$0.3 | 35.1$\pm$0.3 | 34.3$\pm$0.4 |
| Cost | 56.2$\pm$4.8 | 55.2$\pm$14.3 | 56.4$\pm$4.1 | 55.3$\pm$4.0 | 55.5$\pm$3.8 | 53.1$\pm$4.1 |  |
| Point-Button-1 | Return | 25.9$\pm$1.0 | 26.4$\pm$2.2 | 27.0$\pm$0.8 | 22.0$\pm$0.9 | 23.0$\pm$0.9 | 8.8$\pm$0.7 |
| Cost | 92.9$\pm$8.1 | 92.8$\pm$23.5 | 114.5$\pm$6.4 | 61.5$\pm$6.6 | 66.5$\pm$6.5 | 12.2$\pm$2.7 |  |
| Point-Button-2 | Return | 29.2$\pm$0.7 | 29.8$\pm$2.2 | 28.7$\pm$0.8 | 28.3$\pm$0.8 | 28.8$\pm$0.8 | 10.3$\pm$0.9 |
| Cost | 118.7$\pm$7.3 | 123.0$\pm$21.4 | 122.1$\pm$7.3 | 113.2$\pm$7.5 | 114.6$\pm$8.7 | 19.1$\pm$3.0 |  |
| Point-Button-3 | Return | 30.6$\pm$0.7 | 30.8$\pm$1.9 | 29.6$\pm$0.6 | 30.4$\pm$0.6 | 30.3$\pm$0.7 | 14.9$\pm$1.1 |
| Cost | 130.9$\pm$9.0 | 129.2$\pm$23.7 | 129.9$\pm$7.3 | 131.5$\pm$8.8 | 128.6$\pm$8.7 | 55.5$\pm$7.6 |  |
| Car-Button-1 | Return | 14.1$\pm$1.2 | 14.3$\pm$3.6 | 17.6$\pm$1.5 | 10.1$\pm$1.1 | 11.8$\pm$1.3 | 2.3$\pm$0.4 |
| Cost | 132.3$\pm$12.7 | 126.6$\pm$38.3 | 165.7$\pm$14.4 | 101.0$\pm$14.3 | 116.2$\pm$13.1 | 35.9$\pm$5.4 |  |
| Car-Button-2 | Return | 21.0$\pm$1.3 | 20.6$\pm$3.6 | 22.9$\pm$1.1 | 18.7$\pm$1.4 | 21.4$\pm$1.5 | 5.1$\pm$0.7 |
| Cost | 191.4$\pm$13.9 | 189.4$\pm$45.8 | 209.8$\pm$12.2 | 178.0$\pm$14.3 | 198.1$\pm$15.4 | 65.8$\pm$10.1 |  |
| Car-Button-3 | Return | 24.8$\pm$0.9 | 24.3$\pm$3.4 | 25.2$\pm$0.9 | 24.1$\pm$1.1 | 24.7$\pm$1.0 | 14.0$\pm$1.5 |
| Cost | 223.7$\pm$10.8 | 229.5$\pm$41.6 | 230.4$\pm$11.2 | 220.9$\pm$12.8 | 232.5$\pm$10.9 | 144.0$\pm$15.2 |  |

Overall, as the difficulty increases, both the cost and return of all methods rise. This is primarily because the undesired data typically yields a much higher return than the safe data (except in the Point-Goal task, where the returns of safe and undesired data are close in return). BC-mix and IQ-mix struggle to distinguish between safe and undesired behaviors, leading to poor cost performance. IPL also fails to capture the correct preference, as most of its pairwise comparisons involve undesired-undesired pairs, making it unable to infer the correct preference. DWBC and SafeDICE manage to achieve relatively high returns but fail to match the cost performance of the Safe Policy. Our method consistently achieves the lowest cost across all experiments. However, in the Point-Button and Car-Button tasks, the return for our method is lower, as it avoids undesired actions, leaving no high-return options to pursue. The detailed learning curves are shown in Appendix D.1.

### 6.3 Mujoco Velocity benchmarks

Figure: Return
Refer to caption: x2.png

We test our method with two MuJoCo velocity tasks: Cheetah and Ant. We keep the same size of the unlabelled dataset as difficulty level 2 of the Safety-Gym experiments (Section 6.2), which is composed of 400 trajectories from the desired data and 1600 trajectories from the undesired data. Moreover, due to the nature of the environment, we only require a smaller number of labeled undesired datasets, which is $5$. Detailed results are shown in Figure 2, and a more detailed comparison of the MuJoCo domain is provided in Appendix C.5. In general, UNIQ outperforms other baselines in both tasks with a significantly lower cost.

### 6.4 Ablation Study - Performance with Different Sizes of Undesired Dataset

In this experiment, we evaluate our method using varying sizes of the undesired dataset (25, 50, 100, 200, 300, and 500 trajectories), while keeping the unlabelled dataset fixed (400 desired and 1600 undesired trajectories) across two Safety-Gym environments. The results, reported in Figure 3, include the return, cost, and CVaR $10\%$ cost for each undesired dataset size, where CVaR $10\%$ cost is the mean cost of the worst $10\%$ runs in the evaluation. The training curves are shown in the Appendix D.2. Here, BC-safe refers to Behavioral Cloning with only the desired (or expert) demonstrations, which serves as the highest safety performance benchmark. In general, increasing the size of the undesired dataset tends to reduce the cost for all approaches, and UNIQ shows the greatest effect in utilizing the undesired data. Interestingly, UNIQ is able to achieve a lower cost than BC-safe, which can be explained by the fact that the main goal of UNIQ is to avoid the undesired (i.e., high-cost) demonstrations, while BC-safe lacks this avoidance capability.

Figure: Return
Refer to caption: x7.png

## 7 Conclusion, Future Work, and Broader Impacts

Conclusion. We have developed UNIQ, a principled framework based on inverse Q-learning to facilitate learning from undesirable demonstrations. Our algorithm can be seen as a reverse version of standard imitation learning frameworks, where the goal is to maximize the statistical distance between the learning policy and the undesirable policy. UNIQ requires minimal hyper-parameter tuning, as it does not introduce any additional hyperparameters beyond those typically used in inverse Q-learning algorithms. Moreover, it demonstrates superior performance in producing safe policies in several safe reinforcement learning experiments, outperforming other baseline methods.

Limitations and Future work. There are several aspects that have not been addressed in this paper, as they are too significant to be fully explored here. For instance, we generally assume the presence of only one set of undesirable demonstrations, whereas in practice, multiple datasets of varying quality could be leveraged to enhance the training. Additionally, each undesirable trajectory may not be undesirable in its entirety, as it could contain some good actions. Extracting the good parts from undesirable demonstrations could improve sample efficiency but introduces new challenges that warrant further investigation. Another open question is how to extend the framework to multi-agent settings, which would be both relevant and interesting to explore in future research.

Broader Impact. Beyond the standard impacts of imitation learning, our work is particularly useful in scenarios where undesirable demonstrations are richer, clearer, or more reliable than desirable ones. This is especially valuable in critical applications like healthcare and autonomous driving, where avoiding harmful actions is essential. However, our approach also carries potential negative impacts. If not carefully designed, the system might unintentionally reinforce undesirable behaviors or learn harmful actions from poorly curated data. Misinterpreting bad demonstrations could result in unintended consequences, particularly in safety-critical contexts. Additionally, there is a risk that malicious actors could misuse this framework to deliberately train AI systems with harmful or unethical behaviors.

## Ethical Statement

Our work on the UNIQ framework addresses learning from undesirable demonstrations. This research holds potential for impactful real-world applications, including areas such as autonomous systems, healthcare, and robotics, where avoiding harmful behaviors is crucial. However, it is important to acknowledge the ethical implications and risks associated with our work.

While the UNIQ framework aims to learn safe policies by avoiding undesirable behaviors, there is a potential risk that the algorithm could be misused in ways that reinforce unintended or harmful actions if trained on poorly curated data. For example, in scenarios where undesirable demonstrations are not clearly defined, the model could inadvertently reinforce biased or harmful behaviors. Furthermore, there is the risk of deploying the framework in environments where safety-critical decisions are made without sufficient validation or human oversight, leading to unintended consequences.

To mitigate these risks, we emphasize the importance of using well-curated datasets that accurately represent undesirable behaviors and conducting thorough testing in controlled environments before applying the system to real-world scenarios. Moreover, we encourage transparency and collaboration with domain experts to ensure the framework is used responsibly, particularly in safety-critical applications such as healthcare and autonomous driving. Finally, we advocate for the inclusion of human oversight in the deployment of policies learned using the UNIQ framework to ensure ethical and safe outcomes.

## Reproducibility Statement

To ensure the reproducibility of our work, we have submitted the source code for the UNIQ framework along with the datasets used to generate the experimental results reported in this paper (this source code and datasets will be made publicly available if the paper gets accepted).
We also provide comprehensive details of our algorithm in the appendix, including implementation details and key steps required to reproduce the experimental outcomes. Additionally, we have included the hyper-parameter configurations used in all experiments, ensuring that others can replicate the results under the same conditions. We encourage the research community to build on our work and test the UNIQ framework across different environments to validate and extend the findings presented in this paper.

## Appendix

The appendix contains the following details:

Missing Proofs: Refer to Appendix A for proofs omitted from the main paper.

Experimental Details: We provide information on:

- •
Baseline implementation (Appendix B.3)
- •
Hyper-parameter selection for each task domain (Appendix B.4)
- •
Task descriptions (Appendix B.1)
- •
Generation of undesirable and unlabelled datasets (Appendix B.2)

Additional Experiments: We address the following remaining questions:

- •
(Q3) What happens if the policy is directly extracted from the Q-function, rather than by solving the WBC? (Appendix C.2)
- •
(Q4) How does the policy learned by UNIQ compare to a policy trained purely from expert demonstrations? (Appendix C.3)
- •
(Q5) How do UNIQ and other baselines perform when being trained without access to the unlabeled dataset? (Appendix C.4)

Additionally, we present experiments demonstrating the performance of UNIQ and other baseline methods on the dataset from the safeDICE paper (see Appendix C.6) and provide comparison results using CVaR costs (see Appendix C.1).

Supplementary Learning Curves: We provide learning curves to for the results reported in:

- •
Table 1 (Appendix D.1)
- •
Figure 3 (Appendix D.2)

## Appendix A Missing Proofs

### A.1 Proof of Proposition 4.1

##### Proposition.

For a non-restricted feasible set of the reward function:

$$ $\min_{r}\min_{\pi}\left\{L(\pi,r)\right\}=-\max_{\pi}\{d_{\psi}(\rho^{\pi}, \rho^{\textsc{UN}})-H(\pi)\}\$ $$

where $d_{\psi}(\rho^{\pi},\rho^{\textsc{UN}})=\psi^{*}(\rho^{\pi}-\rho^{\textsc{UN}})$, and $\psi^{*}$ is the convex conjugate of the convex function $\psi$, i.e., $\psi^{*}(t)=\sup_{z}(tz-\psi(z))$.

###### Proof.

We write the objective function as

$$ $L(\pi,r)=\sum_{s,a}r(s,a)\left(\rho^{\textsc{UN}}(s,a)-\rho^{\pi}(s,a)\right)+ \psi(r)-H(\pi)$ $$

So we can write the minimization of $L(\pi,r)$ over $r$ as:

$$ $\displaystyle\min_{r}L(\pi,r)$ $\displaystyle=H(\pi)+\min_{r}~{}~{}\sum_{s,a}r(s,a)\left(\rho^{\textsc{UN}}(s, a)-\rho^{\pi}(s,a)\right)+\psi(r)$ $\displaystyle=H(\pi)-\max_{r}\left\{\sum_{s,a}\Big{\{}r(s,a)\left(\rho^{\pi}(s ,a)-\rho^{\textsc{UN}}(s,a)\right)\Big{\}}-\psi(r)\right\}$ $$

Since the feasible set of the reward function $r$ is unrestricted, we know that

$$ $\max_{r}\Big{\{}\langle r,\rho^{\pi}-\rho^{\textsc{UN}}\rangle-\psi(r)\Big{\}} =\psi^{*}(\rho^{\pi}-\rho^{\textsc{UN}})$ $$

which allows us to rewrite the training problem as:

$$ $\displaystyle\min_{r}\min_{\pi}\left\{L(\pi,r)\right\}$ $\displaystyle=\min_{\pi}\left\{H(\pi)-\psi^{*}\left(\rho^{\pi}-\rho^{\textsc{ UN}}\right)\right\}$ $\displaystyle=-\max_{\pi}\left\{\psi^{*}(\rho^{\pi}-\rho^{\textsc{UN}})-H(\pi)\right\}$ $\displaystyle=-\max_{\pi}\{d_{\psi}(\rho^{\pi},\rho^{\textsc{UN}})-H(\pi)\}$ $$

We obtain the desired equation and complete the proof.
∎

###### Proof.

### A.2 Proof of Proposition 4.2

##### Proposition.

The following statements hold:

- (i)
The function $L(\pi,Q)$ is convex in $\pi$ and the problem $\min_{\pi}L(\pi,Q)$ has a unique optimal solution at $\pi^{Q}(s,a)=\frac{\exp(Q(s,a))}{\sum_{a^{\prime}}\exp(Q(s,a^{\prime}))}$.
- (ii)
The learning objective function can be simplified as:
$\min_{Q}\min_{\pi}\left\{L(\pi,Q)\right\}=\min_{Q}\left\{{\mathcal{F}}(Q)=
\mathbb{E}_{\rho^{\text{UN}}}[r^{Q}(s,a)]-(1-\gamma)\mathbb{E}_{s_{0}}[V^{Q}(s
_{0})]+\psi(r^{Q})\right\}$
where $V^{Q}(s)=\log\left(\sum_{a}\exp(Q(s,a))\right)$ and $r^{Q}(s,a)=Q(s,a)-\gamma\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}V^{Q}(s^{
\prime})$.

###### Proof.

We first express the second and third terms of the objective function $L(\pi,Q)$ in 5 as:

$$ $\mathbb{E}_{\rho_{\pi}}[\mathcal{T}^{\pi}[Q](s,a)]+H(\pi)=\mathbb{E}_{\rho_{ \pi}}[Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{\prime})]]-\mathbb{E}_{ \rho^{\pi}}[\log\pi(s,a)]$ $$

$$ $=\mathbb{E}_{\rho_{\pi}}[Q(s,a)-\log\pi(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{ \pi}(s^{\prime})]]=\mathbb{E}_{\rho_{\pi}}[V(s)-\gamma\mathbb{E}_{s^{\prime} \sim P(\cdot|s,a)}[V^{\pi}(s^{\prime})]]$ $$

$$ $=(1-\gamma)\mathbb{E}_{s_{0}\sim P_{0}}[V^{\pi}(s_{0})].$ $$

Thus, the objective function becomes:

$$ $L(\pi,Q)=\mathbb{E}_{\rho^{\text{UN}}}[Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^ {\pi}(s^{\prime})]]-(1-\gamma)\mathbb{E}_{s_{0}\sim P_{0}}[V^{\pi}(s_{0})]+ \sum_{s,a}\psi(Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{\prime})]).$ $$

We now observe that $V^{\pi}(s)=\mathbb{E}_{a\sim\pi(a|s)}[Q(s,a)-\log\pi(a|s)]$ is concave in $\pi$. Therefore, both terms $\mathbb{E}_{\rho^{\text{UN}}}[Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{
\prime})]]$ and $-(1-\gamma)\mathbb{E}_{s_{0}\sim P_{0}}[V^{\pi}(s_{0})]$ are convex in $\pi$. Additionally, since $\psi(t)$ is convex and non-increasing in $t$, and $Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{\prime})]$ is convex in $\pi$, each function $\psi(Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{\prime})])$ is convex in $\pi$. Thus, combining all terms, we conclude that $L(\pi,Q)$ is convex in $\pi$.

Furthermore, each term $Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{\prime})]$, $-(1-\gamma)\mathbb{E}_{s_{0}\sim P_{0}}[V^{\pi}(s_{0})]$, and $\psi(Q(s,a)-\gamma\mathbb{E}_{s^{\prime}})$ strictly decreases in $V^{\pi}$, implying that the minimization of $L(\pi,Q)$ over $\pi$ is achieved when $V^{\pi}(s)$ is maximized for all $s$. Since $V^{\pi}(s)$ is strictly concave in $\pi$, maximizing $V^{\pi}(s)$ over $\pi$ has a unique optimal solution:

$$ $\pi^{Q}(a|s)=\frac{\exp(Q(s,a))}{\sum_{a}\exp(Q(s,a))}.$ $$

This validates part (i) of the theorem.

For part (ii), we observe that the problem $\max_{\pi}V^{\pi}(s)$ has the optimal solution $\pi^{Q}$ as shown above, and the optimal value is:

$$ $\displaystyle\max_{\pi}V^{\pi}(s)$ $\displaystyle=\max_{\pi}\left\{\sum_{a}\pi(a|s)Q(s,a)-\pi(a|s)\log\pi(a|s)\right\}$ $\displaystyle=\log\left(\sum_{a}\exp(Q(s,a))\right)\stackrel{{\scriptstyle \text{def}}}{{=}}V^{Q}(s).$ $$

This directly leads to:

$$ $\min_{Q}\min_{\pi}\{L(\pi,Q)\}=\min_{Q}\left\{\mathcal{F}(Q)=\mathbb{E}_{\rho^ {\text{UN}}}[r^{Q}(s,a)]-(1-\gamma)\mathbb{E}_{s_{0}}[V^{Q}(s_{0})]+\psi(r^{Q} )\right\},$ $$

as required.

∎

###### Proof.

### A.3 Proof of Proposition 4.3

Proposition:
$g(\mu_{1},\mu_{2})$ is strictly concave in $\mu_{1},\mu_{2}$. Furthermore, let $\mu^{*}_{1}$ and $\mu^{*}_{2}$ be the unique optimal solutions to 7, then we have:

$$ $\tau(s,a)=\frac{\mu^{*}_{1}(s,a)}{\mu^{*}_{2}(s,a)}.$ $$

###### Proof.

We begin by expressing the objective function as follows:

$$ $\displaystyle g(\mu_{1},\mu_{2})$ $\displaystyle=\mathbb{E}_{\rho^{\textsc{Mix}}}\left[\log(1-\mu_{1}(s,a))\right ]+\mathbb{E}_{\rho^{\textsc{UN}}}\left[\log(\mu_{1}(s,a))\right]+$ $\displaystyle\mathbb{E}_{\rho^{\textsc{UN}}}\left[\log(1-\mu_{2}(s,a))\right]+ \mathbb{E}_{\rho^{\textsc{Mix}}}\left[\log(\mu_{2}(s,a))\right].$ $$

Each term, $\log(1-\mu_{1}(s,a))$, $\log(\mu_{1}(s,a))$, $\log(1-\mu_{2}(s,a))$, and $\log(\mu_{2}(s,a))$, is strictly concave in $\mu_{1}$ and $\mu_{2}$. Therefore, the objective function $g(\mu_{1},\mu_{2})$ is also strictly concave in $\mu_{1}$ and $\mu_{2}$. To find the unique optimal solution to this problem, we compute the first-order derivatives of $g(\mu_{1},\mu_{2})$ with respect to $\mu_{1}$ and $\mu_{2}$ and set them to zero:

$$ $\frac{\rho^{\textsc{UN}}(s,a)}{\mu_{1}(s,a)}-\frac{\rho^{\textsc{Mix}}(s,a)}{1 -\mu_{1}(s,a)}=0,$ $$

$$ $\frac{\rho^{\textsc{Mix}}(s,a)}{\mu_{2}(s,a)}-\frac{\rho^{\textsc{UN}}(s,a)}{1 -\mu_{2}(s,a)}=0.$ $$

This leads to the following system of equations:

$$ $\rho^{\textsc{UN}}(s,a)(1-\mu_{1}(s,a))=\rho^{\textsc{Mix}}(s,a)\mu_{1}(s,a),$ $$

$$ $\rho^{\textsc{Mix}}(s,a)(1-\mu_{2}(s,a))=\rho^{\textsc{UN}}(s,a)\mu_{2}(s,a).$ $$

Solving this system yields the unique solutions:

$$ $\mu_{1}^{*}(s,a)=\frac{\rho^{\textsc{UN}}(s,a)}{\rho^{\textsc{UN}}(s,a)+\rho^{ \textsc{Mix}}(s,a)},$ $$

$$ $\mu_{2}^{*}(s,a)=\frac{\rho^{\textsc{Mix}}(s,a)}{\rho^{\textsc{UN}}(s,a)+\rho^ {\textsc{Mix}}(s,a)}.$ $$

We observe that both $\mu_{1}^{*}(s,a)$ and $\mu_{2}^{*}(s,a)$ lie in the interval $(0,1)$, confirming that they are the unique solutions that maximize $g(\mu_{1},\mu_{2})$. Moreover, we validate the equality:

$$ $\frac{\mu_{1}^{*}(s,a)}{\mu_{2}^{*}(s,a)}=\frac{\rho^{\textsc{UN}}(s,a)}{\rho^ {\textsc{Mix}}(s,a)}.$ $$

∎

###### Proof.

## Appendix B Experiment settings

### B.1 Environment Details

#### B.1.1 Safe-Gym

Safe-Gym is a collection of reinforcement learning environments designed with a focus on safety, built on top of the OpenAI Gym framework. It introduces constraints that simulate safety-critical scenarios commonly encountered in real-world applications. In Safe-Gym, agents are rewarded for completing task-specific objectives but face penalties for violating safety constraints, such as surpassing speed limits, colliding with obstacles, or entering restricted zones. These constraints enable Safe-Gym to replicate environments where safety is paramount, including robotic navigation in congested areas, autonomous vehicle control, and industrial automation. The environment features two types of agents: Point (an easy agent) and Car (a more challenging agent), as well as two types of tasks: Goal (easy) and Button (hard). Additionally, the environment dynamics change with each new episode, introducing variability and increasing the complexity of the tasks. Illustrations of these tasks are shown in Figure 4.

Figure: Point-Goal
Refer to caption: x14.png

#### B.1.2 Mujoco-Velocity

Mujoco-Velocity is a specialized environment within the Mujoco physics simulation suite, focusing on controlling the velocity of two specific agents: Cheetah and Ant. These agents must complete locomotion tasks while adhering to safety constraints on their speed. The goal is to balance task performance with maintaining safe velocity limits. For instance, Cheetah must run as fast as possible while staying within predefined speed bounds to avoid penalties, mimicking real-world scenarios where exceeding speed limits can cause system failure or unsafe operations. Similarly, Ant must navigate through its environment without violating velocity constraints, ensuring stability and safety. The illustrations are shown in Figure 5.

Figure: Cheetah
Refer to caption: x18.png

### B.2 Dataset generation details

In this paper, we tackle the problem of Offline Reverse Imitation Learning by utilizing two datasets:

- •
Unlabelled dataset ${\mathcal{D}}^{MIX}$: A large dataset comprising both desired and undesired demonstrations, reflecting real-world data (e.g., chat conversations, driving behaviors, treatment decisions, etc.).
- •
Undesired dataset ${\mathcal{D}}^{UN}$: A smaller, accurated dataset containing demonstrations that exhibit behaviors we aim to avoid.

We simulate this scenario using the Safety-gym and Mujoco-velocity environments. First, we train both unconstrained and constrained policies using PPO and SIM (Hoang et al., ) (SIM is an incremental method that achieves significantly higher returns while incurring lower costs compared to other safe methods). We then collect the training datasets as follows:

- •
Unlabelled dataset ${\mathcal{D}}^{MIX}$: We roll out the constrained and unconstrained policies, mixing them at ratios of $(1:1,1:4,1:16)$ to progressively increase task difficulty.
- •
Undesired dataset ${\mathcal{D}}^{UN}$: We roll out the unconstrained policy, gathering trajectories that violate the constraint.

#### B.2.1 Safety-gym dataset detailed quality

The details information of Safety-Gym datasets are shown in Table 2.

**Table 2: Safety-Gym expert policies performance.**
|  | Point-Goal | Car-Goal | Point-Button | Car-Button |
| --- | --- | --- | --- | --- |
| Mean Unconstrained return | 26.8 $\pm$ 1.1 | 35.2 $\pm$ 2.1 | 30.0 $\pm$ 5.3 | 25.5 $\pm$ 8.4 |
| Mean Constrained return | 25.3 $\pm$ 2.1 | 25.7 $\pm$ 6.6 | 15.6 $\pm$ 7.3 | 4.1 $\pm$ 5.2 |
| Mean undesired return | 26.8 $\pm$ 1.1 | 34.7 $\pm$ 1.8 | 29.5 $\pm$ 5.3 | 25.3 $\pm$ 8.0 |
| Mean Unconstrained cost | 57.8 $\pm$ 38.9 | 61.9 $\pm$ 48.1 | 129.9 $\pm$ 77.3 | 245.5 $\pm$ 115.9 |
| Mean Constrained cost | 22.3 $\pm$ 28.0 | 21.0 $\pm$ 31.0 | 24.4 $\pm$ 34.1 | 51.6 $\pm$ 78.2 |
| Mean undesired cost | 80.9 $\pm$ 23.5 | 90.9 $\pm$ 32.2 | 139.0 $\pm$ 80.4 | 251.2 $\pm$ 105.7 |
| Cost Threshold | 25.0 | 25.0 | 25.0 | 50.0 |

#### B.2.2 Mujoco-velocity dataset detailed quality

The details information of safety-gym datasets are shown in Table 3.

**Table 3: Mujoco-velocity expert policies performance.**
|  | Cheetah | Ant |
| --- | --- | --- |
| Mean Unconstrained return | 3027.4 $\pm$ 400.6 | 2972.2 $\pm$ 1020.0 |
| Mean Constrained return | 2751.2 $\pm$ 11.6 | 2830.0 $\pm$ 145.5 |
| Mean undesired return | 3010.3 $\pm$ 424.2 | 2996.5 $\pm$ 991.5 |
| Mean Unconstrained cost | 626.7 $\pm$ 95.0 | 624.0 $\pm$ 231.0 |
| Mean Constrained cost | 14.1 $\pm$ 4.5 | 18.7 $\pm$ 4.4 |
| Mean undesired cost | 622.5 $\pm$ 99.7 | 629.6 $\pm$ 221.4 |
| Cost Threshold | 25.0 | 25.0 |

### B.3 Baseline implementation details

#### B.3.1 BC

We use the orginal BC objective:

$$ $\min_{\pi}-\mathbb{E}_{s,a\sim{\mathcal{D}}}\log\pi(a|s)$ (11) $$

#### B.3.2 IQlearn

We use the official implementation of IQ-Learn (Garg et al., ) from [this link](https://github.com/Div99/IQ-Learn). Moreover, to improve stability in offline training, we modify the actor update part to be the same as in UNIQ (Algorithm 1). The performance comparison between the two versions is provided in Appendix C.2.

#### B.3.3 IPL

We use the official implementation of IPL (Hejna & Sadigh, ) from [this link](https://github.com/jhejna/inverse-preference-learning/). The only difference between our setting and IPL is the pairwise dataset. We create pairwise comparisons from the unlabelled dataset and the undesired dataset and train the Q-function with the following new loss function:

$$ $P_{Q^{\pi}}[\sigma^{MIX}>\sigma^{UN}]=\frac{\exp\sum_{t}(\mathcal{T}^{\pi}Q)(s _{t}^{MIX},a_{t}^{MIX})}{\exp\sum_{t}(\mathcal{T}^{\pi}Q)(s_{t}^{MIX},a_{t}^{ MIX})+\exp\sum_{t}(\mathcal{T}^{\pi}Q)(s_{t}^{UN},a_{t}^{UN})},$ $$

where:

$$ $(\mathcal{T}^{\pi}Q)(s,a)=Q(s,a)-\gamma\mathbb{E}_{s^{\prime}}[V^{\pi}(s^{ \prime})].$ $$

#### B.3.4 DWBC

We use the official implementation of DWBC (Xu et al., ) from [this link](https://github.com/ryanxhr/DWBC/). We modify the algorithm to train a discriminator that assigns $1$ to the undesired dataset ${\mathcal{D}}^{UN}$ and 0 0 to the unlabelled dataset ${\mathcal{D}}^{MIX}$ while keeping the Positive Unlabeled learning technique from the paper:

$$ $\displaystyle\min_{\theta}\,\eta\mathbb{E}_{(s,a)\sim{\mathcal{D}}^{UN}}\left[ -\log d_{\theta}(s,a,\log\pi)\right]$ $\displaystyle+\mathbb{E}_{(s,a)\sim{\mathcal{D}}^{MIX}}\left[-\log\left(1-d_{ \theta}(s,a,\log\pi)\right)\right]$ $\displaystyle-\eta\mathbb{E}_{(s,a)\sim{\mathcal{D}}^{UN}}\left[-\log\left(1-d _{\theta}(s,a,\log\pi)\right)\right].$ $$

We then learn the policy by optimizing:

$$ $\min_{\psi}-\mathbb{E}_{(s,a)\sim{\mathcal{D}}^{MIX}}\left[\left(\frac{1}{d_{ \theta}(s,a)}-1\right)\log\pi_{\psi}(a|s)\right].$ $$

#### B.3.5 SafeDICE

We use the official implementation of SafeDICE (Jang et al., ) from [this link](https://github.com/jys5609/SafeDICE). As the algorithm has been designed to solve this problem, we do not make any further modifications.

### B.4 Hyper-parameter selection

For fair comparison, we keep the same basic hyper-parameters across all the baselines which are detailed as follow for Safety-gym and Mujoco-velocity tasks:

**Table 4: Hyper parameters.**
| Hyper Parameter | Safety-gym | Mujoco-velocity |
| --- | --- | --- |
| Actor Network | [256,256,256] | [256,256] |
| Critic Network | [256,256,256] | [256,256] |
| Training step | 1,000,000 | 1,000,000 |
| Gamma | 0.99 | 0.99 |
| lr actor | 0.0001 | 0.0001 |
| lr critic | 0.0003 | 0.0003 |
| lr Discriminator | 0.0001 | 0.0001 |
| batch size | 256 | 256 |
| soft update critic factor | 0.005 | 0.005 |

Lastly, we apply state normalization for Mujoco-velocity datasets as follow:

$$ $s_{\text{normalized}}=\frac{s-\mu}{\sigma}$ $$

Where:

$$ $\mu=\frac{1}{|\mathcal{D}|}\sum_{s^{\prime}\in\mathcal{D}}s^{\prime}$ $$

$$ $\sigma=\sqrt{\frac{1}{|\mathcal{D}|}\sum_{s^{\prime}\in\mathcal{D}}(s^{\prime} -\mu)^{2}}$ $$

## Appendix C Additional Experiments

### C.1 CVaR cost of Safety-gym comparison

We also report the CVaR $10\%$ cost, supporting the result of the Table 1 with CVaR is the mean of $10\%$ highest in cost trajectories during the evaluation process. The full results are shown in Table 5.

**Table 5: Full comparison results in Return, Cost, and CVaR $10\%$.**
|  |  | BC-mix | IQ-Mix | IPL | DWBC | SafeDICE | UNIQ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Point-Goal-1 | Return | 26.8$\pm$0.1 | 26.7$\pm$0.6 | 26.9$\pm$0.1 | 26.5$\pm$0.2 | 26.6$\pm$0.2 | 20.7$\pm$0.7 |
| Cost | 42.7$\pm$3.5 | 43.9$\pm$11.9 | 53.7$\pm$3.7 | 33.0$\pm$3.2 | 36.3$\pm$2.9 | 23.5$\pm$4.5 |  |
| CVaR | 106.3$\pm$10.3 | 103.8$\pm$54.4 | 119.3$\pm$7.8 | 90.1$\pm$9.6 | 94.8$\pm$10.9 | 101.0$\pm$38.0 |  |
| Point-Goal-2 | Return | 27.1$\pm$0.1 | 27.0$\pm$0.4 | 26.9$\pm$0.1 | 26.9$\pm$0.1 | 27.0$\pm$0.1 | 23.4$\pm$0.4 |
| Cost | 48.8$\pm$2.9 | 46.7$\pm$10.4 | 52.7$\pm$3.4 | 45.8$\pm$3.4 | 46.8$\pm$3.1 | 27.1$\pm$3.0 |  |
| CVaR | 115.4$\pm$7.7 | 105.3$\pm$24.1 | 117.9$\pm$8.2 | 110.4$\pm$7.8 | 111.0$\pm$7.6 | 85.9$\pm$18.9 |  |
| Point-Goal-3 | Return | 27.1$\pm$0.1 | 43.8$\pm$29.1 | 26.9$\pm$0.1 | 27.1$\pm$0.1 | 27.1$\pm$0.1 | 26.4$\pm$0.2 |
| Cost | 51.0$\pm$3.4 | 55.6$\pm$27.6 | 53.6$\pm$3.7 | 50.2$\pm$3.6 | 50.7$\pm$3.6 | 40.6$\pm$3.1 |  |
| CVaR | 116.8$\pm$7.6 | 106.6$\pm$21.7 | 119.5$\pm$8.3 | 115.7$\pm$7.8 | 116.6$\pm$8.1 | 102.6$\pm$10.3 |  |
| Car-Goal-1 | Return | 32.0$\pm$0.6 | 31.7$\pm$1.6 | 33.6$\pm$0.5 | 28.1$\pm$1.2 | 29.8$\pm$0.8 | 21.0$\pm$0.8 |
| Cost | 43.4$\pm$4.1 | 42.0$\pm$12.5 | 51.1$\pm$3.9 | 30.5$\pm$3.3 | 36.4$\pm$2.9 | 15.4$\pm$2.1 |  |
| CVaR | 117.5$\pm$10.9 | 103.4$\pm$30.5 | 129.4$\pm$9.7 | 90.5$\pm$10.3 | 103.8$\pm$9.9 | 62.1$\pm$8.7 |  |
| Car-Goal-2 | Return | 34.1$\pm$0.5 | 34.2$\pm$1.5 | 34.7$\pm$0.3 | 32.8$\pm$0.7 | 33.5$\pm$0.7 | 27.9$\pm$0.8 |
| Cost | 52.0$\pm$4.2 | 49.7$\pm$13.2 | 54.4$\pm$3.7 | 47.4$\pm$3.8 | 50.5$\pm$4.0 | 31.0$\pm$2.8 |  |
| CVaR | 132.8$\pm$10.6 | 114.5$\pm$29.0 | 134.9$\pm$8.7 | 123.2$\pm$10.6 | 128.5$\pm$10.2 | 93.3$\pm$8.4 |  |
| Car-Goal-3 | Return | 35.2$\pm$0.3 | 35.3$\pm$0.7 | 35.2$\pm$0.2 | 35.0$\pm$0.3 | 35.1$\pm$0.3 | 34.3$\pm$0.4 |
| Cost | 56.2$\pm$4.8 | 55.2$\pm$14.3 | 56.4$\pm$4.1 | 55.3$\pm$4.0 | 55.5$\pm$3.8 | 53.1$\pm$4.1 |  |
| CVaR | 140.3$\pm$12.3 | 127.9$\pm$35.0 | 139.5$\pm$10.9 | 139.5$\pm$10.9 | 138.8$\pm$9.9 | 134.8$\pm$9.4 |  |
| Point-Button-1 | Return | 25.9$\pm$1.0 | 26.4$\pm$2.2 | 27.0$\pm$0.8 | 22.0$\pm$0.9 | 23.0$\pm$0.9 | 8.8$\pm$0.7 |
| Cost | 92.9$\pm$8.1 | 92.8$\pm$23.5 | 114.5$\pm$6.4 | 61.5$\pm$6.6 | 66.5$\pm$6.5 | 12.2$\pm$2.7 |  |
| CVaR | 241.7$\pm$30.7 | 209.2$\pm$64.6 | 277.2$\pm$27.3 | 182.8$\pm$24.8 | 187.3$\pm$22.4 | 61.5$\pm$15.6 |  |
| Point-Button-2 | Return | 29.2$\pm$0.7 | 29.8$\pm$2.2 | 28.7$\pm$0.8 | 28.3$\pm$0.8 | 28.8$\pm$0.8 | 10.3$\pm$0.9 |
| Cost | 118.7$\pm$7.3 | 123.0$\pm$21.4 | 122.1$\pm$7.3 | 113.2$\pm$7.5 | 114.6$\pm$8.7 | 19.1$\pm$3.0 |  |
| CVaR | 273.6$\pm$30.4 | 263.6$\pm$97.7 | 281.6$\pm$29.0 | 267.4$\pm$27.0 | 270.1$\pm$33.8 | 81.1$\pm$14.9 |  |
| Point-Button-3 | Return | 30.6$\pm$0.7 | 30.8$\pm$1.9 | 29.6$\pm$0.6 | 30.4$\pm$0.6 | 30.3$\pm$0.7 | 14.9$\pm$1.1 |
| Cost | 130.9$\pm$9.0 | 129.2$\pm$23.7 | 129.9$\pm$7.3 | 131.5$\pm$8.8 | 128.6$\pm$8.7 | 55.5$\pm$7.6 |  |
| CVaR | 273.6$\pm$30.4 | 267.9$\pm$74.5 | 293.5$\pm$24.6 | 267.4$\pm$27.0 | 270.1$\pm$33.8 | 81.1$\pm$14.9 |  |
| Car-Button-1 | Return | 14.1$\pm$1.2 | 14.3$\pm$3.6 | 17.6$\pm$1.5 | 10.1$\pm$1.1 | 11.8$\pm$1.3 | 2.3$\pm$0.4 |
| Cost | 132.3$\pm$12.7 | 126.6$\pm$38.3 | 165.7$\pm$14.4 | 101.0$\pm$14.3 | 116.2$\pm$13.1 | 35.9$\pm$5.4 |  |
| CVaR | 387.0$\pm$44.2 | 334.6$\pm$125.3 | 430.4$\pm$47.2 | 345.5$\pm$49.7 | 363.8$\pm$47.4 | 183.6$\pm$30.7 |  |
| Car-Button-2 | Return | 21.0$\pm$1.3 | 20.6$\pm$3.6 | 22.9$\pm$1.1 | 18.7$\pm$1.4 | 21.4$\pm$1.5 | 5.1$\pm$0.7 |
| Cost | 191.4$\pm$13.9 | 189.4$\pm$45.8 | 209.8$\pm$12.2 | 178.0$\pm$14.3 | 198.1$\pm$15.4 | 65.8$\pm$10.1 |  |
| CVaR | 449.2$\pm$45.8 | 421.1$\pm$136.6 | 482.7$\pm$48.7 | 451.3$\pm$43.7 | 465.3$\pm$36.5 | 294.6$\pm$46.7 |  |
| Car-Button-3 | Return | 24.8$\pm$0.9 | 24.3$\pm$3.4 | 25.2$\pm$0.9 | 24.1$\pm$1.1 | 24.7$\pm$1.0 | 14.0$\pm$1.5 |
| Cost | 223.7$\pm$10.8 | 229.5$\pm$41.6 | 230.4$\pm$11.2 | 220.9$\pm$12.8 | 232.5$\pm$10.9 | 144.0$\pm$15.2 |  |
| CVaR | 474.5$\pm$39.3 | 461.9$\pm$136.4 | 500.3$\pm$53.3 | 473.1$\pm$36.6 | 492.4$\pm$48.8 | 432.0$\pm$44.1 |  |

### C.2 Comparison with Policies Directly Extracted from Q-functions

As our method use a different policy update compared to the original policy update of IQlearn (Garg et al., ), we want to show the reason why we modify policy update method into Weighted-BC.
Here, we keep the same data amount of level 2 for all four safety-gym environments. the comparison results are shown in Figure 6. From 5 training seeds, the result showing that the performance of direct extraction (original policy update of IQlearn) are not stable across different task while our new Weighted BC version have better stability.

Figure: Return
Refer to caption: x20.png

### C.3 Comparison with Policies Learned from Expert (or Desired) Demonstrations

We aim to evaluate the performance when learning directly from desired demonstrations (or expert demonstrations), without the noise of unlabelled undesirable trajectories in the unlabelled dataset. This can be considered the upper bound of performance without the need to handle undesired demonstrations. The results of the safe policy are presented in Table 6. Overall, UNIQ emerges as the algorithm closest to the Safe policy. In Section 6.4, we further demonstrate that when the amount of undesired data becomes sufficiently large, UNIQ can outperform and achieve a lower cost than a policy trained solely on desired demonstrations.

**Table 6: Compare baselines with Safe policy**
|  |  | DWBC | SafeDICE | UNIQ | Safe Policy |
| --- | --- | --- | --- | --- | --- |
| Point-Goal-1 | Return | 26.5$\pm$0.2 | 26.6$\pm$0.2 | 20.7$\pm$0.7 | 25.9$\pm$0.2 |
| Cost | 33.0$\pm$3.2 | 36.3$\pm$2.9 | 23.5$\pm$4.5 | 26.0$\pm$2.6 |  |
| Point-Goal-2 | Return | 26.9$\pm$0.1 | 27.0$\pm$0.1 | 23.4$\pm$0.4 | 25.9$\pm$0.2 |
| Cost | 45.8$\pm$3.4 | 46.8$\pm$3.1 | 27.1$\pm$3.0 | 26.0$\pm$2.6 |  |
| Point-Goal-3 | Return | 27.1$\pm$0.1 | 27.1$\pm$0.1 | 26.4$\pm$0.2 | 25.9$\pm$0.2 |
| Cost | 50.2$\pm$3.6 | 50.7$\pm$3.6 | 40.6$\pm$3.1 | 26.0$\pm$2.6 |  |
| Car-Goal-1 | Return | 28.1$\pm$1.2 | 29.8$\pm$0.8 | 21.0$\pm$0.8 | 26.2$\pm$0.7 |
| Cost | 30.5$\pm$3.3 | 36.4$\pm$2.9 | 15.4$\pm$2.1 | 23.6$\pm$2.8 |  |
| Car-Goal-2 | Return | 32.8$\pm$0.7 | 33.5$\pm$0.7 | 27.9$\pm$0.8 | 26.2$\pm$0.7 |
| Cost | 47.4$\pm$3.8 | 50.5$\pm$4.0 | 31.0$\pm$2.8 | 23.6$\pm$2.8 |  |
| Car-Goal-3 | Return | 35.0$\pm$0.3 | 35.1$\pm$0.3 | 34.3$\pm$0.4 | 26.2$\pm$0.7 |
| Cost | 55.3$\pm$4.0 | 55.5$\pm$3.8 | 53.1$\pm$4.1 | 23.6$\pm$2.8 |  |
| Point-Button-1 | Return | 22.0$\pm$0.9 | 23.0$\pm$0.9 | 8.8$\pm$0.7 | 16.4$\pm$0.9 |
| Cost | 61.5$\pm$6.6 | 66.5$\pm$6.5 | 12.2$\pm$2.7 | 29.1$\pm$3.5 |  |
| Point-Button-2 | Return | 28.3$\pm$0.8 | 28.8$\pm$0.8 | 10.3$\pm$0.9 | 16.4$\pm$0.9 |
| Cost | 113.2$\pm$7.5 | 114.6$\pm$8.7 | 19.1$\pm$3.0 | 29.1$\pm$3.5 |  |
| Point-Button-3 | Return | 30.4$\pm$0.6 | 30.3$\pm$0.7 | 14.9$\pm$1.1 | 16.4$\pm$0.9 |
| Cost | 131.5$\pm$8.8 | 128.6$\pm$8.7 | 55.5$\pm$7.6 | 29.1$\pm$3.5 |  |
| Car-Button-1 | Return | 10.1$\pm$1.1 | 11.8$\pm$1.3 | 2.3$\pm$0.4 | 4.4$\pm$0.6 |
| Cost | 101.0$\pm$14.3 | 116.2$\pm$13.1 | 35.9$\pm$5.4 | 56.7$\pm$7.6 |  |
| Car-Button-2 | Return | 18.7$\pm$1.4 | 21.4$\pm$1.5 | 5.1$\pm$0.7 | 4.4$\pm$0.6 |
| Cost | 178.0$\pm$14.3 | 198.1$\pm$15.4 | 65.8$\pm$10.1 | 56.7$\pm$7.6 |  |
| Car-Button-3 | Return | 24.1$\pm$1.1 | 24.7$\pm$1.0 | 14.0$\pm$1.5 | 4.4$\pm$0.6 |
| Cost | 220.9$\pm$12.8 | 232.5$\pm$10.9 | 144.0$\pm$15.2 | 56.7$\pm$7.6 |  |

### C.4 Learning without the Unlabeled Dataset

Since our algorithm is based on Objective 6, which focuses solely on avoiding undesired demonstrations, we want to compare its performance with a version that only avoids these undesired demonstrations:

- •
IQlearn-UN: Here, IQlearn are only have Undesired dataset. We solve the Objective 6 which only avoid the bad demonstration.
- •
BC-UN: similar to IQlearn-UN, here, we minimize the log prob of the bad demonstrations.

The results, shown in Figure 7, indicate IQlearn-UN and BC-UN have inconsistent performance. Without support from desired trajectories in the unlabelled dataset, this version fails to learn a correct policy.

Figure: Return
Refer to caption: x33.png

### C.5 Full Numerical Experiment for Mujoco Velocity Tasks

We evaluate our method on two MuJoCo velocity tasks: Cheetah and Ant. In addition to using a fixed set of 5 trajectories in the undesired dataset, we also test the method with varying sizes of the undesired dataset, annotated as ”env-UN$={\{1,5,10\}}$”. The detailed results are summarized in Table 7 and learning curves are shown in Figure 8 and Figure 9. Overall, increasing the size of the undesired dataset helps SafeDICE and DWBC achieve higher performance, while UNIQ reaches its peak performance with just a single undesired trajectory.

**Table 7: Full comparison between UNIQ and other baselines in Mujoco-velocity domain. With decreasing of undesirable dataset size, the performance of DWBC and SafeDICE become worse. In contrast, UNIQ able to achieve highest performance with just a single undesirable trajectory.**
|  |  | DWBC | SafeDICE | UNIQ |
| --- | --- | --- | --- | --- |
| Cheetah-UN=10 | Return | 3135.6$\pm$127.4 | 2841.9$\pm$56.1 | 2662.0$\pm$33.1 |
| Cost | 311.0$\pm$116.0 | 550.2$\pm$13.5 | 0.0$\pm$0.0 |  |
| CVaR | 897.7$\pm$10.0 | 682.2$\pm$14.4 | 0.0$\pm$0.0 |  |
| Cheetah-UN=5 | Return | 3430.9$\pm$107.5 | 2860.8$\pm$57.8 | 2661.2$\pm$29.7 |
| Cost | 578.8$\pm$89.6 | 553.9$\pm$25.3 | 0.0$\pm$0.0 |  |
| CVaR | 909.2$\pm$6.3 | 686.7$\pm$17.3 | 0.0$\pm$0.0 |  |
| Cheetah-UN=1 | Return | 3720.7$\pm$39.2 | 2910.0$\pm$61.8 | 2755.3$\pm$23.8 |
| Cost | 823.0$\pm$17.5 | 575.5$\pm$23.0 | 0.0$\pm$0.0 |  |
| CVaR | 916.4$\pm$5.0 | 702.2$\pm$20.0 | 0.0$\pm$0.0 |  |
| Ant-UN=10 | Return | 2225.0$\pm$759.3 | 2713.0$\pm$56.2 | 2850.5$\pm$177.5 |
| Cost | 470.5$\pm$162.8 | 439.7$\pm$57.7 | 15.2$\pm$10.8 |  |
| CVaR | 795.0$\pm$103.7 | 668.7$\pm$14.6 | 24.6$\pm$13.7 |  |
| Ant-UN=5 | Return | 2210.0$\pm$655.7 | 2727.4$\pm$49.8 | 2838.2$\pm$177.9 |
| Cost | 494.5$\pm$146.8 | 464.4$\pm$35.3 | 13.1$\pm$7.5 |  |
| CVaR | 805.7$\pm$16.4 | 671.2$\pm$16.0 | 22.1$\pm$10.5 |  |
| Ant-UN=1 | Return | 2259.4$\pm$653.8 | 2724.4$\pm$90.7 | 2841.4$\pm$214.9 |
| Cost | 507.5$\pm$147.8 | 506.5$\pm$40.3 | 16.9$\pm$7.1 |  |
| CVaR | 789.3$\pm$91.3 | 685.8$\pm$16.4 | 27.2$\pm$9.6 |  |

Figure: Return
Refer to caption: x46.png

Refer to caption: x54.png

### C.6 Performance with the Dataset Employed in the SafeDICE Paper

As we are using a different dataset from the SafeDICE (Jang et al., ) dataset, we requested the authors of SafeDICE to provide their dataset for comparison. The detailed performance of the expert dataset is shown in Table 8:

**Table 8: SafeDICE dataset performance.**
|  | Point-Goal | Point-Button |
| --- | --- | --- |
| Mean non-preferred demonstrations cost | 20.018 | 21.933 |
| Mean preferred demonstrations return | 19.911 | 8.286 |
| Mean non-preferred demonstrations cost | 107.977 | 166.099 |
| Mean preferred demonstrations return | 13.798 | 12.085 |

We mix 300 preferred demonstrations and 1200 non-preferred demonstrations for the unlabelled dataset and use 100 non-preferred demonstrations for the undesired dataset. The performance is shown in Figure 10. It is clearly that our method can achieve lower cost than SafeDICE.

Figure: Return
Refer to caption: x63.png

## Appendix D Detailed Learning Curves

### D.1 Learning Curves for the Results Reported in Table 1

Learning curve of Table 1 are shown in Figure 11,Figure 12, and Figure 13.

Figure: Return
Refer to caption: x70.png

Figure: Return
Refer to caption: x83.png

Figure: Return
Refer to caption: x96.png

### D.2 Learning Curves for the Results Reported in Section 6.4

Learning curve of Figure 3. The detailed results are shown in Figure 14 (Point-Goal) and Figure 15 (Car-Goal).

Figure: Return
Refer to caption: x109.png

Figure: Return
Refer to caption: x128.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Have you considered testing UNIQ in real-world domains? What specific challenges do you foresee in transferring the method from simulated benchmarks to real-world datasets?
soundness: 4
strengths: Problem novelty - This paper tackles the distinct challenge of learning policies that actively avoid undesirable behaviors, rather than merely replicating expert behaviors. This approach is valuable in many applications where high-quality expert data might be scarce or unavailable.

Crisp and well-rounded introduction - Good summary of the problem space, the traditional approach, and the gap it's trying to address. Clearly explained frameworks that the algorithm is based on (inverse Q-learning).

Implementation details - The paper provides source code along with comprehensive details on hyper parameter settings and experimental configurations, improving both reproducibility and transparency.
summary: This paper introduces a novel approach to learning from "undesirable" demonstrations. Instead of conventional imitation learning, which aims to imitate expert demonstrations, i.e. minimizing the distance with desired demonstrations, this method focuses on avoiding negative behaviors, i.e. maximizing the distance with undesired policy.

Based on MaxEnt RL framework and inverse Q-learning, the authors propose UNIQ, an offline inverse Q-learning algorithm designed to maximize the statistical distance between the target policy and undesirable policy. UNIQ utilizes both demonstrations of undesirable behavior and a broader pool of unlabeled data to develop a learning policy that effectively avoids unwanted actions.
weaknesses: Lack of validation using real-world data - While UNIQ is assessed on well-known benchmarks, evaluating it with real-world datasets from fields such as healthcare or autonomous driving could greatly enhance its value and demonstrate its applicability beyond simulated settings.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Q1. In line 220, the manuscript assumes that “$D^{MIX}$ may contain a mix of random, undesired, and expert demonstrations.” However, datasets used in experiments comprised with undesired and expert demonstrations. Could you provide additional exprimental results with mixing (safe) random demonstrations to the dataset?

Q2. Could you clarify what is meant by “the quality of the unlabeled dataset” in line 244? Does this refer to the proportion of preferred and non-preferred demonstrations in $D^{MIX}$? Additionally, why does UNIQ rely less on this compared to SafeDICE?
soundness: 2
strengths: This work integrates the IQ-Learn framework into the problem setting of learning from undesirable demonstrations, addressing a challenge in safe offline imitation learning. 
By extending IQ-Learn to handle scenarios with undesired behaviors, the authors provide a novel application that moves beyond traditional offline imitation learning and into the realm of safe policy learning, which is important in real-world applications.

The study goes beyond merely applying IQ-Learn to this setting; it rigorously justifies the suitability of IQ-Learn for learning from undesirable demonstrations. 
Through in-depth analysis, the authors clarify how the proposed objective captures the characteristics needed to discourage unsafe behaviors, and this thorough justifications strengthen the reliability for applications of UNIQ.

Additionally, the authors propose a practical version of the UNIQ algorithm, making it applicable to real-world scenarios where undesirable demonstrations are given. 
This implementation, combined with theoretical insights, enables safe policy learning in some practical offline learning scenarios.
summary: This work tackles the challenge of offline learning in scenarios with undesirable demonstrations, aiming to train agents that avoid unsafe or undesired behaviors in environments where undesirable and mixed demonstrations are given.
The paper presents UNIQ (**UN**desirable demonstrations-driven **I**nverse **Q**-Learning), an approach built on an inverse Q-learning (IQ-Learn) framework. 
By exploiting IQ-Learn framework, the study aims to learn a reward function that penalizes actions similar to those in the undesirable dataset, effectively steering the policy away from risky behaviors.
Instead of directly optimizing within the policy and reward spaces, UNIQ operates in the Q-space, following a similar approach to IQ-Learn. 
The optimized Q-function is then used to derive the policy through weighted behavior cloning, with weights informed by an advantage function computed from the Q-values.
Through this approach, UNIQ efficiently learns a policy that emphasizes safety by penalizing actions aligned with undesirable demonstrations, making it a robust method for offline learning in risk-sensitive environments.
weaknesses: **1. Questionable Design of Training Objective**

The training objective (Eq. 3) of UNIQ is focused solely on maximizing the discrepancy between the state-action distributions of non-preferred and imitator demonstrations. 
However, this objective lacks a mechanism to guide the policy toward with expert behaviors contained in mixed demonstration datasets. 
Consequently, the Q-function is primarily trained to penalize undesired demonstrations rather than to increase the Q-value of expert demonstrations.
This implies that when the unlabeled dataset contains a large number of safe yet non-optimal random actions in $D^{MIX} \setminus D^{UN}$, UNIQ may struggle to identify safe expert behaviors.

While this may result in a "safe" agent but it raises concerns regarding its usefulness for real-world applications. 
For instance, in an autonomous driving context, a policy that avoids non-preferred behaviors by simply stopping at the starting state could technically be safe,  yet it would lack the functional behaviors that essential for practical deployment.

**2. Limited Consideration of Experimental Scenarios** 

The experimental results also highlights a limitation of the current objective design, as UNIQ achieves relatively low costs and returns, indicating safety but at the expense of effectiveness.
These results appear insufficient to demonstrate that UNIQ is practically beneficial, as all evaluations are conducted on constrained RL benchmarks that primarily aim to maximize returns while keeping costs within predefined thresholds. 

As discussed in Section 1, the authors may argue that UNIQ is advantageous in certain scenarios — such as when expert demonstrations are entirely absent in the mixed dataset or when specific undesirable actions must be avoided by pretrained agents without compromising performance — the absence of experiments on these contexts weakens the argument for its applicability. 
The authors are encouraged to explore and present alternative scenarios and corresponding experiments that are suitable for applying the method's objectives and this would strengthen the justification of UNIQ’s approach and its practical relevance.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - What is the divergence optimized here?
    -  [1] demonstrated that IQ-Learn minimizes the chi-squared divergence between the policy and a mixture distribution. Here, however, it seems that the approach maximizes the chi-squared divergence between a mixture distribution and another mixture distribution that includes both desired and undesired behaviors.
- How were the baselines trained, like BC-Mix and IQ-learn Mix? Where they trained to imitate only the Mix distribution? If yes, I would argue that this is an unfair since you actually have access to the dataset you would like to imitate (the safe trajectories), and IQ-learn and BC should rather try to imitate that one.

[1] Al-Hafez et. al, LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning

Minor Points:
- try to write more compactly:
    - huge tables should be in appendix, only a reduced version should be in the main paper
- many of the propositions are very trivial extension previous ones, and probably not worth to specifically include them in the actual paper.
soundness: 2
strengths: - the concept is interesting and could indeed have important applications
- minor modification to existing approach, making application more straightforward
- the paper is well written
summary: The authors present a new framework for training a desired policy using two datasets: one containing undesirable trajectories and another with unlabeled data. The goal is to train the policy without an explicit definition of optimal behavior, relying instead on a concept of behaviors to avoid. The authors argue that this approach is crucial in fields like autonomous driving and healthcare, where unsafe actions could lead to catastrophic outcomes.
weaknesses: - By merely avoiding certain behaviors, the resulting solution space appears vast and highly ambiguous.
- It seems that the undesired dataset needs to be quite large to effectively narrow the space of desired behaviors. The experiments suggest that a substantial number of trajectories were used.
- The experiments also indicate that the unlabeled dataset may actually contain safe trajectories rather than genuinely unlabeled data, implying a notion of optimality within the datasets. These safe trajectories could be viewed as an expert distribution to match, making them suitable for standard imitation learning methods. (see questions)
- I would argue that the datasets used in this paper are not truly unlabeled but instead consist of trained behaviors—one representing safe behavior and the other unsafe—leading to a narrow solution space. I doubt that this approach would scale effectively to settings with genuinely unlabeled data. It would be valuable for the authors to include an ablation study addressing this point. They could, for instance, test scalability by using a small set of undesired behaviors along with some of the untrained D4RL dataset to validate their approach.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Can you provide theoretical or empirical evidence to demonstrate the stability of the proposed approach, particularly in complex tasks?
- How is the performance of the proposed method on DSRL benchmark datasets?
- What are the performances if you use the original offline-version IQ-Learn as well as the original DWBC learning objective as baselines?
soundness: 2
strengths: - The proposed problem setting is meaningful in practice, which can enable IL policy learning to avoid certain undesirable behaviors.
- Using DICE to convert the IQ-Learn framework to an offline method is interesting.
- The paper is well-organized and easy to read.
summary: This paper proposes an offline IL method that can leverage both expert or near-optimal demonstrations, as well as a collection of undesirable (sub-optimal) demonstrations. The proposed method aims to imitate the expert behavior while avoiding the undesired behaviors. The proposed method can be seen as a flipped version of IQ-Learn, which instead of minimizing the reward gap, it enlarges the gap with the undesired data distribution. As the IQ-Learn is derived under the online setting, the authors also borrow ingredients from the DICE method to estimate the occupancy ratio as importance weights to enable sampling from the data distribution. Although the problem setting is meaningful, I also have some concerns with the paper. Please see the strengths and weaknesses for detailed comments.
weaknesses: - The proposed method basically flipped the formulation in IQ-Learn, from minimizing the divergence with respect to expert distribution, to maximizing the divergence with respect to undesirable data distribution. Although it sounds reasonable, maximizing a divergence is often not a good idea, as it can produce ill-behaved optimization problems and should be avoided in most algorithm designs. Minimizing a statistical divergence may produce stable optimization towards a unique solution, but maximizing a divergence would encourage optimization in an arbitrary direction. Even under the exact same problem setting, SafeDICE still chooses to minimize some divergence to learn their policy. I have some concerns regarding the stability and solution quality of the proposed method in complex tasks. 
- The technical novelty of this paper is incremental, as it is basically an offline, flipped version of IQ-Learn. One new item proposed by the paper is the use of DICE-based method to estimate the occupancy ratio, in order to turn the original IQ-Learn framework from online sampling to offline sampling, but this is still somewhat incremental.
- I also have concerns regarding the heavy dependency on the IQ-Learn framework, as based on my past empirical experience, as well as results reported in many recent IL works, IQ-Learn often performs poorly in many tasks. One reason for this is that the optimization procedure of IQ-Learn is overly complex, which often negatively impacts policy learning stability. The proposed method is even more complex than IQ-Learn (has additional occupancy ratio estimation steps).
- The experiments also have a number of problems.
  - First, the authors use a very strange way to construct the unsafe and safe data (one is collected by an unconstrained PPO policy, and the other is collected with SIM). I don't think this is reasonable as compared to most real-world scenarios. data generated from an RL policy typically have a very narrow data distribution. The datasets generated by the authors would create a data distribution that has two easily separable peaks in distribution, which favors discriminator learning, and also makes the problem easy to solve. The authors should consider evaluating on some more commonly used safe offline RL benchmark datasets, like DSRL[1], which could make the conclusion more convincing.
  - Some baselines are also problematic, which are different from the versions reported in their original paper. The authors need to explain and justify why these changes were made.
    - The original IQ-Learn is derived under the online setting, which is not compatible with the offline setting. In the original IQ-Learn paper, their authors did provide an offline version (see Section 5.1 in the IQ-Learn paper), which changed the way how advantage term is estimated, but this version has some theory-to-practice gaps. The authors mentioned that they use the official implementation of IQ-Learn, but modified the actor update part. I'm not sure if the right offline version of IQ-Learn is compared, since comparing with the online version is meaningless. 
    - The "DWBC" baseline is also far from the version reported in the original DWBC paper, with completely different weighting schemes for policy learning. 

[1] Liu et al. Datasets and Benchmarks for Offline Safe Reinforcement Learning.

Meta-review: (a) Summary: This work tackles the challenge of offline learning in scenarios with undesirable demonstrations, aiming to train agents that avoid unsafe or undesired behaviors in environments where undesirable and mixed demonstrations are given. The paper presents UNIQ (UNdesirable demonstrations-driven Inverse Q-Learning), an approach built on an inverse Q-learning (IQ-Learn) framework. 
(b) Strengths: The paper is generally well-written and easy to follow. The contributions seem to be interesting and reasonable.
(c) Weaknesses: The reviewers pointed out a few major concerns and issues. The novelty is incremental. Some important experimental comparisons are insufficient. The authors did not reply to address the remaining issues after the reviewers replied to the rebuttal.

Decision: Reject