PN-GAIL: Leveraging Non-optimal Information from Imperfect Demonstrations
Authors: Qiang Liu
Huiqiao Fu
Kaiqiang Tang
Chunlin Chen
Daoyi Dong
Venue: ICLR 2025 Poster (2025)
Keywords: Generative adversarial imitation learning
imperfect demonstrations
reinforcement learning
Primary area: reinforcement learning
TL;DR: We introduce a Positive-Negative Generative Adversarial Imitation Learning (PN-GAIL) method within the framework of Generative Adversarial Imitation Learning (GAIL) to leverage non-optimal information from imperfect demonstrations.

Abstract: Imitation learning aims at constructing an optimal policy by emulating expert demonstrations. However, the prevailing approaches in this domain typically presume that the demonstrations are optimal, an assumption that seldom holds true in the complexities of real-world applications. The data collected in practical scenarios often contains imperfections, encompassing both optimal and non-optimal examples. In this study, we propose Positive-Negative Generative Adversarial Imitation Learning (PN-GAIL), a novel approach that falls within the framework of Generative Adversarial Imitation Learning (GAIL). PN-GAIL innovatively leverages non-optimal information from imperfect demonstrations, allowing the discriminator to comprehensively assess the positive and negative risks associated with these demonstrations. Furthermore, it requires only a small subset of labeled confidence scores. Theoretical analysis indicates that PN-GAIL deviates from the non-optimal data while mimicking imperfect demonstrations. Experimental results demonstrate that PN-GAIL surpasses conventional baseline methods in dealing with imperfect demonstrations, thereby significantly augmenting the practical utility of imitation learning in real-world contexts. Our codes are available at https://github.com/QiangLiuT/PN-GAIL.

Published as a conference paper at ICLR 2025
PN-GAIL: LEVERAGING NON-OPTIMAL INFORMA- TION FROM IMPERFECT DEMONSTRATIONS
Qiang Liu, Huiqiao Fu, Kaiqiang Tang & Chunlin Chen
School of Management and Engineering Nanjing University
Nanjing, China
{qiangliu, hgfu, kgtangg)smail njj.
.jj. edu. en, cichenênju. edu, cn
Daoyi Dong
The Australian Artificial Intelligence Institute University of Technology Sydney
Sydney, Australia daoyidong@ggaill. com
ABSTRACT
Imitation learning aims constructing optimal policy by emulating expert demonstrations. However, the prevailing approaches this domain typically pre- sume that the demonstrations in the complexities of real-world applications. The data collected in practical sce- narios often contains imperfections, encompassing both optimal and non-optimal examples. In this study, we propose Positive-Negative Generative Adversarial Im- itation Learning (PN-GAIL), a novel approach that falls within the framework of Generative Adversarial Imitation Learning (GAIL). PN-GAIL innovatively lever- ages non-optimal information from imperfect demonstrations, allowing the dis- criminator to comprehensively assess the positive and negative risks associated with these demonstrations. Furthermore, it requires only a small subset labeled confidence scores. Theoretical analysis indicates that PN-GAIL deviates from the non-optimal data while mimicking imperfect demonstrations. Experimental re- sults demonstrate that PN-GAIL. surpasses conventional baseline methods in deal- ing with imperfect demonstrations, thereby significantly augmenting the practical utility of imitation learning in real-world contexts. Our codes are available https:/ /ttt:////ttu...../ com hwtm///aaagnin/nanng GAIL
optimal, assumption that seldom holds true
INTRODUCTION
In recent years, Reinforcement Learning (RI.) has achieved significant success in addressing sequen- tial decision-making problems (Sutton & Barto (018)Xin, et al. 2020 /ha et al. (021). Its primary goal is optimize policies maximize cumulative rewards. However, designing an appropriate reward function can be quite challenging; poorly designed reward function can lead to suboptimal performance of RL agents. contrast, Imitation Learning (IL) presents a more practical approach, as it learns solely from demonstrations, eliminating the need for explicitly defined reward functions. Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon] 2016). which employs the frame- work of Generative Adversarial Networks (GANs) (Goodfellow et al.| directly learns a policy from demonstrations. Following the development of GAII,, many variants have been proposed to
enhance algorithmic performance across different problem domains (Li et al. 2017 Fu al. 2018
Dadashi et al. 2020 Fu et al. 2024).
The imitation learning methods mentioned above can learn an optimal policy given optimal demon- strations. However, most imitation learning methods tend to fail when faced with data filled with imperfect demonstrations. Especially the real world, the assumption that the provided demonstra- tions are of high quality may not always be valid (Yang et al. (024). For instance, due to factors
"Corresponding author

---

Published as a conference paper at ICLR 2025
such fatigue and distractions, decisions made by human experts may not always be optimal. In such cases, simply assigning equal weight to all data can lead to a decrease in the quality of the learned policy. Therefore, we need method that can extract useful information from imperfect demonstrations to learn an optimal policy.
Existing methods for imitation learning from imperfect demonstrations can be broadly divided into two categories: weighting-base based methods (Wu et al.) [019 Wang et al. 2021b a Tangkaratt et al.
2020 Zhang et al. 2021 Wang et al. 2023) and ranking-based methods (Brown et al. 2019; 2020,
Chen et al. 2021 Huo et al. 2023 Taranovic el al. 2022). Weighting-base based methods achieve imitation optimal demonstrations through reweighting different demonstrations, while ranking- based methods aim to guide the recovery of the reward function with additional ranking information, thereby learning an optimal policy based on the rewards. contrast, weighting- -based methods are more computationally efficient since they do not require trajectory sorting. Additionally, they are more flexible to use, as they do not necessitate demonstrations to be in a trajectory form.
In order to solve the problem of learning from imperfect demonstrations using GAIL, Wu et al.) (2019) proposed two methods: two-step importance weighting IL (2IWIL) and generative adver- sarial IL with imperfect demonstration and confidence (IC-GAIL). The former trains a classifier to forecast confidence scores and subsequently proceeds with weighted imitation learning, employing two-step learning approach. The latter introduces an end-to aad-to-end learning method but at slower pace of learning. However, as discussed in Section 4.1 2IWII. is susceptible to the influence of preferences inherent in imperfect demonstrations during training. In the learning process of the discriminator, 2IWIL tends to assign higher "reward" to the state-action pair with a greater prob- ability of occurrence in imperfect demonstrations. This discrepancy in "rewards" diverges from our intended objectives, potentially resulting in the acquisition of a suboptimal policy.
To tackle the aforementioned challenge, we propose a new method, Positive-Negativy Generative Adversarial Imitation Learning (PN-GAIL), building upon the framework of GAIL. Different from 2IWIL, we leverage non-optimal information from imperfect demonstrations, enabling the discrim- inator to weigh both positive and negative risks of imperfect demonstrations comprehensively and requiring only a small subset of labeled confidence scores. In this way, it can provide more accurate rewards for subsequent RL methods. Theoretical analysis reveals that PN-GAIL not only mimics imperfect demonstrations but also avoids imitating non-optimal ones, illustrating the ability of PN- GAIL to learn an optimal policy. Additionally, to get more accurate confidence scores, propose an improved semi- cemi-supervised confidence classifier. Experiments on six control tasks are conducted to show the efficiency of our method in dealing with imperfect demonstrations compared to baseline methods. In particular, the main contributions of this work are threefold:
1. We propose a new method called PN-GAIL, which can leverage non-optimal information to learn an optimal policy from imperfect demonstrations.
2. We theoretically analyze the output of the optimal discriminator in PN-GAIL, demonstrat- ing that PN-GAIL learns an optimal policy by deviating from the non-optimal demonstra- tions.
3. We demonstrate the efficiency of our method across six control tasks, with results showing superior performance compared to other baseline methods.
2 RELATED WORK
Imitation Learning Imitation learning methods can learn an optimal policy when given optimal
demonstrations. Behavior cloning (BC) (Pomerlcau) 1988) learns policies directly through su- pervised learning paradigm and is mostly used in autonomous driving tasks (Hawke et al. 2020). While straightforward, it suffers from compounded errors due to covariate shift (Ross & Bagnell 2010) and typically demands extensive data for effective training. Inverse Reinforcement Learning (IRL) (bbbell & Ng 2004 Ziebart et al., 2008) first seeks to recover the underlying reward function and then learns policy through RL. On the other hand, GAIL views imitation learning problem through the lens of occupancy measures (Puterman) 2014). and can learn a policy directly from the demonstrations. GAIL has demonstrated success across various imitation tasks, including multi- agent scenarios (Song et al. 2018). robot control (Peng et al. 2021). human motion simulation (Wei)
et al. 2021), and imitation of driver behavior (Bhattacharyya et al. 2022 Ruan & Di 2022). How-
2

---

Published as a conference paper at ICLR 2025
ever, these methods presuppose access to optimal demonstrations. When provided with imperfect demonstrations, they struggle to learn a good policy.
Weighting- imitation learning from imperfect demonstrations Weighting-based imita- tion learning from imperfect demonstrations learns an optimal policy by reweighting different demonstrations and amplifying the significance of the optimal ones. 2IWIL and IC-GAIL (Wu al., 2019) first propose to reweight imitation learning based on confidence. WGAIL (Wang et al. (202 connects confidence with the agent policy and discriminator without requiring additional prior infor- mation on confidence. However, it needs a high proportion of optimal demonstrations in imperfect demonstrations. VILD (Tangkaratt ct al., 2020) employs a variational method to jointly estimate demonstration quality and reward, but assumes that the quality of demonstrations be correlated with variance. CAIL (Zhang et al. 2021) guides confidence estimation by introducing trajectory ranking. UID (Wang et al.] (023) treats imperfect demonstrations as unlabeled data, based on the idea of PU Learning (Du Plessis et al. 2014), mitigating the impact of non-optimal demonstrations. Nevertheless, this relies on the assumption that non-optimal demonstrations within the imperfect demonstrations can well match agent demonstrations. Additionally, some studies address imper- feet demonstrations in offline imitation learning (Sasaki & Yamashina [2000] Xu al., [022)] Kim itt al. [22TT; Yu et al. [2023] Li et al. 2024). However, these methods either similarly assume that the proportion of the optimal demonstrations is dominant, or require an additional set of optimal demonstrations.
Ranking-based imitation learning from imperfect demonstrations Ranking-based imitation learning from imperfect demonstrations utilizes additional ranking information to guide the recovery of the reward function, thereby learning a policy based on the rewards. T-REX (Brown et al.) 2019) infers the reward function from the given ranking trajectories and expects the reward function to conform to the given ranking order. However, this approach demands a substantial quantity of rank- ing trajectories to enhance its generalization capacity. D-REX (Brown et al.. (020) automatically generates ranking trajectories by introducing varying degrees of noise. SSRR (Chen et al., 2021) revises the structure of the reward function in D-REX to accommodate different levels of noise in- fluence better. LERP (Huo et al., 2023) views suboptimal demonstrations as additive noise on the reward function, establishing a quantifiable relationship between noise and reward based on D-REX. However, the automatic generation of the ranking trajectories requires the assumption that the tra- jectory will receive lower rewards with the addition of noise, which is not necessarily true in cases where random demonstrations exist. AILP Taranovvcc et al., (002) necessitates the teacher's access to the true reward function, thereby providing real-time correct ranking between two trajectories. Nevertheless, this condition is challenging to meet in practice.
PRELIMINARIES
In this section, we provide a brief background on RL, GAIL, and 2IWIL.
Reinforcement learning We consider the standard Markov Decision Process (MDP) (Sttonn & Barto 2018). An MDP typically comprises six components, denoted as M = A, R, Po. 7). where S is the state space, A is the action space, P(s az) ) is the transition probability from state 8t and action a; at time step t to state $t+1 at time step t + 1, R(s, a) is the reward func- tion, Po is the distribution of initial states, and 7 (0, 1) stands for the discount factor. In an RL process, the agent aims to learn a policy TTass) to maximize its expected discounted rewards Eegrrpor
Dee 7 *R(ss,, a.)]. For any given policy æ, there exists corresponding occupancy mea- sure Px: S.S. X A + R. establishing a one-to-one relationship between them.
GAIL and 2IWIL GAIL integrates GANs framework into imitation learning, leading the follow- ing min-max optimization problem by minimizing the Jensen-Shannon divergence between pa and PE [ee et al. 202I):
min max E (ssa))>pp [log (s, w
(ss a)] + Bes.))-p [lo([1 (s,
where pa and pr: are the corresponding normalized occupancy measures for the agent policy Ty and the expert policy me, respectively. The discriminator D,, attempts to discern these distributions from
3

---

Published as a conference paper at ICLR 2025
TE and Te. while Th aims to "trick" the discriminator, thereby minimizing ((s.a)~pp.l llog D (s, a)]- Ultimately, the output of the discriminator, log (ur a),, serves as reward, which can then be utilized learn the policy ne through RI. methods such as TRPO (Schulman al. 2013). PPO (Schulman et al. 2017) and SAC (Haarnoja al. 2018).
Since GAIL assigns the same weights to demonstrations, if the given demonstrations are non- optimal, then the learned policy will also be non-optimal. To address this issue, 2IWIL considers the following setup:
D. _ ((ra re)] Ltd. ((x,)),
Du 2 { pi Ltd. p(x),
where z is the state-action pair, r denotes confidence score, indicating the probability that belongs to the optimal demonstrations, g(z, r) = p(z)pe(r(a) and (r(rz(z) = 5(r, r(x)) is Dirac delta function. De and D,, represent confidence data and unlabeled data, respectively.
2IWIL first trains probabilistic classifier, which forecasts the confidence scores of demonstrations in Du through semi-conf (SC) classification, leveraging the knowledge of confidence scores De. The probabilistic classifier is trained with the loss function as follows:
Rgc, = Er [r(((())) + +1 - (()r)((-g(()) - 3((-)(>))] + -xxxp((((g(())1
(2)
where g is prediction function, f a loss function which uses logistic loss and B tna After obtaining confidence scores for all demonstrations, 2IWIL uses Bayes' rule reweight the GAIL objective. The final objective becomes
nu
r(()) log(1 - -Du(t)))
min max Exwpo [log (x)) + Ez~p
(3)
where 7) is a class-prior denoting the proportion of optimal demonstrations within the imperfect demonstrations, and p is the corresponding normalized occupancy measures for D. and D...
4 APPROACH
In this section, we begin by elucidating the motivation behind our method. We illustrate the problem of 2IWIL through an example and then introduce our method PN-GAIL with theoretical analysis. Details of derivations and proofs in this section can be found in Appendix
4.1 MOTIVATION
2IWIL aims to reweight demonstrations based on confidence, assigning greater weights to those with high confidence so that the discriminator can give higher rewards. However, it is worth noting that this weighting behavior can be influenced by the preferences inherent in imperfect demonstrations. As shown in Fig. [ the top half of the graph represents the actual confidence scores, while the bottom half represents the equivalent weights during the discriminator's training. When imperfect demonstrations favor a low-confidence state-action pair, we consider the goals of GAIL, 2IWIL:
T((x) log(1 - Du (1:)
min max Ezaop [log (x)) + Ezzop 9
n
r(x) 1.0. For the GAIL assigns the same weights all given demonstrations, which means given expert demonstrations, we only need to consider the second term of the above equation, which can be expanded as: Σ p(x) r(a) log(1 - (=)).. Here, the coefficient of log(1 - (x)) is riz) Therefore, if there is a higher probability of æ appearing in imperfect demonstrations, e.g., p(x1)= 5p(sother) (assuming that the probabilities of other state-action pairs are the same), then, for 21, the coefficient of log(1 (x)) is p(r1) r(a,) = p(Fother) Gr(aa) This can also be explained as that the probability of 21 appearing is the same as for other demonstrations, but the confidence score is 5 times higher, since is constant across all demonstrations, In the case of GAIL, since r(x) 1.0, the confidence score becomes 5 times of the original, which is calculated as 1.0 5.0. This
"
4

---

Published as a conference paper at ICLR 2025
GAIL
ZIWIL
PN-GAIL((urs))
True confidence
0..5
0.5
0.5
82
0.2
0.2
0.7
0.7
9.7
0.4
0.4
0.4
0.5
0.2
0.9
0.2
0.9
Q.2
0.9
Equivalent weights
0.5
0.5
50
1.0
1.0
0.2
0.2
0.7
0.7
1.0
0.4
0.4
5.0
8.8
0.8
0.3
6.3
3.0
100
0.9
Highhconffieenee data
Low-cenfidence data
Figure 1: Schematic diagram of the difference between PN-GAIL, 2IWIL and GAIL. The top half of the graph is the actual confidence score, the bottom half the equivalent weight when training the discriminator, and the red line is distinguished by threshold of 0.5.
can lead to low-confidence data being treated as high-confidence data, not aligning with the actual situation. In addition, for clearer explanation, also provide simple example.
Suppose a state 81 has two actions #1 (81, a1) and (2($1, 02). Fig. [ the circle with a confidence score of 0.8 represents 22, and the five circles with confidence score of 0.2 all represent æ1. which means p(x)) ) = (2),, indicating a higher probability of ø occurring in imperfect demonstrations. It is clear that 22 is better than æ1- However, in imperfect demonstrations where p(æ1) 5 and the prior v) is the same, according to Eq. (3), the equivalent weight of #1 will be 1.0 compared to 22 (0.2 5 1.0). This means that the discriminator will consider 2j be more likely the optimal demonstration than T2, resulting in a poor policy.
4.2 POSITIVE-NEGATIVE GENERATIVE ADVERSARIAL IMITATION LEARNING
To tackle the problem above, we propose Positive-Negatiio Generative Adversarial Imitation Learn- ing (PN-GAIL). This method leverages non-optimal information from imperfect demonstrations. allowing the discriminator to comprehensively assess the positive and negative risks associated with these demonstration. By doing it mitigates the influence of preferences inherent in imperfect demonstrations on the discriminator, thus ensuring that its evaluations better reflect actual condi- tions. This, in turn, provides more accurate rewards for the subsequent RL process, leading the learning of a better policy.
We begin by focusing on the training of the discriminator, denoting optimal demonstrations pos- itive examples and non-optimal demonstrations as negative examples. In 2IWII., the discriminator only considers the positive risk of imperfect demonstrations, while ignoring negative risk. Therefore, the discriminator will heavily prioritize the positive risk training for state-action pairs frequently appearing in imperfect demonstrations, leading to incorrect results. For this reason, we aim to incor- porate the negative risk into the training of the discriminator when dealing with imperfect demon- strations. Specifically, following Xu & Denil (2021). let (X, Y) represent the input and output of binary classification problem, where X denotes the state-action pair and Y E {0, 1}. We label opti- mal data as and non-optimal data as 1. The imperfect demonstrations denoted as D. comprising Dops (optimal demonstrations) and Dnon (non-optimal demonstrations), where D Dopt + D We aim to train a discriminator De, using a loss function ø IR x {0, 1} + Utilizing the labeled risk operator as follows:
R% = (x)
(4)
We expect the discriminator to provide accurate evaluation scores for both the dataset generated by the agent policy and the given imperfect demonstrations. To achieve this, we consider the risk associated with the dataset generated by the agent policy and the risk associated with the imperfect
5

---

Published as a conference paper at ICLR 2025
demonstrations, respectively. The overall risk of the discriminator is
RD (D... D) = (......... + RD. (D);
(5)
where DTo is the demonstrations generated by agent policy me. We can write the risk associated with the imperfect demonstrations as the sum of positíve and negative risks:
P (D) D. (Dopt: Pnon) n, (Popt) + (1 R) D,. (Dnon).
(6)
where 7 ply 0) a class-prior, denoting the proportion of optimal demonstrations within the imperfect demonstrations.
Based on Eq. ( the overall risk of the discriminator can be rewritten as Re (D, D=,) )- Rb (Dn,) 17 n (Popt) + (1 17) Rb_ (D
(7)
Replacing the loss function with the standard logistic loss and tidying up the statement, the objective of the discriminator becomes
max Exrpo [log (w)) + rBBapppp [log(1 - (w))))] + (1 E Dw
(8)
w
Since r(z) denotes the probability that x belongs to the optimal demonstrations, which means r(x) p(y 0/æ) and - r(z) p(u 1/x), according to the Bayes' rule we have
Popt (x) = p<xly = 0) r(z)p(z) Puon (x) = p(zly = 1) (1 - r(x)))))
(9)
1
1
Then we can rewrite the objective of the discriminator in the following theorem.
Theorem 4.1. Based on Eq. ( the objective of the discriminator can be rewritten as
max Ex-ro
[log (x)) + Ez-p (r(x) log(1 (x)]] + Ezzp [11 r(x)) log (x)) (10)
The agent receives a reward equivalent log D (z)), and then the final objective be optimized becomes
min max Ep [log (x)] + Expp [r(z) log(1 1 Dw + Ex-p [(1 (o
(1I)
Furthermore, recall that 2IWIL adopts two-step learning approach, where D. and D, represent confidence data and unlabeled data, respectively. To get more accurate confidence scores, we refine the semi-conf (SC) classification proposed in 2IWIL, which is trained by minimizing the following
risk:
Rsc,e(g) 1 Er (r((g(z)) + (1- (---((())-((---(----)----------- (Er--((()))
(12)
We note that for state-action pair occurring solely in D., once 1 r O, where r > 1- B, the coefficient of ((-g(z)) becomes negative. In order to minimize the risk, the classifier would then forecast g(z) as positive infinity, leading to excessively high estimation of confidence for demonstrations in D.. Concurrently, Eq. (12) tends to predict data in D, as negative, resulting in an underestimated confidence for demonstrations in D.. To balance this effect, we propose balanced semi-conf (BSC) classification. We introduce [x=p(a((g(z)) - Ex=glal(g(x)) the theoretical value of which is 0 since De and Du are drawn from the same distribution p(z). The final risk is as follows:
Rg) Ex [r((g(x)) + (1 -
(13) where the loss function uses the logistic loss. Next, similar to 2IWIL, we seek derive the optimal values of a and 3 for minimizing the variance of the empirical unbiased estimator (Bsc, through the following theorem.
+ HEEpplal (C)=pla((g())))1 (((-)(z))),
Theorem 4.2, Let di denote Var(((-g(z))), d2 denote Var(e(g(z))) Jcovl denote the covari- ance between 1 Ere (E (Fe,s) (-9 (xr,))) and Ene (-gg (Fe,i)), aoo22 denote the covariance between ne
(g (Fe,))) and 1 E. Egg (ze,)), cov de- note Cov (e (-9 (x)), (g (x:)). The estimator (gsc,r(g) has the minimum variance when
EP= (1 (1-r;) (r) (=.,
ne
nn
dicov 2 nu
+ dioem2 coUo R2,
a ne + nu
d1d2 - 2o12 The + nu
d1d2 cov2 Ne + Tn
nu
d2c0v cov2
Nu
d2d cOvO DeDuu
3
+ d1dz cor2
ne + nu
d1 2ov2 ne + nu
ne + nu

---

Published as a conference paper at ICLR 2025
Since d1, d2, covll cov2, difficult to calculate, in practice, assume that these covariances are sufficiently small for computational convenience. Consequently, we have a = 1iu
and / = The During the training process, as we assume that the data from De and Du are drawn from netn,
ne+tu
the same distribution p(z), we guarantee this condition via the clip function (see more details in Appendix C.2
4.3 THEORETICAL ANALYSIS
We consider the reward given by the optimal discriminator (x)) In 2IWIL, when the discrimi- nator is optimal, the reward is - log D= (2r) log ((rp + 1/pe) (Tpa)). Consequently, if imperfect demonstrations exhibit a pronounced preference for certain state-action pair, it results in a sig- nificantly higher probability of compared to other state-action pairs. The discriminator tends to provide an inflated reward, hindering the learning of an optimal policy. Conversely, in our method, first give the following theorem:
Theorem 4.3. Given a fixed agent policy no, the optimal discriminator D (x) of Eq. 17 can be written as
(x) (1 r)p + Pe
(14)
p+ pa
As a result, when the optimal discriminator D (x) is given, the optimization of Tg is equivalent to minimizing
2JSD(pollp) - KL(ppllpe) - (1 KL (hsooallp))))))))))) P1) + C,
(15)
where P1
(po
(1 1)) 122 17). C nBBarpppp
log P
Nopt
(1
1)) ) Bix-poo llog (1- (1m))aam log(2 - 11) - -11 log(1 n1) n1) 2 log 2, which is a constant
p
for TT.
According Theorem 4.3 since P1 is weighted sum of pe and Pnon. subtracting the second and third terms of the Kullbcck-eebble (KL) divergence is equivalent to letting pe deviate from Puon- Thus, PN-GAIL aims align po with p and ensure that po deviates from Pnon- This illus- trates that our method is able to avoid mimicking non-optimal data within imperfect demonstrations, thereby solely imitating the optimal ones. Additionally, the reward given by the optimal discrim- inator (xxx) in our method is log (x) = log ((p + po) / [(1 - r)p + po)). In cases where imperfect demonstrations exhibit a pronounced preference for certain state-action pair, resulting in significantly higher probability compared to other state-action pairs, the presence of the term (1 r)p in the denominator mitigates the impact of an excessively high p. Furthermore, even in extreme scenarios where p is much greater than po. the maximum reward provided by the discrimi- nator in our method is -log -r), rather than approaching positive infinity as in 2IWIL. As a result, in PN-GAIL, the discriminator can offer more accurate rewards, thereby facilitating the subsequent RL process to learn a better policy.
In the following theorem, we demonstrate that the estimation error of Eq. (3 is bounded, indicating that we can obtain a classifier by minimizing RBsc. We provide the estimation error bound with Rademacher complexity (Bartlett & Mendelson 2002).
Theorem 4.4. Denote G the hypothesis class being utilized and Rn as the Rademacher com- plexity of the function class G with a sample size of n. Assume that the loss function is pr- pipiiphchitz continuous, and there exists a constant Ce > 0 such that for any G, sup
(((g(())) s Ce Define 9 as the minimizer of Rescce(g) over 9 E G and g" as the minimizer of Rbsc,e(g) over g E For 10, with probability at least 8 when repeatedly sampling data to train we huve
(0) RBsc,e(9") <16p1 ( a ByR, (9) + (a + B3) (9)))))))) (9))
log (12/8) (3 a B)m + +(a + 9)nūt). (16)
/

---

Published as a conference paper at ICLR 2025
4.4 OVERALL ALGORITHM
Through the aforementioned classifier, we can obtain the confidence scores for all demonstrations in the unlabeled data D... Subsequently. we treat both De and Du imperfect demonstrations and optimize the discriminator Dw- Finally, we utilize Trust Region Policy Optimization (TRPO) (Schulman et al. 2015) to learn a policy my hased on the rewards provided by the discriminator. The pseudocode for the overall algorithm can be found in Appendix
EXPERIMENTS
In this section, we validate our method by conducting experiments six control tasks, including Pendulum-vl and five challenging MuloCo (Todorov et al. 2012) environments. We aim to answer three questions: (1) Is 2IWIL influenced by the preferences inherent in imperfect demonstrations, and can our method alleviate such influence? (2) Does our proposed BSC outperform the SC pro- posed in 2IWII?? (3) How robust our method?
Task setup We conduct experiments across six environments (Pendulum-vl, Ant-v2, Walker2d- v2, Hopper-v2, Swimmer-v2. and HalfCheetah-v2) Each experiment is conducted using five differ- ent random seeds. Additionally, to better showcase the performance of imitation, we normalize the cumulative rewards of the policies, where 1.0 represents the optimal policy and 0.0 represents the random policy. Due to space constraints, we place the details of the experiments, the performance of the optimal and the random policies and the uncropped figures of Ant-v2 in Appendix C.1 C.4
Demonstrations For the Pendulum-v environment, we train an optimal policy Topt and an inter- mediate policy иi using TRPO. To highlight the preferences inherent imperfect demonstrations, we aim for higher proportion of samples to be drawn from Tı. In that way, we ensure that the number of demonstrations generated by Wi four times that of Topts resulting in a final demonstra- tions ratio of Topt πi = 1 4, which are then merged together. Afterward, all demonsirations are annotated with confidence scores, utilizing normalized rewards. For the Ant-v2, Walker2d-v2, Hopper-v2, Swimmer-v2, v2, and HalfCheetah-v2 environments, to maintain fairness, directly uti- lize the demonstrations and confidence scores provided by the code of 2IWIL. During the practical experiments across all six environments, 20% of the given demonstrations randomly selected to be assigned confidence scores, which means that the label ratio 0.2.
Baselines We choose GAIL, 2IWIL, IC-GAIL, and WGAIL as our baseline methods. Among these methods, since GAIL and WGAIL do not require confidence information, only provide them with demonstrations. Furthermore, we conduct ablation experiments, including 21WIL: Orig- inal 2IWII.. PN-GAILA BSC: BBSC PN-GAIL. with semi-conf (SC) classification. PN-GAIL PN: 2TWIL. with balanced semi-conf cocccoonBBBCCCC (BSC) classification. PN-GAIL: Our final method. All methods are trained jointly using both D. and D,. Meanwhile, we also test the performance CAIL, ranking-basse methods (T-REX, D-REX) and f-IRL (Ni et al. 202T) by constructing trajectory rankings from con- fidence scores in the Pendulum-vi and Ant-v2 environments (due to the demonstrations provided by the 2IWIL's code is not in trajectory form). The results can be seen in Appendix C.3
5.1 PERFORMANCE
In our experiments, we use different numbers of D. + Du for different tasks, and the specific values are shown in Appendix C.I Fig. 2 and Fig. show the normalized average returns during training. The results in Fig. 2 demonstrate that our method outperforms other baseline methods, achieving the highest returns in all six environments. Of particular note its performance in Pendulum-vi. Here, imperfect demonstrations exhibit preference for certain state-action pairs with lower con- fidence scores, adversely affecting the learning process of 2IWIL and leading to a poor policy. In contrast, our method addresses this issue by incorporating the negative risk of imperfect demonstra- tions. Experimental results demonstrate that our method is able to learn a near-optimal policy in the Pendulum-v/ environment while other baseline methods fail.
We observe that the performance of GAIL generally falls below that of other methods. This is because GAIL treats all demonstrations optimal, unable to allocate distinct weights to different
8

---

Published as a conference paper at ICLR 2025
OPTIMAL
PN GAIL (Ours)
2IWIL
IC GAIL
GAIL
WGAIL
Avernge
Ant-V2
230 Hal heetalh-V2
Hopper-v2
Pendulum-vi
Swinmer--Y
Walker2d-v2
Figure 2: Normalized average returns of PN-GAIL and baseline methods during training. The x-axis the number of training steps.
OPTIMAL
(Ours)
PN- GA-GBII
GAILIPN
Average Retum
26
26
Ant-v2
HalfChoottbb-22
ES Hopper-v2
100 280
Pendulum-vV
Swimmer--v2
2R 280
Walker2d-v2
Figure 3: Normalized average returns of ablation experiments during training. The x-axis is the number training steps.
demonstrations. However, in Walker2d-v2, neither 2IWIL nor IC-GAIL outperforms GAIL. We feel this might be due to the relatively low average confidence of demonstrations in Walker2d-v2. Mean- while, we notice that WGAIL performs worse than GAIL in Walker2d-v2, which we attribute to its assumption of higher proportion of optimal demonstrations within the imperfect demonstrations. Since the demonstrations provided Walker2d-v2 not align with this assumption, the confidence estimation of WGAIL would no longer be accurate.
Additionally, Fig. shows the normalized average returns of the ablation experiments. In Fig. 3 the large difference between the performance of PN-GAIL and EN-GAIL\PPN indicates that there is preference in the imperfect demonstrations, resulting in the poor performance of the 2IWII follow- up method. The large performance gap between the performance of PN-GAIL and PN-GAILY BSC indicates that the prediction confidence of SC classification not accurate enough, which affects the subsequent training. the performance gap is not significant, means that the above problems are not obvious or do not affect the final results. Our method outperforms other methods across all environments, thus confirming the performance enhancement brought by incorporating the negative risk of imperfect demonstrations and employing balanced semi-conf (BSC) classification
5.2 ACCURACY OF CLASSIFIER
( BSC as depicted in Fig. 3 it is clear that the performance of PN-GAIL can be improved by using the BSC classifier. This observation demonstrates the supe- rior capability of the BSC classifier over the SC classifier in accurately predicting confidence scores. To illustrate the disparity between these classifiers more clearly, calculate the Mean Abso- lute Error (MAE) and Root Mean Square Error (RMSE) the prediction confidence scores. Here, MAE represents the average of absolute errors, while RMSE denotes the square root of the average of squared differences between predicted and true values. As shown in Table the MAE and RMSE of the BSC classifier are notably lower than those of the SC classifier indicating that the predictions of the BSC classifier are closer to the ground truth. Consequently, BSC classifier provides more accurate confidence scores for subsequent imitation learning.
By comparing PN-GAIL with PN-GAIL
5.3 ROBUSTNESS OF PN-GAIL
To test the robustness of method, We evaluate the performance of PN-GAIL at different label ratios in Ant-v2 and Hopper-v2 environments, the results are shown in Fig. 4 (a) and (b). As the label ratio decreases, PN-GAIL exhibits only marginal decline in performance. This indicates that

---

Published as a conference paper at ICLR 2025
Table Accuracy of classifier measured by MAE and RMSE.
Classifier
Metries
Ant-v2
HaliChestah-v2
Hopper-v2
Pendulum-vl
Swimmer-v2
Walker2d-v2
MAE
0.213 # 0.023
0.184 + 0.011
0.307 a: 0.025
0.126 + 0.014
0.362 a 0.049
0.132 # 0.015
SC
RMSE
0.033
0.272 0.009
0.519 0.022
0.164 0.013
0.595 0.040
0.246 0.032
MAE
0.056
0.011
0.057 0.012
0.169
0.126
0.097 + 0.006
0.286 0.179
0.014 0.002
BSC
RMSE
0.212 0.026
0.173 0.013
0.371 0.138
0.138 0.005
0.472 0.188
0.101 0.010
Ant-v2
Hopper-v2
Pendulum-vi
Average Retum
Average Retum
Averago Return
0.5
0.6
Label mato
Like cado
3.1 115
0.2
02
aB
0.01
0m
0.4
1950
2006
3000 Steps
4600
3000
1000
2000
3000
1000
3006
5000 Steps
5060
4005
Steps
(a)
(b)
(c)
Figure 4: (a) Ant-v2 experiments with different label ratios. (b) Hopper-v2 v2 experiments with differ- ent label ratios. (c) Pendulum-vl experiments with different standard deviations of Gaussian noise.
PN-GAIL is not highly dependent on the label ratio, maintaining excellent performance even as the label ratio decreases.
In practice, considering that confidence scores are typically provided by human annotators, varia- tions in their standards for labeling confidence may arise due individual differences and factors such fatigue. To assess the robustness of our method against noise in confidence scores, we con- duct additional experiments. In Pendulum-vl, we introduce Gaussian noise to the confidence scores: f(x) = clipjo.1) (r(1)= + <). where (00 o²), clipp,,. = min min(max'd max I, u). As shown in Fig. (c), the numbers indicate the standard deviations of Gaussian noise. Even when confidence scores are subject to noise, our method still demonstrates satisfactory performance, indicating its robustness to noisy confidence scores.
We also test the performance PN-GAIL two scenarios: first, by reducing the number of unla- beled demonstrations; and second, by observing how PN-GAIL performs when the average optimal- ity of imperfect demonstrations changes. Due to space constraints, we present the details these experiments and the corresponding figures in Appendix C.3
CONCLUSION
In this work, we proposed a novel algorithm termed PN-GAIL for imitation learning from imperfect demonstrations. PN-GAIL. leverages non-optimal information embedded in these demonstrations, enabling the discriminator to weigh both positive and negative risks in holistic manner. This approach facilitates the assignment of more refined reward signals. To enhance the precision of confidence estimation, we have integrated an advanced semi-supervised confidence classifier into our framework. Our theoretical investigations demonstrate that PN-GAIL is not merely capable of mimicking imperfect demonstrations but also adept at circumventing the imitation of suboptimal be- haviors, thereby ensuring the acquisition of an optimal policy. Comprehensive experimental results indicate that approach surpasses existing baselines in performance and exhibits remarkable bustness, thereby establishing a robust foundation for the practical deployment of imitation learning in real-world scenarios.
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China (Nos. 72394363 & 62073160), the Nanjing University Integrated Research Platform of the Ministry of Education-Top Talents Program and the Australian Research Council (FT220100656)
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, pp- 1, 2004.
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463448,, 2002.
Raunak Bhattacharyya, Blake Wulfe, Derek Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, and Mykel Kochenderfer. Modeling human driving behavior through generative adversarial imitation learning. IEEE Transactions Intelligent Transportation Systems, 24(3): 2874-28871 2022.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, pp. 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. Satter-than-demmostraac imitation learning via automatically--ranked demonstrations. In Conference Robot Learning, pp. 330-359, PMLR, 2020.
Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on Robot Learning, pp. 1662-1277. PMLR, 2021.
Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imita- tion learning. International Conference on Learning Representations, 2020.
Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. Advances Neural Information Processing Systems, 27, 2014.
Huiqiao Fu, Kaiqiang Tang, Yuanyang Lu, Yiming Qi. Guizhou Deng, Flood Sung, and Chunlin Chen. Ess-infogail: Semi-supervised imitation learning from imbalanced demonstrations. Ad- vances in Neural Information Processing Systems, 36, 2024.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference Learning Representations, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- fion Processing Systems, 27, 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemyslaw Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, al. Urban driving with conditional imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 251-257. IEEE, 2020.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
Liangyu Huo, Zulin Wang, and Mai Xu. Learning noise-induced reward functions for surpassing demonstrations imitation learning. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 37, pp. 7953-7961 2023.
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning f-divergence minimization. In Algorithmic Foundations of Robotics XIV: Proceedings of the Fourteenth Workshop on the Algorithmic Foundations of Robotics 14, pp- 313-329. Springer, 2021.
11

---

Published as a conference paper at ICLR 2025
Geon-Hyeong Kim, Seokin Sco, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demon- strations. In International Conference on Learning Representations, 2021.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30, 2017.
Ziniu Li, Tian Xu, Zeyu Qin, Yang Yu, and Zhi-Quar Luo. Imitation learning from imperfection: Theoretical justifications and algorithms. Advances in Neural Information Processing Systems, 36, 2024.
Tianwei Ni, Harshit Sikchi, Yufei Wang. Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference Robot Learning, pp. 529- 551. PMLR, 2021.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4):1-20, 2021.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in Neural Information Processing Systems, 1, 1988.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and stafistics, pp. 661-666 JMLR Workshop and Conference Proceedings, 2010.
Kangrui Ruan and Xuan Di. Learning human driving behaviors with sequential causal imitation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4583- 4592, 2022.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- fional Conference on Learning Representations, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint IrXivv1707.06330 2017.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. Advances in Neural Information Processing Systems, 31, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation learning with diverse-quality demonstrations. In International Conference on Machine Learning, pp. 9407-9417. PMLR, 2020.
Aleksandar Taranovic, Andras Gabor Kupesik, Niklas Freymuth, and Gerhard Neumann. Adversar- ial imitation learning with preferences. In International Conference on Learning Representations, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Yunke Wang, Chang Xu, and Bo Du. Robust adversarial imitation learning via adaptively-selectte demonstrations. In IJCAI, pp. 3155-3161, 2021a.
Yunke Wang. Chang Xu, Bo Du and Honglak Lee. Learning to weight imperfect demonstrations. In International Conference on Machine Learning, pp. 10961-10970. PMLR, 202
12

---

Published as a conference paper at ICLR 2025
Yunke Wang, Bo Du, and Chang Xu. Unlabeled imperfect demonstrations in adversarial imitation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10262- 10270, 2023.
Hua Wei, Dongkuan Xu, Junjie Liang, and Zhenhui Jessie Li. How do move: Modeling human movement with system dynamics. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pp. 4445 4452, 2021.
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imi- tation learning from imperfect demonstration. In International Conference Machine Learning, pp. 6818-6827. PMLR, 2019.
Fei Xia, Chengshu Li, Roberto Martín-Martin, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation. arXiv preprini arXiv:2008.07790 07792, 2020.
Danfei Xu and Misha Denil. Positive-unlabeled reward learning. In Conference on Robot Learning, pp. 205-212221 219. PMLR, 2021.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. seetriiinnato-wweighee offline imitation learning from suboptimal demonstrations. In International Conference Machine Learning, pp- 14725-24742. PMLR, 2022.
Hanlin Yang, Chao Yu, Siji Chen, et al. Hybrid policy optimization from imperfect demonstrations. Advances in Neural Information Processing Systems, 36, 2024.
Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offline imitation learning with suboptimal demonstrations via relaxed distribution matching. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 1016-11024, 2023.
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In International Conference Machine Learning, pp. 12333-1222 12344. PMLR, 2021.
Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learn- ing from demonstrations with varying optimality. Advances in Neural Information Processing Systems, 34:12340-122 2021.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
13

---

Published as a conference paper at ICLR 2025
Appendices
A ALGORITHM
Algorithm 1 PN-GAIL
1: Input: Imperfect demonstrations and confidence D. {< ri)
r.) Du {2 Rm
Ji=1
2: Train a probabilistic classifier by minimizing Eq. (13) with or
Ry
3 +
Bry
3: Predict confidence scores (f for ( J
4: for i 0, 2.. do
5: Sample trajectories
Te.
(e: PuF
6: Update Dw by maximizing Eq.
7: Update wo by TRPO with reward log Dw (x)
8: end for
B DERIVATIONS AND PROOFS
B. PROOF OF THEOREM 4.1
Theorem. Based on Eq. (.. the objective of the discriminator can be rewritten as
max Exwpo log D (x) + Exop log(( Dwo())) + Ezap [(1 r(x)) log D (x)]
16
Proof. Since (x) = p( y 0) r(x)p(x) 7
Pnon (x) = p(rlyy 1) (1-r(=))p(e) 1-n
we have
1) log(1 - (=) = 17 / Pope(z) log(1 (w(x))drr
r(z) p(x) log(1 Dw(xx)dd
7
p(z)r(x) log(1 (x)
Exep
- (w))))
- 11) Err (log Dw (1 1)) Puon (x)) log Dan (x))dx
(1 - 17) /
r(x) 1-r(x) p(x) log (w)
1 1-11
/ p(x)(1 -
p(z)(1-r(zt) r(x)) log Due (x))dr
Expp [(1 - r(x)) log (w)),
According to Eq. (8) the objective of the discriminator can be rewritten as
max Exrpo log D., (x)] + Exp log (=)) + Ezep [(1 r(x)) log D (z))
B.2 DERIVATION BALANCED SEMI-CONF (BSC) CLASSIFICATION Recall that in 2IWIL:
Rg),e(g) Ex,ra
+ + - (1-r)((-<(()))
= Exyrong re(g(x)) + (1- + 38 33((-(())) - (8(-9(x))
=0
14

---

Published as a conference paper at ICLR 2025
By introducing Exxplal(g(z)) Ez=glaa(g(2))) with theoretical values of 0, we have
(99) = Berrng r((g(z)) +1 (1--))((-9(a))) al(g(z)) al(g(=))) +
(a(lar))))((((((() -((-=(=))) - 30( 9(x))
Beryn (r((g(z))+ + (1--))(--g()) -a(((())))))))))))))))))))))()>) =((- + Exp (amplal((())) + 3e(-9(2)))
B.3 PROOF OF THEOREM
Theorem. Let d1 denote (en((((g(x))). d2 denote Var Var(E(g(z))), Jeuvl denote the covariance the between The Ein (E (Te,a)) - (-9 (Ke, and Toe -i=1 (-9 (Ie,,)), Jon22 denote the co- variance between 1 Ei=1 (1 - r:) (e(-g (se,)) i (g (Fe,))) and ne E e (g (fea)), cou de- no note Cov (e (-g(()) e (g (x))). The estimator &gsc,e(g) has the minimum variance when
ny
dicov con2 nu
+ d1 coud
a ne + fy
dıdg con2 The + nuu
d1d2 cor2
Ne + Tlu ne
nu
d2cov con? Nu
+ d20 coVo
B
ne + nu
d1dg cor2 ne + nu
d1d2 - cor2
ne + Ru
Proof. Denote
# a ED..D.
RBsc.e(s)::
111 ED,
(--g (Fea))
= ED,
1
e e (Bus))
= Ex-ple(-glee (((())))))
nu
i=1
1/2 A ED
e (g (fte,))
= -ED..
(g (Ju.)) = =B
d1 VarD. (e(-g(z))) VarD, 1e (c(-g(( (xr))) = Var (e (E(9
d2 VarD. (x.))) Varp. [6 (g (zu))] (a (x))
Ru
(-9 (zc,)) (g (Zc..) ED. cou & CoVv, ((-))((x)) l (g (xx))) = CoVo,
e ED.
Th Σ ((gg( =[
Te
Cov ( = w - 41
(g (x
Ocorl & Cov
rs (e (x).)) {( (-g (Ec,)))
{(-g (sea))
Re iml
To
A Cov
Σ (1 r;) (e( (-0 (zc.:)) - (g (fea)))
Σ e (g (Fea))
Ocov2
Tee
1=1
Next, we adopt the symbols defined above to express several formulas that will be used:
27
1 The E(
1
The
Te
ED.
ED (--g(ra))) +2
Σ
Σ 4 (((((re)))((9((aa
Nee
no
i=1
noBN [((-9(2)))] + nnlone 1) 1C[ (((-)(x))1})
n
Var(e(-g((2) +f
Ne
d1 + n
(17)
Te
15

---

Published as a conference paper at ICLR 2025
Similarly, obtain
27
Ru
ED.
Σ
{ ((~g(zu,))
d1 pi.
(18)
nu
na
i=1
1 nce
27
1
ED
e(g(Te,)))
d2 + 12
(19)
ne
T
1
DD
- d2 + 12
(20)
Th
Thu
1=1
1
1
D
8( p ((
((g(Fa.))
cov + P1/2.
(21)
nu
Ru
nu
i=1
i=1
DD
e( (--g(ae;))
((o((e()))
coe + 11122
(22)
ne
i=1
We also have
(
Epc,Da
1 -(-
1 Σ
el e(-g(zu.i)
Σ
el - g(
Te
nh
i=1
i=1
1
Ep.
- -
Ep.
{( ((-g(Tu.i))
Tee
Nu
i=1
Ep.
-
-
{( - (Zc,,))
Tee i=1
1=1
(
1
Ep.
- ((-o((())))
ED.
Σ
ne
nee
i=1
E.D
(x:) - ((-9((E())))
Σ
E( - (FF())
nee i=1
1 Σ ((-g
Cov
(g - ((( (Xcr)))
nee
nee 1=1
-coool
(23)
Similarly, we obtain
ED
[
1 me 1=1
Σ
((:)))(((-((((e))) -((((()))))
(
(1 Tim
((g(zn,)))
m ((((()e)))
ne ==1
-ooov2
(24)
Hence, express the estimator variance Var (Bssc,(9)) as follows:
16

---

Published as a conference paper at ICLR 2025
Var(Pesc..
(Rnscc.(o)) 11
=,p.D,
= =D
(el((()))))) + (1)r)))((-(((e))
+B(
1
Σ
( (F
Σ
E( ((-g(Fe;i))
Du
1=1
1=1
==1
27
1
1 [g((ac)))
+
(g(aa.)))
-12
7h
The
Ep, [19] + BB D., [Ba] + a
De, 1. + 2,33ED, D. [AB] + 22BBD D. [AC| +
n1²
ED.,D,IA1.
+ DD [BC]
E EB [BC]
ED..D, [B²] (+
+ RD..D, 102
a²
ED., D. [B2]
ED. D. [B2]
ED, D,, [B
+ ED.D,IAC ED..D. D. LABBBBB [BC]
E2
D. [AB
a + D...D.IAA [A?]
112
ED..D. [B2
E ED. B
ED, D0, [AB]
+ ED..D, (BC)
ED.,,, [B
B+
ED,,D, [Ba] a
ED.,D.I [Ba]
10 EBD,,, [BC]]
a+ ED..D, [AC|ED,, [Ba] ED.,,, [AB]E E. D.D,D [BC]
2
ED.,D,IC**
ED..D. D., [
ED.,D. [B2 D., [C
B2_. D. [BC]
E2
E3 [BC]
ED..P.
[AC] D.P.P.L.CIBBB
D. D. [B3] ED.,D. (AB]
LABBEE DD [BC]
2
const
ED. D., [²a]
Ep, B E ED, D C
E2
D., BC
ED.,D. [AB]
ED.,D, [BC]
E.. [B²]
(+
ED..D. [B2]
+
ED.. [2a B2
a
E2 D.. D. [BC]]
ED. D. [AB]Ep [ABBEE
D. [BC] - ED. D. (AC], Bp, D. [B
2
const.
ED. D., [B?]
Ep. B ED, [C² B2, [C]]
(25)
By Eq. 21 and Eq.
22 we can obtain
2 ll (1-g(zn.))
1 Σ {l - gl (Feii)
(
1
Ep.,p,, [BC] = ED. Di
E(g(En.)))
{gg(Te.)))
nu 1=1
The 1=1
nu im1
cov + Mill2 - P1422
M1l22 + cov + 11l22
nu
ne
1
1
cov.
nn
nc
17

---

Published as a conference paper at ICLR 2025
By Eq. 17 and Eq. 18
obtain
27
E D. [B²] = DD. Do
1
El
(--((zu,))
Σ no 1=1
" - g((c,1)))
nu
1=1
(
27
1
ED,
Σ
e( ((-g((nn)))
+ EDd
( ( g((ras))))
Pu
im1
iml
nu
1
nhe
2..
E( 0 g(Zui))
ED.
Thu i=1
Re
1
di
pi
d1 + 1r 2p
The
dl.
The
Similarly,
we
obtain
D. SD..D.119
(
1 + d2.
1
Hence, we have
An
the
E [BC] (, ED. [8
2 since díd2 2 cov?. By Eq. 117) and Eq. 23 we can
obtain:
(
t (ass((((((ees))) ((-g((Fc,))
ED,,D, [AB] DD
Σ
ll ((g((rc)))
ne
ne
1 Tu nu
1 Te
(l ((-g(sn.))
Σ
El ((1g(te,)))
i=1
-covl + 1h - - d1 +
ne
d1 Ocovl.
Ne
Similarly, D. (AC)
D. |C9 1 da Tcoo2. Since ED. [B]] 0. DE
EB. [B0] Ev, [B3]
ne
0, and 8 E R. according to Eq.
25
Var RBSC. (g)) is minimized when
8 ED..D. (AB]
Ep..D. [BC)
ED_ D.. [Ba]
D.D.D... [ a2
ED_D. [AB] D...E.D.. D. [BC] ED..D, [AC E DD P. [Ba]
ED.,D, D. [2a ED. C2]
Ey Eb,,, [C]
Substitute a, obtain
B ED..D. B.D.,.A(A[EE [AC[ED. BC1 Ep.,D. [AB] D. [C2]
ED. D. [Ba] ED. [Ca] E2 E..DD [BC]
Through plugging in the above formula, have
nu
dicov 2 nu
d1o cOVo ne
ne + nu
d1dg cov2 nhe + nu
d1d2 cov2 The + Tuu
Nu d2c0v - 2on2 nu
+ d2o covOenv2
DeD0n
3
+ Ra
d1dg 2ov + nu
d1d2 cov2
ne +
18

---

Published as a conference paper at ICLR 2025
B.4 PROOF OF THEOREM 4.3
Theorem.
Given a fixed agent policy no, the optimal discriminator D (I) of Eq. 17
can be writlen
as
(x) (1 1r-)pp+p
p po
As result, when the optimal discriminator (x)) given, the optimization of πTy equivalent to minimizing
2pSD(ppol1y (p) - (KLppollpp) [P1)) (1 1)) (Puoul (oaiual/Pi/>> + C,
where
P1
(po
(1 1))poon) 1/22 17), C 1BBerppps
log
PPPtpt
(1
1) Be log (1- (1-n)pase for TT.
+ +log(2 7) (1 (1-1)) log(1 71/ -1) - -2logg 2. which is constant
P
Proof. Denote
(( Dw) = Exppo (log Dw (z)) + Eump (r(x) log(1 - Du (x))] + Egup [(1 r(x))(()))))))))))) (x)) (w(x)].
We denote D" (x) = arg min ( Dw=) and have
D
(To, Dw,) Po
rp (1 r)p
0D The maximum value of (TT. Dw) occurs when its partial derivative with respect to zero, given by D, (1-r)p+pe Consequently, obtain D (2)
D
1-D
D
(1-r)ptpe
p+Pe
p+Po
When the optimal discriminator D is given, V(To, Dw) can be rewritten as
V(Ta, Dw) = Ex-pe
log (1 r)p + Pe
+ Epep
log rp
+ Ex-p
(1-r) log
(1 - r)p + pe
+ Pe
pe //
+ Pe
According to Eq. 9
and denoting P1 (pe + (1 P)on)
17), we obtain
log (1 Pnon + Po
Moptt
Exxpaon (1 (1- n) log
(1 11) + pa
V Dw) = Exppo
+ Euspoopt
llg
p + po
p po
p po
Po
Po
Mopt
= Exppo
log
+ log
+ nBaarppp
log
(1 1))nn + p0
p pa
p po
(1 - 1])
(1 1))
+ (1 Trr
log
log
(1
Puon + po
p po
Po
MPppt
(1 17) Pnon
Exxpo
log
+ Bnpoppppe
log
+ (1 17 1)
log
p + po
p po
p pa
logg22 11) +(1 - 11) Eo log Pe log P1
Pnon
1 17
Exanp
(1 1) log
P1
2
const.w.rt.ar
constW.LL..O
po
log Poptt
+ llog p
log (1 17)
Ex-op
log
+ +n. -1EEppppr
+ +(1 Bxm
p + pe
p
+ log
p
(Toloolpz)
- (1-1)) KT (proallaa l1p1) ) + Ci
+
po
P
MPptt
17) Pnon
Egaoo
log
+ Eznp
log
+ E
log
+ (1 Exmppann
log
p + pe
p Po
p
KL (e(pollpi)) lpp)
n]K (PnoallPi) ( +
Ex
log
(p+ (p)pa Ezzp log D 2llgg KL(pel -1oo)))L(ppnoullll Cz
Da
(p+po) 22
2JSD(ppoll
KT. (p1) -11 KT) (pt) + C,
19

---

Published as a conference paper at ICLR 2025
where C = T/EpgPapp
log Whops
+ (1 77 Exmpuon
log (1 Poons
+ +log(2 - -1) - (1 --n log(1
(2/(2 - 11) 2 log is constant for ne.
B.5 PROOF OF THEOREM 4.4
Theorem. Denote G as the hypothesis class being utilized and Rn as the Rademacher complex- ity of the function class 9 with sample size of n. Assume that the loss function & is pr-Lipschitz continuous, and there exists a constant Ce > 0 such that for any
G. w(( [e(yg(x))) s C.. Define 9 as the minimizer of (g) over 9 E G and g" as the minimizer of (g) over g E For 1), with probability least ó when repeatedly sampling data train we have
PBs RBsc,e(o)) S16pL -393nn (9) + (a (3,.,,
(og((2/0)
(3 + a 3)me = + (ax B)n
+ +4CrV
2
Proof. Like 2IWIL, since I and g" are the minimizers of Rasc,e()) and RBsc,e()), respectively, we have
Rpsc,e(0) - RBsc,e(g") = (p (g)) + RBsc,e(0) RBsc,e(g") + Resc,elg") RBsc.e(g")
(Bsc,e(0) Roscce(O)) + 0 + sup (RBsc,e(9) Rasc,e(9))
S sup gEg
sEg
sup 96g
Rasc,elg) RBSCCe(0)
Hence, we just need bound the uniform deviation supyeg RBsc,e(o) R We have
RBs) el9)
sup geg
1 Th
S sup nEG
-((g(re)))) ((- (1-))(((-11()))))))))) al(g(Te))))
ne 1=1
Egrro (r(((((z)) El (((((x))))) + (1- 3)((-9(z)) al(g(x))]
1 nuu
sup geG
ll ((-g(zu,)) Egep [e( (-v(())]]
a susup
(g(( [(((())
nu
DEG
Yu
1 ne
nhe
< sup 909
r(((g(Ze.))
E En.rnglrt(ggn
+ sup geg
(e((-g(Fe.)) E Ex (alpnnr((((gg1))
7e i=1
ne
Tee Σ ((- ((-g(Ie,)) E Begnnng ((((()))]
ns
(1
B) sup gEg
sup geg
Σ ( ((-g(aux)) Bemp(e(-99a))
ne
nu
nle
Thu
sup gEg
((g(x..)) E ( + a sup
((g(x.,))) E En-p((((()))
The
9Eg
nhu
According to 2IWIL (Theorem 4.3 in Wu et al. (0119)). ), since the above six terms are the bounded differences with constants Ct/nn, Cr//nes Ct, ne. Cr/nus Cr/ne, and Cz/Das respectively,
20

---

Published as a conference paper at ICLR 2025
can bound them with probability at least 1 8/6
Th
log (12/0)
log
sup gEg
re((((e))) Ex.r~g(r(((()))
$ Rn, +C)
Re
2nc
log
(12/6)
sup nEg
(((()(((e))) B Be.
R (9) (G) +
Te
2n
i=1
log (12/0)
sup gEg
(( ((-g(Fe))) Berrog
$ 8m, (9) +0
2nc
1=1
Ra
log (12/6)
sup geg
( ())
$ Rm, (9) (9)+2CL V
Tu
2na
log(
(12/0)
sup gEg
((o(xe,)) E (
< 8p (9)+ + 2C1
The
2nc
==1
Ru (g((z..)) - -Expp((((()))
(og(12/5)
sup gEg
< R (9) + 20
nu
2nh
In the end, we can bound the initial estimation error with probability of at least 6:
e RBsc,elg") $16p1 ((3 + a 3). (g) + (a (B)
a- + +ACL log (2g()2/5) (3 a 3)me + (ax + B)
2
C DETAILS OF THE EXPERIMENTS AND ADDITIONAL EXPERIMENTS
C.I PAPEARPARRMETEES SETTINGS AND TASK INFORMATION
All of our experiments the architectures of all neural networks, we employ two hidden layers of size 100 each, using Tanh the activation function. Across all tasks, utilize the same hyper-panee seper-parameters listed in Table Table 3 shows the number of confidence data and unlabeled data used for each task, along with the cumulative rewards corresponding the optimal and random policies.
single machine with 4 NVIDIA GeForce RTX 3080 GPUs. For
Table 2: Hyper-paramette settings
Hyper-parameter:
value
?
0.995
(Generalized Advantage Estimation)
0.97
Batch size
5, 000
Learning rate (Value network)
3 x 10-4
Learning rate (Discriminator)
1 x 10-3
Learning rate (Classifier)
3 10-4
Optimizer
Adam
C.2 CLIP FUNCTION IN BSC
In the process of deriving Eq. 13 two terms with theoretical values of are introduced: If we directly min- imize Eq. (L)). the two terms may deviate far from 0. To mitigate this issue, we use the clip function to limit the sum of these items a neighborhood of 0. Specifically, denote
Ex>plal(g(z)) - [z=g(ol(g(z)) and (3
21

---

Published as a conference paper at ICLR 2025
Table 3: Task information
Tasks
Thu
Optimal policy
Random policy
Ant-v2
120
480
4143.10
-72.30
HalfCheetah-v2
500
2000
3467.32
-288.44
Hopper-v2
20
3250.67
18.04
Pendulum-vl
200
800
-116.81
-1200.96
Swimmer-v2
5
348.99
2.31
Walker-v2
400
1600
3694.13
1.91
R1 E Eg-plate Dx==paa(g(<))] - Exsgladig(a [al[g(>))] + Easp138(-gg(2) -Ex>g(((--g())) According
to Eq. 13
the final empirical risk can be written as
RBsc,e(g) = Ro(g) + clep R2,
where
Ro(g) Σ (re((((())))) + + (1-r)))(-ggaaa)
In Walker2d-v2, the epoch for classifier training is set to 50000, with e configured to 0.05, other experiments, the epoch for classifier training is 25000, with configured to 0.01.
C.3 ADDITIONAL EXPERIMENTS
Pendulum-vi
1.0
Average Return
0.8
OPTIMAL
0.6
(Ours)
0.4
2AWIL
IC-GAIL
0.2
0.0
T-REX
0.2
D-REX
-04
1000
2000
3000 4000 Steps
5000
Figure 5: Experiments with baselines including weighting and ranking-based methods.
CAIL uses the average reward after training convergence, and the implementation is based CAIL codebase. As shown Fig. 5) PN-GAIL outperforms other baseline methods, achieving the highest return.
22

---

Published as a conference paper at ICLR 2025
Ant-v2
1.0
Average Return
0.8
0.6-
0.4
0.2
Unlabeled
0.0
1.0
0.2
0.6
0.4
0.4
0.2
-0.6
1000
2000
3000
4000
5000
Steps
Figure 6: Experiments on reducing the number of unlabeled confidence demonstrations.
We gradually reduce the proportion of unlabeled confidence demonstrations. In Fig. 6 the numbers in the legend indicate the proportion of unlabeled data used as demonstrations. The performance of PN-GAIL improves as the amount of unlabeled data increases, illustrating how the use of unlabeled data can enhance the performance of our method.
Pendulum-vl
Average Return
1.0
0.8
0.6
0.44
Optimality
0.2
40%
0.0
1000
2000
3000 Steps
4000
5000
Figure 7: Experiments under different optimalities of imperfect demonstrations.
We evaluate the performance of PN-GAIL under different optimalities of imperfect demonstrations in the Pendulum-vl environment. In Fig. 7 the numbers in the legend illustrate the proportion of demonstrations generated by the optimal policy when collecting datasets (Topt m : 9). The higher the proportion of demonstrations generated by the optimal policy is, the better the performance of PN-GAIL.
Topt = 1 : 4, Topt
Ant-v2
Average Return
1.0
0.8
OPTIMAL
0.6
PN-GAIL (Ours)
00.4
2IWIL
IC-GAIL
0.2
GAIL
0.0
WGAIL
1000
2000
3000 4000 5000 6000
Steps
Figure 8: Experiments with extreme demonstration ratio.
23

---

Published as a conference paper at ICLR 2025
In the Ant-v2 environment, train optimal policy Topt and an intermediate policy using TRPO. We collect demonstrations with ratio of Topt #i = 1 10 and all demonstrations use normalized rewards to annotate confidence scores. We then evaluate the performance of PN-GAIL and other baseline methods using the collected demonstrations. As shown in Fig. & under this extreme demonstration ratio, all other methods fail, the final result is close GAII,, and only PN- GAIL learns valid information from the extreme demonstrations.
Ant-v2
Pendulum-vl
Average Return
1.0
Average Return
0.8
0.8
OPTIMAL
0.6
0.6
PN-GAIL (Ours)
0.4
9..4
21WIL
IC-GAIL
0.2
0.2
GAIL
0.0
0.0
WGAIL
0
1000
2000
3000 4000 5000 Steps
6000
1000
2000
3000
4000
Steps
Figure 9: Experiments when optimal demonstrations are dominant.
conduct the experiments the demonstration ratio of Topt 2ı = 2: 1. As shown in Fig. 9 it can be seen that when the optimal demonstrations dominant, PN-GAIL still shows robust and excellent performance.
In the Ant-v2 and Pendulum-vl environments,
Ant-v2
1.0
Average Return
0.8-
0.6-
Non Optimal 1.0
0.4
0.8
0.2
0.6
0.0
0.4
1000
2000
3000
4000
5000 6000
Steps
Figure 10: Experiments under different number of non-optimal demonstrations.
We conduct experiments with different numbers of non-optimal demonstrations in the Ant-v2 en- vironment. Fig. I0 the numbers in the legend indicate the proportion of non-optimal demon- strations used demonstrations, and the blue dotted line represents the performance of the optimal policy. When the number of non-optimal demonstrations decreases, the performance of PN-GAIL does not decrease significantly. We speculate that PN-GAIL does not take advantage of additional non-optimal demonstrations now because the number of optimal demonstrations in given expert demonstrations is sufficient to learn optimal policy.
Since the ranking-based methods including CAIL. and f-IRI. all require demonstrations to be stored in the form of trajectory, we only evaluate them in the Pendulum-vi and Ant-v2 environments. The dataset used in the Pendulum-vl environment is the same as in Fig. 2 and the dataset used in the Ant-v2 environment the same in Fig. g The implementation of CAIL, T-REX and D-REX is based on the CAIL codebase, and the implementation of f-IRL is based on the f-IRL codebase. All methods use the average reward after training convergence. We construct trajectory rankings based on confidence scores, and the average returns (only for evaluation) across all methods are shown in Table 4 PN-GAIL outperforms other methods, achieving the highest returns.
24

---

Published as a conference paper at ICLR 2025
Table 4: Average returns of PN-GAIL, CAIL, ranking-based methods and f-IRL during training (only for evaluation).
Methods
Pendulum-vl
Ant-v2
PN-GAIL (Ours)
465.978 +: 132.710
2960,458 +: 851.465
CAIL
-697.821 91.459
2336.546 + 735.378
T-REX
-1074.435 + 258.078
D-REX
-1532.958 + 114.138
-2495.473 220.7733
FFL((-IRL)
-698.064 = 300.793
1563.796 = =3372.482
RKL((-IRL)
-603.760 = 189.727
962.785 = 816.547
JS(f-IRL.
-581.602 + 185.096
882.026 + 690.371
Optimal Policy
-116.81
4271.79
Table 5: The variance of the estimator RBsc,elg)
Var
Ant-v2
HalfChoctah-v2
Hopper-v2
Pendulum-vl
Swimmer-v2
Walker2d-v2
Origin
0.12640.054
0.03340.009
1.47640.574 1.412=0.566
0.00064#0.0000
1.664&1 .475
0.093+0.029
PN-GAIL(Ours)
0.100=0.050
0.025=0.006
0.00013+0.00004
0.007=0.015
0.084=0.0288
Origin indicates the lack and (i.c., a=0 and B=0). The results in Table show that the variance of PN-GAIL is consistently smaller, demonstrating the validity of the chosen values for and B.
C.4 UNCROPPED FIGURES OF ANT-V2
In our experiments, we observe decrease followed by an increase in performance within the Ant-v2 environment. For better comparison, have cropped the figures of Ant-v2 the main text, while the uncropped figures are presented below:
Ant-v2
1.0
Average Return
0.8
0.6
OPTIMAL
0.4
PN-GAIL (Ours)
0.2
2IWIL
0.0
IC-GAIL
0.2
GAIL
-0.4
WGAIL
-0.6-
1000
2000
3000 Steps
4000 5000 6000
Figure 11: Normalized average returns of PN-GAIL and baseline methods during training.
25

---

Published as a conference paper at ICLR 2025
Ant-v2
1.0
Average Return
0.8-
0.6
0.4
OPTIMAL
0.2
PN-GAIL (Ours)
0.0
PN-GAIL BSC PN-GAILPN 2IWIL
0-
-0.4
0.6
1000
2000 3000 4000 5000 6000 Steps
Figure 12: Normalized average returns of ablation experiments during training.
26


Reviews:
[{"review_id": "gSJ076hs9H", "reviewer": "Reviewer_H316", "summary": "This work addresses imitation learning from imperfect demonstrations, utilizing both confidence-labeled noisy demonstrations and unlabeled noisy demonstrations. It aims to overcome two primary limitations of prior work, specifically 2IWIL [A], a representative approach to this problem that employs a two-step learning process: (i) semi-confidence labeler training on the unlabeled dataset, and (ii) confidence-based generative imitation learning.\n\nThe proposed method, PN-GAIL (Positive-Negative GAIL), tackles the limitations of 2IWIL as follows:\n\n1. **Incorporating Negative Risk to Objective**: 2IWIL overlooks the negative risk associated with imperfect demonstrations, leading the discriminator to disproportionately prioritize the positive risk of frequent samples. PN-GAIL addresses this by incorporating both positive and negative risks into the confidence-based imitation learning objective, ensuring a more reliable evaluation regarding to demonstration quality.\n2. **Balanced Semi-Confidence (BSC) Classification**: In 2IWIL, semi-confidence (SC) classification is used to train a confidence labeler for unlabeled demonstrations. However, SC classification tends to overestimate the confidence of labeled data and underestimate the confidence of unlabeled data. To address this, PN-GAIL introduces a balanced semi-confidence (BSC) objective and further suggests near-optimal values for hyperparameters $\\alpha$ and $\\beta$, enhancing practical applicability.\n\n[A] Wu et al., \"Imitation Learning from Imperfect Demonstration,\" ICML 2019.", "questions": "Q1. In contrast to scenarios where $\\pi_1$ dominates, how would the results in Figure 2 be affected if $\\pi_{opt}$ were dominant, a condition under which existing methods are known to perform well? Results for PN-GAIL in the Pendulum-v1 task with various $\\pi_{opt}:\\pi_1$ ratios are presented in Figure 7 of the supplementary material, but a more systematic comparison between PN-GAIL and baseline methods could strengthen the manuscript. If PN-GAIL demonstrates robust performance and outperforms the baseline methods in such scenarios, it would confirm the method's reliability. This evidence would support the assertion that PN-GAIL performs consistently well across a range of dataset quality distributions, thus setting it apart from existing methods.\n\nQ2. Instead of modifying the $\\pi_{opt}:\\pi_1$ ratio while maintaining a fixed total number of demonstrations, what would occur if the optimal demonstrations was kept invariant while the number of suboptimal demonstrations was varied? This setup would illustrate how PN-GAIL effectively utilizes additional suboptimal demonstrations, isolating the influence of optimal demonstrations on imitation performance. It would offer valuable insights into PN-GAIL’s ability to adapt to and leverage diverse demonstration qualities effectively.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This work is well-motivated and effectively addresses the challenges presented in the previous study using sound methods.\n\nA notable strength of this work is its practice-oriented design of the objective functions, which enhances applicability in real-world scenarios.\nThis study removes the dependence on $\\eta$, the class prior $p(y=0)$ for imperfect demonstration datasets, from the primary objective. \nSince $\\eta$ is generally unknown and challenging to estimate, prior work treated it as a hyperparameter, requiring practitioners to invest substantial effort in tuning it. \nBy eliminating this reliance, the proposed approach reduces the overhead associated with hyperparameter optimization.\n\nAdditionally, the authors introduce near-optimal and practical choices for the parameters $\\alpha$ and $\\beta$ in the Balanced Semi-Confidence (BSC) objective, which can be straightforwardly calculated based on the dataset sizes $n_c$ (confidence-labeled) and $n_u$ (unlabeled). \nThis adjustment simplifies the implementation process and supports the broader applicability of imitation learning with imperfect demonstrations in practical settings.\n\nFurthermore, the manuscript includes theoretical analyses showing that (i) the proposed objective helps avoid the imitation of non-optimal data and (ii) derives a sample complexity bound for the BSC method, providing a rigorous foundation for the proposed improvements.", "weaknesses": "Despite the many strengths of this work, the empirical results presented in the manuscript do not stand out as particularly impressive.\n\nSpecifically, Figure 1 shows that the performance difference between PN-GAIL and the most competitive baseline across tasks is not significant. In my opinion, as discussed in Section 2, since baseline methods typically assume a dominant proportion of $\\pi_{opt}$, exploring scenarios with a more skewed ratio (e.g., $\\pi_{opt}:\\pi_1=1:10$) might provide a more notable results where conventional methods fail while PN-GAIL successes. \nI think conducting experiments with more extreme demonstration ratios could more clearly demonstrate the scenarios in which PN-GAIL offers a distinct advantage over baseline methods.\n\n**[Minor Comments]**\n1. For Figures 2, 3, and 4, using distinct line colors for different methods would enhance readability.\n2. In the ablation study presented in Figure 3, it would be advantageous to include results from 2IWIL—even though they are already provided in Figure 2. Since 2IWIL represents a variant of PN-GAIL that excludes both the PN objective and the BSC, its direct comparison within the same figure would clarify the incremental benefits of these components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HLG9qyTtzc", "reviewer": "Reviewer_eg39", "summary": "This paper introduces an IL method that handles imperfect demos by including both the optimal and non-optimal data in training, assuming that the demos come with a confidence score. The goal is to improve policy learning when demos aren’t perfectly optimal. The method assigns weights to optimal and non-optimal examples through a semi-supervised confidence classifier.  Experiments on six control tasks show that PN-GAIL performs better than standard GAIL and other baselines under these imperfect conditions.", "questions": "* Theorem 1’s bound relies on knowing the variances. However, the variances can be difficult to estimate in real-world applications. Given this dependency, how practically useful is the bound in scenarios where variance values are unknown or hard to assess?\n*  Theorem 2’s bound may become quite loose if the Rademacher complexity is high, as is typical with deep and wide neural networks. Could this have negative implications for the method's reliability when using complex models?\n* Related to my disagree on the claim that 'GAIL treats non-optimal demonstrations as if they were optimal' in the weaknesses section, please could you provide a counterargument?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: The use of positive and negative risks to manage imperfect demos tackles some of the real-world problems of non-optimal data in a practical way.\n* Quality: The paper provides theoretical analysis for the positive-negative risk approach and shows experimental results across multiple control tasks.\n* Clarity: Most theoretical ideas are clearly presented with good notation.\n* Significance: The approach is relevant for real-world applications. It might have a real impact on the usability of IL in practical scenarios.", "weaknesses": "* The method’s reliance on confidence scores may limit its applicability when it’s difficult to assign confidence levels directly. Human annotation of preferences over trajectories, for instance, might be more feasible than assigning explicit confidence scores.\n* The fundamental motivation of this paper is based on a claim that 'GAIL treats non-optimal demonstrations as if they were optimal'. I disagree with this claim. GAIL’s objective is to minimize the JS divergence between the expert and agent trajectory distributions, aiming to reproduce the overall distribution of expert demonstrations. When the agent policy is close to the expert policy, the discriminator's output tends to be 0.5 everywhere. If expert demos include a mix of optimal and sub-optimal trajectories, GAIL should naturally capture this mixture without necessarily assuming optimality. Could you provide a counterargument or clarification on why PN-GAIL’s approach is necessary, given this perspective?\n* Lacks comparison with other advanced IL algorithms, such as f-IRL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "phg8J7OfIS", "reviewer": "Reviewer_XEhc", "summary": "The authors propose PN-GAIL, an extension of the GAIL framework designed to handle the imperfect expert demonstrations. By predicting confidence scores for unlabeled data, PN-GAIL allows for more accurate imitation learning without relying solely on optimal examples. Theoretical analysis is provided for the output of the optimal discriminator in the proposed method.", "questions": "1. Could the authors highlight expert performance more prominently in the figures to enhance clarity and interpretability?\n\n2. Would varying values of $n_{u}$ and $n_{c}$ significantly impact the performance of the proposed method? Additionally, could the authors provide guidelines or criteria for selecting optimal values for these parameters in practice?\n\n3. Figure 1 offers a valuable comparison of the proposed method against GAIL and 2IWIL. To further support this comparison, it would be better if the authors could also provide more intuition of why GAIL and 2IWIL fail but the proposed method succeed. E.g., gradient colors for different confidence scores, and why 2IWIL fails to predict them. Additionally, Why GAIL predicts 5.0 for some of theses data points?\n\n4. To further contextualize this work within imitation learning (specifically, driver behavior imitation), it would be beneficial to incorporate additional relevant studies, such as:\n\n[2] Ruan, Kangrui, and Xuan Di. \"Learning human driving behaviors with sequential causal imitation learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 4. 2022.\n\n[3] Hawke, Jeffrey, et al. \"Urban driving with conditional imitation learning.\" 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.\n\nand so on.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a new algorithm (based on 2IWIL and IC-GAIL) supported by theoretical analysis and demonstrates its effectiveness across multiple tasks. Experiments are extensive, including different benchmarks, different $\\pi_{OPT} : \\pi_{1}$ ratios, different standard deviations of Gaussian noise.", "weaknesses": "Although the proposed method is based on [1], e.g., some techniques to prove Theorem 4.1, Theorem 4.2, have been explored in [1], this study sill broadens the scope of [1]. One potential weakness is: although $\\alpha$ and $\\beta$ are intended to play distinct roles in Theorem 4.2, they are selected to be identical in Algorithm 1, which may affect the overall applicability of the algorithm. \n\n[1] Wu, Yueh-Hua, et al. \"Imitation learning from imperfect demonstration.\" International Conference on Machine Learning. PMLR, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a0IE0qLUCC", "reviewer": "Reviewer_1f3M", "summary": "Motivated by the limitation of 2IWIL in a type of scenario where certain non-optimal demonstrations have high probabilities of appearing in a set of imperfect (unlabeled) demonstrations, the paper proposes a new method named PN-GAIL, better leveraging optimal and non-optimal information from imperfect demonstrations to learn optimal reward and policy by assessing both positive and negative risks. Moreover, the paper modifies semi-conf classification in 2IWIL to establish balanced semi-conf classification to handle better the cases where certain demonstrations only exist in the confidence (known, labeled) data.\n\nThe authors conduct experiments, comparing their PN-GAIL to four baseline methods across six environments. The results show that PN-GAIL alleviates the impact of the unbalanced frequency in impact demonstrations, outperforms other methods, and maintains relatively good performances given the decreasing number of labels. Also, the outcomes demonstrate that the balanced semi-conf classifier improves performances, particularly in three out of six environments.", "questions": "1. About clarification: PN-GAIL\\BSC: PN-GAIL without balanced semi-conf (BSC) classification--does this mean no classification used or SC used? Without a probabilistic classifier, how to obtain confidence scores?   \n\n2. To verify BSC outperform SC in 2IWIL, straightforward comparisons are PN-GAIL(BSC) vs. PN-GAIL(switch to SC) vs. 2IWIL(switch to BSC) vs. 2IWIL(SC). Why do you consider the way of comparisons that are presented in the paper?\n\n3. It is better to provide more detailed explanations and discussions regarding those figures. For example, the statement about Figure 3 is quite short. The audience would like to learn about what possible reasons are that result in the varying performance patterns across six environments in Figure 3. We can observe that in some cases three colors (methods) are similar; in some cases blue and green are close; in some cases, blue and orange are close; while in some cases, blue works best. Could you provide a systematic analysis?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The main objective/purpose of this paper is articulated explicitly with the existing methods and their limitations highlighted clearly.\n2. The literature review on IL, GAIL, and IL with imperfect information is comprehensive. The preliminaries provide the essential information of 2IWIL.\n3. The theoretical derivations regarding the discriminator modification and classification refinement are straightforward and concise in the main body, which makes audience easy to follow; meanwhile, the supplemental materials in the appendices provide necessary, detailed explanations.\n4. The experiments with three goals setup are tightly related to the main achievements that the paper wants to claim. The experiments are conducted with representative benchmarks across various environments, effectively showing the performances with well-formatted figures and table in the main context.\n5. Overall, the authors identify the potential challenges of 2IWIL about certain non-optimal data with high frequencies in an unlabeled demonstration set significantly affecting reward and policy generation. The topic is likely to be of interest to a large proportion of the community. The proposed PN-GAIL creatively update the previous methods to successfully remove limitations of prior IL results to a certain extent.", "weaknesses": "1. Missing definition: in Section 3, what δ represents. Please define δ when it is first introduced in the paper.\n\n2. Needing clarification: PN-GAIL\\BSC: PN-GAIL without balanced semi-conf (BSC) classification--does this mean no classification used or SC used? Without a probabilistic classifier, how to obtain confidence scores? Please explicitly state whether a SC classification is used, and to explain how confidence scores are obtained if no classifier is used. \n\n3. Lack of analysis for experimental outcomes: it is necessary to provide more detailed explanations and discussions regarding those figures. For example, in Figure 3 what possible reasons (e.g., characteristics of each environment? limited number of demonstrations?) are that result in the varying performance patterns across six environments. We can observe that in some cases three colors (methods) are similar; in some cases blue and green are close; in some cases, blue and orange are close; while in some cases, blue works best. Please provide a systematic analysis of how different factors might contributing to these patterns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)