Imitation Learning from Vague Feedback
Authors: Xin-Qiang Cai
Yu-Jie Zhang
Chao-Kai Chiang
Masashi Sugiyama
Venue: NeurIPS 2023 poster (2023)
Keywords: imitation learning
vague feedback
risk rewriting
mixture propotion estimation

Abstract: Imitation learning from human feedback studies how to train well-performed imitation agents with an annotator's relative comparison of two demonstrations (one demonstration is better/worse than the other), which is usually easier to collect than the perfect expert data required by traditional imitation learning. However, in many real-world applications, it is still expensive or even impossible to provide a clear pairwise comparison between two demonstrations with similar quality. This motivates us to study the problem of imitation learning with vague feedback, where the data annotator can only distinguish the paired demonstrations correctly when their quality differs significantly, i.e., one from the expert and another from the non-expert. By modeling the underlying demonstration pool as a mixture of expert and non-expert data, we show that the expert policy distribution can be recovered when the proportion $\alpha$ of expert data is known. We also propose a mixture proportion estimation method for the unknown $\alpha$ case. Then, we integrate the recovered expert policy distribution with generative adversarial imitation learning to form an end-to-end algorithm. Experiments show that our methods outperform standard and preference-based imitation learning methods on various tasks.

Imitation Learning from Vague Feedback
Xin-Qiang Cai', Yu-Jie Zhang', Chao-Kai Chiang', Masashi Sugiyama The University of Tokyo, RIKEN AIP, Tokyo, Japan
Abstract
Imitation learning from human feedback studies how train well-performed imitation agents with an annotator's relative comparison of two demonstrations (one demonstration is better/worse than the other), which is usually easier collect than the perfect expert data required by traditional imitation learning. However, many real-world applications, it is still expensive even impossible to provide clear pairwise comparison between two demonstrations with similar quality. This motivates to study the problem of imitation learning with vague feedback, where the data annotator can only distinguish the paired demonstrations correctly when their quality differs significantly, i.e., from the expert and another from the expert. By modeling the underlying demonstration pool as a mixture of expert and non-expert data, we show that the expert policy distribution can be recovered when the proportion a of expert data is known. We also propose a mixture proportion estimation method for the unknown a case. Then, integrate the recovered expert policy distribution with generative adversarial imitation learning to form an end-to-ent algorithm"] Experiments show that our methods outperform standard and preference-based imitation learning methods on various tasks.
Introduction
Imitation learning (IL) is popular approach for training agents to perform tasks by learning from expert demonstrations [1.3] It has been applied successfully variety of tasks, including robot control [1 autonomous driving 4 and game playing [5]. However, traditional IL methods struggle when presented with both expert and non-expert demonstrations, as the agents may learn incorrect behaviors from the non-expert demonstrations [6]. This problem, which researchers refer to as "Imitation Learning from Imperfect Demonstration" (ILfID), arises when the demonstrations used to train the model may contain some non-expert data 6.9
One popular solution to ILfID is resorting to oracle to provide specific information, such as explicit labels (confidence from an expert) of each demonstration, as in Figure Ta However, such a specific oracle is quite expensive. A more recent framework, Reinforcement Learning from Human Feedback (RLHF) [10 incorporates human feedback into the learning process, including a key part of the pipeline used for training ChatGPT 12 There exist two widely studied paradigms of feedback: full ranking and global binary comparison, as in Figures [b and Ic] Methods that use full-ranked demonstrations assumed that the feedback between every pair trajectories available, and further methods to solve the problem [13 14]. Other studies have investigated
employed preference-bart situations where demonstrations can be divided into two global binary datasets, allowing the learner However, data processed by any feedback of Figures| la [b] Ind requirr the guarantee that clear ranking information or least one set consisting of pure expert demonstrations exists, so that off the-hhh-shll IL methods can be applied immediately. The availability of expert demonstrations raises the question of what if, practice the feedback is not clear enough to provide purely expert demonstrations but mixed with non-expert demonstrations? The question poses a challenge that previous methods fail to address since none of
to filter out non-expert data from these global comparisons 7 g
'The code is available on /ttpppps///gihthaa..////aaagagga http://thhhb.coocccaaaaa//eeeeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Amolator info Wopller snee))
Ancolitton on info thai ronking
07
End
5mh
(a) Explicit label
(b) Full ranking
Annotstor infe (Pebd comperisun
Annetsher info (Paga cumpensoot
Dama
Dema
(c) Global (clear) comparison
(d) Vague comparison (our setting)
(e) A maze example
Figure I: (d): The comparisons among the imitation learning with the explicit label, full ranking, global comparison, and our setting. II denotes the expert data while denotes the non-expert one. (e) An example of labeling route navigation. In the example, Path I and Path 2 share the same distance. similar situation lies in Path 3 and Path 4. The data collectors cannot provide explicit, ranking, or pairwise information as in (a). (b), and (c), while they can only provide vague comparisons that r+ can be more expert-like than 1"
the previous algorithms handles the issue of the mixture of expert and non-expert demonstrations without other information. Their conventional strategies are to deal with the vagueness in the data processing stage (relying on human feedback) not during the learning stage (by the learning agent).
Therefore, we consider a kind of weaker feedback than those previously formulated in the liter- ature, which unintentionally mixes expert and non-expert demonstrations due vague feedback. Specifically, the human annotator demonstrates in pairwise way and only distinguishes the expert demonstration from the non-expert one. However, the annotator cannot tell the origin when both are from an expert or non-expert. As depicted in Figure Idl this annotation process results in two datasets: r+ containing demonstrations the annotator believes to be most likely from experts, and r- containing von-expert-like demonstrations. If one only places the distinguishable demonstrations into the pools and discards the indistinguishable ones, results Figure [Ic Since we aim to investigate the potential of IL. under the presence of non non-expert demonstrations, we distribute the indistinguishable demonstrations randomly, one to I+ and the other to r- This step embodies the vagueness we wish to study in this paper. Note that either dataset contains non-expert demonstrations, and thus, a direct application of the existing method might be inappropriate due to the danger of learning from undesirable demonstrations.
Vague feedback commonly exists in many scenarios, especially regarding RLHF In a crowd- sourcing example of routes navigation, as shown in Figure Ic most of the time crowd-workers may lack domain expertise. Meanwhile, (when facing two trajectories from the same source, such as (I.0) or (C,D)). Therefore, we cannot obtain high hugh-quality crowd-sourcing labels when the data are annotated [15] as in Figures Ia| and Ic On the other hand, it is natural for workers to provide vague comparisons as in Figure ld
quite difficult distinguish every pair of demonstrations
In this work, we formulate the problem of Vaguely Pairwise Imitation Learning (VPIL), in which the human annotator can only distinguish the paired demonstrations correctly when their quality differs significantly. In Section 4 we analyze two situations within this learning problem: VPIL with known expert ratio a and unknown a. For VPIL with known a, we provide rigorous mathematical analysis and show that the expert policy distribution can be empirically estimated with the datasets I+ and r=; for the more challenging situation of VPIL with unknown a, we propose reduction of the problem of estimating a to a mixture proportion estimation problem [16] and develop an iterative update procedure to estimate a. In Section
5 we integrate our algorithm with an off-the-shelf IL
method to solve VPIL problems. In Section 6 we evaluate our methods with state-of-the-art IL
methods on variety of tasks of MuloCo 17 with different ratios and find that our methods obtained the best performance.
2 Related Work
In imitation learning scenarios, a model trained to mimic the actions expert demonstrations. One of the main challenges of imitation learning is that the gathered demonstrations can be imprecise, making it difficult for the learner accurately replicate the underlying policy of the demonstrations. To overcome this issue, various works have utilized annotator, source of supplementary supervi-

---

Demonstration Pool
Data Collection Process
Training Data
ratio
D
distributio
Expert policy
B Q
*whan D T
I' contains vaguely expertillee
policies
0000
"when T CD-CI
on Proz
randomily Into 1" and
I" contains vaguely non expert like data
Figure 2: The description the data collection process.
sion that can aid the learner in better understanding the expert's ' intent. For example, [8] used explicit action signals of the annotator; [14). [19), and [13] used ranking/preference information from the annotator to learn the policy; [6] utilized confidence information from the annotator to re- re-weight the unlabeled demonstrations and further learned the policy. However, illustrated in Section | cases, the annotator may not be able to provide explicit information for the imperfect demonstrations, especially when the demonstrations come from different sources.
in some
Alternative strategies for by imposing prior knowledge instead an annotator have also been put forward, such as state density estimation for importance weighting (20), state-action extrapolat- ing (21) and adding noise to recover ranking 22 But this research line relies on certain assumptions about the state/state-action spaces, which may not always hold in many real-works applications. In this work, we focus on using an annotator with vague information, which also low-cost.
Our methods drew inspiration from the risk rewriting technique in weakly supervised learning literature [33. which aims establish an unbiased risk estimator for evaluating the model's quality only with weak supervision. Although the technique has been successfully applied to various specific weakly supervised learning problems (2427), these methods typically require knowledge of parameter called the mixture proportion, i.e., the expert ratio in our problem [28]. the estimation of which can be challenging in the imitation learning problem (see Section 66.3 for more details). To overcome this issue, we introduce an iterative data selection procedure to better estimate the expert ratio, leading to superior empirical performance. It is worth noting that [6] also used the risk rewriting technique to identify the expert demonstrations, but their algorithm does not require estimation of the expert ratio as the confidence score given. However, such information is unavailable in VPIL, making the problem more challenging.
3 Preliminaries and Problem Setting
In this section, we first introduce the IL process. Then we formulate the VPIL problem.
3.1 Preliminaries
Markov Decision Process. In policy learning problems, a Markov Decision Process (MDP) can be represented by a tuple AS,A, P, r, T", where S is the state space; AA is the action space; P: S x A x S > [0, is the transition probability distribution; y is discount factor in the range (0, 11: r: : - R is the reward function; and T is the horizon. The goal to learn a policy æ that maximizes the expected returns E rrr(s,, at)], where E denotes the expectation.
In II, the learner does not have access to the reward function r. Instead, the learner is given m expert demonstrations TE,1, TE.2,..., TEmm, where TE,,, E (1....,m) m) is the i-th trajectory (a series of statc-action pairs) drawn independently following the demonstrator's policy mE. The goal of the learner is to learn policy Te to mimic TE.
Occupancy Measure. Since in II we not have reward functions, we need to measure the policy performance in the state-action space, i.e., occupancy measure Ps. The occupancy measure
Pa : x A + was introduced to characterize the distribution of state-action pairs generated by given policy π, which was defined with the discount factor: P (* a) :Taas) Σi= 7 PPlss mT)
where Pr[s: = $ π is the probability of reaching state s at time t following policy π 29
Moreover, we know that there is one-to-one correspondence between the occupancy measure and the policy. Specifically, we have the following theorem.
3

---

Theorem 1 (Theorem 2 of (29). Suppose is the occupancy measure for Tp(a () = L p(a,a)) Then, up the only policy whose occupancy measure is p.
p(a,a)
We can also define normalized version of the occupancy measure by (=( a
E' Pe(s'a = (1- (1-1)) p1 a). If we have m expert demonstrations TE,1, 7E,2. TE
pale.a)
the occupancy measure PT can be empirically estimated by the trajectories as P (8, =
(1) aa(') (s, a) where [[:] is an indicator function that returns 1 if is true
m
and returns 0 otherwise. Finding out the underlying Prg is the key for solving II. problems 1 30
3.2 Vaguely Pairwise Imitation Learning
In this work, we focus on the ILfID problem with pairwise information, where the learner aims to learn a good policy with pairwise dataset (T+, [--), generated from a mixture demonstration
pool performed by an expert policy TE and a set of non-expert policies { NE
(k) } A==1' In this work, we
consider the mixed occupancy measure of the non-expert policy set as PTNL The proportion of the
expert data within the pool is denoted as a E (0. 1]. The data collection process shown in Figure 2 To clearly reveal the effect of mixed demonstrations and explain the proposed framework, we choose to assume that the data collector is not an attacker and will make mistakes, so there will be no noise during the collection process. In Section 6.4, we discuss how our method can easily be extended to handle the presence of human error.
Also, in this work, we do not consider that the data collector could be an attacker or make mistakes, so there will be no noise during the collection process.
Collection of Pairwise Datasets. The data collector would first sample a pair of trajectories (Tr, TT) independently from the mixture pool. If (Ti, T2) are from different sources, i.e., one is from the expert, and another is not, then the collector will take the expert one тi into r and the non-expert one Tj into r- Otherwise, the collector randomly puts them into I and r- Under such data generation process, the expertise probabilities of this pair are as follows:
{
Pr[m: ~ Pre Tj ~ P = a²
Pr(m: ~ PANE: T3 PANNE = (1 a)
(1)
Pr(m: r Prrg Tj ~ Pawe] = 2a(1 oi)
In such case, I is always "not worse" than r' contains more expert data.
4 Learning frameworks for Solving VPIL Problems
In this section, we analyze the core challenge in VPIL problems with the pairwise datasets I and r- i.e., recovering the occupancy measure of the expert policy Prg. To this end, we propose two learning frameworks for VPIL with known and unknown a, respectively.
4.1
VPIL with Known &
Most of the methods based on the occupancy measure need to assume that the demonstrations are sampled only from the expert policy, so that they can directly estimate Png and match the learner's s distribution with PaE- However, the occupancy measure of the expert policy Pag is inaccessible in the VPIL problem, since both I'+ and I' contain non-expert data while the label of each data is unavailable. Below, we attempt to approximate Pre with {I', 1'
n4 be the empirical occupancy measure of trajectory T, I+,, where (st, ac,e) Ta is state action pair at time t. Let (s. Σm p. (s.a)/n.. Define Pr. (s, a) similarly. We have the following theorem.
Let n. |[ Dt (s) (1 7) Eio 7'1 [( ar,;) (s,a)) for 1,...,
Theorem 2. Assume the pairwise datasets (T+,--) are generated following the procedure in Section 3.2 Let Pr,, (s.a) a) Er! [pr, (s, (s,a)) and Pr_ (s,a) = Er Pr_ (s, as) | be the expected occupancy measures If and r-, where the randomness is taken over the draws of I and I-. Then, we have
Pry (s,a) (2a a²) (s, + (1 P)p (S (s,a), Pr. (s, a²pr, (s, + (1 a²) (s,
(2)

---

Prre (oe(s,,0) a), =
1ta 2a Pn+ (s, a)
1
2a Px_ (s, a)
(3)
PaNe (8,a)= =
2 a Pa,
Pa (8,
a)
+ 2-2n (5, a).
2-
proof can be found in Appendix A occupancy measure of the expert policy from contaminated data pools r and r= Thus, we can use an off-the- shelf IL. method learn the policy.
The Theorem provides a feasible way to recover the unknown
Once we have recovered Prr by Eq. (10), we can use an off-the-shelf IL method to learn the policy,
4.2
VPIL with Unknown a
When facing more challenging problem in which the expert ratio a is unknown, a straightforward way is to estimate the ratio first and then reconstruct the expert policy by the approach developed for known a cases. Here, we will first introduce how to estimate the expert ratio by reducing into mixture proportion estimation (MPE) problem [28) and then identify that direct application of the estimated ratio is not accurate enough to reconstruct the expert policy. To this end, we further propose an iterative sample selection procedure to exploit the estimated expert ratio, which finally leads to a better approximation the expert policy.
Estimation of the Expert Ratio a. In this paragraph, show that the estimation of the expert ratio a can be reduced to two MPE problems 22 Let P and N be two probability distributions, and
u 3 + (1 3)NV
(4)
be a mixture of them with certain proportion 8 (0, 1). The MPE problem studies how to estimate mixture proportion .3 with the empirical observations sampled from V and P (not from M). Over the decades, various algorithms were proposed with sound theoretical and empirical studies [31 32
To see how the estimation of the expert ratio a is related to the MPE problem, we rewrite 2 as
Px; as 3,p= (8,a) a) + (1- 31) and Pr. (s, 82px, (s, a) (1 32)
(5)
1-a and B2 = 2-a By taking Pr_ as li and P as P, the first line of (5) shares the where 31 = 1+a
a
same formulation as the MPE problem ④ Since P++ and Pr_ are empirically accessible via r+ and r+, we can estimate B1 by taking I and r- as the input of any MPE solver and obtain the expert ratio by a
1-81 1+01 The same argument also holds for the second line of (5). where we can take Pr_
as ll and Pr, as P to estimate 3. Then, the expert ratio can also be obtained by I+,92"
29g
One might worry about the identifiability of the expert ratio a by estimating it with MPE tech- niques (28). We can show that the true parameter is identifiable if the distribution Pr+ and distribution Pr_ are mutually irreducible [66 as follows:
Proposition 1. Suppose the distributions PTg and Prue are mutually irreducible such that there exists no decomposition of the form P=g = (1 1) + 1pPTTe and Pone (1 n') +11 Pr for any
probability distributions Q, Q' and scalars n. 1/ E (0, 1]. Then, the true mixture proportions B₁ and 32 unique and can be identified by
{
B1 sup(n/pr. = -P=_ +1 -)] KEC,,
(6)
82 = sup(n) Ipx = r'pm, + (1-r/)) K' C),
where C is the set containing all possible probability distributions. Thus, a identifiable by a1 -=11-))/(11 + 31)
==222/(1 + 32).
Proposition I demonstrates that the true mixture proportion 31 (resp. 32) can be identified by finding the maximum proportion of Pr_ contained in Pr,, (resp. Pr. contained in Pr_). This idea can be empirically implemented via existing MPE methods 28 33 We note that the expert ratio is identifiable by either estimating 31 or 82 when we have infinite samples. In the finite sample case, the two estimators coming from different distribution components could lead to different estimation biases. Since the MPE solutions tend to have a positive estimation bias on the true value 3, and /3 as shown by 28 Theorem 12] and [31 Corollary 1]. the estimation = (1 /1 + 81) tends to yield an underestimated a while that with B2 will lead to an overestimation. Besides, when the number of expert data is quite small, the underlying true parameter B2 would also have small value,
5

---

Algorithm
oEstimation
Input: Pairwise Demonstrations 1', -"; Numbers of Iterations I.
Output: Estimated Expert's Ratio σ.
I: function ESTIMATION([** ()-
2: Initialize D+ < I+ and +- r-
3: for each iteration until do
4:
Train a binary classifier fo : x A [0, by taking D+ as positive data and D as
negative.
5:
Assign a score for each data S + (s | (8, E D+} and S + (3, }
(s,a)
6:
Estimate 8 with mixture proportion estimator by taking S as U and S as P in 4 > We choose (1 (1-33) B1 + B) for better estimation.
7:
Estimate by (7).
8:
Data selection for : D <- -12 α² fraction of [+ of with top scores S Data selection for r- D + fraction of r- with bottom scores 5-).
9:
10: end for return
12: end function
Algorithm 2 COMPILER/COMPILER
Input: Pairwise Demonstrations r+ P Expert ratio a (Unknown for COMPILER-E); Environ-
ment env.
Output: The learner policy Initialize the learner policy To and the discriminator Da
2: if a is unknown then
COMPILER-E COMPILER
3: Obtain the estimated expert ratio a + ExpertRatiioEstimmaioo
4: else
5:
a v inpuuta(...
6: end if
7: for each training steps do
Sample a batch of learner's 's data (s,a) Png by the interactions between To and env. 9: Sample a batch demonstrations data (s, ~ in, and (s, a) ~ Pr_ from I and I"
8:
10: Update Da by maximizing (8).
11: Update To with (s, a) ~ Px. and the reward loog (w a) using off-the-shelf RL algorithm.
12:
end for
and its estimation would be highly unstable. Thus, we choose estimate a with 31 in our algorithm. The empirical studies in Section 6.3 also supported our choice.
The relationship between the expert and non-expert data shares similar formulation that of the unlabeled-unlabell (UU) classification 34 which is specific kind of weakly supervised learning problem studying how to train classifier from two unlabeled datasets and requires the knowledge of mixture proportions. However, how to estimate these mixture proportions is still an open problem in the weakly supervised learning literature. Although our reduction is developed for estimating the expert ratio for IL problems, it can be applied to a UU learning setting for independent interest.
Reconstruct Expert Policy by Data Selection. To handle the MPE task for high dimensional data, a practice is to first train classifier with probability output and then conduct the MPE on probability outputs of samples 35 However, the quality of the trained classifier turns out to depend on the estimation accuracy of a. On the other hand, we found that
possible to filter out the undesired component of the pairwise datasets (pmg in I' and PTNEE in I't and keep the desired data to enhance estimating a. Therefore, also inspired by [31] (3] of the distribution shift problem and weakly supervised learning, we propose we throw away the non-expert data with higher confidence in r and the expert data with higher confidence in r- after estimating a, and train classifier with the datasets after selection. The detailed learning process can be found in Algorithm !
deration-based learning framework. In each iteration,
After we obtained the estimated o we can recover Prg as by Eq. 10

---

5 COMParative Imitation LEarning with Risk rewriting (COMPILER)
We have illustrated how to estimate the occupancy measure the expert policy Pse from (I+, r under known and unknown respectively. Here we describe how to adopt these learning frameworks into end-to-end algorithms for learning an policy. We name our algorithm for VPIL with known COMParative Imitation LEarning with Risk-rewriting (COMPILER), and with unknown COMParative Imitation LEarning with Risk-rewriting by Estimation (COMPILER-E).
Current ofate-of-the-ard adversarial IL methods [37 [ aim to learn a policy by matching the occu- pancy measure between the learner and the expert policies. We fuse our methods with one of the representative adversarial II. methods, Generative Adversarial Imitation Learning (GAIL) [ whose optimization problem is given as follows:
min max (s,0)>>== log (s) B(s,a)-==E log(1 - Dw (s, a)), AE WEW
where Due : S A [0, parameterized by w is a discriminator, Pn. parameterized by 0 is the occupancy measure of policy. In setting, the expert demonstration of Prg is unavailable.
COMPILER. As suggested by Theorem 2 we can recover Paz with the occupancy measures of
1+a I+ and r Then, we can train the policy Pro by mincing Pr. 2g Pr+ 2n P=_ in Eq. (TO). Specifically, plugging (2) into (8) and approximating Pr+ and pr_ with their empirical versions, we can obtain the desirable target. However, such rewriting can introduce a negative term and lead to To avoid such an undesired phenomenon, instead of training the policy my to
1-a
overfitting 38
39
make PTy match 2a Pr.
1-a
2a PT_.. we twist the objective function of GAII. ( to minimize the 1app-_ and pr_.. which gives the following optimization problem
discrepancy between
2a
Pro
without the negative term:
min max 2a (e log (s, + (1 - a) () log(Dwr(s,))) a)) AE wEW
+ (1 + a) (sa)) log(1 (s,
Let V(TO. Dw) be the objective function in (8) and V(TT, Dw) be its expectation established Pr. and _=_ We have the following theorem for the estimation error of the discriminator trained by (8)
Theorem Let W be a parameter space for training the discriminator and Dw (D w W} be the hypothesis space. Assume the functions |log D., (s, a)] < B and |log(1 -w((s < B are upper-bounded for any state-action puir (8,a) S x A and w E W. Further assume both the functions log (8, a) and log(1 - (m, a)) are L-Lipschitz continuous in the state-action space. For a fixed policy ne. let 0 = {-( ] be trajectories generated from ng. Then, for any d E (0, 1), with probability at least
we have
(ITT D - V(TO, Dw)) & 4L(1 + R.)R, (Dw) + 4L(1 - RR) (Dw))
1
1
+ (Dw) + C(5)
+
Vno
VT+
where D., = arg max (TT. Dw,) and D = arg max,,ew V( Dwo). The constants n. and n_ are the number of trajectories in + and r- We define C(ß) 4B Alg(66//) The empirical Radamacher complexities [40] datasets P+ annadd 1 are denoted by Ra, Rn.. and Rag
proof is given in Appendix A The theorem shows that the discriminator trained by 8 converges to the one optimized with the true distribution Pr_ and p=_ at each step of the training.
COMPILER-E. For the VPIL problem with unknown a, first we estimate â by Algorithm then we learn the policy by with ã.
We integrate the two algorithmic processes COMPILER and COMPILER-E into Algorithm
6 Experiments
In this section, we conducted extensive experiments under the setting of VPIL. Through the experi- ments, want to investigate the following questions: (1) Can COMPILER and COMPILER-E solve VPIL problems under various expert ratios in demonstrations? using different MPE estimators to obtain @? (3) How is the estimation with Bn and 3g as in 7
(2) Is COMPILER-E still valid when
7

---

Table 1: The detailed information of demonstrations in the empirical studies.
Dimension of
Dimension of
Non-Expert
Non-Expen 2
Expert
Hopper
8817.80=819.69
3606. 114443.95
[0.1.02.00,
Swimmet
46.28111.55
62.61=16.82
100.6344..99
[0.1.02. 0.3. 0..
Walker2d
17
410.981156.91
396.67-5998.85
22013771888.5
[0.1.02.0. 0.3.
HalfCheetah
17
532. 0022500::88.66
864.18=335.90
1599.062-1.97
(0.1.02,0.3 0.3. 0..00.
6.1 Setup
To investigate the answer to the first question, we chose 20 different VPIL tasks with four MuJoCo benchmark environments evaluate the performance of the contenders and our approaches under five different a levels. The detailed setups are reported as follows.
Environments and Demonstrations. We Hallcheetaah, Hopper, Swimmer, and Walker2d in MuJoCo as the basic environments. For each experiment, the demonstration pool contains 100 trajectories with different expert ratios a {0.1,0.2 0.2, 0.3, 0.4, 0.53, in which a 0.1 is the most difficult situation of the VPIL problem. We also trained an expert-level RL agent and two non-expert The details of the environment and the
RL agents as demonstrators by DDPG algorithm demonstrations can be found in Table 1
41
Dataset generation process. We start with the demonstration pool {T.) which contains a. trajectories sampled from the optimal policy "E and (1 a). trajectories from the non-optimal policy MNE. For notation simplicity, we denote by D+ the part containing expert demonstrations only and by D_ the part for the non-expert demonstrations. Then, we can generate the dataset follows,
[+ sample(D,,, (2a a²) U (100))))) N))
r-- sample(D4. a N2 U sample(D_,, (1-aa)))
where sample(D, (D,N) is the function that samples N trajectories from the dataset D.
Contenders. Since [+ contains more expert demonstrations, here we use Behavior Cloning GAIL M.. and AIRL 43 with I*+ only as basic baselines. Also, we provide GAIL with expert-level demonstrations
42
as the skyline of all methods. Besides, we set T-REX [13. state-of- the-art preference-based IL method, by taking that every data in I is more expert-like than that in T as form of preference to train its reward function. We also the variants of T-REX algorithms, D-REX (22) and SSRR [44]. that directly generate the ranking information through the imperfect datasets. CAIL proposed a confidence-basse
method imitate from mixed demonstrations [14. In the experiment, we provide 5% trajectory labels to meet its requirement as suggested in their paper. Each method ran 4e7 steps. trials were conducted with different seeds for each task.
Meanwhile, to investigate the answer to the second question, we conducted Best Bin Estimation (BBE) 31 and Kernel Mean (KM) (28) algorithm estimate 3 in algorithm of COMPILER-E, as COMPILER-E (BBE) and COMPILER-E (KM) respectively. We note that COMPILER-E cannot obtain the true a during the whole experiment.
Implementation of a estimation. We implement a four-layer fully connected neural network as the binary classifier f& Algorithm 1. The neurons each layer are 1000, 1000, 100, and 50 respectively. The activation function of each layer is ReL.U, and the output value will go through the sigmoid function to obtain the score for I and r as S and S respectively. The optimizer for f is stochastic gradient descent. We totally train fø for 1000 epochs with bath size 1024. The initial learning rate is 5e-4, with an exponential decay rate of 0.99 at each step.
Implementation of policy training. We choose Proximal Policy Optimization (PPO) 45 the basic RI. algorithm, and set all hyper-parameters, update frequency, and network architectures of the policy part the same as [46). Besides, the hyper-parameters of the discriminator for all methods were the same: The discriminator was updated using Adam with decayed learning rate of 3 x 10-4; the batch size was 256. The ratio of update frequency between the learner and discriminator was 3: 1. For T-REX algorithm [13, we use the codes with default hyperparameters and model architecture of their official implementation.
6.2 Empirical Results
The final results are gathered in Table 2 algorithms, they cannot adaptively filter out non-expert data, so the underlying non-expert data reduces the performance of them. T-REX did not obtain promising results on VPIL tasks, which
We can see that since AIRL and GAII, are standard IL
8

---

Table 2: The episodic returns of each method on different tasks with five random seeds.
Aputäx a414
4 ---
0.3
0- -04
GAIL
115.0 1039
11.55
110000
540
520
130
Gall
11:00:110
180/0--011 16
45
3000
100 94
N00 20-00-000000 30
24.6 05
TAEN
-80.81-9..6
-30.49--6000 16:00
-55U0 47.70
2110 2111-99-255 25..5
148.47
200 26881
2196231-22 23.34
5526 11--44 26.37
531-34
24.55 115.3
18600 122.4
100575
177619
4.15
COMPEER SURRP)
02203310
01- 12:33
1923
08000-10 11
10000000 13
1-0 03 15.01
1111 N- 50
3394 PO
2071 08-
30
FO0A 18 190
3228
1.1.0
0.2
6.4
10000 L120
40.0 204
345531
20025
S4DKED 6 155
CLMB
11:09
00 12.00
91.3 5/5
711.4
2599
1:15
6589 9 1:7
-0-02
1-8X
1 07.01
0201 34
150
BRUST
100
2091 0
76.41
5/22
14538
2
01275-1113
DMBEX
11 2:5.0
9432
55.22
13722
15.15
32.41
179
12.15
5 125
102.13
262.3
100:00
583131-200
COMPILES
19:13 0:35
102.7 1.21
100.56
0.20
13235
112.58
138.20
1882)
29.88 5.23
152.43..80
3.00
36.754 4 3-21
-31.892 6-41
1900-1
3041.25
3548.73
240224
COMPLLSR
102.37
5464.6 221.4
4:34333
75.65
verifies that vague pairwise comparison can not be considered as full-ranking information, so the preference-based method cannot be used to tackle such problems. In addition, both SSRR and D-REX are updated versions of T-REX using noise to generate ranking information. They still cannot achieve good results due to its inability to address the core challenges of the VPIL problem. This phenomenon further demonstrates that preference- based algorithms cannot solve VPIL problems without explicit preference information, and some assumptions may not be held in VPIL problems. As grows, can see that the performances of almost all the contenders increased more or less, except for the skyline GAll__expert while COMPILER and COMPILER-E got the biggest performance boost and achieved the best performances. Even under the a = 0.1 situation, COMPILER and COMPILER-E have achieved expert-level performance on Hopper and Walker2d environments. For CAIL, it shows competitive performance with COMPILER and COMPILER- under high a situations, but still falls short in low a cases. The reason for this phenomenon that when a is relatively low, the trajectory quality corresponding to the provided trajectory labels may also be poor. Under such circumstances, estimating confidence is extremely difficult. As a result, CAIL generally performs suboptimally in low a cases. On the other hand, our algorithms can achieve comparable results with CAIL in high alpha scenarios. This indicates that our methods reasonably utilize the structure of the VPIL problem and can achieve results on par with the state-of-the-a algorithm without introducing additional information. Meanwhile, COMPILER (BBE) and COMPILER (KM) both achieved promising and comparable performances, which indicates that COMPILER-E is a general algorithm and can be operated with any effective MPE method.
6.3 Case Study: Estimation Effect on Data Selection
Overestimation and underestimation comparisons. To investigate if the superiority of COMPILER-E comes from precisely estimating a, and also to answer the third question about using two different estimations in (7) in Section 6 reported the estimation curves under different true a on HalfCheetah with BBE and KM estimators, as shown in Figure 3 We can see that the overestimation method obtained ā that far exceeded the true a, especially when a was smaller; while the underestimation method was very accurate despite obtaining a slightly lower ā value compared to the true a. for both BBE and KM estimators. The results also suggested that we indeed need to use the underestimation method for COMPILER when the expert ratio a is small. It also reflects that the power of COMPILER-E for solving VPIL problems indeed comes from accurately estimating the a value. More environmental results can be found in Appendix B
Thrown data effect. To further investigate what caused the performance gap between the overesti- mation and underestimation, we analyze the effect on the proportion of data selections in Algorithm [] Also connected to the expert and non-expert proportions of I and I' in ( we can calculate the theoretical value of the ratio differences of thrown data under different estimations with that under true a, as shown in Figure 3
We can see that the overestimated method threw away more data in r- and fewer data in I+ (corre- sponding to the upper left triangles of the red boxes in the figures); meanwhile, the underestimated method did the opposite (corresponding to the lower right triangles). However, as analyzed in Section 4.2 the overestimated method relied heavily on Pre of r- dataset, whose components, were quite small. In the case of finite data, once Prg of r- were overthrown, the remaining part of Prg became less, making the estimation further high, which led to a vicious circle. The phenomenon was especially severe in the case of low a. This is the reason why â of the overestimated method became

---

04
9 03 6
R.GI 21411
-08
42 405
estimates
6158
4050
01 02 08
02 04 0. 0. ca tue experts ratio
tus expera r
trs eoparts natc
tre exports raca
(a) BBE estimator HalfCheetah
on(b)
KM estimator HalfCheetah
on(c) Thrown data ratio differ(d) Thrown data ratio differ- ence on I dataset
ence on dataset
Figure 3: (a, b) The comparisons of overestimated I+ and underestimated 1- methods on T+.
various tasks with BBE and KM estimators. (c.d) The distribution map of thrown data ratio using estimated ã and true The thrown data ratio difference I+ is (1 a)² (1 a)),, while that on r a² a². The range marked by the red box is considered in our experiments, in which the
expert ratio is smaller than (or equal to) the non-expert
bigger as true a decreased. On the other hand, although the underestimated method also overthrew the non-expert data in r+ the proportion of these data was relatively high. So even more were thrown away at the beginning, it will not affect the accuracy of the estimation. This is the reason why using 1-8
to estimate ā is relatively stable and accurate.
1+3
6.4 Case Study: Assessing the Robustness of the COMPILER Algorithm
To evaluate the robustness of the COMPILER algorithm, comprehensive set of experiments were conducted. These were carried out in four distinct environments, implementing tasks with a = 0.5. Each experiment was repeated five times ensure the reliability of the results. To emulate real-world conditions, different levels of noise were introduced to the parameter a, resulting in the generation of corresponding noisy datasets. This noise was denoted s, thereby resulting in ā a + 6.
Figure presents the findings from these experiments. Remarkably, the COMPILER algorithm demonstrated consistent performance, achieving at least 96% of the standard performance presence various noise levels. This robustness, as exhibited in the results, underlines the capacity of COMPILER to effectively handle and perform under noisy conditions, specifically when the provided a subject to disturbances. This emphasizes not only the resilience of the algorithm but also its potential for deployment in real-world situations where data is seldom perfect.
in the
Matir Eovel
Sacur Loyer
(a) Hopper
(b) Walker2d
(c) HalfCheetah
(d) Swimmer
Figure 4: The performance COMPILER under different noise levels.
7 Conclusion
In this work, we formulated the problem of Vaguely Pairwise Imitation Learning (VPIL., in which mixed expert and non-expert demonstrations are present, and the data collector only provides vague pairwise information of demonstrations. To solve this problem, we proposed two learning paradigms, with risk rewriting and mixture proportion estimations (MPE), recover the expert distribution with the known expert ratio a and unknown one respectively. Afterward, we showed that these paradigms can be integrated with off off-the-shelf IL methods, such as GAIL, form the algorithm COMParative Imitation LEearning with Risk rewriting (COMPILER) and that by Estimation (COMPILER-F to solve the VPIL. problem with known and unknown a respectively. The experimental results showed that our methods outperformed standard and preference-based II. methods on variety of tasks. In the future, we hope to use our algorithms to address more challenging problems, such VPII. with multiple and noisy annotators.
10

---

Acknowledgments
This research was supported by the Institute for Al and Beyond, UTokyo; JST SPRING, Grant Number JPMJSP2108. The authors would like to thank Johannes Ackermann and the anonymous reviewers for their insightful comments and suggestions.
References
[1] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information 2016.
Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages
[2] Zi-Xuan Chen, Xin- Qiang Cai, Yuan Jiang, and Zhi-Hur Zhou. Anomaly guided policy learning from imperfect demonstrations. In 21 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, pages 244 252.
[3] Xin-Qiang Cai, Yao-Xiang Xiang Ding, Zi-Xuan Chen, Yuan Jiang, Masashi Sugiyama, and Zhi-Hua Zhou. Seeing differently, acting similarly: Heterogeneously observable imitation learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[4] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 1. 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 66-75. PMLR, 2019.
[5] Xin- Xin-Qiang Cai, Yao-Xiang Ding, Yuan Jiang, and Zhi-Hua Zhou Imitation learning from pixel-level demonstrations by hashreward. In AAMAS '21: 20th International Conference on Autonomous Agents and Mulliagent Systems, Virtual Event, United Kingdom, May 3-7, 2021, pages 279-287. ACM, 2021.
[6] Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6818-6827. PMLR, 2019.
[7] Yunke Wang. Chang Xu, and Bo Du. Robust adversarial imitation learning via adaptively- selected demonstrations. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3155-3161. ijcai.org, 2021.
[8] Mostafa Hussein and Momotaz. Begum. Detecting incorrect visual demonstrations for improved policy learning. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 1817-1827. PMLR, 2022.
[9] Yunke Wang, Bo Du, and Chang Xu. Unlabeled imperfect demonstrations in adversarial im- itation learning. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thir- teenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 10262 10270. AAAI Press, 2023.
[10] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton. Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from Al feedback. CoRR, abs/22 12. 0200000. 2022.
11

---

[11] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1302-4310) 2017.
[12] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, November 28 December 9, 2022, New Orleans, Louisiana, USA, 2022.
[13] Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 783-792, 2019.
[14] Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-awart imitation learning from demonstrations with varying optimality. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2,21, December 6- 14, 2021, virtual, pages 12340-12350 2021.
[15] Wei Wang and Zhi-Hua Zhou. Crowdsourcing label quality: theoretical analysis. Sci. China Inf. Sci., 58(11): 1-12 2015.
[16] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi- Semi-supervised novelty detection. Journal of Machine Learning Research, 13:2973-3009, 2010.
[17] Emanuel Todorov. Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pages 5026-5033, 2012.
[18] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 201 1, Fort Lauderdale, USA, April 11-13, 201 I, pages 627-635, 2011.
[19] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada., pages 8022-8034, 2018.
[20] Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10961-10970. PMLR, 2021.
[21] Yuanying Cai, Chuheng Zhang, Wei Shen, Xiaonan He, Xuyun Zhang, and Longbo Huang. Imitation learning to outperform demonstrators by directly extrapolating demonstrations. In Pro- ceedings of the st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pages 128-137. ACM, 2022.
[22] Daniel S. Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstraato imitation learning via automatically-ranked demonstrations. In Proceedings of the 3rd Conference on Robot Learning, 2019.
[23] Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, Tomoya Sakai, and Niu Gang. Machine Learning from Weak Supervision: An Empirical Risk Minimization Approach. MIT Press, 2022.
[24] Takashi Ishida, Gang Niu, and Masashi Sugiyama. Binary classification from positive- confidence data. In Advances in Neurul Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada,, pages 5921-5932, 2018
12

---

[25] Takashi Ishida, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. Complementary- label learning for arbitrary losses and models. In Proceedings of the 36th International Confer- ence on Machine Learning (ICML., pages 2971-2980, 2019.
[26] Nan Lu, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[27] Lei Feng, Senlin Shu, Nan Lu, Bo Han, Miao Xu, Gang Niu, Bo An, and Masashi Sugiyama. Pointwise binary classification with pairwise confidence comparisons. In Proceedings of the 38th International Conference on Machine Learning. 1CML 2021, 18-24 July 2021. Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3252- 3262. PMLR, 2021.
[28] Harish G. Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 2052-2060. JMLR.org, 2016.
[29] Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008, volume 307 of ACM International Conference Proceeding Series, pages 1032-1033 1039. ACM, 2008.
[30] Chelsea Finn, Paul F. Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. CoRR, abs/161 0tbs//611.0882 1.0385,, 2016.
[31] Saurabh Garg, Yifan Wu, Alexander J. Smola, Sivaraman Balakrishnan, and Zachary C. Lipton. Mixture proportion estimation and PU learning: A modern approach. In Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 8532-8544, 2021.
[32] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong. Gang Niu, Masashi Sugiyama, and Dacheng Tao. Rethinking class-prior estimation for positive-unlabeles learning. In The Tenth Interna- lional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[33] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Rethinking class-prior estimation for positive-unlabeled learning. In 10th International Conference on Learning Representations, ICLR 2022, virtual, April 25-29, 2022. OpenRe- view.net, 2022.
[34] Nan Lu, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, Louisiana, US, May 6-9, 2019, 2019.
[35] Yu-Jie Zhang, Peng Zhao, Lijun zhang, and Zhi-Hua Zhou. An unbiased risk estimator for learning with augmented classes. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, pages 10247-10258. 2020.
[36] Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 62, 2020, virtual, 2020.
[37] Justin Fu, Katie I Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3. 2018. OpenReview.net, 2018.
13

---

[38] Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positive- unlabeled learning with non -enattatiee risk estimator. In Advances in Neural Information Processing Systems 30: Anmual Conference Neural Information Processing Systems 2017,
NIPS 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1675-1685, 2017.
[39] Nan Lu, Tianyi Zhang, Gang Niu, and Masashi Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1115-1125, 2020.
[40] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, Foundations of Machine Learning. The MIT press, 2012.
[41] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, 2016.
[42] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15, Intelligent Agents (St. Catherine's 's College, Oxford, UK, July 1995], pages 103-129. Oxford University Press, 1995.
[43] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference Learning Representations, 2018.
[44] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demon- stration via self-supervised reward regression. In 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pages 1262-1277. PMLR, 2020.
[45] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707447777 2017.
[46] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.
[47] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
14

---

A Proof for the Theorems
In this section, we provide the proof for Theorem 2, Proposition and Theorem 3 in the main body of the paper.
Theorem 2. Assume the pairwise datasets (T+, r-) are generated following the procedure in Section
3.2 in the main paper. Let Px+ (s,a) Er [a_ (s. and Px_ (s.a) = Er [Pn_ (3, a)] be the
expected occupancy measures of and r_ where the randomness is taken over the draws of I and p Then, we have
Pri (8, a) = (2a- (re( a) + (1 PonE (8, Pr_ (8,a) = a Prp a) + (1 -a2) (8)
(9)
Prre (8,a)= =
1+a
2a pT+ (8, a) 1-a pr (8, a),
2a
(10)
PTnne (s,a)= 2(-a a) Pa+ (s + 2-a
2 2a Pr (s.
Proof. We first prove the first line of (9). Considering the data collection process introduced in Section 3.2, where a pair of trajectories (r(1, 7()) is independently sampled from the demon- strator pool. When one trajectory enjoys better quality than another, the collector would put the better one into rt_ Otherwise, the two trajectories will be randomly allocated. Let P (s.a) (1
7) Eee [ at,s) = a)] be the empirical occupancy measure for the trajectory Tj E r+. Then, further define the even Es E = {T; is sampled from J We have
{Ti is sampled from TE} and event
Bp. (s)
Prr (s, Ei] + Pr
[p. (s.a) [g]]
-Pr(Fr]p> Pr PTNEE
(11)
The second equality is due to т; is i.i.d. sampled from Prş under event E₁ and is i.id. sampled from Pr_ under event Eg. Then, let the even A {both ((1), 72) are sampled from TE), A2 =
(both ,11), 7(2) are sampled from TTE) and A3- {other situation). Then, we have
Pr[Ei] = Pr[A AA] + Pr|A2|Pr(Es A2] + Pr/ss/Pr[E Ass
= = 21 a) 2a a²
where the second inequality is due Pr(F,, |A] Pr(FEs Ag] = and Pr Pr[Fy Ag] 0 according to the data collection process. We can also show the probability of event Fg is Pr Pr(Fr] = then we have
B (s, a)] = (2a - a²) (s, + (1 a)2 ())
Since pr. a)) = Em p. (s,a)/m+, we have BF (s, a)] E Elp (s, [s,a))/n++
(2a a²) Prre (s. (s (1- PPm (s, a). which completes the proof for the first equality of (9) The second equality of (9) can be proved following a similar arguments
Proposition 2. Suppose the distributions Pxg and Puns mutually irreducible such that there exists no decomposition of the form Pr+ = (1 11 + TTp and Prwe = (1 - '/' + n'Pre for any probability distributions Q, Q' and scalars n, r/ E (0, 1]. Then, the true mixture proportions B1 and B2 are unique and can be identified by
{
31 sup(rlp=, -p= +1 (1-1)K, EC),
(12)
B2= = sup(n')pe Ipx_. r'pa,, (1-1')K K' E eC),
where C is the set containing all possible probability distributions. Thus, a is identifiable by
a a= (1 B1) + 31) or a 23222111 + 32).
(13)
Proof. We first prove the first line of (2). Since prr is irreducible w.r.t. PIre such that there exists no decomposition P.g (1 Q 7/PTNE for any 7) E (0, 1] and distribution Q, we have Pre also irreducible w.r.t. Pr_,, such that Pr: can not be rewritten as a mixture Pr. = (1 11-10 -/')Q' + 17 Pr_ for any distribution Q' and o/' E (0. 1]. Then, according to the relationship (5) of the main paper and
Proposition 5 of [16], we can show 31 is identifiable and 81 sup(rlp=, .P+- + (1 n) K, K e
C}. We can prove the second line of (12) by similar argument. Then, combined with B1 I+a 32 Z-a in (5) of the main paper, can obtain identifiable estimation in [3
I-a and
15

---

Theorem 3. Let W be a parameter space for training the discrininator and = w W W) be the hypothesis space. Assume the functions |log D. (s, a)l S B and [ogg(1-D(ss, a))l B are upper-bounded for any state-action pair (8, a) S x A and w E W. Further assume both the functions log Dw(s, (o(8,)) a) and log(1 Dw a)) are I--Lipschitz continuous in the state-action space. For fixed policy ne, let 1 { Jim1 be trajectories generated from Tg. Then, for any ò E (0, 1), with probability at least 1 5, we have
V (TT; D=) - (TTO, Dw)) & 4L(1 + a.)R, (Dw) + 4T((1 - R)) (Dw))
1
+ ST. (Dw) + C(6)
Vna
v
where D arg waw (ITE Dw) and D arg max (TTT, Dw). The constants n4 and n. are the number of trajectories I and r We define C(6) = 4B /log(6//6). The empirical Radamacher complexities [40) on datasets D+, r and r are denoted by Rny, Rm, and Rus-
Proof. Since Dw and D the maximizer of the objective functions (TTe, Dw) and V(To, Dw). respectively.
V (m)) (ITT, Dw) = V(To, D() - V (T) + VTT, Dw) - (TT,, (14)
+ V(TT, Dw)) - (TT, Dw))
S V (TT, Dw) - V(TT, D + V (TTO Dw) - V(TO, D
2 sup (TT, Dw) ((TT. Dw)l,
(15)
wEW
where the inequality is due to the optimality of D.,. The according to the definition of V( Dw)
and VTTO Dw). VTTA, Dw.) 2cB(e.)) Dog D (8, a)] + (1 a) [log(D..ss, a))]
(16)
(1 a (
[log(1 - D (s.
and
(TTO Dw) = 2aB (a log Du (s, a)] + (1 - a ( [log (w a))l
(17)
+ (1 + a (5,))= [log(1 Dw a))],
where (s. a) 1 E.s (s, a) and p. (s. a) = (1 - E 7 [(s,,,, a,,) = (s,a)) is the empirical occupancy measure for the trajectory ti sampled from कe The empirical occupancy measure unbiased w.r.t. Te such that Elp a]] Pro (s, a) Then, can decompose the R.H.S. of (15
as
sup V(To, Dw) - (TO. Dw)l term (a) + term (b) + term (c).
weW
In above, the first
term (a) = 2xx ((,,,) Dlog (sn a)) E,,,))y,,, flog D (8, a)l
1 no
2a
Dw (s, a)] (,,aa)pp [log D (s, E (. log D
measures the generalization gap between P., and Pas. The second term
1 Σ E (.,0) [log (s, - B [log (s,
term (b) (1 - a)
measures the generalization gap between Pr_ and Pr_ and the last term
term (c) = (1 a)
E (na)) log(1 Du B(s.a)=ps) [log(1 D (s, a))l
n4 im1
16

---

measures the gap between P.++ and Pr..
Under the condition that llg (s)( S B for any (8,a) 8 x A and W, the standard analysis generalization analysis with Rademacher complexity shows (e.g. Theorem 26.5 of (47)) for all w W, have
2 2mm(2/))) term (a) S 8aRnn (log o Dw) + 4a
no
with probability least 1 - 8', where Rng is the empirical Rademacher complexity. Then, according to the Talagrand's contraction inequality (Lemma 26.9 of [47)) we have
Ry, (log Dww S LRn, (Dw),
Then, obtain
2 2ln(2/8))
term (a) S Ra (Dw) + 4aB/
(18)
116
Since llo((1 Dw)l & B and log(1 - Dw.) is L-Lipschitz continuous, the a similar arguments ensures,
2 1ln(2/))
term (b) v 4(1 aR) (Dw) + a)),
(19)
71
and
2
(2/8")
term (c) < a)) (Dw) + 2(1 + aa) B
(20)
with probability least 1' Let 8' 8/3, we complete the proof by combining (18). with (15).
19 and 20
B Full Results of Estimations
In the main body of the paper, we only reported the effect of overestimation and underestimation on HalfCheetah environment due the space limitation. Here we provide the full results on four environments in our experiments, as shown in Figure The experimental results demonstrated the same conclusion as in the main body of the paper, that underestimation method is better than the overestimation one.
17

---

edtnetted eaper meo.
(a) BBE
estimator
on(b) KM
estimator
on(c) BBE estimator on Hop(d) KM estimator Hopper per
HalfCheetah
HalfCheetah
(e) BBE estimator on Swim-()) KM estimator on Swim (g) mer
BBE Walker2d
estimator on(h)
KM
estimator on
mer
Walker2d
28 Figure 5: The comparisons of overestimated
I+3
and underestimated methods various
tasks with BBE and KM estimators.
18


Reviews:
[{"review_id": "WocwZg0QDM", "reviewer": "Reviewer_zY91", "summary": "The paper studies the problem of Imitation Learning (IL) when the provided demonstrations come from experts and non-experts. The dataset may then not divided exactly into expert and non-expert subsets, but instead a data collection procedure may only vaguely separate the data. Thus, the authors pose the problem of Vague pairwise imitation learning, where the goal is finding a policy that approximates the expert policy that provided the expert demonstrations. \nTo tackle the problem, two cases are considered: the ratio of expert to non-expert demonstrations is known or unknown. In the first case the authors show how the occupancy measure (commonly used in IL) of the expert policy can be estimated, which then allows to solve the problem using standard IL methods. In the second case, the ratio is estimated using a lower bound.\nThe approach is evaluated in 4 different simulation setups and compared against numerous state-of-the-art IL methods. For various ratios of expert to non-expert data, the proposed method shows clear advantages over the baselines. Lastly, the evaluation provides a robustness analysis for the estimator, as well as empirical results motivating the use of the lower bound as opposed to an upper bound. \n", "questions": "Questions\n\n1.\tProblem statement (Sect. 3.2): There is no definition of an expert and non-expert policy. How do they differ? Is an expert always consistent, i.e., do their demonstrations optimizes the same reward function (without any noise)? It would be good to define the policy types at least on a high level to obtain a well-posed problem.\n\n2.\tNo clear statement of contribution: The introduction would benefit from a contributions section, highlighting the novelty of the work.\n\n3.\tResults: Line 292 “As α grows, […] COMPILER and COMPILER-E got the biggest performance boost and achieved the best performances.” This claim does not seem to fit the data reported in the table. On two of four tasks it is CAIL that achieves the best performance for the largest alpha=0.5. Furthermore, the relative increase between alpha 0.1 and 0.5 is only up to 5% for COMPILER, while CAIL increases the return by 50-100%. Either I misunderstand something, or the claim is wrong, please clarify.\n\n4.\tThe paper should have limitations section, discussing how restrictive the model for the data collection is. Further, related to my first question, some discussion on the differences between the expert and non-expert policies would be required, and how accurate the assumed policies are.\n\n\nMinor Comments\n\n5.\tWhat are the objectives in the maze example in Figure 1e? Only distance is mentioned, but with only one objective the task becomes rather trivial, even for non-experts\n\n6.\tEnd of problem statement: It would be good to conclude the section with a concise problem statement (in a theorem-style environment). Earlier it is mentioned that the goal is to learn a policy $\\pi_{\\theta}$ that mimics the expert policy – does that mean it minimizes the difference in the occupancy measure? It would help to make such an objective explicit.\n\n7.\tLines 135-136  seem to be redundant w.r.t. to the paragraph before\n\n8.\tLine 160. What algorithm is referred to? Is (3) referring to equation 3? I cannot really follow the sentence, this should be clarified.\n\n9.\tResults: The problem statement defined alpha to be in (0,1], yet the experiments only consider alpha from .1 to .5. What happens for larger alpha – do the baselines catch up eventually? These results might not be as interesting as for the lower alpha reported, but this could at least be mentioned. Also it is peculiar in Fig 3 a and b that the over and underestimate already converge for alpha=.5 – is there a reason for that?\n", "limitations": "The paper does not have a limitations section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper is very well written and addresses an interesting and active research area. \nThis work addresses limitations of state-of-the-art IL algorithms when the dataset is a mixture of expert and non-expert demonstrations.\nThe proposed approach is - to the best of my knowledge - original  and the technical contribution is sound and provides theoretical results on the way. The simulation results are overall convincing and effectively showcase the benefits of the proposed method.", "weaknesses": "The structure of the paper could be improved by adding a clear problem statement and list the main contributions.\nIn the problem description the paper should add some more detail, in particular definitions of the expert and non-expert policies.\nSome parts of the discussion of the results should be clarified.\nFurther, the paper does not have a discussion of its limitations and how restrictive the assumptions / model for the data collector are.\n Please find my detailed comments below.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for their positive remarks and in-depth engagement with our problem setting. We will now address the concerns raised.\n### Q1:\n**Query:** \"no definition of an expert and non-expert policy. How do they differ?\"\n\n**Response:** Our approach adheres to the foundational definitions from the imitation learning from imperfect demonstrations community, where both the expert and non-expert operate within the same environment with a unified goal (the same reward function) [1,2,3]. Specifically, for our experiments, we trained converged RL agents as experts to produce expert demonstrations, while the unconverged agents were treated as non-experts. The performance metrics of both experts and non-experts can be found in Appendix B.\n### Q2:\n**Query:** \"No clear statement of contribution.\"\n\n**Response:** Our primary contributions are:\n\n1. Introduction of the Vaguely Pairwise Imitation Learning (VPIL) problem. In VPIL, data annotators, when faced with the challenge of distinguishing between two trajectories, can only provide ambiguous feedback. The agent's objective is to derive an effective policy from such nebulous information.\n2. We proposed methods to extract the expert policy distribution from a mix of expert and non-expert demonstrations using risk rewriting and mixture proportion estimation techniques.\n3. Integration of our methodology with existing imitation learning algorithms, such as GAIL, resulted in two end-to-end solutions, COMPILER and COMPILER-E, for VPIL problems with and without \\( $\\alpha$ \\). Additionally, we conducted rigorous mathematical analyses to ensure method convergence.\n4. A comprehensive set of experiments were undertaken to validate our claims. The results showcased the superiority of our techniques over traditional and preference-based imitation learning methods, emphasizing their effectiveness in handling ambiguous feedback.\n\nWe acknowledge the oversight of not explicitly listing these contributions in the original submission due to space constraints. We will ensure their inclusion in the revision.\n### Q3:\n**Query:** \"'COMPILER and COMPILER-E got the biggest performance boost and achieved the best performances.' This claim does not seem to match the data reported in the table.\"\n\n**Response:** We appreciate your attention to this matter. An oversight led to this erroneous statement, and we will rectify it in our revision. Our algorithm exhibits a diminishing performance boost as \\( $\\alpha$ \\) increases. This is because even at \\($ \\alpha = 0.1$ \\), our proposed method had already achieved expert-level performance, comparable to GAIL_expert, which is trained exclusively with expert data.\n### Q4:\n**Query:** \"The paper should have a limitations section\"\n\n**Response:** We have briefly touched upon the limitations in lines 135-136 and 343-344 of our manuscript. Here, we indicated that our framework doesn't factor in noise introduced by potential attackers or annotator errors. Additionally, we only considered a single annotator for VPIL problems, whereas real-world scenarios might involve multiple annotators with distinct preferences. In response to your feedback, we will restructure and elaborate on these limitations in our revision.\n### Q5:\n**Query:** \"What are the objectives in the maze example in Figure 1e?\"\n\n**Response:** The primary goal in this example was to demonstrate that even with a simplistic objective, like distance, annotators might struggle to provide a clear preference between two trajectory pairs. This serves as an illustrative motivation. For our experiments, we incorporated more intricate continuous control tasks to validate the efficacy of our methods.\n### Q6:\n**Query:** \"It is mentioned that the goal is to learn a policy $\\pi_\\theta$ that mimics the expert policy – does that mean it minimizes the difference in the occupancy measure?\"\n\n**Response:** The aim of learning a policy $\\pi_\\theta$ to mimic the expert policy amidst mixed demonstrations is central to VPIL problems. However, the methodology we adopted, which involves minimizing the difference in the occupancy measures between the learner's and the expert's policies, is distinct from the main objective. Your observation is astute, and we will clarify this distinction in our revised manuscript.\n### Q7:\n**Query:** \"Line 160... Is (3) referencing equation 3?\"\n\n**Response:** Indeed, in our manuscript, indicators within parentheses, such as (3), denote references to specific equations. In this context, we aimed to highlight that Equation 3 plays a pivotal role in enabling us to retrieve the expert policy's occupancy measure, which is integral to our algorithm.\n### Q8:\n**Query:** \"What happens for larger alpha\" and \"It is peculiar in Fig 3 a and b that the over and underestimate already converge for alpha=.5\"\n\n**Response:** Our focus was primarily on challenging tasks, with \\( $\\alpha = 0.1$ \\) presenting the most arduous scenario, where merely 10% of the data originates from expert sources. Consequently, we restricted our experiments to \\($ \\alpha$ \\) values within (0, 0.5]. For larger \\( $\\alpha$ \\) values, our algorithm can still emphasize expert data imitation. In scenarios with unknown \\( $\\alpha$ \\), the estimation of \\( $\\alpha$ \\) becomes more precise since the target proportion (pertaining to the expert) exceeds the non-expert proportion. Regarding Fig 3 a and b, the over and underestimation methods converge at \\( $\\alpha = 0.5$ \\) due to their estimation values aligning. Through Equation 7, we deduced that \\( $\\beta_1 = \\frac{1-\\alpha}{1+\\alpha}$ \\) and \\( $\\beta_2 = \\frac{\\alpha}{2-\\alpha}$ \\). When \\( $\\alpha = 0.5$ \\), \\( $\\beta_1 = \\beta_2 = \\frac{1}{3}$ \\), leading both estimation methods to converge at this point.\n\n## References\n[1] Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality. NeurIPS 2021.\n\n[2] Learning to Weight Imperfect Demonstrations. ICML 2021.\n\n[3] Imitation Learning from Imperfect Demonstration. ICML 2019."}, {"review_id": "ERJIAa8mq5", "reviewer": "Reviewer_B2bu", "summary": "The paper formulates the problem of imitation learning from imperfect demonstrations with vague comparisons named Vaguely Pairwise Imitation Learning (VPIL) which contains both expert and non-expert demonstrations, and the data collector only provides vague pairwise information of demonstrations. The authors proposed two learning paradigms to solve VPIL. One with risk rewriting and mixture proportion estimations (MPE), to recover the expert distribution with the known expert ratio α and unknown one respectively. They evaluated the proposed method on simulate MuJoCo based tasks.", "questions": "Mentioned above in weakness", "limitations": "- I don't think authors have mentioned the limitations of the proposed method. They are encouraged to mention them in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper attempts to solve a pretty interesting problem.\n- The experimental results demonstrates that the proposed method is able to outperform standard and preference-based IL methods.\n- The paper presents good theoretical analysis of the problem and proposed method.\n- Extensive comparison to baselines.\n  ", "weaknesses": "- The paper has some typos and grammatical errors. For ex. In line 333, \"the\" is written twice; In line 36, \"off-the-shall\" -> \"off-the-shelf\". I would highly recommend a senior author to proof-read the paper.\n- Some parts of the paper are a little difficult to follow. For ex. Line 43-53.\n- Also, the problem of vague feedback is a little unclear.\n- The paper evaluates the method on 4 different MuJoCo tasks. However, problem is motivated from real-world robotics/imitation based task. It would be good for the authors to at least discuss how there method would work in real-world. If possible they can perform some simple experiment on real-world task that would be great.\n- There is only one small future extension mentioned in the paper. The authors are encouraged to at least mention few more and have breif discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're appreciative of the reviewer's feedback, especially the emphasis on enhancing clarity. We will address the technical and motivation-related queries below.\n\n### Q1:\n**Query:** \"Some parts of the paper are difficult to follow, e.g., Line 43-53 and the problem of vague feedback is unclear.\"\n\n**Response:** In lines 43-53, we described a real-world data collection scenario. Contrasting with Figure 1c, where the annotator provides a clear global comparison, in our problem, the annotator can only offer a vague comparison. This process is elucidated in Figure 2. We acknowledge the ambiguity and will enhance clarity regarding vague feedback in our revision.\n\n### Q2:\n**Query:** \"Discussion on how the method would work in the real-world.\"\n\n**Response:** In real-world applications, the data collection process illustrated in Figure 2 can be directly employed to curate \\( $\\Gamma^+$ \\) and \\($ \\Gamma^- $\\) datasets. This approach minimizes the annotator's burden as they only need to provide vague feedback for data pair differentiation. Subsequently, our algorithms (COMPILER for known \\($ \\alpha$ \\) scenarios and COMPILER-E for unknown \\( $\\alpha$ \\) cases) can be employed to derive the desired policy. We value this suggestion and will incorporate it into our revised conclusion.\n\n### Q3:\n**Query:** \"There is only one small future extension mentioned in the paper. The authors are encouraged to at least mention few more and have breif discussion..\"\n\n**Response:** We concur that the exploration of the broader applicability of our work is valuable. In addition to the two potential extensions in our manuscript, we envisage other promising directions, such as:\n\n1. **Active Learning Integration:** An active learning paradigm could enable the agent to judiciously query human annotators for clear feedback on demonstrations that would most refine the imitation model.\n2. **Hierarchical Imitation Learning with Vague Feedback:** Implementing hierarchical models might offer a nuanced approach to assimilate vague feedback across varying task granularities.\n3. **Integration with Other Feedback Mechanisms:** Exploring the synergy between the vague feedback mechanism and other feedback types, like reward shaping or advice-giving, could augment the learning process.\n\nWe aim to incorporate these discussions into our revised manuscript, contingent on space availability.\n"}, {"review_id": "oDXQ7tFQdA", "reviewer": "Reviewer_g4mG", "summary": "The manuscript considers a popular learning framework, learning from human feedback/preference. In real-world applications, the feedback can be inaccurate and detrimental to the training process. The manuscript formulates the imitation learning (IL) from vague feedback process, in which the data annotator can only provide accurate preference when the paired demonstrations are significantly different. The authors analyze the problem for two different levels of difficulty, i.e., known or unknown expert proportion $\\alpha$, and propose two algorithms based on risk rewriting and mixture proportion estimation to solve the two problems separately. The authors also conduct experiments and case studies with various IL tasks and state-of-the-art baselines to verify the effectiveness and generality of the algorithms.", "questions": "Is it possible to separate expert and non-expert data first, and then use T-REX to learn the policy?", "limitations": "Although the authors have illustrated VPIL with multiple annotators as the future work, I recommend the authors to discuss it as a limitation of this work with more details.", "rating": 9, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "### Originality\nThe manuscript introduces a realistic problem in feedback-based imitation learning, accounting for the potential noisiness in human feedback. The proposed scenario is distinguished by its realism and poses novel challenges to the reinforcement learning and imitation learning communities. Furthermore, the authors ingeniously develop a learning framework, which, when integrated with existing imitation learning techniques, effectively addresses the stated problem.\n\n### Quality\nThe authors adeptly formulate the problem, lending credence to its practical significance. The learning framework presented is versatile, facilitating integration with various imitation learning (IL) and mixture proportion estimation (MPE) methods. Commendably, the manuscript offers rigorous theoretical analyses regarding the interaction between MPE and the Generative Adversarial Imitation Learning (GAIL) approach. Moreover, the experimental section is robust, featuring comparisons across an extensive array of tasks and benchmarks. This establishes the efficacy and generalization capabilities of the proposed algorithms.\n\n### Clarity\nThe writing is lucid, and the claims advanced are substantiated effectively. The incorporation of a comprehensive collection of case studies furnishes empirical validation, thus bolstering the merits of this work. This meticulous detailing is, in my estimation, one of the strong suits of this manuscript.\n\n### Significance\nThis manuscript addresses a salient issue within the reinforcement learning and imitation learning domains - learning from human feedback. Through an adept modeling of ambiguous feedback and the introduction of an efficacious learning framework, this manuscript renders important contributions to the field. These are all very important contributions to the community.", "weaknesses": "The text size in Table 1 is a little too small. Also, I recommend the authors to put the robustness results in Appendix D to the main manuscript. This is also an important case study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the reviewer's encouraging comments and constructive feedback. We will address some of the technical inquiries below.\n\n### Q1:\n**Query:** \"I recommend the authors to put the robustness results in Appendix D to the main manuscript. This is also an important case study.\"\n\n**Response:** We acknowledge the importance of the robustness results when encountering the noise on $\\alpha$. We will endeavor to integrate these results into the main manuscript during our revision, ensuring they receive the attention they merit.\n\n### Q2:\n**Query:** \"Is it possible to separate expert and non-expert data first, and then use T-REX to learn the policy?\"\n\n**Response:** Your suggestion offers an intriguing perspective. However, conventional preference-based algorithms, including T-REX, do not yield satisfactory results in VPIL scenarios when applied in this manner. It is because that T-REX necessitates comprehensive ranking data, but post-separation, we only have binary datasets, which are informationally inferior to full-ranking datasets.\n"}, {"review_id": "rjGCYN838E", "reviewer": "Reviewer_DWbY", "summary": "This paper falls into imitation learning with human preference. Specifically, this paper deals with a more challenging setting, where the annotator information is wrong, i.e., some data collected by non-experts could be labeled as positive demonstrations, and some data collected by experts could be labeled as negative demonstrations.\nThis paper proposed two methods to solve this challenge, one with a known ratio of real demonstration data, and one with an unknown ratio.\nThe proposed approach is theoretically justified, and evaluated in the mujoco domain.", "questions": "If the authors could show that the proposed method can work well in a more challenging task, such as humanoid running, this paper could be greatly improved.", "limitations": "This paper has not discussed the limitations in the main body of the submission.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is theoretically justified.", "weaknesses": "The experiment results in rather confusing, e.g., in Table 1, for the HalfCheetah environment, apparently, the return of COMPILER is not the largest, but it is with the bold font, and the return of GAIL if significantly larger than the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the inconsistencies in Table 1. An updated version of Table 1 has been made available in the general response's PDF file. We have also ensured reproducibility by providing the code and demonstrations from our experiments via anonymous links. We are committed to addressing these concerns in our revised manuscript. Below, we delve into a discussion on potential further experiments.\n\n### Q1:\n**Query:** \"If the authors could show that the proposed method can work well in a more challenging task, such as humanoid running, this paper could be greatly improved.\"\n\n**Response:** We concur that validating our proposed method on more demanding tasks would be beneficial. We have earmarked such tasks for the project's scaling-up phase. In the current submission, our objective was to showcase the efficacy of handling mixed demonstrations from both experts and non-experts in VPIL scenarios. While we have incorporated multiple continuous control tasks analogous to humanoid running in MuJoCo, the proposed method has displayed commendable results in VPIL scenarios. However, the sheer volume of required experiments, given the numerous baselines and our methods across various \\( $\\alpha$ \\) levels in a single environment (and the need for multiple trials for credibility), precluded the inclusion of these tasks in the rebuttal timeframe. Nonetheless, we are keen on validating our methods on more challenging tasks in the future.\n"}, {"review_id": "UgWoWtvSOQ", "reviewer": "Reviewer_CNdS", "summary": "Edit: Changed the score from a 3 to a 6 due to the discussion below.\n\nThe paper focuses on tackling the challenge of Imitation Learning from imperfect demonstrations that come with vague pairwise comparisons. The authors make the key assumption that a competent annotator is available, but that they become uncertain when presented with either two expert or two non-expert samples, randomly assigning them as \"good\" or \"bad\" examples. To address this problem, they introduce a novel learning paradigm called Vaguely Pairwise Imitation Learning. By modeling the demonstration pool as a mixture of expert and non-expert data, the authors show that it is possible to recover the expert policy distribution when the proportion of expert data is known. Furthermore, they propose a method for estimating the mixture proportion in cases where this expert ratio is unknown.\n\nTo effectively leverage the recovered expert policy distribution, the authors integrate it with Generative Adversarial Imitation Learning (GAIL), resulting in end-to-end algorithms designed for scenarios with known and unknown expert data proportions. The performance of these algorithms is evaluated through experiments conducted on various Mujoco tasks. The results showcase that the proposed methods consistently outperform standard and preference-based imitation learning approaches for expert ratios of up to $0.5$.", "questions": "- Importance of Equation 8: Equation 8 rephrases the GAIL loss in terms of the estimated positive and negative data distributions to mitigate overfitting. How crucial is this rephrasing for the overall performance of the approach? Conducting an ablation study to assess the impact of this choice would provide valuable insights and clarify its significance.\n- Alternatives to Algorithm 1: Algorithm 1 includes the design choices of iteratively training a discriminator to approximately separate expert from non-expert data. What would happen if a mixture proportion estimator were directly applied to \\Gamma^+ and \\Gamma^- instead? Explaining the reasoning behind the chosen approach and comparing it to alternative strategies would enrich the discussion and highlight the advantages of the proposed algorithm.\n- Intuition Behind the Step from Equation 2 to Equation 3: Equations 2 and 3 explain the relations between the (non)-expert the pairwise dataset’s occupancy measures. It would be helpful to gain some intuition on what the transition from e.g., $p_{\\pi_+}$ to $p_{\\pi_E}$ actually means. A deeper explanation or examples illustrating this step would make this easier to understand.\n- Clarification on Expert Ratio Estimation: Lines 96-98 state that \"[…] [5] also used the risk rewriting technique to identify the expert demonstrations, but their algorithm does not require estimation of the expert ratio as the confidence score is given.\" For the presented approach, the expert ratio is also given in some of the experiments. How does the approach differ from this particular related work in this case?", "limitations": "The limitations or potential negative societal impact of the work are currently not addressed. It should be acknowledged that the method assumes a perfect annotator who randomly assigns samples when they come from the same distribution. However, in real-world scenarios, annotators may possess biases or preferences towards certain types of solutions, which are not accounted for in the proposed approach. Addressing and discussing this limitation would strengthen the submission.\n\nIn terms of ethical considerations, aspects such as fairness, transparency, and bias in the training and deployment of imitation learning models is crucial to ensure responsible and ethical AI development. It could be beneficial for the authors to include a section that highlights these potential ethical implications.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- Novel Approach for Vaguely Pairwise Imitation Learning: The paper proposes a novel approach to estimate precise expert occupancy measures from essentially imperfect preferences. This addresses the challenge of working with imperfect data and provides a valuable new approach for imitation learning tasks.\n- Sound Theory and Proofs: The paper demonstrates a solid theoretical foundation, supported by proofs in the appendix. The proposed algorithm is very closely related to the algorithm, using few additional assumptions or relaxations.\n- Strong Experimental Performance: The experimental evaluation of the proposed method showcases its superior performance compared to a large number of standard and preference-based imitation learning methods. The results presented in the paper demonstrate the efficacy and practical relevance of the approach across various Mujoco environments.\n\nOverall, the paper exhibits originality in its approach, a solid theoretical foundation for the presented algorithm, and significance in its experimental results. These strengths contribute to the advancement of the field of Imitation Learning and may offer valuable insights for both researchers and practitioners.", "weaknesses": "The following weaknesses are listed in order of severity.\n\n- Suspected Issue with Overestimation: The values of the overestimation Figure 3a, b, and Appendix Figure 1a-h, are very close to 1 - the true ratio. Additionally, the matrix plots in Figure 3c, d appear as negative transposes of each other. This raises suspicion and suggests a potential issue with the code implementation.\n- Inconsistent Experimental Results in Table 1: The experimental results presented in Table 1 exhibit inconsistencies. The HalfCheetah and Hopper experiments seem to use results from Swimmer and Walker2d for all methods, except for Behavioral Cloning and GAIL. Consequently, there are no results provided specifically for these tasks in the current state of the submission, which reduces the completeness of the evaluation.\n- Evaluation of GAIL_expert and Confidence Intervals: The evaluation of GAIL_expert in Table 1 is performed separately for different alpha values, despite its independence from alpha. Moreover, while GAIL_expert is presented as an upper bound for performance, the presented methods outperform it in several instances. This may indicate that the number of random seeds used (5) is insufficient, especially considering the relatively large confidence intervals.\n- Unclear Advantages of COMPILER-E: It is unclear why COMPILER-E would outperform COMPILER. Exploring potential reasons, for example that COMPILER-E may remove \"bad\" expert demonstrations through an implicit de-noising procedure in its estimation of the expert ratio could provide clarification and improve the understanding of its advantages.\n- Limited Experiment Scope: The experiments conducted in the paper are limited to Mujoco environments. Including experiments on other environments, such as Meta-world, would enhance the comprehensiveness of the evaluation.\n- Code Unavailability: The paper does not provide the code, which can hinder reproducibility and limit further exploration of the proposed methods.\n- Missing Hyperparameters: The paper lacks information about the hyperparameters used for some of the baseline methods. For example, for D-Rex the number of noise levels utilized to estimate the distributions during training is missing.\n- Proof Derivations Lack Readability: The derivations for the proofs are presented in a compact manner, making them difficult to follow. Providing additional steps and explanations would greatly improve the readability of the proofs.\n- Typos and Style:\n    - Line 133 references Section 6.4, which does not exist in the paper.\n    - The layout of the caption for Figure 3 makes it challenging to read and comprehend the information provided.\n    - Figure 2 could benefit from a descriptive caption to aid understanding and provide context for the information presented.\n    - There are instances of redundancy and language issues throughout the paper, such as the information in Line 163, which was already mentioned in the previous paragraph and e.g., the over-use of \"the\" in the first sentence of the description of Figure 1. Addressing these issues would improve the overall clarity and readability of the paper.\n    - Factorization in Equation 3: In Equation 3, the factor $2(1-\\alpha)$ is factored out only once, even though it appears twice in the equation. Ensuring consistent notation would enhance the equation's clarity.\n\nOverall, the suspected problems with the overestimation method for the expert ratio, and the missing results for $2$ out of $4$ tasks are a clear reason to reject the paper. Addressing these (and other) issues during the rebuttal period would significantly increase the quality of the submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the comprehensive feedback and deep dive into our work. We value the acknowledgment of our work's novelty, theoretical rigor, and experimental robustness. We have provided the correct results in the general response's PDF file and have shared the code and demonstrations from our experiments for reproducibility. We hope this assuages any concerns. For brevity, we will focus our response on addressing technical queries.\n### Q1:\n**Query:** \"Suspected Issue with Overestimation\"\n\n**Response:** We appreciate your meticulous examination of our empirical results. The phenomenon observed in Figure 3a,b is attributed to the inherent limitations of the overestimation method, particularly when $\\alpha$ is less than 0.5. During overestimation, as guided by Eq.(7), we discern the maximum proportion of $p_{\\pi_+}$ within $p_{\\pi_-}$ as $\\beta$ , subsequently computing $\\alpha = \\frac{2\\beta}{1+\\beta}$ . In scenarios where $\\alpha$ is diminutive, the estimation of $\\beta$ can be notably unreliable. This discrepancy is further elucidated with a detailed example: For example, in the case when $\\alpha = 0.1$, we have $p_{\\pi_+} \\approx 0. 2p_{\\pi_E} + 0.8p_{\\pi_{NE}} $ and $p_{\\pi_{-}} \\approx 0. 01p_{\\pi_E} + 0.99p_{\\pi_{NE}}$. Since the proportion of $\\pi_E$ in $p_{\\pi_+}$ is larger than that in $p_{\\pi_-}$, we expected the estimator to return the $\\beta \\approx 0. 01p_{\\pi_E} / 0. 2p_{\\pi_E} = 0.05$, which finally lead to the right estimation $\\alpha \\approx 0.1$. However, in practice, the mixture proportion will be highly affected by the data from $\\pi_{NE}$. Since both $p_{\\pi_+}$ and $p_{\\pi_-}$ contain a large proportion of $p_{\\pi_{NE}}$, the MPE estimator will take them as almost the same distribution and return the value $\\beta$ close to 1, thus leading to $\\alpha$ close to 1, which is in accordance with the results in Figure 3(a) and Figure 3(b). \n\nRegarding Figure 3(c and d), we apologize for the confusion. The heatmap values essentially represent theoretical computations rather than algorithmic outputs. These figures aim to elucidate the relationship between actual $\\alpha$ values and the estimated $ \\hat{\\alpha} $. We will clarify this in our subsequent revision.\n### Q2:\n**Query:** \"Inconsistent Experimental Results in Table 1\"\n\n**Response:** We are grateful for highlighting this oversight. The corrected results have been furnished in the general response's pdf. Furthermore, for the sake of reproducibility, we have made available the codes and all experimental demonstrations. We sincerely hope that these corrective actions address the concerns, and we pledge to rectify these errors in our revised version.\n### Q3: \n**Query:** \"The evaluation of GAIL_expert in Table 1 is performed separately for different alpha values... the number of random seeds used (5) is insufficient\"\n\n**Response:** It's crucial to note that GAIL_expert exclusively utilizes expert data extracted from mixed data. Consequently, with a diminishing $\\alpha$ , the volume of expert data may reduce, necessitating separate evaluations for distinct $\\alpha$ values in Table 1. Besides, five seeds is a customary selection in the IL domain [1,2,3]. Results are deemed acceptable as long as the method doesn't considerably surpass the expert's performance.\n### Q4:\n**Query:** \"It is unclear why COMPILER-E would outperform COMPILER.\"\n\n**Response:** We have included t-tests comparing COMPILER and COMPILER-E in the general response's pdf. It's evident that COMPILER-E's superiority over COMPILER is restricted to specific tasks and $\\alpha$ values. In a majority of cases, COMPILER outshines COMPILER-E. Thus, instances where COMPILER-E outperforms COMPILER are mere anomalies.\n### Q5:\n**Query:** \"Limited Experiment Scope: The experiments conducted in the paper are limited to Mujoco environments. Including experiments on other environments, such as Meta-world, would enhance the comprehensiveness of the evaluation.\"\n\n**Response:** We concur on the importance of method verification across diverse tasks. In the present submission stage, our primary objective is to demonstrate our algorithm's prowess in handling mixed demonstrations from both experts and non-experts within VPIL problems. Notably, Meta-world, driven by MuJoCo's physics engine, primarily serves Meta-Learning scenarios. Considering VPIL problems, we have showcased the efficacy of COMPILER and COMPILER-E across various continuous control tasks.\n### Q6: \n**Query:** \"How crucial is this rephrasing for the overall performance of the approach?\"\n\n**Response:** The reframing of the GAIL loss in Eq. 8 is pivotal for our algorithm's success in VPIL problems. In the conventional IL paradigm, policies can be optimized by minimizing the original GAIL loss based on the expert policy $p_{\\pi_{E}}$ . However, in VPIL problems, the expert policy remains elusive due to the inherent ambiguity of feedback, thereby complicating the evaluation of the original GAIL loss. The equivalence between the original GAIL loss and the rephrased version (Eq. 8) allows us to evaluate the original loss using the vague datasets $\\Gamma^+$ and $\\Gamma^-$ .\n### Q7:\n**Query:** \"It would be helpful to gain some intuition on what the transition from e.g., $p_{\\pi_+}$ to $p_{\\pi_E}$ actually means.\"\n\n**Response:** Eq. 2 can be interpreted as a linear system with two variables. Within our framework, both $p_{\\pi_E}$ and $p_{\\pi_{NE}}$ are the elusive variables we aim to decipher, while all other metrics, including p_{\\pi_+} and p_{\\pi_-} , are empirically estimable. Thus, the transition from Eq. 2 to Eq. 3 mirrors the linear system resolution process. Through this mechanism, we can represent $p_{\\pi_E}$ and $p_{\\pi_{NE}}$ as linear combinations of $p_{\\pi_+}$ and $p_{\\pi_-}$ .\n\n## Reference\n[1] Learning to Weight Imperfect Demonstrations. ICML 2021.\n\n[2] Learning from Suboptimal Demonstration via Self-Supervised Reward Regression. CoRL 2020.\n\n[3] Imitation Learning from Imperfect Demonstration. ICML 2019.\n"}]

Author rebuttal:
We are profoundly grateful to all the reviewers for their constructive comments and suggestions. We will address each technical concern or misunderstanding raised by individual reviewers in their respective rebuttal sections. In this general response, we would like to highlight the following updates:

- **PDF file**: An updated version of the HalCheetah and Hopper results has been uploaded to address the queries from Reviewers DWbY and CNdS. We have also included t-tests between COMPILER and COMPILER-E to further address concerns from Reviewer CNdS.
  
- **Codes and Demonstrations**: The codes for our work are available at this anonymized link: [https://anonymous.4open.science/r/COMPILER-7B6A](https://anonymous.4open.science/r/COMPILER-7B6A). Additionally, the demonstrations used in our experiments can be accessed at: [https://drive.google.com/file/d/1-LWep6U_FHdEWuEJxDaluQXBQWGC086L/view?usp=sharing](https://drive.google.com/file/d/1-LWep6U_FHdEWuEJxDaluQXBQWGC086L/view?usp=sharing).
  
- **Discussion on Limitations**: We have discussed the limitations of our work in lines 135-136, which address the omission of noise during data collection. Experiments involving noisy $\alpha$ have been included in the Appendix. Furthermore, as mentioned in lines 343-344, we considered a single annotator in VPIL problems. However, real-world scenarios might involve multiple annotators with varying preferences. Based on the reviewers' feedback, we will incorporate a dedicated section to discuss our work's limitations more comprehensively.


Meta-review: The paper proposes an approach that allows imitation learning from mixed expert and non-expert demonstrations. The method allows to disentangle the dataset both when the percentage is known and when it needs to be estimated. The approach is studied with extensive experiments and comparisons against the sota.

The reviewers found the method innovative and sound. The theoretical and experimental analysis are nicely done. The main concerns were about missing details, clarifying the contributions and limitations, inconsistencies/unexplained effects in the results, and some unclear theoretical implications.

The rebuttal managed to overcome all (major) objections of the reviewers. Some reviewers did not react to the rebuttal, but they were already positive about the paper and their concerns have been nicely addressed. Providing the anonymized code was highly appreciated.

All reviewers are now in favor accepting the paper.
Decision: Accept (poster)