Title: Imitation Learning from Vague Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: imitation learning

vague feedback

risk rewriting

mixture propotion estimation
Abstract: Imitation learning from human feedback studies how to train well-performed imitation agents with an annotator's relative comparison of two demonstrations (one demonstration is better/worse than the other), which is usually easier to collect than the perfect expert data required by traditional imitation learning. However, in many real-world applications, it is still expensive or even impossible to provide a clear pairwise comparison between two demonstrations with similar quality. This motivates us to study the problem of imitation learning with vague feedback, where the data annotator can only distinguish the paired demonstrations correctly when their quality differs significantly, i.e., one from the expert and another from the non-expert. By modeling the underlying demonstration pool as a mixture of expert and non-expert data, we show that the expert policy distribution can be recovered when the proportion $\alpha$ of expert data is known. We also propose a mixture proportion estimation method for the unknown $\alpha$ case. Then, we integrate the recovered expert policy distribution with generative adversarial imitation learning to form an end-to-end algorithm. Experiments show that our methods outperform standard and preference-based imitation learning methods on various tasks.

Imitation Learning from Vague Feedback

Abstract

Imitation learning from human feedback studies how to train well-performed imitation agents with an annotator's relative comparison of two demonstrations (one demonstration is better/worse than the other), which is usually easier to collect than the perfect expert data required by traditional imitation learning. However, in many real-world applications, it is still expensive or even impossible to provide a clear pairwise comparison between two demonstrations with similar quality. This motivates us to study the problem of imitation learning with vague feedback, where the data annotator can only distinguish the paired demonstrations correctly when their quality differs significantly, i.e., one from the expert and another from the nonexpert. By modeling the underlying demonstration pool as a mixture of expert and non-expert data, we show that the expert policy distribution can be recovered when the proportion α of expert data is known. We also propose a mixture proportion estimation method for the unknown α case. Then, we integrate the recovered expert policy distribution with generative adversarial imitation learning to form an end-to-end algorithm 1 . Experiments show that our methods outperform standard and preference-based imitation learning methods on various tasks.

1 Introduction

Imitation learning (IL) is a popular approach for training agents to perform tasks by learning from expert demonstrations [1-3]. It has been applied successfully to a variety of tasks, including robot control [1], autonomous driving [4], and game playing [5]. However, traditional IL methods struggle when presented with both expert and non-expert demonstrations, as the agents may learn incorrect behaviors from the non-expert demonstrations [6]. This problem, which researchers refer to as 'Imitation Learning from Imperfect Demonstration' (ILfID), arises when the demonstrations used to train the model may contain some non-expert data [6-9].

One popular solution to ILfID is resorting to an oracle to provide specific information, such as explicit labels (confidence from an expert) of each demonstration, as in Figure 1a. However, such a specific oracle is quite expensive. A more recent framework, Reinforcement Learning from Human Feedback (RLHF) [10, 11], incorporates human feedback into the learning process, including a key part of the pipeline used for training ChatGPT [12]. There exist two widely studied paradigms of feedback: full ranking and global binary comparison, as in Figures 1b and 1c. Methods that use full-ranked demonstrations assumed that the feedback between every pair of trajectories is available, and further employed preference-based methods to solve the problem [13, 14]. Other studies have investigated situations where demonstrations can be divided into two global binary datasets, allowing the learner to filter out non-expert data from these global comparisons [7, 8]. However, data processed by any feedback of Figures 1a, 1b, and 1c require the guarantee that clear ranking information or at least one set consisting of pure expert demonstrations exists, so that off-the-shall IL methods can be applied immediately. The availability of expert demonstrations raises the question of what if, in practice, the feedback is not clear enough to provide purely expert demonstrations but mixed with non-expert demonstrations? The question poses a challenge that previous methods fail to address since none of

(a) Explicit label

(b) Full ranking

(c) Global (clear) comparison

(d) Vague comparison (our setting)

(e) A maze example

the previous IL algorithms handles the issue of the mixture of expert and non-expert demonstrations without other information. Their conventional strategies are to deal with the vagueness in the data processing stage (relying on human feedback) not during the learning stage (by the learning agent).

Therefore, we consider a kind of weaker feedback than those previously formulated in the literature, which unintentionally mixes expert and non-expert demonstrations due to vague feedback. Specifically, the human annotator demonstrates in a pairwise way and only distinguishes the expert demonstration from the non-expert one. However, the annotator cannot tell the origin when both are from an expert or a non-expert. As depicted in Figure 1d, this annotation process results in two datasets: Γ + , containing demonstrations the annotator believes to be most likely from experts, and Γ -, containing non-expert-like demonstrations. If one only places the distinguishable demonstrations into the pools and discards the indistinguishable ones, it results in Figure 1c. Since we aim to investigate the potential of IL under the presence of non-expert demonstrations, we distribute the indistinguishable demonstrations randomly, one to Γ + and the other to Γ -. This step embodies the vagueness we wish to study in this paper. Note that either dataset contains non-expert demonstrations, and thus, a direct application of the existing IL method might be inappropriate due to the danger of learning from undesirable demonstrations.

Vague feedback commonly exists in many scenarios, especially regarding RLHF [12]. In a crowdsourcing example of routes navigation, as shown in Figure 1e, most of the time crowd-workers may lack domain expertise. Meanwhile, it is quite difficult to distinguish every pair of demonstrations (when facing two trajectories from the same source, such as ( , ) or ( , )). Therefore, we cannot obtain high-quality crowd-sourcing labels when the data are annotated [15] as in Figures 1a, 1b, and 1c. On the other hand, it is natural for workers to provide vague comparisons as in Figure 1d.

In this work, we formulate the problem of Vaguely Pairwise Imitation Learning (VPIL) , in which the human annotator can only distinguish the paired demonstrations correctly when their quality differs significantly. In Section 4, we analyze two situations within this learning problem: VPIL with known expert ratio α and unknown α . For VPIL with known α , we provide rigorous mathematical analysis and show that the expert policy distribution can be empirically estimated with the datasets Γ + and Γ -; for the more challenging situation of VPIL with unknown α , we propose a reduction of the problem of estimating α to a mixture proportion estimation problem [16] and develop an iterative update procedure to estimate α . In Section 5, we integrate our algorithm with an off-the-shelf IL method to solve VPIL problems. In Section 6, we evaluate our methods with state-of-the-art ILfID methods on a variety of tasks of MuJoCo [17] with different α ratios and find that our methods obtained the best performance.

2 Related Work

In imitation learning scenarios, a model is trained to mimic the actions of expert demonstrations. One of the main challenges of imitation learning is that the gathered demonstrations can be imprecise, making it difficult for the learner to accurately replicate the underlying policy of the demonstrations. To overcome this issue, various works have utilized an annotator, a source of supplementary supervi-

sion that can aid the learner in better understanding the expert's intent. For example, [18] used explicit action signals of the annotator; [14], [19], and [13] used ranking/preference information from the annotator to learn the policy; [6] utilized confidence information from the annotator to re-weight the unlabeled demonstrations and further learned the policy. However, as illustrated in Section 1, in some cases, the annotator may not be able to provide explicit information for the imperfect demonstrations, especially when the demonstrations come from different sources.

Alternative strategies for IL by imposing prior knowledge instead of an annotator have also been put forward, such as state density estimation for importance weighting [20], state-action extrapolating [21], and adding noise to recover ranking [22]. But this research line relies on certain assumptions about the state/state-action spaces, which may not always hold in many real-world applications. In this work, we focus on using an annotator with vague information, which is also low-cost.

Our methods drew inspiration from the risk rewriting technique in weakly supervised learning literature [23], which aims to establish an unbiased risk estimator for evaluating the model's quality only with weak supervision. Although the technique has been successfully applied to various specific weakly supervised learning problems [24-27], these methods typically require knowledge of a parameter called the mixture proportion, i.e., the expert ratio in our problem [28], the estimation of which can be challenging in the imitation learning problem (see Section 6.3 for more details). To overcome this issue, we introduce an iterative data selection procedure to better estimate the expert ratio, leading to superior empirical performance. It is worth noting that [6] also used the risk rewriting technique to identify the expert demonstrations, but their algorithm does not require estimation of the expert ratio as the confidence score is given. However, such information is unavailable in VPIL, making the problem more challenging.

3 Preliminaries and Problem Setting

In this section, we first introduce the IL process. Then we formulate the VPIL problem.

3.1 Preliminaries

Markov Decision Process. In policy learning problems, a Markov Decision Process (MDP) can be represented by a tuple ⟨S , A , P , γ, r, T ⟩ , where S is the state space; A is the action space; P : S × A × S → [0 , 1] is the transition probability distribution; γ is a discount factor in the range (0 , 1] ; r : S → R is the reward function; and T is the horizon. The goal is to learn a policy π that maximizes the expected returns E [ ∑ ∞ t =0 γ t r ( s t , a t )] , where E denotes the expectation.

In IL, the learner does not have access to the reward function r . Instead, the learner is given m expert demonstrations τ E , 1 , τ E , 2 , . . . , τ E ,m , where τ E ,i , i ∈ { 1 , . . . , m } is the i -th trajectory (a series of state-action pairs) drawn independently following the demonstrator's policy π E . The goal of the learner is to learn a policy π θ to mimic π E .

Occupancy Measure. Since in IL we do not have reward functions, we need to measure the policy performance in the state-action space, i.e., occupancy measure ρ π . The occupancy measure ρ π : S × A ↦→ R was introduced to characterize the distribution of state-action pairs generated by a given policy π , which was defined with the discount factor: ρ π ( s, a ) = π ( a | s ) ∑ ∞ t =1 γ t Pr[ s t = s | π ] , where Pr[ s t = s | π ] is the probability of reaching state s at time t following policy π [29].

Moreover, we know that there is a one-to-one correspondence between the occupancy measure and the policy. Specifically, we have the following theorem.

Theorem 1 (Theorem 2 of [29]) . Suppose ρ is the occupancy measure for π ρ ( a | s ) := ρ ( s,a ) ∑ a ′ ∈A ρ ( s,a ′ ) . Then, π ρ is the only policy whose occupancy measure is ρ .

We can also define a normalized version of the occupancy measure by p π ( s, a ) ≜ ρ π ( s,a ) ∑ s ′ ∈S ,a ′ ∈A ρ π ( s ′ ,a ′ ) = (1 -γ ) ρ π ( s, a ) . If we have m expert demonstrations τ E , 1 , τ E , 2 , . . . , τ E ,m , the occupancy measure p π E can be empirically estimated by the trajectories as ̂ p π E ( s, a ) = 1 -γ m m ∑ i =1 ∞ ∑ t =0 γ t ✶ [ ( s ( i ) t , a ( i ) t ) = ( s, a ) ] , where ✶ [ · ] is an indicator function that returns 1 if · is true and returns 0 otherwise. Finding out the underlying p π E is the key for solving IL problems [1, 30].

3.2 Vaguely Pairwise Imitation Learning

In this work, we focus on the ILfID problem with pairwise information, where the learner aims to learn a good policy with a pairwise dataset (Γ + , Γ -) , generated from a mixture demonstration pool performed by an expert policy π E and a set of non-expert policies { π ( k ) NE } K k =1 . In this work, we consider the mixed occupancy measure of the non-expert policy set as p π NE . The proportion of the expert data within the pool is denoted as α ∈ (0 , 1] . The data collection process is shown in Figure 2. To clearly reveal the effect of mixed demonstrations and explain the proposed framework, we choose to assume that the data collector is not an attacker and will make mistakes, so there will be no noise during the collection process. In Section 6.4, we discuss how our method can easily be extended to handle the presence of human error.

Also, in this work, we do not consider that the data collector could be an attacker or make mistakes, so there will be no noise during the collection process.

Collection of Pairwise Datasets. The data collector would first sample a pair of trajectories ( τ i , τ j ) independently from the mixture pool. If ( τ i , τ j ) are from different sources, i.e., one is from the expert, and another is not, then the collector will take the expert one τ i into Γ + and the non-expert one τ j into Γ -. Otherwise, the collector randomly puts them into Γ + and Γ -. Under such a data generation process, the expertise probabilities of this pair are as follows:

In such a case, Γ + is always 'not worse' than Γ -as it contains more expert data.

4 Learning frameworks for Solving VPIL Problems

In this section, we analyze the core challenge in VPIL problems with the pairwise datasets Γ + and Γ -, i.e., recovering the occupancy measure of the expert policy p π E . To this end, we propose two learning frameworks for VPIL with known α and unknown α , respectively.

4.1 VPIL with Known α

Most of the IL methods based on the occupancy measure need to assume that the demonstrations are sampled only from the expert policy, so that they can directly estimate p π E and match the learner's distribution with p π E . However, the occupancy measure of the expert policy p π E is inaccessible in the VPIL problem, since both Γ + and Γ -contain non-expert data while the label of each data is unavailable. Below, we attempt to approximate p π E with { Γ + , Γ -} .

Let n + = | Γ + | ; ̂ p + i ( s, a ) = (1 -γ ) ∑ ∞ t =0 γ t ✶ [( s t,i , a t,i ) = ( s, a )] for i = 1 , . . . , n + be the empirical occupancy measure of a trajectory τ i ∈ Γ + , where ( s t,i , a t,i ) ∈ τ i is a state action pair at time t . Let ̂ p π + ( s, a ) = ∑ n + i =1 ̂ p + i ( s, a ) /n + . Define ̂ p π -( s, a ) similarly. We have the following theorem.

Theorem 2. Assume the pairwise datasets (Γ + , Γ -) are generated following the procedure in Section 3.2. Let p π + ( s, a ) = E Γ + [ ̂ p π + ( s, a ) ] and p π -( s, a ) = E Γ -[ ̂ p π -( s, a ) ] be the expected occupancy measures of Γ + and Γ -, where the randomness is taken over the draws of Γ + and Γ -. Then, we have

A proof can be found in Appendix A. The Theorem provides a feasible way to recover the unknown occupancy measure of the expert policy from contaminated data pools Γ + and Γ -. Thus, we can use an off-the-shelf IL method to learn the policy.

Once we have recovered p π E by Eq. (10), we can use an off-the-shelf IL method to learn the policy.

4.2 VPIL with Unknown α

When facing a more challenging problem in which the expert ratio α is unknown, a straightforward way is to estimate the ratio first and then reconstruct the expert policy by the approach developed for known α cases. Here, we will first introduce how to estimate the expert ratio by reducing it into a mixture proportion estimation (MPE) problem [28] and then identify that a direct application of the estimated ratio is not accurate enough to reconstruct the expert policy. To this end, we further propose an iterative sample selection procedure to exploit the estimated expert ratio, which finally leads to a better approximation of the expert policy.

Estimation of the Expert Ratio α . In this paragraph, we show that the estimation of the expert ratio α can be reduced to two MPE problems [28]. Let P and N be two probability distributions, and

be a mixture of them with a certain proportion β ∈ (0 , 1) . The MPE problem studies how to estimate mixture proportion β with the empirical observations sampled from U and P (not from N ). Over the decades, various algorithms were proposed with sound theoretical and empirical studies [31, 32].

To see how the estimation of the expert ratio α is related to the MPE problem, we rewrite (2) as

where β 1 = 1 -α 1+ α and β 2 = α 2 -α . By taking p π + as U and p π -as P , the first line of (5) shares the same formulation as the MPE problem (4). Since p π + and p π -are empirically accessible via Γ + and Γ + , we can estimate β 1 by taking Γ + and Γ -as the input of any MPE solver and obtain the expert ratio by α = 1 -β 1 1+ β 1 . The same argument also holds for the second line of (5), where we can take p π -as U and p π + as P to estimate β . Then, the expert ratio can also be obtained by α = 2 β 2 1+ β 2 .

One might worry about the identifiability of the expert ratio α by estimating it with MPE techniques [28]. We can show that the true parameter is identifiable if the distribution p π + and distribution p π -are mutually irreducible [16] as follows:

Proposition 1. Suppose the distributions p π E and p π NE are mutually irreducible such that there exists no decomposition of the form p π E = (1 -η ) Q + ηp π NE and p π NE = (1 -η ′ ) Q ′ + η ′ p π E for any probability distributions Q,Q ′ and scalars η, η ′ ∈ (0 , 1] . Then, the true mixture proportions β 1 and β 2 are unique and can be identified by

where C is the set containing all possible probability distributions. Thus, α is identifiable by

Proposition 1 demonstrates that the true mixture proportion β 1 (resp. β 2 ) can be identified by finding the maximum proportion of p π -contained in p π + (resp. p π + contained in p π -). This idea can be empirically implemented via existing MPE methods [28, 33]. We note that the expert ratio is identifiable by either estimating β 1 or β 2 when we have infinite samples. In the finite sample case, the two estimators coming from different distribution components could lead to different estimation biases. Since the MPE solutions tend to have a positive estimation bias on the true value β 1 and β 2 as shown by [28, Theorem 12] and [31, Corollary 1], the estimation α = (1 -β 1 ) / (1 + β 1 ) tends to yield an underestimated α while that with β 2 will lead to an overestimation. Besides, when the number of expert data is quite small, the underlying true parameter β 2 would also have a small value,

and its estimation would be highly unstable. Thus, we choose to estimate α with β 1 in our algorithm. The empirical studies in Section 6.3 also supported our choice.

The relationship (2) between the expert and non-expert data shares a similar formulation to that of the unlabeled-unlabeled (UU) classification [34], which is a specific kind of weakly supervised learning problem studying how to train a classifier from two unlabeled datasets and requires the knowledge of mixture proportions. However, how to estimate these mixture proportions is still an open problem in the weakly supervised learning literature. Although our reduction is developed for estimating the expert ratio for IL problems, it can be applied to a UU learning setting for independent interest.

Reconstruct Expert Policy by Data Selection. To handle the MPE task for high dimensional data, a practice is to first train a classifier with probability output and then conduct the MPE on the probability outputs of samples [35]. However, the quality of the trained classifier turns out to depend on the estimation accuracy of α . On the other hand, we found that it is possible to filter out the undesired component of the pairwise datasets ( ̂ p π E in Γ -and ̂ p π NE in Γ + ) and keep the desired data to enhance estimating α . Therefore, also inspired by [31, 36] of the distribution shift problem and weakly supervised learning, we propose an iteration-based learning framework. In each iteration, we throw away the non-expert data with higher confidence in Γ + and the expert data with higher confidence in Γ -after estimating α , and train a classifier with the datasets after selection. The detailed learning process can be found in Algorithm 1.

After we obtained the estimated ̂ α , we can recover p π E as by Eq. (10).

5 COMParative Imitation LEarning with Risk rewriting (COMPILER)

We have illustrated how to estimate the occupancy measure of the expert policy p π E from (Γ + , Γ -) under known and unknown α respectively. Here we describe how to adopt these learning frameworks into end-to-end algorithms for learning an policy. We name our algorithm for VPIL with known α COMParative Imitation LEarning with Risk-rewriting (COMPILER) , and with unknown α COMParative Imitation LEarning with Risk-rewriting by Estimation (COMPILER-E) .

Current state-of-the-art adversarial IL methods [37, 1] aim to learn a policy by matching the occupancy measure between the learner and the expert policies. We fuse our methods with one of the representative adversarial IL methods, Generative Adversarial Imitation Learning (GAIL) [1], whose optimization problem is given as follows:

where D w : S × A ↦→ [0 , 1] parameterized by w is a discriminator. p π θ parameterized by θ is the occupancy measure of a policy. In our setting, the expert demonstration of p π E is unavailable.

COMPILER. As suggested by Theorem 2, we can recover p π E with the occupancy measures of Γ + and Γ -. Then, we can train the policy p π θ by mincing p π E = 1+ α 2 α p π + -1 -α 2 α p π -in Eq. (10). Specifically, plugging (2) into (8) and approximating p π + and p π -with their empirical versions, we can obtain the desirable target. However, such rewriting can introduce a negative term and lead to overfitting [38, 39]. To avoid such an undesired phenomenon, instead of training the policy π θ to make p π θ match 1+ α 2 α ̂ p π + -1 -α 2 α ̂ p π -, we twist the objective function of GAIL (8) to minimize the discrepancy between 2 α 1+ α p π θ + 1 -α 1+ α ̂ p π -and ̂ p π + , which gives the following optimization problem without the negative term:

Let ̂ V ( π θ , D w ) be the objective function in (8) and V ( π θ , D w ) be its expectation established on p π + and p π -. We have the following theorem for the estimation error of the discriminator trained by (8).

Theorem 3. Let W be a parameter space for training the discriminator and D W = { D w | w ∈ W} be the hypothesis space. Assume the functions | log D w ( s, a ) | ≤ B and | log(1 -D w ( s, a )) | ≤ B are upper-bounded for any state-action pair ( s, a ) ∈ S × A and w ∈ W . Further assume both the functions log D w ( s, a ) and log(1 -D w ( s, a )) are L -Lipschitz continuous in the state-action space. For a fixed policy π θ , let Γ θ = { τ θ i } n θ i =1 be trajectories generated from π θ . Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

where ̂ D w = arg max w ∈W ̂ V ( π θ , D w ) and D ∗ w = arg max w ∈W V ( π θ , D w ) . The constants n + and n -are the number of trajectories in Γ + and Γ -. We define C ( δ ) = 4 B √ log(6 /δ ) . The empirical Radamacher complexities [40] on datasets Γ + , Γ -, and Γ θ are denoted by R n + , R n -, and R n θ .

A proof is given in Appendix A. The theorem shows that the discriminator trained by (8) converges to the one optimized with the true distribution p π + and p π -at each step of the training.

COMPILER-E. For the VPIL problem with unknown α , first we estimate ̂ α by Algorithm 1, then we learn the policy by (8) with ̂ α .

We integrate the two algorithmic processes COMPILER and COMPILER-E into Algorithm 2.

6 Experiments

In this section, we conducted extensive experiments under the setting of VPIL. Through the experiments, we want to investigate the following questions: (1) Can COMPILER and COMPILER-E solve VPIL problems under various expert ratios in demonstrations? (2) Is COMPILER-E still valid when using different MPE estimators to obtain ̂ α ? (3) How is the α estimation with β 1 and β 2 as in (7)?

6.1 Setup

To investigate the answer to the first question, we chose 20 different VPIL tasks with four MuJoCo benchmark environments to evaluate the performance of the contenders and our approaches under five different α levels. The detailed setups are reported as follows.

Environments and Demonstrations. We set HalfCheetah, Hopper, Swimmer , and Walker2d in MuJoCo as the basic environments. For each experiment, the demonstration pool contains 100 trajectories with different expert ratios α = { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } , in which α = 0 . 1 is the most difficult situation of the VPIL problem. We also trained an expert-level RL agent and two non-expert RL agents as demonstrators by DDPG algorithm [41]. The details of the environment and the demonstrations can be found in Table 1.

Dataset generation process. We start with the demonstration pool { τ i } N i =1 , which contains αN trajectories sampled from the optimal policy π E and (1 -α ) N trajectories from the non-optimal policy π NE . For notation simplicity, we denote by D + the part containing expert demonstrations only and by D -the part for the non-expert demonstrations. Then, we can generate the dataset as follows,

where sample( D , N ) is the function that samples N trajectories from the dataset D .

Contenders. Since Γ + contains more expert demonstrations, here we use Behavior Cloning [42], GAIL [1], and AIRL [43] with Γ + only as basic baselines. Also, we provide GAIL with expert-level demonstrations (GAIL_expert) as the skyline of all methods. Besides, we set T-REX [13], a state-ofthe-art preference-based IL method, by taking that every data in Γ + is more expert-like than that in Γ -as a form of preference to train its reward function. We also set the variants of T-REX algorithms, D-REX [22] and SSRR [44], that directly generate the ranking information through the imperfect datasets. CAIL proposed a confidence-based method to imitate from mixed demonstrations [14]. In the experiment, we provide 5% trajectory labels to meet its requirement as suggested in their paper. Each method ran 4e7 steps. 5 trials were conducted with different seeds for each task.

Meanwhile, to investigate the answer to the second question, we conducted Best Bin Estimation (BBE) [31] and Kernel Mean (KM) [28] algorithm to estimate β in algorithm 1 of COMPILER-E, as COMPILER-E (BBE) and COMPILER-E (KM) respectively. We note that COMPILER-E cannot obtain the true α during the whole experiment.

Implementation of α estimation. We implement a four-layer fully connected neural network as the binary classifier f ψ in Algorithm 1. The neurons in each layer are 1000, 1000, 100, and 50 respectively. The activation function of each layer is ReLU, and the output value will go through the sigmoid function to obtain the score for Γ + and Γ -as S + and S -respectively. The optimizer for f ψ is stochastic gradient descent. We totally train f ψ for 1000 epochs with bath size 1024. The initial learning rate is 5e-4, with an exponential decay rate of 0.99 at each step.

Implementation of policy training. We choose Proximal Policy Optimization (PPO) [45] as the basic RL algorithm, and set all hyper-parameters, update frequency, and network architectures of the policy part the same as [46]. Besides, the hyper-parameters of the discriminator for all methods were the same: The discriminator was updated using Adam with a decayed learning rate of 3 × 10 -4 ; the batch size was 256. The ratio of update frequency between the learner and discriminator was 3: 1. For T-REX algorithm [13], we use the codes with default hyperparameters and model architecture of their official implementation.

6.2 Empirical Results

The final results are gathered in Table 2. We can see that since AIRL and GAIL are standard IL algorithms, they cannot adaptively filter out non-expert data, so the underlying non-expert data reduces the performance of them. T-REX did not obtain promising results on VPIL tasks, which

verifies that vague pairwise comparison can not be considered as full-ranking information, so the preference-based method cannot be used to tackle such problems. In addition, both SSRR and D-REX are updated versions of T-REX using noise to generate ranking information. They still cannot achieve good results due to its inability to address the core challenges of the VPIL problem. This phenomenon further demonstrates that preference-based algorithms cannot solve VPIL problems without explicit preference information, and some assumptions may not be held in VPIL problems. As α grows, we can see that the performances of almost all the contenders increased more or less, except for the skyline GAIL_expert, while COMPILER and COMPILER-E got the biggest performance boost and achieved the best performances. Even under the α = 0 . 1 situation, COMPILER and COMPILER-E have achieved expert-level performance on Hopper and Walker2d environments. For CAIL, it shows competitive performance with COMPILER and COMPILER-E under high α situations, but still falls short in low α cases. The reason for this phenomenon is that when α is relatively low, the trajectory quality corresponding to the provided trajectory labels may also be poor. Under such circumstances, estimating confidence is extremely difficult. As a result, CAIL generally performs suboptimally in low α cases. On the other hand, our algorithms can achieve comparable results with CAIL in high alpha scenarios. This indicates that our methods reasonably utilize the structure of the VPIL problem and can achieve results on par with the state-of-the-art algorithm without introducing additional information. Meanwhile, COMPILER (BBE) and COMPILER (KM) both achieved promising and comparable performances, which indicates that COMPILER-E is a general algorithm and can be operated with any effective MPE method.

6.3 Case Study: Estimation Effect on Data Selection

Overestimation and underestimation comparisons. To investigate if the superiority of COMPILER-E comes from precisely estimating α , and also to answer the third question about using two different estimations in (7) in Section 6, we reported the estimation curves under different true α on HalfCheetah with BBE and KM estimators, as shown in Figure 3. We can see that the overestimation method obtained ̂ α that far exceeded the true α , especially when α was smaller; while the underestimation method was very accurate despite obtaining a slightly lower ̂ α value compared to the true α , for both BBE and KM estimators. The results also suggested that we indeed need to use the underestimation method for COMPILER-E when the expert ratio α is small. It also reflects that the power of COMPILER-E for solving VPIL problems indeed comes from accurately estimating the α value. More environmental results can be found in Appendix B.

Thrown data effect. To further investigate what caused the performance gap between the overestimation and underestimation, we analyze the effect on the proportion of data selections in Algorithm 1. Also connected to the expert and non-expert proportions of Γ + and Γ -in (2), we can calculate the theoretical value of the ratio differences of thrown data under different estimations with that under true α , as shown in Figure 3.

We can see that the overestimated method threw away more data in Γ -and fewer data in Γ + (corresponding to the upper left triangles of the red boxes in the figures); meanwhile, the underestimated method did the opposite (corresponding to the lower right triangles). However, as analyzed in Section 4.2, the overestimated method relied heavily on p π E of Γ -dataset, whose components, were quite small. In the case of finite data, once p π E of Γ -were overthrown, the remaining part of p π E became less, making the estimation further high, which led to a vicious circle. The phenomenon was especially severe in the case of low α . This is the reason why ̂ α of the overestimated method became

̂

̂

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

Underestimation(Ours)

Overestimation

True expert ratio

α

(a)

BBE

estimator

on

HalfCheetah

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(b)

KM

estimator

on

HalfCheetah

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

true expert's ratio

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

estimated expert's ratio

-0.8 -0.63-0.48-0.35-0.24-0.15-0.08-0.03

0

-0.77 -0.6 -0.45-0.32-0.21-0.12-0.05

0

0.03

-0.72-0.55 -0.4 -0.27-0.16-0.07

0

0.05 0.08

-0.65-0.48-0.33 -0.2 -0.09

0

0.07 0.12 0.15

-0.56-0.39-0.24-0.11

0

0.09 0.16 0.21 0.24

-0.45-0.28-0.13

0

0.11

0.2

0.27 0.32 0.35

-0.32-0.15

0

0.13 0.24 0.33

0.4

0.45 0.48

-0.17

0

0.15 0.28 0.39 0.48 0.55

0.6

0.63

0

0.17 0.32 0.45 0.56 0.65 0.72 0.77

0.8

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

(c) Thrown data ratio differ-

ence on

Γ

+

dataset

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

true expert's ratio

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

estimated expert's ratio

0.8

0.77 0.72 0.65 0.56 0.45 0.32 0.17

0

0.63

0.6

0.55 0.48 0.39 0.28 0.15

0

-0.17

0.48 0.45

0.4

0.33 0.24 0.13

0

-0.15-0.32

0.35 0.32 0.27

0.2

0.11

0

-0.13-0.28-0.45

0.24 0.21 0.16 0.09

0

-0.11-0.24-0.39-0.56

0.15 0.12 0.07

0

-0.09 -0.2 -0.33-0.48-0.65

0.08 0.05

0

-0.07-0.16-0.27 -0.4 -0.55-0.72

0.03

0

-0.05-0.12-0.21-0.32-0.45 -0.6 -0.77

0

-0.03-0.08-0.15-0.24-0.35-0.48-0.63 -0.8

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

(d) Thrown data ratio differ-

ence on

Γ

-

dataset

Figure 3: (a, b) The comparisons of overestimated ( 2 β 1+ β ) and underestimated ( 1 -β 1+ β ) methods on various tasks with BBE and KM estimators. (c,d) The distribution map of thrown data ratio using estimated ̂ α and true α . The thrown data ratio difference on Γ + is (1 -̂ α ) 2 -(1 -α ) 2 , while that on Γ -is ̂ α 2 -α 2 . The range marked by the red box is considered in our experiments, in which the expert ratio is smaller than (or equal to) the non-expert one.

bigger as true α decreased. On the other hand, although the underestimated method also overthrew the non-expert data in Γ + , the proportion of these data was relatively high. So even if more were thrown away at the beginning, it will not affect the accuracy of the estimation. This is the reason why using 1 -β 1+ β to estimate ̂ α is relatively stable and accurate.

6.4 Case Study: Assessing the Robustness of the COMPILER Algorithm

To evaluate the robustness of the COMPILER algorithm, a comprehensive set of experiments were conducted. These were carried out in four distinct environments, implementing tasks with α = 0 . 5 . Each experiment was repeated five times to ensure the reliability of the results. To emulate real-world conditions, different levels of noise were introduced to the parameter α , resulting in the generation of corresponding noisy datasets. This noise was denoted as ε , thereby resulting in ̂ α = α + ε .

Figure 4 presents the findings from these experiments. Remarkably, the COMPILER algorithm demonstrated consistent performance, achieving at least 96% of the standard performance even in the presence of various noise levels. This robustness, as exhibited in the results, underlines the capacity of COMPILER to effectively handle and perform under noisy conditions, specifically when the provided α is subject to disturbances. This emphasizes not only the resilience of the algorithm but also its potential for deployment in real-world situations where data is seldom perfect.

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

Noise Level

0.92

0.94

0.96

0.98

1.00

1.02

Normalized Episodic Returns

(a) Hopper

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

0.94

0.96

0.98

1.00

1.02

Normalized Episodic Returns

Noise Level

(b) Walker2d

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

0.975

0.980

0.985

0.990

0.995

1.000

1.005

Normalized Episodic Returns

Noise Level

(c) HalfCheetah

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

0.94

0.96

0.98

1.00

1.02

1.04

Normalized Episodic Returns

Noise Level

(d) Swimmer

7 Conclusion

In this work, we formulated the problem of Vaguely Pairwise Imitation Learning (VPIL) , in which mixed expert and non-expert demonstrations are present, and the data collector only provides vague pairwise information of demonstrations. To solve this problem, we proposed two learning paradigms, with risk rewriting and mixture proportion estimations (MPE), to recover the expert distribution with the known expert ratio α and unknown one respectively. Afterward, we showed that these paradigms can be integrated with off-the-shelf IL methods, such as GAIL, to form the algorithm COMParative Imitation LEearning with Risk rewriting (COMPILER) and that by Estimation (COMPILER-E) to solve the VPIL problem with known and unknown α respectively. The experimental results showed that our methods outperformed standard and preference-based IL methods on a variety of tasks. In the future, we hope to use our algorithms to address more challenging problems, such as VPIL with multiple and noisy annotators.

Acknowledgments

This research was supported by the Institute for AI and Beyond, UTokyo; JST SPRING, Grant Number JPMJSP2108. The authors would like to thank Johannes Ackermann and the anonymous reviewers for their insightful comments and suggestions.

References

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 4565-4573, 2016.

Zi-Xuan Chen, Xin-Qiang Cai, Yuan Jiang, and Zhi-Hua Zhou. Anomaly guided policy learning from imperfect demonstrations. In 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022 , pages 244-252.

Xin-Qiang Cai, Yao-Xiang Ding, Zi-Xuan Chen, Yuan Jiang, Masashi Sugiyama, and Zhi-Hua Zhou. Seeing differently, acting similarly: Heterogeneously observable imitation learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings , volume 100 of Proceedings of Machine Learning Research , pages 66-75. PMLR, 2019.

Xin-Qiang Cai, Yao-Xiang Ding, Yuan Jiang, and Zhi-Hua Zhou. Imitation learning from pixel-level demonstrations by hashreward. In AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021 , pages 279-287. ACM, 2021.

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 6818-6827. PMLR, 2019.

Yunke Wang, Chang Xu, and Bo Du. Robust adversarial imitation learning via adaptivelyselected demonstrations. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 3155-3161. ijcai.org, 2021.

Mostafa Hussein and Momotaz Begum. Detecting incorrect visual demonstrations for improved policy learning. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand , volume 205 of Proceedings of Machine Learning Research , pages 1817-1827. PMLR, 2022.

Yunke Wang, Bo Du, and Chang Xu. Unlabeled imperfect demonstrations in adversarial imitation learning. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023 , pages 10262-10270. AAAI Press, 2023.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from AI feedback. CoRR , abs/2212.08073, 2022.

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 4302-4310, 2017.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, November 28 - December 9, 2022, New Orleans, Louisiana, USA , 2022.

Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , pages 783-792, 2019.

Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 12340-12350, 2021.

Wei Wang and Zhi-Hua Zhou. Crowdsourcing label quality: a theoretical analysis. Sci. China Inf. Sci. , 58(11):1-12, 2015.

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of Machine Learning Research , 13:2973-3009, 2010.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012 , pages 5026-5033, 2012.

Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011 , pages 627-635, 2011.

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada. , pages 8022-8034, 2018.

Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 10961-10970. PMLR, 2021.

Yuanying Cai, Chuheng Zhang, Wei Shen, Xiaonan He, Xuyun Zhang, and Longbo Huang. Imitation learning to outperform demonstrators by directly extrapolating demonstrations. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022 , pages 128-137. ACM, 2022.

Daniel S. Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Proceedings of the 3rd Conference on Robot Learning , 2019.

Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, Tomoya Sakai, and Niu Gang. Machine Learning from Weak Supervision: An Empirical Risk Minimization Approach . MIT Press, 2022.

Takashi Ishida, Gang Niu, and Masashi Sugiyama. Binary classification from positiveconfidence data. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada. , pages 5921-5932, 2018.

Takashi Ishida, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. Complementarylabel learning for arbitrary losses and models. In Proceedings of the 36th International Conference on Machine Learning (ICML) , pages 2971-2980, 2019.

Nan Lu, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019.

Lei Feng, Senlin Shu, Nan Lu, Bo Han, Miao Xu, Gang Niu, Bo An, and Masashi Sugiyama. Pointwise binary classification with pairwise confidence comparisons. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 3252-3262. PMLR, 2021.

Harish G. Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 2052-2060. JMLR.org, 2016.

Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 , volume 307 of ACM International Conference Proceeding Series , pages 1032-1039. ACM, 2008.

Chelsea Finn, Paul F. Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. CoRR , abs/1611.03852, 2016.

Saurabh Garg, Yifan Wu, Alexander J. Smola, Sivaraman Balakrishnan, and Zachary C. Lipton. Mixture proportion estimation and PU learning: A modern approach. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 8532-8544, 2021.

Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Rethinking class-prior estimation for positive-unlabeled learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Rethinking class-prior estimation for positive-unlabeled learning. In 10th International Conference on Learning Representations, ICLR 2022, virtual, April 25-29, 2022 . OpenReview.net, 2022.

Nan Lu, Gang Niu, Aditya Krishna Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, Louisiana, US, May 6-9, 2019 , 2019.

Yu-Jie Zhang, Peng Zhao, Lijun zhang, and Zhi-Hua Zhou. An unbiased risk estimator for learning with augmented classes. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , pages 10247-10258, 2020.

Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018 . OpenReview.net, 2018.

Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positiveunlabeled learning with non-negative risk estimator. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NIPS 2017, December 4-9, 2017, Long Beach, CA, USA , pages 1675-1685, 2017.

Nan Lu, Tianyi Zhang, Gang Niu, and Masashi Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1115-1125, 2020.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . The MIT press, 2012.

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016 , 2016.

Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, UK, July 1995] , pages 103-129. Oxford University Press, 1995.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations , 2018.

Letian Chen, Rohan R. Paleja, and Matthew C. Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA , volume 155 of Proceedings of Machine Learning Research , pages 1262-1277. PMLR, 2020.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning - From Theory to Algorithms . Cambridge University Press, 2014.

A Proof for the Theorems

In this section, we provide the proof for Theorem 2, Proposition 1, and Theorem 3 in the main body of the paper.

Theorem 2. Assume the pairwise datasets (Γ + , Γ -) are generated following the procedure in Section 3.2 in the main paper. Let p π + ( s, a ) = E Γ + [ ̂ p π + ( s, a ) ] and p π -( s, a ) = E Γ -[ ̂ p π -( s, a ) ] be the expected occupancy measures of Γ + and Γ -, where the randomness is taken over the draws of Γ + and Γ -. Then, we have

Proof. We first prove the first line of (9). Considering the data collection process introduced in Section 3.2, where a pair of trajectories ( τ (1) , τ (2) ) is independently sampled from the demonstrator pool. When one trajectory enjoys a better quality than another, the collector would put the better one into Γ + . Otherwise, the two trajectories will be randomly allocated. Let ̂ p + i ( s, a ) = (1 -γ ) ∑ ∞ t =0 γ t ✶ [( s t,i , a t,i ) = ( s, a )] be the empirical occupancy measure for the trajectory τ i ∈ Γ + . Then, further define the even E 1 = { τ i is sampled from π E } and event E 1 = { τ i is sampled from π NE } . We have

The second equality is due to τ i is i.i.d. sampled from p π E under event E 1 and is i.i.d. sampled from p π -under event E 2 . Then, let the even A 1 = { both τ (1) , τ (2) are sampled from π E } , A 2 = { both τ (1) , τ (2) are sampled from π NE } and A 3 = { other situation } . Then, we have

where the second inequality is due to Pr [ E 1 | A 1 ] = Pr [ E 3 | A 3 ] = 1 and Pr [ E 1 | A 2 ] = 0 according to the data collection process. We can also show the probability of event E 2 is Pr [ E 1 ] = (1 -α ) 2 , then we have

Since ̂ p π + ( s, a ) = ∑ n + i =1 ̂ p + i ( s, a ) /n + , we have E [ ̂ p π + ( s, a )] = ∑ n + i =1 E [ ̂ p + i ( s, a )] /n + = ( 2 α -α 2 ) p π E ( s, a ) + (1 -α ) 2 p π NE ( s, a ) , which completes the proof for the first equality of (9). The second equality of (9) can be proved following a similar arguments

Proposition 2. Suppose the distributions p π E and p π NE are mutually irreducible such that there exists no decomposition of the form p π E = (1 -η ) Q + ηp π NE and p π NE = (1 -η ′ ) Q ′ + η ′ p π E for any probability distributions Q,Q ′ and scalars η, η ′ ∈ (0 , 1] . Then, the true mixture proportions β 1 and β 2 are unique and can be identified by

where C is the set containing all possible probability distributions. Thus, α is identifiable by

Proof. We first prove the first line of (12). Since p π E is irreducible w.r.t. p π NE , such that there exists no decomposition p π E = (1 -η ) Q + ηp π NE for any η ∈ (0 , 1] and distribution Q , we have p π E is also irreducible w.r.t. p π -, such that p π E can not be rewritten as a mixture p π E = (1 -η ′ ) Q ′ + η ′ p π -for any distribution Q ′ and η ′ ∈ (0 , 1] . Then, according to the relationship (5) of the main paper and Proposition 5 of [16], we can show β 1 is identifiable and β 1 = sup { η | p π + = ηp π -+(1 -η ) K,K ∈ C} . We can prove the second line of (12) by a similar argument. Then, combined with β 1 = 1 -α 1+ α and β 2 = α 2 -α in (5) of the main paper, we can obtain identifiable α estimation in (13).

Theorem 3. Let W be a parameter space for training the discriminator and D W = { D w | w ∈ W} be the hypothesis space. Assume the functions | log D w ( s, a ) | ≤ B and | log(1 -D w ( s, a )) | ≤ B are upper-bounded for any state-action pair ( s, a ) ∈ S × A and w ∈ W . Further assume both the functions log D w ( s, a ) and log(1 -D w ( s, a )) are L -Lipschitz continuous in the state-action space. For a fixed policy π θ , let Γ θ = { τ θ i } n θ i =1 be trajectories generated from π θ . Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

where ̂ D w = arg max w ∈W ̂ V ( π θ , D w ) and D ∗ w = arg max w ∈W V ( π θ , D w ) . The constants n + and n -are the number of trajectories in Γ + and Γ -. We define C ( δ ) = 4 B √ log(6 /δ ) . The empirical Radamacher complexities [40] on datasets Γ + , Γ -, and Γ θ are denoted by R n + , R n -, and R n θ .

Proof. Since ̂ D w and D ∗ w are the maximizer of the objective functions ̂ V ( π θ , D w ) and V ( π θ , D w ) , respectively.

where the inequality is due to the optimality of ̂ D w . The according to the definition of V ( π θ , D w ) and ̂ V ( π θ , D w ) ,

and

where ̂ p π θ ( s, a ) = 1 n θ ∑ n θ i =1 ̂ p θ i ( s, a ) and ̂ p θ i ( s, a ) = (1 -γ ) ∑ ∞ t =0 γ t ✶ [( s t,i , a t,i ) = ( s, a )] is the empirical occupancy measure for the trajectory τ i sampled from π θ The empirical occupancy measure is unbiased w.r.t. π θ such that E [ ̂ p θ i ( s, a )] = p π θ ( s, a ) . Then, we can decompose the R.H.S. of (15) as

In above, the first term

measures the generalization gap between ̂ p π θ and p π θ . The second term

measures the generalization gap between ̂ p π -and p π -and the last term

measures the gap between ̂ p π + and p π + .

Under the condition that | log D w ( s, a ) | ≤ B for any ( s, a ) ∈ S × A and w ∈ W , the standard analysis generalization analysis with Rademacher complexity shows (e.g. Theorem 26.5 of [47]), for all w ∈ W , we have

with probability at least 1 -δ ′ , where R n θ is the empirical Rademacher complexity. Then, according to the Talagrand's contraction inequality (Lemma 26.9 of [47]), we have

Since | log(1 -D w ) | ≤ B and log(1 -D w ) is L -Lipschitz continuous, the a similar arguments ensures,

and

with probability at least 1 -δ ′ . Let δ ′ = δ/ 3 , we complete the proof by combining (18), (19), and (20) with (15).

B Full Results of α Estimations

In the main body of the paper, we only reported the effect of overestimation and underestimation on HalfCheetah environment due to the space limitation. Here we provide the full results on four environments in our experiments, as shown in Figure 5. The experimental results demonstrated the same conclusion as in the main body of the paper, that the underestimation method is better than the overestimation one.

Then, we obtain

̂

̂

̂

̂

̂

̂

̂

̂

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

Underestimation(Ours)

Overestimation

True expert ratio

α

(a)

BBE

estimator

on

HalfCheetah

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(b)

KM

estimator

on

HalfCheetah

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(c) BBE estimator on Hop-

per

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(d) KM estimator on Hopper

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(e) BBE estimator on Swim-

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(f) KM estimator on Swim-

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(g)

BBE

estimator

on

Walker2d

0.1

0.2

0.3

0.4

0.5

true expert's ratio

α

0.0

0.2

0.4

0.6

0.8

1.0

estimated expert's ratio

α

(h)

KM

estimator

on

Walker2d

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEdit: Changed the score from a 3 to a 6 due to the discussion below.\n\nThe paper focuses on tackling the challenge of Imitation Learning from imperfect demonstrations that come with vague pairwise comparisons. The authors make the key assumption that a competent annotator is available, but that they become uncertain when presented with either two expert or two non-expert samples, randomly assigning them as \"good\" or \"bad\" examples. To address this problem, they introduce a novel learning paradigm called Vaguely Pairwise Imitation Learning. By modeling the demonstration pool as a mixture of expert and non-expert data, the authors show that it is possible to recover the expert policy distribution when the proportion of expert data is known. Furthermore, they propose a method for estimating the mixture proportion in cases where this expert ratio is unknown.\n\nTo effectively leverage the recovered expert policy distribution, the authors integrate it with Generative Adversarial Imitation Learning (GAIL), resulting in end-to-end algorithms designed for scenarios with known and unknown expert data proportions. The performance of these algorithms is evaluated through experiments conducted on various Mujoco tasks. The results showcase that the proposed methods consistently outperform standard and preference-based imitation learning approaches for expert ratios of up to $0.5$.\n\nSTRENGTHS:\n- Novel Approach for Vaguely Pairwise Imitation Learning: The paper proposes a novel approach to estimate precise expert occupancy measures from essentially imperfect preferences. This addresses the challenge of working with imperfect data and provides a valuable new approach for imitation learning tasks.\n- Sound Theory and Proofs: The paper demonstrates a solid theoretical foundation, supported by proofs in the appendix. The proposed algorithm is very closely related to the algorithm, using few additional assumptions or relaxations.\n- Strong Experimental Performance: The experimental evaluation of the proposed method showcases its superior performance compared to a large number of standard and preference-based imitation learning methods. The results presented in the paper demonstrate the efficacy and practical relevance of the approach across various Mujoco environments.\n\nOverall, the paper exhibits originality in its approach, a solid theoretical foundation for the presented algorithm, and significance in its experimental results. These strengths contribute to the advancement of the field of Imitation Learning and may offer valuable insights for both researchers and practitioners.\n\nWEAKNESSES:\nThe following weaknesses are listed in order of severity.\n\n- Suspected Issue with Overestimation: The values of the overestimation Figure 3a, b, and Appendix Figure 1a-h, are very close to 1 - the true ratio. Additionally, the matrix plots in Figure 3c, d appear as negative transposes of each other. This raises suspicion and suggests a potential issue with the code implementation.\n- Inconsistent Experimental Results in Table 1: The experimental results presented in Table 1 exhibit inconsistencies. The HalfCheetah and Hopper experiments seem to use results from Swimmer and Walker2d for all methods, except for Behavioral Cloning and GAIL. Consequently, there are no results provided specifically for these tasks in the current state of the submission, which reduces the completeness of the evaluation.\n- Evaluation of GAIL_expert and Confidence Intervals: The evaluation of GAIL_expert in Table 1 is performed separately for different alpha values, despite its independence from alpha. Moreover, while GAIL_expert is presented as an upper bound for performance, the presented methods outperform it in several instances. This may indicate that the number of random seeds used (5) is insufficient, especially considering the relatively large confidence intervals.\n- Unclear Advantages of COMPILER-E: It is unclear why COMPILER-E would outperform COMPILER. Exploring potential reasons, for example that COMPILER-E may remove \"bad\" expert demonstrations through an implicit de-noising procedure in its estimation of the expert ratio could provide clarification and improve the understanding of its advantages.\n- Limited Experiment Scope: The experiments conducted in the paper are limited to Mujoco environments. Including experiments on other environments, such as Meta-world, would enhance the comprehensiveness of the evaluation.\n- Code Unavailability: The paper does not provide the code, which can hinder reproducibility and limit further exploration of the proposed methods.\n- Missing Hyperparameters: The paper lacks information about the hyperparameters used for some of the baseline methods. For example, for D-Rex the number of noise levels utilized to estimate the distributions during training is missing.\n- Proof Derivations Lack Readability: The derivations for the proofs are presented in a compact manner, making them difficult to follow. Providing additional steps and explanations would greatly improve the readability of the proofs.\n- Typos and Style:\n    - Line 133 references Section 6.4, which does not exist in the paper.\n    - The layout of the caption for Figure 3 makes it challenging to read and comprehend the information provided.\n    - Figure 2 could benefit from a descriptive caption to aid understanding and provide context for the information presented.\n    - There are instances of redundancy and language issues throughout the paper, such as the information in Line 163, which was already mentioned in the previous paragraph and e.g., the over-use of \"the\" in the first sentence of the description of Figure 1. Addressing these issues would improve the overall clarity and readability of the paper.\n    - Factorization in Equation 3: In Equation 3, the factor $2(1-\\alpha)$ is factored out only once, even though it appears twice in the equation. Ensuring consistent notation would enhance the equation's clarity.\n\nOverall, the suspected problems with the overestimation method for the expert ratio, and the missing results for $2$ out of $4$ tasks are a clear reason to reject the paper. Addressing these (and other) issues during the rebuttal period would significantly increase the quality of the submission.\n\nQUESTIONS:\n- Importance of Equation 8: Equation 8 rephrases the GAIL loss in terms of the estimated positive and negative data distributions to mitigate overfitting. How crucial is this rephrasing for the overall performance of the approach? Conducting an ablation study to assess the impact of this choice would provide valuable insights and clarify its significance.\n- Alternatives to Algorithm 1: Algorithm 1 includes the design choices of iteratively training a discriminator to approximately separate expert from non-expert data. What would happen if a mixture proportion estimator were directly applied to \\Gamma^+ and \\Gamma^- instead? Explaining the reasoning behind the chosen approach and comparing it to alternative strategies would enrich the discussion and highlight the advantages of the proposed algorithm.\n- Intuition Behind the Step from Equation 2 to Equation 3: Equations 2 and 3 explain the relations between the (non)-expert the pairwise dataset’s occupancy measures. It would be helpful to gain some intuition on what the transition from e.g., $p_{\\pi_+}$ to $p_{\\pi_E}$ actually means. A deeper explanation or examples illustrating this step would make this easier to understand.\n- Clarification on Expert Ratio Estimation: Lines 96-98 state that \"[…] [5] also used the risk rewriting technique to identify the expert demonstrations, but their algorithm does not require estimation of the expert ratio as the confidence score is given.\" For the presented approach, the expert ratio is also given in some of the experiments. How does the approach differ from this particular related work in this case?\n\nLIMITATIONS:\nThe limitations or potential negative societal impact of the work are currently not addressed. It should be acknowledged that the method assumes a perfect annotator who randomly assigns samples when they come from the same distribution. However, in real-world scenarios, annotators may possess biases or preferences towards certain types of solutions, which are not accounted for in the proposed approach. Addressing and discussing this limitation would strengthen the submission.\n\nIn terms of ethical considerations, aspects such as fairness, transparency, and bias in the training and deployment of imitation learning models is crucial to ensure responsible and ethical AI development. It could be beneficial for the authors to include a section that highlights these potential ethical implications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-16"}

comments: SUMMARY:
This paper falls into imitation learning with human preference. Specifically, this paper deals with a more challenging setting, where the annotator information is wrong, i.e., some data collected by non-experts could be labeled as positive demonstrations, and some data collected by experts could be labeled as negative demonstrations.
This paper proposed two methods to solve this challenge, one with a known ratio of real demonstration data, and one with an unknown ratio.
The proposed approach is theoretically justified, and evaluated in the mujoco domain.

STRENGTHS:
The proposed method is theoretically justified.

WEAKNESSES:
The experiment results in rather confusing, e.g., in Table 1, for the HalfCheetah environment, apparently, the return of COMPILER is not the largest, but it is with the bold font, and the return of GAIL if significantly larger than the proposed approach.

QUESTIONS:
If the authors could show that the proposed method can work well in a more challenging task, such as humanoid running, this paper could be greatly improved.

LIMITATIONS:
This paper has not discussed the limitations in the main body of the submission.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript considers a popular learning framework, learning from human feedback/preference. In real-world applications, the feedback can be inaccurate and detrimental to the training process. The manuscript formulates the imitation learning (IL) from vague feedback process, in which the data annotator can only provide accurate preference when the paired demonstrations are significantly different. The authors analyze the problem for two different levels of difficulty, i.e., known or unknown expert proportion $\\alpha$, and propose two algorithms based on risk rewriting and mixture proportion estimation to solve the two problems separately. The authors also conduct experiments and case studies with various IL tasks and state-of-the-art baselines to verify the effectiveness and generality of the algorithms.\n\nSTRENGTHS:\n### Originality\nThe manuscript introduces a realistic problem in feedback-based imitation learning, accounting for the potential noisiness in human feedback. The proposed scenario is distinguished by its realism and poses novel challenges to the reinforcement learning and imitation learning communities. Furthermore, the authors ingeniously develop a learning framework, which, when integrated with existing imitation learning techniques, effectively addresses the stated problem.\n\n### Quality\nThe authors adeptly formulate the problem, lending credence to its practical significance. The learning framework presented is versatile, facilitating integration with various imitation learning (IL) and mixture proportion estimation (MPE) methods. Commendably, the manuscript offers rigorous theoretical analyses regarding the interaction between MPE and the Generative Adversarial Imitation Learning (GAIL) approach. Moreover, the experimental section is robust, featuring comparisons across an extensive array of tasks and benchmarks. This establishes the efficacy and generalization capabilities of the proposed algorithms.\n\n### Clarity\nThe writing is lucid, and the claims advanced are substantiated effectively. The incorporation of a comprehensive collection of case studies furnishes empirical validation, thus bolstering the merits of this work. This meticulous detailing is, in my estimation, one of the strong suits of this manuscript.\n\n### Significance\nThis manuscript addresses a salient issue within the reinforcement learning and imitation learning domains - learning from human feedback. Through an adept modeling of ambiguous feedback and the introduction of an efficacious learning framework, this manuscript renders important contributions to the field. These are all very important contributions to the community.\n\nWEAKNESSES:\nThe text size in Table 1 is a little too small. Also, I recommend the authors to put the robustness results in Appendix D to the main manuscript. This is also an important case study.\n\nQUESTIONS:\nIs it possible to separate expert and non-expert data first, and then use T-REX to learn the policy?\n\nLIMITATIONS:\nAlthough the authors have illustrated VPIL with multiple annotators as the future work, I recommend the authors to discuss it as a limitation of this work with more details.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper formulates the problem of imitation learning from imperfect demonstrations with vague comparisons named Vaguely Pairwise Imitation Learning (VPIL) which contains both expert and non-expert demonstrations, and the data collector only provides vague pairwise information of demonstrations. The authors proposed two learning paradigms to solve VPIL. One with risk rewriting and mixture proportion estimations (MPE), to recover the expert distribution with the known expert ratio α and unknown one respectively. They evaluated the proposed method on simulate MuJoCo based tasks.\n\nSTRENGTHS:\n- The paper attempts to solve a pretty interesting problem.\n- The experimental results demonstrates that the proposed method is able to outperform standard and preference-based IL methods.\n- The paper presents good theoretical analysis of the problem and proposed method.\n- Extensive comparison to baselines.\n\nWEAKNESSES:\n- The paper has some typos and grammatical errors. For ex. In line 333, \"the\" is written twice; In line 36, \"off-the-shall\" -> \"off-the-shelf\". I would highly recommend a senior author to proof-read the paper.\n- Some parts of the paper are a little difficult to follow. For ex. Line 43-53.\n- Also, the problem of vague feedback is a little unclear.\n- The paper evaluates the method on 4 different MuJoCo tasks. However, problem is motivated from real-world robotics/imitation based task. It would be good for the authors to at least discuss how there method would work in real-world. If possible they can perform some simple experiment on real-world task that would be great.\n- There is only one small future extension mentioned in the paper. The authors are encouraged to at least mention few more and have breif discussion.\n\nQUESTIONS:\nMentioned above in weakness\n\nLIMITATIONS:\n- I don't think authors have mentioned the limitations of the proposed method. They are encouraged to mention them in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of Imitation Learning (IL) when the provided demonstrations come from experts and non-experts. The dataset may then not divided exactly into expert and non-expert subsets, but instead a data collection procedure may only vaguely separate the data. Thus, the authors pose the problem of Vague pairwise imitation learning, where the goal is finding a policy that approximates the expert policy that provided the expert demonstrations. \nTo tackle the problem, two cases are considered: the ratio of expert to non-expert demonstrations is known or unknown. In the first case the authors show how the occupancy measure (commonly used in IL) of the expert policy can be estimated, which then allows to solve the problem using standard IL methods. In the second case, the ratio is estimated using a lower bound.\nThe approach is evaluated in 4 different simulation setups and compared against numerous state-of-the-art IL methods. For various ratios of expert to non-expert data, the proposed method shows clear advantages over the baselines. Lastly, the evaluation provides a robustness analysis for the estimator, as well as empirical results motivating the use of the lower bound as opposed to an upper bound.\n\nSTRENGTHS:\nOverall, the paper is very well written and addresses an interesting and active research area. \nThis work addresses limitations of state-of-the-art IL algorithms when the dataset is a mixture of expert and non-expert demonstrations.\nThe proposed approach is - to the best of my knowledge - original  and the technical contribution is sound and provides theoretical results on the way. The simulation results are overall convincing and effectively showcase the benefits of the proposed method.\n\nWEAKNESSES:\nThe structure of the paper could be improved by adding a clear problem statement and list the main contributions.\nIn the problem description the paper should add some more detail, in particular definitions of the expert and non-expert policies.\nSome parts of the discussion of the results should be clarified.\nFurther, the paper does not have a discussion of its limitations and how restrictive the assumptions / model for the data collector are.\n Please find my detailed comments below.\n\nQUESTIONS:\nQuestions\n\n1.\tProblem statement (Sect. 3.2): There is no definition of an expert and non-expert policy. How do they differ? Is an expert always consistent, i.e., do their demonstrations optimizes the same reward function (without any noise)? It would be good to define the policy types at least on a high level to obtain a well-posed problem.\n\n2.\tNo clear statement of contribution: The introduction would benefit from a contributions section, highlighting the novelty of the work.\n\n3.\tResults: Line 292 “As α grows, […] COMPILER and COMPILER-E got the biggest performance boost and achieved the best performances.” This claim does not seem to fit the data reported in the table. On two of four tasks it is CAIL that achieves the best performance for the largest alpha=0.5. Furthermore, the relative increase between alpha 0.1 and 0.5 is only up to 5% for COMPILER, while CAIL increases the return by 50-100%. Either I misunderstand something, or the claim is wrong, please clarify.\n\n4.\tThe paper should have limitations section, discussing how restrictive the model for the data collection is. Further, related to my first question, some discussion on the differences between the expert and non-expert policies would be required, and how accurate the assumed policies are.\n\n\nMinor Comments\n\n5.\tWhat are the objectives in the maze example in Figure 1e? Only distance is mentioned, but with only one objective the task becomes rather trivial, even for non-experts\n\n6.\tEnd of problem statement: It would be good to conclude the section with a concise problem statement (in a theorem-style environment). Earlier it is mentioned that the goal is to learn a policy $\\pi_{\\theta}$ that mimics the expert policy – does that mean it minimizes the difference in the occupancy measure? It would help to make such an objective explicit.\n\n7.\tLines 135-136  seem to be redundant w.r.t. to the paragraph before\n\n8.\tLine 160. What algorithm is referred to? Is (3) referring to equation 3? I cannot really follow the sentence, this should be clarified.\n\n9.\tResults: The problem statement defined alpha to be in (0,1], yet the experiments only consider alpha from .1 to .5. What happens for larger alpha – do the baselines catch up eventually? These results might not be as interesting as for the lower alpha reported, but this could at least be mentioned. Also it is peculiar in Fig 3 a and b that the over and underestimate already converge for alpha=.5 – is there a reason for that?\n\nLIMITATIONS:\nThe paper does not have a limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}