Title: Imitation Learning from Imperfection: Theoretical Justifications and Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: imitation learning

distribution shift

policy optimization

data selection
Abstract: Imitation learning (IL) algorithms excel in acquiring high-quality policies from expert data for sequential decision-making tasks. But, their effectiveness is hampered when faced with limited expert data. To tackle this challenge, a novel framework called (offline) IL with supplementary data has been proposed, which enhances learning by incorporating an additional yet imperfect dataset obtained inexpensively from sub-optimal policies. Nonetheless, learning becomes challenging due to the potential inclusion of out-of-expert-distribution samples. In this work, we propose a mathematical formalization of this framework, uncovering its limitations. Our theoretical analysis reveals that a naive approach—applying the behavioral cloning (BC) algorithm concept to the combined set of expert and supplementary data—may fall short of vanilla BC, which solely relies on expert data. This deficiency arises due to the distribution shift between the two data sources. To address this issue, we propose a new importance-sampling-based technique for selecting data within the expert distribution. We prove that the proposed method eliminates the gap of the naive approach, highlighting its efficacy when handling imperfect data. Empirical studies demonstrate that our method outperforms previous state-of-the-art methods in tasks including robotic locomotion control, Atari video games, and image classification. Overall, our work underscores the potential of improving IL by leveraging diverse data sources through effective data selection.

Imitation Learning from Imperfection: Theoretical Justifications and Algorithms

Abstract

Imitation learning (IL) algorithms excel in acquiring high-quality policies from expert data for sequential decision-making tasks. But, their effectiveness is hampered when faced with limited expert data. To tackle this challenge, a novel framework called (offline) IL with supplementary data has been proposed [25, 61], which enhances learning by incorporating an additional yet imperfect dataset obtained inexpensively from sub-optimal policies. Nonetheless, learning becomes challenging due to the potential inclusion of out-of-expert-distribution samples. In this work, we propose a mathematical formalization of this framework, uncovering its limitations. Our theoretical analysis reveals that a naive approach-applying the behavioral cloning (BC) algorithm concept to the combined set of expert and supplementary data-may fall short of vanilla BC, which solely relies on expert data. This deficiency arises due to the distribution shift between the two data sources. To address this issue, we propose a new importance-sampling-based technique for selecting data within the expert distribution. We prove that the proposed method eliminates the gap of the naive approach, highlighting its efficacy when handling imperfect data. Empirical studies demonstrate that our method outperforms previous state-of-the-art methods in tasks including robotic locomotion control, Atari video games, and image classification. 1 Overall, our work underscores the potential of improving IL by leveraging diverse data sources through effective data selection.

1 Introduction

Imitation learning (IL) [3, 34] is an essential technique in artificial intelligence, allowing machines to enhance their performance by imitating expert behaviors. This technique has showcased remarkable achievements across various domains such as robotics [27], self-driving cars [35], and language models [55, 8]. Among the IL approaches, behavioral cloning (BC) [40] stands out as a popular method. In particular, BC leverages temporally-correlated state-action pairs extracted from expert demonstrations, employing them as training samples to learn a mapping from states to actions via maximum likelihood estimation. In this manner, IL expands upon the traditional supervised learning framework, enabling the acquisition of sequential decision-making capabilities.

The quantity of expert trajectories plays a crucial role in achieving satisfactory performance. Previous studies have shown that BC works well when the dataset contains a large number of expert-level

trajectories [48]. However, the well-known compounding errors issue [44] renders any offline IL algorithm, including BC, ineffective when the number of expert trajectories is small [42, 60]. Furthermore, collecting more trajectories from the expert is costly and impractical in many domains, such as robotics and healthcare.

To overcome the challenge of scarce expert data, we propose to use an additional yet imperfect dataset to supplement the expert data; see Figure 1 for illustration. In particular, this additional dataset can be cheaply obtained by executing sub-optimal policies. 2 However, the incorporation of supplementary dataset introduces a distribution shift issue due to the presence of out-of-expertdistribution trajectories. 3 The distribution shift issue may hamper the model's performance in utilizing the supplementary data, as we will argue later.

We realize that a number of empirical successes have been reported in this direction [25, 24, 61, 32]. Most algorithms rely on a discriminator to distinguish between expert-style and sub-optimal samples, followed by optimization of a weighted BC objective to learn a good policy. For example, DemoDICE [25] uses a regularized state-action distribution matching objective to train the discriminator, while DWBC [61] employs a cooperative training framework for the policy and discriminator. Despite the empirical successes in certain scenarios, there remains a notable absence of systematic theoretical studies, particularly in terms of imitation gap (i.e., performance difference between the expert and learner), which may hinder deep understanding and impede future algorithmic advances. Framework 3: Cross-dataset Transfer

limited

expert data

rich

supplementary data

but

OOD

model training

data

selection

data

augment

6

We aim to bridge the gap between theory and practice in the (offline) IL with supplementary data framework by developing effective algorithms and providing rigorous theoretical guarantees. To the best of our knowledge, only [9] provided imitation gap bounds for a model-based adversarial imitation learning approach in a similar problem. However, our focus lies on the widely used and simpler BC and its variants, which are model-free in nature. Our contributions are summarized below.

We develop a formal mathematical framework of the IL with supplementary data framework and conduct corresponding theoretical analysis. Our findings highlight the impact of the distribution shift between expert and supplementary data. Our results are summarized in Table 1. 4

In particular, our analysis shows that naively applying BC on the union of expert and supplementary data (referred to as the NBCU method in this paper) has a non-vanishing error term in the imitation gap bound (see the second row of Table 1). This means that the direct use of additional data might yield inferior performance compared with solely using the expert data with BC. This necessitates the development of more advanced algorithms.

To address the distribution shift issue, we propose a new approach called ISW-BC, which uses the importance sampling technique to select data within the expert distribution. In contrast to prior methods [25, 61], ISW-BC re-weights data in an unbiased way. We develop a new imitation gap bound for ISW-BC (see the last row of Table 1), revealing that it not only eliminates the gap exhibited by the naive approach but also offers a superior guarantee over BC.

Our theoretical analysis has been validated through extensive experiments, including robotic locomotion control, Atari video games, and image classification. The results affirm the superiority of ISW-BC over existing methods, thus demonstrating the potential of our method in addressing the distribution shift issue in IL with supplementary data.

2 Related Work

We review briefly relevant studies in the main text and provide a detailed discussion in Appendix B.

Behavioral Cloning. Behavioral cloning (BC) is a popular algorithm in the offline setting, where the learner cannot interact with the environment. According to the learning theory in [42], only using an expert dataset, BC has an imitation gap of O ( |S| H 2 /N E ) , where |S| is the state space size, H is the planning horizon, and N E is the number of expert trajectories. Our work investigates the use of a supplementary dataset to enhance the dependence on the data size.

Adversarial Imitation Learning. In contrast to BC, adversarial imitation learning (AIL) methods, such as GAIL [21], perform imitation through state-action distribution matching. It has been demonstrated both empirically and theoretically that AIL methods do not suffer from the compounding errors issue when the expert data is limited [21, 16, 26, 59, 60, 28, 63]. Under mild conditions, [62] provided a horizon-free bound of O (min { 1 , √ |S| /N E } ) , which is much better than BC in terms of dependence on H . However, AIL methods work naturally in the online setting (i.e., the interaction is allowed), which is not directly applicable in the offline setting that we study in this paper. Although the proposed method has a discriminator and a policy like AIL, our discriminator and policy are not designed to compete with each other adversarially, as we will explain in detail later.

Imitation Learning with Supplementary Data. Our theoretical study is motivated by recent empirical successes in IL with supplementary data [25, 61, 32, 24]. Compared with [25, 61], a related setting, learning from observation, is studied in [32, 24]. In this setting, expert actions are absent, and only expert states are observed. The importance sampling technique used in our method for addressing distribution shift is also studied in (semi-)supervised learning [49, 11, 30, 15]. Our contribution is to show this technique is also effective in the imitation learning, where data has a Markovian structure.

3 Preliminaries

Markov Decision Process. In this paper, we consider the episodic Markov decision process (MDP) framework [41]. An MDP is defined by the tuple M = ( S , A , P , r, H, ρ ) , where S and A are the state and action space, respectively. H is the maximum length of a trajectory, and ρ is the initial state distribution. The non-stationary transition function is specified by P = { P 1 , · · · , P H } , where P h ( s h +1 | s h , a h ) determines the probability of transiting to state s h +1 given the current state s h and action a h in time step h , for h ∈ [ H ] . Here the symbol [ x ] means the set of integers from 1 to x . Similarly, the reward function r = { r 1 , · · · , r H } specifies the reward received at each time step, where r h : S × A → [0 , 1] for h ∈ [ H ] . A policy in an MDP is a function that maps each state to a probability distribution over actions. We consider time-dependent policies π h : S → ∆( A ) , where ∆( A ) is the probability simplex. The policy at each time step h is denoted as π h , and we use π to denote the collection of time-dependent policies { π h } H h =1 when the context is clear.

We measure the quality of a policy π by the policy value (i.e., environment-specific long-term return): V ( π ) = E [ ∑ H h =1 r h ( s h , a h ) | s 1 ∼ ρ ; a h ∼ π h ( ·| s h ) , s h +1 ∼ P h ( ·| s h , a h ) , ∀ h ∈ [ H ] ] . To facilitate later analysis, we need to introduce the state-action distribution d π h ( s, a ) = P ( s h = s, a h = a | π ) . We use the convention that d π is the collection of time-dependent state-action distributions.

Imitation Learning. Imitation learning (IL) aims to learn a policy that mimics an expert policy based on expert demonstrations. In this paper, we assume that there exists a good expert policy π E that generates a dataset D E consisting of N E trajectories of length H .

The learner aims to imitate the expert using the expert dataset D E . The quality of the imitation is measured by the imitation gap , defined as E [ V ( π E ) -V ( π ) ] , where the expectation is taken over the randomness of data collection. It is worth noting that in the training phase, the IL learner does not have access to reward information. A good learner should closely mimic the expert, resulting in a small imitation gap. We assume that the expert policy is deterministic, a common assumption in the literature [42, 43, 60], and applicable to tasks such as MuJoCo locomotion control.

Behavioral Cloning. Behavioral cloning (BC) is a popular imitation learning algorithm that aims to learn a policy from an expert dataset D E via supervised learning. Specifically, BC seeks to find a policy π BC that maximizes the log-likelihood of the expert actions in the dataset:

where ̂ d E h ( s, a ) is the empirical state-action distribution in the expert dataset. By the maximum likelihood estimation (MLE), BC can make good decisions by duplicating expert actions on states visited in D E . However, BC may take sub-optimal actions on non-visited states, resulting in compounding errors and a large imitation gap. This issue is significant when the expert data is limited.

4 Imitation Learning with Supplementary Data

To address the challenge of limited availability of expert data, we consider an IL with a supplementary dataset framework. We assume that a supplementary yet imperfect dataset D S = { tr = ( s 1 , a 1 , s 2 , a 2 , · · · , s H , a H ) } is collected by a behavior policy π β . A naive approach is to perform MLE on the union of the expert and supplementary dataset D U = D E ∪ D S :

where ̂ d U h ( s, a ) is the empirical state-action distribution in D U . We refer to this approach as NBCU (naive BC with the union dataset). NBCU treats these two datasets equally and is brittle to distribution shift, as we will demonstrate later. For theoretical analysis purpose, we assume expert data represents a η ∈ [0 , 1] fraction of the total union data.

Assumption 1. The expert dataset D E and supplementary dataset D S are collected in the following way: each time, we roll-out a behavior policy π β with probability 1 -η and the expert policy with probability η . Such an experiment is independent and identically conducted N tot times.

Under Assumption 1, we slightly overload our notations: we use N E to denote the expected number of expert trajectories, which is given by N E = ηN tot , and N S to denote the expected number of supplementary trajectories, which is given by N S = (1 -η ) N tot . Note that the probabilistic sampling procedure does not change the nature of our theoretical insights. In practice, one may collect a fixed number of expert and supplementary trajectories, respectively.

To establish a common ground, we begin by specifying the policy representations. Here, we adopt tabular representations, which assume that the parameterized value functions can take any possible form. Specifically, we define π h ( a | s ; θ ) = ⟨ ϕ ( s, a ) , θ ⟩ , where ϕ ( s, a ) ∈ R d is the feature representation and θ ∈ R d is the parameter to optimize. In tabular representations, we use one-hot features for ϕ ( s, a ) . For a discussion on general function approximation schemes, please refer to Appendix E.

Imitation Gap of BC. In order to evaluate the usefulness of the supplementary dataset, we use BC with only the expert dataset as a baseline. The analysis of this approach has been done in the conventional IL set-up in [42], and we re-state their results in our setting.

Theorem 1. Under Assumption 1, if we apply BC only on the expert dataset, we have that E [ V ( π E ) -V ( π BC ) ] = O ( |S| H 2 N E ) , where the expectation is taken over the randomness in the dataset collection (same as other expectations).

Proofs of Theorem 1 and other theoretical results are deferred to the Appendix. The proof of Theorem 1 builds on [42], with the main difference being that the number of expert trajectories is a random variable in our set-up. We handle this difficulty by using Lemma 3 in the Appendix. The quadratic dependence on the planning horizon H indicates the compounding errors issue of BC.

If the expert data is limited (i.e., N E is small), the performance gap can be large, suggesting poor performance of BC in this scenario.

Imitation Gap of NBCU. Guarantees of naively using the supplementary data are presented below. Theorem 2. Under Assumption 1, if we apply BC on the union dataset, we have E [ V ( π E ) -V ( π NBCU ) ] = O ((1 -η )( V ( π E ) -V ( π β )) + |S| H 2 log( N tot ) N tot ) .

Remark 1. In case when the behavior policy is inferior to the expert policy, we have V ( π E ) -V ( π β ) > 0 . In this case, even if N tot is large enough to make the second term negligible, there is still a non-vanishing gap of V ( π E ) -V ( π β ) due to the behavior policy's potential to collect non-expert actions. In other words, the recovered policy may select wrong actions even on expert states, leading to the sub-optimal performance of NBCU.

It is worthy to note that this non-vanishing error term can also be interpreted from the viewpoint of the distribution shift. Specifically, using the analysis in [62], we can show that

where ε d = max h TV( d π E h , d π β h ) is the state-action distribution total variation (TV) distance and ε π = max h max s TV( π E h ( ·| s ) , π β h ( ·| s )) is the policy distribution TV distance. Hence, we can also view Theorem 2 in the context of state-action or policy distribution shifts.

Note also that we do not claim that NBCU is always worse than BC. Instead, Theorem 2 implies that if the distribution shift between two data datasets is small, NBCU could be better than BC by leveraging more data, which we will also show in experiments later.

The next proposition establishes the inevitability of the gap V ( π E ) -V ( π β ) in the worst case. Proposition 1. Under Assumption 1, there exists an MDP M , an expert policy π E and a behavior policy π β , such that we have E [ V ( π E ) -V ( π NBCU ) ] = Ω((1 -η )( V ( π E ) -V ( π β ))) .

The hard instance in Proposition 1 builds on the following idea: NBCU considers all action labels in the union dataset equally important. Therefore, we can build an instance where the expert π selects a good action with an one-step reward of 1 , while the behavior policy π β chooses a bad action with an one-step reward of 0 . The noise introduced by π β results in incorrect learning goals, causing NBCU to make a mistake with probability 1 -η , which is the fraction of the noise in the union dataset. By a carefully designed transition probability, we can obtain the expected bound in Proposition 1.

5 Addressing Distribution Shift with Importance Sampling

In this section, we propose a data selection approach to alleviate the distribution shift issue between expert data and supplementary data. Our approach is inspired by recent works [25, 61], where a discriminator is trained to re-weight samples, and a weighted BC objective is used for policy optimization. Specifically, we define the weighted BC objective as follows:

where ̂ d U h ( s, a ) is the empirical state-action distribution of the union dataset, and w h ( s, a ) ∈ [0 , ∞ ) is the weight decided by the discriminator. We introduce a hyper-parameter δ ∈ [0 , ∞ ) to better select samples.

We propose using the importance sampling technique [46, Chapter 9] to transfer samples in the union dataset to the expert policy distribution, which is the key idea behind our method. This technique helps address the failure mode of NBCU. In an ideal scenario where there are infinite samples (i.e., at the population level), ̂ d U h would equal d U h . By setting w h ( s, a ) = d E h ( s, a ) /d U h ( s, a ) , we obtain ̂ d U h ( s, a ) w h ( s, a ) = d E h ( s, a ) , and the objective (3) enables the learning of a policy as if samples were collected by the expert policy. However, in practice, d E h ( s, a ) and d U h ( s, a ) are unknown, and we only have a finite number of samples from each of these distributions. Therefore, we must estimate the grounded importance sampling ratio d E h ( s, a ) /d U h ( s, a ) from the expert data and union data.

Weemphasize that estimating the probability densities of high-dimensional distributions separately for expert and union data and then calculating their quotient can be a challenging task. We take a different

approach. Inspired by [17], we directly train a discriminator to estimate the importance sampling ratio d E h ( s, a ) /d U h ( s, a ) . To this end, we introduce time-dependent parameterized discriminators { c h : S × A → [0 , 1] } H h =1 , each of which is optimized according to the objective function

Solving the optimization problem in (4) is equivalent to training a binary classifier that assigns positive labels to expert data and negative labels to union data. We can obtain the optimal discriminator at the population level, from which we can derive the importance sampling ratio formula:

Based on the previous discussion, we present the implementation of our proposed method, named ISW-BC (importance-sampling-weighted BC), in Algorithm 1. It is worth noting that ISW-BC employs an unbiased weighting rule since it directly estimates the importance sampling ratio. In contrast, previous approaches such as [25, 61] use regularized weighting rules that may fail to recover the expert policy even with infinite samples. For further details on the differences between our method and previous ones, please refer to Appendix B.

5.1 Negative Result of ISW-BC with Tabular Representations of Discriminator

We have not yet specified the representations of the discriminator. One natural choice is to use tabular representations, which correspond to linear function approximation with one-hot features. Tabular representations have a strong representation power since they can span all possible functions. However, surprisingly, we show that tabular representations can fail when considering generalization.

Proposition 2. If the discriminator uses the one-hot feature with δ = 0 , we have π ISW-BC = π BC .

Proposition 2 suggests that even if we have a large number of supplementary samples and use importance sampling, ISW-BC is not guaranteed to outperform BC based on tabular representations. To illustrate, suppose we have a sample ( s, a ) that is an expert-style sample but only appears in the supplementary dataset, meaning that d E h ( s, a ) > 0 , ̂ d E h ( s, a ) = 0 and ̂ d U h ( s, a ) > 0 . Using tabular representations, we can compute the closed-form solution c ⋆ h ( s, a ) = ̂ d E h ( s, a ) / ( ̂ d E h ( s, a ) + ̂ d U h ( s, a )) = 0 . This implies that the importance sampling ratio w h ( s, a ) = c ⋆ h ( s, a ) / (1 -c ⋆ h ( s, a )) = 0 , so this good sample does not contribute to the learning objective (3). The failure of tabular representations is due to their discrete treatment of data, ignoring internal correlations. Consequently, although they work well in minimizing the empirical loss, they are not good at generalization . This kind of failure mode is also mentioned in the GAN literature [4].

Algorithm 1 ISW-BC

Input: Expert data D E and supplementary data D S . U E S

2: c ← Train a binary classifier with positive labels for D E and negative labels for D U .

1: D ←D ∪ D .

3: w ← Compute importance sampling ratio by Eq. (5).

4: π ISW-BC ← Apply weighted BC to learn a policy by Eq. (3) with D U and w .

Output:

Policy π ISW-BC .

+/-

+/-

-

-

-

-

-

-

-

-

-

in-distribution

out-of-distribution

5.2 Positive Result of ISW-BC with Function Approximation of Discriminator

In this section, we address the issue raised in the previous section by investigating ISW-BC with a specific function approximation. To avoid the limitations of tabular representations, we consider that the discriminator is parameterized by c h ( s, a ; θ h ) = 1 1+exp( -⟨ ϕ h ( s,a ) ,θ h ⟩ ) , where ϕ h : S×A → R d is a fixed feature mapping and θ h ∈ R d is the parameter to be trained. Note that we require d < |S||A| to avoid the tabular representations. Let g ( x ) = log(1 + exp( x )) . Then, the optimization problem of

the discriminator becomes:

Let θ ⋆ = { θ ⋆ 1 , · · · , θ ⋆ H } be the optimal solution obtained from Eq. (6). With the feature vector, samples are no longer treated independently, and the discriminator can perform structured estimation. To be consistent with the previous results, the policy is still based on tabular representations.

In the context of general linear function approximation, it is no longer possible to obtain a closed-form solution for c ⋆ as in Eq. (5). This raises the question: what can we infer about c ⋆ ? Our intuition can be described as follows. We can envision the supplementary dataset containing two types of samples: some that were in-expert distribution, and others that were out-of-expert distribution. We expect that w h ( s, a ) is large in the former case and small in the latter case. Note that w h ( s, a ) is monotonic with respect to the inner product ⟨ ϕ h ( s, a ) , θ ⟩ . Therefore, we conclude that a larger value of ⟨ ϕ h ( s, a ) , θ ⟩ implies a more significant contribution to the learning objective (3). In the following part, we demonstrate that the aforementioned intuition can be achieved under mild assumptions.

Assumption 2. Let D S h denote the set of state-action pairs in D S in h . Define D S , 1 h = { ( s, a ) ∈ D S h : d π E h ( s ) > 0 , a = π E h ( s ) } as the in-expert-distribution dataset in D S h and D S , 2 h = D S h \ D S , 1 h as the out-of-expert-distribution dataset. There exists a ground truth parameter ¯ θ h ∈ R d , for any ( s, a ) ∈ D E h ∪ D S , 1 h and ( s ′ , a ′ ) ∈ D S , 2 h , it holds that ⟨ ¯ θ h , ϕ h ( s, a ) ⟩ > 0 and ⟨ ¯ θ h , ϕ h ( s ′ , a ′ ) ⟩ < 0 .

Readers may realize that Assumption 2 is closely related to the notion of 'margin' in the classification problem. Define ∆ h ( θ ) ≜ min ( s,a ) ∈D E h ∪D S , 1 h ⟨ θ, ϕ h ( s, a ) ⟩ -max ( s ′ ,a ′ ) ∈D S , 2 h ⟨ θ, ϕ h ( s ′ , a ′ ) ⟩ . From Assumption 2, we have ∆ h ( ¯ θ h ) > 0 . This means that there exists a classifier that recognizes samples from both D E h and D S , 1 h as in-expert-distribution samples and samples from D S , 2 h as out-of-expertdistribution samples. Note that such a nice classifier is assumed to exist, which is not identical to what is learned via Eq. (6). Before further discussion, we note that ¯ θ h is not unique if it exists. Without loss of generality, we define ¯ θ h as that can achieve the maximum margin (among all unit vectors).

Let us delve into the technical challenge that arises from Assumption 2. Although we assume two modes in the supplementary dataset, the learner is not aware of them beforehand. To gain a better understanding, refer to Figure 2, where the 'star' corresponds to the expert data and the 'triangle' corresponds to the supplementary data. The green and red parts of the triangle represent D S , 1 and D S , 2 , respectively. While training the discriminator, we assign positive labels (shown in '+') to the expert data and negative labels (shown in '-') to the union data. Consequently, it becomes challenging to determine the learned decision boundary theoretically. To address this challenge, we develop the landscape properties, Lipschitz continuity and quadratic growth conditions, in Lemma 1 and Lemma 2, respectively. These terminologies are from the optimization literature; see [23, 13]. Incorporating these properties will enable us to infer the learned decision boundary.

Lemma 1. For any θ ∈ R d , the margin function is L h -Lipschitz continuous in the sense that ∆ h ( ¯ θ h ) -∆ h ( θ ) ≤ L h ∥ ∥ ¯ θ h -θ ∥ ∥ , where L h = ∥ ϕ h ( s 1 , a 1 ) -ϕ h ( s 2 , a 2 ) ∥ with ( s 1 , a 1 ) ∈ argmin ( s,a ) ∈D E h ∪D S , 1 h ⟨ θ, ϕ h ( s, a ) ⟩ and ( s 2 , a 2 ) ∈ argmax ( s,a ) ∈D S , 2 h ⟨ θ, ϕ h ( s, a ) ⟩ .

Lemma 2. For any h , let A h ∈ R N tot × d be the matrix that aggregates the feature vectors of samples in D U h . Assume that rank ( A h ) = d , then L h (defined in Eq. (6) ) has a (one-sided) quadratic growth condition. That is, there exists τ h > 0 such that L h ( θ h ) ≥ L h ( θ ⋆ h ) + τ h 2 ∥ ∥ θ h -θ ⋆ h ∥ ∥ 2 .

Using Lemma 1 and Lemma 2, we are ready to obtain the imitation gap bound of ISW-BC.

Theorem 3. Under Assumptions 1 and 2, let µ = max ( s,h ) ∈S× [ H ] d π E h ( s, π E h ( s )) /d π β h ( s, π E h ( s )) < ∞ , if the feature is designed such that √ 2 ( L h ( ¯ θ h ) -L h ( θ ⋆ h ) ) τ h < ∆ h ( ¯ θ h ) L h holds, then we have ∆ h ( θ ⋆ h ) > 0 . Furthermore, we have the imitation gap bound E [ V ( π E ) -V ( π ISW-BC )] = O ( |S| H 2 N E + N S /µ ) .

In order to interpret Theorem 3, it is important to note that ∆ h ( θ ⋆ h ) > 0 means that there exists a δ > 0 such that w h ( s, a ; θ ⋆ h ) > δ for ( s, a ) ∈ D E h ∪D S , 1 h and w h ( s, a ; θ ⋆ h ) < δ for ( s, a ) ∈ D S , 2 h . As a result, all samples from D E h and D S , 1 h are assigned with large weights, which allows ISW-BC to make use of additional samples and outperform BC.

We remark that the imitation gap bound of ISW-BC is dependent on the number of expert-style state-action pairs presented in the union of D E h and D S , 1 h . This number is represented as N E + N S /µ , where µ is a state-action coverage parameter. It is important to mention that a similar notation is used in the literature of offline RL, as seen in [33, 10]. Additionally, ISW-BC has the ability to eliminate the gap of NBCU, meaning there is no non-vanishing error in Theorem 3. Moreover, ISW-BC can perform well even when D S , 2 h has noisy action labels, a scenario where NBCU may fail.

Although Theorem 3 produces desirable outcomes, it does have some limitations. First, the theoretical analysis necessitates knowledge of δ , which is typically challenging to determine beforehand. However, our empirical findings in Section 6 demonstrate that setting δ = 0 is effective in practice. Second, Theorem 3 mandates the use of good smooth features to ensure the required inequality holds, thereby avoiding the undesirable case presented in Proposition 2. Our paper does not offer a solution for finding such feature representations. Nevertheless, our experiments indicate that neural networks can usually learn suitable features. We present a simple mathematical example corresponding to Theorem 3 in Appendix D.5. We leave more general results of ISW-BC to future work.

6 Experiments

To validate the theoretical claims, we perform numerical experiments. We provide a brief overview of the experiment set-up below, and the details can be found in Appendix H.

6.1 Robotic Locomotion Control

In this section, we present our experiment on locomotion control, where we train a robot to run like a human in four environments from the Gym MuJoCo suite [14]: Ant , Hopper , Halfcheetah , and Walker . We adopt online SAC [19] to train an agent for each environment with 1M steps, and consider the resultant policy as the expert. For each environment, the expert data contains 1 trajectory collected by the expert policy. We consider two types of supplementary datasets:

Full Replay (small distribution shift): the supplementary dataset (1 million samples) is directly sampled from the experience replay buffer of the online SAC agent, which is suggested by [25]. This setting has a small distribution shift as the online agent quickly converges to the expert policy (see Figure 5 in the Appendix), resulting in abundant expert trajectories in the replay buffer.

Noisy Expert (large distribution shift): the supplementary dataset consists of 10 clean expert trajectories and 5 noisy expert trajectories where the action labels are corrupted (i.e., replaced by random actions). This introduces a large state-action distribution shift, as discussed in Remark 1. For further discussion on dataset corruption and distribution shift, please refer to Appendix E.2.

Besides our proposed methods, we also evaluate two state-of-the-art methods in the locomotion control domain: DemoDICE [25] and DWBC [61]. We report the experiment results in Table 2.

We observe that BC suffers since the amount of expert data is limited. In the full replay task, NBCU performs well due to the small distribution shift. However, in the noisy expert task, NBCU's average

performance is inferior to that of BC 5 , while ISW-BC outperforms NBCU in average significantly, demonstrating the robustness of ISW-BC to distribution shift. It is worth noting that among all evaluated methods, only our proposed method ISW-BC performs well in both settings. Prior methods such as DemoDICE and DWBC only perform well in one of the two settings.

6.2 Atari Video Games

In this section, we evaluate algorithms on Atari games [5], which involve video frames as inputs and discrete controls as outputs. Furthermore, environment transitions are stochastic for these games. We consider 5 games, namely Alien , MsPacman , Phoenix , Qbert , and SpaceInvaders . We obtain the offline expert data and supplementary data from the replay buffer of an online DQN agent, as released by [2]. We use the expert data from the buffer with the last index, which only contains 50k frames, to create a challenging learning setting. To augment this data, we use earlier replay buffer data to obtain supplementary data with approximately 200k frames. We consider the same baselines as in Section 6.1. All methods build on the classical convolutional neural networks used in DQN.

Similar to Section 6.1, we consider two types of supplementary data. The full replay setting involves supplementary data that is close to the expert data, exhibiting a small distribution shift. The noisy expert setting has noisy action labels, leading to a large distribution shift. Experiment details can be found in Appendix H.1.2. We report the game scores of the trained policies in Table 3.

Our observations are consistent with those of the previous experiments. NBCU performs well when the distribution shift is small, while only ISW-BC is robust when the distribution shift is large.

6.3 Image Classification

(c) Painting

(d) Quickdraw

(e) Real

(f) Sketch

TIGERWOODSByTNs

72

15

3

3

3

8

DAILYFILL

In our final experiment, we tackle an image classification task. This task is a special type of imitation learning where the planning horizon is 1 and there are no environment transitions. The reward is classification accuracy. Please note that our main purpose here is to use this degraded one-step task to verify the theoretical results.

We use a famous dataset, DomainNet [36], which comprises 6 sub-datasets ( clipart , infograph , painting , quickdraw , real , and sketch ) that have different feature patterns and hence distribution shifts; see Figure 3 for an illustration. Following [22], our task is to perform 10-class classification ( bird , feather , headphones , ice_cream , teapot , tiger , whale , windmill , wine_glass , and

zebra ) using 80% of the images for training and 20% for test. Each sub-dataset has roughly 2000-5000 images.

We build the classifier on the pretrained ResNet-18 [20], as directly training ResNet-18 on the DomainNet dataset failed. We then optimize a 2-hidden-layer neural network, where inputs are from the feature representations extracted by the pretrained and fixed ResNet-18. We create 6 sub-tasks, where one of the 6 sub-datasets is used as the expert data while the other 5 sub-datasets are used as the supplementary datasets. We evaluate the classification accuracy on the expert test data. Note that there is no natural extension of DemoDICE for this task. More details can be found in Appendix H.1.3.

The results of our experiment are presented in Table 4. We observe that due to the presence of distribution shifts, NBCU performs even worse than BC, even though NBCU use more data than BC. On the other hand, ISW-BC can improve the performance over BC on 5 out of 6 tasks by re-weighting the supplementary data. At the same time, ISW-BC is more effective than DWBC.

7 Discussion and Conclusion

This paper introduces a formal mathematical framework for imitation learning with a supplementary yet imperfect dataset, which is designed to address the scarcity of expert data. Within this framework, we present new theoretical insights that illuminate the distribution shift challenge between expert and supplementary data. To deal with this challenge, we devise a new method named ISW-BC, employing the importance sampling technique to select data within the expert distribution. Through both theoretical analysis and empirical evaluations, we show the superiority of the proposed approach.

Our research is closely connected with data-centric artificial intelligence (AI) [39, 54, 64]. Here, the emphasis lies in the quality, availability, and management of data as foundational elements for constructing effective AI models and applications. The importance sampling technique developed in this paper proves valuable for processing imperfect data.

Furthermore, our methods can extend beyond the scope of consideration in this paper. To illustrate, the core concept of data re-weighting and selection can find utility in the realm of large language models. In specific downstream tasks with limited expert data, we can judiciously select a set of pre-training tasks to construct supplementary data and to enhance overall performance of language models; for additional insights, please refer to recent progress in [58, 18]. Other potential avenues for future exploration include the extension to multi-task imitation learning, as well as the unsupervised case where the expert data is not available. 6

Overall, our findings demonstrate the potential of improving imitation learning performance by leveraging diverse data sources through effective data selection. We aspire for this work to serve as inspiration for future advancements in the field.

Acknowledgment

We thank Congliang Chen for discussing a technical lemma. The work of Yang Yu is supported by National Key Research and Development Program of China (2020AAA0107200), NSFC (61876077), and Collaborative Innovation Center of Novel Software Technology and Industrialization. The work of Zhi-Quan Luo is supported in part by the National Key Research and Development Project under grant 2022YFA1003900, and in part by the Guangdong Provincial Key Laboratory of Big Data Computing.

References

Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in Neural Information Processing Systems 33 , pages 20095-20107, 2020.

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning , pages 104-114, 2020.

Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems , 57(5):469-483, 2009.

Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning , pages 224-232, 2017.

Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279, 2013.

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Proceedings of the 36th International Conference on Machine Learning , pages 783-792, 2019.

Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning , pages 330-359. PMLR, 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 , pages 1877-1901, 2020.

Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. In Advances in Neural Information Processing Systems 34 , pages 965-979, 2021.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning , pages 1042-1051, 2019.

Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In Advances in Neural Information Processing Systems 23 , pages 442-450, 2010.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

Dmitriy Drusvyatskiy and Adrian S Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of Operations Research , 43(3):919-948, 2018.

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Proceedings of the 33rd International Conference on Machine Learning , pages 1329-1338, 2016.

Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In Advances in Neural Information Processing Systems 33 , pages 11996-12007, 2020.

Seyed Kamyar Seyed Ghasemipour, Richard S. Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Proceedings of the 3rd Annual Conference on Robot Learning , pages 1259-1277, 2019.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27 , pages 2672-2680, 2014.

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644 , 2023.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning , pages 1856-1865, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29 , pages 4565-4573, 2016.

Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. Outsourcing training without uploading data via efficient collaborative open-source sampling. In Advances in Neural Information Processing Systems 35 , pages 20133-20146, 2022.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the polyak-łojasiewicz condition. In Proceeddings of The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases , pages 795-811, 2016.

Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang, and Kee-Eung Kim. Lobsdice: Offline learning from observation via stationary distribution correction estimation. In Advances in Neural Information Processing Systems 35 , pages 8252-8264, 2022.

Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In Proceedings of the 10th International Conference on Learning Representations , 2022.

Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In Proceedings of the 7th International Conference on Learning Representations , 2019.

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 17(39):1-40, 2016.

Ziniu Li, Tian Xu, Yang Yu, and Zhi-Quan Luo. Rethinking valuedice: Does it really improve performance? arXiv preprint arXiv:2202.02468 , 2022.

Anqi Li, Byron Boots, and Ching-An Cheng. Mahalo: Unifying offline reinforcement learning and imitation learning from observations. arXiv preprint arXiv:2303.17156 , 2023.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence , 38(3):447-461, 2015.

Liu Liu, Ziyang Tang, Lanqing Li, and Dijun Luo. Robust imitation learning from corrupted demonstrations. arXiv preprint arXiv:2201.12594 , 2022.

Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. In Prooceedings of the 39th International Conference on Machine Learning , pages 14639-14663, 2022.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotic , 7(1-2):1-179, 2018.

Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning. arXiv preprint arXiv:1709.07174 , 2017.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1406-1415, 2019.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Machine Learning, Proceedings of the 24th International Conference on Machine Learning , pages 745-750, 2007.

Neoklis Polyzotis and Matei Zaharia. What can data-centric ai learn from data and ml engineering? arXiv preprint arXiv:2112.06439 , 2021.

Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation , 3(1):88-97, 1991.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, 2014.

Nived Rajaraman, Lin F. Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. In Advances in Neural Information Processing Systems 33 , pages 2914-2924, 2020.

Nived Rajaraman, Yanjun Han, Lin Yang, Jingbo Liu, Jiantao Jiao, and Kannan Ramchandran. On the value of interaction and function approximation in imitation learning. In Advances in Neural Information Processing Systems 34 , pages 1325-1336, 2021.

Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the 13rd International Conference on Artificial Intelligence and Statistics , pages 661-668, 2010.

Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Proceedings of the 9th International Conference on Learning Representations , 2021.

Alexander Shapiro. Monte carlo sampling methods. Handbooks in operations research and management science , 10:353-425, 2003.

Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. In Sixteenth European Workshop on Reinforcement Learning , 2023.

Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872 , 2021.

Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in Neural Information Processing Systems 20 , pages 1433-1440, 2007.

Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, Proceedings of the 25th International Conference on Machine Learning , pages 1032-1039, 2008.

Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation learning with diverse-quality demonstrations. In Proceedings of the 37th International Conference on Machine Learning , pages 9407-9417, 2020.

Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations. In Proceedings of the 38th International Conference on Machine Learning , pages 10961-10970, 2021.

Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the l1 deviation of the empirical distribution. Hewlett-Packard Labs, Techical Report , 2003.

Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal , pages 1-23, 2023.

Grover J Whitehurst and Ross Vasta. Is language acquired through imitation? Journal of Psycholinguistic Research , 4:37-59, 1975.

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In Proceedings of the 36th International Conference on Machine Learning , pages 6818-6827, 2019.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems 34 , pages 27395-27407, 2021.

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169 , 2023.

Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. In Advances in Neural Information Processing Systems 33 , pages 15737-15749, 2020.

Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In Prooceedings of the 39th International Conference on Machine Learning , pages 24725-24742, 2022.

Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. Understanding adversarial imitation learning in small sample regime: A stage-coupled analysis. arXiv preprint arXiv:2208.01899 , 2022.

Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. Provably efficient adversarial imitation learning with unknown transitions. In Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence , pages 2367-2378, 2023.

Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158 , 2023.

A Broader Impacts

This study delves into the theoretical aspects of offline imitation learning with supplementary data, and we verify our findings through experiments on established benchmarks. While this paper does not present any immediate, direct social impacts, the potential practical applications of our research could bring about positive change. By expanding the reach of imitation learning algorithms, our work may facilitate the development of more efficient and effective solutions in fields such as robotics, autonomous vehicles, and healthcare. However, we must also acknowledge that the misuse of such technology could have negative consequences, such as the manipulation of information to influence people's behavior. Therefore, it is crucial to remain vigilant in ensuring that the benefits of imitation learning are harnessed in a responsible and ethical manner.

B Additional Related Work

B.1 Imitation Learning from Imperfection

The problem considered in this paper is related to IL with a single imperfect dataset [56, 6, 51, 52, 45, 31]. In particular, the supplementary dataset in our set-up can also be viewed as imperfect demonstrations. However, our problem setting differs from IL with imperfect demonstrations in two key aspects. First, in IL with imperfect demonstrations, they either pose strong assumptions [51, 45, 31] or require auxiliary information (e.g., confidence scores on imperfect trajectories) on the imperfect dataset [56, 6]. In contrast, we assume access to a small number of expert trajectories to identify in-expert-distribution data. Second, most works [56, 6, 51, 52] in IL with imperfect demonstrations require online environment interactions while we focus on the offline setting. Additionally, in a related study, [7] employed imperfect data to infer a reward function. Under the assumption of the expert being strictly sub-optimal, [7] demonstrated that it is possible to learn a policy that surpasses the performance of the demonstrator by utilizing the recovered reward function.

Our research also bears relevance to the recent works by [29, 47], which aimed to unify offline Reinforcement Learning (RL) and IL. Different from our approach, these studies adopt a principle akin to adversarial imitation learning [50]. For a more detailed discussion, we refer readers to [29, 47]. We also observe a connection between the objective of importance sampling weighted behavior cloning (ISW-BC) and the reward-weighted regression (RWR) framework introduced in [38, 37]. Specifically, the training objective of ISW-BC can be likened to reward-weighted regression (RWR) with γ = 0 and a reward function denoted as r ( s, a ) = log d E ( s,a ) d U ( s,a ) . Nevertheless, there exist two distinctions between the two approaches. First, RWR primarily focuses on the online setting, while our work centers around the offline setting. Second, RWR finds its application in the context of RL, where rewards are readily available. Conversely, in our imitation learning setup, the reward (or the importance sampling ratio) needs to be inferred.

B.2 Difference with DemoDICE and DWBC

Our work builds upon previous research in IL with supplementary data, specifically the algorithms DemoDICE [25] and DWBC [61]. A significant distinction arises between ISW-BC and these two methods in terms of the weighting rule design. While DemoDICE and DWBC employ regularized weighting rules, our method directly estimates the importance sampling ratio. This fundamental difference can be critical as regularized weighting rules may struggle to recover the expert policy exactly even with infinite samples. We provide further elaboration on this point below.

First, DemoDICE also uses the weighted BC objective in Eq. (3). But, DemoDICE uses the weighting rule of ˜ w ( s, a ) ∝ d ⋆ ( s, a ) /d U ( s, a ) (refer to the formula between Equations (19)-(20) in [25]), where d ⋆ ( s, a ) is computed by the expert's state-action distribution matching objective regularized by a divergence to the union data distribution (refer to [25, Equations (5)-(7)]): 7

where γ ∈ [0 , 1) is the discount factor, α > 0 is a hyper-parameter. Due to the regularization term in the objective, it holds that d ⋆ ( s, a ) = d π E ( s, a ) , resulting in a biased weighting rule ˜ w ( s, a ) .

̸

Second, DWBC considers a different policy learning objective (refer to [61, Equation (17)]):

where α > 0 , λ > 0 are hyper-parameters, and c is the output of the discriminator that is jointly trained with π (refer to [61, Equation (8)]):

Since its input additionally incorporates log π , the discriminator is not guaranteed to estimate the state-action distribution. Thus, the weighting in Eq. (7) loses a connection with the importance sampling ratio.

C Proof of Results in Section 4

Recall the objective of BC in Eq. (1):

where ̂ d E h ( s, a ) = n E h ( s, a ) /N tot is the empirical state-action distribution in the expert dataset, and n E h ( s, a ) is the number of expert trajectories such that their state-action pairs are equal to ( s, a ) in time step h . With the tabular representations, we can obtain a closed-formed solution to the above optimization problem.

where n E h ( s ) ≜ ∑ a ′ n E h ( s, a ′ ) . Analogously, we also have a closed-form solution for NBCU in the tabular setting:

We will discuss the generalization performance of NBCU later.

In the proof, we frequently use the notation ≲ and ≳ . In particular, a ( n ) ≲ b ( n ) means that there exist C, n 0 > 0 such that a ( n ) ≤ Cb ( n ) for all n ≥ n 0 . In our context, n usually refers to the number of trajectories. For any two distributions P and Q over a finite set X , we define the total variation distance as

C.1 Proof of Theorem 1

When |D E | ≥ 1 , by [42, Theorem 4.2], we have the following imitation gap bound for BC:

When |D E | = 0 , we simply have that

Therefore, we have the following unified bound.

The last inequality follows that max { x, 1 } ≥ ( x +1) / 2 for any x ≥ 0 . Finally, notice that |D E | follows a binomial distribution by Assumption 1, i.e., |D E | ∼ Bin( N tot , η ) . By Lemma 3, we have that E [1 / ( |D| E +1)] ≤ N tot η , so which completes the proof.

C.2 Proof of Theorem 2

For analysis, we first define the mixture state-action distribution as follows.

By Assumption 1, in the population level, the marginal state-action distribution of union dataset D U in time step h is exactly d mix h . That is, d U h ( s, a ) = d mix h ( s, a ) , ∀ ( s, a, h ) ∈ S × A × [ H ] . Then we define the mixture policy π mix induced by d mix as follows.

From the theory of Markov Decision Processes, we know that (see, e.g., [41])

Therefore, we can obtain that the marginal state-action distribution of union dataset D U in time step h is exactly d π mix h . Then we have the following decomposition.

For V ( π E ) -V ( π mix ) , we have that

2

Bandit

The last equation follows the dual formulation of policy value (see, e.g., [41]), i.e., V ( π ) = ∑ H h =1 ∑ ( s,a ) d π h ( s, a ) r h ( s, a ) for any policy π . Besides, notice that E [ V ( π mix ) -V ( π NBCU ) ] is exactly the imitation gap of BC when regarding π mix and D U as the expert policy and expert dataset, respectively. Note that π mix may be a stochastic policy. By [42, Theorem 4.4], we have the following imtiation gap bound

Combining Eq. (11) and Eq. (12) yields that

C.3 Proof of Proposition 1

The hard instance in Proposition 1 builds on the Standard Imitation MDP proposed in [60]; see Figure 4 for illustration. For this MDP, each state is an absorbing state, i.e., P h ( s | s, a ) = 1 for any s and a . This property is mainly used to facilitate probability calculation and does not change the nature of our analysis. Furthermore, by only taking the action a 1 (shown in green), the agent can obtain a reward of +1 . Otherwise, the agent obtains a reward of 0 for the other action a = a 1 . The initial state distribution is a uniform distribution, i.e., ρ ( s ) = 1 / |S| for any s ∈ S .

̸

1

· · ·

|S|-

1

|S|

0

1

0

1

0

1

QRQGLYPH<c=16,font=/WDFZII+PingFangSC-Regular>H[SHUWGLYPH<c=3,font=/WDFZII+PingFangSC-Regular>DFWLRQ

%OXHGLYPH<c=3,font=/MYMHWG+PingFangSC-Semibold>DUURZGLYPH<c=29,font=/MYMHWG+PingFangSC-Semibold>

*UHHQGLYPH<c=3,font=/MYMHWG+PingFangSC-Semibold>DUURZGLYPH<c=29,font=/MYMHWG+PingFangSC-Semibold>

GLYPH<c=3,font=/WDFZII+PingFangSC-Regular>GLYPH<c=3,font=/WDFZII+PingFangSC-Regular>GLYPH<c=3,font=/WDFZII+PingFangSC-Regular>GLYPH<c=3,font=/WDFZII+PingFangSC-Regular>H[SHUWGLYPH<c=3,font=/WDFZII+PingFangSC-Regular>DFWLRQ

Weconsider that the expert policy π E always takes the action a 1 (shown in green) while the behavioral policy π β always takes another action a 2 (shown in blue). Formally, π E h ( a 1 | s ) = 1 and π β h ( a 2 | s ) = 1 for any s ∈ S and h ∈ [ H ] . It is direct to calculate that V ( π E ) = H and V ( π β ) = 0 . The supplementary dataset D S and the expert dataset D E are collected according to Assumption 1. The mixture state-action distribution (introduced in Appendix C.2) can be calculated as for any s ∈ S and h ∈ [ H ] :

Note that in the population level, the marginal distribution of the union dataset D U in time step h is exactly d mix h . The mixture policy induced by d mix (introduced in Appendix C.2) can be formulated as

Just like before, we have d π mix h ( s, a ) = d mix h ( s, a ) . The policy value of π mix can be calculated as

Recall from Eq. (9) that π NBCU can be formulated as

We can view that the BC's policy learned on the union dataset mimics the mixture policy π mix . In the following part, we analyze the lower bound on the imitation gap of π NBCU .

Then we consider the term E [ V ( π mix ) V ( π NBCU ) ] .

We take expectation over the randomness in D U on both sides and obtain that

For the first term in RHS, we have that

The last equation follows the fact that π NBCU h ( a | s ) is an unbiased estimation of π mix h ( a | s ) , so E [ π mix h ( a | s ) -π NBCU h ( a | s ) | n U h ( s ) > 0] . For the second term in Eq. (15), we have that

In the equation ( a ) , we use the fact that r h ( s, a 1 ) = 1 but r h ( s, a ) = 0 for any a = a 1 . In the equation ( b ) , since each state is an absorbing state, we have that P ( n U h ( s ) = 0) = P ( n U 1 ( s ) = 0) for any h ∈ [ H ] . We consider two cases to address RHS of equation (b). In the first case of η ≥ 1 / |A| , we directly have that

̸

By Eq. (15), we have that which implies that

In the second case of η < 1 / |A| , we have that

In the inequality ( a ) , we use that

The inequality ( b ) holds since we consider the range where N tot ≥ |S| log(2) . By Eq. (15), we have that

This implies that

In both cases, we prove that E [ V ( π E ) -V ( π NBCU ) ] ≳ (1 -η )( V ( π E ) -V ( π β )) and thus complete the proof.

D Proof of Results in Section 5

D.1 Proof of Proposition 2

In the tabular case, with the first-order optimality condition, we have c ⋆ h ( s, a ) = ̂ d E h ( s, a ) / ( ̂ d E h ( s, a ) + ̂ d U h ( s, a )) . By Eq. (5), we have

Hence, the learning objective (3) reduces to (1).

D.2 Proof of Lemma 1

Recall that

Then we have that

In inequality ( a ) , we utilize the facts that ( s 1 , a 1 ) ∈ argmin ( s,a ) ∈D E h ∪D S , 1 h ⟨ θ h , ϕ h ( s, a ) ⟩ and ( s 2 , a 2 ) ∈ argmax ( s,a ) ∈D S , 2 h ⟨ θ h , ϕ h ( s, a ) ⟩ . Inequality ( b ) follows the Cauchy-Schwarz inequality. Let L h = ∥ ∥ ϕ h ( s 1 , a 1 ) -ϕ h ( s 2 , a 2 ) ∥ ∥ and we finish the proof.

D.3 Proof of Lemma 2

First, by Taylor's Theorem, there exists θ ′ h ∈ { θ ∈ R d : θ t = θ ⋆ h + t ( θ h -θ ⋆ h ) , ∀ t ∈ [0 , 1] } such that

The last equality follows the optimality condition that ∇L h ( θ ⋆ h ) = 0 . Then, our strategy is to prove that the smallest eigenvalue of the Hessian matrix ∇ 2 L h ( θ ′ h ) is positive, i.e., λ min ( ∇ 2 L h ( θ ′ h )) > 0 . We first calculate the Hessian matrix ∇ 2 L h ( θ ′ h ) . Given D E and D U , we define the function G : R ( |D E | + |D U | ) → R as

where v i is the i -th element in the vector v ∈ R ( |D E | + |D U | ) and g ( x ) = log (1 + exp( x )) is a real-valued function. Besides, we use B h ∈ R ( |D E | + |D U | ) × d to denote the matrix whose i -th row B h,i = -y i ϕ h ( s i , a i ) ⊤ , and y i = 1 if ( s i , a i ) ∈ D E h , y i = -1 if ( s i , a i ) / ∈ D E h . Then the objective function can be reformulated as

Then we have that ∇ 2 L h ( θ h ) = B ⊤ h ∇ 2 G ( B h θ h ) B h , where

Here g ′′ ( x ) = σ ( x )(1 -σ ( x )) , where σ ( x ) = 1 / (1 + exp( -x )) is the sigmoid function. The eigenvalues of ∇ 2 G ( B h θ h ) are

Notice that θ ′ h ∈ { θ ∈ R d : θ t = θ ⋆ h + t ( θ h -θ ⋆ h ) , ∀ t ∈ [0 , 1] } . For a matrix A , we use λ min ( A ) to denote the minimal eigenvalue of A . Here we claim that the minimum of the minimal eigenvalues of ∇ 2 G ( B h θ t ) over t ∈ [0 , 1] is achieved at t = 0 or t = 1 . That is,

We prove this claim as follows. For any t ∈ [0 , 1] , we use { λ 1 ( t ) , . . . , λ |D E | + |D U | ( t ) } to denote the eigenvalues of ∇ 2 G ( B h θ t ) . For each i ∈ [ |D E | + |D U | ] , we consider λ i ( t ) : [0 , 1] → R as a function of t . Specifically,

We observe that g ′′′ ( x ) = σ ( x )(1 -σ ( x ))(1 -2 σ ( x )) which satisfies that ∀ x ≤ 0 , g ′′′ ( x ) ≥ 0 , and ∀ x ≥ 0 , g ′′′ ( x ) ≤ 0 . Therefore, we have that the minimum of λ i ( t ) over t ∈ [0 , 1] must be achieved at t = 0 or t = 1 . That is,

For any t ∈ [0 , 1] , we define i t ∈ [ |D E | + |D U | ] as the index of the minimal eigenvalue of ∇ 2 G ( B h θ t ) , i.e., λ i t ( t ) = λ min ( ∇ 2 G ( B h θ t )) . Then we have that

Equality ( a ) follows (17) and equality ( b ) follows that λ i 0 (0) and λ i 1 (1) are the minimal eigenvalues of ∇ 2 G ( B h θ 0 ) and ∇ 2 G ( B h θ 1 ) , respectively.

In summary, we derive that min { λ min ( ∇ 2 G ( B h θ t )) : ∀ t ∈ [0 , 1] } = min { λ min ( ∇ 2 G ( B h θ 0 )) , λ min ( ∇ 2 G ( B h θ 1 )) } , (18) which proves the previous claim.

Further, we consider λ min ( ∇ 2 L h ( θ h ) ) .

Here Im( B h ) = { z ∈ R d : z = B h x, ∥ x ∥ = 1 } . The last inequality follows Eq. (18).

Recall we assume that rank ( A h ) = d , so we have that rank ( B h ) = d . Thus, Im( B h ) is a set of vectors with positive norms, i.e., inf z ∈ Im( B h ) ∥ z ∥ > 0 . Besides, since g ′′ ( x ) = σ ( x )(1 -σ ( x )) > 0 , we also have that

In summary, we obtain that

λ min ( ∇ 2 L h ( θ h ) ) ≥ ( inf z ∈ Im( B h ) ∥ z ∥ ) 2 min { λ min ( ∇ 2 G ( B h θ 0 )) , λ min ( ∇ 2 G ( B h θ 1 )) } > 0 .

Then, with Eq. (16), there exists

such that

which completes the proof.

D.4 Proof of Theorem 3

First, invoking Lemma 1 with θ = θ ⋆ h yields that

Here L h = ∥ ϕ h ( s, a ) -ϕ h ( s ′ , a ′ ) ∥ with ( s, a ) ∈ argmin ( s,a ) ∈D E h ∪D S , 1 h ⟨ θ ⋆ h , ϕ h ( s, a ) ⟩ and ( s ′ , a ′ ) ∈ argmax ( s,a ) ∈D S , 2 h ⟨ θ ⋆ h , ϕ h ( s, a ) ⟩ . Then, by Lemma 2, there exists τ h > 0 such that

This directly implies an upper bound of the distance between θ h and θ ⋆ h .

If the feature is designed such that √ 2 ( L h ( ¯ θ h ) -L h ( θ ⋆ h ) ) τ h < ∆ h ( ¯ θ h ) L h holds, we further have that ∥ ∥ θ h -θ ⋆ h ∥ ∥ < ∆ h ( ¯ θ h ) /L h . Then we get that

which completes the proof of the first statement.

Then we proceed to prove the imitation gap bound. We first identify the property of π ISW-BC . Recall the objective of WBCU.

For any state s with ∑ a ∈A ̂ d U h ( s, a ) w h ( s, a ) I [ w h ( s, a ) ≥ δ ] > 0 , with the first-order optimality condition, we have

For an expert state s with d π E h ( s ) > 0 , if ( s, π E h ( s )) ∈ D E h ∪ D S , 1 h , we have that

This is due to the first statement that ∆ h ( θ ⋆ h ) > 0 in this theorem. Recall that

We can further obtain that w h ( s, π E h ( s )) > w h ( s, a ) for any ( s, a ) ∈ D S , 2 h . This implies that we can find a δ such that I [ w h ( s, π E h ( s )) ≥ δ ] = 1 for any ( s, π E h ( s )) ∈ D E h ∪ D S , 1 h and I [ w h ( s, a ) ≥ δ ] = 0 for any ( s, a ) ∈ D S , 2 h . Based on the above analytical form of π ISW-BC ,

we have that π ISW-BC ( π E h ( s ) | s ) = 1 for any ( s, π E h ( s )) ∈ D E h ∪ D S , 1 h . In summary, for any state s with ( s, π E h ( s )) ∈ D E h ∪ D S , 1 h , we have that π ISW-BC h ( π E h ( s ) | s ) = 1 .

With the above property of π ISW-BC , we proceed to analyze the policy value gap. According to [42, Lemma 4.3], we have

Since π E is assumed to be deterministic, we have

̸

Inequality ( a ) follows the property of π ISW-BC derived above. In particular, for any state s with ( s, π E h ( s )) ∈ D E h ∪ D S , 1 h , we have that π ISW-BC h ( π E h ( s ) | s ) = 1 . Equation ( b ) holds due to the Assumption 2. In particular, for an expert state s that d π E h ( s ) > 0 , the events of ( s, π E h ( s )) / ∈ D E h ∪ D S , 1 h and ( s, π E h ( s )) / ∈ D U h are equivalent.

Moreover, we take the expectation over D U on both sides and obtain that

According to Assumption 1, we have that

Inequality ( a ) follows the definition of µ in Theorem 3: for any ( s, h ) ∈ S × [ H ] , we have d π E h ( s, π E h ( s )) /d π β h ( s, π E h ( s )) ≤ µ . Then we obtain that

For each ( s, h ) ∈ S × [ H ] , we observe that

Here the last inequality follows Lemma 5. Consequently, we can derive that

which further implies that

We complete the proof.

D.5 An Example Corresponding to Theorem 3

In this section, we provide an example that illustrates the required feature design in Theorem 3 can hold.

Example 1. To illustrate Theorem 3, we consider an example in the feature space R 2 . In particular, for time step h ∈ [ H ] , we have the expert dataset and supplementary dataset as follows.

The corresponding features are

Notice that the set of expert-style samples is D E h ∪D S , 1 h = { ( s (1) , a (1) ) , ( s (2) , a (2) ) , ( s (4) , a (4) ) } and the set of non-expert-style samples is D S , 2 h = { ( s (3) , a (3) ) } . It is direct to calculate that the groundtruth parameter that achieves the maximum margin among unit vectors is θ h = ( - √ 2 / 2 , √ 2 / 2) ⊤ and the maximum margin is ∆ h ( θ h ) = √ 2 / 2 . According to Eq. (6) , for θ h = ( θ h, 1 , θ h, 2 ) ⊤ , the optimization objective is

We apply CVXPY [12] to calculate the optimal solution θ ⋆ h ≈ ( -0 . 310 , 0 . 993) ⊤ and the objective values L h ( θ ⋆ h ) ≈ 1 . 287 , L h ( θ h ) ≈ 1 . 309 . Furthermore, we calculate the Lipschitz coefficient L h appears in Lemma 1.

Then we calculate the parameter of strong convexity τ h appears in Lemma 2. Based on the proof of Lemma 2, our strategy is to calculate the minimal eigenvalue of the Hessian matrix.

First, for θ h = ( θ h, 1 , θ h, 2 ) ⊤ , the gradient of L h ( θ h ) is

Here σ ( x ) = 1 / (1 + exp( -x )) for x ∈ R is the sigmoid function. Then the Hessian matrix at θ h is

where f ( x ) = σ ( x )(1 -σ ( x )) and f ( x ) = f ( -x ) . For any t ∈ [0 , 1] , the eigenvalues of the Hessian matrix at θ t h = θ h + t ( θ ⋆ h -θ h ) are

Now, we calculate the minimal eigenvalues of ∇ 2 L h ( θ t h ) . We consider the function

The gradient is

We observe that ∀ x ≤ 0 , g ′ ( x ) ≥ 0 , and ∀ x ≥ 0 , g ′ ( x ) ≤ 0 . Thus, we have that the minimum of g ( x ) must be achieved at x = a or x = b . Besides, we have that g ( x ) = g ( -x ) . With the above arguments, we know that the minimal eigenvalue is g (0 . 993) ≈ 0 . 163 and τ h ≈ 0 . 163 . Then we can calculate that

The inequality in Theorem 3 holds.

E Discussion

In the main text, we focus on the tabular representations for policies. Furthermore, we consider a trajectory sampling procedure for behavior policy in collecting the supplementary dataset. We present two possible extensions in this section.

E.1 Function Approximation of Policies

In the main text, the theoretical analysis for BC-based algorithms considers the tabular setting in policy learning where a table function represents the policy. Here we provide an analysis of BC with general function approximation in policy learning. Notice that the algorithms considered in this paper (i.e., BC, NBCU and ISW-BC) can be unified under the framework of maximum likelihood estimation (MLE) 8 . Therefore, the theoretical results in the main text can also be extended to the setting of general function approximation by a similar analysis.

Assume that the learner is access to a finite function class Π = { π = ( π 1 , π 2 , . . . , π h ) } , where π h : S → ∆( A ) could be any function (e.g., neural networks). For simplicity of analysis, we assume that Π is a finite class. Notice that the algorithms considered in this paper are BC and its variants, which all take the principle of maximum likelihood estimation (MLE). The theoretical analysis of these algorithms is based on the following inequality:

Therefore, the key is to upper bound the TV distance. Take BC as an example (i.e., π = π BC ). By using the concentration inequality in [1, Theorem 21], we obtain that for any δ ∈ (0 , 1) , when

|D E | ≥ 1 , with probability at least 1 -δ over the randomness within D E ,

With additional efforts (by using union bound and Jensen's inequality), we have the following result.

Theorem 4 (BC with Function Approximation) . Under Assumption 1. In the general function approximation setting, additionally assume that π E ∈ Π . If we apply BC on the expert data, we have

where the expectation is taken over the randomness in the dataset collection.

The detailed proof is deferred to Appendix F. Compared with Theorem 1, we notice that the change in theoretical bound is that O ( |S| /N E ) is replaced by O ( √ log( | Π | HN E ) /N E ) . Such a change is expected for other algorithms (e.g., NBCU and WBCU), so our theoretical implications still hold. We leave the detailed analysis for future works.

E.2 Supplementary Data with Corruption

In the main text, we consider the trajectory sampling procedure in Assumption 1. However, in some cases, the supplementary data can be poisoned and corrupted by an adversary. For example, although the human expert demonstrates an optimal trajectory, the recorder or the recording system possibly corrupts the data by accident or on purpose. Data corruption is one of the main security threats to imitation learning methods [31]. Therefore, it is valuable to investigate the robustness of the presented algorithms in this poison setting. Supplementary data with corruption is partially investigated in our experiments under the noisy expert setting, which we argue have a large state-action distribution shift.

Assumption 3 (Poison Setting) . The supplementary dataset D S and expert dataset D E are collected in the following way: each time, with probability η , we rollout the expert policy to collect a trajectory. With probability 1 -η , we still rollout the expert policy to collect a trajectory but with probability 1 -η ′ , the actions along the sampled trajectory are replaced with actions uniformly sampled from the action space. Such an experiment is independent and identically conducted by N tot times.

Theorem 5 (NBCU in the Poison Setting) . Under Assumption 3. In the tabular case, for any η ∈ (0 , 1] , we have

where the expectation is taken over the randomness in the dataset collection.

Theorem 6 (ISW-BC in the Poison Setting) . Under Assumptions 2 and 3, if the feature is designed such that √ 2 ( L h ( ¯ θ h ) -L h ( θ ⋆ h ) ) τ h < ∆ h ( ¯ θ h ) L h holds, we have the imitation gap bound

Proofs of Theorem 5 and Theorem 6 can be found in Appendix F. Compared with the imitation gap of NBCU, there is no non-vanishing gap due to the corrupted actions in the imitation gap of ISW-BC. This means that ISW-BC is still robust in this setting.

F Proof of Results in Section E

F.1 Proof of Theorem 4

According to [42, Lemma 4.3], we have

With [1, Theorem 21], when |D E | ≥ 1 , for any δ ∈ (0 , 1) , with probability at least 1 -δ over the randomness within D E , we have that

With union bound, with probability at least 1 -δ , for all h ∈ [ H ] , it holds that which implies that

Inequality ( a ) follows Jensen's inequality. Taking expectation over the randomness within D E yields that

Equation a holds due to the choice that δ / E . For E , we directly have that

Therefore, for any |D E | ≥ 0 , we have that

( ) = 1 (2 |D | ) |D | = 0 E D E [ V ( π E ) -V ( π BC ) ] ≤ H.

We consider a real-valued function f ( x ) = log( cx ) /x for x ≥ 1 , where c = 4 | Π | H > 4 . Its gradient function is f ′ ( x ) = (1 -log( cx )) /x 2 ≤ 0 when x ≥ 1 . Then we know that f ( x ) is decreasing as x increases. Furthermore, we have that max {|D E | , 1 } ≥ ( |D E | +1) / 2 when |D E | ≥ 0 . Then we obtain

Taking expectation over the random variable |D E | ∼ Bin ( N tot , η ) yields that

Inequality ( a ) follows Jensen's inequality. We consider the function g ( x ) = -x log( x/c ) for x ∈ (0 , 1] , where c = 4 | Π | H .

Thus, g ( x ) is a concave function. By Jensen's inequality, we have that E [ g ( x )] ≤ g ( E [ x ]) . Then we can derive that

In inequality ( a ) , we use the facts that g ′ ( x ) ≥ 0 and E [ 1 / ( |D E | +1) ] ≤ 1 /N E from Lemma 3. We complete the proof.

F.2 Proof of Theorem 5

We first analyze the data distribution in D U . According to Assumption 3, we summarize the sampling procedure of trajectories in D U as follows. Each time, we rollout the expert policy to collect a trajectory. Furthermore, with the probability of (1 -η )(1 -η ′ ) , the actions along the sampled expert trajectory are replaced with actions uniformly sampled from the action space. Then we put this poisoned expert trajectory into D U . Otherwise, with the probability of 1 -(1 -η )(1 -η ′ ) , we directly put the original expert trajectory into D U . Therefore, we can formulate the marginal distribution of the state-action pairs in time step h in D U . For each ( s, a, h ) ∈ S × A × [ H ] ,

Then we proceed to analyze the imitation gap. Similar to the proof of Theorem 2, according to [42, Lemma 4.3], we have

Again, we introduce the definition of the policy π mix .

In particular, if d U h ( s ) > 0 , we have that

Then we decompose the imitation gap into two parts.

We first analyze the first term in RHS. For certain ( s, h ) such d U h ( s ) = d π E h ( s ) > 0 , we have that

̸

Therefore, we can derive that

Now we analyze the second term of

Recall the formula of π NBCU .

Notice that π NBCU is the maximum likelihood estimation of π mix . According to the concentration inequality of total variation [53], for each ( s, h ) ∈ S × [ H ] , for any fixed δ ∈ (0 , 1) , when n U h ( s ) > 0 , with probability at least 1 -δ , we have

When n U h ( s ) = 0 , we have that

By combining the above two inequalities, for each ( s, h ) ∈ S × [ H ] , with probability at least 1 -δ , we have

Applying union bound yields that with probability at least 1 -δ/ 2 , for all ( s, h ) ∈ S × [ H ] ,

Then we have that

̸

Here the last inequality follows Cauchy-Swartz inequality. Notice that n U h ( s ) is the number of times that the state s appears in D U in time step h and thus follows the Binomial distribution of Bin( N tot , d π E h ( s )) . By applying Lemma 4, for each ( s, h ) , with probability at least 1 -δ , we have

By union bound, with probability at least 1 -δ/ 2 , for all ( s, h ) ∈ S × [ H ] ,

Then, with probability at least 1 -δ , we have

Finally, we upper bound the imitation gap. With probability at least 1 -δ , we have

On the other hand, we directly have E [ V ( π E ) -V ( π NBCU )] ≤ H . We complete the proof.

F.3 Proof of Theorem 6

In the poison setting, we can conduct the same analysis as in the proof of Theorem 3 and demonstrate that π ISW-BC ( π E h ( s ) | s ) = 1 , ∀ ( s, π E h ( s )) ∈ D E h ∪ D S , 1 h , where D E h is the set of state-action pairs in D E in time step h and D S , 1 h = { ( s, a ) ∈ D S h : d π E h ( s ) > 0 , a = π E h ( s ) } . According to [42, Lemma 4.3], we have

Since the expert policy is assumed to be deterministic, we can obtain

̸

Let D S , clean denote the non-corrupted dataset in D S . Then we can obtain that

where D S , clean h denotes the set of state-action pairs in D S , clean in time step h . Inequality ( a ) follows that D S , clean h ⊆ D S , 1 h since D S , clean is collected by the expert policy. Taking expectation over the randomness in D E and D S , clean on both sides yields that

Notice that both D E and D S , clean are collected by the expert policy. Then if |D E | + |D S , clean | ≥ 1 , we can calculate that for each ( s, h ) ∈ S × [ H ] , where the last inequality follows Lemma 5. If |D E | + |D S , clean | = 0 , we directly have that

We unify the above two inequalities and get that

Now we proceed to upper bound the imitation gap.

Note that |D E | + |D S , clean | ∼ Bin( N tot , η + (1 -η ) η ′ ) . Taking expectation with respect to |D E | + |D S , clean | yields that

Inequality ( a ) follows Lemma 3. We finish the proof.

G Technical Lemmas

Lemma 3. For any N ∈ N + and p ∈ (0 , 1) , if the random variable X follows the binomial distribution, i.e., X ∼ Bin( N,p ) , then we have that

Proof.

Lemma 4 (Binomial concentration (Lemma A.1 in [57])) . For any N ∈ N + and p ∈ (0 , 1) , suppose X ∼ Bin( N,p ) . Then with probability at least 1 -δ , we have

Lemma 5. For any N ∈ N + and x ∈ [0 , 1] , consider the function f ( x ) := x (1 -x ) N , then we have

Proof. We calculate that f ′ ( x ) = (1 -x ) N -1 (1 -( N +1) x ) . It is direct to have that f ( x ) achieves its maximum at x ⋆ = 1 / ( N +1) . Furthermore, we have

Inequality ( a ) follows that (1 + x/N ) N ≤ exp( x ) , ∀ N ≥ 1 , | x | ≤ N . We complete the proof.

H Experiments Details and Additional Results

H.1 Experiment Details

In this section, we present the experiment details to facilitate the replication of our results. The experiments are conducted on a machine comprising 48 CPU cores and 4 V100 GPU cores. We repeat each experiment 5 times using different random seeds (2021, 2022, 2023, 2024, and 2025).

H.1.1 Robotic Locomotion Control

In this study, we evaluate the performance of various imitation learning algorithms on four locomotion control tasks from the MuJoCo suite: Ant-v2 , HalfCheetah-v2 , Hopper-v2 , and Walker2d-v2 . These tasks are widely used in the literature and are considered challenging benchmarks.

To train the expert policy, we use the online Soft Actor-Critic (SAC) algorithm [19] with 1 million training steps. We implement the algorithm using the rlkit codebase, which is available at https: //github.com/rail-berkeley/rlkit . The training curves of the online SAC agent are shown in Figure 5. We treat the resulting policy as the expert policy and use it to generate expert trajectories.

0.00

0.25

0.50

0.75

1.00

Interaction Step

1e6

0

1000

2000

3000

4000

5000

6000

Evaluation Return

0.00

0.25

0.50

0.75

1.00

Interaction Step

1e6

2500

0

2500

5000

7500

10000

12500

HalfCheetah-v2

0.00

0.25

0.50

0.75

1.00

Interaction Step

1e6

0

800

1600

2400

3200

4000

Hopper-v2

0.00

0.25

0.50

0.75

1.00

Interaction Step

1e6

0

1000

2000

3000

4000

5000

In our experimental setup, we utilize an expert dataset comprising of 1 expert trajectory collected by the trained SAC agent. Additionally, all algorithms are provided with a supplementary dataset. There are two setting of the supplementary data.

Full Replay . The supplementary dataset is obtained from the replay buffer of the online SAC agent, which has over one million samples, equivalent to 1000+ trajectories. The rapid convergence of online SAC, as illustrated in Figure 5, implies that the replay buffer is enriched with a substantial number of expert-level trajectories. As a result, we expect that utilizing the supplementary data without any modification may lead to desirable results.

Noisy Expert . The supplementary dataset comprises of 10 clean expert trajectories and 5 noisy expert trajectories. In this case, we replace the action labels in the noisy trajectories with random actions drawn from [ -1 , 1] . This replacement creates noisy action labels for the expert states, leading to a significant distribution shift at the state-action level, as noted in Remark 1. The high degree of distribution shift makes it challenging for using the supplementary data.

We use a 2-hidden-layer multi-layer perceptron (MLP) with hidden size 256 and ReLU activation for all algorithms, as the state information in locomotion control tasks is informative by design. The codebase of DemoDICE is based on the original authors' work, which can be accessed at https://github.com/KAIST-AILab/imitation-dice . For DWBC, we also use the authors' codebase, which is available at https://github.com/ryanxhr/DWBC . We experimented with different hyper-parameters for both algorithms but found that the default parameters provided by the authors work well. We normalize state observations in the dataset before training all algorithms, following [25]. This is crucial for achieving satisfactory performance.

In training the discriminator of ISW-BC, we use the gradient penalty (GP) regularization, as recommended by [25]. We add the following loss to the original loss (4) to enforce 1 -Lipschitz continuity:

where g is the gradient of the discriminator c ( s, a ; θ ) , and B is a mini-batch. This promotes the learning of smooth features and can improve generalization performance.

In our implementation of ISW-BC, we employ 2-hidden-layer MLPs with 256 hidden units and ReLU activation for both the discriminator and policy networks. We use a batch size of 256 and Adam optimizer with a learning rate of 0 . 0003 for training both networks. The training objective is to maximize the log-likelihood. We set δ to 0 and use a gradient penalty coefficient of 8 by default, unless otherwise stated. The training process is carried out for 1 million iterations. We evaluate the performance every 10k iterations with 10 episodes. The normalized score in the last column of Table 2 is computed in the following way:

H.1.2 Atari Video Games

We evaluate algorithms on a set of 5 Atari video games from the standard benchmark: Alien , MsPacman , Phoenix , Qbert , and SpaceInvaders . We preprocess the game environments using a standard set of procedures, including sticky actions with a probability of 0.25, grayscaling, downsampling to an image size of [84, 84], and stacking frames of 4. These procedures follow the instructions provided by the dopamine codebase, which is available at https://github.com/google/ dopamine/blob/master/dopamine/discrete_domains/atari_lib.py . The final image inputs are of shape (84, 84, 4).

We use the replay buffer data from an online DQN agent, which is publicly available at https: //console.cloud.google.com/storage/browser/atari-replay-datasets , thanks to the work of [2]. The dataset consists of 200 million frames, divided into 50 indexed buckets (ranging from 0 to 49). However, using the entire dataset is computationally infeasible 9 and unnecessary for our task. Therefore, we select specific buffer buckets for imitation learning.

We choose the expert data from bucket index 49, using only the first 400K frames for training. This makes the task challenging (we find that BC performs well with 1M frames of expert data). For the full replay setting, we select supplementary data from buffer indices 45 to 48, using the first 400K frames from each bucket. This yields a supplementary dataset that is 4 times larger than the expert data. In the noisy task setting, we follow the same procedure for selecting supplementary data, but replace the action labels with random labels on buffer index 45.

All agents employ the same convolutional neural network (CNN) architecture as the DQN agent, consisting of three convolutional blocks. The first block applies a filter size of 8, a stride of 4, and has a channel size of 32. The second block uses a filter size of 4, a stride of 4, and a channel size of 64, while the third block applies a filter size of 3, a stride of 4, and has a channel size of 64. All blocks use the ReLU activation function. The feature representations are flattened to a vector, on which a 1-hidden-layer MLP with a hidden size of 512 and ReLU activation function is applied. Finally, the outputs are passed through a softmax function to obtain a probability distribution.

Atari games are not considered in [25, 61] and public implementations of DemoDICE and DWBC for Atari games are not available. To use these methods in the Atari environment, we extend their original implementation by replacing the MLP used in locomotion control with the CNN described earlier. Implementing ISW-BC is a little more complicated. We use the same CNN policy network as in the other methods, but find that directly training the discriminator from scratch is less effective. This is because the discriminator tends to focus on irrelevant background information instead of the decision-centric part. To overcome this issue, we build the discriminator upon the feature extractor of the policy network, leveraging its ability to extract useful information. The discriminator is an MLP with ReLU activation and a hidden size of 1024: the image feature representation has a dimension 512 and the action feature representation also has a dimension 512 (we randomly project one-hot discrete actions to a 512-dimension space). We find that the depth of the MLP is crucial for performance, using a depth of 1 for the full replay setting and 3 for the noisy expert setting. We clip the importance sampling ratio for numerical stability, using a minimum value of 0 and a maximum value of 5 for the full replay setting, and a minimum value of 0.2 and a maximum value of 5 for the noisy expert setting. We provide ablation studies of these hyperparameters in Appendix H.2.2.

All methods were optimized using the Adam optimizer with a learning rate of 0.00025 and a batch size of 256. The training objective is to maximize the log-likelihood. The training process consisted of 200K gradient steps. Every 2K gradient steps, the algorithms were evaluated by running 10 episodes and computing the raw game scores. The normalized score in the last column of Table 3 is computed by Eq. (20).

H.1.3 Image Classification

Weutilize the publicly available DomainNet dataset [36] for our experiments, which can be accessed at http://csr.bu.edu/ftp/visda/2019/multi-source . This dataset comprises six sub-datasets: clipart , infograph , painting , quickdraw , real , and sketch , with 2103, 2626, 2472, 4000, 4864, and 2213 images, respectively. Our task involves recognizing objects from 10 different classes: bird , feather , headphones , ice_cream , teapot , tiger , whale , windmill , wine_glass , and zebra . We divided the images into training and test sets, with 80% for training and 20% for testing.

We employ a 2-hidden-layer neural network with a hidden size of 512 and ReLU activation as the classifier. To extract features from images, we utilize the pretrained ResNet-18 model (trained on ImageNet), which has a feature dimension of 512. The ResNet-18 model can be accessed at https://pytorch.org/vision/main/models/generated/torchvision.models. resnet18.html . We opted for this approach as training such a large convolutional neural network directly on the DomainNet dataset proved to be ineffective. The training objective is to minimize the cross-entropy loss. To optimize the network parameters, we use the stochastic gradient descent (SGD) optimizer with a learning rate of 0.01 and momentum of 0.9. Additionally, we apply

weight decay with a coefficient of 0.0005. The models are trained for 100 epochs with a batch size of 100, following the standard practice.

The discriminators used in ISW-BC and DWBC are implemented as 2-hidden-layer neural networks with ReLU activation. It's important to note that these discriminators take both the image and label as inputs. The image input is processed by the pre-trained and fixed ResNet-18, while the label input is projected to the same dimension (512) by a random projection matrix. The hidden size for the discriminator is set to 1024 for ISW-BC and 1025 for DWBC, as the discriminator in DWBC also takes the log-likelihood as an input. For ISW-BC, the discriminator is trained independently for 100 epochs with the same optimization configuration as the classifier. Afterward, the discriminator is fixed, and its output is used to compute the importance sampling ratio, which is then used to train the classifier.

H.2 Additional Results

H.2.1 Training Curves

The training curves on the robotic locomotion control tasks are displayed in Figure 6, Figure 7 and Figure 8. The training curves on Atari video games are displayed in Figure 9 and Figure 10. The training curves on the image classification task are displayed in Figure 11.

0

1500

3000

4500

6000

Evaluation Return

0

2500

5000

7500

10000

HalfCheetah-v2

BC

NBCU

DWBC

DemoDICE

ISW-BC

0.00

0.25

0.50

0.75

1.00

Iteration

1e6

0

800

1600

2400

3200

4000

Hopper-v2

0.00

0.25

0.50

0.75

1.00

Iteration

1e6

1000

0

1000

2000

3000

4000

5000

H.2.2 Ablation Study

In this section, we present ablation studies conducted on Atari games, aiming to provide insights into the underlying working scheme of our method. We specifically emphasize Atari games due to their high-dimensional image inputs, making these tasks particularly challenging. In contrast, the other two tasks, locomotion control and image classification, involve informative vector inputs, setting them apart from the unique characteristics of Atari games.

Ablation Study on Feature Representations of Discriminator Network. Our study reveals that employing a separate CNN for the discriminator yields inferior results compared to utilizing the feature extractor of the policy network. Please refer to Figure 12. Our conjecture is that training the discriminator independently may cause it to fit noise information (e.g., background). In contrast, the policy CNN network is capable of learning decision-centric information, enabling an effective approach to building the discriminator network through the feature extractor of the policy network.

3000

1500

0

1500

3000

4500

Evaluation Return

0

2500

5000

7500

10000

HalfCheetah-v2

BC

NBCU

DWBC

DemoDICE

ISW-BC

0.00

0.25

0.50

0.75

1.00

Iteration

1e6

0

800

1600

2400

3200

4000

Hopper-v2

0.00

0.25

0.50

0.75

1.00

Iteration

1e6

1000

0

1000

2000

3000

4000

5000

0

1500

3000

4500

6000

Evaluation Return

0

2500

5000

7500

10000

HalfCheetah-v2

BC

NBCU

DWBC

DemoDICE

ISW-BC

0.00

0.25

0.50

0.75

1.00

Iteration

1e6

0

800

1600

2400

3200

4000

Hopper-v2

0.00

0.25

0.50

0.75

1.00

Iteration

1e6

1000

0

1000

2000

3000

4000

5000

0

400

800

1200

1600

2000

Evaluation Return

Alien

0

600

1200

1800

2400

3000

MsPacman

BC

DWBC

DemoDICE

ISW-BC

NBCU

0

50000

100000

150000

200000

Iteration

0

500

1000

1500

2000

2500

3000

Phoenix

0

50000

100000

150000

200000

Iteration

0

2000

4000

6000

8000

10000

0

50000

100000

150000

200000

Iteration

0

150

300

450

600

750

SpaceInvaders

0

300

600

900

1200

1500

Evaluation Return

Alien

0

500

1000

1500

2000

2500

3000

MsPacman

BC

DWBC

DemoDICE

ISW-BC

NBCU

0

50000

100000

150000

200000

Iteration

0

400

800

1200

1600

2000

Phoenix

0

50000

100000

150000

200000

Iteration

0

2000

4000

6000

8000

10000

0

50000

100000

150000

200000

Iteration

0

150

300

450

600

SpaceInvaders

0

20

40

60

80

100

76

78

80

82

84

86

88

90

92

Test Accuracy

0

20

40

60

80

100

44

46

48

50

52

54

56

58

60

Test Accuracy

Infograph

BC

DWBC

ISW-BC

NBCU

0

20

40

60

80

100

78

80

82

84

86

88

90

92

94

Test Accuracy

Painting

0

20

40

60

80

100

Epoch

60

64

68

72

76

80

84

88

Test Accuracy

Quickdraw

0

20

40

60

80

100

Epoch

93.0

93.6

94.2

94.8

95.4

96.0

96.6

Test Accuracy

Real

0

20

40

60

80

100

Epoch

72

75

78

81

84

87

90

Test Accuracy

Sketch

600

900

1200

1500

1800

2100

Evaluation Return

Alien

500

1000

1500

2000

2500

3000

MsPacman

ISW-BC(shared_feature=false)

ISW-BC(shared_feature=true)

0

50000

100000

150000

200000

Iteration

0

500

1000

1500

2000

2500

3000

Phoenix

0

50000

100000

150000

200000

Iteration

3000

4500

6000

7500

9000

0

50000

100000

150000

200000

Iteration

150

300

450

600

750

SpaceInvaders

Ablation Study on Depth of Discriminator Network. We have discovered that the number of discriminator layers plays a crucial role in the performance of Atari games. The training curves, depicted in both Figure 13 and Figure 14, illustrate the performance variation based on the number of layers in the discriminator network. Notably, a 1-hidden-layer neural network yields the best results for the full replay setting, while a 3-hidden-layer neural network performs optimally in the noisy expert setting. It is important to note that this phenomenon is specific to Atari games. We do not have a good explanation yet. We believe this deserves further investigation in the future work.

600

900

1200

1500

1800

Evaluation Return

Alien

0

500

1000

1500

2000

2500

3000

MsPacman

ISW-BC(num_of_layers=1)

ISW-BC(num_of_layers=2)

ISW-BC(num_of_layers=3)

0

50000

100000

150000

200000

Iteration

0

500

1000

1500

2000

2500

3000

Phoenix

0

50000

100000

150000

200000

Iteration

0

2000

4000

6000

8000

10000

12000

0

50000

100000

150000

200000

Iteration

200

300

400

500

600

700

800

SpaceInvaders

H.2.3 Results on the Mix of Expert and Random Data

In the main text, we consider two types of supplementary datasets: full replay and noisy expert. Here we consider another type of supplementary datasets which consists of expert trajectories and trajectories generated by a random policy [25, 47]. We evaluate all methods on this new type of supplementary datasets on the robotic locomotion control tasks. In particular, the supplementary dataset contains 10 expert trajectories and 10 random trajectories. We report the experimental results on Table 5. The results show that ISW-BC outperforms NBCU significantly, demonstrating

300

600

900

1200

1500

Evaluation Return

Alien

0

500

1000

1500

2000

2500

3000

MsPacman

ISW-BC(num_of_layers=1)

ISW-BC(num_of_layers=2)

ISW-BC(num_of_layers=3)

0

50000

100000

150000

200000

Iteration

0

400

800

1200

1600

2000

Phoenix

0

50000

100000

150000

200000

Iteration

0

2000

4000

6000

8000

10000

0

50000

100000

150000

200000

Iteration

100

200

300

400

500

600

700

SpaceInvaders

the robustness of ISW-BC to distribution shift. Furthermore, ISW-BC also performs better than DemoDICE and DWBC.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of offline imitation learning with supplementary data with optimality not guaranteed. The paper gives theoretical analysis on the performance gap bound between expert policy and learner's policy for behavior cloning (BC) on expert data only and naively using BC over the union of expert and supplementary data. Based on the analysis, the paper proposes a provably better method than BC, which is ISW-BC. ISW-BC uses importance sampling (with a lower threshold) between state-action pairs to \"correct\" the learning from non-expert state-action occupancy onto expert state-action occupancy. In several mujoco and Atari environments, ISW-BC works comparably well or better than the SOTA methods, DemoDICE and DWBC.\n\nSTRENGTHS:\n**1. Good writing that is easy to follow and clearly conveys the idea.** The paper is a math-heavy one with many pages of theoretical proofs; however, the core result is well-summarized by Tab. 1, the theorems, and Eq. 3-5. Besides, for readers that are interested in theoretical results with function approximators, the theorems are kindly summarized in Sec. E in the appendix, leaving the long proof details in the next section. The limitations, broader impact and computational resource are all well-discussed. \n\n**2. Simple but effective idea.** The idea of ISW-BC proposed by the paper is very simple; it only requires separate training of a discriminator and an actor, which is quite easy to implement. However, such method is theoretically guaranteed to be better than BC and is indeed better than many baselines in multiple environments.\n\n**3. Solid theoretical and practical results.** The theoretical results clearly shows that how bad could it be when we are treating non-expert data as expert data in behavior cloning (BC), and how the quality (value function) of the non-expert data affects the result. Based on this, the author proposes ISW-BC, a method that is theoretically proved to be better, and indeed achieves superior performance over multiple baselines (DWBC, DemoDICE) on multiple environments (Atari, mujoco, and even non-RL tasks).\n\nWEAKNESSES:\n**1. The proposed method, ISW-BC, might be non-robust to stochastic environment.** Consider a simple tabular MDP with five states $s_{begin}, s_1, s_2, s_{success}, s_{fail}$; \n\nthe agent always begin at $s_{begin}$, and there is only one action for $s_{begin}$, which has 50% probability of leading to $s_1$ and 50% probability of leading to $s_2$; \n\nthere is only one action for $s_1$ that 100% leads to $s_{success}$, and two actions for $s_2$ that 100% leads to $s_{success}$ and $s_{fail}$ respectively;\n\n $s_{success}$ is the success state that terminates the episode with $+1$ reward; $s_{fail}$ terminates the episode with $-1$ reward.\n\nNow, consider the scenario where we only have one expert trajectory (1-shot is a common case) that goes from $s_{begin}$ to $s_1$ and finally $s_{success}$. The supplementary data acts uniformly random. By definition, the discriminator now gives a close-to-zero ratio for $d_h^E/d_h^U$ with any history that involves $s_2$ (let us ignore the numerical stability issue for now because there are engineering solutions). DICE methods have an equivalence of value function (which is the Lagrange dual function), which can guide the agent back from $s_2$ to $s_{success}$. ISW-BC, however, append no or very little weight to supplementary data on $s_2$, and, because expert has never experienced $s_2$, does not know what to do on $s_2$.\n\n**2. It strikes me as a little strange how we \"justify\" the use of weight in imitation with imperfect data from theoretical analysis**, because the story of the paper seems to be improve over NBCU (later proved to be empirically better than DICE/DWBC), but the NBCU analyzed in the paper is too unintuitive to ever work; it is natural, rather than with theoretical analysis for one to know that (s)he cannot treat non-expert data that is arbitrarily bad as expert ones. **(Despite of this, I am still convinced that the theoretical analysis on NBCU is a notable contribution.)**\n\n**3. Other minor problems:**\n\na) Besides the empirical advance on offline IL mentioned in the paper, there are more theoretical advances in offline IL (more specifically, the unification of offline IL and RL), which are MAHALO [1] and ReCOIL [2]. I encourage the author to briefly discuss them in the related work section.   \n\nb) The color of the curve for each method should be unified throughout the paper. For example, the color of ISW-BC in Fig.9 and Fig.10 in the appendix are not unified and might mislead the readers.\n\n  \n\n**References:**\n\n[1] A. Li et al. MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations. In ICML, 2023.\n\n[2] H. S. Sikchi et al. Imitation from Arbitrary Experience: A Dual Unification of Reinforcement and Imitation Learning Methods. In ArXiv, 2023.\n\nQUESTIONS:\nI have one question: can we actually comprehend ISW-BC as reward-weighted regression [1, 2] with $\\gamma=0$ and reward $r(s,a)=\\log\\frac{d^E(s,a)}{d^U(s,a)}$? If so, then I think more discussion is needed on the relationship between ISW-BC and RWR [1] / AWR [2].\n\nThe suggestions for the authors are listed as follows. I am open to increase my score if the authors can improve their paper as suggested:\n\n1. Modify the paper as suggested in the weaknesses (discussion of point 1, modification with point 2, 3) and limitation section;\n\n2. It would be great for the authors to try the supplementary data consists of some expert trajectories and many random trajectories or medium-level trajectories, like those tested in [3]. The component of the supplementary data can largely affect the performance of the algorithm [3];\n\n3. I strongly recommend the authors to open-source their code upon acceptance (or next submission).\n\n**References:**\n\n[1] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In ICML, 2007.\n\n[2] Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. X. Peng et al. in ArXiv, 2019.\n\n[3] H. S. Sikchi et al. Imitation from Arbitrary Experience: A Dual Unification of Reinforcement and Imitation Learning Methods. In ArXiv, 2023.\n\nLIMITATIONS:\n**Limitations:** The authors have discussed the theoretical limitations in line 280 and line 746-750. I think, however, there are more limitations that the authors could consider to add, and concentrated in a single limitation section:\n\n1) non-robustness to stochasticity (elaborated in point 1 of the weakness section);\n\n2) the assumption that $d^U$ covers $d^E$, which is also a weakness that DICE possesses but still a concern in practical use.\n\nNote these limitations do not necessarily mean that the work is not valuable enough for the conference; however, they do pose concern for readers who want to apply ISW-BC in the future.\n\n**Potential Negative Societal Impact:** The paper does a good job in discussing the broader impact at the beginning of the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides derivations of the imitation gap, the gap in performance between the trained agent and expert who provided the data, when traditional behavioural cloning (BC) is used. The specific setting assumes there is plentiful of supplementary data to train the BC agent on, but since this supplementary data may (and probably is) of poor quality (i.e., not as good as expert demonstrations), traditional BC produces less than optimal agents. The paper proposes a importance-sampling correction by training a discriminator on the dataset (to distinguish the high quality and low quality demonstrations), and then train BC using the importance-sampling correction. The results indicate the proposed method improves over BC and existing methods, while not reducing original BC performance in settings with high amount of expert data.\n\nSTRENGTHS:\n- Theoretical backing and derivation of the method.\n- Both theoretical and empirical improvements over baselines, and proposed method is more applicable than baselines (e.g., no natural extension of DemoDICE to image recognition task, as it lacks rewards).\n- Empirical results in three different settings (MuJoCo, Atari and object recognition).\n- Method is simple to implement. With a good baseline code base shared, I could see other people adapting this method and trying it out.\n\nWEAKNESSES:\n- Proposed method has rather small/noisy improvements in terms of metrics in the experiments.\n\t- In \"noisy expert\" setting, the proposed method is clearly better than the baselines. However this setting seems rather unrealistic (proper trajectories from a policy but actions are random). A more realistic scenario would be rollouts from a poorly trained policy, or a random agent.\n- Not a very novel setting (as evident by the number of baselines) and the solution, while well executed, is a combination of existing works in somewhat simple way.\n- Training discriminators may be problematic (which is a shared difficulty with baselines). Authors note this in the Appendix for the Atari experiments.\n- (Minor) No code available, but paper lists references to libraries and datasets used. Nevertheless, replicating the results as presented in the paper will be near-impossible, given the earlier works in ML field. I urge authors to share the code, even if it is \"messy\", so that others can build on the contributions of this work.\n\nQUESTIONS:\n1) What is the \"BC\" method shown in Tables 2, 3 and 4? Is this just \"BC\" row trained on the expert dataset only? If so, please clarify in main paper text.\n2) The \"one expert trajectory in expert dataset\" setup throughout experiments seems overly restrictive. Have you experiment with other settings, like with 5 or 10 expert trajectories in the expert dataset? Was it ensured that the \"one expert trajectory\" in the expert dataset was indeed a good demonstration? Even with a good policy, some trajectories may end up being bad.\n3)  In the object recognition task, the supplementary data is data with valid labels but with a different style. However, the focus is on \"sub-optimal policy\" data, and in other environments the supplementary data consists of different policies (\"full dataset\" setting) and outright wrong labels (\"noisy expert\"), where some actions are replaced with random ones. Following this, it seems like a more natural setup for object recognition task would be to do everything in single domain (e.g., \"Real\"), but randomly replace the sample label with a wrong label in the supplementary dataset. What is the reason behind using different sub-sets as supplementary data?\n\n### Comments\n\nNote: I do not have the expertise to comment on the mathematical derivations. I work on the assumption these are correct/throughout, and comment on the experimental setup and results.\n\n- Footnotes come after commas\n- Term \"imitation gap\" is used early on (e.g., lines 39 and 66), but not formally or informally defined. A quick definition after first use of the term would help readability.\n- Not required, but in future, please use same colors consistently between figures (e.g., same method always gets the same color). This improves paper readability quite considerably.\n- Section 3 header sounds like it is missing something (e.g., should it be \"Preliminaries\" or \"Problem setup\"?)\nNumbers in the tables are in bit different format; some (Random, Expert) seem to be in normal text mode, while rest are in math mode. Formatting should be consistent.\n\nLIMITATIONS:\nAuthors have broader impact section, and correctly report the limited societal impact. Authors list some of the limitations of the method in Appendix, e.g., difficulties regarding training discriminator in the Atari domain.\n\n## Rebuttal acknowledgement\n\nI have read authors' rebuttal and new results which did address my concerns, and I updated my score from 6 to 7 (before discussion period closed).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of offline imitation learning (IL) with a supplementary dataset, which can address the scarce expert data issue in pure IL. In this setting, the challenge is that the supplementary dataset may have out-of-distribution samples. This paper considers the classical method Behavioral Cloning (BC) and its variants, and proves their imitation gap bounds in offline IL with a supplementary dataset. The theoretical results show that the naïve BC on union dataset (NBCU) method suffers a non-vanishing gap, and thus may be worse than BC which only learns from the expert dataset. To address this issue, the authors propose the method Importance-sampling weighted BC (ISW-BC), which can select in-distribution samples in supplementary dataset. They prove that ISW-BC can eliminate the gap in NBCU. The experimental results also show that ISW-BC outperforms existing methods on a variety of tasks.\n\nSTRENGTHS:\n1. This paper conducts a systematic theoretical study of offline IL with a supplementary dataset. The developed theory closes the gap between theory and practice and lays a foundation for further studies of this problem.\n\n2. This paper proposes a simple and effective method ISW-BC. The authors validate that ISW-BC can address the distribution shift issue in both theory and practice, which makes advances over existing methods.\n\nWEAKNESSES:\n1. This paper is a bit dense to read. I believe that this paper would benefit from providing more intuitions and proof sketch for the theoretical results. Besides, the authors should give more analysis of the experimental results, which can give the reader an intuitive idea about how and where the proposed algorithm improves upon existing methods.\n\nQUESTIONS:\nThe theoretical results for ISW-BC in lines 260-270 are quite complicated and difficult to understand. Can the authors give more explanations and proof sketch for these results?\n\nLIMITATIONS:\nThe authors have discussed the limitations and broader impacts of this paper in the conclusion part and appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper, the authors focus on imitation learning (IL) when working with supplementary imperfect data. They conduct a thorough theoretical analysis to understand the limitations of IL under various dataset compositions. The authors' theoretical analysis provides insights into the bounds and constraints of IL when dealing with different types of datasets.\n\nTo address this problem, they propose a novel method called importance-sampling behavior cloning ISW-BC. The proposed method is designed to mitigate the issues associated with imperfect data in IL. This technique leverages importance sampling to assign appropriate weights to different samples, thereby effectively reducing the impact of imperfections in the training data.\n\nTo validate the effectiveness of their approach, the authors conduct extensive evaluations on a diverse set of tasks. The results indicate that the proposed method outperforms the current state-of-the-art techniques in most cases. This suggests that the importance-sampling behavior cloning method is a promising solution for tackling the problem of imitation learning with supplementary imperfect data.\n\nSTRENGTHS:\nOne strength of the paper is the authors' meticulous exploration of the various theoretical bounds that arise when working with imperfect data within the framework of BC. By dissecting these limitations, they provide a deep understanding of the challenges faced in practice, enabling researchers and practitioners to make more informed decisions when applying BC to real-world datasets.\n\nFurthermore, the authors introduce a novel method based on importance sampling, which offers a clear and intuitive approach for addressing the imperfections in the data. \n\nIn addition to their theoretical contributions, the authors demonstrate the practical relevance of their proposed method by conducting a thorough analysis on diverse tasks. This empirical evaluation validates the effectiveness of their approach across various application domains, further strengthening the paper's findings.\n\nWEAKNESSES:\nThe authors' analysis lacks consideration of alternative methods that can effectively learn from imperfect data. [1]\n\nIn order for the method to be applied, a dataset of labeled expert demonstrations is required. In many practical applications, we do not have access to this information.\n\n[1]Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations,Daniel S. Brown, Wonjoon Goo and Scott Niekum, CoRL 2019\n\nQUESTIONS:\nIn line 310, authors claim that NBCU performs worse than BC. This would be the expected results, however in Table 2 for Ant and HalfCheetah NBCU with Noisy Expert data performs significantly better than BC. Can the authors explain this in more detail?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}