Title: Data Quality in Imitation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Imitation Learning

Robotics

Data Quality
Abstract: In supervised learning, the question of data quality and curation has been sidelined in recent years in favor of increasingly more powerful and expressive models that can ingest internet-scale data. However, in offline learning for robotics, we simply lack internet scale data, and so high quality datasets are a necessity. This is especially true in imitation learning (IL), a sample efficient paradigm for robot learning using expert demonstrations. Policies learned through IL suffer from state distribution shift at test time due to compounding errors in action prediction, which leads to unseen states that the policy cannot recover from.
Instead of designing new algorithms to address distribution shift, an alternative perspective is to develop new ways of assessing and curating datasets. There is growing evidence that the same IL algorithms can have substantially different performance across different datasets. 
This calls for a formalism for defining metrics of "data quality" that can further be leveraged for data curation.
In this work, we take the first step toward formalizing data quality for imitation learning through the lens of distribution shift: a high quality dataset encourages the policy to stay in distribution at test time. We propose two fundamental properties that are necessary for a high quality datasets: 
i) action divergence: the mismatch between the expert and learned policy at certain states; and ii) transition diversity: the noise present in the system for a given state and action. We investigate the combined effect of these two key properties in imitation learning theoretically, and we empirically analyze models trained on a variety of different data sources. We show that state diversity is not always beneficial, and we demonstrate how action divergence and transition diversity interact in practice.

Data Quality in Imitation Learning

Abstract

In supervised learning, the question of data quality and curation has been overshadowed in recent years by increasingly more powerful and expressive models that can ingest internet-scale data. However, in offline learning for robotics, we simply lack internet scale data, and so high quality datasets are a necessity. This is especially true in imitation learning (IL), a sample efficient paradigm for robot learning using expert demonstrations. Policies learned through IL suffer from state distribution shift at test time due to compounding errors in action prediction, which leads to unseen states that the policy cannot recover from. Instead of designing new algorithms to address distribution shift, an alternative perspective is to develop new ways of assessing and curating datasets. There is growing evidence that the same IL algorithms can have substantially different performance across different datasets. This calls for a formalism for defining metrics of 'data quality' that can further be leveraged for data curation. In this work, we take the first step toward formalizing data quality for imitation learning through the lens of distribution shift: a high quality dataset encourages the policy to stay in distribution at test time. We propose two fundamental properties that shape the quality of a dataset: i) action divergence: the mismatch between the expert and learned policy at certain states; and ii) transition diversity: the noise present in the system for a given state and action. We investigate the combined effect of these two key properties in imitation learning theoretically, and we empirically analyze models trained on a variety of different data sources. We show that state diversity is not always beneficial, and we demonstrate how action divergence and transition diversity interact in practice.

1 Introduction

Supervised learning methods have seen large strides in recent years in computer vision (CV), natural language processing (NLP), and human-level game playing [18, 26, 43, 8, 51, 13, 44]. These domains have benefited from large and complex models that are trained on massive internet-scale datasets. Despite their undeniable power, biases present in these large datasets can result in the models exhibiting unexpected or undesirable outcomes. For example, foundation models such as GPT-3 trained on uncurated datasets have resulted in instances of racist behavior such as associating Muslims with violence [1, 5]. Thus offline data curation is immensely important for both safety and cost-effectiveness, and it is gaining in prominence in training foundation models [22, 10, 40].

Data curation is even more important in the field of robotics, where internet-scale data is not readily available and real-world datasets are small and uncurated. Noise or biases present in the data can lead to dangerous situations in many robotics tasks, for example injuring a person or damaging equipment. In such scenarios, deciding which data to collect and how best to collect it are especially important [37, 34, 4]. Of course, the quality of data depends on the algorithm that uses the data. A common paradigm in robot learning from offline datasets is imitation learning (IL), a data-driven, sample efficient framework for learning robot policies by mimicking expert demonstrations. However, when learning from offline data using IL, estimating data quality becomes especially difficult, since the 'test set' the robot is evaluated on is an entirely new data distribution due to compounding errors

incurred by the model, i.e., action prediction errors take the model to unseen states. This phenomenon is studied and referred to as the distribution shift problem, and prior work has viewed and addressed it through several angles [46, 49].

Broadly, prior work address distribution shift either by taking an algorithm-centric approach to account for biases in the dataset, or by directly modifying the dataset collection process in a datacentric manner. Algorithm-centric approaches learn robust policies by imposing task-specific assumptions [23, 28, 25], acquiring additional environment data [21, 41], or leveraging inductive biases in representing states [54, 30, 38, 12, 12, 37] and actions [53, 48, 12]. What these algorithm-centric works overlook is that changing the data can be as or more effective for policy learning than changing the algorithm. In prior data-centric methods, the goal is usually to maximize state diversity through shared control [46, 45, 16, 27, 32, 36, 47], noise injection [34], or active queries [15, 6, 42, 29]. By only focusing on state coverage, these works are missing the role that actions (i.e., the expert) plays in the quality of data. A more complete understanding of data quality that integrates both state and action quality would not only improve performance but also save countless hours of data collection.

To better understand the role of both states and actions in the data for learning good policies, consider a single state transition - the three core factors that affect distribution shift are: the policy action distribution, the the stochasticity of transitions, and the previous state distribution. Note that the previous states are also just a function of the policy and dynamics back through time. Through this we extract two fundamental properties that can influence data quality in IL: action divergence and transition diversity . Action divergence captures how different the learned policy action distribution is from the expert's actions at a given state: for example, if the expert is very consistent but the policy has high variance, then action divergence will be high and distribution shift is likely. Transition diversity captures the inherent variability of the environment at each state, which determines how the state distribution evolves for a given policy: for example, noisy dynamics in the low data regime can reduce overlap between the expert data and the learned policy distribution. Importantly, these factors can compound over time, thus greatly increasing the potential for distribution shift. While state coverage has been discussed in prior work, we are the first work to formalize the roles of both state and action distributions in data quality, along with how they interact through time: this new data-focused framing leads to insights about how to curate data to learn more effective policies.

To validate and study these properties empirically, we conduct two sets of experiments: (1) Data Noising, where we ablate these properties in robotics datasets to change the policy success rates, and (2) Data Measuring, where we observe human and machine generated datasets and approximately measure these properties in relationship to the policy success rates. Both experiments show that state diversity, a commonly used heuristic for quality in prior work, is not always correlated with success. Furthermore, we find that in several human generated datasets, less consistent actions at each state is often associated with decreases in policy performance.

2 Related Work

Data quality research dates back to the early time of computing and the full literature is out of the scope of this paper. Existing literature in machine learning has proposed different dimensions of data quality including accuracy, completeness, consistency, timeliness, and accessibility [33, 14, 20]. We could draw similarities between action divergence with the concept of data consistency, and state diversity with completeness. However, instantiation of these metrics is non-trivial and domain-specific.

Our work is closely related to the imitation learning literature that explicitly addresses the distribution shift problem in various ways. Prior work can be divided into two camps: ones that take an algorithmcentric approach to account for biases in the dataset, and ones that employ data-centric methods for modifying the data collection process.

Algorithm-centric. Robust learning approaches including model-based imitation learning methods learn a dynamics model of the environment and therefore can plan to go back in distribution when the agent visits out of distribution states [21, 41]. Data augmentation is a useful post-processing technique when one could impose domain or task specific knowledge [23, 28, 25]. Prior work has also investigated how to explicitly learn from sub-optimal demonstrations by learning a weighting function over demonstrations either to guide BC training [9, 3] or as a reward function for RL algorithms [7, 11]. These methods either require additional information such as rankings of trajectories [9], demonstrator identity [3], or collecting additional environment data [7, 11]. Recent efforts on demonstration retrieval augment limited task-specific demonstrations with past robot experiences [39]. A recent

body of work build better inductive biases into the model for learning robust state features, like pretrained vision or language representations [54, 30, 38, 12]. Some approaches modify the action representation to be more expressive to capture all the expert's actions, for example using Gaussian or mixture model action spaces [12, 37]. Others consider temporally abstracted action spaces like waypoints or action sequences to reduce the effective task length and thus mitigate compounding errors [53, 48, 12]. What these algorithm-centric works overlook is that changing the data can be as or more effective for policy learning than changing the algorithm. However, we still lack a comprehensive understanding of what properties in the data matter in imitation learning.

Data-centric. In more data-centric prior works that discuss data quality, the primary goal is often just to maximize state diversity. A large body of research focuses on modifying the data collection process such that the expert experience a diverse set of states. Ross et al. [46] proposed to iteratively collect on-policy demonstration data with shared-control between the expert and the robot, randomly switching with a gradually decreasing weight on the expert's input, such that the training data contains direct samples of the expert policy at states experienced by the learned policy. However, randomly switching the control can make it unnatural for the human demonstrator and leads to noisier human control. To mitigate this issue, methods have been proposed to gate the control more effectively by evaluating metrics such as state uncertainty and novelty [16, 27, 50]. Other methods allow the human to gate the control and effectively correct the robot's behavior only when necessary [32, 36, 24, 47]. Closely related to our work, Laskey et al. [34] takes an optimal control perspective and showed that injecting control noise during data collection can give similar benefits as DAgger-style iterative methods. Active learning methods have also been developed to guide data collection towards more informative samples [15, 6, 42, 29]. By only focusing on state coverage, these works are missing the role that actions (i.e., the expert) plays in the quality of data.

3 Preliminaries

In imitation learning (IL), we assume access to a dataset D N = { τ 1 , . . . , τ N } of N successful demonstrations. Each demonstration τ i consists of a sequence of continuous state-action pairs of length T i , τ i = { ( s 1 , a 1 ) , . . . , ( s T i , a T i ) } , with states s ∈ S and actions a ∈ A . Demonstrations are generated by sampling actions from the demonstration policy π E through environment dynamics ρ ( s ′ | s, a ) . The objective of imitation learning is to learn a policy π θ : S → A that maps states to actions. Standard behavioral cloning optimizes a supervised loss maximizing the likelihood of the state-action pairs in the dataset:

which is optimizing the following objective under finite samples from the demonstration policy:

The C term here captures the entropy of the demonstration state-action distribution, which is constant with respect to θ and thus it does not affect optimality of θ . ρ π E ( s ) is the state visitation of the demonstration policy, defined for any policy π as follows:

3.1 Distribution Shift in IL

Behavioral cloning methods as in Eq. (2) often assume that the dataset distribution is a good approximation of the demonstration policy distribution. In most applications, however, the learned policy is bound to experience novel states that were not part of the training data due to stochasticity of the environment dynamics and the learned policy. Herein lies the fundamental challenge with imitation learning, i.e., state distribution shift between training and test time. Consider the training sample ( s, a, s ′ ) at timestep t in demonstration τ i . If the learned policy outputs ˜ a ∼ π ( ·| s ) which has a small action error ϵ = ˜ a -a , the new state at the next time step will also deviate: ˜ s ′ ∼ ρ ( s ′ | s, a + ϵ ) , which in turn affects the policy output at the next step. In practice, the change in next state can be

highly disproportionate to || ϵ || , so small errors can quickly lead the policy out of the data distribution. Stochastic dynamics, for example system noise, can compound with the distribution shift caused by policy errors, and as we continue to execute for T -t steps, both these factors can compound over time to pull the policy out of distribution, often leading to task failure.

We can address this distribution shift problem by minimizing the mismatch between state visitation distribution of a learned policy π and the state visitation distribution of an demonstration policy π E under some f -divergence D f :

Next, we connect this distribution shift problem to the question of data quality.

3.2 Formalizing Data Quality

How can we define and measure the quality of a dataset D in IL? In existing literature, data quality has been used interchangeably with either the proficiency level of the demonstrator at the task or the coverage of state space. However, these notions of quality are loose and incomplete: for example, they do not explain what concretely makes a demonstrator better or worse, nor do they consider how stochasicity in the transitions impacts state space coverage and thus downstream learning. Our goal is to more formally define a measure of quality so that we can then optimize these properties in our datasets.

We posit that a complete notion of dataset quality is one that minimizes distribution shift in Eq. (5). This suggests that we cannot discuss data quality in isolation. The notion of data quality is heavily tied to the algorithm A that leads to the learned policy π A as well as the demonstration policy π E . We thus formalize the quality of a dataset D N conditioned on the demonstration policy π E that generates it along with a policy learning algorithm A , as the negative distribution shift of a learned policy under some f -divergence D f :

Here, π A is the policy learned by algorithm A using dataset D N of size N , which is generated from demonstrator π E . Note that quality is affected by several components: the choice in demonstrator, the choice of dataset size, and the algorithm. The choice of demonstration policy π E changes the trajectories in D N (which in turn affects policy learning) along with the desired state distribution ρ π E ( s )) . The dataset size controls the amount of information about the demonstration state distribution present in the dataset: note that ρπ A ( s ) should match ρ π E ( s ) , but A learns only from D N not from the full ρ π E ( s ) . The algorithm controls how the data is processed to produce the final policy π A and thus the visited distribution ρ π A ( s ) .

To optimize this notion of quality, we might toggle any one of these factors. Prior work has studied algorithm modifications at length [37, 54, 4], but few works study how π E and the dataset D N should be altered to perform best for any given algorithm A . We refer to this as data curation . In practice we often lack full control over the choice of π E , since this is usually a human demonstrator; still, we can often influence π E , for example through prompting or feedback [24], or we can curate the dataset derived from π E through filtering [19].

4 Properties of Data Quality

To identify properties that affect data quality in imitation learning, we can study how each state transition ( s, a, s ′ ) ∈ D in the dataset affects the dataset quality in detail. As we defined in Eq. (6), the dataset quality relies on distribution shift given an demonstration policy π E and an imitation learning policy π A . This relies on the state visitation distribution ρ t π A ( s ) , which based on Eq. (4) depends on three terms: π A ( a | s ) , ρ ( s ′ | s, a ) , and ρ t -1 π A ( s ) . Intuitively, three clear factors emerge for managing distribution shift: how different the policy π A ( a | s ) is from the demonstrator π E ( a | s ) - which we refer to as action divergence , how diverse the dynamics ρ ( s ′ | a, s ) are - which we refer to as transition diversity , and how these factors interact over time to produce past state visitation distribution ρ t -1 π A ( s ) . Importantly, the past state visitation can be controlled through action divergence and transition diversity at previous time steps, so in this section, we will formalize these two properties and discuss their implication on data curation.

4.1 Action Divergence

Action divergence is a measure of distance between the learned policy and the demonstration policy D f ( π A ( ·| s ) , π E ( ·| s )) . This can stem from biases in the algorithm or dataset such as mismatched action representations or lack of samples. While this is typically viewed in prior work as a facet of the algorithm or dataset size, importantly action divergence and thus data quality can also be influenced by the demonstration policy itself. For example, if the demonstrator knows the action representation used by the learning agent a priori , then the policy mismatch can be reduced by taking actions that are consistent with that action space. In Theorem 4.1 1 , we illustrate the importance of action divergence by showing that our notion of data quality is lower bounded by the action divergence under the visited state distribution.

Theorem 4.1. Given a policy π A and demonstrator π E and environment horizon length H , the distribution shift:

When using KL divergence for distribution shift in Eq. (5), we can see how the quality is lower bounded by the policy action divergence from the optimal policy under the visited state distribution, weighted at each step by time-to-go. Many prior works have noted the compounding error problem [46, 45], and here too action divergence at earlier timesteps has an out-sized effect on the overall distribution shift. Thus to optimize for the quality of a dataset, we should reduce the aggregated policy mismatch across visited trajectories.

Optimality : Many prior ideas around the 'optimality' of a demonstration are intimately related to this action divergence property. Suboptimalities that are naturally present in demonstrations like pauses or other action noise have been a source of difficulty when learning imitation policies [35, 4]. These factors make it harder for the model to learn the demonstration action distributions, thus increasing action divergence. Suboptimality can also come from multi-modal demonstration policy distributions, and more expressive policy representations (i.e., reducing action divergence between the demonstrator and the learned policy) has been shown to help [17, 12, 24]. The speed of demonstration is another common notion of optimality, which decreases total action divergence by reducing horizon length H , provided it does not increase the per time step action divergence to compensate.

State Visitation : Critically, the visitation distribution in Theorem 4.1 determines where action divergence should be low, whereas in the standard BC objective in Eq. (2), the divergence is only minimized under samples from the demonstration distribution ρ t π E . To better understand how the visitation distribution evolves, we now analyze state transitions in greater detail.

4.2 Transition Diversity

Transition diversity encompasses the diversity of next state transitions seen at a given state for a certain policy. What role does transition diversity play in minimizing distribution shift? Intuitively, if we consider the upper bound in Theorem 4.1, the demonstrator's state coverage should overlap as much as possible with the visited state distribution, but without increasing the action divergence.

Lemma 4.2. Given a learned policy π A and an demonstration policy π E , assume that the policy is learned such that when s ∈ supp( ρ t π E ) , D KL ( π A ( ·| s ) , π E ( ·| s )) ≤ β . Then:

✶ Lemma 4.2 shows that with a good learning algorithm (i.e., when β is small), it is important to maximize the overlap between the visited policy and the demonstration data (minimizes the contribution of the unbounded right term), but as shown in Theorem 4.1 this should not come at the expense of increasing the action divergence of the policy. Rather than broadly maximizing state diversity, as prior works do, a more sensible approach is for the demonstrator to maximize the two other factors that affect ρ t π A : system noise and initial state variance in the demonstrator data.

Finite Data and Low Coverage : While the above analysis holds for infinite data, maximizing transition diversity can lead to thinly spread coverage in the finite data regime. To analyze the finite data regime in isolation, we consider Gaussian system noise for simplicity. First, we formulate an empirical estimate of state 'coverage' of the demonstrator (i.e. the deviation from states in the demonstration dataset). We then show that higher system noise can cause the learned policy π A to deviate more from states sampled in the dataset.

Definition 1. For a given start state s , define the next state coverage probability over N samples from π E for tolerance ϵ as P S ( s ; N,ϵ ) := P (min i ∈{ 1 ...N } || s ′ -s ′ ∗ ,i || ∞ ≤ ϵ ) , where { s ′ ∗ ,i ∼ ρ t π E ( ·| s ) } is a set of N sampled next state transitions under the demonstration policy π E starting at s , and s ′ ∼ ρ t π A ( ·| s ) is a sampled next state under the policy π A starting at S .

We can think of P S ( s ; N,ϵ ) as the probability under potential datasets for π E of seeing a next state at test time (under π A ) that was nearby the next states in the dataset, conditioned at a starting state s (i.e., coverage). This is related to a single step of distribution shift, but measured over a dataset rather than a distribution. Defining coverage as the distance to the dataset being below some threshold is reasonable for function approximators like neural networks [2].

Theorem 4.3. Given a policy π A and demonstrator π E , assume that for state s, if ρ t -1 π E ( s ) > 0 , then π A ( a | s ) = π E ( a | s ) . Assume that transitions are normally distributed with fixed and diagonal variance, ρ ( s ′ | s, a ) = N ( µ ( s, a ) , σ 2 I ) , then the next state coverage probability is P S ( s ; N,ϵ ) ≥ 1 -( 1 -( cϵ σ ) d exp ( -α 2 d )) N -exp ( -( α -1) 2 d ) , where d is the dimensionality of the state, c is a constant, and α ≥ 1 is a parameter chosen to maximize the bound.

In Theorem 4.3, we see that even under a policy that is perfect when in distribution, the lower bound of next state coverage probability increases as the variance of system noise σ 2 decreases, for a fixed sampling budget N (and fixed α ). However, increasing N has a much stronger positive effect on this lower bound of state coverage than decreases in σ , suggesting that system noise is only an issue when the dataset size is sufficiently small. Furthermore, ϵ here represents some simplistic form of the generalization capacity of the policy, and we see here that increasing ϵ makes us more robust to proportional increases in σ .

Finite Data and Good Coverage : However, if we assume N is large enough to mitigate the coverage effect of system noise, are there any other effects of transition diversity in the dataset? The answer lies in generalization properties of the algorithm, which as provided in Theorem 4.3 are fairly conservative (i.e., in the definition of P S ( s ; N,ϵ ) ). In Theorem A.1 in Appendix A, we use a more loose generalization definition and relax the assumption that the learned policy is perfect. We show as system noise increases, the resulting boost in sample coverage can actually replicate and overcome the effects of high learned policy noise, suggesting that more system noise can actually be beneficial for learning. This finding sheds light on results from prior work and our own experiments in Section 5 that show the benefits of collecting data in the presence of high system noise [34].

4.3 Implications for Data Curation

We have shown how action divergence and transition diversity are both tied to distribution shift and thus data quality. Based on action divergence and transition diversity, we now examine downstream implications of these properties on what matters for data curation , where the goal is to collect and then select high quality demonstrations in our dataset D N for good policy learning.

Action Consistency : To minimize action divergence, the algorithm action representation should align with the demonstrator's action distribution, for the given dataset size. One algorithmic solution is to improve the expressiveness of the policy action space so it can capture the exact actions at every state that was demonstrated. However, in practice, individual states might only be visited a few times in the dataset, and so if the entropy of the demonstration policy at those states is high, the learned policy will find it difficult to perfectly match the actions even for the most expressive action spaces. Instead, we argue that demonstration data should be curated to have more consistent actions, e.g., reducing the entropy of the demonstration policy: E s ∼ ρ π A ( · ) [ H ( π E ( ·| s ))] . Since knowing the visited state distribution is impossible beforehand, the best we can do is to encourage low entropy in the demonstration data distribution:

Low:

State

High Action

High State

LowAction

Action Variance

Scripted

Coverage

consistency

Human

Coverage

consistency

State Diversity : As discussed previously, the state visitation of the policy depends on the trajectory action distribution, the initial state distribution, and the system noise distribution. Many prior works seek to improve the state coverage of the dataset, using some metric similar to Definition 1 [6, 29]. However, the required state diversity is a function of the learned policy mismatch (action divergence) and the noise present in the system (transition diversity), and is thus a coarse representation of data quality. Uniformly increasing coverage over the state space is not necessarily a good thing - as shown in Section 4.1, if state coverage comes at the expense of action consistency, the result might be worse than a policy trained on less state coverage but more consistent actions. Instead, we argue that we should optimize for system noise.

System Noise : Although system noise at a given state cannot be controlled, demonstrators can control system noise at the trajectory level (e.g. visiting states with more or less system noise). Should system noise be encouraged or discouraged in a dataset? Based on the discussion in Section 4.2, we hypothesize that when the dataset size is fixed, increasing the system noise can increase the coverage of the demonstrations and thus improve robustness in the learned policy (Theorem A.1), but only up to a certain point, when the data becomes too sparse to generalize (Theorem 4.3). Thus, in addition to consistent actions, we posit that demonstrators should encourage paths with high system entropy for learning better policies, such that the overall state entropy stays below some threshold γ to avoid coverage that is too sparse.

Horizon Length : The length of trajectories in the demonstration data also can have a large effect on the demonstration and visited state distributions, and thus on the quality of the dataset. However, while horizon length certainly plays a role, like state diversity, it is a downstream effect of action divergence and transition diversity. Based on previous analysis, what really matters is minimizing the aggregated action divergence and transition diversity produced by the demonstrator across time. Horizon length alone only crudely measures this, but is often correlated in practice as we show in Section 5.2.

While prior work in data curation primarily aim to maximize state diversity alone, our analysis of data quality reveals several new properties that matter for data curation, such as action consistency, system noise, and their combined effect over time. As we demonstrate in the next section, understanding these properties and the inherent tradeoffs between them is vital for data curation in imitation learning.

5 Analysis

To empirically analyze how different data properties affect imitation learning, we conduct two sets of experiments. In the first set, we generate various quality demonstration datasets by adding different types of noise to scripted policies, i.e., policies that are hand-designed to be successful at the task. An example scripted policy compared to human demonstrations is shown in Fig. 1. We study both noise in the system - considering transition diversity - and noise added to the demonstration policy - considering both transition diversity and action divergence - as a function of dataset size. In the second set of experiments, we evaluate the empirical properties (see Appendix B) for real human collected datasets.

SystemNoise(PolicyNoise=0)

Policy Noise (System Noise = 0)

Policy Noise (System Noise = 0.03)

- 00I

0,=0.01

0,=0.02

0,=0.04

100 -

[0'0=²0

0,=0.02

0=0.04

100 -

0,=0.01

0,=0.02

0′0=0

Point Mass Obstacle

90

-06

90-

80 -

80

80

do

+

10'0

0.02

0.03

0.04

0.01

0.02

0.03

0.04

0.01

0.02

0.03

0.04(

0.01

0.02

80'0

0.04

10'0

0.02

0.03

100

I0'0

0.02

0.03

0.04

(a)1000 demos

(b)10 demos

(e)1000 demos

(f)10 demos

(i)1000 demos

G)10 demoes

100 -

0=0.05

0=0.2

0,=0.4

100 -

0,=0.05

0,=0.2

0,=0.4

100 -

0=0.01

0,=0.02

0,=0.04

中

80 -

80 -

80 -

a ssss

60-

60-

60-

40 -

40

40-

20 -

20 -

20

do

+

0 -

0.1

0.2

0.3

0.4

0.1

0.2

0.3

0.4

0-

0.005

0.010

0.015

0.0200.005

0.010

0.015

0.020

0-

2000

0100

0.015

0.0200.005

0.010

0.015

0.020

(c)200 demos

(d)50 demoes

(g)200 demos

(h)50 demoes

(k)200 demos

(1)50 demoes

5.1 Data Noising

We train Behavior Cloning (BC) with data generated with system noise and policy noise in two environments: PMObstacle , a 2D environment where a point mass agent must reach a target point without hitting an obstacle in its way, where demonstrations are multimodal (left and right around the obstacle); and Square , a 3D environment where a robot arm must pick up a square nut and insert it onto a peg (shown in Fig. 1). In both, system and policy noise are Gaussian random variables added to the dynamics and scripted policy, respectively, at each time step, and BC uses an MLP architecture. Fig. 2 shows results in PMObstacle (top row) and Square (bottom row). We additionally show Transformer architecture results in Fig. 3.

State Diversity through Transition Diversity improves performance, up to a point . The left plots in Fig. 2 (a)-(d) show policy success rates as we vary system noise ( σ s ) (more noise for lighter color shades), where (a)(c) show the high-data regime and the (b)(d) show the low data regime. Higher system noise tends to improve policy performance in the high data regime, but only up to a point for Square -after σ s = 0 . 3 in the demonstration data (c), performance starts to drop off, which we hypothesize is due to the low coverage, based on the analysis of transition diversity in Section 4.2. In the low data regime, the story is similar but even more exaggerated, with increasing system noise leading to comparable performance as the high data regime. Once again for Square , increasing transition diversity (d) helps until the state coverage is too thin. See Table 2, Table 5, and Table 6 in Appendix C for the full sweep of system noise values in each environment.

State Diversity at the cost of Action Divergence hurts performance . Plots in Fig. 2 (e)(f) and Fig. 2 (i)(j) show policy success rates for increasing the policy noise ( σ p ) in the dataset, where the (e)(g) show the high data regime and the (f)(h) show the low data regime. Note that each value of policy noise yields the same demonstration state distribution as the corresponding amount of system noise. Since this noise is zero-mean and the learned policy is deterministic, policy noise in the high data regime (e)(g) is only moderately worse as compared to equivalent amounts of system noise (a)(c), suggesting that the action divergence is minor. In fact, due to the added state diversity, adding policy noise in the high data regime helps up to a certain point. However in the low data regime (f)(h), performance is substantially worse compared to comparable system noise, since the policy can not recover the unbiased demonstration policy from just a few noisy examples (note that in (c)(d), the x-axes are not aligned with those in (g)(h), and σ s = 0 . 2 corresponds to σ p = 0 . 02 as shown by the dotted line). This illustrates that state diversity coming from the noise in the demonstration policy

can increase action divergence, and thus high state diversity is not universally desirable. See Table 3, Table 7, and Table 8 in Appendix C for the full sweep of policy noise values in each environment.

Transition Diversity can counteract the effects of policy noise . In the right two plots (i)(j) for PMObstacle in Fig. 2, the dataset combines both system and policy noise. The system noise is fixed ( σ s = 0 . 03 ) and the policy noise is varied in the same range as the middle two plots (just policy noise). Given the analysis in Section 4.2 and Theorem A.1 in Appendix A, we would expect that having some system noise could actually make the policy much more robust to policy noise, since the state coverage provided in the dataset can help learn a more robust policy (generalization). Indeed we find that just by adding system noise, the learned policy becomes very robust to added policy noise in both the high (i) and low (j) regimes. This suggests that adding transition diversity can help mitigate the affects of action divergence incurred by noisy or sub-optimal demonstrators. See Table 4 in Appendix C for the full sweep of policy noise values for fixed system noise in each environment.

System Noise (Policy Noise = 0)

Policy Noise (System Noise = 0)

0=0.05

0=0.2

0=0.4

0=0.01

0=0.02

0=0.04

100

100 -

Obstacle

90

90

rate

success

80-

80-

Point Mass (

70-

70-

0.01

0.02

0.03

0.04

0.01

0.02

0.03

0.04

0.01

0.02

0.03

0.04

0.01

0.02

0.03

0.04

(b) 10 demos

(c)1000demos

(d) 10 demos

5.2 Data Measuring

Next we empirically instantiate several metrics from Section 4.3 that capture various facets of data quality, and measure different datasets: (1) action variance (higher means less action consistency) measured through clustering nearby states (2) horizon length T (higher means longer trajectories), and (3) state similarity (opposite of state diversity) measured as the average cluster size. See Appendix B for the exact empirical definitions. We leave out system noise since these human datasets were collected in deterministic environments.

In Table 1, we consider single and multi-human datasets from the Square and Can tasks from robomimic [37]. We see that higher action variance and horizon length are often accompanied by decreases in success rates. The Worse multi-human datasets have seemingly lower action variance but less state similarity, and yet qualitatively we observed highly multi-modal behaviors in these datasets (e.g., grasping different parts of the nut instead of the handle). We suspect that this type of multi-modality is not measured well by the single time step action variance metric, but rather requires a longer term view of the effects of that action on the scene - as indicated in Theorem 4.1. Additionally, state diversity once again is not correlated with performance on the task.

We emphasize that these metrics likely do not provide a complete picture of data quality, but they do provide us insights into why performance is low and how data collection can be improved in future datasets. Through future efforts from the community, we envision a comprehensive set of data metrics that practitioners can use to quickly evaluate the quality of datasets without needing to first train and evaluate models on those datasets.

6 Discussion

Data curation is incredibly important for maximizing the potential of any given algorithm, and especially so in imitation learning. To curate data, one needs to establish a formalism for assessing data quality. The predominant notions of data quality today are solely centered around maximizing state diversity in a dataset, which ignore the quality of the demonstrated actions and how these factors interplay. In this work, we propose a holistic view of data curation centered around minimizing distribution shift. We demonstrate that action divergence and transition diversity are crucial factors in data quality that can often be controlled or measured, and we draw several valuable insights from this shedding light on effective strategies for curating demonstration datasets. We find that making actions more consistent tends to increase policy success. Furthermore, we observe a fundamental tradeoff between state diversity and action divergence - increasing state diversity can often come at the expense of action divergence. Instead of uniformly maximizing state diversity, we show that increasing transition diversity improves performance until the data coverage becomes too sparse. While the metrics presented in our experiments are often good measures of final performance, we find that comprehensively measuring data quality in practice for real world datasets can be quite challenging. Finally, we envision a broader set of metrics informed by our formalism for data quality that practitioners can use to curate datasets, and we believe our work is an important first step in formulating and evaluating these metrics.

Limitations and Future Work: We hope our analysis will inspire future research in data quality for imitation learning. Firstly, we hope to relax several theoretical assumptions like Gaussian noise in future work. The metrics defined in Section 5.2 do not necessarily capture the full scope of factors that comprise data quality. Future work should refine these metrics or find alternate ways of measuring data quality without requiring running the algorithm first, or potentially with only limited algorithm training. While we show the performance of several simple metrics on two interesting control settings, future work is needed in determining how best to use these metrics for guiding data collection. One option is to use the metrics as a score function that demonstrators might use as feedback. Another is to explicitly filter or weight data using the metrics. Furthermore, it would be beneficial to test the insights in this paper on real world settings, for example under more realistic models of system noise than simply Gaussian noise. Nonetheless, we believe our work to be an important first step in defining and measuring data quality.

Fairness in robotics : As discussed in our introduction, foundation models like GPT-3 trained on uncurated datasets have learned harmful biases and stereotypes from uncurated datasets. Efforts in the broader ML community have started to focus on curation as a means to solve this problem. In the policy learning setting in robotics, fairness and safety are less clearly defined. Our primary focus in this paper is on data curation for learning high performing policies, but we believe future work should find analogous notions of fairness in the robotics setting, and to consider how metrics of data quality that we define will affect this notion of fairness. For example, we might want to ensure data is collected over a diverse range of institutions rather than in just a select few locations. Our action consistency metric might artificially select only demonstrators from a small range of institutions, and so this should also be considered during data curation.

Acknowledgments

This research was supported by ONR, DARPA YFA, Ford, and NSF Awards #1941722, #2218760, and #2125511. We also give special thanks to Stephen Tu for helping to streamline and improve several proofs for theorems in the text.

References

Abubakar Abid, Maheen Farooqi, and James Zou. Large language models associate muslims with violence. Nature Machine Intelligence , 3(6):461-463, 2021.

Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research , 12:149-198, 2000.

Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning , pages 1732-1748. PMLR, 2022.

Suneel Belkhale and Dorsa Sadigh. Plato: Predicting latent affordances through object-centric play. In 6th Annual Conference on Robot Learning , 2022.

Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible textto-image generation amplifies demographic stereotypes at large scale. ACM Conference on Fairness, Accountability, and Transparency , 2023.

Erdem Bıyık, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv:1910.04365 , 2019.

Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning , pages 330-359. PMLR, 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Zhangjie Cao, Zihan Wang, and Dorsa Sadigh. Learning from imperfect demonstrations via adversarial confidence transfer. In 2022 International Conference on Robotics and Automation (ICRA) , pages 441-447. IEEE, 2022.

Ting-Yun Chang and Robin Jia. Careful data curation stabilizes in-context learning, 2022.

Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning , pages 1262-1277. PMLR, 2021.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS) , 2023.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Corinna Cichy and Stefan Rass. An overview of data quality frameworks. IEEE Access , 7: 24634-24648, 2019.

Yuchen Cui and Scott Niekum. Active reward learning from critiques. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 6907-6914. IEEE, 2018.

Yuchen Cui, David Isele, Scott Niekum, and Kikuo Fujimura. Uncertainty-aware data aggregation for deep imitation learning. In 2019 International Conference on Robotics and Automation (ICRA) , pages 761-767. IEEE, 2019.

Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=c7rM7F7jQjN .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. In Proceedings of Robotics: Science and Systems (RSS) , 2023.

Lisa Ehrlinger and Wolfram Wöß. A survey of data quality measurement and monitoring tools. Frontiers in big data , page 28, 2022.

Peter Englert, Alexandros Paraschos, Marc Peter Deisenroth, and Jan Peters. Probabilistic model-based imitation learning. Adaptive Behavior , 21(5):388-403, 2013.

Hugging Face. Starcoder. https://github.com/bigcode-project/starcoder , 2023.

Alexandre Galashov, Josh S Merel, and Nicolas Heess. Data augmentation for efficient learning from parametric experts. Advances in Neural Information Processing Systems , 35:31484-31496, 2022.

Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, and Dorsa Sadigh. Eliciting compatible demonstrations for multi-human imitation learning. In Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022.

Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. In Conference on Robot Learning , pages 175-187. PMLR, 2023.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning , pages 598-608. PMLR, 2022.

Stephen James and Andrew J Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters , 7(2):1612-1619, 2022.

Hong Jun Jeon, Smitha Milli, and Anca D Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Conference on Neural Information Processing Systems , 2020.

Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. In Proceedings of Robotics: Science and Systems (RSS) , 2023.

Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha S. Srinivasa. Imitation learning as f-divergence minimization. CoRR , abs/1905.12888, 2019. URL http://arxiv.org/abs/1905.12888 .

Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Hgdagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8077-8083. IEEE, 2019.

Nuno Laranjeiro, Seyma Nur Soydemir, and Jorge Bernardino. A survey on data quality: classifying poor data. In 2015 IEEE 21st Pacific rim international symposium on dependable computing (PRDC) , pages 179-188. IEEE, 2015.

Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning , pages 143-156. PMLR, 2017.

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning , pages 1113-1132. PMLR, 2020.

Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human-in-the-loop imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733 , 2020.

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298 , 2021.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning , 2022. URL https://openreview.net/forum?id=tGbpgz6yOrI .

Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In 6th Annual Conference on Robot Learning , 2022.

Xu Ouyang, Shahina Mohd Azam Ansari, Felix Xiaozhu Lin, and Yangfeng Ji. Efficient model finetuning for text classification via data filtering, 2022.

Carl Qi, Pieter Abbeel, and Aditya Grover. Imitating, fast and slow: Robust learning from demonstrations via decision-time planning. arXiv preprint arXiv:2204.03597 , 2022.

Mattia Racca, Antti Oulasvirta, and Ville Kyrki. Teacher-aware active robot learning. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 335-343. IEEE, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022.

Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 661-668. JMLR Workshop and Conference Proceedings, 2010.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Mariah L Schrum, Erin Hedlund-Botti, Nina Moorman, and Matthew C Gombolay. Mind meld: Personalized meta-learning for robot-centric imitation learning. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 157-165. IEEE, 2022.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In 6th Annual Conference on Robot Learning , 2022.

Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872 , 2021.

Gokul Swamy, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Sequence model imitation learning with unobserved contexts. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=3nbKUphLBg5 .

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 , 2022.

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/9781108627771.

Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS) , 2023.

Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for visionbased manipulation with object proposal priors. 6th Annual Conference on Robot Learning (CoRL) , 2022.

A Theoretical Results

Theorem 4.1: Given a policy π A and demonstrator π E and environment horizon length H , the distribution shift:

Proof. Using the log-sum inequality:

Note that Ke et al. show a similar theorem, however they make the strong assumption of the fdivergence satisfying the triangle inequality, which is not true for the KL divergence we use in Theorem 4.1, and also yields a different bound. Furthermore, the implications of the divergence relationship on data quality (i.e. the data generating policy distribution) is not examined within this prior work. They are focusing more on the algorithmic perspective, as is common in prior work.

Lemma 4.2: Given learned policy π A and expert π E , define assume that the policy is learned such that when s ∈ supp ( ρ t π E ) , D KL ( π A ( ·| s ) , π E ( ·| s )) ≤ β . Then E s ∼ ρ t π A [ D KL ( π A ( ·| s ) , π E ( ·| s ))] ≤ E s ∈ ρ t π A [ β ✶ ( s ∈ supp ( ρ t π E )) + ✶ ( s / ∈ supp ( ρ t π E )) D KL ( π A ( ·| s ) , π E ( ·| s ))] Proof. This follows by simple substitution:

Theorem 4.3: Given a policy π A and demonstrator π E , assume that for state s, if ρ t -1 π E ( s ) > 0 , then π A ( a | s ) = π E ( a | s ) . Assume that transitions are normally distributed with fixed and diagonal variance, ρ ( s ′ | s, a ) = N ( µ ( s, a ) , σ 2 I ) , then the next state coverage probability is P S ( s ; N,ϵ ) ≥ 1 -( 1 -( cϵ σ ) d exp ( -α 2 d )) N -exp ( -( α -1) 2 d ) , where d is the dimensionality of the state, c is a constant, and α ≥ 1 is a parameter chosen to maximize the bound.

Proof. The proof follows by first discretizing the state space into length ϵ bins. Denote the probability mass of bin b as p b . First we note that for two independent samples from the same distribution s ′ and s ′ i, ∗ , we can say that P ( | s ′ -s ′ i, ∗ | ∞ ≥ ϵ ) is upper bounded by the probability that neither sample lands in the same bin. This is because of the fact that if | s ′ -s ′ i, ∗ | ∞ ≥ ϵ , then that implies the samples did not land in the same bin (if they did the infinity norm would be less than epsilon). Thus, we can also say that P (min i | s ′ -s ′ i, ∗ | ∞ ≥ ϵ ) is upper bounded by the probability that all of the samples s ′ i, ∗ land in a different bin than s ′ . Thus:

Next define a ball of radius R around the mean µ ( s, a ) as Ball( R ) . Using this ball we can partition the above inequality into two terms to yield an upper bound.

We can upper bound everything inside the ball using the minimum Gaussian mass for the given radius R , or p b ≥ ( cϵ σ ) d exp -R 2 2 σ 2 where c = 1 √ 2 π . For the second term, if R is large enough, then we can assume p b is sufficiently small such that (1 -p b ) N ≈ 1 .

The second term is upper bounded by the probability that the d -dim Gaussian s is farther in euclidean distance than R from the mean, which can be written using the tail probability of the χ 2 distribution.

We know that for a χ 2 random variable Y over d sub Gaussians, P ( Y d ≥ (1 + δ ) 2 ) ≤ exp ( -dδ 2 2 ) for δ ≥ 0 [52]. Thus assuming R ≥ σ √ d , we can write:

Now, for any α = R σ √ d ≥ 1 , we can rewrite the above bound as:

Starting from Definition 1:

A.1 Generalization under System Noise

Definition 2. Given a policy π A , a data generating policy π E and a starting state s , define the probability of next state coverage P B ( s, µ ; N ) = 1 - ∩ i P ( || s ′ i -µ || 2 ≥ || s ′ i, ∗ -µ || 2 ) , where s ′ i, ∗ ∼ ρ t π E ( ·| s ) are next state samples from the expert starting at s , and s ′ i ∼ ρ t π A ( ·| s ) are next state samples from π A starting at s .

Intuitively P B ( s ; N ) is the probability that given N chances, the hyper-sphere defined by the L2 distance from µ to a sampled data point contains a sample from the learned policy. Here the hypersphere represents the set of next states that the policy can generalize to. The N chances approximate the effect of having more data samples to leverage for generalization. For policies learned with neural networks, this aims to represent the 'interpolation' capacity of these models among the training samples.

Theorem A.1. Given a policy π A and deterministic demonstrator π E , assume that for state s, if ρ t -1 π E ( s ) > 0 , then π A ( a | s ) = N ( π E ( s ) , σ 2 π I ) . Assume that transitions are normally distributed with fixed and diagonal variance, ρ ( s ′ | s, a ) = N ( µ ( s, a ) , σ 2 I ) , where µ ( s, a ) = s + αa are simplified linear dynamics for scalar α ∈ R . Then the next state coverage probability is P B ( s, µ ( s, a ); N ) =

Proof. First we note that both ∆ i = s ′ i -µ ( s, a ) and ∆ i, ∗ = s ′ i, ∗ -µ ( s, a ) are zero mean, and the events for ∆ i ≥ ∆ i, ∗ for all i are independent. Thus:

Note that || ∆ i || 2 is just a χ 2 d random variable with d degrees of freedom, and likewise for || ∆ i, ∗ || 2 . The former has variance σ 2 + ( ασ p ) 2 (transition plus learned policy noise), while the latter has variance σ 2 (just transition noise). The f-distribution is defined for two χ 2 d variables X and Y of dimension d as the distribution of Z = X/d Y/d . Thus with a change of variables to X and Y , and denoting F d as the CDF of the f-distribution of dimension d :

Plugging the above expression into the expression for P B :

In Theorem A.1, while the probability of next state coverage is not immediately interpretable, we can still intuitively recognize that high σ s can improve the coverage likelihood of the learned policy even under notable σ p , and as N gets bigger even less σ s is needed, despite the fact that system noise is present even under the learned policy. Intuitively, we once again see that increasing N has a significant effect on the coverage likelihood. In terms of noise, what matters is the ratio of policy to system noise, where increasing this ratio leads to sharp drops in performance at some cutoff based on N . We visualize this coverage probability in Fig. 4 under increasing ratios of policy to system noise for different values of N .

B Metrics of Data Quality

Having formalized action divergence and transition diversity in Sec. 4 as two fundamental considerations in a dataset, how can we measure these properties in a given dataset?

Action Variance : To measure action consistency, the empirical form of the objective in Eqn. 7 is intractable without access to the underlying expert action distribution π E . Instead we propose using the empirical variance of the action distribution in the data to approximate the 'spread' of the data. In continuous state spaces, we can estimate variance using a coverage distance ϵ to cluster nearby states, and then measuring the per dimension variance across the corresponding actions within said cluster.

Defining a cluster to be C ( s, D ) = { ˜ s, ˜ a, ˜ s ′ ∈ D : || s -˜ s || ≤ ϵ } , we can compute the variance as:

N=100

N=50

N=10

N=2

0.5

1.5

2

2.5

3

3.5

4

The choice in ϵ corresponds to the generalization of the learning model to nearby states, similar to the notion of coverage in Definition 1. We use this metric of action consistency in Sec. 5 to study human generated datasets of various quality.

State Similarity : To measure the consistency of states, we approximate the number of 'nearby' states using the same clustering process as in the Action Variance metric, and measure the expected cluster size as a fraction of the overall data.

While these approximate forms do not encapsulate the full spectrum of possible metrics, we believe these metrics help advance our empirical understanding of data quality for imitation learning. In section 5.2 in the main text, we analyze these metrics of data quality in several environments across different dataset sources.

C Results

The performance results under system noise, policy noise, and both noises are shown with a broader sweep for both PMObstacle and Square in the tables below.

Peer reviews:
comments: SUMMARY:
The paper analyze several theoretical property of the expert demonstration distribution to the performance of the learnt policy in imitation learning. The paper claims that people should provide consistent and transition and state diversity.

STRENGTHS:
The paper defines several quantity to clear express the meaning of the properties. The theorems are sound to me.

WEAKNESSES:
The results claimed by the paper are straightforward to me, which makes the contribution less significant. The theoretical results are basically quick derivation of standard machinery of MDP analysis, which are trivial and straightforward.

The paper claims that expert should provide consistent demo based on BC analysis. However, the paper does provide experiment of the types of imitation learning methods like transformer architecture to handle multi-modal distribution.

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers how to evaluate  the quality of a given dataset for an imitation learning task.  It is well-known that IL suffers from distribution shift. While much work on IL focuses on improving algorithms, the authors propose that another important approach to making IL more effective is to curate better datasets and demonstrations from experts. Existing approaches to evaluating the quality of datasets focus on state coverage / diversity, trying to ensure that the expert demonstration contains instances of a variety of different states.\nThe authors discuss in some depth how state diversity is an incomplete measure of dataset quality, specifically because it ignores the actions taken in the dataset; the goal of the paper is to find an improved metric for evaluating dataset quality. he paper proposes action diversity and transition diversity.  The authors then consider  the distribution shift from the expert policy to the learned policy under a given f-divergence (KL divergence by default), as having a large distribution shift can cause problems in IL.\nThey then show a number of properties of their measure and relate it to action diversity and transition diversity , and discuss how these metrics can be applied to improve IL in general (4.3, Implications for Data Curation). They claim that their work suggests that action consistency in the expert demonstrations leads to better outcomes, while state diversty and system noise in the dataset are good or bad depending on context.\nFinally, they perform experiments on two simple simulationtasks, comparing the effects of adding policy noise to adding system noise in the training set and comparing how demonstration datasets of varying (known) quality correlate with various metrics.\n\nSTRENGTHS:\nThe main idea of finding improved metrics for dataset quality, above and beyond state space coverage, is interesting. Furthermore, the dichotomy that their analysis exposes between expanding state space coverage via higher action divergence versus higher transition divergence is an interesting one, showing that not all state space coverage is created equal. The theoretical results linking Q with action divergence are also interesting and well-presented.\nWhile there are questions left by the experiments (see Weaknesses and Questions) one extremely interesting finding is that the additon of a small amount of system noise vastly improves the effectiveness of training with policy noise (compare Figure 2 (c)(d) with Figure 2 (i)(j)), especially when the number of demos is low. The effect of this is so strong that it would have been better to focus more on it when discussing the results.\n\nWith a few minor exceptions, the paper is also very well written. The essential ideas are clearly stated and easy for the reader to absorb.\n\nWEAKNESSES:\nThis paper would be greatly improved by reducing repetition and adding more experiments.  The points made in the Intro are repeated at least 5 times.  The paper would be much better if the authors agree to remove this redundancy and add more thorough experiments.\n\nhe biggest weakness is a key property of the proposed measure (equation (6)), which is tied to a strength of the work (\"data quality is heavily tied to the algorithm A that leads to the learned policy \\pi_A as well as the expert policy \\pi_E\"): since it takes the learned policy \\pi_A as well as the expert policy \\pi_E, actually measuring the quality Q may require running A, which somewhat diminishes the value of the data quality metric since the it may not be usable when collecting the data (note that most common metric, state diversity in the training dataset, can be measured without running A). Even one of the proposed proxies for Q, action divergence, requires having the learned policy to measure against the expert policy.\nThere is an inconsistency in lines 196 and 199-200 with Theorem 4.1: it says the data quality is \"lower bounded by the action divergence\" but Theorem 4.1 shows that action divergence provides an upper bound to the data quality. Since upper and lower bounds for dataset quality have different implications this is very important to fix.\nWhile Figure 2 (i)(j) represents the most interesting finding (in my opinion) of this work, it doesn't escape notice that there is no corresponding Figure 2 (k)(l) showing the same plots for the Square nut task. These should absolutely be included to show whether or not the finding in (i)(j) generalizes to at least one other task (otherwise there is the possibility that (i)(j) is simply a property of one particular setting).\nThe authors do not include a limitations section.\nMinor weaknesses and typos:\nIn (3), the sum should go for t=1 to T, not H\nFigure 2 caption: In \"X-axis corresponds…\" should probably write \"injected system noise\" twice (rather than \"injected Gaussian noise\" the first time and \"injected system noise\" the second time). They're both system noise and both Gaussian, right? Writing it differently may convey the mistaken impression that they might not be both Gaussian, whereas the difference is just when the noise is added. Notation is also confusing, it seems \\sigma_s is used both to refer to system noise at evaluation (in the figure itself) and to system noise added in the training set (line 342).\nTable 1: instead of merely stating that state similarty is the \"opposite of state diversity\" (line 360), it would be much better to have a more precise definition. Additionally, it is claimed that \"state diversity once again is not correlated with performance on the task\" (line 370, just above Table 1) but Better/Okay/Worse all seem to have similar state similarity values while Proficient Human (PH) has lower state similarity (i.e. higher state diversity) and has significantly better success rates – doesn't this mean that it is correlated? Claims of no correlation should probably come with the actual correlation number, since it isn't obvious from the table that there really is no correlation.\nEquation (6) should probably be in a Definition since it's one of the key contributions of the work.\n4.3 Implications for Data Curation might be better placed after the experiments section.\n\nQUESTIONS:\nThis paper would be greatly improved by reducing repetition and adding more experiments.  The points made in the Intro are repeated at least 5 times.  The paper would be much better if the authors agree to remove this redundancy and add more thorough experiments.\n\nA number of the authors' conclusions (regarding what their results imply about creating and evaluating IL datasets) do not seem to follow obviously from their results and experiments. For instance, they claim in 4.3 (line 294-295): \"we posit that expert demonstrators should encourage paths with high system entropy [up to a point]\" – this doesn't really seem to follow from their experiments, in which system noise is added to a fixed policy, rather than changing the policy itself.\nThe authors claim to show that action divergence is undesirable and leads to worse outcomes, while transition diversity (in moderation) can improve results; this is demonstrated by adding policy noise during training and comparing the resulting policy to that learned from a dataset with system noise. However, this only shows that action divergence is harmful when it results from noise being artificially added (in which case the expert can be viewed as only partly trying to achieve the task, while also being partly random).\nWhat about the effect of action divergence resulting from multi-modality (i.e. when many very different courses of action can achieve the goal in a reasonable way)? It is not at all clear that the proposed measures of data quality will work for multi-modal tasks –  which have different solutions, e.g. even simple ones like moving around an obstacle (where you can go to the left or right).\nIt would be interesting to consider how this can be used to guide human expert demonstrations; while I'm not convinced by the discussion in section 4.3, at least without more details (see the first item under Questions), there are possibly other interesting avenues to consider, such as adding noise to the human expert's controls when conducting demonstrations.\n\nLIMITATIONS:\nThere is no Limitations section in the paper.\n\nThe limitations of the work are only very briefly addressed (lines 390-392) and need more expanding; in particular, the weakness mentioned (that the data quality metric proposed is very hard to measure in real datasets) is a serious one and deserves a much broader discussion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies imitation learning (IL) from a theoretical perspective. It proposes the idea that we should curate the data to improve performance, rather than solely relying on designing more robust algorithms, which is a more common practice in the field. The authors propose data quality for a particular IL algorithm, which is the negative of the divergence between the expert policy and the learned policy using that algorithm. They then show theoretically how two properties, action divergence and transition diversity, are linked to data quality. The authors then study the finite-data regime under a very simplistic setting, and finally show some results of studying their metrics on PMObstacle, Square, and Robomimic domains.\n\nSTRENGTHS:\nThe presentation / quality of the writing is top-notch. I really appreciated the clear descriptions throughout the paper, starting from the well-written introduction to the explicit problem setting to the way the authors use English to describe and contextualize all the math.\n\nRelated work section is comprehensive. I like how it addresses both algorithmic and data robustness.\n\nI also really like the framing of Equation 6. The insight that data quality is a function of the learning algorithm is very nice.\n\nWEAKNESSES:\nI am unfortunately not well-versed in IL theory, so I cannot comment on the novelty of the work with respect to the rest of the field. Therefore, I will only evaluate the paper on the merits of the contributions stated by the authors.\n\nMy only major weakness is about the unrealistic conditions underlying some of the theory.\n* One of the conditions of Lemma 4.2 is that states in the support of the expert policy would have low divergence between actions under the expert policy and the learned policy. The authors say in L226 that with a good learning algorithm, this bounding constant \\beta would be small, but I don't see why this would be true in general. For BC, maybe it makes sense, but for other IL algorithms, a \"good\" algorithm could sacrifice exactly matching the expert at some states, for the sake of getting better interpolation/generalization to states outside of the support of the expert policy.\n* Theorem 4.3 studies only a very simple setting where the environment dynamics are fixed Gaussian with diagonal variance. I'm not sure how much we can take away from this Theorem as this condition will almost never hold in practice. As a consequence, I find takeaways like that in L254 to be misleading because one cannot draw general conclusions from such a restricted Theorem.\n\nOther comments:\n* One more broad concern I have is that most of the robotics domains studied in the RL literature have deterministic dynamics, including Robomimic (ref: https://arxiv.org/abs/2305.14550) and Mujoco. So how can we obtain transition diversity in such settings?\n* Theorem 4.1 only makes sense with stochastic policies due to the KL divergence. It's not clear how to apply these findings to deterministic policies, which are quite common (and in fact, are used in the experiments of this paper).\n* L230, L290: It's unclear what the authors mean when they say the data should maximize system noise. That's a property of the environment dynamics, not the expert policy; we don't have control over it!\n* It is hard to gain too many insights from Table 1. A simpler explanation for the observed success rates is that as task horizon increases while data size stays fixed, BC simply has less data for each state, leading to worse policies. I'm not really sure why action variance and transition diversity come into play here.\n\nMinor things:\n* The authors use the word \"expert\" throughout the paper. I would rather say the \"behavior policy\" or \"data generation policy\", because the whole point of this paper is that we are trying to curate the dataset, e.g. by introducing more diversity, which means the data may not end up looking like what an expert would do.\n* It would be good to say in the preliminaries that you're in a continuous state/action space setting.\n* L73: remove \"how\"\n* L167: \"\\pi_A should match \\rho_{\\pi_E}(s)\" --> I think you mean \"\\rho_{\\pi_A} should match \\rho_{\\pi_E}\"?\n* L167: \"learns only\" --> \"learns only from\"\n* Lemma 4.2 RHS, subscript of the expectation: \\in should be \\sim?\n\nQUESTIONS:\nL147: I don't see why it's a \"test time\" objective. You would be optimizing your policy with this f-divergence loss, so it comes into play during training time, right?\n\nTheorem 4.1: What does it mean to have a superscript on D_KL? Is this term the same as the LHS of Lemma 4.2?\n\nDo any of the intuitions in this paper transfer to state-only IL, where our dataset doesn't have actions at all? This is becoming an increasingly common setting in the new world of large-scale pre-training (e.g., from human videos).\n\nLIMITATIONS:\nYes, addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
The paper addresses a key issue in Imitation Learning (IL) called as distribution shift. The paper formalize data quality and data curation for IL. The paper propose two fundamental properties to evaluate/measure data quality as "action divergence" and "transition divergence". The authors investigate these two properties in context of data quality for IL and presented theoretical and empirical results. They conducted an experiment on human/expert demonstration data and provide analysis on data noising and data measuring.

STRENGTHS:
- The paper is well-written
- The idea presented in the paper is very important for IL

WEAKNESSES:
- The work doesn't seems to fit for a high quality conference like NeurIPS
- On measuring the data quality and its effect on success rate, a lot of research is being done on learning from mixed data as its really hard to classify expert vs non-expert data.
- The paper seems more like a white paper or can be submitted to AI Safety conferences but not for NeurIPS
-

QUESTIONS:
- addressed in Weakness section

LIMITATIONS:
- Authors are encouraged to explicity mention the limitations of the paper
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nData quality is an important aspect of offline robotics because there is not a large quantity of data. In particular, in imitation learning, it has been shown that naive training policies on low-quality data can lead to distribution shift issues. Most researchers focused on dealing with this issue using algorithm-centric approaches. Instead, this paper advocates for a data-centric approach for addressing this issue. The data-centric approach modifies the data collection process, where the data collection can be broken down into two categories: action divergence and transition diversity. The authors formalize these two categories of the data collection process and provide empirical evidence of their significance.\n\nSTRENGTHS:\nFormalizing and addressing the data quality issues in imitation learning is challenging. This paper proposed interesting ideas to this challenging problem and has the following strengths:\n- Clarity: The overall paper is well written, and the problem the authors would like to address is clear. The two categories of data collection are also very clear and make it easy to understand the underlying problem dataset attributes.\n- Empirical Justification: The author's performance experience using synthetic noise, single and multi-human datasets. All of the experiments performed across these datasets provide insight into various aspects of the data collection discussed in the author's paper. These insights helped enhance the author's discussion of data collection attributes proposed by the authors.\n\nWEAKNESSES:\nAlthough the authors proposed an interesting perspective to address a difficult problem in imitation learning, there exist several weaknesses of the paper:\n- Lack of novelty: Theorem 4.1 was already proposed and proven in [1] (see theorem 3 appendix). That means the action divergence interpretation was already discussed in the literature. Furthermore, transition diversity was already proposed and discussed to address distribution shift issues in [2].\n- Lack of explanation: The authors emphasize that past work only focuses on state coverage while ignoring that action's role. But if the policy providing the demonstrations is an expert,  then the states provided from the expert should inherently have low action divergence because this is an expert;  and an expert provides the optimal action for a given state.\n- Lack of citation: The section on distributions shift in IL needs proper citation.\n\n\n[1] Imitation Learning as f-Divergence Minimization by Ke et al.\n[2] Sequence Model Imitation Learning with Unobserved Contexts\n\nQUESTIONS:\nBelow are several questions that I have regarding the proposed idea:\n- What is the difference between noise in the expert policy and sub-optimal expert? If there is no difference, then is action divergence focuses on suboptimal experts, not experts?\n- In the system noise section, the authors posit that \"the expert demosntrators should encourage paths with high system entropy\";  how is this concept different than DAgger[1]?\n- Why is data quality affected by the algorithm? The algorithm simply uses the data while the expert provides the data. In other words, the algorithms consumes the data while the expert produces the data.\n- If the expert is optimal and the transition is not deterministic -- that means that when the expert takes action in state s it will, with some probability end up one of many states, for example. Expert data with a noisy transition function means you get more state coverage. This implies that the expert and policy will have more overlap. What is the difference of this idea and the idea discussed in [2] -- which cover the case when the expert and the learner policy have state distribution overlap. \n\n[1] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ross et al.\n[2] Feedback in Imitation Learning: The Three Regimes of Covariate Shift Spencer et al.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}

comments: SUMMARY:
This paper systematically studied the data quality assessment problem in imitation learning, proposing new metrics based on test-time distribution shift.

STRENGTHS:
The paper is generally well-organized and well-written. The analysis seems to be in fair detail.

WEAKNESSES:
One of the drawn conclusions seem straightforward to me ("increasing state diversity can often come at the expense of action divergence"). Increasing state diversity will bring more challenges for the learning and thus may cause increased action divergence. 
    
The paper lacks practical insights, e.g., how the proposed data quality evaluation metrics can help guide the data collection and thus help the imitation learning is unclear. Currently I dot not see the benefits of introducing such data quality evaluation metrics.

QUESTIONS:
From Eq.6, the proposed data quality metric is calculated based on the expert policy \pi_E and the learned policy \pi_A by applying an imitation learning algorithm A. If one needs to perform imitation learning first and only after that can get to know how the data quality is, then what is the point of data quality assessment?

Can the authors provide some explanations on how the proposed data quality metrics can be used to benefit the imitation learning?

LIMITATIONS:
Not discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-21