Data Quality in Imitation Learning
Authors: Suneel Belkhale
Yuchen Cui
Dorsa Sadigh
Venue: NeurIPS 2023 poster (2023)
Keywords: Imitation Learning
Robotics
Data Quality
TL;DR: We formalize data quality in imitation learning and analyze performance for models trained on different data sources in several environments.

Abstract: In supervised learning, the question of data quality and curation has been sidelined in recent years in favor of increasingly more powerful and expressive models that can ingest internet-scale data. However, in offline learning for robotics, we simply lack internet scale data, and so high quality datasets are a necessity. This is especially true in imitation learning (IL), a sample efficient paradigm for robot learning using expert demonstrations. Policies learned through IL suffer from state distribution shift at test time due to compounding errors in action prediction, which leads to unseen states that the policy cannot recover from.
Instead of designing new algorithms to address distribution shift, an alternative perspective is to develop new ways of assessing and curating datasets. There is growing evidence that the same IL algorithms can have substantially different performance across different datasets. 
This calls for a formalism for defining metrics of "data quality" that can further be leveraged for data curation.
In this work, we take the first step toward formalizing data quality for imitation learning through the lens of distribution shift: a high quality dataset encourages the policy to stay in distribution at test time. We propose two fundamental properties that are necessary for a high quality datasets: 
i) action divergence: the mismatch between the expert and learned policy at certain states; and ii) transition diversity: the noise present in the system for a given state and action. We investigate the combined effect of these two key properties in imitation learning theoretically, and we empirically analyze models trained on a variety of different data sources. We show that state diversity is not always beneficial, and we demonstrate how action divergence and transition diversity interact in practice.

Data Quality in Imitation Learning
Suneel Belkhale Stanford University bel delkhale@sstaffrrd edu
Yuchen Cui
Dorsa Sadigh Stanford University
Stanford University yuchenc@stanff ord. edu
dorsa@stanfore edu
Abstract
In supervised learning, the question of data quality and curation has been over- shadowed in recent years by increasingly more powerful and expressive models that can ingest internet-scale data. However, in offline learning for robotics, we simply lack internet scale data, and so high quality datasets are a necessity. This is especially true in imitation learning (IL.,, sample efficient paradigm for robot learning using expert demonstrations. Policies learned through IL suffer from state distribution shift at test time due to compounding errors in action prediction, which leads to unseen states that the policy cannot recover from. Instead of designing new algorithms address distribution shift, alternative perspective is to develop new ways of assessing and curating datasets. There is growing evidence that the same IL algorithms can have substantially different performance across different datasets. This calls for formalism for defining metrics of "data quality" that can further be leveraged for data curation. In this work, we take the first step toward formalizing data quality for imitation learning through the lens of distribution shift: high quality dataset encourages the policy to stay in distribution at test time. We propose two fundamental properties that shape the quality of dataset: i) action divergence: the mismatch between the expert and learned policy at certain states; and ii) transition diversity: the noise present in the system for given state and action. We investigate the combined effect of these two key properties in imitation learning theoretically, and empirically analyze models trained different data sources. We show that state diversity is not always beneficial, and demonstrate how action divergence and transition diversity interact in practice.
variety of
Introduction
Supervised learning methods have large strides in recent years in computer vision (CV), natural language processing (NLP), and human-level game playing [18, 26, 43, 8, 51, 13, 44]. These domains have benefited from large and complex models that trained on massive internet-scale datasets. Despite their undeniable power, biases present in these large datasets can result in the models exhibiting unexpected or undesirable outcomes. For example, foundation models such as GPT-3 trained on uncurated datasets have resulted in instances of racist behavior such as associating Muslims with violence [1, 5]. Thus offline data curation is immensely important for both safety and cost-effectiveness, and it is gaining in prominence in training foundation models [22, 10, 40].
Data curation even more important in the field of robotics, where internet-scale data is not readily available and real-worhh datasets small and uncurated. Noise biases present in the data can lead to dangerous situations in many robotics tasks, for example injuring a person or damaging equipment. In such scenarios, deciding which data to collect and how best to collect it are especially important [37, 34, 4]. Of course, the quality of data depends on the algorithm that uses the data. A common paradigm in robot learning from offline datasets is imitation learning (IL), data-driven, sample efficient framework for learning robot policies by mimicking expert demonstrations, However, when learning from offline data using IL, estimating data quality becomes especially difficult, since the "test set" the robot is evaluated on is an entirely new data distribution due to compounding errors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

incurred by the model, i.e., action prediction errors take the model to unseen states. This phenomenon studied and referred to as the distribution shift problem, and prior work has viewed and addressed it through several angles [46, 49].
Broadly, prior work address distribution shift either by taking an algorüthn-eenriic approach to account for biases in the dataset, or by directly modifying the dataset collection process data- centric manner. segorithm-centiic approaches learn robust policies by imposing task-specific as- sumptions [23, 22,,28.22,, 25], acquiring additional environment data [21, 411, or leveraging inductive biases in representing states [54, 30, 38, 12, 12, 37] and actions [53, 48, 12]. What these algorithm-centric works overlook is that changing the data can be as or more effective for policy learning than changing the algorithm. In prior data-centric methods, the goal is usually to maximize state diversity through
shared control [46, 45, 16, 27, 32, 36, 47J, noise injection [34], or active queries [15, 6, 42, 29I. By only focusing state coverage, these works are missing the role that actions (i.e., the expert) plays in the quality of data. A more complete understanding of data quality that integrates both state and action quality would not only improve performance but also save countless hours of data collection.
To better understand the role of both states and actions in the data for learning good policies, consider a single state transition - the three core factors that affect distribution shift are: the policy action distribution, the the stochasticity of transitions, and the previous state distribution. Note that the previous states are also just function of the policy and dynamics back through time. Through this we extract two fundamental properties that can influence data quality in IL: action divergence and transition diversity. Action divergence captures how different the learned policy action distribution is from the expert's 's actions at a given state: for example, if the expert is very consistent but the policy has high variance, then action divergence will be high and distribution shift is likely. Transition diversity captures the inherent variability of the environment at each state, which determines how the state distribution evolves for a given policy: for example, noisy dynamics in the low data regime can reduce overlap between the expert data and the learned policy distribution. Importantly, these factors can compound over time, thus greatly increasing the potential for distribution shift. While state coverage has been discussed in prior work, are the first work to formalize the roles of both state and action distributions in data quality, along with how they interact through time: this new data-focused framing leads to insights about how to curate data to learn more effective policies.
To validate and study these properties empirically, we conduct two sets of experiments: (1) Data Noising, where we ablate these properties in robotics datasets to change the policy success rates, and (2) Data Measuring, where observe human and machine generated datasets and approximately measure these properties in relationship to the policy success rates. Both experiments show that state diversity, commonly used heuristic for quality in prior work, is not always correlated with success. Furthermore, we find that in several human generated datasets, less consistent actions at each state is often associated with decreases in policy performance.
2 Related Work
Data quality research dates back to the early time of computing and the full literature is out of the scope of this paper. Existing literature in machine learning has proposed different dimensions of data quality including accuracy, completeness, consistency, timeliness, and accessibility [33, 14, 20]. We could draw similarities between action divergence with the concept of data consistency, and state diversity with completeness. However, instantiation of these metrics is non- non-trivial and domain-specific specific.
Our work is closely related the imitation learning literature that explicitly addresses the distribution shift problem in various ways. Prior work can be divided into two camps: ones that take an algorithm- centric approach to account for biases in the dataset, and ones that employ data-centric methods for modifying the data collection process.
Algorithm-centric Robust learning approaches including model-based imitation learning methods learn dynamics model of the environment and therefore can plan to go back in distribution when the agent visits out of distribution states [21, 41]. Data augmentation is a useful post-processing technique when one could impose domain or task specific knowledge [23, 28, 255. Prior work has also investigated how to explicitly learn from sub-optimal demonstrations by learning a weighting function over demonstrations either to guide BC training [9,3] or as a reward function for RL algorithms [7, 11]. These methods either require additional information such as rankings of trajectories [9]. demonstrator identity [3], or collecting additional environment data [7, 11]. Recent efforts on demonstration retrieval augment limited task-specific demonstrations with past robot experiences [39]. A recent

---

body of work build better inductive biases into the model for learning robust state features, like pretrained vision or language representations [54, 30, 38, 12]. Some approaches modify the action representation to be more expressive to capture all the expert's actions, for example using Gaussian or mixture model action spaces [12, 37]. Others consider temporally abstracted action spaces like waypoints or action sequences to reduce the effective task length and thus mitigate compounding errors [53, 48, 12]. What these algorithm-centric works overlook is that changing the data can be as more effective for policy learning than changing the algorithm. However, we still lack comprehensive understanding of what properties in the data matter in imitation learning.
Data-centric. In more data-centric prior works that discuss data quality, the primary goal is often just to maximize state diversity. A large body of research focuses on modifying the data collection process such that the expert experience diverse set of states. Ross et al. [46] proposed iteratively collect on-policy demonstration data with shared-control between the expert and the robot, randomly switching with gradually decreasing weight on the expert's 's input, such that the training data contains direct samples of the expert policy at states experienced by the learned policy. However, randomly switching the control can make unnatural for the human demonstrator and leads to noisier human control. To mitigate this issue, methods have been proposed to gate the control more effectively by evaluating metrics such as state uncertainty and novelty [16, 27, 50]. Other methods allow the human to gate the control and effectively correct the robot's s behavior only when necessary [32, 36, 24, 47]. Closely related to our work, Laskey et al. [34] takes an optimal control perspective and showed that injecting control noise during data collection can give similar benefits as DAgger-style iterative methods. Active learning methods have also been developed to guide data collection towards more informative samples [15, 6, 42, 29]. By only focusing on state coverage, these works are missing the role that actions (i.e., the expert) plays in the quality of data.
3 Preliminaries
In imitation learning (IL), we assume access to dataset DN { JN} of N successful demonstrations. Each demonstration т; consists of a sequence of continuous state-action pairs of length T, T; {(51 o1), ... ($T; (r:)). with states E S and actions a E .A. Demonstrations are generated by sampling actions from the demonstration policy Tg through environment dynamics p(s')s s, a). The objective of imitation learning to learn a policy me S + A that maps states to actions. Standard behavioral cloning optimizes supervised loss maximizing the likelihood of the state-action pairs in the dataset:
1
(())
Σ (s.a)EDy
log To (a)s),
[D
which is optimizing the following objective under finite samples from the demonstration policy:
E EgpP=E ()) [Dx (T=(-))), m=()>))))))))))) (-(+)))) -R ()) awwe(-la) [log [u[()))] +
The C term here captures the entropy of the demonstration state-action distribution, which is constant with respect to 0 and thus it does not affect optimality of 0. Px (8) is the state visitation of the demonstration policy, defined for any policy π as follows:
p=(s)= T
Σ (s)
(3)
1=1
0,(5') = [
P,, a, s' ds da - / =(als))(811s a)s (s)dss "s)d da
3.1 Distribution Shift in IL
Behavioral cloning methods as in Eq. (2) often assume that the dataset distribution is a good approximation of the demonstration policy distribution. In most applications, however, the learned policy is bound to experience novel states that were not part of the training data due to stochasticity of the environment dynamics and the learned policy. Herein lies the fundamental challenge with imitation learning, i.e., state distribution shift between training and test time. Consider the training sample (s, u, 8') at timestep I in demonstration т. If the learned policy outputs à ~ rT-ss) which has a small action error e = ā a, the new state the next time step will also deviate: g' ~ p(s'|s, + e). which in turn affects the policy output at the next step. In practice, the change in next state can be
3

---

highly disproportionate to ||e], so small errors can quickly lead the policy out of the data distribution. Stochastic dynamics, for example system noise, can compound with the distribution shift caused by policy errors, and as we continue to execute for T - / steps, both these factors can compound over time to pull the policy out of distribution, often leading to task failure.
We can address this distribution shift problem by minimizing the mismatch between state visitation distribution of learned policy л and the state visitation distribution of an demonstration policy Te under some f --dieergence Dy:
TTT =
(5)
Next, we connect this distribution shift problem to the question of data quality.
3.2 Formalizing Data Quality
How can we define and measure the quality of a dataset D in IL? In existing literature, data quality has been used interchangeably with either the proficiency level of the demonstrator at the task or the coverage of state space. However, these notions of quality are loose and incomplete: for example, they do not explain what concretely makes a demonstrator better or worse, nor do they consider how stochasicity in the transitions impacts state space coverage and thus downstream learning. Our goal is to more formally define a measure of quality so that we can then optimize these properties in our datasets.
We posit that a complete notion of dataset quality is one that minimizes distribution shift in Eq. (5). This suggests that we cannot discuss data quality in isolation. The notion of data quality is heavily tied to the algorithm A that leads to the learned policy A as well as the demonstration policy Tg. We thus formalize the quality of dataset Dx conditioned on the demonstration policy Te that generates it along with a policy learning algorithm A, as the negative distribution shift of a learned policy under some fddivergence Dy:
AE DD (Dan (s), PTg (5)), where TA- A(Dx)
(6)
Here, TA is the policy learned by algorithm A using dataset DN of size N, which is generated from demonstrator gg Note that quality is affected by several components: the choice in demonstrator, the choice of dataset size, and the algorithm. The choice of demonstration policy gg changes the trajectories in Dy (which in turn affects policy learning) along with the desired state distribution Proe (s)). The dataset size controls the amount of information about the demonstration state distribu- tion present in the dataset: note that pm (s) should match Prg (s), but A learns only from Dx not from the full Prg (s). The algorithm controls how the data is processed to produce the final policy a and thus the visited distribution P (s).
To optimize this notion of quality, we might toggle any one of these factors. Prior work has studied algorithm modifications length [37, 54, 4], but few works study how Te and the dataset Dy should be altered to perform best for any given algorithm A. We refer to this as data curation. In practice we often lack full control over the choice of Tæ. since this usually a human demonstrator; still, we can often influence nE. for example through prompting or feedback [24]. or we can curate the dataset derived from ag through filtering [19].
4 Properties of Data Quality
To identify properties that affect data quality in imitation learning, we can study how each state transition (s, a, s') E D in the dataset affects the dataset quality in detail. As we defined in Eq. (6), the dataset quality relies on distribution shift given an demonstration policy TT and an imitation learning policy TA. This relies on the state visitation distribution p"r, (s), which based on Eq. (4) depends on three terms: A (a) s). p(s'|s, a). and p² (s). Intuitively, three clear factors emerge for managing distribution shift: how different the policy x. Aa(as) is from the demonstrator Te ss) - which we refer to as action divergence, how diverse the dynamics p(s' [a, s) are which we refer to as transition diversity, and how these factors interact over time to produce past state visitation distribution p4-1 Importantly, the past state visitation can be controlled through action divergence and transition diversity at previous time steps, so in this section, we will formalize these two properties and discuss their implication on data curation.

---

4.1 Action Divergence
Action divergence is measure of distance between the learned policy and the demonstration policy Dy( (-11s), (=)+)))). This can stem from biases in the algorithm or dataset such as mismatched action representations or lack of samples. While this is typically viewed in prior work as facet of the algorithm or dataset size, importantly action divergence and thus data quality can also be influenced by the demonstration policy itself. For example, if the demonstrator knows the action representation used by the learning agent a priori, then the policy mismatch can be reduced by taking actions that are consistent with that action space. In Theorem 4.1" 1 we illustrate the importance of action divergence by showing that our notion of data quality is lower bounded by the action divergence under the visited state distribution.
Theorem 4.1. Given policy TA and demonstrator ag and environment horizon length H, the distribution shift:
H-1 -H - [DK((T(()))
PRE) S H
1=0
When using KL divergence for distribution shift in Eq. (5), we can see how the quality is lower bounded by the policy action divergence from the optimal policy under the visited state distribution, weighted at each step by time-to-go. Many prior works have noted the compounding error prob- lem [46, 45]. and here too action divergence at earlier timesteps has an out-sized effect on the overall distribution shift. Thus to optimize for the quality of dataset, we should reduce the aggregated policy mismatch across visited trajectories.
Optimality: Many prior ideas around the "optimality" of a demonstration are intimately related to this action divergence property. Suboptimalities that are naturally present in demonstrations like pauses or other action noise have been a source of difficulty when learning imitation policies [35, 4]. These factors make harder for the model learn the demonstration action distributions, thus increasing action divergence. Suboptimality can also come from multi-modal demonstration policy distributions, and more expressive policy representations (i.e., reducing action divergence between the demonstrator and the learned policy) has been shown to help [17, 12, 24]. The speed of demonstration another common notion of optimality, which decreases total action divergence by reducing horizon length H. provided it does not increase the per time step action divergence to compensate.
State Visitation: Critically, the visitation distribution in Theorem 4.1 determines where action divergence should be low, whereas in the standard BC objective Eq. (2), the divergence is only minimized under samples from the demonstration distribution pº To better understand how the visitation distribution evolves, we now analyze state transitions in greater detail.
4.2 Transition Diversity
Transition diversity encompasses the diversity of next state transitions seen at given state for certain policy. What role does transition diversity play in minimizing distribution shift? Intuitively, we consider the upper bound in Theorem 4.1, the demonstrator's 's state coverage should overlap as much as possible with the visited state distribution, but without increasing the action divergence.
Lemma 4.2. Given learned policy A and an demonstration policy Tg, assume that the policy is learned such that when s E supp(png) DRI. ( =<(+ss)) S B. Then:
Ex-th, [Dxa
& Bo-et [B1(s E supp(p?e))
1(s & () (T
Lemma 4.2 shows that with good learning algorithm (i.e., when ,9 is small), it is important to maximize the overlap between the visited policy and the demonstration data (minimizes the contribution of the unbounded right term), but as shown in Theorem 4.1 this should not come at the expense of increasing the action divergence of the policy. Rather than broadly maximizing state diversity, as prior works do, more sensible approach is for the demonstrator to maximize the two other factors that affect P== system noise and initial state variance in the demonstrator data.
Proof for all theorems and lemmas in Appendix A
5

---

Finite Data and Low Coverage: While the above analysis holds for infinite data, maximizing transition diversity can lead to thinly spread coverage in the finite data regime. To analyze the finite data regime in isolation, we consider Gaussian system noise for simplicity. First, we formulate an empirical estimate of state "coverage" of the demonstrator (i.e. the deviation from states in the demonstration dataset). We then show that higher system noise can cause the learned policy R to deviate more from states sampled in the dataset.
Definition 1. For a given start state s, define the next state coverage probability over N samples from TE for tolerance as Ps(s; N,e) == P(mming(1L..NN I|s' s=,,ll> I é), where (&4, p {-1s) is a set of N sampled next state transitions under the demonstration policy Te starting at s, and s' P (-1s) is sampled next state under the policy AA starting at S.
We can think of (s(x; N,e) as the probability under potential datasets for N of seeing next state at test time (under TA) that was nearby the next states in the dataset, conditioned at a starting state 8 (i.e., coverage). This is related to single step of distribution shift, but measured over a dataset rather than a distribution. Defining coverage as the distance to the dataset being below some threshold is reasonable for function approximators like neural networks [2].
Theorem 4.3. Given a policy N and demonstrator TE assume that for state s, if PTT > O, then n (a|s) (a(als). Assume that transitions are normally distributed with fixed and diagonal variance, p(s'| s, a) = N(u(s, a), then the next state coverage probability is Ps(5; N,,) Z 1 (1- ( exp (aa²d))
exp (-(a- 1)2d), where d is the dimensionality of the state, c is a constant, and o 2 1 is a parameter chosen to maximize the bound.
that even under policy that is perfect when in distribution, the lower bound of next state coverage probability increases as the variance of system noise o² decreases, for a fixed sampling budget N (and fixed a). However, increasing N has a much stronger positive effect on this lower bound of state coverage than decreases in a, suggesting that system noise is only an issue when the dataset size is sufficiently small. Furthermore, e here represents some simplistic form of the generalization capacity of the policy, and we see here that increasing makes us more robust to proportional increases in Ø.
In Theorem 4.3,
Finite Data and Good Coverage: However, if we assume N is large enough to mitigate the coverage effect of system noise, are there any other effects of transition diversity in the dataset? The answer lies in generalization properties of the algorithm, which as provided in Theorem 4.3 are fairly conservative (i.c., in the definition of (s; N, e)). In Theorem A.I 1 in Appendix A, use more loose generalization definition and relax the assumption that the learned policy is perfect. We show as system noise increases, the resulting boost in sample coverage can actually replicate and overcome the effects of high learned policy noise, suggesting that more system noise can actually be beneficial for learning. This finding sheds light on results from prior work and our own experiments in Section that show the benefits of collecting data in the presence of high system noise [34].
4.3 Implications for Data Curation
We have shown how action divergence and transition diversity are both tied distribution shift and thus data quality. Based on action divergence and transition diversity, we now examine downstream implications of these properties on what matters for data curation, where the goal is to collect and then select high quality demonstrations in our dataset D for good policy learning.
Action Consistency: To minimize action divergence, the algorithm action representation should align with the demonstrator's 's action distribution, for the given dataset size. One algorithmic solution is improve the expressiveness of the policy action space it capture the exact actions at every state that was demonstrated. However, in practice, individual states might only be visited a few times in the dataset, and so if the entropy of the demonstration policy those states is high, the learned policy will find it difficult to perfectly match the actions even for the most expressive action spaces. Instead, we argue that demonstration data should be curated have more consistent actions, e.g., reducing the entropy of the demonstration policy: E.-pr a ()) [. e (=]s)) Since knowing the visited state distribution is impossible beforehand, the best we can do is to encourage low entropy in the demonstration data distribution:
min E (-) [+((T(())s)
RE

---

Age
Low State
High Actian consistency
High State Coverage
Lorn Action conseniency
Scripted
Autionn Varianas
Human
Figure Case Study: Trajectories and action variance for scripted (left two plots) compared to human demonstration data (right two plots). Even though the human data (right) has high state coverage, the action variance is high, leading high action divergence, and vice versa.
State Diversity: As discussed previously, the state visitation of the policy depends on the trajectory action distribution, the initial state distribution, and the system noise distribution. Many prior works seek to improve the state coverage of the dataset, using some metric similar to Definition 1 [6, 29]. However, the required state diversity is a function of the learned policy mismatch (action divergence) and the noise present in the system (transition diversity), and is thus coarse representation of data quality. Uniformly increasing coverage over the state space is not necessarily good thing - as shown in Section 4.1, if state coverage comes the expense of action consistency, the result might be worse than a policy trained on less state coverage but more consistent actions. Instead, we argue that we should optimize for system noise.
System Noise: Although system noise at a given state cannot be controlled, demonstrators can control system noise the trajectory level (e.g. visiting states with more or less system noise). Should system noise be encouraged or discouraged in a dataset? Based on the discussion in Section 4.2, we hypothesize that when the dataset size fixed, increasing the system noise can increase the coverage of the demonstrations and thus improve robustness in the learned policy (Theorem A.1), but only up certain point, when the data becomes too sparse to generalize (Theorem 4.3). Thus, in addition to consistent actions, we posit that demonstrators should encourage paths with high system entropy for learning better policies, such that the overall state entropy stays below some threshold y to avoid coverage that is too sparse.
max ()an))(())))(()))(()()
s.t. ( (s)) S
(8)
Horizon Length: The length of trajectories in the demonstration data also can have a large effect on the demonstration and visited state distributions, and thus on the quality of the dataset. However, while horizon length certainly plays role, like state diversity, it is a downstream effect of action divergence and transition diversity. Based on previous analysis, what really matters minimizing the aggregated action divergence and transition diversity produced by the demonstrator across time. Horizon length alone only crudely measures this, but is often correlated in practice as we show in Section 5.2.
While prior work in data curation primarily aim to maximize state diversity alone, our analysis of data quality reveals several new properties that matter for data curation, such as action consistency, system noise, and their combined effect over time. As we demonstrate in the next section, understanding these properties and the inherent tradeoffs between them vital for data curation in imitation learning.
5 Analysis
To empirically analyze how different data properties affect imitation learning, we conduct two of experiments. In the first set, we generate various quality demonstration datasets by adding different types of noise to scripted policies, i.e., policies that are hand-designed be successful at the task. An example scripted policy compared to human demonstrations is shown in Fig. 1. We study both noise in the system considering transition diversity - and noise added to the demonstration policy considering both transition diversity and action divergence as a function of dataset size. In the second set experiments, we evaluate the empirical properties (see Appendix B) for real human collected datasets.
7

---

System Noise (Pollcy Noise 0)
Policy Noise (System Noise 0)
Policy Noise (System Noise (.03)
Poim PointMass Mass Obstacle
6.5
0/0
0/0
000
111
0lc
(4) 1000 demos
(b) 10 domoc
1000 damos
(6) 10 demos
i 1000 damas
10 domozs
Square
1.03
10.0
0.50
(c) 200 dkemon
(4) 50 demmoox
200 demos
50 dermoox
(03.200 denons
(3) 10.50 dessoss
BC Success rates in PMObstacle (top row) for 1000 and 10 episodes of data, and in Square (bottom row) for 200 and 50 episodes of data (error hars over datasets). X-axis corresponds injected system or policy noise in the dataset (o," or o7) and each line corresponds to injected noise (a., or σp) during evaluation. System Noise (left): for large datasets (a)(c), higher system noise during evaluation decreases performance, but more system noise during training does the best. For small datasets (b)d), we note a similar but exaggerated effect. Policy Noise (mid) For large datasets (eXg), unlike system noise, more demonstration policy noise often hurts performance despite similar state coverage. For small datasets (f)h), adding policy noise exaggerates this effect and produces large variance in performance. For Square, the dotted lines mark comparable values of noise in terms of state coverage. System + Policy Noise (i-l): Adding some system noise can make the policy more robust to action divergence.
Figure 2:
5.1
Data Noising
We train Behavior Cloning (BC) with data generated with system noise and policy noise in two environments: PMObstacle, a 2D environment where a point mass agent must reach a target point without hitting an obstacle in its way, where demonstrations are multimodal (left and right around the obstacle); and Square, a 3D environment where robot arm must pick up square nut and insert onto a peg (shown in Fig. 1). In both, system and policy noise are Gaussian random variables added to the dynamics and scripted policy, respectively, at each time step, and BC uses an MLP architecture. Fig. 2 shows results in PMObstacle (top row) and Square (bottom row). We additionally show Transformer architecture results in Fig. 3.
State Diversity through Transition Diversity improves performance, up to a point. The left plots in Fig. 2 (a)-(d) show policy success rates as we vary system noise (ar,) (more noise for lighter color shades), where (a)(c) show the high-data regime and the (b) show the low data regime. Higher system noise tends to improve policy performance in the high data regime, but only up to a point for Square after a_ = 0.3 in the demonstration data (c), performance starts to drop off, which we hypothesize due to the low coverage, based on the analysis of transition diversity in Section 4.2. In the low data regime, the story is similar but even more exaggerated, with increasing system noise leading to comparable performance as the high data regime. Once again for Square, increasing transition diversity (d) helps until the state coverage is too thin. See Table 2, Table 5, and Table 6 in Appendix C for the full sweep of system noise values in each environment.
State Diversity at the cost of Action Divergence hurts performance. Plots in Fig. 2 (e) and Fig. 2 (i() show policy success rates for increasing the policy noise (ap) in the dataset, where the (c)(g) show the high data regime and the (f)hh show the low data regime. Note that each value of policy noise yields the same demonstration state distribution as the corresponding amount of system noise. Since this noise is zero-mean and the learned policy is deterministic, policy noise in the high data regime (e)(g is only moderately worse as compared to equivalent amounts of system noise (a)(c), suggesting that the action divergence is minor. In fact, due the added state diversity, adding policy noise in the high data regime helps up to certain point. However in the low data regime (f),h, performance is substantially worse compared to comparable system noise, since the policy can not recover the unbiased demonstration policy from just few noisy examples (note that in (c)(d, the x-axes are not aligned with those in (g),,, and a. = 0.2 corresponds to op = 0.02 as shown by the dotted line). This illustrates that state diversity coming from the noise in the demonstration policy
8

---

can increase action divergence, and thus high state diversity is not universally desirable. See Table 3, Table 7, and Table in Appendix C for the full sweep of policy noise values in each environment.
Transition Diversity can counteract the effects of policy noise. In the right two plots (i))) for PMObstacle in Fig. the dataset combines both system and policy noise. The system noise is fixed (a, = =0.03) and the policy noise is varied in the same range as the middle two plots (just policy noise). Given the analysis in Section 4.2 and Theorem A. in Appendix A, would expect that having some system noise could actually make the policy much more robust to policy noise, since the state coverage provided in the dataset can help learn a more robust policy (generalization), Indeed we find that just by adding system noise, the learned policy becomes very robust to added policy noise in both the high (i) and low (i) regimes. This suggests that adding transition diversity can help mitigate the affects of action divergence incurred by noisy or sub-optimal demonstrators. See Table 4 in Appendix C for the full sweep of policy noise values for fixed system noise in each environment.
System Noise (Policy Noise
Policy Noise (System Noise 0)
0,0005
5,202
00.00
0.00
-0004
100
100
Point Mass Obstacle
90-
80-
70-
0.00
d.1
0.01
0.00
0.6
0.01
0.02
0.03
0.01
0.01
0.02
0.00
0.01
(a) 1000 demos
(b) demos
(c) 1000 demos
(d) 10 demos
Figure 3: Transformer-Be Success rates in PMObstacle for 1000 and 10 episodes. X-axis corresponds injected system or policy noise the dataset and each line corresponds to injected noise (a, or σp) during evaluation. System Noise (left): for large datasets (a), higher system noise during evaluation decreases performance, but more system noise during training does the best. For small datasets (b), we note a similar but exaggerated effect. Policy Noise (mid) For large datasets (c), unlike system noise, more demonstration policy noise often hurts performance despite Transformers being more expressive than simple MLPs. For small datasets (d), adding policy noise exaggerates this effect and produces large variance in performance. Overall, we find that Transformers can suffer from similar data quality issues as MLPs despite being more expressive.
5.2
Data Measuring
Next we empirically instantiate several metrics from Section 4.3 that capture various facets of data quality, and measure different datasets: (I) action variance (higher means less action consistency) measured through clustering nearby states (2) horizon length T' (higher means longer trajectories), and (3) state similarity (opposite of state diversity) measured as the average cluster size. See Appendix B for the exact empirical definitions. We leave out system noise since these human datasets were collected in deterministic environments.
In Table 1, consider single and multi-human datasets from the Square and Can tasks from robomimic [37]. We see that higher action variance and horizon length are often accompanied by decreases in success rates. The Worse multi-human datasets have seemingly lower action variance but less state similarity, and yet qualitatively we observed highly multi-modal behaviors in these datasets (e.g., grasping different parts of the nut instead of the handle). We suspect that this type of multi-modality is not measured well by the single time step action variance metric, but rather requires longer term view of the effects of that action on the scene as indicated in Theorem 4.1. Additionally, state diversity once again is not correlated with performance on the task.
We emphasize that these metrics likely do not provide a complete picture of data quality, but they do provide us insights into why performance is low and how data collection can be improved in future datasets. Through future efforts from the community, we envision a comprehensive set of data metries that practitioners can use to quickly evaluate the quality of datasets without needing to first train and evaluate models on those datasets.

---

Square
Can
PH
Better
Okay
Worse
PH
Better
Okay
Worse
Success Rate
58
36
12
96
56
40
22
Dataset Size (N)
200
100
100
100
200
100
100
100
Action Variance
0.073
0.062
0.099
0.061
0.051
0.066
0.079
0.063
Horizon Length (7)
150
190
250
350
115
140
180
300
State Similarity
8.2e-5
1.8e4
1.7e-4
1.2e-4
1.0e-4
2.1e-4 2.4e-4
2.0e-4
Table 1: Data Quality metrics evaluated on Square (left) and Can (right) for proficient human (PHD, and multi human (Better, Okay, Worse) datasets. Note that state similarity (defined in Appendix B) is lower for PH than Better, however importantly Better actually represents overlapping data with the PH dataset (Better consists of 2 proficient demonstrators). We use this as calibration between different dataset sizes, and we see that there some effect of dataset size on our chosen metrics. Note that datatset size corresponds to number of episodes.
Discussion
Data curation is incredibly important for maximizing the potential of any given algorithm, and especially so in imitation learning. To curate data, one needs to establish a formalism for assessing data quality. The predominant notions of data quality today are solely centered around maximizing state diversity in a dataset, which ignore the quality of the demonstrated actions and how these factors interplay. In this work, we propose holistic view of data curation centered around minimizing distribution shift. We demonstrate that action divergence and transition diversity are crucial factors in data quality that can often be controlled or measured, and we draw several valuable insights from this shedding light on effective strategies for curating demonstration datasets. We find that making actions more consistent tends to increase policy success. Furthermore, we observe fundamental tradeoff between state diversity and action divergence increasing state diversity can often come at the expense of action divergence. Instead of uniformly maximizing state diversity, we show that increasing transition diversity improves performance until the data coverage becomes too sparse. While the metrics presented in our experiments are often good measures of final performance, we find that comprehensively measuring data quality in practice for real world datasets can be quite challenging. Finally, we envision a broader set of metrics informed by our formalism for data quality that practitioners can use to curate datasets, and we believe our work is an important first step in formulating and evaluating these metrics.
Limitations and Future Work:
We hope our analysis will inspire future research in data quality for imitation learning. Firstly. we hope to relax several theoretical assumptions like Gaussian noise in future work. The metrics defined in Section 5.2 do not necessarily capture the full scope of factors that comprise data quality. Future work should refine these metrics or find alternate ways of measuring data quality without requiring running the algorithm first, or potentially with only limited algorithm training. While we show the performance of several simple metrics on two interesting control settings, future work is needed in determining how best to these metrics for guiding data collection. One option is to use the metrics as a score function that demonstrators might use as feedback. Another to explicitly filter or weight data using the metrics. Furthermore, it would be beneficial test the insights in this paper on real world settings, for example under more realistic models of system noise than simply Gaussian noise. Nonetheless, we believe our work to be an important first step in defining and measuring data quality.
As discussed in our introduction, foundation models like GPT-3 trained on uncurated datasets have learned harmful biases and stereotypes from uncurated datasets. Efforts in the broader ML community have started focus on curation policy learning setting in robotics, faimess and safety are less clearly defined. Our primary focus in this paper is on data curation for learning high performing policies, but we believe future work should find analogous notions of fairness in the robotics setting, and to consider how metrics of data quality that we define will affect this notion of fairness. For example, we might want to ensure data is collected over a diverse range of institutions rather than in just a select few locations. Our action consistency metric might artificially select only demonstrators from a small range of institutions, and so this should also be considered during data curation.
Fairness in robotics:
means to solve this problem. In the
10

---

Acknowledgments
This research was supported by ONR, DARPA YFA, Ford, and NSF Awards #1941722, #2218760, and #2125511. We also give special thanks to Stephen Tu for helping to streamline and improve several proof' for theorems the text.
References
[1] Abubakar Abid, Maheen Farooqi, and James Zou. Large language models associate muslims with violence. Nature Machine Intelligence, 3(6):461-463, 2021.
[2] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198, 2000.
[3] Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators, In International Conference on Machine Learning, pages 1732-1748. PMLR, 2022.
[4] Suneel Belkhale and Dorsa Sadigh. Plato: Predicting latent affordances through object-centric play. In 6th Annual Conference Robot Learning, 2022.
[5] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text- to-image generation amplifies demographic stereotypes at large scale. ACM Conference on Fairness, Accountability, and Transparency, 2023.
[6] Erdem Bryik, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv:1910.044 04365, 2019.
[7] Daniel S Brown, Wonjoon Goo, and Scott Niekum. Beter--hann-demonstate imitation learning vía automatically-rankeed demonstrations. In Conference on robot learning, pages 330-359. PMLR, 2020.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1919 2020.
[9] Zhangjie Cao, Zihan Wang, and Dorsa Sadigh. Learning from imperfect demonstrations via adversarial confidence transfer. 2022 International Conference on Robotics and Automation (ICRA), pages 441-447. IEEE, 2022.
[10] Ting- Tung-Yun Chang and Robin Jia. Careful data curation stabilizes in-context learning, 2022.
[11] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning, pages 1262-1277 PMLR, 2021.
[12] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[13] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
[14] Corinna Cichy and Stefan Rass. overview of data quality frameworks. IEEE Access, 24634-24648, 2019.
[15] Yuchen Cui and Scott Niekum. Active reward learning from critiques. In 2018 IEEE interna- tional conference robotics and automation (ICRA), pages 6907-6914, IEEE, 2018.
11

---

[16] Yuchen Cui, David Isele, Scott Nickum, and Kikuo Fujimura. Uncertainty aware data aggrega- tion for deep imitation learning. In 2019 International Conference on Roboties and Automation (ICRA), pages 761-767. IEER, 2019.
[17] Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. In International Conference on Learning Representations, 2023. URL http:t/ /ttptts:///ppeeeveeeeee wwweeeervieeeeee/forreeeeeee net/forum? id=C7
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations 2020.
[19] Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[20] Lisa Ehrlinger and Wolfram Wö8. A survey of data quality measurement and monitoring tools. Frontiers in big data, page 28, 2022.
[21] Peter Englert, Alexandros Paraschos, Marc Peter Deisenroth, and Jan Peters. Probabilistic model-based imitation learning. Adaptive Behavior, 21(5):388-444 40388 2013.
[22] Hugging Face. Starcoder.
com/ httpa//co..oooooooooooooo//taaaa 2023.
[23] Alexandre Galashov, Josh S Merel, and Nicolas Heess. Data augmentation for efficient learning from parametric experts. Advances in Neural Information Processing Systems, 35:31484-23 1484-31496, 2022.
[24] Kanishk Gandhi, Siddharth Karamcheti, Madeline Liao, and Dorsa Sadigh. Eliciting compatible demonstrations for multi-human imitation learning. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[25] Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven histery-aware policies for robotic manipulations. In Conference on Robot Learning, pages 175-187, PMLR, 2023.
[26] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[27] Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning, pages 598-608. PMLR, 2022.
[28] Stephen James and Andrew Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 112:161221619, 2022.
[29] Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Conference Neural Information Processing Systems, 2020.
[30] Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-drive -diven representation learning for robotics. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[31] Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning f-divergence minimization. CoRR, abs/1195.12288 2019. URL absg//bb//ae 19159195.111 12868.
http://arxiv..rg//
[32] Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEEE, 2019.
12

---

[33] Nuno Laranjeiro, Seyma Nur Soydemir, and Jorge Bernardino. A survey on data quality: classifying poor data. In 2015 IEEE 21st Pacific rim international symposium on dependable computing (PRDC), pages 179-188. IEFF, 2015.
[34]
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 143-156. PMLR, 2017.
[35] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[36] Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. mamma--n--the-ooy imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733, 2020.
[37] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2 2021.
[38] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning, 2022. https:::://
net/forum?? tGbpgz6yOrI.
[39] Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In 6th Annual Conference on Robot Learning, 2022.
[40] Xu Ouyang, Shahina Mohd Azam Ansari, Felix Xiaozhu Lin, and Yangfeng Ji. Efficient model finetuming for text classification via data filtering, 2022.
[41] Carl Qi, Pieter Abbeel, and Aditya Grover. Imitating, fast and slow: Robust learning from demonstrations via decision-tima planning. arXiv preprint wxXiv:220.003599 2022.
[42] Mattia Racca, Antti Oulasvirta, and Ville Kyrki. Teacher-aware active robot learning. In 2019 14th ACM/IEE International Conference on Human-Rokee Interaction (HRI), pages 335-343. IEEE, 2019.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[44] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Marth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06117. 2022.
[45] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial infelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[46] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[47] Mariah L Schrum, Erin Hedlund-Botti, Nina Moorman, and Matthew C Gombolay. Mind meld: Personalized meta-learning for robot-centric imitation learning. In 2022 17th ACM/IEEE International Conference on Hluman-Robol Interaction (HRI), pages 157-165. IEEE, 2022.
[48] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022.
[49] Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift, arXiv preprint 2xXiv:2102.0287.. 2021.
13

---

[50] Gokul Swamy. Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Sequence model imitation learning with unobserved contexts. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttps:/opeereeviee hett:forunniddddtbiiaa......
[51] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.082339 2022.
[52] Martin J. Wainwright. High-Dimensiooal Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/978 108627771.
[53] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. Proceedings of Robotics: Science and Systems (RSS), 2023.
[54] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision- based manipulation with object proposal priors. 6th Annual Conference on Robot Learning (CoRL), 2022.
14

---

A Theoretical Results
Theorem 4.1: Given policy лa and demonstrator ag and environment horizon length H, the distribution shift:
PRE PTB) HH- B) (
Proof. Using the log-sum inequality:
Pi (5)) DRIlPEs, = [. Sra log)
log P (5))
=1o (1/ Pata ( log
Sn (3) a, 8)
Sa PiE (s)
( (4 log
Pne (8) (als)p(s'1a P
+ log TA(a|s)
$ (s) (a, la, slog
PTg
(s)
Te(als)
$ 1 PEr (T)
(s)
TE(a|s) TA [a(ss)
-log
PeP (8)
[a(a)))
P
P (s) log Ta(als)
$ (5) log
PRE (8)
I Dra :Pam + Esnuph (T((=a(())):TT((11))
S (t) (-)) + Σ
E Banpt (
Recursive
E
Bo
1=0
H
-== Pra (s) log
DRI(PEAS Pag)
M (8)
Σ P/an (s)
11
11
Pz (8)
<
Prc (5) log
P% (8)
H
1
DKL (
H
H
t-1
E
snuple
#-1
1
Σ
(H D)
Note that Ke et al. show similar theorem, however they make the strong assumption of the f- divergence satisfying the triangle inequality, which is not true for the KL divergence we use in Theorem 4.1, and also yields different bound. Furthermore, the implications of the divergence relationship on data quality (i.e. the data generating policy distribution) not examined within this prior work. They are focusing more on the algorithmic perspective, as is common in prior work.
15

---

Lemma 4.2: Given learned policy TA and expert RE, define assume that the policy is learned such that when supp(pnn)) DxL (Ta(- --(-s)) S B. Then Ex-rt, [ ((( v Exert [81(s E supp( )) + 1(s supp( Dr
Proof. This follows by simple substitution:
Eanpt (T.A((TA(-18)) x-{+|s)]] = Eseph [1(s
[(s &
(T))
I E [$1 supp(pag))
+1ss & DxL +=(1)))))))
Theorem 4.3: Given policy aa and demonstrator TE, assume that for state if PTT 0, then A (a|s) (a(as)) Assume that transitions are normally distributed with fixed and diagonal variance, p(s' s, a) =N(p(s, a). I2I, then the next state coverage probability is Ps(s; N.)) Z 1 1- -1 (d exp (-a"d)) exp (-(a- 1))d), where d is the dimensionality of the state, c is a constant, and 2 a parameter chosen to maximize the bound.
Proof. The proof follows by first discretizing the state space into length e bins. Denote the probability mass of bin b as Pa- First note that for two independent samples from the same distribution s' and s²,,, we can say that P([s' - s 2 <) is upper bounded by the probability that neither sample lands in the same bin. This is because of the fact that if Is' - 5 loc 2 e, then that implies the samples did not land in the same bin (if they did the infinity norm would be less than epsilon). Thus, can
also say that P(min; is' - -% 2 c) is upper bounded by the probability that all of the samples
S/, land in a different bin than s'. Thus:
P(min |s' - o, 2 E) S P(none lands in same bin)
Σ pb(1 - pb)
Next define ball of radius R around the mean μuss Ball(R). Using this ball partition the above inequality into two terms to yield an upper bound.
P(min|s Is' s/ 20 Σ pb - Pb)
Σ
po pb)
Σ
Pb pb)
BEBall(R)
bgBall(R)
We can upper bound everything inside the ball using the minimum Gaussian mass for the given radius -πt R, or pa 2 ( yd exp 20² where c = 1 . For the second term, if R is large enough, then we can assume p6 sufficiently small such that (1 pb)
R2
N
P(min|s - -,,,0000 26)5
Σ ph
CE
exp
+
Σ
pb
202
bg (a)l(
ballal(
d
R²
N
Σ
Σ bgBall(R)
exp
20
pb
Pb
(EBall(R)
1
d
exp
-
R22
Σ
Pb
20
bgBall( R2)
16

---

where Fa(x) is the CDF of the f-distribution of dimension d. The second term is upper bounded by the probability that the d-dim Gaussian is farther in euclidean distance than R from the mean, which can be written using the tail probability the x² distribution.
08 d
R$
N +
P(min |s' - 1,,000 20) i
1 -
exp
P(Iss
0(8,0)1122 R²))
20
d
R²
N
1 (1-X2
R2
1
exp
+
2
We know that for x2 random variable Y over d sub Gaussians, P( 2 (1+ S exp
-d
for 5 2 0 [52]. Thus assuming R 2 ovd, we can write:
R2
N
R 2 (-(8-
P(min |s' s4,0 20) S 1
exp
+ exp
vdl
2a
0
Now, for any a ovd 2 1, we can rewrite the above bound as:
P(min (8' - 1,,10 26) s (11-()) exp (-a2a)) N
+ exx (-(a- 1)2d)
Starting from Definition 1:
N,( = P(min 11 - s',, ()
= - 11-P(mmin/ |s' 14, 2 c)
2 1 1 _ exp (-a2d)) exp (-(a -))2d)
A.1 Generalization under System Noise
Definition Given a policy TA a data generating policy TH and a starting state s, define the
probability of next state coverage Pa(s, pi N) - N,P([ss pul² Z || pul3),, where 1/2x (+)(()s) (-ls) are next state samples from the expert starting al s, and sf Pl (+|s) are next state samples from A starting at s.
Intuitively (s(s; N) the probability that given chances, the hyper-sphere defined by the L.2 distance from μ to a sampled data point contains a sample from the learned policy. Here the hyper- sphere represents the set of next states that the policy can generalize to. The chances approximate the effect of having more data samples to leverage for generalization. For policies learned with neural networks, this aims to represent the "interpolation" capacity of these models among the training samples.
Theorem A.1. Given a policy TA and deterministic demonstrator TE, assume that for state if (s) > O, then (als) = =N(TE(s), 1) Assume that transitions are normally distributed with fixed and diagonal variance, p(s'ls, = [u(ss, o2), where p(s. a) = + aa are simplified linear dynamics for scalar E. Then the next state coverage probability is PB(s, u(s, a); NN
1 1 (1
1+02(033) ))
Proof. First we note that both A; = p(s, , a) and Ai,- 8/ - ((8,0) are zero mean, and the events for A, > Ais x for all i are independent. Thus:
17

---

Pn(s,IEN) = N2 P2P(I 2 (1A.-119) =1- (1-(1-((AA11) & 11A),=13))
1 1- -1-
|A ( S
1|A.1P
N
Note that ||A; 1| is just a x2 random variable with degrees of freedom, and likewise for ||A,, 11 The former has variance o² (axr,) (transition plus learned policy noise), while the latter has variance o² (just transition noise). The f-distribution is defined for two X² variables X and Y of dimension d as the distribution of Z X/d
V7a Thus with a change of variables to X and Y, and
denoting Fa the CDF of the f-distribution of dimension d:
||11
(a² (1o
$ = P 1
o²Y
0²
P
Y
0² + (aop))
o²
Fa
0²
+ (aop)3
Fa
1+ (3²)
Plugging the above expression into the expression for PB:
1
N) =1 1 Pa
1 a+ (oe)
In Theorem A.I, while the probability of next state coverage is not immediately interpretable, can still intuitively recognize that high d, can improve the coverage likelihood even under notable op. and as N gets bigger even less a, is needed, despite the fact that system noise is present even under the learned policy. Intuitively, we once again see that increasing has a significant effect the coverage likelihood. In terms of noise, what matters is the ratio of policy to system noise, where increasing this ratio leads to sharp drops in performance at some cutoff based on N. We visualize this coverage probability in Fig. 4 under increasing ratios of policy to system noise for different values of N.
learned policy
B Metrics of Data Quality
Having formalized action divergence and transition diversity in Sec. 4 as two fundamental considera- tions dataset, how can we measure these properties given dataset?
Action Variance: To measure action consistency, the empirical form of the objective in Eqn. is intractable without access to the underlying expert action distribution æg. Instead we propose using the empirical variance of the action distribution in the data to approximate the "spread" of the data. In continuous state spaces, we can estimate variance using a coverage distance e to cluster nearby states, and then measuring the per dimension variance across the corresponding actions within said cluster. Defining cluster to be C'ss, D) {5, ā 5 E D ||s
all I E),
compute the variance
Action Variance (D)
Σ (a
Σ
a)2
D
s
(.
18

---

N-100
N=50
N=2 N-10
16
2.5
Figure PB(s, ie; N) (y-axis) from Theorem plotted for four-dimensional state under [2, 10, 50, 100]. but varying the ratio of policy to system noise (x-axis is =t). We see that under this more loose coverage model, with lots of samples, adding system noise can make coverage likely even under double or triple the noise in the learned policy.
The choice in corresponds to the generalization of the learning model to nearby states, similar to the notion of coverage in Definition We use this metric of action consistency in Sec. study human generated datasets of various quality.
State Similarity: To measure the consistency of states, we approximate the number of "nearby" states using the same clustering process in the Action Variance metric, and measure the expected cluster size as a fraction the overall data.
StateSimilarity(D) (D) = D
Σ IC(s, D) 8.ED
(10)
While these approximate forms do not encapsulate the full spectrum of possible metrics, believe these metrics help advance our empirical understanding of data quality for imitation learning. In section 5.2 the main text, we analyze these metries of data quality in several environments across different dataset sources.
C
Results
The performance results under system noise, policy noise, and both noises are shown with a broader sweep for both PMObstacle and Square in the tables below.
= 0.01
0.02
0.03
0.04
0.01
0.02
0.03
0.04
SCRIPTED
100
100
99
96
Tx = 0.01
97.7(1.5)
95.7(0.7)
96.7(1.1)
93.3(0.7)
90.3(7.1)
90.0(8.2)
94.0(2.9) 87.7(3.7)
a. = 0.02
98.7(0.5)
98.0(0.5)
97.7(1.)
92.7(1.2)
99.7(0.3)
98.0(0.9)
94.3(1.4)
92.3(2.0)
0.03
98.3(0.7)
98.0(0.8)
99.0(0.5)
95.0(0.9)
99.7(0.3)
98.7 98.7(0.5)
97.7(0.5)
95.7(1.1)
0.04
100.0(0.0) 100.0(0.0) 99.3(0.3) 96.7(0.7)
100.0(0.0)
99.0(0.5)
98.7(0.5)
96.7(1.4)
Table 2: System Noise: Success rates (and standard error) for BC in PMObstacle, for 1000 episodes (left) and 10 episodes (right) of data, under system noise. Rows correspond to injecting gaussian system noise (a,) into the dataset of increasing variance, and columns correspond injecting noise during evaluation. The diagonal both sub-tables represents evaluating in distribution. Left: For large datasets, higher system noise during evaluation tends to decrease the performance of each model (rows left to right), but more system noise during training generally produces the best models (columns to bottom). Right: For small datasets, we observe similar but exaggerated effect the left table.
19

---

a. 0.01 0, = 0.02 o, = 0.03 0, 0.04
0, = 0.01 0, = 0.02 o, = 0.03 o2 0.04
SCRIPTED
100
100
99
96
op 0.01
94.0(1.7)
94.0(2.4)
94.7(1.8)
91.3(1.4)
78.0(8.6)
81.3(5.0) 78.7(6.7)
81.3(5.8)
0p 0.02
87.7(2.0)
(2.3(2.0)
90.7(2.0)
91.3(2.2)
88.0(9.4)
78.7(4.4)
80.7(3.1)
80.3(3.2)
Op = 0.08
97. (7.9( (9.5( 97.0(0.0)
95.0(0.8)
88.7(3.2)
82.7(5.4)
88.7(5.2)
85.7(4.4)
= 0.04
86.7(1.3)
93.3(1.4) (1.0(2.4)
92.7(1.5)
88.3(6.5)
88.0(1.5)
86.0(5.9)
82.3(2.0)
Table Policy Noise: Success rates (and standard error) for BC in PMObstacle, for 1000 episodes (left) and 10 episodes (right) of data, under learned policy noise. Rows correspond to injecting gaussian policy noise (o,,) into the expert of increasing variance, and columns correspond to injecting system noise during evaluation. Left: For large datasets, unlike system noise in Table 2, more policy noise during training often produces the worst models (columns top to bottom). Right: For small datasets, adding policy noise produces large variance in performance across runs, Importantly, the datasets each row have the same observed state diversity as the corresponding row in Table 2, but performance is almost universally lower in both sub-tables here, supporting the idea that state diversity is coarse metric for success.
= 0.01 0, = 0.02 0, = 0.03 0, = 0.04
0. 0.01 O. 0.02 = 0.03 a. = 0.04
SCRIPTED
100
100
99
96
0p 0.01
96.3(2.2)
99.3(0.5)
97.7(0.3) 93.3(1.0)
99.7(0.3)
98.0(1.2) 96.7(0.5) 96.3(1.4)
Pp = 0.02
98.0(0.5)
98.3(0.5)
97.7(0.7)
94.7(1.0)
99.3(0.5)
98.7(1.1) 96.0(2.2) 94.7(1.7)
0p = 0.03
98.0(0.8)
96.7(1.0) 98.3(1.0) 96.3(0.7)
95.0(2.1)
97.7(0.5)
(7.7(
93.3(2.2)
0p = 0.04
98. 98.7(0.5) 99.0(0.5)
97.3(0.3)
95.0(0.8)
100.0(0.0) 99.7(0.3) 99.0(0.8)
93. (3.1(
Table 4: System Noise Policy Noise: Success rates (and standard error) for BC in PMObstacle, for 1000 episodes (left) and 10 episodes (right) of data, under learned policy noise for fixed amount of system noise (op 0.03). Here how system noise improves the robustness of the model added policy noise.
0, 0.05
= 0.1
0. = 0.2
0.3
= 0.4
0 0.05
55.7(5.3)
50.0(5.4)
27.0(3.9)
12.0(2.4)
7.3(2.2)
a, = 0.1
69.7(5.5)
69.0(3.7)
57.3(6.0)
50.3(6.0)
22.7(3.5)
0, 0.2
67.7(9.6)
68.7(12.6)
82.0(3.4)
74.3(2.7)
50.3(1.7)
0, = 0.3
47.0(4.9)
53.3(5.9)
54.3(3.0)
50.7(4.3)
38.7(7.3)
0.4
31.3(5.0)
37.7(8.2)
48.3(9.7)
49.0(8.7)
11.0(5.3)
Table 5: System Noise, 200ep: Success rates for BC in Square, for 200 episodes of data, under system noise. Rows correspond to injecting gaussian system noise (a,)) into the dataset of increasing variance, and columns correspond injecting noise during evaluation. The diagonal in both sub-tables represents evaluating in distribution. In both sub-tables we see how policies with low data coverage (low system noise) generalize the worst to increasing noise at test time. More system noise during training generally produces the best models (columns top to bottom).
a. = 0.05
0, = 0.1
0. 0.2
a. = 0.3
0. = 0.4
0.05
40.0(1.2)
33.7(3.1)
16.7(1.4)
1.3(1.0)
2.0(0.5)
G, 0.1
42.3(4.0)
39.7(3.8)
31.3(3.7)
19.7(3.1)
10.0(1.6)
O, 0.2
70.0(7.0)
73.7(5.4)
69.7(0.7)
55.3(1.2)
27.0(2.2)
O, = 0.3
57.3(3.1)
58.7(8.1)
64.7(1.4)
60.7(0.3)
44.7(2.2)
G. = 0.4
30.0(7.0)
33.7(6.4)
39.7(5.7)
39.7(6.5)
36.7(6.9)
Table 6: System Noise, 50ep: Success rates for BC Square, for 50 episodes of data, under system noise.
Rows correspond to injecting gaussian system noise (a.) into the dataset of increasing variance, and columns correspond to injecting noise during evaluation. The diagonal in both sub-tables represents evaluating in distribution. In both sub-tables we how policies with low data coverage (low system noise) generalize the worst to increasing noise at test time. More system noise during training generally produces the best models (columns top to bottom).
20

---

0.05
0.1
0, 0.2
0.3
0.4
Pp 0.005
69.0(3.7)
59.0(3.7)
34.0(1.7)
20.7(2.4)
7.0(1.6)
P 0.01
80.7(3.7)
78.0(4.2)
57.7(2.4)
38.0(1.7)
23.0(2.2)
0.02
62.8(7.8)
71.7(6.1)
73.0(3.9)
65.3(2.8)
43.3(3.6)
Table 7: Policy Noise, 200ep: Success rates for BC in Square, for 200 episodes of data, under learned policy noise. Rows correspond to injecting gaussian policy noise (ap) into the dataset increasing variance, and columns correspond to injecting noise during evaluation. In the high data regime, tends improve performance (columns top bottom), since the noise unbiased with enough samples from the scripted policy, the model recover an unbiased policy.
that more policy noise
0.05
0.1
0, 0.2
0.3
0.4
Op 0.005
32.7(3.8)
30.3(3.2)
18.0(3.6)
7.0(0.8)
5.7(1.2)
0 0.01
61.3(4.3)
59.0(6.7)
48.3(3.7)
29.7(1.7)
19.3(1.4)
0.02
57.7(3.2)
58.3(3.1)
49.3(0.5)
41.3(0.5)
28.3(2.4)
Table Policy Noise, 50ep: Success rates for BC Square, for 50 episodes of data, under learned policy noise. Rows correspond to injecting gaussian policy noise (a₂) into the dataset of increasing variance, and columns correspond to injecting noise during evaluation. As the amount of data is reduced, there is a significant drop in performance for added policy noise in the dataset, along with higher performance variation compared to 200eps, the policy can longer recover unbiased policy.
21


Reviews:
[{"review_id": "SNR7rTmXYF", "reviewer": "Reviewer_oFdr", "summary": "This paper systematically studied the data quality assessment problem in imitation learning, proposing new metrics based on test-time distribution shift. \n", "questions": "From Eq.6, the proposed data quality metric is calculated based on the expert policy \\pi_E and the learned policy \\pi_A by applying an imitation learning algorithm A. If one needs to perform imitation learning first and only after that can get to know how the data quality is, then what is the point of data quality assessment?\n\nCan the authors provide some explanations on how the proposed data quality metrics can be used to benefit the imitation learning?", "limitations": "Not discussed in the paper.\n\n", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well-organized and well-written. The analysis seems to be in fair detail.", "weaknesses": "One of the drawn conclusions seem straightforward to me (\"increasing state diversity can often come at the expense of action divergence\"). Increasing state diversity will bring more challenges for the learning and thus may cause increased action divergence. \n    \nThe paper lacks practical insights, e.g., how the proposed data quality evaluation metrics can help guide the data collection and thus help the imitation learning is unclear. Currently I dot not see the benefits of introducing such data quality evaluation metrics. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and we appreciate that the reviewer felt our analysis was in fair detail.\n\n**The conclusion on state diversity seems straightforward.**\n\nWe agree that this conclusion is intuitive; however, we disagree that it is trivial: state diversity has been the main metric of “quality” in much of the imitation learning and active learning literature (e.g., when defining a novelty score), and so we strongly believe this point is novel and impactful.\n\nSee our main comment response (Straightforward Results).\n\n**How can the proposed metrics guide data collection?**\n\nWe agree that data quality metrics derived from our theory should eventually be used to guide data collection, and thus we included several practical insights in the “Implications on Data Curation” section (4.3). For example, increasing action consistency during data collection is one potential way to handle action divergence. Furthermore, as mentioned previously, the conclusion on state diversity is extremely practical for data curation. We show the effects of state diversity and action consistency in our experiments. If the reviewer is unclear about how data quality metrics can be useful more generally, we have provided some potential use cases for quality metrics in the future:\n\n1. To filter datasets and only train on high quality examples without expensive human cleaning effort (which currently requires extensive manual labor and is highly subjective).\n2. To improve the quality of a given demonstrator online, e.g. through real time feedback like a score function that they can optimize.\n3. To weight examples during learning.\n\nWe also point the reviewer to the Data Measuring experiments in Section 5.2, where we evaluate some initial relevant metrics on various human collected datasets. We see our work as a first step in defining a comprehensive set of data metrics for practitioners to use before training their model.\n\nWe hope this provides a broader picture of the utility of data quality metrics in imitation learning.\n\n**How is data quality (Eq.6) useful if we need to first run the algorithm to evaluate it?**\n\nSee our main comment response (same title).\n\nPlease let us know if you have any further questions or concerns, and in light of our response, we hope that you will consider raising your score.\n"}, {"review_id": "7HwgkqSIzQ", "reviewer": "Reviewer_zh6Y", "summary": "Data quality is an important aspect of offline robotics because there is not a large quantity of data. In particular, in imitation learning, it has been shown that naive training policies on low-quality data can lead to distribution shift issues. Most researchers focused on dealing with this issue using algorithm-centric approaches. Instead, this paper advocates for a data-centric approach for addressing this issue. The data-centric approach modifies the data collection process, where the data collection can be broken down into two categories: action divergence and transition diversity. The authors formalize these two categories of the data collection process and provide empirical evidence of their significance. ", "questions": "Below are several questions that I have regarding the proposed idea:\n- What is the difference between noise in the expert policy and sub-optimal expert? If there is no difference, then is action divergence focuses on suboptimal experts, not experts?\n- In the system noise section, the authors posit that \"the expert demosntrators should encourage paths with high system entropy\";  how is this concept different than DAgger[1]?\n- Why is data quality affected by the algorithm? The algorithm simply uses the data while the expert provides the data. In other words, the algorithms consumes the data while the expert produces the data.\n- If the expert is optimal and the transition is not deterministic -- that means that when the expert takes action in state s it will, with some probability end up one of many states, for example. Expert data with a noisy transition function means you get more state coverage. This implies that the expert and policy will have more overlap. What is the difference of this idea and the idea discussed in [2] -- which cover the case when the expert and the learner policy have state distribution overlap. \n\n[1] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Ross et al.\n[2] Feedback in Imitation Learning: The Three Regimes of Covariate Shift Spencer et al.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Formalizing and addressing the data quality issues in imitation learning is challenging. This paper proposed interesting ideas to this challenging problem and has the following strengths:\n- Clarity: The overall paper is well written, and the problem the authors would like to address is clear. The two categories of data collection are also very clear and make it easy to understand the underlying problem dataset attributes.\n- Empirical Justification: The author's performance experience using synthetic noise, single and multi-human datasets. All of the experiments performed across these datasets provide insight into various aspects of the data collection discussed in the author's paper. These insights helped enhance the author's discussion of data collection attributes proposed by the authors.", "weaknesses": "Although the authors proposed an interesting perspective to address a difficult problem in imitation learning, there exist several weaknesses of the paper:\n- Lack of novelty: Theorem 4.1 was already proposed and proven in [1] (see theorem 3 appendix). That means the action divergence interpretation was already discussed in the literature. Furthermore, transition diversity was already proposed and discussed to address distribution shift issues in [2].\n- Lack of explanation: The authors emphasize that past work only focuses on state coverage while ignoring that action's role. But if the policy providing the demonstrations is an expert,  then the states provided from the expert should inherently have low action divergence because this is an expert;  and an expert provides the optimal action for a given state.\n- Lack of citation: The section on distributions shift in IL needs proper citation.\n\n\n[1] Imitation Learning as f-Divergence Minimization by Ke et al.\n[2] Sequence Model Imitation Learning with Unobserved Contexts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and we are glad the reviewer felt our empirical evidence supported our claims.\n\n**Lack of novelty**\n\nThe theorem the reviewer cites as similar to 4.1 in [1] looks similar, however it makes the strong assumption of the f-divergence satisfying the triangle inequality, which is not true for the KL divergence we use in 4.1, and also yields a different bound. Furthermore, the implications of the divergence relationship on data quality (i.e. the data generating policy distribution) is not examined within this prior work. They are focusing more on the algorithmic perspective, as is common in prior work. We now explicitly refer to the similarities and differences to this theorem in our discussion. For transition diversity, the concept of transition probabilities is of course not new, however to our knowledge the relationship between state diversity, transition diversity, and action diversity has not been analyzed before to the same extent as our work. We could not find any specific discussion of transition diversity and its role in data quality in the provided citation [2], and so we ask the reviewer to clarify to what specifically they were referring. To our knowledge, [2] instead makes claims about the benefits of state diversity (i.e., low density ratios being beneficial) as is common in prior work.\n\nFurthermore, we believe our conclusion about state diversity is a key contribution: state diversity has been the main metric of “quality” in much of the imitation learning and active learning literature (e.g., when defining a novelty score, or in [2]), and so we strongly believe this point is novel and impactful.\n\nSee our main comment response (Novelty).\n\n**Usage of “Expert” Policy terminology**\n\nThe reviewer correctly notes that a true expert would likely not have high action divergence. This is a flaw in our terminology: the expert policy was not meant to imply a perfect demonstrator, and we apologize for the confusion. In the data quality setting, this policy is just the data generating policy, which is defined solely as one that successfully completes the task (this is a more realistic assumption than having access to a true expert). We have updated the paper to reflect this terminology change. We hope this also answers the reviewer’s other question about the difference between expert and sub-optimal expert policies.\n\n**Lack of Citation**\n\nWe have updated the related work to include [1] and [2], and we are open to any other citations the reviewer believes we should include.\n\n**Difference between System Entropy and DAgger**\n\nWe posit that expert demonstrators might encourage paths where the system entropy is higher in order to increase transition diversity (purely offline). DAgger involves querying an expert at robot visited states (online). DAgger makes no claim of which action “strategy” (high or low transition diversity) is better, but rather just relabels any visited states by the robot. In contrast, we are positing that in the offline setting, experts can choose action strategies that give them more coverage over states without increasing action divergence.\n\n**How is data quality (Eq.6) useful if we need to first run the algorithm to evaluate it?**\n\nSee our main comment response (same title).\n\n**State Distribution Overlap compared to the Goldilock’s Regime in [2]?**\n\nThe high state coverage “Goldilock’s regime” considered in [2] allows them to bound the density ratio between expert and visited state distributions. This is part of a broader class of works that consider the benefit of state diversity, with the loose intuition that state diversity helps us stay in distribution by either expanding the possible successful trajectory space or by introducing recovery behaviors to bring us back in distribution. The difference with our discussion of transition diversity is that we are decoupling action divergence and state diversity. While prior work including [2] broadly consider state diversity as beneficial, the idea that higher state diversity necessarily means we will learn a “recovery” policy is invalid if we have high action divergence. Instead, we look at just how transition diversity can change the state distribution, and thus increase state distribution overlap between expert and policy without incurring high action divergence.\n\nPlease let us know if you have any further questions or concerns, and in light of our response, we hope that you will consider raising your score.\n"}, {"review_id": "VKLtNw8c84", "reviewer": "Reviewer_24NK", "summary": "The paper addresses a key issue in Imitation Learning (IL) called as distribution shift. The paper formalize data quality and data curation for IL. The paper propose two fundamental properties to evaluate/measure data quality as \"action divergence\" and \"transition divergence\". The authors investigate these two properties in context of data quality for IL and presented theoretical and empirical results. They conducted an experiment on human/expert demonstration data and provide analysis on data noising and data measuring.", "questions": "- addressed in Weakness section", "limitations": "- Authors are encouraged to explicity mention the limitations of the paper", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written\n- The idea presented in the paper is very important for IL", "weaknesses": "- The work doesn't seems to fit for a high quality conference like NeurIPS\n- On measuring the data quality and its effect on success rate, a lot of research is being done on learning from mixed data as its really hard to classify expert vs non-expert data.\n- The paper seems more like a white paper or can be submitted to AI Safety conferences but not for NeurIPS\n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and we appreciate that they agree of the importance of the ideas presented in the paper.\n\n**The work doesn't seem to fit for a high quality conference like NeurIPS.**\n\nWe disagree that this work is not fit for NeurIPS. As noted by several other reviewers, the question of data quality is an important one for practitioners of imitation learning, which has been largely ignored in recent years. Many people within research and industry  spend countless hours defining clean demonstration strategies, teaching demonstrators to be “proficient,” and filtering data after collection. Yet as a community we do not have a clear understanding of what makes a demonstration clean. In fact, several works submitted in NeurIPS in recent years have used data diversity as a metric for quality [1, 2]. Our work is the first to actually define data quality in imitation learning beyond simplistic notions of “state diversity”, and derive some useful practical insights about state diversity, action divergence, and transition diversity for data curation, which we also show empirically. We value the reviewer’s opinion, and we ask them to elaborate and to provide us with more concrete suggestions for how we can improve our work.\n\n**It’s hard to classify experts vs. non-experts, and thus algorithmic research is more promising**\n\nWe completely agree on the difficulty in classifying expert vs. non-expert data; however, in our view this does not make it any less valuable or worth pursuing! We respect the reviewer’s opinion that algorithmic research is more promising, and we hope both avenues can be explored in parallel. \nRecent work in data retrieval [3, 4] have also shown that without changing the algorithm, one can get better policy performance by filtering the data. Therefore, we believe improving dataset quality is an important agenda and with more community focus on data quality, what is considered difficult now will eventually become highly impactful.\n\n**The paper seems more like a white paper or can be submitted to AI Safety conferences**\n\nOur paper includes both theoretical results and empirical analysis that is extremely relevant to the machine learning community, and thus does not fit the definition of a white paper. To add, our primary focus is on improving data quality for final policy performance, and thus we are not focusing on implications of safety in our work. Therefore we do not see how our work would be a fit for an  AI Safety conference.\n\nPlease let us know if you have any further questions or concerns, and in light of our response, we hope that you will consider raising your score.\n\n[1] Rajaraman, Nived, et al. \"Toward the fundamental limits of imitation learning.\" Advances in Neural Information Processing Systems 33 (2020): 2914-2924.\n\n[2] Nguyen, Thao, et al. \"Quality not quantity: On the interaction between dataset design and robustness of clip.\" Advances in Neural Information Processing Systems 35 (2022): 21455-21469.\n\n[3] Nasiriany, Soroush, et al. \"Learning and retrieval from prior data for skill-based imitation learning.\" arXiv preprint arXiv:2210.11435 (2022).\n\n[4] Du, Maximilian, et al. \"Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets.\" arXiv preprint arXiv:2304.08742 (2023)."}, {"review_id": "8mLYYtxyHi", "reviewer": "Reviewer_9hHD", "summary": "This paper studies imitation learning (IL) from a theoretical perspective. It proposes the idea that we should curate the data to improve performance, rather than solely relying on designing more robust algorithms, which is a more common practice in the field. The authors propose data quality for a particular IL algorithm, which is the negative of the divergence between the expert policy and the learned policy using that algorithm. They then show theoretically how two properties, action divergence and transition diversity, are linked to data quality. The authors then study the finite-data regime under a very simplistic setting, and finally show some results of studying their metrics on PMObstacle, Square, and Robomimic domains.", "questions": "L147: I don't see why it's a \"test time\" objective. You would be optimizing your policy with this f-divergence loss, so it comes into play during training time, right?\n\nTheorem 4.1: What does it mean to have a superscript on D_KL? Is this term the same as the LHS of Lemma 4.2?\n\nDo any of the intuitions in this paper transfer to state-only IL, where our dataset doesn't have actions at all? This is becoming an increasingly common setting in the new world of large-scale pre-training (e.g., from human videos).", "limitations": "Yes, addressed adequately.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The presentation / quality of the writing is top-notch. I really appreciated the clear descriptions throughout the paper, starting from the well-written introduction to the explicit problem setting to the way the authors use English to describe and contextualize all the math.\n\nRelated work section is comprehensive. I like how it addresses both algorithmic and data robustness.\n\nI also really like the framing of Equation 6. The insight that data quality is a function of the learning algorithm is very nice.", "weaknesses": "I am unfortunately not well-versed in IL theory, so I cannot comment on the novelty of the work with respect to the rest of the field. Therefore, I will only evaluate the paper on the merits of the contributions stated by the authors.\n\nMy only major weakness is about the unrealistic conditions underlying some of the theory.\n* One of the conditions of Lemma 4.2 is that states in the support of the expert policy would have low divergence between actions under the expert policy and the learned policy. The authors say in L226 that with a good learning algorithm, this bounding constant \\beta would be small, but I don't see why this would be true in general. For BC, maybe it makes sense, but for other IL algorithms, a \"good\" algorithm could sacrifice exactly matching the expert at some states, for the sake of getting better interpolation/generalization to states outside of the support of the expert policy.\n* Theorem 4.3 studies only a very simple setting where the environment dynamics are fixed Gaussian with diagonal variance. I'm not sure how much we can take away from this Theorem as this condition will almost never hold in practice. As a consequence, I find takeaways like that in L254 to be misleading because one cannot draw general conclusions from such a restricted Theorem.\n\nOther comments:\n* One more broad concern I have is that most of the robotics domains studied in the RL literature have deterministic dynamics, including Robomimic (ref: https://arxiv.org/abs/2305.14550) and Mujoco. So how can we obtain transition diversity in such settings?\n* Theorem 4.1 only makes sense with stochastic policies due to the KL divergence. It's not clear how to apply these findings to deterministic policies, which are quite common (and in fact, are used in the experiments of this paper).\n* L230, L290: It's unclear what the authors mean when they say the data should maximize system noise. That's a property of the environment dynamics, not the expert policy; we don't have control over it!\n* It is hard to gain too many insights from Table 1. A simpler explanation for the observed success rates is that as task horizon increases while data size stays fixed, BC simply has less data for each state, leading to worse policies. I'm not really sure why action variance and transition diversity come into play here.\n\nMinor things:\n* The authors use the word \"expert\" throughout the paper. I would rather say the \"behavior policy\" or \"data generation policy\", because the whole point of this paper is that we are trying to curate the dataset, e.g. by introducing more diversity, which means the data may not end up looking like what an expert would do.\n* It would be good to say in the preliminaries that you're in a continuous state/action space setting.\n* L73: remove \"how\"\n* L167: \"\\pi_A should match \\rho_{\\pi_E}(s)\" --> I think you mean \"\\rho_{\\pi_A} should match \\rho_{\\pi_E}\"?\n* L167: \"learns only\" --> \"learns only from\"\n* Lemma 4.2 RHS, subscript of the expectation: \\in should be \\sim?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and we are excited that the reviewer found our insight about the algorithm's role in data quality to be very nice.\n\n**Theoretical Assumptions**\n\n1. *Why should $\\beta$ be small in Lemma 4.2?*\n\n$\\beta$ corresponds to the in-distribution loss of the model, and to our knowledge, no imitation learning algorithms would have unbounded in-distribution loss. For example, intervention based methods, adversarial methods, and energy based methods still utilize the training data to model high probability actions. The reviewer states other IL algorithms might sacrifice exactly matching the expert at some states to get better generalization, however once again, to our knowledge, the in-distribution loss is usually still bounded or part of the training objective. We’d be interested in considering any specific algorithms that the reviewer is referring to.\n\n2. *Gaussian system and policy noise*\n\nWe agree with the reviewer that the assumption of Gaussian noise is somewhat simple. We believe that making any further assumptions about the type of system or policy noise would be less general, but this point is well taken and we have included it in our limitations (see one-pager). Modeling and analyzing system noise in real environments is an interesting direction for future work.\n\n**Deterministic Transitions and Policies**\n\nThe reviewer raises a concern about obtaining transition diversity in settings with deterministic dynamics. One manner is similar to DART, cited in our paper, where one can artificially inject system noise through the action itself, while learning from the unaltered action. However, in practice, outside of simulated environments we rarely have deterministic dynamics.\n\nThe reviewer is correct that 4.1 does not directly apply to deterministic policies, and this is an interesting question. Intuitively, many divergence functions are not ideally suited for comparing two deterministic policies or one stochastic and one deterministic policy, but we might still be able to use naive definitions of “distance” like state MSE in place of the state distribution divergence used in 4.1. However, without more assumptions about the transition function, it is hard to translate state MSE into action MSE to derive a similar theorem to 4.1. Instead, one approximate solution might be to replace the deterministic policy with a unimodal gaussian (with constant variance) for sake of analysis: training a Gaussian with fixed variance with NLL is equivalent to action MSE, and then theorem 4.1 can still apply. Of course, the visited state distribution during deployment (deterministic actions) will be a subset of the visited state distribution used for computing the action divergence (probabilistic actions).\n\n**Maximizing System Entropy**\n\nIn the paper we posited that expert demonstrators might encourage paths where the system entropy is higher in order to increase transition diversity. The reviewer is correct that we cannot control the system noise in an environment; however, system noise is conditioned on state and action, and we can still control which part of state/action space we explore during demonstration (i.e., when different parts of state/action space have different system noise). Our claim here is not to increase system noise at a fixed state, but rather that experts can choose different trajectories to demonstrate that are still successful but have more overall system entropy.\n\n**Data size staying fixed in Table 1**\n\nThe reviewer states that a simpler explanation for the observed success rates is that BC has less data at each state when data size stays fixed, and thus performance suffers. We clarify that the dataset size is not fixed in terms of number of transitions, but rather number of episodes. Thus longer horizon datasets actually have *more* transitions, and thus should benefit from more state visitation by the same logic. And yet, performance suffers, so we look to action divergence and transition diversity to help understand the results. We will clarify this in the main text, and apologize for the confusion.\n\n**Usage of “Expert” Policy terminology**\n\nThe reviewer correctly notes that a true expert would likely not have high action divergence. This is a flaw in our terminology: the expert policy was not meant to imply a perfect demonstrator, and we apologize for the confusion. In the data quality setting, this policy is just the data generating policy, which is defined solely as one that successfully completes the task (this is a more realistic assumption than having access to a true expert). We have updated the paper to reflect this terminology change.\n\n**Distribution shift as a Test time objective**\n\nThe objective in Eq.5 is not a training objective, but rather the objective by which we measure the distribution shift at test time. The subtle difference is the order of the f-divergence. We are evaluating the divergence $\\rho_\\pi$ on the distribution induced by $\\rho_\\pi$ (test time distribution) not $\\rho_{\\pi_E}$ (training time distribution).\n\n**Superscript on $D_{KL}$**\n\nYes this is the same as the LHS of Lemme 4.2. We will expand this out in the text, sorry for the confusion.\n\n**State-only IL**\n\nIn state-only IL (e.g. not predicting actions), much of our analysis does not transfer over, since we are primarily dealing with the relationship between actions and states.\n\nPlease let us know if you have any further questions or concerns, and in light of our response, we hope that you will consider raising your score.\n"}, {"review_id": "yRiDkOIoZK", "reviewer": "Reviewer_WXvd", "summary": "This paper considers how to evaluate  the quality of a given dataset for an imitation learning task.  It is well-known that IL suffers from distribution shift. While much work on IL focuses on improving algorithms, the authors propose that another important approach to making IL more effective is to curate better datasets and demonstrations from experts. Existing approaches to evaluating the quality of datasets focus on state coverage / diversity, trying to ensure that the expert demonstration contains instances of a variety of different states.\nThe authors discuss in some depth how state diversity is an incomplete measure of dataset quality, specifically because it ignores the actions taken in the dataset; the goal of the paper is to find an improved metric for evaluating dataset quality. he paper proposes action diversity and transition diversity.  The authors then consider  the distribution shift from the expert policy to the learned policy under a given f-divergence (KL divergence by default), as having a large distribution shift can cause problems in IL.\nThey then show a number of properties of their measure and relate it to action diversity and transition diversity , and discuss how these metrics can be applied to improve IL in general (4.3, Implications for Data Curation). They claim that their work suggests that action consistency in the expert demonstrations leads to better outcomes, while state diversty and system noise in the dataset are good or bad depending on context.\nFinally, they perform experiments on two simple simulationtasks, comparing the effects of adding policy noise to adding system noise in the training set and comparing how demonstration datasets of varying (known) quality correlate with various metrics.", "questions": "This paper would be greatly improved by reducing repetition and adding more experiments.  The points made in the Intro are repeated at least 5 times.  The paper would be much better if the authors agree to remove this redundancy and add more thorough experiments.\n\nA number of the authors' conclusions (regarding what their results imply about creating and evaluating IL datasets) do not seem to follow obviously from their results and experiments. For instance, they claim in 4.3 (line 294-295): \"we posit that expert demonstrators should encourage paths with high system entropy [up to a point]\" – this doesn't really seem to follow from their experiments, in which system noise is added to a fixed policy, rather than changing the policy itself.\nThe authors claim to show that action divergence is undesirable and leads to worse outcomes, while transition diversity (in moderation) can improve results; this is demonstrated by adding policy noise during training and comparing the resulting policy to that learned from a dataset with system noise. However, this only shows that action divergence is harmful when it results from noise being artificially added (in which case the expert can be viewed as only partly trying to achieve the task, while also being partly random).\nWhat about the effect of action divergence resulting from multi-modality (i.e. when many very different courses of action can achieve the goal in a reasonable way)? It is not at all clear that the proposed measures of data quality will work for multi-modal tasks –  which have different solutions, e.g. even simple ones like moving around an obstacle (where you can go to the left or right).\nIt would be interesting to consider how this can be used to guide human expert demonstrations; while I'm not convinced by the discussion in section 4.3, at least without more details (see the first item under Questions), there are possibly other interesting avenues to consider, such as adding noise to the human expert's controls when conducting demonstrations.", "limitations": "There is no Limitations section in the paper.\n\nThe limitations of the work are only very briefly addressed (lines 390-392) and need more expanding; in particular, the weakness mentioned (that the data quality metric proposed is very hard to measure in real datasets) is a serious one and deserves a much broader discussion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main idea of finding improved metrics for dataset quality, above and beyond state space coverage, is interesting. Furthermore, the dichotomy that their analysis exposes between expanding state space coverage via higher action divergence versus higher transition divergence is an interesting one, showing that not all state space coverage is created equal. The theoretical results linking Q with action divergence are also interesting and well-presented.\nWhile there are questions left by the experiments (see Weaknesses and Questions) one extremely interesting finding is that the additon of a small amount of system noise vastly improves the effectiveness of training with policy noise (compare Figure 2 (c)(d) with Figure 2 (i)(j)), especially when the number of demos is low. The effect of this is so strong that it would have been better to focus more on it when discussing the results.\n\nWith a few minor exceptions, the paper is also very well written. The essential ideas are clearly stated and easy for the reader to absorb.", "weaknesses": "This paper would be greatly improved by reducing repetition and adding more experiments.  The points made in the Intro are repeated at least 5 times.  The paper would be much better if the authors agree to remove this redundancy and add more thorough experiments.\n\nhe biggest weakness is a key property of the proposed measure (equation (6)), which is tied to a strength of the work (\"data quality is heavily tied to the algorithm A that leads to the learned policy \\pi_A as well as the expert policy \\pi_E\"): since it takes the learned policy \\pi_A as well as the expert policy \\pi_E, actually measuring the quality Q may require running A, which somewhat diminishes the value of the data quality metric since the it may not be usable when collecting the data (note that most common metric, state diversity in the training dataset, can be measured without running A). Even one of the proposed proxies for Q, action divergence, requires having the learned policy to measure against the expert policy.\nThere is an inconsistency in lines 196 and 199-200 with Theorem 4.1: it says the data quality is \"lower bounded by the action divergence\" but Theorem 4.1 shows that action divergence provides an upper bound to the data quality. Since upper and lower bounds for dataset quality have different implications this is very important to fix.\nWhile Figure 2 (i)(j) represents the most interesting finding (in my opinion) of this work, it doesn't escape notice that there is no corresponding Figure 2 (k)(l) showing the same plots for the Square nut task. These should absolutely be included to show whether or not the finding in (i)(j) generalizes to at least one other task (otherwise there is the possibility that (i)(j) is simply a property of one particular setting).\nThe authors do not include a limitations section.\nMinor weaknesses and typos:\nIn (3), the sum should go for t=1 to T, not H\nFigure 2 caption: In \"X-axis corresponds…\" should probably write \"injected system noise\" twice (rather than \"injected Gaussian noise\" the first time and \"injected system noise\" the second time). They're both system noise and both Gaussian, right? Writing it differently may convey the mistaken impression that they might not be both Gaussian, whereas the difference is just when the noise is added. Notation is also confusing, it seems \\sigma_s is used both to refer to system noise at evaluation (in the figure itself) and to system noise added in the training set (line 342).\nTable 1: instead of merely stating that state similarty is the \"opposite of state diversity\" (line 360), it would be much better to have a more precise definition. Additionally, it is claimed that \"state diversity once again is not correlated with performance on the task\" (line 370, just above Table 1) but Better/Okay/Worse all seem to have similar state similarity values while Proficient Human (PH) has lower state similarity (i.e. higher state diversity) and has significantly better success rates – doesn't this mean that it is correlated? Claims of no correlation should probably come with the actual correlation number, since it isn't obvious from the table that there really is no correlation.\nEquation (6) should probably be in a Definition since it's one of the key contributions of the work.\n4.3 Implications for Data Curation might be better placed after the experiments section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review, and we are glad the reviewer found both the theoretical and empirical analysis important and interesting.\n\n**Reducing repetition**\n\nWe are sorry that the reviewer felt the paper was repetitive! Our intention was to clearly state the takeaways, but we recognize that the text can be shortened. We will fix this in the main text.\n\n**How is data quality (Eq.6) useful if we need to first run the algorithm to evaluate it?**\n\nSee our main comment response (same title).\n\n**Theorem 4.1: Lower bound**\n\nThe reviewer is correct that the Theorem 4.1 shows an upper bound for the KL divergence, however *quality* is the negative of this KL divergence, and is thus lower bounded.\n\n**Adding Figure 2(k)(l)**\n\nWe agree that showing effects of combined system and policy noise for the Square task is important, so we have included findings for this figure in the one pager (see main response), and will add this to the main text. To summarize the results, we once again find that adding system noise to the training data makes the learned policy more robust to policy noise. We thank the reviewer for pointing this out and hope this completes the story of our experimental section.\n\n**Adding a Limitations Section**\n\nWe apologize for not explicitly including our limitations as part of the text. We will address our limitations in the updated text, and we have included a draft of the limitations in the attached one pager.\n\n**Experimental evidence for high system entropy up to a point**\n\nThe claim that we should encourage paths with high system entropy up to a point is backed up by results for the Square dataset (a more complex task than PointMass Obstacle which has a much larger state space), where increasing system noise only helps performance up to a point, for both 200 and 50 episode datasets.\n\n**Action divergence resulting from multi-modality**\n\nThe reviewer raises the point that action divergence resulting from multi-modality is different from action divergence stemming from randomness. We highlight that in the PointMassObstacle task, there is in fact multi-modality already present, where the data consists of going either left or right around the obstacle. Without this multi-modality, we find that performance increases to near 100% even for many values of both system and policy noise. This emphasizes the point that action divergence resulting from multi-modality is equally or more harmful than action divergence stemming from randomness. We will add this result to the Appendix and clarify this in the main text. We acknowledge that better metrics are needed for measuring this multi-modality, and we are excited to pursue this in future research.\n\nWe also evaluate PointMassObstacle with transformer models that can supposedly handle more multimodality, however as seen in the results (one-pager), even transformers suffer from policy noise but benefit from system noise. This provides additional evidence of how existing models struggle to learn from multi-modal and noisy demonstrations.\n\n**State similarity for PH vs. Better/Okay/Worse**\n\nThe reviewer is correct to note that state similarity is lower for PH than Better, however importantly Better actually represents overlapping data with the PH dataset (Better consists of 2 proficient demonstrators). We use this as calibration between different dataset sizes, and we see that there is some effect of dataset size on our chosen metrics. Furthermore, within Better / Okay / Worse we see either the opposite trend (Square) or uncorrelated performance (Can) with state similarity, and thus it would not be fair to say more state diversity leads to better policies. We have included this point in the analysis.\n\n**Miscellaneous**\n\n- We thank the reviewer for the various typos pointed out, and will fix them for the final manuscript. \n- We apologize that notation was confusing for using \\sigma_s as both training and evaluation noise. We have now distinguished these two in the paper and figure. \n- A precise definition for state similarity is present in Appendix B.\n\nPlease let us know if you have any further questions or concerns, and in light of our response, we hope that you will consider raising your score.\n"}, {"review_id": "x4mYfhO8Xk", "reviewer": "Reviewer_AXdc", "summary": "The paper analyze several theoretical property of the expert demonstration distribution to the performance of the learnt policy in imitation learning. The paper claims that people should provide consistent and transition and state diversity.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper defines several quantity to clear express the meaning of the properties. The theorems are sound to me.", "weaknesses": "The results claimed by the paper are straightforward to me, which makes the contribution less significant. The theoretical results are basically quick derivation of standard machinery of MDP analysis, which are trivial and straightforward.\n\nThe paper claims that expert should provide consistent demo based on BC analysis. However, the paper does provide experiment of the types of imitation learning methods like transformer architecture to handle multi-modal distribution.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review.\n\n**Straightforward results**\n\nWe agree with the reviewer that several of the conclusions are intuitive; however, we disagree that they are trivial or straightforward. See our main response comment (Straightforward Results).\n\nThe reviewer also states that the theoretical results are based on simple derivations and the “standard machinery of MDP analysis”, but simplicity is not the same as significance. The impact of theoretical results depends on how well they explain practical results, not how simple the proofs are to understand. Our theoretical claims are actually supported by the practical results on several environments, and as exemplified above this could have high significance in the field of imitation learning.\n\nOn the point of novelty, we emphasize that ours is the first work to form a full picture of data quality and glean useful properties from it. We are hoping this work inspires a paradigm shift to focus on improving data quality rather than just changes to any given algorithm.\n\n**Transformer Architectures for Multimodality**\n\nThe reviewer requests additional experiments involving transformer based architectures that can handle multimodality. We have included these results for PointMassObstacle in the attached one-pager. Overall, we see that transformers still suffer from policy noise (i.e., action divergence), while they benefit from system noise (i.e., transition diversity).\n\nRecently, prior work has studied these transformer architectures in the multi-task setting as well, for example RT-1 [1] and RT-2 [2]. They use massive datasets and incredibly large transformer architectures, for relatively simple tasks (pick and place), and still their performance leaves much to be desired. This is further evidence that large transformer-based architectures can’t overcome challenges associated with uncurated datasets, even as datasets grow to hundreds of thousands or millions of trajectories. As the field trends towards bigger models and datasets, we argue intelligent data curation (e.g. through metrics of data quality like those proposed in Section 5.2) might be a better use of time and resources.\n\nPlease let us know if you have any further questions or concerns, and in light of our response, we hope that you will consider raising your score.\n\n[1] Brohan, Anthony, et al. \"Rt-1: Robotics transformer for real-world control at scale.\" arXiv preprint arXiv:2212.06817 (2022).\n\n[2] Brohan, Anthony, et al. \"RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.\" https://robotics-transformer2.github.io/assets/rt2.pdf\n"}]

Author rebuttal:
We thank all the reviewers for their detailed and constructive feedback. We are excited that the majority of reviewers agree about the importance of the data quality problem in imitation learning and find our analysis sound.

We outline the points of important feedback that came up several times in the reviews below:

**Novelty**

We apologize for any confusing presentation of the work leading some reviewers to fail to see the contributions of our work. Our contributions cannot be properly assessed without understanding the broad literature of current developments in machine learning and the history of IL research. Recently there has been a paradigm shift in NLP and CV from developing large models to curating high-quality datasets, and we believe robotics can learn from these lessons as we are starting to see increasingly large models. A body of IL research relied on near-optimal or machine-generated datasets before the community realized machine-generated data is not representative of human data [1]. The community largely has chosen to collect large amounts of human data and hope algorithmic fixes can overcome the challenges of uncurated data, for example with large transformers like RT-1 [2]. However, as the results of these prior works show, algorithmic changes without close attention to data quality can only take us so far. Instead our work shows that a more efficient and principled approach is to analyze what properties of machine-generated data allow simple methods to work and how to curate human-generated data to be friendly to existing IL models.

We would like to re-emphasize the core contribution of our work: we are the first to formalize data quality in imitation learning in an effort to guide data curation in practice. Our work is a first step towards this paradigm shift in the IL community and we hope the reviewers can see the importance of this message. In addition, our work presents two key insights that challenge common beliefs of the IL community:
- the downstream learning algorithm is an important factor when assessing data quality
- state diversity alone is not sufficient for measuring data quality

Empirically, we study how action divergence and transition diversity in a dataset influence downstream performance in controlled settings. While our analysis informs what properties we want a dataset to have (i.e., higher transition diversity up to a point, lower action divergence), we also show the difficulty in designing universal data quality metrics, opening an important problem for the community. 

**Straightforward Results**

Two reviewers found some results to be straightforward. While we agree several of the conclusions are intuitive, we disagree that they are trivial of straightforward: state diversity has been the main metric of “quality” in much of the imitation learning and active learning literature, and so we strongly believe this point – going beyond simply optimizing for state diversity as done in prior work – is novel and impactful [3,4,5].

Furthermore, simplicity is not the same as significance. The impact of theoretical results depends on how well they explain practical results, not how simple the proofs are to understand. Our theoretical claims are actually supported by the practical results on several environments, and as exemplified above this could have high significance in the field of imitation learning.


**How is data quality (Eq.6) useful if we need to first run the algorithm to evaluate it?**

To be clear, the definition in Eq.6 is a theoretical definition of data quality, and acts more as a north star rather than a tractable metric. Our work is the first to define and study quality at a theoretical level. We spend the rest of the theoretical section using this formula to derive more tractable metrics like action consistency, horizon length, and transition diversity, which do not require an algorithm to evaluate. This question is precisely what makes analyzing data quality so difficult, and our work is the first to recognize this relationship between data quality and the algorithm, formalize it, and analyze it in practice. Our experiments in 5.1 and 5.2 show how this theory can be useful in practice: by understanding properties of a given algorithm *class* (e.g. the benefits of action consistency for behavior cloning methods), we can bypass the need for model training to evaluate data quality.

**Full list of changes and updates:**
- Discussion of fairness and safety implications for data quality metrics (see one-pager)
- Adding a limitations section for theoretical assumptions, metric limitations, etc (see one-pager).
- Adding system noise + policy noise experiments for Square (see one-pager)
- Adding transformer based experiments to Appendix (see one-pager)
- Adding results on system and policy noise without multimodality in PMObstacle
- Motivating how we would use metrics to guide data collection in the text
- Changing “expert” to “demonstrator” policy
- Including missing citations suggested by Reviewer 2 (zh6Y)
- Clarifying dataset details for Section 5.2, and longer analysis of Metrics in Section 5.2
- Rewriting the KL divergence using the same notation as the left hand side of Lemma 4.2
- Various typos presented by the reviewers
- Better explanation of the experiment figure and better labeling

[1] Mandlekar, Ajay, et al. "What matters in learning from offline human demonstrations for robot manipulation.".

[2] Brohan, Anthony, et al. "Rt-1: Robotics transformer for real-world control at scale.".

[3] Judah, Kshitij, et al. "Active lmitation learning: formal and practical reductions to IID learning.".

[4] Hawke, Jeffrey, et al. "Urban driving with conditional imitation learning.".

[5] Antotsiou, Dafni, Carlo Ciliberto, and Tae-Kyun Kim. "Adversarial imitation learning with trajectorial augmentation and correction.".

Meta-review: The authors propose a method to understand the qualities of a *good* dataset for imitation learning, for the purposes of data curation. They find interesting insights regarding the transition diversity and action divergence being good metrics to measure dataset quality. They pair a more formal description of these phenomena with a variety of empirical analyses on imitation learning data in simulated robotics. The resulting paper provides a set of actionable insights for practitioners of imitation learning. 

During the review period, there were 2 very uninformative reviews, which we discarded during the discussion period. All the other reviewers find the insights described quite useful and the formalism introduced will be useful for future imitation learning practitioners. The reviewers brought up valuable points of clarification to be added to the paper, please do incorporate them for a camera-ready version.
Decision: Accept (poster)