SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Abubakar Abid, Maheen Farooqi, and James Zou. Large language models associate muslims with violence. Nature Machine Intelligence, 3(6):461-463, 2021.
[2] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198, 2000.
[3] Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning, pages 1732-1748. PMLR, 2022.
[4] Suneel Belkhale and Dorsa Sadigh. Plato: Predicting latent affordances through object-centric play. In 6th Annual Conference on Robot Learning, 2022.
[5] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. ACM Conference on Fairness, Accountability, and Transparency, 2023.
[6] Erdem Biryik, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv:1910.04365, 2019.
[7] Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning, pages 330-359. PMLR, 2020.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] Zhangjie Cao, Zihan Wang, and Dorsa Sadigh. Learning from imperfect demonstrations via adversarial confidence transfer. In 2022 International Conference on Robotics and Automation (ICRA), pages 441-447. IEEE, 2022.
[10] Ting-Yun Chang and Robin Jia. Careful data curation stabilizes in-context learning, 2022.
[11] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning, pages 1262-1277. PMLR, 2021.
[12] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[13] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[14] Corinna Cichy and Stefan Rass. An overview of data quality frameworks. IEEE Access, 7: 24634-24648, 2019.
[15] Yuchen Cui and Scott Niekum. Active reward learning from critiques. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6907-6914. IEEE, 2018.
[50]Gokul Swamy, Sanjiban Choudhury,Drew Bagnell,and Steven Wu. Sequence model imitation learning with unobserved contexts. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=3nbKUphLBg5.
[51] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
[52] Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/9781108627771.
[53] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[54] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. 6th Annual Conference on Robot Learning (CoRL), 2022.
A Theoretical Results
Theorem 4.1: Given a policy  $\pi_A$  and demonstrator  $\pi_E$  and environment horizon length  $H$ , the distribution shift:
$$
D _ {\mathrm {K L}} \left(\rho_ {\pi_ {A}}, \rho_ {\pi_ {E}}\right) \leq \frac {1}{H} \sum_ {t = 0} ^ {H - 1} (H - t) \mathbb {E} _ {s \sim \rho_ {\pi_ {A}} ^ {t}} \left[ D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) \right]
$$
Proof. Using the log-sum inequality:
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(\rho_ {\pi_ {A}} ^ {t}, \rho_ {\pi_ {E}} ^ {t}\right) = \int_ {s ^ {\prime}} \rho_ {\pi_ {A}} ^ {t} \left(s ^ {\prime}\right) \log \frac {\rho_ {\pi_ {A}} ^ {t} \left(s ^ {\prime}\right)}{\rho_ {\pi_ {E}} ^ {t} \left(s ^ {\prime}\right)} \\ = \int_ {s ^ {\prime}} \left(\int_ {s, a} \rho_ {\pi_ {A}} ^ {t - 1} (s) \pi_ {A} (a | s) \rho (s ^ {\prime} | a, s)\right) \log \frac {\int_ {s , a} \rho_ {\pi_ {A}} ^ {t - 1} (s) \pi_ {A} (a | s) \rho (s ^ {\prime} | a , s)}{\int_ {s , a} \rho_ {\pi_ {E}} ^ {t - 1} (s) \pi_ {E} (a | s) \rho (s ^ {\prime} | a , s)} \\ \leq \int_ {s ^ {\prime}} \int_ {s, a} \rho_ {\pi_ {A}} ^ {t - 1} (s) \pi_ {A} (a | s) \rho (s ^ {\prime} | a, s) \log \frac {\rho_ {\pi_ {A}} ^ {t - 1} (s) \pi_ {A} (a | s) \rho (s ^ {\prime} | a , s)}{\rho_ {\pi_ {E}} ^ {t - 1} (s) \pi_ {E} (a | s) \rho (s ^ {\prime} | a , s)} \\ \leq \int_ {s ^ {\prime}} \int_ {s, a} \rho_ {\pi_ {A}} ^ {t - 1} (s) \pi_ {A} (a | s) \rho (s ^ {\prime} | a, s) (\log \frac {\rho_ {\pi_ {A}} ^ {t - 1} (s)}{\rho_ {\pi_ {E}} ^ {t - 1} (s)} + \log \frac {\pi_ {A} (a | s)}{\pi_ {E} (a | s)}) \\ \leq \int_ {s, a} \rho_ {\pi_ {A}} ^ {t - 1} (s) \pi_ {A} (a | s) \left(\log \frac {\rho_ {\pi_ {A}} ^ {t - 1} (s)}{\rho_ {\pi_ {E}} ^ {t - 1} (s)} + \log \frac {\pi_ {A} (a | s)}{\pi_ {E} (a | s)}\right) \\ \leq \int_ {s} \rho_ {\pi_ {A}} ^ {t - 1} (s) \log \frac {\rho_ {\pi_ {A}} ^ {t - 1} (s)}{\rho_ {\pi_ {E}} ^ {t - 1} (s)} + \int_ {s, a} \rho_ {\pi_ {A}} ^ {t - 1} (s) \pi_ {A} (a | s) \log \frac {\pi_ {A} (a | s)}{\pi_ {E} (a | s)} \\ \leq D _ {\mathrm {K L}} (\rho_ {\pi_ {A}} ^ {t - 1}, \rho_ {\pi_ {E}} ^ {t - 1}) + \mathbb {E} _ {s \sim \rho_ {\pi_ {A}} ^ {t}} \left[ D _ {\mathrm {K L}} (\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)) \right] \\ \leq D _ {\mathrm {K L}} \left(\rho^ {0} (\cdot), \rho^ {0} (\cdot)\right) + \sum_ {j = 0} ^ {t - 1} \mathbb {E} _ {s \sim \rho_ {\pi_ {A}} ^ {j}} \left[ D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) \right] \triangleright \text {R e c u r s i v e} \\ \leq \sum_ {j = 0} ^ {t - 1} \mathbb {E} _ {s \sim \rho_ {\pi_ {A}} ^ {j}} \left[ D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} D _ {\mathrm {K L}} (\rho_ {\pi_ {A}}, \rho_ {\pi_ {E}}) = \int_ {s} \left(\frac {1}{H} \sum_ {t = 1} ^ {H} \rho_ {\pi_ {A}} ^ {t} (s)\right) \log \frac {\sum_ {t = 1} ^ {H} \rho_ {\pi_ {A}} ^ {t} (s)}{\sum_ {t = 1} ^ {H} \rho_ {\pi_ {E}} ^ {t} (s)} \\ \leq \int_ {s} \sum_ {t = 1} ^ {H} \rho_ {\pi_ {A}} ^ {t} (s) \log \frac {\rho_ {\pi_ {A}} ^ {t} (s)}{\rho_ {\pi_ {E}} ^ {t} (s)} \\ \leq \frac {1}{H} \sum_ {t = 1} ^ {H} D _ {\mathrm {K L}} \left(\rho_ {\pi_ {A}} ^ {t}, \rho_ {\pi_ {E}} ^ {t}\right) \\ \leq \frac {1}{H} \sum_ {t = 1} ^ {H} \sum_ {j = 0} ^ {t - 1} \mathbb {E} _ {s \sim \rho_ {\pi_ {A}} ^ {j}} \left[ D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) \right] \\ \leq \frac {1}{H} \sum_ {t = 0} ^ {H - 1} (H - t) \mathbb {E} _ {s \sim \rho_ {\pi_ {A}} ^ {t}} [ D _ {\mathrm {K L}} (\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)) ] \\ \end{array}
$$
Note that Ke et al. show a similar theorem, however they make the strong assumption of the f-divergence satisfying the triangle inequality, which is not true for the KL divergence we use in Theorem 4.1, and also yields a different bound. Furthermore, the implications of the divergence relationship on data quality (i.e. the data generating policy distribution) is not examined within this prior work. They are focusing more on the algorithmic perspective, as is common in prior work.
Lemma 4.2: Given learned policy \(\pi_A\) and expert \(\pi_E\), define assume that the policy is learned such that when \(s \in \mathrm{supp}(\rho_{\pi_E}^t)\), \(D_{\mathrm{KL}}(\pi_A(\cdot | s), \pi_E(\cdot | s)) \leq \beta\). Then \(\mathbb{E}_{s \sim \rho_{\pi_A}^t}[D_{\mathrm{KL}}(\pi_A(\cdot | s), \pi_E(\cdot | s))] \leq \mathbb{E}_{s \in \rho_{\pi_A}^t}[\beta \mathbb{1}(s \in \mathrm{supp}(\rho_{\pi_E}^t)) + \mathbb{1}(s \notin \mathrm{supp}(\rho_{\pi_E}^t)) D_{\mathrm{KL}}(\pi_A(\cdot | s), \pi_E(\cdot | s))]
Proof. This follows by simple substitution:
$$
\begin{array}{l} \mathbb {E} _ {s \sim \rho_ {\pi_ {A}} ^ {t}} \left[ D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) \right] = \mathbb {E} _ {s \in \rho_ {\pi_ {A}} ^ {t}} \left[ \mathbb {1} (s \in \operatorname {s u p p} \left(\rho_ {\pi_ {E}} ^ {t}\right)) D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) \right. \\ + \mathbb {1} (s \notin \operatorname {s u p p} \left(\rho_ {\pi_ {E}} ^ {t}\right)) D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) ] \\ \leq \mathbb {E} _ {s \in \rho_ {\pi_ {A}} ^ {t}} [ \beta \mathbb {1} (s \in \operatorname {s u p p} (\rho_ {\pi_ {E}} ^ {t})) \\ + \mathbb {1} (s \notin \operatorname {s u p p} \left(\rho_ {\pi_ {E}} ^ {t}\right)) D _ {\mathrm {K L}} \left(\pi_ {A} (\cdot | s), \pi_ {E} (\cdot | s)\right) ] \\ \end{array}
$$
Theorem 4.3: Given a policy  $\pi_A$  and demonstrator  $\pi_E$ , assume that for state  $s$ , if  $\rho_{\pi_E}^{t-1}(s) > 0$ , then  $\pi_A(a|s) = \pi_E(a|s)$ . Assume that transitions are normally distributed with fixed and diagonal variance,  $\rho(s'|s,a) = \mathcal{N}(\mu(s,a), \sigma^2 I)$ , then the next state coverage probability is  $P_S(s; N,\epsilon) \geq 1 - (1 - (\frac{c\epsilon}{\sigma})^d \exp(-\alpha^2 d))^N - \exp(-(\alpha - 1)^2 d)$ , where  $d$  is the dimensionality of the state,  $c$  is a constant, and  $\alpha \geq 1$  is a parameter chosen to maximize the bound.
Proof. The proof follows by first discretizing the state space into length  $\epsilon$  bins. Denote the probability mass of bin  $b$  as  $p_b$ . First we note that for two independent samples from the same distribution  $s'$  and  $s_{i,*}^{\prime}$ , we can say that  $P(|s' - s_{i,*}'|_{\infty} \geq \epsilon)$  is upper bounded by the probability that neither sample lands in the same bin. This is because of the fact that if  $|s' - s_{i,*}'|_{\infty} \geq \epsilon$ , then that implies the samples did not land in the same bin (if they did the infinity norm would be less than epsilon). Thus, we can also say that  $P(\min_i |s' - s_{i,*}'|_{\infty} \geq \epsilon)$  is upper bounded by the probability that all of the samples  $s_{i,*}'$  land in a different bin than  $s'$ . Thus:
$$
\begin{array}{l} P \left(\min  _ {i} | s ^ {\prime} - s _ {i, *} ^ {\prime} | _ {\infty} \geq \epsilon\right) \leq P (\text {n o n e l a n d s i n s a m e b i n}) \\ \leq \sum_ {b} p _ {b} (1 - p _ {b}) ^ {N} \\ \end{array}
$$
Next define a ball of radius  $R$  around the mean  $\mu(s, a)$  as  $\operatorname{Ball}(R)$ . Using this ball we can partition the above inequality into two terms to yield an upper bound.
$$
\begin{array}{l} P \left(\min  _ {i} \left| s ^ {\prime} - s _ {i, *} ^ {\prime} \right| _ {\infty} \geq \epsilon\right) \leq \sum_ {b} p _ {b} (1 - p _ {b}) ^ {N} \\ \leq \sum_ {b \in \operatorname {B a l l} (R)} p _ {b} (1 - p _ {b}) ^ {N} + \sum_ {b \notin \operatorname {B a l l} (R)} p _ {b} (1 - p _ {b}) ^ {N} \\ \end{array}
$$
We can upper bound everything inside the ball using the minimum Gaussian mass for the given radius  $R$ , or  $p_b \geq \left(\frac{c\epsilon}{\sigma}\right)^d \exp \frac{-R^2}{2\sigma^2}$  where  $c = \frac{1}{\sqrt{2\pi}}$ . For the second term, if  $R$  is large enough, then we can assume  $p_b$  is sufficiently small such that  $(1 - p_b)^N \approx 1$ .
$$
\begin{array}{l} P \left(\min  _ {i} \left| s ^ {\prime} - s _ {i, *} ^ {\prime} \right| _ {\infty} \geq \epsilon\right) \leq \sum_ {b \in \operatorname {B a l l} (R)} p _ {b} \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp \frac {- R ^ {2}}{2 \sigma^ {2}}\right) ^ {N} + \sum_ {b \notin \operatorname {B a l l} (R)} p _ {b} \\ \leq \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp \left(\frac {- R ^ {2}}{2 \sigma^ {2}}\right)\right) ^ {N} \left(\sum_ {b \in \operatorname {B a l l} (R)} p _ {b}\right) + \sum_ {b \notin \operatorname {B a l l} (R)} p _ {b} \\ \leq \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp \left(\frac {- R ^ {2}}{2 \sigma^ {2}}\right)\right) ^ {N} + \sum_ {b \not \in \mathrm {B a l l} (R)} p _ {b} \\ \end{array}
$$
The second term is upper bounded by the probability that the  $d$ -dim Gaussian  $s$  is farther in euclidean distance than  $R$  from the mean, which can be written using the tail probability of the  $\chi^2$  distribution.
$$
\begin{array}{l} P (\min _ {i} | s ^ {\prime} - s _ {i, *} ^ {\prime} | _ {\infty} \geq \epsilon) \leq \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp \left(\frac {- R ^ {2}}{2 \sigma^ {2}}\right)\right) ^ {N} + P (| | s - \mu (s, a) | | _ {2} ^ {2} \geq R ^ {2})) \\ \leq \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp \left(\frac {- R ^ {2}}{2 \sigma^ {2}}\right)\right) ^ {N} + \left(1 - \chi^ {2} \left(\frac {R ^ {2}}{\sigma^ {2}}\right)\right) \\ \end{array}
$$
We know that for a  $\chi^2$  random variable  $\mathbf{Y}$  over  $d$  sub Gaussians,  $P\left(\frac{Y}{d} \geq (1 + \delta)^2\right) \leq \exp \left(\frac{-d\delta^2}{2}\right)$  for  $\delta \geq 0$  [52]. Thus assuming  $R \geq \sigma \sqrt{d}$ , we can write:
$$
P (\min _ {i} | s ^ {\prime} - s _ {i, *} ^ {\prime} | _ {\infty} \geq \epsilon) \leq \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp \frac {- R ^ {2}}{2 \sigma}\right) ^ {N} + \exp \left(- \left(\frac {R}{\sigma} - \sqrt {d}\right) ^ {2}\right)
$$
Now, for any  $\alpha = \frac{R}{\sigma\sqrt{d}}\geq 1$  , we can rewrite the above bound as:
$$
P (\min _ {i} | s ^ {\prime} - s _ {i, *} ^ {\prime} | _ {\infty} \geq \epsilon) \leq \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp (- \alpha^ {2} d)\right) ^ {N} + \exp (- (\alpha - 1) ^ {2} d)
$$
Starting from Definition 1:
$$
\begin{array}{l} P _ {S} (s; N, \epsilon) = P \left(\min  _ {i} \| s ^ {\prime} - s _ {*, i} ^ {\prime} \| _ {\infty} \leq \epsilon\right) \\ = 1 - P (\min _ {i} | s ^ {\prime} - s _ {i, *} ^ {\prime} | _ {\infty} \geq \epsilon) \\ \geq 1 - \left(1 - \left(\frac {c \epsilon}{\sigma}\right) ^ {d} \exp (- \alpha^ {2} d)\right) ^ {N} - \exp (- (\alpha - 1) ^ {2} d) \\ \end{array}
$$
A.1 Generalization under System Noise
Definition 2. Given a policy  $\pi_A$ , a data generating policy  $\pi_E$  and a starting state  $s$ , define the probability of next state coverage  $P_B(s, \mu; N) = 1 - \cap_i P(||s_i' - \mu||^2 \geq ||s_{i,*}' - \mu||^2)$ , where  $s_{i,*}' \sim \rho_{\pi_E}^t(\cdot|s)$  are next state samples from the expert starting at  $s$ , and  $s_i' \sim \rho_{\pi_A}^t(\cdot|s)$  are next state samples from  $\pi_A$  starting at  $s$ .
Intuitively  $P_B(s; N)$  is the probability that given  $N$  chances, the hyper-sphere defined by the L2 distance from  $\mu$  to a sampled data point contains a sample from the learned policy. Here the hypersphere represents the set of next states that the policy can generalize to. The  $N$  chances approximate the effect of having more data samples to leverage for generalization. For policies learned with neural networks, this aims to represent the "interpolation" capacity of these models among the training samples.
Theorem A.1. Given a policy  $\pi_A$  and deterministic demonstrator  $\pi_E$ , assume that for state  $s$ , if  $\rho_{\pi_E}^{t-1}(s) > 0$ , then  $\pi_A(a|s) = \mathcal{N}(\pi_E(s), \sigma_\pi^2 I)$ . Assume that transitions are normally distributed with fixed and diagonal variance,  $\rho(s'|s, a) = \mathcal{N}(\mu(s, a), \sigma^2 I)$ , where  $\mu(s, a) = s + \alpha a$  are simplified linear dynamics for scalar  $\alpha \in \mathbb{R}$ . Then the next state coverage probability is  $P_B(s, \mu(s, a); N) = 1 - \left(1 - F_d\left(\frac{1}{1 + \alpha^2\left(\frac{\sigma_p}{\sigma}\right)^2}\right)\right)^N$ , where  $F_d(x)$  is the CDF of the  $f$ -distribution of dimension  $d$ .
Proof. First we note that both  $\Delta_{i} = s_{i}^{\prime} - \mu (s,a)$  and  $\Delta_{i,*} = s_{i,*}^{\prime} - \mu (s,a)$  are zero mean, and the events for  $\Delta_i\geq \Delta_{i,*}$  for all  $i$  are independent. Thus:
$$
\begin{array}{l} P _ {B} (s, \mu ; N) = 1 - \cap_ {i} P (\left\| \Delta_ {i} \right\| ^ {2} \geq \left\| \Delta_ {i, *} \right\| ^ {2}) \\ = 1 - \left(1 - P \left(\left\| \Delta_ {i} \right\| ^ {2} \leq \left\| \Delta_ {i, *} \right\| ^ {2}\right)\right) ^ {N} \\ = 1 - \left(1 - P \left(\frac {\left| \left| \Delta_ {i} \right| \right| ^ {2}}{\left| \left| \Delta_ {i , *} \right| \right| ^ {2}} \leq 1\right)\right) ^ {N} \\ \end{array}
$$
Note that  $||\Delta_i||^2$  is just a  $\chi_d^2$  random variable with  $d$  degrees of freedom, and likewise for  $||\Delta_{i,*}||^2$ . The former has variance  $\sigma^2 + (\alpha \sigma_p)^2$  (transition plus learned policy noise), while the latter has variance  $\sigma^2$  (just transition noise). The f-distribution is defined for two  $\chi_d^2$  variables  $X$  and  $Y$  of dimension  $d$  as the distribution of  $Z = \frac{X / d}{Y / d}$ . Thus with a change of variables to  $X$  and  $Y$ , and denoting  $F_{d}$  as the CDF of the f-distribution of dimension  $d$ :
$$
\begin{array}{l} P \left(\frac {\left\| \Delta_ {i} \right\| ^ {2}}{\left\| \Delta_ {i , *} \right\| ^ {2}} \leq 1\right) = P \left(\frac {\left(\sigma^ {2} + (\alpha \sigma_ {p}) ^ {2}\right) X}{\sigma^ {2} Y} \leq 1\right) \\ = P \left(\frac {X}{Y} \leq \frac {\sigma^ {2}}{\sigma^ {2} + (\alpha \sigma_ {p}) ^ {2}}\right) \\ = F _ {d} \left(\frac {\sigma^ {2}}{\sigma^ {2} + (\alpha \sigma_ {p}) ^ {2}}\right) \\ = F _ {d} \left(\frac {1}{1 + \alpha^ {2} \left(\frac {\sigma_ {p}}{\sigma}\right) ^ {2}}\right) \\ \end{array}
$$
Plugging the above expression into the expression for  $P_B$ :
$$
P _ {B} (s, \mu ; N) = 1 - \left(1 - F _ {d} \left(\frac {1}{1 + \alpha^ {2} \left(\frac {\sigma_ {p}}{\sigma}\right) ^ {2}}\right)\right) ^ {N}
$$
In Theorem A.1, while the probability of next state coverage is not immediately interpretable, we can still intuitively recognize that high  $\sigma_{s}$  can improve the coverage likelihood of the learned policy even under notable  $\sigma_{p}$ , and as  $N$  gets bigger even less  $\sigma_{s}$  is needed, despite the fact that system noise is present even under the learned policy. Intuitively, we once again see that increasing  $N$  has a significant effect on the coverage likelihood. In terms of noise, what matters is the ratio of policy to system noise, where increasing this ratio leads to sharp drops in performance at some cutoff based on  $N$ . We visualize this coverage probability in Fig. 4 under increasing ratios of policy to system noise for different values of  $N$ .
B Metrics of Data Quality
Having formalized action divergence and transition diversity in Sec. 4 as two fundamental considerations in a dataset, how can we measure these properties in a given dataset?
Action Variance: To measure action consistency, the empirical form of the objective in Eqn. 7 is intractable without access to the underlying expert action distribution  $\pi_E$ . Instead we propose using the empirical variance of the action distribution in the data to approximate the "spread" of the data. In continuous state spaces, we can estimate variance using a coverage distance  $\epsilon$  to cluster nearby states, and then measuring the per dimension variance across the corresponding actions within said cluster. Defining a cluster to be  $C(s,\mathcal{D}) = \{\tilde{s},\tilde{a},\tilde{s}^{\prime}\in \mathcal{D}:||s - \tilde{s} ||\leq \epsilon \}$ , we can compute the variance as:
$$
\operatorname {A c t i o n V a r i a n c e} (\mathcal {D}) = \frac {1}{| D |} \sum_ {s, a \in \mathcal {D}} \left(a - \sum_ {\tilde {s}, \tilde {a}, \tilde {s} ^ {\prime} \in C (s, \mathcal {D})} \tilde {a}\right) ^ {2} \tag {9}
$$
Figure 4:  $P_B(s, \mu; N)$  (y-axis) from Theorem A.1 plotted for four-dimensional state under  $N \in [2, 10, 50, 100]$ , but varying the ratio of policy to system noise (x-axis is  $\frac{\sigma_p}{\sigma}$ ). We see that under this more loose coverage model, with lots of samples, adding system noise can make coverage likely even under double or triple the noise in the learned policy.
The choice in  $\epsilon$  corresponds to the generalization of the learning model to nearby states, similar to the notion of coverage in Definition 1. We use this metric of action consistency in Sec. 5 to study human generated datasets of various quality.
State Similarity: To measure the consistency of states, we approximate the number of "nearby" states using the same clustering process as in the Action Variance metric, and measure the expected cluster size as a fraction of the overall data.
$$
\operatorname {S t a t e S i m i l a r i t y} (\mathcal {D}) = \frac {1}{| D |} \sum_ {s, a \in \mathcal {D}} | C (s, \mathcal {D}) | \tag {10}
$$
While these approximate forms do not encapsulate the full spectrum of possible metrics, we believe these metrics help advance our empirical understanding of data quality for imitation learning. In section 5.2 in the main text, we analyze these metrics of data quality in several environments across different dataset sources.
The performance results under system noise, policy noise, and both noises are shown with a broader sweep for both PMObstacle and Square in the tables below.
σs=0.01	σs=0.02	σs=0.03	σs=0.04	σs=0.01	σs=0.02	σs=0.03	σs=0.04
SCRIPTED	100	100	99	96
σs=0.01	97.7(1.5)	95.7(0.7)	96.7(1.1)	93.3(0.7)	90.3(7.1)	90.0(8.2)	94.0(2.9)	87.7(3.7)
σs=0.02	98.7(0.5)	98.0(0.5)	97.7(1.0)	92.7(1.2)	99.7(0.3)	98.0(0.9)	94.3(1.4)	92.3(2.0)
σs=0.03	98.3(0.7)	98.0(0.8)	99.0(0.5)	95.0(0.9)	99.7(0.3)	98.7(0.5)	97.7(0.5)	95.7(1.1)
σs=0.04	100.0(0.0)	100.0(0.0)	99.3(0.3)	96.7(0.7)	100.0(0.0)	99.0(0.5)	98.7(0.5)	96.7(1.4)
Table 2: System Noise: Success rates (and standard error) for BC in PMObstacle, for 1000 episodes (left) and 10 episodes (right) of data, under system noise. Rows correspond to injecting gaussian system noise  $(\sigma_s)$  into the dataset of increasing variance, and columns correspond to injecting noise during evaluation. The diagonal in both sub-tables represents evaluating in distribution. Left: For large datasets, higher system noise during evaluation tends to decrease the performance of each model (rows left to right), but more system noise during training generally produces the best models (columns top to bottom). Right: For small datasets, we observe a similar but exaggerated effect as the left table.
σs=0.01	σs=0.02	σs=0.03	σs=0.04	σs=0.01	σs=0.02	σs=0.03	σs=0.04
SCRIPTED	100	100	99	96
σp=0.01	94.0(1.7)	94.0(2.4)	94.7(1.8)	91.3(1.4)	78.0(8.6)	78.7(6.7)	81.3(5.0)	81.3(5.8)
σp=0.02	87.7(2.0)	92.3(2.0)	90.7(2.0)	91.3(2.2)	88.0(9.4)	78.7(4.4)	80.7(3.1)	80.3(3.2)
σp=0.03	97.0(0.9)	99.0(0.5)	97.0(0.0)	95.0(0.8)	88.7(3.2)	82.7(5.4)	88.7(5.2)	85.7(4.4)
σp=0.04	86.7(4.3)	91.0(2.4)	93.3(1.4)	92.7(1.5)	88.3(6.5)	88.0(4.5)	86.0(5.9)	82.3(2.0)