SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, empirically evaluate the effect on policy performance of our proposal to inject noise back into the dynamics at inference time. We consider three challenging robotic manipulation tasks studied in prior work: PushT block-pushing [19]; Robomimic Can Pick-and-Place and Square Nut Assembly [42] (we direct the reader to Chi et al. [19] for an extensive empirical investigation into the performance of diffusion policies in the un-noised  $\sigma = 0$  regime). We display the results of our experiments in Figure 6. Observe that the performance degradation of the replica policy from the unnoised  $\sigma = 0$  variant is minimal across all environments and even leads to a slight but noticeable improvement in the small-noise regime for PushT (and low-data Can Pick and Place). In the presence of non-negligible noise HINT significantly outperforms the conventional policy  $\hat{\pi}$  (obtained by noisy observations at training but not test time), as predicted by our theory.
This work considerably loosened assumptions placed on the expert distribution by introducing a synthesis oracle responsible for stabilization. How best to achieve low-level stabilization remains an open question. We hope that this work encourages further empirical research into improving the stability of imitation learning, either via the hierarchical route proposed in this paper or via new innovations.
$^{3}$ Code for PushT, Robomimic experiments can be found at https://github.com/pfrommerd/diffusion_policy.pt. Quadrotor experiments are in https://github.com/pfrommerd/stanza/
We would like to thank Cheng Chi for his extensive help in running the code in [19] and Benjamin Burchfiel for numerous insightful discussions and intuitions about DDPM policies. Lastly, we thank Alexandre Amice for his helpful feedback on an earlier draft of this manuscript. AB acknowledges support from the National Science Foundation Graduate Research Fellowship under Grant No. 1122374 as well as the support of DOE through award DE-SC0022199. MS and RT acknowledge support from Amazon.com Services LLC grant; PO# 2D-06310236. DP and AJ acknowledge support from the Office of Naval Research under ONR grant N00014-23-1-2299.
[1] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016.
[2] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
[3] A. Ajay, A. Gupta, D. Ghosh, S. Levine, and P. Agrawal. Distributionally adaptive meta reinforcement learning. arXiv preprint arXiv:2210.03104, 2022.
[4] A. Ajay, S. Han, Y. Du, S. Li, A. Gupta, T. Jaakkola, J. Tenenbaum, L. Kaelbling, A. Srivastava, and P. Agrawal. Compositional foundation models for hierarchical planning. arXiv preprint arXiv:2309.08587, 2023.
[5] J. Altschuler and K. Talwar. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. Advances in Neural Information Processing Systems, 35:3788-3800, 2022.
[6] J. M. Altschuler and S. Chewi. Faster high-accuracy log-concave sampling via algorithmic warm starts. arXiv preprint arXiv:2302.10249, 2023.
[7] B. D. Anderson and J. B. Moore. Optimal control: linear quadratic methods. Courier Corporation, 2007.
[8] O. Angel and Y. Spinka. Pairwise optimal coupling of multiple random variables. arXiv preprint arXiv:1903.00632, 2019.
[9] D. Angeli. A lyapunov approach to incremental stability properties. IEEE Transactions on Automatic Control, 47(3):410-421, 2002.
[10] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079, 2018.
[11] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
[12] P. L. Bartlett, A. Montanari, and A. Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30:87-201, 2021.
[13] A. Block, Y. Mroueh, and A. Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
[14] A. Block, Y. Mroueh, A. Rakhlin, and J. Ross. Fast mixing of multi-scale Langevin dynamics under the manifold hypothesis. arXiv preprint arXiv:2006.11166, 2020.
[15] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016.
[16] J. D. Chang, M. Uehara, D. Sreenivas, R. Kidambi, and W. Sun. Mitigating covariate shift in imitation learning via offline data without great coverage. arXiv preprint arXiv:2106.03207, 2021.
[17] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.