[15] M. Paul, S. Ganguli, and G. K. Dziugaite, “Deep learning on a data diet: Finding important examples early in training,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, 2021, pp. 20596-20607. [Online]. Available: https://proceedings.neurips.cc/paper/2021/file/ac56f8fe9eea3e4a365f29f0f1957c55-Paper.pdf 2, 3, 22
[16] K. Chitta, J. M. Álvarez, E. Haussmann, and C. Farabet, “Training data subset search with ensemble active learning,” IEEE Transactions on Intelligent Transportation Systems, 2021. 2, 3, 22
[17] A. H. Jiang, D. L. K. Wong, G. Zhou, D. G. Andersen, J. Dean, G. R. Ganger, G. Joshi, M. Kaminsky, M. Kozuch, Z. C. Lipton, and P. Pillai, "Accelerating deep learning by focusing on the biggest losers," 2019. [Online]. Available: https://arxiv.org/abs/1910.00762 2, 3, 7, 8, 9, 23
[18] K. Killamsetty, S. Durga, G. Ramakrishnan, A. De, and R. Iyer, "Grad-match: Gradient matching based data subset selection for efficient deep model training," in International Conference on Machine Learning. PMLR, 2021, pp. 5464-5474. 2, 3, 7, 8, 23
[19] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Conf. on comp. vision and pattern recognition. IEEE, 2009, pp. 248-255. 2, 7, 15
[20] K. Meding, L. M. S. Buschoff, R. Geirhos, and F. A. Wichmann, “Trivial or impossible-dichotomous data difficulty masks model differences (on ImageNet and beyond),” arXiv preprint arXiv:2110.05922, 2021. 2
[21] K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, A. De, and R. K. Iyer, “GRAD-MATCH: gradient matching based data subset selection for efficient deep model training,” in ICML, ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, pp. 5464–5474. 2
[22] X. Zeng, M. Yan, and M. Zhang, "Mercury: Efficient On-Device Distributed DNN Training via Stochastic Importance Sampling," in Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, 2021, pp. 29-41. 3, 5, 7, 22
[23] C.-Y. Wu, R. Manmatha, A. J. Smola, and P. Krahenbuhl, "Sampling matters in deep embedding learning," in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2840-2848. 3, 22
[24] P. Zhao and T. Zhang, "Stochastic optimization with importance sampling for regularized loss minimization," in international conference on machine learning. PMLR, 2015, pp. 1-9. 3, 22
[25] Z. Allen-Zhu, Z. Qu, P. Richtárik, and Y. Yuan, "Even faster accelerated coordinate descent using non-uniform sampling," in International Conference on Machine Learning. PMLR, 2016, pp. 1110-1119. 3, 22
[26] T. B. Johnson and C. Guestrin, “Training deep models faster with robust, approximate importance sampling,” Advances in Neural Information Processing Systems, vol. 31, 2018. 3, 22
[27] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos, “Beyond neural scaling laws: beating power law scaling via data pruning,” 2022. [Online]. Available: https://arxiv.org/abs/2206.14486 3, 22, 23
[28] Z. Qin, K. Wang, Z. Zheng, J. Gu, X. Peng, D. Zhou, and Y. You, "Infobatch: Lossless training speed up by unbiased dynamic data pruning," ArXiv, vol. abs/2303.04947, 2023. 3, 20
[29] O. Pooladzandi, D. Davini, and B. Mirzasoleiman, "Adaptive second order coresets for data-efficient machine learning," in International Conference on Machine Learning. PMLR, 2022, pp. 17-848-17-869. 3, 23
[30] Q. Meng, W. Chen, Y. Wang, Z. Ma, and T. Liu, "Convergence analysis of distributed stochastic gradient descent with shuffling," Neurocomputing, vol. 337, pp. 46-57, 2019. 5
[31] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in 5th International Conference on Learning Representations (ICLR), 2017. [Online]. Available: https://openreview.net/forum?id=Sy8gdB9xx 5
[32] E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry, “Augment your batch: Improving generalization through instance repetition,” in CVPR. Computer Vision Foundation / IEEE, 2020, pp. 8126–8135. 5, 6