[33] He, H., Huang, G., and Yuan, Y. Asymmetric valleys: Beyond sharp and flat local minima. Advances in neural information processing systems, 32, 2019.
[34] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
[35] Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 646-661. Springer, 2016.
[36] Irandoust, S., Durand, T., Rakhmangulova, Y., Zi, W., and Hajimirsadeghi, H. Training a Vision Transformer from scratch in less than 24 hours with 1 GPU, November 2022. URL http://arxiv.org/abs/2211.05187.arXiv:2211.05187 [cs].
[37] Izsak, P., Berchansky, M., and Levy, O. How to train bert with an academic budget. arXiv preprint arXiv:2104.07705, 2021.
[38] Izsak, P., Berchansky, M., and Levy, O. How to Train BERT with an Academic Budget, September 2021. URL http://arxiv.org/abs/2104.07705.arXiv:2104.07705 [cs].
[39] Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., and Pillai, P. Accelerating Deep Learning by Focusing on the Biggest Losers, October 2019. URL http://arxiv.org/abs/1910.00762.arXiv:1910.00762 [cs, stat].
[40] Kaddour, J. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209.14981, 2022.
[41] Kaddour, J. The minipile challenge for data-efficient language models. arXiv preprint arXiv:2304.08442, 2023.
[42] Kaddour, J., Liu, L., Silva, R., and Kusner, M. J. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577-16595, 2022.
[43] Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., and McHardy, R. Challenges and Applications of Large Language Models, July 2023. URL http://arxiv.org/abs/2307.10169.
[44] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[45] Katharopoulos, A. and Fleuret, F. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534. PMLR, 2018.
[46] Kawaguchi, K. and Lu, H. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669-679. PMLR, 2020.
[47] Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337, 2021.
[48] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[49] Kleinberg, B., Li, Y., and Yuan, Y. An alternative view: When does sgd escape local minima? In International conference on machine learning, pp. 2698-2707. PMLR, 2018.
[50] Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big Transfer (BiT): General Visual Representation Learning, May 2020. URL http://arxiv.org/abs/1912.11370. arXiv:1912.11370 [cs].