[1] Y. Tay, M. Dehghani, S. Abnar, H. W. Chung, W. Fedus, J. Rao, S. Narang, V. Q. Tran, D. Yogatama, and D. Metzler, "Scaling laws vs model architectures: How does inductive bias influence scaling?" 2022. [Online]. Available: https://arxiv.org/abs/2207.105511
[2] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, "Scaling vision transformers," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12 104-12 113. 1
[3] L. scale Artificial Intelligence Open Network (LAION), “LAION 5B Dataset,” https://laion.ai/ laion-5b-a-new-era-of-open-large-scale-\multi-modal-datasets/, retrieved 29 april 2022. 1
[4] T. Kurth, S. Treichler, J. Romero, M. Mudigonda, N. Luehr, E. Phillips, A. Mahesh, M. Matheson, J. Deslippe, M. Fatica et al., "Exascale deep learning for climate analytics," in SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2018, pp. 649-660. 1, 2, 7, 15
[5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020. 1
[6] H. Kataoka, R. Hayamizu, R. Yamada, K. Nakashima, S. Takashima, X. Zhang, E. J. Martinez-Noriega, N. Inoue, and R. Yokota, "Replacing labeled real-image datasets with auto-generated contours," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 21 232-21 241. 1, 15
[7] O. Sharir, B. Peleg, and Y. Shoham, “The cost of training nlp models: A concise overview,” arXiv preprint arXiv:2004.08900, 2020. 1
[8] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, "Green AI," Commun. ACM, vol. 63, no. 12, p. 54-63, nov 2020. [Online]. Available: https://doi.org/10.1145/3381831 1
[9] T. T. Nguyen, F. Trahay, J. Domke, A. Drozd, E. Vatai, J. Liao, M. Wahib, and B. Gerofi, “Why globally re-shuffle? revisiting data shuffling in large scale deep learning,” in IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2022, pp. 1085–1096. 1
[10] N. Dryden, R. Böhringer, T. Ben-Nun, and T. Hoefler, “Clairvoyant prefetching for distributed machine learning I/O,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021, pp. 1–15. 1
[11] A. Katharopoulos and F. Fleuret, “Not all samples are created equal: Deep learning with importance sampling,” in International conference on machine learning. PMLR, 2018, pp. 2525–2534. 1, 2, 4, 5, 7, 8, 9, 21, 22
[12] S. Mindermann, J. M. Brauner, M. T. Razzak, M. Sharma, A. Kirsch, W. Xu, B. Höltgen, A. N. Gomez, A. Morisot, S. Farquhar, and Y. Gal, “Prioritized training on points that are learnable, worth learning, and not yet learnt,” in Proceedings of the 39th International Conference on Machine Learning. PMLR, Jun 2022, p. 15630–15649. 1, 2, 3, 21, 22
[13] M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. J. Gordon, "An empirical study of example forgetting during deep neural network learning," in 7th International Conference on Learning Representations (ICLR), 2019. [Online]. Available: https://openreview.net/forum?id=BJlxm30cKm 2, 3, 5, 7, 8, 17, 22, 23
[14] V. Feldman and C. Zhang, "What neural networks memorize and why: Discovering the long tail via influence estimation," in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS'20, 2020. 2, 3, 5, 22, 23