[22] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2383–2392. DOI: 10.18653/v1/D16-1264. [Online]. Available: https://aclanthology.org/D16-1264.
[23] R. Ri and Y. Tsuruoka, “Pretraining with artificial language: Studying transferable knowledge in language models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 7302-7315. DOI: 10.18653/v1/2022.acl-long.504. [Online]. Available: https://aclanthology.org/2022.acl-long.504.
[24] O. Russakovsky et al., "Imagenet large scale visual recognition challenge," International journal of computer vision, vol. 115, pp. 211-252, 2015.
[25] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019.
[26] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929-1958, 2014. [Online]. Available: http://jmlr.org/papers/v15/srivastava14a.html.
[27] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
[28] Y. Tay et al., "Scale efficiently: Insights from pre-training and fine-tuning transformers," arXiv preprint arXiv:2109.10686, 2021.
[29] Y. Tay et al., "Unifying language learning paradigms," arXiv preprint arXiv:2205.05131, 2022.
[30] R. Taylor et al., "Galactica: A large language model for science," arXiv preprint arXiv:2211.09085, 2022.
[31] H. Touvron et al., "Llama: Open and efficient foundation language models," arXiv preprint arXiv:2302.13971, 2023.
[32] P. Villalobos, J. Sevilla, L. Heim, T. Besiroglu, M. Hobbhahn, and A. Ho, "Will we run out of data? an analysis of the limits of scaling datasets in machine learning," arXiv preprint arXiv:2211.04325, 2022.
[33] Y. Wu, F. Li, and P. S. Liang, "Insights into pre-training via simpler synthetic tasks," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35, Curran Associates, Inc., 2022, pp. 21844-21857. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/89379d5fc6eb34ff98488202fb52b9d0-Paper-Conference.pdf.
[34] F. Xue, X. He, X. Ren, Y. Lou, and Y. You, "One student knows all experts know: From sparse to dense," arXiv preprint arXiv:2201.10890, 2022.
[35] F. Xue, Z. Shi, F. Wei, Y. Lou, Y. Liu, and Y. You, "Go wider instead of deeper," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8779-8787.
[36] L. Xue et al., "Byt5: Towards a token-free future with pre-trained byte-to-byte models," Transactions of the Association for Computational Linguistics, vol. 10, pp. 291-306, 2022.
[37] L. Xue et al., "Mt5: A massively multilingual pre-trained text-to-text transformer," arXiv preprint arXiv:2010.11934, 2020.
[38] B. Zoph et al., "St-moe: Designing stable and transferable sparse expert models," URL https://arxiv.org/abs/2202.08906, 2022.