[1] R. Anil et al., “Palm 2 technical report,” arXiv preprint arXiv:2305.10403, 2023.
[2] S. Biderman et al., Pythia: A suite for analyzing large language models across training and scaling, 2023. arXiv: 2304.01373 [cs.CL].
[3] A. Chowdhery et al., “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311, 2022.
[4] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser, “Universal transformers,” arXiv preprint arXiv:1807.03819, 2018.
[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
[6] A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020.
[7] D. Hernandez et al., "Scaling laws and interpretability of learning from repeated data," arXiv preprint arXiv:2205.10487, 2022.
[8] J. Hoffmann et al., “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556, 2022.
[9] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, “Deep networks with stochastic depth,” in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, Springer, 2016, pp. 646–661.
[10] J. Kaplan et al., "Scaling laws for neural language models," arXiv preprint arXiv:2001.08361, 2020.
[11] Y. J. Kim et al., "Scalable and efficient moe training for multitask multilingual models," arXiv preprint arXiv:2109.10465, 2021.
[12] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "Albert: A lite bert for self-supervised learning of language representations," arXiv preprint arXiv:1909.11942, 2019.
[13] D. Lepikhin et al., "Gshard: Scaling giant models with conditional computation and automatic sharding," arXiv preprint arXiv:2006.16668, 2020.
[14] J. Li, Z. Zhang, and H. Zhao, "Self-prompting large language models for open-domain qa," arXiv preprint arXiv:2212.08635, 2022.
[15] Q. Liu et al., "Tapex: Table pre-training via learning a neural sql executor," arXiv preprint arXiv:2107.07653, 2021.
[16] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," arXiv preprint arXiv:1711.05101, 2017.
[17] Y. Lou, F. Xue, Z. Zheng, and Y. You, "Cross-token modeling with conditional computation," arXiv preprint arXiv:2109.02008, 2021.
[18] OpenAI, “Gpt-4 technical report,” ArXiv, vol. abs/2303.08774, 2023.
[19] L. Ouyang et al., "Training language models to follow instructions with human feedback," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35, Curran Associates, Inc., 2022, pp. 27 730-27 744. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
[20] J. W. Rae et al., "Scaling language models: Methods, analysis & insights from training gopher," arXiv preprint arXiv:2112.11446, 2021.
[21] C. Raffel et al., "Exploring the limits of transfer learning with a unified text-to-text transformer," Journal of Machine Learning Research, vol. 21, no. 140, pp. 1-67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html.