[30] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. "The pile: An 800gb dataset of diverse text for language modeling". In: arXiv preprint arXiv:2101.00027 (2020).
[31] Utkarsh Sharma and Jared Kaplan. "Scaling Laws from the Data Manifold Dimension". In: Journal of Machine Learning Research 23.9 (2022), pp. 1-34. URL: http://jmlr.org/papers/v23/20-1111.html.
[32] Eric J. Michaud, Ziming Liu, and Max Tegmark. "Precision Machine Learning". In: Entropy 25.1 (2023). ISSN: 1099-4300. DOI: 10.3390/e25010175. URL: https://www.mdpi.com/1099-4300/25/1/175.
[33] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. "Explaining neural scaling laws". In: arXiv preprint arXiv:2102.06701 (2021).
[34] Alexander Maloney, Daniel A Roberts, and James Sully. "A Solvable Model of Neural Scaling Laws". In: arXiv preprint arXiv:2210.16859 (2022).
[35] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. "Spectrum dependent learning curves in kernel regression and wide neural networks". In: International Conference on Machine Learning. PMLR. 2020, pp. 1024-1034.
[36] Marcus Hutter. "Learning curve theory". In: arXiv preprint arXiv:2102.04074 (2021).
[37] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. "Beyond the imitation game: Quantifying and extrapolating the capabilities of language models". In: arXiv preprint arXiv:2206.04615 (2022).
[38] Ryan Schaeffer, Brando Miranda, and Sanmi Koyejo. "Are Emergent Abilities of Large Language Models a Mirage?" In: arXiv preprint arXiv:2304.15004 (2023).
[39] Sanjeev Arora and Anirudh Goyal. “A theory for emergence of complex skills in language models”. In: arXiv preprint arXiv:2307.15936 (2023).
[40] Lorenzo Saitta, Attilio Giordana, and Antoine Cornuejols. Phase transitions in machine learning. Cambridge University Press, 2011.
[41] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. "On the stepwise nature of self-supervised learning". In: arXiv preprint arXiv:2303.15438 (2023).
[42] Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. "Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models". In: arXiv preprint arXiv:2307.14430 (2023).
[43] Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. "Data distributional properties drive emergent in-context learning in transformers". In: Advances in Neural Information Processing Systems 35 (2022), pp. 18878-18891.
[44] Marvin Minsky. Society of mind. Simon and Schuster, 1988.
[45] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaepau, M. Brucher, M. Perrot, and E. Duchesnay. "Scikit-learn: Machine Learning in Python". In: Journal of Machine Learning Research 12 (2011), pp. 2825–2830.
[46] Ronen Eldan and Yuanzhi Li. "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" In: arXiv preprint arXiv:2305.07759 (2023).
[47] Pablo Villalobos. Scaling Laws Literature Review. https://epochai.org/blog/scaling-laws-literature-review. Accessed: 2023-01-31. 2023.
[48] Jasha Droppo and Oguz Elibol. "Scaling laws for acoustic models". In: arXiv preprint arXiv:2106.09488 (2021).
[49] Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. "Understanding Scaling Laws for Recommendation Models". In: arXiv preprint arXiv:2208.08489 (2022).