[30] Thomas Paine, Hailin Jin, Jianchao Yang, Zhe Lin, and Thomas Huang. GPU asynchronous stochastic gradient descent to speed up neural network training. arXiv preprint arXiv:1312.6186, 2013. https://arxiv.org/pdf/1312.6186.pdf.
[31] Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Capuchin: Tensor-basedgpu memory management for deep learning. In Proceedings of ASPLOS, pages 891-905, 2020. http://alchem.usc.edu/portal/static/download/capuchin.pdf.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of ICML, pages 8748-8763, 2021. https://arxiv.org/pdf/2103.00020.pdf.
[33] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI blog, 2018. https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf.
[34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervisedMULTITASK_LEARNers.pdf.
[35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. http://jmlr.org/papers/v21/20-074.html.
[36] Samyam Rajbhandari, Jeff Rasley, Olatunjri Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of SC, pages 1-16, 2020. https://arxiv.org/pdf/1910.02054.pdf.
[37] Samyam Rajbhandari, Olatunj Ruwase, Jeff Rasley, Shadeen Smith, and Yuxiong He. Zero-infinity: Breaking thegpu memory wall for extreme scale deep learning. In Proceedings of SC, pages 1-14, 2021. https://arxiv.org/pdf/2104.07857.pdf.
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. https://arxiv.org/pdf/2204.06125.pdf.
[39] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of ICML, pages 8821-8831, 2021. https://arxiv.org/pdf/2102.12092.pdf.
[40] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of KDD, pages 3505-3506, 2020. https://dl.acm.org/doi/10.1145/3394486.3406703.
[41] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunj Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training. In Proceedings of ATC, pages 551-564, 2021. https://arxiv.org/pdf/2101.06840v1.pdf.
[42] Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design. In Proceedings of MICRO, pages 1-13, 2016. https://arxiv.org/pdf/1602.08124.pdf.
[43] Harvey M Salkin and Cornelis A De Kluyver. The knapsack problem: A survey. Naval Research Logistics Quarterly, 22(1):127-144, 1975. https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800220110.
[44] SB Shriram, Anshuj Garg, and Purushottam Kulkarni. Dynamic memory management forgpu-based training of deep neural networks. In Proceedings of IPDPS, pages 200-209, 2019. https://ieeexplore.ieee.org/abstract/document/8820980.