[15] Zhongzhe Hu, Junmin Xiao, Zheye Deng, Mingyi Li, Kewei Zhang, Xiaoyang Zhang, Ke Meng, Ninghui Sun, and Guangming Tan. Megtaichi: Dynamic tensor-based memory management optimization for dnn training. In Proceedings of ICS, pages 1-13, 2022. https://dl.acm.org/doi/pdf/10.1145/3524059.3532394.
[16] Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapadvisor: Pushing deep learning beyond thegpu memory limit via smart swapping. In Proceedings of ASPLOS, pages 1341-1355, 2020. https://dl.acm.org/doi/pdf/10.1145/3373376.3378530.
[17] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan First, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Proceedings of NeurIPS, volume 32, 2019. https://proceedings.neurips.cc/paper/2019/file/093f65e080a295f8076b1c5722a46aa2-Paper.pdf.
[18] Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Checkmate: Breaking the memory wall with optimal tensor rematerialization. In Proceedings of MLSys, volume 2, pages 497-511, 2020. https://proceedings.mlsys.org/paper/2020/file/084b6fbbb10729ed4da8c3d3f5a3ae7c9-Paper.pdf.
[19] Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization. In Proceedings of ICLR, 2021. https://arxiv.org/pdf/2006.09616.pdf.
[20] Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, and Masanori Koyama. A graph theoretic framework of recomputation algorithms for memory-efficient backpropagation. In Proceedings of NeurIPS, volume 32, 2019. https://arxiv.org/pdf/1905.11722.pdf.
[21] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
[22] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of CVPR, pages 12009-12019, 2022. https://arxiv.org/pdf/2111.09883.pdf.
[23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of ICCV, pages 9992-10002, 2021. https://arxiv.org/pdf/2103.14030.pdf.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of ICLR, 2019. https://arxiv.org/pdf/1711.05101.pdf.
[25] George B Mathews. On the partition of numbers. Proceedings of the London Mathematical Society, 1(1):486-490, 1896. https://academic.oup.com/plms/article-abstract/s1-28/1/486/1495829.
[26] Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism. arXiv preprint arXiv:2211.13878, 2022. https://arxiv.org/pdf/2211.13878.pdf.
[27] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of SOSP, pages 1-15, 2019. https://dl.acm.org/doi/pdf/10.1145/3341301.3359646.
[28] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training ongpu clusters using megatron-lm. In Proceedings of SC, pages 1-15, 2021. https://arxiv.org/pdf/2104.04473.pdf.
[29] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. https://arxiv.org/pdf/2303.08774.pdf.