Title: AccLoRT: Efficient Large Language Models Pretraining through Low-Rank Accumulation
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Low-rank Training, Memory Efficient Training

Abstract: Pretraining large language models (LLMs) poses significant computational challenges, particularly due to the memory requirements that exceed the capabilities of standard GPU devices. To address these issues, we introduce a fully low-rank approach for LLMs pretraining to improve the memory efficiency. Specifically, our approach sequentially trains low-rank matrices and accumulates them into a frozen high-rank matrix until convergence. Notably, our approach enables the low-rank traning without a warm up phase with full parameter, therefore achieving memory efficiency in the entire training process. We provide a comprehensive theoretical analysis for our proposed method by establishing the upper and lower bounds for the rank of multiple matrix sums and analyzing the rank dynamics in low-rank adapters. The results show that with finite accumulation steps, the accumulated low-rank training is equivalent to full-rank training. Extensive experiments on both synthetic reduced rank regression and practical Llama models (60M to 1B parameters) validate the effectiveness of the proposed approach in pretraining, demonstrating its potential to make LLM development more accessible and efficient.