Full Parameter Fine-tuning for Large Language Models with Limited Resources (2024)

Abstract: Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) but demand massive GPU resources for training. Lowering the threshold for LLMs training would encourage greater participation from researchers, benefiting both academia and society. While existing approaches have focused on parameter-efficient fine-tuning, which tunes or adds a small number of parameters, few have addressed the challenge of tuning the full parameters of LLMs with limited resources. In this work, we propose a new optimizer, LOwMemory Optimization (LOMO), which fuses the gradient computation and the parameter update in one step to reduce memory usage. By integrating LOMO with existing memory saving techniques, we reduce memory usage to 10.8% compared to the standard approach (DeepSpeed solution). Consequently, our approach enables the full parameter fine-tuning of a 65B model on a single machine with 8×RTX 3090, each with 24GB memory.1

heading: 1 Introduction
text: Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP), demonstrating remarkable abilities such as emergence and grokking (Wei et al., 2022), pushing model size to become larger and larger. However, training these models with billions of parameters, such as those with 30B to 175B parameters, raises the bar for NLP research. Tuning LLMs often requires expensive GPU resources, such as 8×80GB devices, making it difficult for small labs and companies to participate in this area of research.
Recently, parameter-efficient fine-tuning methods (Ding et al., 2022), such as LoRA (Hu et al., 2022) and Prefix-tuning (Li and Liang, 2021), provide solutions for tuning LLMs with limited
*Corresponding author. 1Code and data are available at https://github.com/
OpenLMLab/LOMO.
resources. However, these methods do not offer a practical solution for full parameter finetuning, which has been acknowledged as a more powerful approach than parameter-efficient finetuning (Ding et al., 2022; Sun et al., 2023). In this work, we aim to explore techniques for accomplishing full parameter fine-tuning in resource-limited scenarios.
We analyze the four aspects of memory usage in LLMs, namely activation, optimizer states, gradient tensor and parameters, and optimize the training process in three folds: 1) We rethink the functionality of an optimizer from an algorithmic perspective and find that SGD is a good replacement in terms of fine-tuning full parameter for LLMs. This allows us to remove the entire part of optimizer states since SGD does not store any intermediate state (Sec-3.1). 2) Our proposed optimizer, LOMO as illustrated in Figure 1, reduces the memory usage of gradient tensors to O(1), equivalent to the largest gradient tensor’s memory usage (Sec-3.2). 3) To stabilize mix-precision training with LOMO, we integrate gradient normalization, loss scaling, and transition certain computations to full precision during training (Sec-3.3).
Our technique results in memory usage that equals the usage of parameters plus activation and the largest gradient tensor. We push the memory usage of full parameter fine-tuning to an extreme, making it merely equivalent to the usage of inference. This is because the memory usage of the forward + backward process should not be less than the forward process alone. It is worth noting that, when employing LOMO to save memory, we ensure that the fine-tuning process remains uncompromised, as the parameter update process is still equivalent to SGD.
We empirically assess the memory and throughput performance of LOMO and show that the usage of LOMO enables successful training of a 65B model with only 8 RTX 3090 GPUs. Additionally,
8187
to validate the downstream performance of our proposed technique, we apply LOMO to tune the full parameters of LLMs on the SuperGLUE dataset collection (Wang et al., 2019). The empirical results demonstrate the efficiency and effectiveness of LOMO for optimizing LLMs with billions of parameters. Overall, our contributions are as follows:
• We provide a theoretical analysis suggesting that SGD can successfully fine-tune the full parameters of LLMs. The issues that previously hindered the widespread usage of SGD may no longer be severe problems for finetuning LLMs.
• We propose LOw-Memory Optimization, named LOMO, to significantly save GPU memory usage without harming the finetuning process.
• Through a thorough evaluation of memory usage and throughput performance, we empirically validate the effectiveness of LOMO in optimizing LLMs under resource-constrained scenarios. This is further supported by performance evaluations on downstream tasks.

heading: 2 Related Work
text: In this section, we present related work on memorysaving techniques during full parameter fine-tuning.
These techniques can be effectively combined with LOMO to further reduce memory consumption.
Activation Checkpointing During vanilla backpropagation, all activations from the forward pass are stored in memory to compute gradients. This can be a significant memory overhead, especially for large language models. Alternatively, one could discard all activations and recompute them on demand for gradients computation in order to save memory. However, this can result in a substantial additional computation cost. Activation checkpointing (or gradient checkpointing) takes into account both memory usage and computational cost, providing a compromise solution (Chen et al., 2016). The activations of strategically selected checkpoint nodes in the computational graph are kept in memory after the forward pass, while the activations of remaining nodes are recomputed at most once. The activation memory can be reduced to the square root of the original amount at the cost of one extra forward pass.
Mixed-Precision Training Mixed-precision training has become a prevalent approach for training large language models due to its ability to accelerate training speed and reduce memory footprint (Narayanan et al., 2021; Rajbhandari et al., 2020). By employing half-precision storage for parameters, activations, and gradients,
mixed-precision training enables high-throughput computations. In order to uphold stability and model accuracy, Micikevicius et al. (2018) proposed three techniques which involve the use of full precision copies of weights, loss scaling, and the execution of specific arithmetic operations in full precision.
Heterogeneous Training System Multiple studies (Rhu et al., 2016; Wang et al., 2018; Ren et al., 2021a) have attempted to reduce GPU memory consumption by leveraging heterogeneous memory, such as CPU and NVMe memory. L2L (Pudipeddi et al., 2020) employs a layer-to-layer strategy, where only the tensors necessary for the computation of a particular layer are transferred to the GPU memory, while the remaining tensors are retained in the CPU memory. ZeRO-Offload (Ren et al., 2021b), an extension of ZeRO-2 (Rajbhandari et al., 2020), reserves the gradients and optimizer states in the CPU memory and updates parameters through CPU computation. ZeRO-Infinity (Rajbhandari et al., 2021), a subsequent advancement of ZeROOffload on ZeRO-3 (Rajbhandari et al., 2020), enables further scaling of the model size.
In addition to the methods orthogonal to LOMO mentioned above, recent developments have introduced several memory-efficient optimization techniques. MeZO (Malladi et al., 2023) employs a zero-order optimization approach, estimating gradients using two forward passes and updating parameters in place. GaLore (Zhao et al., 2024) performs low-rank decomposition on gradients and uses these approximated gradients for parameter updates. Other approaches reduce memory usage by quantizing optimizer states (Dettmers et al., 2022; Sun et al., 2020b). Compared to these methods, LOMO neither approximates gradients nor requires low-bit quantization.

heading: 3 Method

heading: 3.1 Rethink the Functionality of Optimizer
text: The optimizer states occupy a large part of the memory used for training LLMs. Modern optimizer like Adam (Kingma and Ba, 2015) stores intermediate states that are twice the size of parameters. As the size of parameters increases, the optimizer states become the dominant term of memory usage.

heading: 3.1.1 Using SGD
text: Although Adam has achieved great success in training deep models, we ask the question “Can we use
a cheaper optimizer for fine-tuning LLMs?" Our answer is SGD, the most basic optimizer. Fortunately, we find that it is an acceptable solution for fine-tuning LLMs when we limit the scope.
Prior works often discuss three challenges of SGD: 1) large curvature loss surface, 2) local optimum, and 3) saddle points (Ruder, 2016; Sun et al., 2020a). Modern optimizers have shown effectiveness in dealing with the 1) problem and can mitigate 2) and 3) in some cases. However, when we limit the scope to fine-tuning LLMs, these three challenges could be different.
Smoother loss surface One important assumption is that the parameter space of LLMs is quite smooth and small perturbations on the parameters will not change the loss too much. There are empirical results and theoretical analyses supporting this assumption (Hao et al., 2019). If we believe that larger models have a smoother loss surface, we can conclude that the 1) problem is not an issue since the loss surface of LLMs should not have a large curvature. Note that this holds only when we teach the LLMs natural language-based tasks (or code-based if pre-trained with code). A synthetic loss function unrelated to pre-training tasks will indeed face the large curvature problem.
Local optimum is good enough The goal of fine-tuning is to adapt LLMs to new tasks and domains without significantly changing the model itself. Therefore, a local optimum is often a good enough solution (Kawaguchi et al., 2019), and the limited training data (compared to pre-training corpus) makes it difficult to push the model to a faraway global optimum.
Distant saddle points Similarly, for a common NLP task, the initial point of LLMs should be in a valley. If the model is pre-trained with instructions (tasks), the phenomenon could be much more apparent since we have more chances of finding pre-trained tasks that are similar to the new task. Saddle points typically appear on ridges and have a distance from valleys, so we may not encounter the saddle point problem if we do not change the parameter too far from the pre-trained value.
However, there is no guarantee that SGD is a powerful optimizer compared to modern optimizers. Our intuition is to create a simple and practical solution for fine-tuning LLMs and identify its flaws to continually improve it.

heading: 3.1.2 Implicit Batch Size
text: Besides the above qualitative discussion, we want to provide a deeper analysis of the stability of finetuning LLMs with SGD. Suppose we have a pretrained model f(·) with the parameter θ, a training set D = {d1, d2, · · · , dn}, and a loss function L. One step update of SGD on a batch with two data points could be,
θ′ = θ − α[∇L(di, f(di,θ)) +∇L(dj , f(dj ,θ))], (1)
where α is the learning rate, and di, dj are two different training samples.
Next, two steps update of SGD on these two training samples di, dj sequentially could be,
θ1 = θ − α∇L(di, f(di,θ)), (2) θ2 = θ1 − α∇L(dj , f(dj ,θ1)). (3)
By differential mean value theorem, we have
L(dj , f(dj ,θ1)) = L(dj , f(dj ,θ)) +∇L(dj , ξ)(f(dj ,θ1)− f(dj ,θ)), (4) θ2 = θ − α∇L(di, f(di,θ)) − α∇L(dj , f(dj ,θ)) − α∇[∇L(dj , ξ)(f(dj ,θ1)− f(dj ,θ))], (5) θ2 = θ − α[∇L(di, f(di,θ)) +∇L(dj , f(dj ,θ))] − α∇[∇L(dj , ξ)(f(dj ,θ1)− f(dj ,θ))], (6)
where ξ is a point between f(dj ,θ) and f(dj ,θ1), and we can see that (6) minus (1) equals the α∇[∇L(dj , ξ)(f(dj ,θ1) − f(dj ,θ))]. Suppose the loss surface is smooth enough, this term become negligible. It suggests that utilizing SGD optimizer over a smooth loss surface could imply a larger batch size.
As we mentioned above, it’s reasonable to assume that the loss surface of LLMs is smooth, and a larger batch size indicates stronger training stability, so we believe that finetuning process of LLMs with the SGD optimizer is stable. This also explains why SGD failed on small models but worked for large models.

heading: 3.2 LOMO: LOw-Memory Optimization
text: The gradient tensor represents the gradient of a parameter tensor and has the same size as the parameter, resulting in a large memory overhead. Modern deep learning training frameworks like
Algorithm 1 Fusion Update in LOMO Require: model f(·) with L layers and p parame-
ters, parameter θ ∈ Rp , learning rate α, max step T , training dataset D, loss function L
1: for t = 1, . . . , T do 2: Sample batch B = (x,y) ⊂ D 3: ŷ ← f(x,θ) ▷ Forward pass 4: ℓ← L(y, ŷ) 5: for l = L, . . . , 1 do ▷ Backward 6: θl ← [θi for θi ∈ layer l] 7: gl ← ∂ℓ∂θl 8: θl ← θl − α ∗ gl 9: gl ← None ▷ Clear gradients
10: end for 11: end for
PyTorch (Paszke et al., 2017) store gradient tensors for all parameters. There are two reasons for storing gradient tensors: computing optimizer states and normalizing gradients.
Since we take SGD as the optimizer, there are no optimizer states relying on gradients, and we have some alternatives to gradient normalization. Thus, we proposed LOw-Memory Optimization (LOMO) as illustrated in Algorithm 1, fusing the gradient computation and parameter update in one step to avoid storing any gradient tensors.
In detail, we can express the vanilla gradient descent as grad = ∂L∂p , p = p−lr∗grad, which is a two-step process, computing the gradients first and updating it to the parameters. The fusion version is p = p− lr ∗ ∂L∂p .
The key idea is to update the parameter immediately when its gradient is computed so that we do not store gradient tensor in memory. This can be achieved by injecting hook functions into the backward propagation.2 PyTorch provides relevant APIs for injecting hook functions, but we cannot implement the exact immediate update with current APIs. Instead, we store at most one parameter’s gradient in memory and update each parameter one by one along with the backward propagation. Our approach reduces the memory usage of gradients from storing of all parameters’ gradients to storing only one parameter’s gradient.
The majority of LOMO memory usage coincides with that of parameter-efficient fine-tuning (PEFT) methods, indicating that combining LOMO with
2We should inject different hook functions accordingly if some of them share the weight.
these methods only introduces a minor increase in memory occupied by gradients. This enables tuning much more parameters for PEFT methods.

heading: 3.3 Stabilize Training with LOMO

heading: 3.3.1 Alternatives to Gradient Normalization and Clipping
text: Gradient normalization and clipping are essential tools to deal with the gradient explosion and vanishing problem (Chen et al., 2018), but their computation process requires using the gradient tensors of all parameters. We propose two alternatives here:
• Clipping gradient tensors by its values rather than the norm.
• Compute the gradient norm in an additional backward pass.
Clipping gradient tensors by their values is a simple but effective solution for gradient explosion before gradient norm approaches. The main concern of clipping by values is that truncating some gradient elements could change the direction of the gradient tensor. For example, a two-dim vector [1.3, 0.8] and its clipped version [1.0, 0.8] (clipped to 1.0) indicate different directions. Our experience is that the clipping by values performs worse when the learning rate is high because truncations happened more often in that case. However, clipping by values performs well for medium and small learning rates. Note that the scale of the learning rate largely depends on the task and data, but in general, we suggest using clipping by values for a learning rate less than 1e− 3.
Our approach cannot directly compute the gradient norm because we update parameters along with the backpropagation, so we do not know the norm of rest parameters when updating a certain parameter. However, we can introduce an additional pass to compute and accumulate each parameter’s gradient norm, resulting in two backward passes, one for computing the gradient norm and one for updating parameters. The memory usage leaves unchanged but sacrifices the speed.
A controversial solution Our current training framework computes the gradient norm based on all parameters and requires two backward passes. One solution to save the additional backward pass is to approximate the norm of gradient tensors with a group of parameters, for example, the adjacent layers. This method is indeed biased, because it
results in different update step sizes for different parameters. When updating, the parameters are multiplied by a scale factor according to the gradient norms. Since the gradient norms differ among parameter groups, such an approximation leads to a difference in scale factors. This grouped gradient clipping method can be considered as applying a dynamic learning rate to different groups of parameters based on their gradient norms. Sun et al. (2020a) suggests that it is not always appropriate to use the same learning rate for all parameters in SGD, thus we believe our approach also holds the potential to further benefit SGD. We leave the exploration as a compelling future direction.

heading: 3.3.2 Mitigating Precision Degradation
text: Mixed-precision training is commonly employed to speed up the training process. To mitigate the degradation in precision, we utilize dynamic loss scaling and transition certain computations to full precision. The approach of loss scaling is crucial in preventing underflows during FP16 training, magnifying the loss with a specific factor prior to the backward pass and diminishing the gradients by the same factor.
In this context, we integrate a dynamic loss scaler with LOMO, which dynamically adjusts the scaling factor throughout the training procedure. If no overflow occurs during a specified number of backward passes, the scale factor is doubled. Otherwise, this step is dropped and the scale factor is halved. This process echoes the scenario encountered during gradient normalization. It is unknown whether there will be an overflow until the backward has completed. Consequently, we perform two backward passes: the first pass to identify any overflow, and the second pass to update the parameters if no overflow is detected. These two backward passes for dynamic loss scaling can be executed simultaneously with gradient normalization. To effectively update parameter and handle gradient for operations like normalization and scaling, the gradient and its associated parameter are converted to full precision within these computations.

heading: 4 Experiment
text: In this section, we evaluate our proposed method from three aspects, namely memory profile, throughput and downstream performance. If not further explained, all our experiments are conducted with LLaMA models (Touvron et al., 2023), ranging from 7B to 65B.

heading: 4.1 Memory Profile
text: We first profile the memory usage of model states and activations during the training under different settings. As demonstrated in Table 1, the usage of the LOMO optimizer leads to a substantial reduction in memory footprint from 102.20GB to 14.58GB, when compared to the AdamW optimizer (Loshchilov and Hutter, 2019), and from 51.99GB to 14.58GB, when compared to SGD, in the context of training the LLaMA-7B model. This significant decrease in memory usage can be attributed primarily to the reduced memory requirements of the gradient and optimizer states. As a result, memory is mostly occupied by parameters in the training process, commensurate with memory usage during inference.
Optimizer States Figure 2 illustrates that employing the AdamW optimizer for LLaMA-7B training, a widely adopted configuration, yields a substantial proportion of memory (73.7%) being allocated to optimizer states. This outcome is a consequence of the mixed-precision training approach, where full-precision copies of weights, momentum, and variance are maintained within the optimizer states for weight updates. Replacing the AdamW
optimizer with the SGD optimizer can effectively reduce the percentage of optimizer states in memory, and therefore alleviate the GPU memory usage (from 102.20GB to 51.99GB). This reduction is due to the fact that the SGD optimizer does not require the storage of full-precision momentums and variances. For LOMO, parameter update and backward are fused into one step, further eliminating the need for optimizer state memory.
Gradients During the training process using LOMO, parameters are immediately updated upon receiving gradients, following which the gradients are discarded from memory. As a result, the upper bound of gradient memory consumption is determined by the gradient associated with the parameter matrix of greatest magnitude. This approach considerably reduces memory usage by almost the size of parameters.
Activations The training of a 7B model with 512×8 tokens in one batch demands a substantial amount of memory for activations. LOMO is compatible with activation memory reduction techniques such as activation checkpointing. By integrating activation checkpointing with LOMO, the memory footprint due to activation can be reduced
from 45.61GB to 1.79GB.

heading: 4.2 Throughput
text: We evaluate the throughput performance of LOMO compared to AdamW and SGD. The experiments are conduct on a server equipped with 8 RTX 3090 GPUs, interconnected via a PCIe motherboard. The sequence length and batch size are set to 1024 and 1, respectively. Throughput is measured in terms of the number of tokens processed per GPU per second (TGS), and parameter partitioning was achieved using ZeRO-3 (Rajbhandari et al., 2020).
For the 7B model, LOMO demonstrates remarkable throughput, surpassing AdamW and SGD by about 11 times. This significant improvement can be attributed to LOMO’s ability to train the 7B model on a single GPU, thereby reducing interGPU communication overhead. The slightly higher throughput of SGD compared to AdamW can be attributed to SGD’s exclusion of momentum and variance calculations.
As for the 13B model, it could not be trained with AdamW on the available 8 RTX 3090 GPUs due to memory limitations. In this scenario where model parallelism is necessary for LOMO, LOMO still outperforms SGD in terms of throughput. This advantage is attributed to LOMO’s memoryefficient properties and the requirement of only two GPUs to train the model with the same settings, resulting in reduced communication costs and greater throughput. Furthermore, when training the 30B model, SGD encounters out-of-memory (OOM) issues with the 8 RTX 3090 GPUs, while LOMO performs well with only 4 GPUs.
Finally, we successfully train the 65B model using 8 RTX 3090 GPUs, achieving a throughput of 4.93 TGS. Utilizing such a server configuration
and LOMO, the training process on 1000 samples, each containing 512 tokens, requires approximately 3.6 hours.

heading: 4.3 Downstream Performance
text: To assess the effectiveness of LOMO in fine-tuning large language models, we conduct an extensive set of experiments. We compare LOMO against two other methods, Zero-shot, which does not require fine-tuning, and LoRA, which is currently one of the most popular parameter-efficient finetuning techniques. As descirbed in (Hu et al., 2022), LoRA reparameterizes the dense layers and only updates low rank matrices while introducing no latency during inference.
We use the SuperGLUE dataset collection to evaluate model performance, specifically focusing on RTE (Dagan et al., 2005), BoolQ (Clark et al., 2019), WSC (Levesque et al., 2012), WIC (Pilehvar and Camacho-Collados, 2019), MultiRC (Khashabi et al., 2018), and COPA (Roemmele et al., 2011). Given the high computational cost associated with running large language models, we follow MeZO (Malladi et al., 2023) to randomly sample 1000 training data from training set and 1000 test data from validation set, and report the best results obtained using the same random seed. The prompts used in our experiments are the same as MeZO, and the hyperparameters are detailed in Appendix-A.
During inference, we insert different labels or candidates into the prompt and calculate the average log-likelihood for each label. The label with the highest score is selected as the model’s answer. To evaluate the performance, we use Accuracy as the evaluation metric.

heading: 4.3.1 Main results
text: The downstream performances of LOMO compared with Zero-shot and LoRA are presented in Table 3. Based on the results, we reach the following observations.
LOMO performs significantly better than Zero-shot. Across all six datasets and model sizes, LOMO consistently achieves superior results over Zero-shot, with average gains of more than 20 points using LLaMA-13B. While previous research has showcased the impressive capabilities of large language models in zero-shot settings, fine-tuning still yields remarkable performance enhancements for specific downstream tasks. The experimental results confirm the effectiveness of LOMO in optimizing large language models of different sizes.
LOMO generally outperforms LoRA in most experiments. We show that LOMO delivers strong performance compared to LoRA, for instance, resulting in average gains of 2.8 points using LLaMA13B. This suggests that the model performance benefits more from full-parameter fine-tuning than parameter-efficient fine-tuning, as the former adjusts more parameters. LOMO strikes a good balance between performance and efficiency, making it a competitive choice for fine-tuning.
In some cases, LOMO performs worse than LoRA. One possible reason is the relatively small training set we use, which may not be sufficient for full-parameter fine-tuning of large models. Additionally, LoRA and LOMO employ different model architectures. To be specific, LoRA offers a shortcut for model tuning, which can be advantageous in
certain scenarios. Actually, these two methods are not conflicting or mutually exclusive. In the next subsection, we validate that combing LoRA with LOMO does not harm model performance and, in most cases, leads to performance gains.
LOMO efficiently scales up to 65 billion parameter models. Despite conducting all experiments on a single machine equipped with 8 × RTX 3090, LOMO consistently exhibits strong performance even on a 65-parameter scale. This further supports the effectiveness of LOMO in optimizing LLMs under resource-constrained scenarios.

heading: 4.3.2 LoRA with LOMO
text: LOMO and LoRA are fundamentally independent of each other. In order to verify this claim, we perform experiments using LLaMA-13B on the BoolQ and MultiRC datasets. Results are shown in Figure 3. We find that LOMO consistently enhances the performance of LoRA regardless of the higher results LoRA achieved. This suggests that different fine-tuning methods employed by LOMO and LoRA are complementary. Specifically, LOMO focuses on fine-tuning the pre-trained models weights, while LoRA tunes additional modules. As a result, LOMO dose not compromise the performance of LoRA; rather, it facilitates better model tuning for downstream tasks.

heading: 5 Conclusion
text: In this paper, we introduce LOw-Memory Optimization (LOMO), a new optimizer designed to facilitate full parameter fine-tuning for large lan-
guage models with limited resources. We have demonstrated the feasibility of fine-tuning a 65B model on a server equipped with consumer GPUs such as RTX 3090. By analyzing the memory usage of LOMO, conducting throughput tests, and performing experiments on SuperGLUE, we have showcased its effectiveness and potential impact.
Looking ahead, our future work aims to further lower the resource threshold required for training large language models, thus enabling wider access and adoption of these models. The majority of memory are currently occupied by parameters when training with LOMO. Thus, one promising direction is the exploration of parameter quantization techniques, which could significantly reduce memory usage. Additionally, we intend to investigate more applicable scenarios for LOMO and delve into theoretical analyses for optimizing large language models, which hold substantial value for advancing the field.
Limitations
In response to the challenges associated with gradient normalization and clipping, we have developed alternative optimization methods. Although gradient normalization for LOMO does not increase memory usage, our current implementation necessitates an additional backward pass, which can slow down the training speed in scenarios where gradient normalization is essential.
Due to time and resource constraints, our experiments were limited to a subset of the SuperGLUE benchmark, and we did not evaluate LOMO’s throughput on advanced GPUs such as A100.
Ethics statement
This paper employs open-source models LLaMA, in compliance with their respective licenses. The datasets utilized, including RTE, BoolQ, WSC, WIC, MultiRC and COPA, permit public and freeusage.

heading: Acknowledgments
text: We would like to thank Qinghui Gao for the kind help on memory profile. This work was supported by the National Key Research and Development Program of China (No.2022ZD0160102). The computations in this research were performed using the CFFF platform of Fudan University.

heading: A Hyperparameters
text: The hyperparameters we use in the experiments are listed in Table 4. Due to limited computational resources, we report the highest results of experiments conducted with the same random seed.

heading: B Training Dynamics
text: To analyze the training dynamics of LOMO, we present the training loss curve and validation accuracy for LLaMA-7B trained on BoolQ (Clark et al., 2019) using LOMO and LoRA in Figure 4 and Figure 5, respectively. During training process with LOMO, the loss converges rapidly in the initial phase and then tends to stabilize and gradually decline. The accuracy on the development set generally shows an upward trend as the number of training steps increases.
0 100 200 300 400 500 600 Step
0
1
2
3
4
5
6
Tr ai
ni ng
L os
s
LoRA LOMO
Figure 4: Training loss curve for LLaMA-7B on BoolQ. 100 200 300 400 500 600 Step
0.78
0.80
0.82
0.84
0.86
0.88
Va lid
at io
n Ac
cu ra
cy
LoRA LOMO
Figure 5: Validation accuracy of LLaMA-7B on BoolQ.

heading: Future Work
text: Looking ahead, our future work aims to further lower the resource threshold required for training large language models, thus enabling wider access and adoption of these models. The majority of memory are currently occupied by parameters when training with LOMO. Thus, one promising direction is the exploration of parameter quantization techniques, which could significantly reduce memory usage. Additionally, we intend to investigate more applicable scenarios for LOMO and delve into theoretical analyses for optimizing large language models, which hold substantial value for advancing the field. Limitations
In response to the challenges associated with gradient normalization and clipping, we have developed alternative optimization methods. Although gradient normalization for LOMO does not increase memory usage, our current implementation necessitates an additional backward pass, which can slow down the training speed in scenarios where gradient normalization is essential. Due to time and resource constraints, our experiments were limited to a subset of the SuperGLUE benchmark, and we did not evaluate LOMO’s throughput on advanced GPUs such as A100