Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
Authors: Jun Zhang
Jue WANG
Huan Li
Lidan Shou
Ke Chen
Yang You
Guiming Xie
Xuejian Gong
Kunlong Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Memory-Efficient LoRA
Primary area: foundation or frontier models, including LLMs
TL;DR: LoRAM is a memory-efficient LoRA training for cost-effective performance gains by training low-rank matrices on a pruned model and merging recoverd them for inference on the original model.

Abstract: Large Language Models (LLMs) have significantly advanced natural language processing with exceptional task generalization capabilities. Low-Rank Adaption (LoRA) offers a cost-effective fine-tuning solution, freezing the original model parameters and training only lightweight, low-rank adapter matrices. However, the memory footprint of LoRA is largely dominated by the original model parameters. To mitigate this, we propose LoRAM, a memory-efficient LoRA training scheme founded on the intuition that many neurons in over-parameterized LLMs have low training utility but are essential for inference. LoRAM presents a unique twist: it trains on a pruned (small) model to obtain pruned low-rank matrices, which are then recovered and utilized with the original (large) model for inference. Additionally, minimal-cost continual pre-training, performed by the model publishers in advance, aligns the knowledge discrepancy between pruned and original models. Our extensive experiments demonstrate the efficacy of LoRAM across various pruning strategies and downstream tasks. For a model with 70 billion parameters, LoRAM enables training on a GPU with only 20G HBM, replacing an A100-80G GPU for LoRA training and 15 GPUs for full fine-tuning. Specifically, QLoRAM implemented by structured pruning combined with 4-bit quantization, for LLaMA-3.1-70B (LLaMA-2-70B), reduces the parameter storage cost that dominates the memory usage in low-rank matrix training by 15.81× (16.95×), while achieving dominant performance gains over both the original LLaMA-3.1-70B (LLaMA-2-70B) and LoRA-trained LLaMA-3.1-8B (LLaMA-2-13B). Code is available at https://github.com/junzhang-zj/LoRAM.

Published as a conference paper at ICLR 2025
TRAIN SMALL, INFER LARGE: MEMORY-EFFICIENT LORA TRAINING FOR LARGE LANGUAGE MODELS
Jun Zhang! Jue Wang Huan Li Lidan Shoul.21, Ke Chen
Yang You', Guiming X Xuejian Gong", and Kunlong Zhou²
The State Key Laboratory of Blockchain and Data Security, Zhejiang University
2 "Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
Cooleee of Computer Science and Technology. Zhejiang University
4 Department of Computer Science, National University of Singapore
6 "I Center, Guangdong OPPO Mobile Telecommunications Corp., Ltd.
[z]. cs, juwang jue, 1 lihuan. es, should, chenk ( ju waw.edduccom cn,
youy@comp. nus. edu. sg, (xieguiming, gongxue zhoukunlong] @pppo com
ABSTRACT
Large Language Models (LLMs) have significantly advanced natural language processing with exceptional task generalization capabilities. Low-Rank Adaption (LoRA) offers cost-effective fine-tuning solution, freezing the original model pa- rameters and training only lightweight, low-rank adapter matrices. However, the memory footprint of LoRA largely dominated by the original model parameters. To mitigate this, propose LORAM, memory -efficceeef LoRA training scheme founded on the intuition that many neurons in over- peer-parameterized LLMs have low training utility but essential for inference. LORAM presents unique twist: it trains on pruned (small) model to obtain pruned low-rank matrices, which are then recovered and utilized with the original (large) model for inference. Ad- ditionally, minimal-cost continual pre-training, performed by the model publish- ers in advance, aligns the knowledge discrepancy between pruned and original models. Our extensive experiments demonstrate the efficacy of LORAM across various pruning strategies and downstream tasks. For a model with 70 billion parameters, LoR AM enables training on a GPU with only 20G HBM, replacing an A100-80G GPU for LoRA training and 15 GPUs for full fine-tuning. Specifi- cally, QLORAM AM implemented by structured pruning combined with 4-bit quanti- zation, for LLaMA-3.1 10B (LLaMA-2-700 2--BB,, reduces the parameter storage cost that dominates the memory usage in low-rank matrix training by 15.81 I (16.95x), while achieving dominant performance gains over both the original LLaMA-3.1 70B -LLaMA-2-708B and LoRA-trained LLaMA-3.1-88 is available at https: /tt:://gittie //gtttub.....///////// com/ hwamn/jjnnaangnne /LORAM.
2-3---1--1B) Code
INTRODUCTION
Large language models (LLMs), such as GPT-4 (OpenAI, 2023), LLaMA (Touvron et al., 2023a;b; Dubey et al., 2024), and PaLM (Chowdhery et al., 2023), have recently revolutionized natural lan- guage applications. These models excel in task generalization, driven by their exponential increase in scale, with some exceeding 400 billion parameters (Dubey et al., 2024). Fine-tuning pre-trained LLMs is critical for task-specific customization, enhancing desired behaviors while mitigating un- desired ones (Qi et al., 2024). However, this process is constrained by substantial memory re- quirements; for instance, fine-tuning 70B LLaMA in 16-bit precision demands over 1178GB' of memory, necessitating an expensive setup of 15 GPUs (A 100 or H100, each with 80GB HBM).
Work done during internship at OPPO AI Center.
1 Huan and Lidan Shou are the corresponding authors. 'Training is performed on sample with length of 4K using BF 1 mixed precision with the Adam optimizer, incorporating gradient checkpointing.

---

Published as a conference paper at ICLR 2025
To mitigate the high cost of fine-tuning LLMs, parameter--effiieet fine-tuning (Li & Liang, 2021; Lester et al., 2021; Liu et al., 2021; Qiu et al., 2023; Liu et al., 2024b; 2022), particularly Low- Rank Adaption (LoRA) (Hu et al., 2022) and its variants (Liu et al., 2024a; Ding et al., 2023; Zi et al., 2023; Zhang et al., 2023b; Kalajdzievski, 2023), freezes the original L.M weights and only updates the injected low-rank matrices to adapt to new tasks under limited resources. However, during training, they still struggle with the significant memory footprint of the parameters of the base LLM, even with quantization (Dettmers et al., 2023; Xu et al., 2024; Li et al., 2024; Guo al., 2024; Frantar et al., 2023; Chai et al., 2023). Typically, they reduce the precision to 4 bits at most due to quality considerations. This memory dilemma raises an interesting and necessary question:
Can we further reduce the memory overhead of the base model during LoRA training while still maintaining the inference accuracy?
Our answer is resounding Yes! In this paper, we propose Memory-efficient LoRA training, coined LORAM, novel scheme to reduce the memory overhead of LoRA fine-tuning for LLMs. We revisit the training and inference process of the LoRA paradigm, building on it with unique twist: Unlike typical LoRA, which uses the same original model for training and inference, LORAM employs different models at each stage, i.e., it trains prumed (small) model by updating the pruned LoRA weights and then performs inference on the original (large) model with the recovered low-rank ma- trices. This recovery process reshapes the pruned matrices ensure be merged into the original model, allowing for the updating of unpruned weights while utilizing the pruned weights during in- ference. The key insight driving our approach comes from reconciling two seemingly contradictory concepts, The scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020; Hernandez et al., 2021) suggest that large number of parameters of LLMs essential for effective model generalization. Conversely, sparsity in LLMs (Zhang et al., 2024b; Ma et al., 2023: Sun et al., 2024; Frantar & Alistarh, 2023a; Xia al., 2024) show that these pre-trained models can be compressed by remov- ing redundant weights. The goal to minimize the difference in the model's output before and after pruning. However, such methods tend to falter at higher pruning ratios and aggressively pruned models lose critical reasoning capabilities, e.g., only prun- ing 10%~~20% (Ma et al., 2023; Frantar & Alistarh, 2023a). Our intuition builds this: some critical parameters contribute signif- icantly to fine-tuning while other parameters, though essential for inference, usually remain unchanged during fine-tuning. Therefore, LORAM leverages this insight by updating the weights retained through pruning from LoRA training (yellow blocks in Fig. 1) significantly reduce memory usage and training time, while em- ploying the pruned weights (blue blocks in Fig. 1) during inference to enhance generation performance (see Section 3.5).
T: Train via X
I: Infer
T&d
T&d
T&dl
T&l
T&d
T&d
T&d
T&d
Pre-trained Weights
Figure 1: Idea of LORAM
Despite the significant reduction in memory cost achieved by the pruning-recovery process of Lo- RAM, maintaining gains at more aggressive pruning rates (e.g., 65% or higher) remains challenging. We attribute this to the knowledge inconsistency between the pruned model used for training and the original model used for inference. To address this, we propose an effective alignment strategy: low-cost continual pre-training of the pruned model on a small dataset. This alignment is performed once offline, allowing the model's 's publisher to execute it. For instance, Meta AI could release set of aligned pruned models for LLaMA-3, enabling low-resource users to fine-tune large models for customized tasks using LORAM. Notably, bonus, LORAM seamlessly integrates with exist- ing quantization schemes designed for LoRA, such as QLoRA, forming QLORAM, which further reduces memory overhead.
The contributions of this work are summarized as follows:
(1) Novel Training Scheme: We propose LORAM, a memory-efficient LoRA training scheme. LORAM trains a pruned model by updating the pruned low-rank matrices and then uses di- mensionally recovered low-rank matrices to integrate with the original model for inference. The process significantly reduces the memory consumption incurred by the model parameters during training, and synergistically boosts performance by leveraging the full original param- eters during inference. Thus, LORAM efficiently enhances performance under limited device memory resources.
2

---

Published as a conference paper at ICLR 2025
(2) Effective Alignment Strategy: We identify that the knowledge inconsistency between the pruned model used for training and the original model used for inference limits the performance gain of LORAM under aggressive pruning rates. To tackle this, train the pruned model on small amount of general corpus to achieve alignment, which a one-shot offiin process and can be easily performed by the model publisher.
(3) Extensive Experimental Evaluation: We conduct comprehensive experiments to validate the ef- fectiveness of LORAM across various pruning algorithms, models of different sizes, and tasks in different domains, Notably, QLORAM which combines LORAM with structured prun- ing and 4-bit quantization reduces the memory cost of LLaMA-2- ---BA parameters by 16.95 x while effectively achieving performance gains superior to both the original LLaMA-2-70B and L.L.MA---13B fine-tuned with LoRA.
MEMORY-ERFICIENT LORA TRAINING - LORAM
2.1 LOW-RANK ADAPTATION
Given a pre-trained weight matrix Wo E Rmxx,, a typical full-parameter fine-tuning (Li et al., 2023b) process adapts to new tasks by updating the entire full-rank matrix Wo. Inspired by the insight that pre-trained weights of LLMs exhibit low "intrinsic dimension" when adapting to specific tasks (Aghajanyan al., 2021), LoRA (Hu et al., 2022) further suggests that the updated weights have low "intrinsic rank". Consequently, LoRA reparameterizes the model weights as Wo + WA = Wo + BA, where B E R and A E RR" and WA = BA represents a low-rank decomposition matrix with the rank r < min(m, 12).
During training, as illustrated in Fig. (a), the pre-trained weight matrix Wo is frozen to avoid gradient computation. Instead, the low-rank matrices B and A are updated to enable parameter- efficient fine-tuning, which defaults to standard supervised fine-tuning, with the objective function Cser defined as the cross-entropy loss between the predicted logits and the ground-truth answers. Given an input feature vector x of length m, the forward pass of LoRA modifies the output activation from fully fine-tuning, represented by h = xWo (of length n.), to:
h xWo xWA xWo + xBA.
Once low-rank matrices B" and A are trained by minimizing the CAFT, as shown in the Fig. (c), the computation of activation h for x is reformulated to improve inference efficiency:
h x xWW W = x(Wo + B*A*).
2.2 CEMMORYEFFICIENT LORA TRAINING
Consider the LLaMA-2-13B model, we introduce low-rank matrices (r 8) for the four projection matrices (W 9. Wz. W.. and W.) in the attention layer, the three projection matrices (W up- W gaate and Wdowa) in the MLP layer, and the weight matrix Wim im head in output layer. Despite the additional 32 million parameters, the number of trainable parameters is reduced by 406x compared to the full parameters. Many LoRA variats (Zhou et al., 2024; Zhang et al., 2023a; Kopiczko et al., 2024; Azizi al., 2024; Wang et al., 2024) aim to address the significant memory overhead associated with W a as Wo scales, but they still necessitate storing a complete copy of Wo in memory, which dominates training memory usage. Even with quantization methods designed for LoRA (Dettmers et al., 2023; Xu et al., 2024; Li et al., 2024; Guo et al., 2024; Frantar et al., 2023; Chai et al., 2023), training performance constraints often limit the representation of W o 4 bits. Consequently, for the LLaMA-2-1333 2-3B, storage requirements are reduced from 26 GB in FP1 6 to 6.5GB in NF4. However, this is still significantly higher than the memory required for W A BF1 6, which occupies only 64MB of storage and has a peak memory requirement of 576MB during training. Thus, the memory needed for the frozen quantized Wo is 11.5x greater than that required for learnable WA.
To mitigate the memory overhead dominated by Wo while achieving inference performance gain, we propose a memory-effiiient LoRA training called LORAM. LORAM first prunes the model to smaller size and performs LoRA fine-tuning on the pruned model. After training. it recovers the LoRA weights, applies them to the original model, and then conducts inference. We now describe the various stages of LORAM. The complete algorithm of LoRAN A is presented in Appendix F.
3

---

Published as a conference paper at ICLR 2025
Stages W.
Online Stges of W2
B
W
W
B
Pre-triined Weights
BC-)
W
W
LoRAM Inference
C.
B
Cert
W W
A
Copy
(8)
id]
(6)
Figure 2: Comparison of LORAM and LoRA: Training (subfigures a and b) and Inference (c and d). Key stages include the offline process of the frozen full-rank matrix Wi (subfigure e) and the online generation of the learnable low-rank matrix W A (f) during LORAM training (b) and inference (d),
Pruned Full-Rank Weight Generation. First, we employ pruning algorithm P(.) to derive the pruned weight matrix W from the original weights Wp. Specifically, W computed as:
Wi = p(W = Wo o M
where Mº
{0, 1] mxn is a binary mask matrix indicating retained parameters ('1') and pruned parameters ('0'), and o denotes the Hadamard product.
Pruned Low-Rank Matrix Training. After obtaining the pruned weight matrix Wi we modify the standard LoRA training process. Instead of updating the low-rank matrices B and A for the original Wo, we train the pruned low-rank decomposition matrix W WA M B'A", while keeping W frozen as shown in Fig. 2 (b). The output activation h for an input feature vector is calculated as:
h xW + xW = xW + x(BAA))
(4)
Recovered Low-Rank Matrix Generation. By optimizing the objective function LSFT, we obtain
the trained pruned low-rank matrix W To fully leverage the original model weights for improved inference performance, we introduce a recovery function R(-), guided by the pruning mask MP. This function recovers the shape of the trained low-rank matrix by filling zeros at pruned positions, resulting in W as follows:
W = B A = R(W = W (1 1 M²).
(5)
This operation ensures that the recovered low-rank matrix W can be seamlessly merged with the original pre-trained weights Wo. forming Wo W as follows:
M [1, = Woli +o]..]] if M [i.j]] 0
Woli.
(Wo W =
(6)
This formula indicates that, for positions where the pruning mask M2 is '1', the merged matrix Wo + w retains the original values from the pre-trained matrix Wo- For positions where the mask '0,, the elements in the merged matrix are updated be the sum of the corresponding values from Wo and the recovered low-rank matrix WA.
Recovered Low-Rank Matrix Inference. Once we obtain the recovered low-rank matrix Wa' during inference, the forward pass output activation h for an input feature x is computed as follows:
h x W. + W = x( + B A²).
Our experiments (see Section 3.2 and Section 3.3) show that LORAM maintains high performance across various pruning strategies P(-), including structured (Ma et al., 2023) and non-structured pruning (semi-structured & unstructured pruning) (Frantar & Alistarh, 2023a). The four stages out- lined above summarize the core steps of LORAM. To avoid notational clutter, we have streamlined the algorithmic details. Nonetheless, three key considerations for deployment must be emphasized:
4

---

Published as a conference paper at ICLR 2025
C Pruned Full-Rank Weight Generation: For non-structured pruning, the matrix dimension remains unchanged, with W compressed into a sparse matrix populated by zeros. In structured pruning, weights are physically removed, yielding a compact, dense W
C2 Recovered Low-Rank Matrix Generation: For non-structured pruning, the weights in W corresponding pruned positions in M" are excluded from backpropagation by setting their gradients to zero, ensuring that only the retained components are updated during training.
A
C3 Recovered Low-Rank Matrix Inference:
For non-structured pruning, the shapes of both the pre-trained and low-rank weights are identical (see Ci), and the gradients of the pruned weights are blocked (sce C2). Consequently, we can bypass the recovery phase, resulting in W = B A = B AP". In the case of structured pruning, the shapes of the weight matrices vary significantly across different pruning strategies. To simplify the definitions, standardize the recovery process using the pruning mask.
To clearly illustrate the evolution of weight matrix dimensions across these stages, we take LLM- Pruner (Ma et al., 2023) as an example in Appendix C, visualizing the transformation from Wo = WR. and W w under LORAM with structured pruning.
WP. WA +
Pruned Full-Rank Weight Alignment. Given the original pre-trained weights Wo. the optimal low-rank matrix learned is W Similarly, for pruned weights W the optimal low-rank counter- part is W If the knowledge encoded in Wo and W is closely aligned, the knowledge embedded in W should approximate that of W Consequently, the recovered matrix W should effec- tively pair with Wo. yielding performance improvements similar those from W However, the pruning function P(-) disrupts some of the knowledge embedded in the original weights, leading to mismatch between Wa and W Such knowledge mismatch causes W A when paired with Wo, deliver suboptimal performance, particularly at higher pruning ratios.
To address the knowledge mismatch, we propose an efficient alignment scheme, namely continual pre-training of pruned weights W into W on a small, general corpus Dz. Formally, we minimize the alignment loss L, defined as following:
LA EEEE
Σ log p (St41 S.a WPAA)
where p(5 Sct; W represents the model's predict likelihood of generating the next token S:41 given the first tokens Ser of input sequence s (token number is [s) and current parameter matrices P.P.A. This alignment process is a one-time, offline operation on relatively small corpus (about 105 million tokens), making it cost-effective solution for model publishers. Alongside the base model, they can release the aligned pruned model, enabling low-resource users to fine-tune the base model via LORAM for specific downstream tasks.
Pruned Full-Rank Weight Quantization. The design of LORAM inherently supports the seam- less integration of quantization techniques, further reducing memory consumption during training by applying quantization to pruned models. For example. by adapting the LoR quantiza- tion scheme QLoRA (Dettmers al., 2023), LORAM extends into QLORAM, the pruned full-rank weight matrix is quantized into the NE 4 format, while the pruned low-rank matrices B" and A re- main in full or half precision, striking a balance between memory efficiency and fine-tuning quality.
Formally, given a quantization function Q(-), during training, the forward pass output activation vector h for an input feature vector is computed as:
= xQ(W)) + xB AP xW 0 + X
where W represents the full-rank weight Wo after undergoing pruning via P(-), followed by quantization using Q(-). We can also quantize the pruned full-rank weight after alignment A.A.A. For inference, unless additional quantization is required, QLORAM operates identically to Lo- RAM, as shown in Fig. 2 (d). It utilizes the original full-rank weights Wo alongside the recovered low-rank matrices Ba and A3" to perform the forward pass according to Eq. (7). In summary, for
5

---

Published as a conference paper at ICLR 2025
LORAM, as shown in Fig. 2 (e), the offline processing path of the frozen full-rank matrix W which minimizes GPU memory usage, is Wo P(-) W بكك
Q(-)
W.
Fig. 2 (f) shows that the
online generation path for the trained low-rank matrix W is WA W
P
Part
w
q()
W
3 EXPERIMENTS
3.1 SETUP
Pre-train Corpus. To align the inconsistent knowledge between the pruned model during training and the original model during inference, we apply LORAM to continual pre-training LLMs on mixed corpus of Fine (Penedo et al., 2024) and Open OpenWebMath (Paster et al., 2023). Notably, this alignment process is a one-time, offline operation that can be executed by model publishers.
Fine-tuning Data. Following the fine-tuming setup of LoRA (Hu et al., 2022), we primarily con- duct supervised fine-tuning (SFT) on the OpenHerrmes-2.5 (Teknium, 2023) (referred to as Open- Hermes) and OpenOrea (Lian et al., 2023) datasets. To effectively assess the overall fine-tuning performance, evaluate test perplexity not only in-domain test sets constructed from the in- struction fine-tuning data but also on out-of-domain test sets built from Alpaca (Taori et al., 2023).
Downstream Task. We focus on the performance of LORAM in various downstream tasks, in- cluding MathQA (Amini et al., 2019) and GSM8K (Cobbe et al., 2021) in mathematical reason- Easy (Clark et al., 2018), HellaSwag (Zellers et al., 2019), Open-
ing, six tasks-Are Challenge BookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), and WinoGrande (Sakaguchi al., 2021) in common sense reasoning, and HumanEval (Chen et al., 2021) in code generation.
Sparsification & Quantization. For sparsification P(.), we first establish a variant LORAM- RAND randomly structured pruning and adapt LORAM to another three variants based on lead- ing approaches: LORAM-STER with the structured pruning LLM-Pruner* (Ma et al., 2023) and LORAM-SEMI and LORAM-UNST with the non-structured (semi-structured & unstructured) prun- ing SparseGPT* (Frantar & Alistarh, 2023a). For quantization Q(-), we achieve QLORAM by com- bining LORAM with the LoRA- -oRA-tailored quantization algorithm QLoRA (Dettmers et al., 2023). The storage cost of the original model primarily drives the memory consumption during LoRA weights training. Thus, we define the parumeter reduction ratio as the count of parameters before and after pruning, to evaluate the memory efficiency of baselines. The details of our experiment setups and hyperparameters are provided in Appendix B.
3.2 FINE-TUNING CONVERGENCE
We investigate the convergence trends of LORAM across varying model scales (LLaMA-2-13B & LLaMA-2-70B) and different instruction-tuning datasets (OpenHermes & OpenOrca). To assess training performance, we track perplexity over training iterations on both out-of-domain (Alpaca) and in-domain (OpenHermes or OpenOrea) test sets, shown in Fig. and Fig. 4.
Out-of-Domaal Performance. LORAM consistently achieves out-of-domain performance with similar trends, positioned between the LoRA fine-tuned models of the same scale and smaller mod- els, across different models and datasets. As shown in Figs. and (a), for the 13B model, the per- plexity of LORAM variants pruned by different algorithms is lower than that of the A-Rt-trad 7B model but higher than the LoRA-trained 13B model, with LORAM- RAND and LORAM-STRE achieving 2. 1.17x parameter reduction. Similarly, shown Figs. 3 and 4 (c), for the 70B model, this reduction extends to 12.84x under similar convergence trends.
In-Domain Performance. LORAM shows limited improvement in-domain performance, likely due to overfitting when the base models are fine-tuned with LoRA, resulting in relatively lower
Pruner (Apache-2.0 license) cttptttcooo/////sSssiaaaaaaaae spp.ssggppppppppppeeaoeapep (Apache-2.0 license)
'https://e //tpss///gi thub. IST-

---

Published as a conference paper at ICLR 2025
(aT LLa 2A.2 1.5 Alpaca
121 14-23-2-138 Opantarmes
[C] 2-700 Alpo
IdI LLLNMA 2-728 Oppriiaamnsss
LORA 40X1
LSKA Fal 2010
10 (ARU 2010 130
L'0
Strs
TE0 CLORAM Sira (22200)
IARAM Card
120 LENAN Sant 6E-7
Heeratons
Beralions
(brral
Figure 3: The test perplexity of training LLaMA-2-13B & LLaMA-2-70B on OpenHermes.
140 1 18 AAAACA
(h) 2- Open
2MA-2-70R Ploara
L006-2-70R (Opeettra
(L
Lie LORK 30xx Tho DustAS: Sand BE (22300
215 LITAN 3TM
LEGAN Bur (OLEDA)
115 Ausleniann 220-
310 LURAA (ho 12.10A4
355 Lo Una
Thaning lleratoos
Traning BaTTETTTT
Thening Derations
Thiniing Iar
Figure 4: The test perplexity of training LLaMA-2-13B & LLaMA-2-70B on OpenOrca.
perplexity. This is further supported by downstream evaluations, where models that excel in in- domain perplexity often underperform in downstream tasks. As shown in Figs. 3 and 4 (b), while the LoRA-trained 7B model outperforms 13B FORAM-RAND and LORAM-STRUS tests, it underperforms on several downstream tasks as shown in Section 3.3.
in-domain
Non-Structured LORAM Excels in In-Domain. The non-structured variants (LORAM-SEMI
& LORAM-UNST) consistently outperform their structured counterparts (LORAM-RAN) & LORAM-STRE STRU) in-domain test sets. As shown in Fig. 3 (a) vs. Fig. 3 (b) and Fig. 4 (a) Fig. 4 (b), the in-domain perplexity of LORAM- SEMI and LORAM-UNST is notably lower, while their out-of-domain performance shows less pronounced differences. This advantage likely arises from the more selective weight pruning in the non-structured variants, which preserves information cap- ture capabilities similar to the original model, thus enhancing in-domain performance.
LORAM Benefits from Scaling. The performance gains of the non-random Lo- RAM become more evident the model size grows. As shown in (a.b) of Figs. 3 and 4 vs. (c,d) of Figs. 3 and 4, LORAM- AMSTRU outperforms LORAM-RAND considerably on the 70B model, while the difference is marginal on the 13B model. This indicates that larger models exhibit greater differences in the redundancy of individual weights, making selective pruning more effective".
3.3 DOWNSTREAM TASK PERFORMANCE
We evaluate the performance of various models trained with LORAM on different instruction data across three downstream tasks: mathematical reasoning, common sense reasoning (CSR), and code generation. Results are summarized in Tables to 3. We highlight the core competition scenario with a gray background, which includes the untrained original model and smaller sibling model trained with LoRA. For instance, for LoR A M-trained LLaMMA-2-33B, we report the scores of the 13B without fine-tuning and the L-Rretr 7B model. Blue backgrounds of varying intensity indicate the degree of improvement for each LORAM variant relative to the core competition sce- nario: darker shades indicate greater improvements, while lighter shades signify smaller gains.
Overall, we observe that most LORAM variants outperform the core competitive baseline across all downstream tasks, particularly in mathematical and common sense reasoning. This improve- ment is further amplified by increasing the model scale. Specifically, as shown in Table 1, the 70B LORAM RAND and LORAM-
models achieve a 12.84x reduction parameters compared to the original 70B model (70B w/o FT), exceeding the 5.30x reduction of the LoRA-trained 13B model. terms of performance, LORAM improves the original 70B model's score on GSM8K from 52% to 57% significantly outperforming the LoRAetrained 13B model, which only achieved 37%. These results demonstrate that updating low-rank matrices on pruned models effectively re-
"The trained low-rank matrices are visualized in Appendix D, and the update patterns they exhibit somewhat align with these insights.
7

---

Published as a conference paper at ICLR 2025
Table 1: Accuracy (%) of the MathQA (1-shot) & GSM8K (8-shots) in the mathematical domain under LLaMA-2. indicates the theoretical parameters reduction of non-structured pruning. How- ever, these parameters are filled with zeros in actual training, so the memory footprint is not reduced.
OPENHERMES
OPENORCA
METHOD
PARAMETER MATHQA GSM8K REDU. RATIO
MATHQA GSM8K
13B W/O FT
32.60
24.26
32.93
23.35
1.00
7B LORA
29.61
22.82
30.95
13.87
1.93x
13B LORAM M-RNAND
33.77
27.22
32.83
25.93
2.17x
1 13B LORAM-S
33.80
24.64
33.07
24.49
2.17x
13B LORAM- M-MMII
SEMI
31.76
36.92
33.07
27.29
1.95
13B LORAM UNST
30.12
31.92
32.70
26.61
2.16x
70B W/O FT
39.53
$2.01
39.53
$2.01
1.00x
13B LORA
32.03
36.69
33.63
25.70
5.30x
70B QLORAM-RAN RAND
39.66
57.62
39.40
55.72
12.84
70B QLORAM-STKE STRU
39.77
57.16
39.73
54.44
12.84x
Table 2: Average accuracy (%) of the CSR in the common sense reasoning domain (1-shot) under the I.L.MA22. Baseline results for each subtask of CSR are detailed in Appendix E.
OPENHERMES
OPENORCA MEAN STD
METHOD
PARAMETER REDU. RATIO
MEAN + STD
13B W/O
64.28+1.30 61.51+1.22
64.28 30
1.00x
7B LORA
31.42+1..30
1.93x
13B LORAM-E AM-RAND
64.64-1.29
64.49- 1.30
2.17x
13B LORAN PERAMMSTREE STRU
64.42.1.22
64.32.1..29
2.17x
13B LOR AM-SEMI SEMI
64.38-11.29
64.73+1.30
1.95x
13B LORAM-UNST
64.12+1..29
64.68+1.29
2.16x
70B w/o FT
68.69+1.27
68.69+1.27
1.00x
13B LORA
65.05+1.229
65.40+1.22
5.30x
70B OLORAM-RAND
68. 68.99++1.5 1.22
68.46 1.27
12.84
70B QLORAM-STRU
69.10-11.
68.94 1.27
12.84x
Table 3: PASS @ 1() and PASS @ 10(%) of HumanEval in the code generation domain under LLaMA-2. The best results for all baselines are reported, selected from TEMPERATURE settings in {0.0, 0.2, 0.4, 0.6, 0.8) with TOP, fixed 0.95.
METHOD
OPENHERMES
OPENORCA PASS
PARAMETER REDU. RATIO
PASS@ PASS PASS
13B W/O
17.68
35.37
17.68
35.37
1.00x
7B LORA
15.24
28.04
15.85
26.21
1.93
LORAM- AM--AN
19.51
33.54
19.51
32.32
2.17x
13B LORAM STRU
17.68
35.37
17.07
31.71
2.17x
13B LORAM- SEMI
20.12
35.37
18.29
39.63
1,95x
13B LORAM-UNST
22.56
34.15
18.29
37.20
2.16x
70B w/o FT
31.71
58.54
31.71
58.54
1.00x
13B LORA
18.29
35.98
18.29
39.02
5.30x
70B OLORAM-RAND
29.27
57.32
31.71
56.71
12.84
70B QLORAM-STRU
32.32
58.54
32.32
59.15
12.84x
duces memory requirements during training. Merging the recovered low-rank matrices into the original model yields substantial performance gains during inference.
3.4 ADAPTION TO LLAMA-3.1
Here, we extend LoRAM-Str to LLaMA-3.1 herds and investigate two key questions: (1) How does LoRAM perform terms of perplexity and downstream tasks within this model series? (2) What the effect of continued pre-training iteration steps (proportional to corpus size) on performance?
As shown in Fig. 5 (a.b), QLORAM-S STRU for the LLaMA-3.1-70B model exhibits consistent trends across both out-of-domain and in-domain test sets. It achieves 15.81 x parameters reduc- tion while its perplexity falls between that of the smaller LoRA-traines 8B and LoRA-trained 70B.
8

---

Published as a conference paper at ICLR 2025
w/o Recovery & w/o Alignment --- w/ Recovery & we Alignment
we Recovery & w/ Alignment --- w/ Recovery & w/ Alighim entt Lo
RalAAM
Saru
Tel 26 To nonning Itoraions
24 Itcrocerss
ring Ecrations
Troiningg Itcrttoos
Figure 6: Necessity of Recovery & Alignment across different pruning strategies on LLaMA-2-13B.
In downstream tasks (Fig. 5 (c)), QLORAM-STRU sig-
LLa79--33-700 Appace
80 IT
53.27
64:27
34.76
01:59
nificantly exceeds the 8B w/o FT, 8B LoRA, and 70B w/o FT, even surpassing the LoRA-trained 70B on MathQA and HumanEval @Pass 10). Moreover, we observe that min- imal pre-training corpus can yield substantial per- formance gains. For in- stance, QLORAM-STRE 200, with just 200 updates (about 13 million tokens), achieves a 15.81 x parame- ter reduction alongside performance improvements. This one-time, low-cost alignment allows pub- lishers to offer aligned pruned models for low-resource users to customize tasks.
200
72.28
Trarr ng Itaratorss
700 LMRA
$1.66
00:74
70:63
02.80
64:15
(5) 11-N6311-05 Opentiammss
TER GloRaM Smu 200
$4.67
75:30
73.35
57.97
83:54
$4.64
00:36
57.37
AR 458
70.00
73.07
53.08
11:54
20 Iterations
Hahoo GaMEK
C5R
Peerg3
Perggi
Figure 5: The test perplexity & downstream performance of training LLaMA-3. 1-70B on OpenHermes.
3.5 NECESSITY OF RECOVERY& & ALIGNMENT
We conduct an ablation study on two critical phases of LORAM: recovery and alignment. To assess their necessity, we analyze the convergence trends of various pruning variants on the Alpaca test set using LLaMA-2-13B.
Impact of Recovery. we compare the standard approach with an alternative setup where the pruned low-rank matrices are directly combined with the pruned full-rank model weights (w/o Recovery) and track perplexity changes over iterations. As shown Fig. 6, for all four pruning strategies, models without the recovery phase (solid lines, w/o Recovery & *) consistently exhibit higher perplexity compared to those with recovery (dashed lines, w/ Recovery & *), particularly in structured LoRAM (see in Fig. (a) and (b)). This highlights that the recovery phase leverages relatively redundant neurons during training to enhance inference performance significantly.
Impact of Alignment. We also introduce a variant of LORAM without continual pre-training for alignment (w/o Alignment). As shown in Fig. 6, aligned pruned models (yellow lines, * & w/ Alignment) consistently achieve lower perplexity than unaligned counterparts (blue lines, & w/o Alignment), irrespective of the pruning strategy or recovery phase. This highlights that even low- cost continual pre-training on small general corpus effectively narrows the knowledge gap between pruned and original models, enhancing the overall performance of LORAM.
3.6 SCALING LAWS FOR PARAMETER REDUCTION ON LORAM
We explore the impact of scaling the parameter reduction ratios in Fig. 7. The LoRA-trained LLaMA-2-13B (triangles) achieves a 5.30 parameter reduction, while QLORAM-STRU maintains superior perplexity on the Alpaca and further reduces parameters across both instruction datasets. In contrast, naive pruning leads to a significant increase in perplexity with minimal pruning. When the parameter reduction ratio reaches 28.56x, QLORAM- -TRUU sustains an effective perplexity of approximately 2.5, whereas naive pruning escalates 621.98. These highlight LoR AM's ability

---

Published as a conference paper at ICLR 2025
drastically reduce memory of the base model by updating LoRA weights in the pruned model, while seamlessly integrating with the full model to preserve inference performance.
We then evaluate the performance of models trained with LORAM OpenHermes across var- ious downstream tasks under different pruning ratios. As shown in Fig. 8, overall performance improves as the parameter reduction ratio increases from 9.82x to 16.95 before declining. Notably, tasks achieve op- timal performance between parameter reduction ratios of 12.84X and 16.95x, consistently outperforming 13B LoRA and 70B w/o
600
138 OpenOrca
13B LoRA OpenHermes
621.88
400
QLeRMM-SU OpenOrca
708 GtoRAM-Siru Opentermes
pmpllexiyy Alpaca
200
708
(Nalva Pruning)
92.20
10.36
38.09
FT. However, at pa-
rameter reduction ratio of
2,50
9.82x, despite the larger memory capacity available for LORAM, downstream performance does not al-
2.58
2.37
2.49
2:33
2.31
2.32
2.38
2.22
2.27
2.30
ways exceed that of higher parameter reduction ratios. We attribute this to the fact that lower parameter reduction ratios fine-tune more parameters, potentially degrading the pre- trained model's performance on certain tasks (e.g., Fig. (a.c.e)). This effect is also reflected in MathQA, where a fully fine-tuned LoRA model underperforms the pre-trained model without fine- tuning (see Fig. (b)). Moreover, excessive pruning at a ratio of 28.56x leaves too few neurons to capture the rich information needed for downstream improvements, particularly in tasks like code generation (see Fig. (d.e)).
Parameter Reduction Ratio (t)
Figure 7: Effect of scaling parameter reduction ratio.
(a) GSMBK
(6) MathiQ
(4] CSR
(d) HumanEval (Passi])
(e) HumannEall (Pasig10)
3.35
3.55
0.68
0.43
030
708
130 LoRA
6.36
2.25
0.50
0.54
0.32
0.55
Reduction
Reduction
Paramater Reduction Rabio
Parameter
Figure 8: Performance of downstream tasks across different parameter reduction ratios.
CONCLUSION
We propose LORAM, memory-efficient LoRA training scheme for large language models. Lo- RAM significantly reduces the count of parameters of the original model by 16.95x, while main- taining the performance of large-scale LLM fine-tuning. We identify several open questions for LORAM, including the potential for reduced inference costs through context-aware computational graph recovery and its applicability diffusion models (Ho et al., 2020). We hope our work inspires further research memory-efficien LoRA training from a sparsity perspective and believe LORAM will serve as a valuable tool for the community, enabling LoRA training of large-scale models on consumer-grade hardware.
models like vision transformers (Dosovitskiy et al., 2021) and
ACKNOWLEDGMENTE
This work is supported by the Pioneer R&D Program of Zhejiang (No. 2024C01021), "Lead- ing Talent of Technological Innovation Program" of Zhejiang Province (No. 2023R5214), OPPO Research Fund, the Major Research Program of Zhejiang Provincial Natural Science Foundation (No. LD24F020015), and NSFC Grant No. 62402420.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 7319-7328. URL
Association for Computational Linguiistiiiiii Linguisties, 2021. doi: 10. 111665//1/2222AAALLLLLL https: doi www./orgooooo.oooooooooooo 186.:/ vttp///////222/2 acl- -ong. 568.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-
Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Humun Language Technologies, Volume I (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, June 2019, Association for Computational Lin- URL https: / /
guistics. doi: 18.18653/v 1//19-1145
9- 1244.
Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram. Lamda: Large model fine-tuning via spectrally decomposed low-dimensional adaptation, 2024. UtL.httss://arxii http://xxxvv.rggb//////o 2406. 12832.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In ICLR 2024 Workshop: How Far Are We From AGI, 2024. URL https: http///ppnneeeeeeeeeeeeeeeee net forum?
id=v9L38gCohb.
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability- based case study at 66 billion scale. arXiv preprint arXiv:2212.0000 2022.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Yuji Chai, John Gkountouras, Glenn G. Ko, David Brooks, and Gu-Yeon Wei. Int2.1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation, 2023. URL https: / / /awxiv.vvvv.////i///a.a/aaavtv org/ aba///a//// /2006. 081 62.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021.
Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. Lorashear: Efficient large language model structured pruning and knowledge recovery, 2023. URL https://arxiv. orgg/bb/// 2310 18356.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4794-4802, 2019.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin

---

Published as a conference paper at ICLR 2025
Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Samkaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsterm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240:133, 2023. URL http: jmlr org/ papers/v24) 22-11 1144 html.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: arxiv. org/ abs 1183.00557.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10.2210.111 14168, 2021.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint #xXiv2208.003333 2022.
Tim Dettmers, Artidoro Pagnoni, Ani Holtzman, and Luke Zettlemoyer. Qlora: Ef- ficient finetuning of quantized llms. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz. Hardt, and Sergey Levine (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, wwp.. wwwpcppppppppllieee 2ttp://eeeeeeeepee//22////2//// html.
2023, 2023. URL
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. https// httpp///ew///eeeee..... net forum? ad=YicbFdNTTy.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, and et al. The llama herd of models, 2024.
URL https:://// http///xxvvv.////// org/ abs//407 21783.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp- 10323-10337. PMLR, 2023a. URL. https: httttps:/procceed..ttttt /ptpc/edino.....ceee/ee. .mlr. press// v202/frante wwr.//aanaaaa3a2 html.
Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2001.00774 2023b.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers, arXiv preprint arXiv:2210.1332. 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: accurate quantization
for generative pre-trained transformers. In The Eleventh International Conference Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: http:::///peerevieee net forum? d fww..
12

---

Published as a conference paper at ICLR 2025
Naibin Gu, Peng Fu, Liu,
Liu, Bowen Shen, Zheng Lin, and Weiping Wang. Light-PEFT: Light- ening parameter-efffiient fine-tuning via early pruning. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 7528-7541, Bangkok, Thailand, August 2024a. Association for Computational Linguis- tics. doi: 10. 0188663/112222/ 4f22224ffnnings-all..... URL https: //tp:://alaaaaa htl///caanttoooo....o org/2024. findings-aad aeciinngsscl.... 447/.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Reprexentations, 2024b.
URL https:ttp////...t. /ttps/////ppeeeeeeeeeeeee net/ /tto//oddudmmuoorrro
Han Guo, Philipp Greengard, Eric Xing, and Yoon Kim. LQ-IoRA: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: wwwpppppeeeeew..eom/ee net / forum?id- xw2 Xw29VvOMmU.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 2015.
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 4340-4349, 2019.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer.
CoRR, abs/2102.01293, 2021. URL https: /arxivivvvv..//aa wwtg/bs///////oo////..//eee/aa// /bs//1/2 01293.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: 02531, 2(7), 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpt: merrip htt.t:pss//pproceednng.g neurips. /ttpppper/222//asttt html.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22(241):1-124, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Henni- gan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy. Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
Cheng- Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujäi, Alex Ratner, Ranjay Krishna, Yhen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. Anna Rogers, Jordan L.. Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 8003-8017. Association for Computational Linguisties, 2023. doi: 10. 10.18653/V1/2023 FINDINGNN 18653 1//lv112200 2111//222..2 f mondingg-aacl. 507.
URL https: //doi org/
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL htt.s::/// http:://ooeeeeeeeeee
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arine arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018
13

---

Published as a conference paper at ICLR 2025
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tumine iin-ttniing with lora, 2023. URL pettps///axivv.om wwwp//tgg/./.ooooootttooot...tst
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https::///// //tp://aa/vvv///ii httg/abs//000 08361.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Tweifth International Conference on Learning Representations, 2024. URL
https:/ httpps:///peereeee wpwpreveeweeweeee/e/eeeee..oeewe .et/ / forum? -o-
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: beaech-based pseudocode to code. In Advances in Neural Information Processing Systems, 2019.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic. 7-11 November, 2021, pp. 3045- 3059. Association for Computational Linguistics, 2021. doi:
E.N...--MMLLMMMMEEE MAMNN. 243. URL https: / /ooi .rgg O 8633 v1 221//22212222211 emn lommnn--maa p- 243.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society, 2023a.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Pa- pers), Virtual Event, August 1-6, 2021, pp. 4582-4597. Association for Computational Linguis- tics, 2021. doi: 10. 10.18653/V1/2093 11611////2211AAALLLNGG..... ACL-LONG.333., URL https: /tt.///o...o///////o.oo org/10..g1111000 18653/ v1/2021.
353.
Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: CeRA--fne-tuning-- aware quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
Zhikai Li, Xiaoxuan Liu, Banghua Zhu, Zhen Dong, Qingyi Gu, and Kurt Keutzer. Qft: Quantized full-parameter tuning of Ilms with affordable resources. arXiv preprint 2rXiv:2310.07147, 2023b.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. bttps::/https// huggingface. co/ /tt////ennnnaa/aeeeeeeeeeeeeeeee Saaan--rca//ppenreee OpenOrra,, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. ACL, 2017.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficienn fine-tuning is better and cheaper than in-context learn- ing. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: wtwpssppppssaaaaapaaap nips ce wow.ractttooooeeeneeeeccec html. Ocde695b83ad Btw.aaaolBB6611d446888888888888 Abstract
Shih-Yan Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICML, 2024a.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue. Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng. Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR, 2024b.
14

---

Published as a conference paper at ICLR 2025
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. CoRR, abs/2103.10385, 2021. URL attpp://arxiv. org/ abss21033 10385.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.0522 2018.
Zichang Liu, Jue Wang. Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pp. 22137-22176. PMLR, 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httph:ttp//// pttttps://ppnreeiews wwwp//wennneeeeeeeeeeeeeeeeeeeep net forum? id 1dannajjJJJJBB
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. 2016.
Markus Nagel, Mart Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1325-1334, 2019.
OpenAl. GPT-niechnican report. CoRR, abs/2303.08877, 2023. doi: 10448500ARRXIV22333007777 URL https://dol www.ooog....oooooooooo.ooo/..e.o
Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models.
arXiv preprint arXiv:2206.09533 2022.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. URL https://azxiv.org: /
1310.067860
Guilherme Penedo, Hynek Kydličck, Loubna Ben allal, Anton Lozhkov. Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https: /tl::///rxxee
wwwpaaiivvo.oo//...aeaaaaaev.aaa org/ at///bb//2244 /2406 10006.75555 17557.
Xiangyu Qi. Yi Zeng. Tinghao Xie, Pin-Yo Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024. URL https: / //openreview net./ formm/orrmmmmm diw.mmmcnddhtheeddc.....
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 papers nips wwwpppppppppppccpccccecceeeeeeee files/ hatees/aapp///ee 2203 //aah// html.
16, 2023, 2023. URL http: /ttptttpp//ppppssppppp
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an ad- versarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, aug 2021. ISSN 0001-0782. doi: 10. 00/1445/3474381. URL https:t//t......ooo 0
Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https:
/ //aaxiiv.rrg/aba//222 /2000 05202.
15

---

Published as a conference paper at ICLR 2025
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Tweifth International Conference on Learning Representations, 2024. URL https: opwnreview.eeeeeeeeeeeeee.eeeeee o
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprini arXiv: 12136, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. htt//tatsu-lall saab/stanfont astaanfodd___aaaee
https: pitess:/ggiees /ggthub.
2023.
Teknium. Openhermes 2.5: An open dataset of symthetic data for generalist Ilm assistants, 2023.
URL httpst::
co co/dttsees http//teeiiiimmieet wtwpp///ww/epeeeeeeeeeeeeee
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 0os//332..1397,, 2023a. doi: 10.485500arX...2302.11000 139711. URL bttpss://oot orggooggg/ooooooooooo 10..00000000 48550/ arxiv 2302 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CORR, abs/2307.09288, 2023b. doi: 10.48550/ARXIV.330..2 092888. URL https:: //tp:////o../.... wrg. 48550/ arxiv 2307 09288.
Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, and Chuan Wu. Prolora: Partial rotation empowers more parameter-efficeen lora. In ACI., 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. URL https: arxia. org/ abs/ /2222222222 10560.
Mengzhou Xia. Tianyu Gao. Zhiyuan Zeng. and Danqi Chen. Sheared LLaMA: Accelerat- ing language model pre-training via structured pruning. In The Twelfth International Confer- ence Learning Representations, 2024. URL https: htttps://openreiiii net forum? f 0910dae0zp.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: arXiv preprint
Accurate and efficient post-training quantization for large language models. arXiv:2211.10438 2022.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xi- aopeng Zhang, and Qi Tian. Qa-lora: Quantization-awan aware low-rank adaptation of large language models. In ICLR, 2024.
Suorong Yang, Peng Ye, Wanli Ouyang, Dongzhan Zhou, and Furao Shen. A powered frame- work for robust and generalizable data selection. arXiv preprint arXiv:2410. 12 15, 2024.
16

---

Published as a conference paper at ICLR 2025
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.0/ 01861, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Marquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-48000 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 18653/v1/P19 1472. URL htwps://pp/// aaclllll antho loge wtt/
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Hanna M. Wallach, Hugo Larochelle. Alina Beygelzimer, Florence d' d'Alché-Bus Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12360-12371, 2019. URL htttps://proceede/ www.eeeeeeeenn.ee.e.......ee.. neurips. ps..ip........ tttppp.ccccpppppc cc/paper/ Abstract html.
2019 1019/hassn http/hash/hae wwwesh///Sali http:/hhhhheeaa92222eeee646 wew oet.aaaa00..2222aa0eeeeee3e11112
Longteng Zhang. Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient
low-rank adaptation for large language models fine-tuning, 2023a. URL https://arxiv. org/abs/ 2308 03303.
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 3013-3026, Bangkok, Thailand and virtual meeting, August 2024a. Association for Computational Linguistics. URL https: / wnt.anhthooo.ooooooo org/2024. findings-acl. 178.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiiiet fine-tuning. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. jiY.
OpenReview.net, 2023b. URL https: :ttpttt::///pppppeeeeee http//pppeeeeeeeeeeeeeeeeeeeeeee net/ www.mmiddoige200
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse LLMs. In The Twelfth International Conference Learning Representations, 2024b. URL https:
/ /ppenreview www.nneeieww.ee/forr.eeeeeee met/forrmiiddiaaeeeeeeeeteeeeeee 1adm2ddd1ndDZZd@444
Jiawei Zhao, Florian Schäfer, and Anima Anandkumar. Zero initialization: Initializing neural networks with only zeros and ones. Trans. Mach. Learn Res., 2022, 2022. URL https:
/ //openzeview.... net. fort/ffrm?? wttpmmmmmmm1axxxmmmmm1
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pp. 7543-7555 PMLR, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, lon Stoica, and Hao Zhang. Lmsys-chat-Imi A large-scale real-world I1m conversation dataset, 2023.
Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, and Tiejun Zhao. Lora-drop: Efficient lora parameter pruning based on output evaluation. CoRR, abs/2402.00721, 2024. doi: 10.48550/ ARXIV.2402.07721. URL https:tpp........p. //t//////////// htt.d/i..orgg//// www.ooogg...... 48500 /ariiv..vvvvv. 2402 07721.
Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices, 2023. URL https:// a arxiv. org/ abs /2309. 02401.021991........22222 02411.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORK
Low-Rank Adaptation. LoRA (Low-Rank Adaptation) (Hu et al., 2022) has emerged promi- nent technique for parameter-efficeent fine-tuning (PEFT) (Li & Liang, 2021; Lester et al., 2021; Liu al., 2021: Qiu al., 2023; Liu et al., 2024b; 2022). By injecting lightweight, trainable low-rank decomposition matrices into frozen pre-trained weights, LoRA enables efficient task customization, especially in ressurce-constrraiee settings. Some LoRA variants (Liu et al., 2024a; Ding et al., 2023; Zi et al., 2023; Zhang et al., 2023b; Kalajdzievski, 2023) have been developed to enhance its generalization and robustness, while others (Zhou al., 2024; Zhang et al., 2023a; Kopiczko et al., 2024; Azizi et al., 2024; Wang et al., 2024; Yang et al., 2024) address the increased memory overhead associated with scaling up model sizes. However, during training, these efficient LoRA variants still struggle with the substantial memory footprint of the original LLM parameters.
LoRA-related Compression. Model compression techniques like quantization (Han et al., 2015; Jacob et al., 2018; Nagel et al., 2019; Zhao et al., 2019; Yao et al., 2022; Park et al., 2022; Dettmers et al., 2022; Xiao al., 2022; Frantar et al., 2022), sparsification (Molchanov al., 2016; Liu al., 2018; He et al., 2019; Hoefler et al., 2021; Frantar & Alistarh, 2023b; Liu et al., 2023; Bansal et al., 2022), and distillation (Hinton et al., 2015; Cho & Hariharan, 2019; Tang et al., 2019; Touvron et al., 2021; Hsieh al., 2023; Gu et al., 2024b) have proven effective in reducing the memory footprint of LLM during training and inference. Naturally, the concept of compression has been adapted LoRA to alleviate the substantial memory consumption dominated by pre-trained model parameters. In particular, LoRA-related quantization schemes (Dettmers et al., 2023; Xu et al., 2024; Li al., 2024: Guo et al., 2024; Frantar al., 2023; Chai al., 2023) have been widely explored, but they still face the limitations of 1-bit precision, typically quantize weights to 4-bit to balance training efficiency with performance. Our work aims to push the boundaries of memory-efficient LoRA training by leveraging sparsification to achieve cost-effective performance improvements. Notably, existing LoRA-related sparsification works (Chen et al., 2023; Zhang al., 2024a) focus on designing pruning algorithms to slim down models and use LoRA to recover the knowledge of pruned models, thereby producing compact but high-quality models. In contrast, LORAN enables effective general pruning under high base-model sparsity, whereas (Gu et al., 2024a) focuses on task-specific LoRA sparsification with limited impact on base model memory reduction.
B EXPERIMENTAL DETAILS
Pre-train Corpus. To align the inconsistent knowledge between the pruned model during training and the original model during inference, we apply LORAM continual pre-training L.LMs in teacher-foreing manner (Bachmann & Nagarajan, 2024) on mixed corpus of FineWeb (Penedo al., 2024) and OpenWebMath (Paster al., 2023). FineWeb, containing over 15TB of cleaned and deduplicated English web data from Common Crawl. Open WebMath, extracted from over 200 billion HTML files on Common Crawl, provides high-quality mathematical text. Mixing these datasets enhances the pruned model's capabilities in both general and mathematical domains.
Unless specified otherwise, we randomly sample 102,400 instances from both Fin and Open- WebMath to construct a mixed dataset with a sequence length of 512, yielding approximately 105 million tokens. The default training batch size is 128, allowing up to 1,600 update steps. We train without data repetition over sufficiently large corpus to simulate a realistic pre-training scenario. Notably, this alignment process is a one-time, offline operation that model publishers can exccute.
Fine-tuning Data. Following the fine-tuning scenario of LoRA (Hu et al., 2022), we primarily conduct supervised fine-tuning (SFT) on the OpenHermes-2.55 (Teknium, 2023) (referred to as Open- Hermes). OpenHermes is large-scale dataset constructed from synthetically generated instructions and chat samples, encompassing diverse sources such as Airoboros 2.2 (Wang et al., 2023), Came- IAI Domain Expert Dataset (Li et al., 2023a), ChatBot Arena (GPT-4 Only) (Zheng et al., 2023), and more. To further demonstrate the general effectiveness of the LORAM alignment process, we also evaluate LoRAM on the OpenOrca (Lian et al., 2023) dataset. OpenOrca is widely used
18

---

Published as a conference paper at ICLR 2025
instruction fine-tuning dataset where each data instance represents entries from the FLAN collec- tion (Longpre et al., 2023). augmented by submitting the listed questions to either GPT-4 or GPT-3.5.
By default, we train SFT the instruction dataset with batch size of 128 and a sequence length of 512 for 400 steps, totaling approximately 26.2 million tokens. To effectively evaluate the overall fine-tuning performance,
assess the perplexity of the fine-tuned model on an out-of-domain test set. This out-of-domain test set is constructed by randomly sampling 2,000 instances from the Alpaca (Taori et al., 2023) test set, truncated sequence length of 512.
Downstream Task. We focus on the performance of LORAM in various downstream tasks, in- cluding mathematical reasoning, common sense reasoning, and code generation. All our down- stream task evaluations are performed on Im-evaluation-hanl harness' and code-eval 6 with VLLM
For mathematical reasoning, we benchmark the accuracy of baseline models using greedy decoding on MathQA (Amini et al., 2019) with a 1-shot setting and GSM8K (Grade School Math 8K) (Cobbe et al., 2021) with 8-shots, Chain of Thought (CoT) prompting and strict match MathQA is large- scale dataset comprising 37k English multiple-choice math word problems, covering diverse math domains. It extends the AQuA-RAS dataset (Ling et al., 2017) by annotating problems with fully specified operational programs using a new representation language, building on the questions, op- tions, rationale, and correct answers provided by AQuA-RAT. The GSM8K is dataset of 8.5K high-quality, linguistically diverse grade school math word problems, designed to evaluate multi- step reasoning in basic arithmetic operations (+-x+). We conduct evaluations on its 1.3K test set with strict-match to assess logical and mathematical reasoning in language models.
For commonsense reasoning (CSR), we report the average accuracy across six tasks Challenge & Easy (Clark et al., 2018), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), and WinoGrande (Sakaguchi et al., 2021)-under 1-shot and greedy de- coding settings. These benchmarks comprehensively assess the model's ability to apply "common- sense" or world knowledge for reasoning, rather than relying on pattern recognition.
For code generation, we compare two pass rates, PASS @ and PASS 10 (Kulal et al., 2019), on HumanEval (Chen et al., 2021) of each baseline in zero-shot setting with sampling parameters of TEMPERATURE = (0.0, 0.2, 0.4, 0.6. 0.8}. and TOP, 0.95. The HumanEyal dataset released by OpenAl consists of 164 handwritten Python programming problems, each with a function signature, docstring, body, and unit tests. Serving as benchmark, HumanEval assesses models on range of Python coding skills, from basic syntax to complex problem-solving. offering insights into their programming capabilities alongside songuage-focusee tasks.
Sparsification & Quantization. LORAM incorporates two model compression techniques: spar- sification, which generates a pruned model for low-rank matrix updates, and quantization, which forms QLORAM further reduce the memory footprint of the pruned model. For sparsification, to validate the general effectiveness of LORAM. we benchmark its performance across various pruning strategies P(-). Specificallyy, we first establish a variant using randomly structured pruning and adapt LORAM AM to another three variants based on leading approaches: the structured pruning LLMM-Pruner (Ma et al., 2023) and the non-structured (semi-structured & unstructured) pruning SparseGPT (Frantar & Alistarh, 2023a). These baselines are summarized below, with the corre- sponding configurations presented Tables to 6.
LORAM-RAND: We adhere to the pruning settings of LORAM-STRU modifying only by randomly removing weights instead of the original gradient- based pruning criterion.
We follow LLM-Pruner and employ a block-wise strategy for local structured pruning. Attention and MLP layers treated separate blocks, with non-critical coupling weights pruned based on gradient information at a uniform ratio. We retain the first four and last two layers of both blocks, focusing pruning on the intermediate layers.
LORAM-STRU:
httter///ggthaacccm///aaattaaa/s harness (MIT License).
6 htttg:///gtthaa..commmbc//////es (MIT License).
hattaa///hhaabb.cm////.rrpppssss (Apache-2.0 license).
(Apccca
hattta///ggttbbb.ccmm//ssee////s (Apache-2.0 license)
(Apache-2.0 0 license)
19

---

Published as a conference paper at ICLR 2025
LORAM-SEMI
We utilize SparseGPT with 4:8 semi-structured sparsity pattern to prune pre-trained weights across all model layers.
LORAM-UNST: We prune individual weights uniformly across layers using predefined pruning ratio based on an unstructured version of SparseGPT.
For quantization Q.), to further reduce memory usage during training, especially when dealing with
models exceeding 70 billion parameters, achieve QLORAM by combining LORAM with the
LoRA-tailored quantization algorithm QLoRA (Dettmers et al., 2023). While LORAM is compati- ble with the quantization of other customized LoRA methods (Xu et al., 2024; Li et al., 2024; Guo et al., 2024; Frantar et al., 2023; Chai et al., 2023), this falls outside the scope of this article.
Architecture & Hyperparameters. We adopt a LLaMA architecture with RMSNorm (Zhang & Sennrich, 2019) and SwiGLU activations (Shazeer, 2020; Zhao et al., 2022). We run all experiments with BF16 format to reduce memory usage. For all configurations, we default to a learning rate le- 3. However, the downstream performance of models fine-tuned OpenOrea is relatively sensitive to the learning rate. Therefore, in this evaluation, we tune the learning rates for each baseline within the range of [1e-5, 1e-3] and report their respective optimal downstream scores. Specifically, le-5 for the 7B LoRA and 13B & 70B LoRAM models, and lc-4 for the 13B LoRA model. All experiments rum on NVIDIA A A100-80GB GPUs with environments of CUDA 12.2, PyTorch 2.4.0, and Transformer 1.45.1. For LLaMA-2 herds, we set low-rank matrices B and A of rank r = for W q. WE. W, and W. o in the attention layer, W up. W gate and W Wdown in the MLP layer, and the head embedding matrix W head; for LLaMA-3 herds, we exclude the injection of the low-rank matrix of W Im head-
Table LoRAM configures ALaMA-2-13B. Comparison of different pruning methods in terms of parameter reduction ratio (Reduction) and HBM footprint (GB) of pruned parameters (HBM), ignoring low-rank matrix overhead.
Method
#Orig. Params Pruning Ratio #Pruned Params Reduction
HBM
LoRAM-Semi LoRAM-Unst
13015864320
0.50
6738415616
1.93x
12.55
13015864320
0.55
6037628912
2.16x
11.25
MoRAM-Rand & Stru
13015864320
0.65
6005662720
2.17x
11.19
Table 5: LoRAM configures LLaMA-2-70B and LLaMA-3,1-700 with different pruning ratios.
Method
#Orig. Params
Pruning Ratio
#Pruned Params
Reduction
HBM
Lon & Stru 68976648192
0.65
28099436544
2.45x
52.34
LoRAM-Rand & Stru 68976648192
0.75
21488738304
3.21x
40.03
LoRdM-Rand & Stru 68976648192
0.85
16272924672
4.24x
30.31
LoRAM-Rand & Stru 68976648192
0.95
9662226432
7.14x
18.00
LoRAM-Rand & Stru 70553706496
0.85
17849982976
3.95x
33.25
Table QLoRAM configures on I.LMMA--70B and I.LMMA3.1-70B with demonstrating more aggressive parameter compression.
Method
#Orig. Params Pruning Ratio
#Pruned Params Reduction
HBM
QLoRAM-Rand & Stru 68976648192
0.65
7024859136
9.82
13.08
QLoRAM-Rand & Stru 68976648192
0.75
5372184576
12.84x
10.01
OLoRAM-Rand Stru 68976648192
0.85
4068231 168
16.95x
7.58
OLoRAM-Rann Stru 68976648192
0.95
2415556608
28.56x
4.50
OLo & Stru 70553706496
0.85
4462495744
15.81
8.31
20

---

Published as a conference paper at ICLR 2025
C VISUALIZATION OF DIMENSION EVOLUTION
To clearly illustrate the evolution of weight matrix dimensions across the multiple stages in the proposed scheme, take LLM-Pruner (Ma al., 2023) an example in (e.g.. LORAM-STRU) in Fig. 9, visualizing the transformation from Wo = W WA = W and W W under LORAM with structured pruning. For LORAN variants employing non-structured pruning, the parameter dimensionality remains unchanged during training due to the use of mask matrix. Therefore, these visualizations are omittted.
(a) W W
(b) W W
Original Full-Rank Rank Weight (+E u.u gare)) [+E (p.low)))
Original Low- Rank-Rank Matrix R. (+E (q, un, pitte))) A, down))
Pruned Low-Rank Matrix up.eatej)
A. fee pg..cc..ppaaar B. (-e (o. downl)
A. (+E gate)) B. (+e (o. dosm))
A. (+E (o, dawn))
31
|nx(1 p2
5--5: hidden dize
n=d Intermediate size
m=2
W
pruning
W
(c) W W
Pruned Full-Rank Weight w? (se sulk Canpiiee) w!" (+E (oodn))))
Trained Pruned Low Rank Matrix
Trained Recovered Low- Rank Matrix |+E (. Al (vE (u upp.pp tel)
af" (+E (a i.g. gateee Al (er (s,tnee))
A." [+E (q.k; ug.gaa)))
B." dawws))
(se do.doa)]].
B. (+E (o,dovn])
0
0
pil
[NXCI PI11
W
WR R
Figure 9: Dimensional evolution of the weight matrices: W. = W (a), WA = WP (b), and
W + W (c) during LORAM-STRU training. This includes updates for Wg. Wt. W., and up- W gate- and W down in the MLP layer.
Wo in the attention layer, well W
21

---

Published as a conference paper at ICLR 2025
D VISUALIZATION OF LOW-RANK MATRICES
In this section, utilize the L2-norm evaluate variations in low-rank matrices trained with different LORAM variants. This metric facilitates the visualization of captured features and allows for analysis of LORAM's effectiveness. Specifically, we examine the updated low-rank matrices in the self-attention and MLP layers of LLaMA-2-13B and L.aMA-2-70BB, trained with LOR variants on OpenHermes.
D.1 HEAD-WISE NORM OF ATTENTION
For low-rank matrices in the attention layer, denoted as WA where * {q. k, v, o}, we compute the L2 norms for each attention head. Let H" represent the number of heads. The L2 norms for each head h (where = 0,1..... H* 1) are defined as follows:
W W = W A [k,]]l2
if E kg
(10)
[ hill2
if ==
The results visualized through heatmaps in Figs. 10 and 12, effectively illustrating the distribu- tion of features captured by different attention heads.
D.2 LAYER-WISE NORM OF MIP
For the low-rank matrices in the MLP layers, denoted as Wa where A E {up. gate, down), we denote the number of layers as L. The average L2 norm for specific layer (where 0, 1,.,, L 1) is computed as follows, excluding elements equal to zero using a mask, ensuring that only active parameters contribute to the average:
E n EE
[. 2
1(W
# 0)
i gat)
W 2
(11)
I(W
[.
+ if down
Here, (I() denotes the indicator function, which returns only when the corresponding element is non-zero, effectively excluding zero elements from the average calculation. The average norms for the MLP layers are visualized in Figs. 11 and 13, clearly depicting the trends in updating amplitudes across the various projections.
D.3 ATTENTION UPDATE PATTERNS
Layer Update Patterns in LORAM and LoRA. Figs. 10 and 12 reveal that both LoRA and Lo- RAM display similar layer update behaviors. In any low-rank matrix Wa where {q, k, v, v,o), deeper colors predominantly concentrate in either shallow or deep layers, while middle layers re- ceive relatively few updates. This suggests that training primarily focuses optimizing the shallow layers to capture semantic information, with deeper layers refining this knowledge, rendering middle layers somewhat redundant.
More Uniform Projection Updates in LORAM. Figs. 10 and 12 further indicates that updates the LoRA-trained low-rank matrices, particularly for Way A', are relatively uniform, exhibiting substantial deep colors across multiple heads. In contrast, other matrices emphasize specific rows and heads. For instance, in the 70B model's WAY, only the heads the uppermost layers experience significant updates, while lower layers show minimal changes. This suggests that the unpruned model retains rich knowledge, requiring only minor adjustments to a few heads certain layers for task adaptation. Conversely, LORAM demonstrates more uniform distribution of deep colors across each low-rank matrix, indicating that the pruned model must effectively utilize every limited neuron capture knowledge, thereby enhancing downstream performance.
22

---

Published as a conference paper at ICLR 2025
(a) LoRA W
LoRA W
(a) LoRA W
(a) LoRA W
Layer Index
Layer Index
Layer Index
18
-44
466
-1
-1
84
SRA ARR
3.eeg: RAR 6RR
5x 000255855 RAX
oneed: SA2 5 RXR
Head Index
Head Index
Head Index
Head Index
(b) LoRAM-Rand W
(b) LoRAM-Rand W.
LoRAM-Rand W
LoRAM-Rand W
Layer Index
Sayer Index
12
Layer Index
Layer Index
11
-12
-22
GHERSSS AAR
-60
3ioogg RAR
-16
onengg RAR
-50
Head Index
Head Index
Head Index
Head Index
(c)
LoRAM-Stru w
(c) LoRAM-Stru. W
(c) 10LoRAM-Stro w
(c) LLRRA--Stru W
Layer Index
Layer Index
Layer Index
Layer
34oogd
oreedd
Head Index
Head Index
Head Index
Head Index
(d)
LoRAM-Semi w
(d) LoRAM-Semi w
(d) LoRAM-Semi W
(d) LoRAM-Semi W
Layer Index
Layer Index
Layer Index
34
Head Index
Head Index
Head Index
Head Index
(e) LoRAM-Unst W
(e) LoRAM-Unst
(e) LoRAM-Unst W
(e) LoRAM--Unst W
Layer Index
32
Layer Index
Layer Index
110
11
Layer
22 502 2. SPARA
megg ARA 2:4
25 RAR
444 RM2 ARA
Head Index
Head Index
Head Index
Head Index
Figure 10: Visualization of low-rank matrices in the attention layers of LLaMA-2-13B.
(a) w
(b) W
(c) w
Mean L.-Norm
Mean NorN
Mean Norm
2.05
0..5
LokAk Band LakAY
2.4
10
30
30
Layer Index
Layer Index
Layer Index
Figure 11: Average La norms of low-rank matrices in the ML. layers of LLaMA-2-70B.
23

---

Published as a conference paper at ICLR 2025
(a) LoRA W
LoRA W
(a) LoRA W
(a) LoRA
W.
Layer Index
30
Layer Index
Layer Index
36
-04
125
53
10
-1.
-se
C50
-35
S.50AN R 8 &
5. 10 9
Head Index
Head Index
Head Index
Head Index
(b) LoRAM-Rand W
(b) LoRAM-Rand W
(b) LoRAM-Rand W
(b) LoRAM-Rand W
Layer Index
1A
Layer Index
laver Index
-10
Layer
-06
-64
-02
-20
Head Index
Head Index
Head Index
Head Index
(c) LORAM--SrL Stru W
(c) LoRAM-Stru W
(c) LLRAM--Str. W
(c) MORAM-Stru
Layer Index
1aye Index
Layo Index
1344 Index
4.4
0.1
2000002
Head Index
Head Index
Head Index
Head Index
Figure 12: Visualization of low-rank matrices in the attention layers of LLaMA-2-70B.
Mean Ly-Nomm
(a) W
Mean La -orm
(b) W gen
Mean Norm
(c) WS....
0.14
Lotk
0.0
2.00
0.000
2.00
0.00
2.00
4
60
Layer Index
Layer Index
Layer Index
Figure 13: Average La norms low-rank matrices in the MLP layers of LLaMA-2-70B.
D.4 MLP UPDATE PATTERNS
LORAM Exhibits Greater Update Amplitude than LoRA. For both the 13B and 70B models, LORAM consistently exhibits greater update amplitude across each layer compared to LoRA, as shown in Figs. and 13. This increased amplitude indicates that LORAM is more effective in adjusting the weights in all layers, thus enhancing the adaptability and overall performance.
Distinet Update Trends in Layer Amplitudes. The amplitude changes reveal distinct pattern in Figs. 11 and 13: first decreasing, then increasing, and finally decreasing again. Shallow layers (0-3) and deeper layers (25-35 for the 13B model and 50-75 for the 70B model) undergo intensive updates. This behavior indicates that model prioritizes foundational feature extraction in shallow layers and the refinement complex representations in deeper layers. Such a strategic update dis- tribution optimizes the learning process, ensuring effective capture of basic and advanced features.
D.5 ANALYSIS OF UNCHANGED WEIGHTS
Here, try analyze the unchanged weights support the motivation of LoRAM.
24

---

Published as a conference paper at ICLR 2025
Fine-Grained Visualizations. As the above visualization, conducted detailed visualizations comparing the updated magnitudes of pruned and unpruned weights across layers. The results demonstrate that unpruned weights in both attention and MLP layers exhibit consistently smaller updates during fine-tuning as shown in Fig. 12, indicating their critical role in preserving the model's capacity for inference.
Theoretical Perspective. The phenomenon can be explained by the gradient- based importance of these weights, which prioritize parameters with minimal updates but high sensitivity during re- covery. These weights stabilize inference outputs, making them indispensable despite their limited fine-tuning updates.
Quantitative Evidence Our analysis reveals strong correlation between weight update magni- tudes and downstream performance. Pruning weights with smaller updates significantly degrades performance, highlighting their importance for inference and validating intuition.
Impact on Large Models such LLaMA-2-70B. where outperforms random pruning by substantial margin. Retaining critical parameters ensures effective task adaptation and generalization across diverse domains.
The selective pruning strategy shows notable benefits in larger models
25

---

Published as a conference paper at ICLR 2025
E PERFORMANCE OF SUB-TASKS IN CSR
We report the performance of six sub-tasks in CSR, with Figs. 14 and 15 showcasing the results for LoRAM-traiee LLaMA-2-13B and LLaMA-2-70B, respectively. Our findings indicate that various LORAM variants outperform core competitive benchmarks: for the 13B model, LOR AM surpasses both the untrained 13B and the LoRA-trained 7B, while for the 70B model, it exceeds the untrained 70B and the LoRA-traings 13B. This demonstrates that AM consistently achieves performance gains across models of different scales while effectively reducing memory usage. Fur- thermore, selective weight contributions in the 70B model significantly enhance performance, as evidenced by LORAM-STRES STRU'S marked improvement, particularly the challenging Are Challenge multi-choice question leverages weight differences, focusing on the most trainable weights compared to MORAM-RANN
question-answerinn answering task. This suggests that LORAM-STER effectively identifies and
(a) LLaMA-2-138 138 OpenHermes
(b) LLaMA 2-18B OpenOrca
Swal
Arc Easy
Swala
Arc Easy
OpenBookOA
0.30
Challenge
OpenBookQA
9.30
Challenge
LoRA
LoRA
13B w/o FT
13B w/o FT
136 LoRAM-Stru
135 LoRAM-Siru
136 LoRAM-Uns!
136 St
PIOA
WinoGrandd
PIOA
WinoGrande
Figure 14: Performance of six CSR sub-tasks on the trained LLaMA-2-13B using LORAM.
(a) LLaMA-2-708
OpenHermes
LLaMA-2-708/ OpenOrca
HellaSwag
Arc Easy
HellaSwag.
Arc Easy
0.80
OpenBookQA
0.40
Arc Challenge
OpenBookQA
0.40
Arc Challenge
13B LoRA
13B LoRA
70B wio
70B w/o
136 LoRAM-Rand
150 LoRAM-Rand
138 LoRAM-Stru
LoRAM-Stru
PIOA
WnnoGrande
PIQA
WinoGrande
Figure 15: Performance of six CSR sub-tasks on the trained LLaMA-2-700 2-B0B using LORAM.
26

---

Published as a conference paper at ICLR 2025
F ALGORITHM OF LORAM
Here, we present the complete algorithm of LORAM in Algorithm 1.
Algorithm LORAM (ilemor--ffficiee LoRA Training)
Require: original full-rank pre-trained weight Wo, alignment corpus Da, and flags F², F", F FR
1: Offline W. Process Stage:
2: if _F" then
3: W P(W.) Wo M
Pruned Full-Rank Weight Generation.
if 1 then
5: W < argmin (Dx;; W
Pruned Full-Rank Weight Alignment.
6:
if 2 then W P.
7:
Pruned Full-Rank Weight Quantization.
end if
else if F then
10:
W Q(W))
11: end if
12: else if 7 then
W (W))
Standard Quantization LoRA
14:
end if
15:
Record the processing result of Wo as W.. * P,Q Q). (P, A), (P, Q)).
16:
Online W Training Stage:
F" then
Pruned Low-Rank Matrix Generation.
19:
W B = P(Wa)= = WA M
BA M
20:
while TRAINING do
Pruned Low-Rank Matrix Training. Update low-rank matrix via objective Cafr with the forward pass h xW2 xW Return trained low-rank matrix W = "B" A=".
21:
22:
23:
end while
24:
if F then
Recovered Low-Rank Matrix Generation. Structured LORAM
25:
W
B A R(W2)= = W o (1-
26:
else
27:
W B A
A
> Non-structured LORAM
28:
end if
29:
else
30:
while TRAINING do
> Standard LoRA Training.
31:
Update low-rank matrix via objective LaFr with the forward pass h = xW6 xWA. Return trained low-rank matrix W = A"
32:
33: end while
34: end
35: Record the trained low-rank matrix as W * E (R*,*).
36:
37: Online Wa. W Inference Stage:
38: while INFERENCE
Recovered Low-Rank Matrix Inference. 39: Perform inference with the forward pass h XWWWxxt W = x( (w B A² ).
is do
40: end while
41: while INFERENCE with is x do
Standard LoRA Inference.
42: Perform Inference with the forward pass x(Wo W x B*A*).
43: end while
27

---

Published as a conference paper at ICLR 2025
G TUNING OF LEARNING RATE
We provide additional details on the learning rate tuning process for full LoRA applied to LLaMA- 2-7B and LLaMA-2-13B models, trained the OpenHermes dataset. These experiments in Fig. 16 demonstrate that a learning rate of le-3 consistently achieves the best perplexity across both in- domain and out-of-domain datasets, further validating the reliability of our comparison.
Alpaca/ / SFFT
Openltermes SFT
0.95
0.90
LLaMA-2:78 Full 16-4
Valication
LL Full LoRA 3e
Valliation
0.85
LL7 Full LoRA 3e-4
1.00
LLaMA-2-78
LLaMA 2 78 Full LORA 3e
0.80
LLaMA 2 7B Full LORA 3e
0.75
0.95
0.70
350
Update Steps
Updete Steps
Alpaca/ /
OpenHermas
1.05
0.85
Validation Loss
Validition
1.00
LLaMA-2-138
0.80
LLaMA-2-138
LLaMA-2-136 Full LoRA 3e-4
LLaMA-2-138 Full LORA 3e-4
0.95
LoRA
LLaMA-2-138
LLaMA-2-130 Full LoRA 3e-3
LLaMA-2-130 Full LORA 3e-3
0.90
0.03
Update Steps
Update Steps
Figure 16: Learning rate tuning for LLaMA-2-7B and LLaMA - 1-B on OpenHermes using LoRA.
H PERFORMANCE OF DOMAIN-SPECIIII TASK
To assess the effectiveness of LoRAM in Nomain-speciffe tasks, we conducted experiments on GSM8K (using the training set for tuning and the test set for evaluation), mathematical reason- ing benchmark known for sensitivity to sparsification. Specifically, we trained LLaMA-3.1-70B using QLoRAM under various configurations.
The results, summarized Table 7, highlight that LoRAM achieves excellent performance this domain-specific setting. Notably, LoRAM-basse models maintain high accuracy with substantial parameter reduction ratios, showcasing their robustness and efficiency domain-specific tasks. These findings emphasize LoRAM's broad applicability beyond general-purpose instruction fine- tuning.
Table 7: Evaluation of LoRAM on the GSM8K dataset for domain-specific fine-tuning. Results show accuracy (%) and parameter reduction ratios for different configurations.
LLaMA-3.1
GSM8K
Parameter Reduction Ratio
8B w/o Fine-Tuning
55.27
8.79x
8B LoRA (OpenHermes 400)
55,80
8.79x
70B w/o Fine-Tuning
75.28
1.00x
70B QLoRAM-Shi 400 (OpenHermes 400)
80.36
15,81x
70B QLoRAM -Sru 400 (GSM8K 100)
77.18
15.81x
70B QLoRAM-SHi 400 (GSM8K 200)
79.15
15.81x
70B LoRA (OpenHermes 400)
80.74
1.00x
28

---

Published as a conference paper at ICLR 2025
I ANALYSIS OF LORAM COST
Identifying the costs LoRAM is indeed important, which is why we report both the number of training tokens used during the alignment phase and the parameter reduction ratios in the low-rank training phase. Below, we clarify the two stages of LoRAM:
Offiin Knowledge Alignment Phase. The offline phase is task-agnostic and can conducted by the model publisher prior to deployment, making its cost negligible for end users. To quantify the offline cost, we measured the number training tokens (as in Xia et al. (2024)) rather than end- to-end latency, which can vary based on hardware configurations. As shown in Figure 5, LoRAM achieves sigmificant performance gains using only 13 million tokens, demonstrating the efficiency of the alignment phase.
Online Low-Rank Matrix Training Phase. For the online phase, the memory and latency costs are primarily determined by the size of the base model parameters, which dominate resource con- sumption during training. To avoid redundancy in reporting, we focused on parameter reduction ratios instead of absolute time or memory usage.
Comparative Metries for Online Training. Here, provide additional metries, including mem- ory and latency comparisons for the online training phase. We conducted experiments using a work- load of 1024 samples (batch size 128, micro-batch size 4, sequence length 512) randomly selected from OpenHermes. The results Table demonstrate that LoRAM with structured pruning ratio of 2.17x (13B 6B) achieves comparable peak memory, latency, and throughput 7B LoRA, with only minor trade-offs. These differences arise due to the larger layer count in 13B LORAM, introducing more non DonGEMM operations, slightly affecting latency and throughput.
These results underscore the advantages of LoRAM's design in achieving substantial resource effi- ciency without significant trade-off's in memory or latency.
Table 8: Comparison of peak memory (MiB), latency (s), and throughput (samples/s) during the online training phase for LoRAM and LoRA models. Results are based on a workload of 1024 samples (batch size 128, micro-batch size 4, sequence length 512).
LLaMA-2
#Model Params
Reduction Ratio
Memory
Latency
Throughput
7B LoRA
6.73B
1.93x
30,517
134.27
7.626
13B LoRA
13.02B
1.00x
51,661
206.07
4.969
13B LoRAM-Stru
6.01B
2.17x
29,799
147.86
6.925
29

---

Published as a conference paper at ICLR 2025
J ANALYSIS OF CHANGES IN PERFORMANCE TRENDS
We analyze performance at two stages: after fine-tuning but before recovery, and after both fine- tuning and recovery.
After Fine-Tuning but Before Recovery. At this stage, the results of LoRAM align with prior work (e.g., SparseGPT, Wanda, and L.MM-Pruner). Unstructured and semi-structured pruning con- sistently outperform structured pruning (see Fig. 6, solid lines). This trend holds true across both aligned and unaligned settings, with the performance order as follows: LORAM-SENT < LORAM-UNST < LORAM-STRU < LORAM-RAND The slight advantage of LORAM-SEMI over LORAM-URE UNST can be attributed to its smaller pruning ratio, which retains more parameters and mitigates performance degradation.
After Mine-Tuning and Recovery. Post-recovery results show that structured pruning outperforms unstructured pruning. This can explained by factors:
Preserved Structure for Recovery:
Structured pruning maintains the organization of the pruned weights into coherent structures (e.g.. rows and columns in MLP layers, attention heads attention layers), ensuring that activations after recovery are aligned with those of the original model. This alignment improves the recovery process.
Pruned Weight Quality:
The quality of pruned weights influences the recovery effective- ness. Structured pruning tends to remove less critical weights, leaving more recoverable parameters. In contrast, unstructured pruning can remove weights that are more difficult to recover, which negatively impacts performance post-recovery.
These results highlight the interplay between pruning and recovery, suggesting that structured prun- ing, despite initial performance disadvantages, facilitates more effective recovery.
30


Reviews:
[{"review_id": "UamgwXNOX1", "reviewer": "Reviewer_EtZ8", "summary": "The paper proposes LORAM, a memory-efficient training approach for Low-Rank Adaptation (LoRA) fine-tuning on large language models (LLMs). LORAM aims to reduce the memory burden during training by pruning the model parameters and then “recovering” them during inference. The technique includes an offline alignment process to minimize knowledge discrepancies between the pruned and original models, as well as integration with quantization to further optimize memory usage. Extensive experiments showcase LORAM’s performance across various tasks using models like LLaMA-2.", "questions": "1. Could the authors provide specific examples of how LORAM compares to standard LoRA without any pruning? Same goes for QLORAM vs QLORA?\n2. What influenced the choice of pruning ratios, and could different ratios impact the effectiveness of LORAM’s alignment step? Could the authors also clarify if any theoretical or empirical basis underlies the decision to use certain pruning ratios in LORAM? For instance, what drives the choice of an optimal parameter reduction ratio?\n3. Are there cases where LORAM underperforms compared to standard LoRA? This would provide insights into LORAM’s limitations. As it is, I find it hard to wrap my head around the possibilities of LORAM winning in all scenarios.\n4. How was the the alignment corpus chosen, and what effect does corpus size have on the performance of aligned models? \n5. How consistent are LORAM’s improvements across different tasks? Further analysis here would help illustrate LORAM’s potential.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- LORAM aims to make LLM fine-tuning feasible on lower-memory devices, addressing a relevant challenge for the NLP community.\n- The introduction of an alignment phase to handle pruning inconsistencies is conceptually innovative and could provide insights for future memory-efficient models.\n- The authors conduct a broad range of experiments, which helps to illustrate LORAM’s performance across multiple tasks.", "weaknesses": "- The paper contains frequent grammatical errors, inconsistent terminology, and formatting issues that detract from readability.\n- The main technical idea—pruning during training and recovering for inference—lacks fundamental novelty and could be seen as an incremental step on existing methods rather than a breakthrough.\n- The experimental setup would be more convincing with comparisons to simpler or standard pruning approaches, helping to illustrate LORAM’s distinct benefits more clearly.\n- Testing LORAM beyond LLaMA models would make the approach more broadly applicable and better showcase its robustness across architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K6R0TVYk82", "reviewer": "Reviewer_aWyt", "summary": "This paper proposes “LoRAM,” a novel memory-efficient LoRA fine-tuning process that significantly reduces required storage by pruning base weights—an approach largely overlooked in prior LoRA research. To mitigate the discrepancy introduced by pruning, the authors introduce a recovery step to ensure alignment at inference time, thereby improving accuracy.\n\nCompared to models with a similar effective parameter reduction ratio, LoRAM achieves a higher compression rate while demonstrating lower training/test loss and higher downstream accuracy on the LLaMA-2 family and LLaMA-3.1 models. Additionally, an ablation study shows that the proposed recovery and alignment steps contribute to improved training performance.", "questions": "- Was a comparable grid search conducted for hyper-parameter tuning (learning rate, epochs) for both baseline LoRA and LoRAM? Fine-tuning can be sensitive to settings, so the degree of granularity in hyper-parameter tuning might impact the final performance.\n- I am curious about whether this method would also yield effective results for domain-specific fine-tuning tasks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a novel approach to optimizing the heavy base weights in LoRA-based fine-tuning, an area not well-focused in existing LoRA studies. Moving beyond the widely studied quantization, the authors address the challenges of achieving high compression through pruning by introducing recovery and alignment steps, which help to offset pruning’s drawbacks and yield promising performance results. The method shows scalability potential, and the presentation and explanation are clear, making it easier to follow what might otherwise be a complex process. The experiments also appear to be thoroughly conducted.", "weaknesses": "- The paper lacks a discussion of the cost implications of the proposed method. Unlike standard one-stage LoRA fine-tuning, the multi-stage process in LoRAM may involve trade-offs in latency, but this is not adequately addressed. Given the focus on efficient training, a detailed comparison of memory and latency costs with baseline methods should be discussed.\n\n- While the main table’s comparisons by compression ratio are understandable, the results should also include fine-tuning outcomes on same LoRAM’s target LLM, as seen in Figure 8. Including the performance upper bound of fine-tuning the same model without memory reduction would allow readers to better assess the trade-off between memory compression and performance.\n\n- To elaborate on the need for comparison within the same model family: from the perspective of a model publisher, different sizes of LLMs are not just scaled versions but may be trained with distinct capabilities in mind, considering future usability (ex. LLaMA-3.1-8/70B vs. LLaMA-3.2-3B). Thus, comparing models with similar reduction ratios but different original capabilities may not be appropriate. For those who performing fine-tuning, it would be more informative to see how LoRAM compares to standard LoRA on the same model, as this would directly reflect the practical value of the method. (I’m open to counterarguments if there’s a rationale for this choice.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1EBfORdwMR", "reviewer": "Reviewer_asc9", "summary": "This paper introduces LORAM, a training method that refines a pruned model by adjusting the low-rank pruned matrices and then reintegrates dimensionally restored low-rank matrices with the original model for inference. This approach greatly reduces memory demands from model parameters during training and enhances performance by utilizing all original parameters during inference. Consequently, LORAM effectively improves performance on devices with limited memory capacity. Extensive experiments across diverse pruning techniques, model sizes, and task domains demonstrate LORAM's effectiveness.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and comprehensively analyzes various adaptations of the proposed method to accommodate different pruning techniques.\n2. This paper conducted thorough experiments across various pruning algorithms, models of different sizes, and tasks.", "weaknesses": "1. The novelty of this paper is limited as the proposed approach essentially combines existing pruning techniques with LoRA.\n2. The claim that LoRAM substantially reduces the number of trainable parameters compared to standard LoRA is somewhat misleading. In the case of unstructured or semi-unstructured pruned models, there is no actual reduction in trainable parameters, as noted in the paper. Meanwhile, the reduction in trainable parameters when fine-tuning structured pruned models is due to the smaller dimensions of these pruned models compared to the original model, rather than any change in the LoRA component, which remains the same as in standard LoRA.\n3. Could you explain why, after fine-tuning, structured pruned models outperform semi-structured and unstructured pruned models? In previous studies, like SparseGPT, Wanda, and LLM-pruned, unstructured pruning has consistently shown the least accuracy degradation. However, in your case, the results seem to be the opposite after finetuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0nttRrpLqp", "reviewer": "Reviewer_H1mX", "summary": "The paper proposes LoRAM, a memory-efficient training scheme that combines pruning, LoRA-based training on the pruned model and subsequent integration of the found solution back in the original model. This procedure yields significant memory reduction compared to LoRA-based fine-tuning, which in turn was already an improvement over the full-model fine-tuning.\n\nAuthors further observed that under aggressive pruning rates their method did not work well and identified that the main culprit is inconsistency between the pruned model used for training and the original model used for inference. To combat this, they train the pruned model on a relatively small corpus to achieve alignment, which is a one-shot offline process.\n\nThe effectiveness of LoRAM is evaluated across several model sizes, pruning algorithms, and tasks in different domains.", "questions": "* In Figure 3 & 4, any reason for instability for QLoRAM-Rand & QLoRAM-Stru for 70B model when evaluated on Alpaca?\n* Seems there is a typo in Table 3: 0.3415 should be 34.15", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper is fairly well structured and well written. Visualizations are quite helpful for quickly grasping the method.\n* LoRAM enables training a 70B model on a GPU with only 20GB, which is an impressive result.\n* The proposed method is complementary with other memory reduction techniques such as quantization and pruning strategies.", "weaknesses": "* Some comparisons in Tables 1-3 seem unfair: It would be nice to include a baseline that uses the same total training time/compute as LoRAM (including alignment & LoRA-based fine-tuning of the pruned model), but for full fine-tuning of the original model.\n* Aside from the trivial “w/o FT” baseline, it seems that LoRA is the only method LoRAM is compared against. It would be nice to include more baselines from the literature. For instance, would be interesting to compare LoRAM to the respective pruning method used to see the effect of subsequent LoRA training.\n* The paper did not report the training speed LoRAM compared to LoRA and full-model fine-tuning. Having a comparison of wall-clock time and/or throughput (e.g. tokens/second) for LoRAM vs LoRA and full fine-tuning across a few model sizes would give a more complete picture of the trade-offs.\n* It would greatly help to increase the reproducibility of this work by releasing the code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0cPZuYx5ct", "reviewer": "Reviewer_BSMa", "summary": "This work proposes a memory-efficient training scheme for LLMs, named LoRAM. Based on an intuition that some parameters remain unchanged during finetuning, but are important during inference, the authors train a pruned model by updating the pruned low-rank matrices and then uses recovered matrices for inference. In this way, the parameters need to be finetuned can be largely reduced while achieving similar accuracy compared with previous LoRA methods.", "questions": "-What is the training throughput of your method? Updating pruned parameters seems time-consuming generally.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-Memory efficient training is an important topic in LLM-related fields.\n\n-This paper proposes an easy but effective method to reduce the number of parameters which need to be finetuned.\n\n-The potential this sparsity-based method shows when combined with QLoRA seems a promising way for this field.\n\n-The paper is easy to follow, experiments presented are good.", "weaknesses": "-My major concern is that why cannot train small and infer small? Training small is convincing enough, but it sounds more reasonable to me that we train on the pruned model, and then also infer on that model. I think it is indispensable for this paper to show the certain benefits of inferring large over inferring small. Detailedly, experimental results (e.g. model accuracy or inference latency) are needed for the comparison of inferring small and inferring large, so that the inference design can be convincing.\n\n-This idea is based on the intuition that some parameters remain unchanged during finetuning, but are important during inference, although the proposed method seems to work, there is no evidence to show the phenomenon and no theoretical analysis to show the reason of the phenomenon. What is the property of those unchanged weights? This paper will be much better with more effort on this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)