EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction
Authors: Hsi-Che Lin
Yu-Chu Yu
Kai-Po Chang
Yu-Chiang Frank Wang
Venue: NeurIPS 2025 poster ()
Keywords: Memory-efficient fine-tuning
Low-rank approximation
Primary area: applications
TL;DR: EMLoC is a memory-efficient fine-tuning framework that uses a lightweight emulator and LoRA correction to enable fine-tuning within an inference-level memory budget.

Abstract: Open-source foundation models have seen rapid adoption and development, enabling powerful general-purpose capabilities across diverse domains. However, fine-tuning large foundation models for domain-specific or personalized tasks remains prohibitively expensive for most users due to the significant memory overhead beyond that of inference. We introduce EMLoC, an Emulator-based Memory-efficient fine-tuning framework with LoRA Correction, which enables model fine-tuning within the same memory budget required for inference. EMLoC constructs a task-specific light-weight emulator using activation-aware singular value decomposition (SVD) on a small downstream calibration set. Fine-tuning then is performed on this lightweight emulator via LoRA. To tackle the misalignment between the original model and the compressed emulator, we propose a novel compensation algorithm to correct the fine-tuned LoRA module, which thus can be merged into the original model for inference. EMLoC supports flexible compression ratios and standard training pipelines, making it adaptable to a wide range of applications. Extensive experiments demonstrate that EMLoC outperforms other baselines across multiple datasets and modalities. Moreover, without quantization, EMLoC enables fine-tuning of a 38B model, which originally required 95GB of memory, on a single 24GB consumer GPU—bringing efficient and practical model adaptation to individual users.

EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction
Hsi-Che Lin'
Yu-Chu Yu'
Kai-Po Chang!.2 Yu-Chiang Frank Wang!
GGaduate Institute of Communication Engineering, National Taiwan University 2NVIDIA
r13942079entu. .ww.::ttttuuuuuuu.....eee...e... frankwang@nvidia. com
Abstract
Open-source foundation models have seen rapid adoption and development, en- abling powerful general-purpose capabilities across diverse domains. However, fine-tuning large foundation models for domain-specific or personalized tasks remains prohibitively expensive for most users due the significant memory overhead beyond that of inference. We introduce EMLoC, an Emulator-based Memory-efficieen fine-tuning framework with LoRA Correction, which enables model fine-tuning within the same memory budget required for inference. EMLoC constructs a task-specific fight-weigh emulator using activation-ais aware singular value decomposition (SVD) on small downstream calibration set. Fine-tuning then performed on this lightweight emulator via LoRA. To tackle the misalign- ment between the original model and the compressed emulator, we propose novel compensation algorithm to correct the fine-tuned LoRA module, which thus can be merged into the original model for inference. EMLoC supports flexible compres- sion ratios and standard training pipelines, making it adaptable to a wide range applications. Extensive experiments demonstrate that EMLoC outperforms other baselines across multiple datasets and modalities. Moreover, without quantization, EMLoC enables fine-tuning of 38B model, which originally required 95GB of memory, on a single 24GB consumer GPU- bringing efficient and practical model adaptation individual users. Project Page: hhs-chee-lin, www.gggittti...ii.......
Introduction
cennera-puupoose foundation models have demonstrated impressive zero-shot capabilities across wide range of benchmarks [2] 16,88 8 II 38). For real-world deployment, such as domain-specine tasks [22]
46] or personalized user behavior 22 331 further customization by fine-tuning is still required. However, fine-tuning typically incurs significantly more memory overhead than inference [30].
Consequently, if users have fixed amount of available computing resources, they will forced to choose between two unfavorable options. First, they can use small model that fits within their memory budget for fine-tuning as shown Fig. aa), but this sacrifices the emergent capabilities 441 of larger models and underutilizes hardware during inference. Alternatively, they can opt for large model that fully utilizes resources during inference but exceeds memory limits fine-tuning as shown in Fig. [b), making user-specific adaptation infeasible and potentially limiting performance specialized applications. This paper addresses central research question: Is it possible to design a fine-tuning strategy such that users can fine-tune model under the same memory budget inference?
The memory cost of fine-tuning can be broadly attributed to three components: optimizer states, intermediate activations, and the model parameters themselves, as marked in Fig. [ with different colors. Initial efforts reduce the memory usage of fine-tuning concentrated on the first two components. The first component, optimizer states, stores auxiliary information such momentum and variance in the Adam optimizer [6 for each trainable parameter. This overhead can be
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Small Model
(b) Large Model
(c) Large Model + Ours
GPU Memory Usage (GB)
OOM
aplimizer state activation model parameters
Can only zero-shot
Max available
Resource unutilized
Inference
Fine-tune
Inference
Fine-tune
Inference
Fine-tune
Emergent ability
Special domain
Enable fine-tuning
Resource utilization
Personalization
Resource utilization
Figure 1: The dilemma caused by additional memory overhead during fine-tuning. (a) Users opt for smaller 8B model, sacrificing emergent capabilities and underutilizing available hardware. (b) Use of a larger 26B model requiring memory exceeding the hardware limit even with LoRA [14] and gradient checkpointing [4] techniques.
(c) Our EMLoC utilizes smaller model during fine-tuning, allowing the same budget for both training and inference.
mitigated using parameter-efficient fine-tuning (PEFT) methods [4 21 45), such as LoRA [1]] which reduces trainable parameters by only updating lightweight low-rank matrices. The second component, intermediate activations, refers to the values retained during the forward pass for use in backpropagation. Previous works reduced it by modifying model architectures [TO 20 37 or
gradient checkpointing [4) However, since the components reduced by both strategies still presented during fine-tuning, reducing them alone, without addressing the memory usage of model parameters. cannot fully close the memory gap between fine-tuning and inference (Fig. ())).
Few recent works have taken initial steps toward addressing the third component, the memory cost of model parameters. For example, Offsite-Tuning Tuning entirely during fine-tuning and to update only the top and bottom layers, thereby saving some memory occupied by model parameters. LORAM (49) introduces row-pruning as method to reduce model parameters during fine-tuning, allowing the updating of the unpruned rows in the intermediate layers. Despite their promise, these methods typically rely on memory-intensive full-model training. This entails strong and often impractical assumption that well-resourced institutions must be involved, making them unfeasible for individual users. In addition, they impose strict constraints on which subsets of weights can be updated, limiting their flexibility and general applicability across tasks and architectures. Morcover, they overlook the fact that different base models are used during fine-tuning and inference, introducing potential mismatches that degrade performance.
43 proposes to drop some of the intermediate layers
In this paper, we propose EMLoC, an Emulator-based Memory-efficient fine-tuning framework with LoRA Correction. EMLoC completely eliminates the memory cost gap between inference and training and can be carried out solely by individual users. The central idea is to perform fine- tuning a low-rank model which we call the emulator so that the memory taken up by model parameters can be reduced as depicted in Fig. Cc). To make this approach effective, we introduce emulator construction procedure based on activation-awe aware singular
an efficient, downstream-aww value decomposition (SVD), using a small subset of downstream data for calibration. This yiclds lightweight emulator tailored the downstream data, without requiring costly full-model training. Fine-tuning is then conducted entirely on the emulator without any assumptions about the fine-tuning process any standard training pipeline can be used. However, since the LoRA modules are fine- tuned on the emulator rather than the original model, naively merging the LoRA modules into the original model will suffer from the misalignment between the full model and the compressed emulator. To mitigate this issue, we further introduce novel LoRA correction algorithm, which adjusts the parameters to compensate such discrepancy during inference.
Our contributions can be summarized as follows:
We introduce pipeline that closes the memory cost gap between inference and fine-tuning, enabling users to fine-tune model under the same memory budge as inference.

---

We propose a novel amulator-basel fine-tuning framework that significantly reduces memory consumption by operating on a downstream aware low-rank approximation of the model.
We develop a LoRA correction algorithm that addresses the misalignment between emulator- based fine-tuning and full-model inference, improving final performance and transferability.
Related Works
2.1 Reducing Optimizer State
There are mainly two ways to reduce the memory required for optimizer states: by reducing the number of trainable parameters required per parameter 19
45 47 50], or by reducing auxiliary information A prominent example of the first approach is LoRA [14), which introduces trainable low-rank adapters into frozen pre-trained models to achieve parameter- efficient fine-tumin tuning. For the second approach, Adafactor [34] eliminates the need to store full-rank second- recond-moment moment estimators by using factored approximation. While these methods are effective, non-trivial number of trainable parameters are required to maintain acceptable performance. As result, the optimizer states associated with those parameters continue contribute a non-negligible portion of the overall memory footprint.
14
13
34
53
2.2 Reducing Intermediate Activation
Another way to fine-tune with less memory overhead is to reduce memory from intermediate ac- tivations. Two explored strategies are shortening the backpropagation path [27 37 45 48 and recomputing activations during the backward pass [4] I0 20). An example of the first strategy is Ladder Side-Tuning 37, where trainable modules are placed outside the backbone avoid back- propagation through the backbone. [20] carefully design the position and initialization of the adapter modules so that the pre-trained model becomes reversible, which enables recomputing intermediate activation from the output during backpropagation. However, these methods cannot completely eliminate activation memory due to the fundamental need for gradients, and they typically incur additional computational overhead.
2.3 Reducing Model Parameters
Beyond optimizing optimizer states and intermediate activations, very recently reducing the memory footprint of the model parameters during fine-tuning has been explored. It is worth noting that quantization-base methods 9 (2) are largely orthogonal to our focus: they do not reduce fine-tuning memory relative inference, and can therefore combined with our method. An approach to reducing parameter memory during fine-tuning involves using pruned model for training while maintaining a full model for inference [43]]]] 49 For example, LORAM [49 assumes that the model publisher provides row-pruned and continually pre-trained variant of the original model. Users fine-tune this reduced model and then transfer the learned LoRA modules back to the full model. However, their setting requires continual pre-training of the pruned model, making it inaccessible to individual users. Furthermore, only the unpruned weights can be fine-tuned, limiting flexibility in selecting which parameters to adapt. In contrast. our work introduces pipeline that can be performed by users individually and allows flexible selection of tunable parameters.
3 Method
Our proposed method, EMLoC, is memory-efficient fine-tuning framework that enables individual users fine-tune model with the same memory cost inference. An overview of EMLoC is shown in Fig. 2 EMLoC consists of three main stages. First, we construct a downstream matcaamawwae aware lightweight emulator by performing activation-ans aware singular value decomposition (SVD) [40] on small calibration set drawn from the downstream data. Second, we fine-tune the emulator with LoRA modules using any standard training pipeline. The reduced parameter size of the emulator is crucial for enabling memory-efficient fine-tuning. Finally, to address the misalignment issue between the original model and the emulator, we present novel LoRA correction algorithm to correct the learned LoRA modules, and transfer them back to the original model before inference.
3

---

Original Model (8B)
Emulator (2B)
Original Model M
Layer N
Parameters 4x
Layer N
Misalign
Layer N
Downstream
Wi
data Do
W
W
W;
Truc, SVD
Correct
W
Reducing Parameters
Mitigate Misalignment
Layer
Layer 1
Layer 1
Slage 1: Downstream-swwst Emulator Construction
Stage 2: Fine-luning
Stage 3: LoRA Correction
Inference
lightweight emulator. Stage 2: Fine-tune the emulator via LoRA, allowing reduced memory costs. Stage 3: Update the LoRA module to compensate the misalignment between the full model and emulator.
Figure Overview of EMLoC. Stage 1: Construct a downstream-awen
3.1 Memory-Efficient Fine-Tuning: Problem Formulation
Given a pre-trained model M and a downstream dataset Ddown. the goal is to fine-tune M with the same memory constraint as inference. We denote the collection of linear weights M as W [W:)- In EMLoC, we introduce an lightweight emulator, denoted as E, constructed by replacing each W; W with its counterpart W e wp. The subscript here is retained to emphasize the positional correspondence between W; and W in the architecture. For simpleness, when the context is clear. Fine-tuning is then performed on the emulator & using any standard training pipeline. During training, a set of LoRA modules A {A,] J learned on top of W resulting the fine-tuned model weights W + A. At inference time, the learned LoRA modules are transferred back the original model M,, with the corresponding weights W + A, which is used for inference. Note that conventional parameter-efffcien fine-tuning (PEFT) methods are special case of this framework, where we W and hence 3 M.
may omit the index i
3.2 Downstream-aware Emulator Construction and Fine-tuning
Downstream-aware Emulator Construction. The goal of emulator construction is to create a lightweight replacement & for the original model M to be used during fine-tuning. To ensure that EMLoC is memory-efficient, generalizable, and effective, the emulator must satisfy three key criteria. First, it should have fewer parameters than the original model reduce memory consumption during fine-tuning. Second, should support flexible placement of LoRA modules, enabling all weights to be fine-tunable. In other words, if LoRA module A can be used to fine-tune weight matrix W in the original model, then EMLoC must support placing and training A beside the corresponding W in the emulator, thereby enabling effective fine-tuning of W. Third, it should preserve knowledge relevant to the downstream task to ensure the fine-tuning process is effective. To meet all three criteria simultaneously, we propose constructing the emulator using activation-awa aware SVD, which naturally balances parameter reduction, flexibility, and task relevance.
Specifically, construct the set of emulator weights wy by applying the activation-aware SVD method proposed in SVD-LLM 440] to each weight matrix W E W
Wy = ( W Wv = SVD-LLM(W,,) W E W].
where n is the number of kept singular values, chosen to control the parameter count of the emulator. The emulator E is then constructed by replacing each W in M with its corresponding low-rank approximation W Although we denote each 3 as a single matrix, it stored in its factored form Wo Wy. which significantly reduces the memory footprint of model parameters, thus satisfying the first criterion. Importantly, each W retains the same position, and the input/output dimensions as W. differing only in the rank of the transformation As result, any LoRA module A that could be applied to W can equivalently be applied to W enabling all weights to be fine-tunable, the second criterion. Finally, SVD-LLM minimizes the output reconstruction error
| W T x ||r,
(2)
where X is the intermediate activation computed from calibration data Dc- This ensures that the emulator preserves task-relevant knowledge, making fine-tuning on it effective, thereby fulfilling the third criterion.

---

aclive input subspace
WE
W
Base Model
LoRA
Merged Output
A
VA
W
A
Diff.
Modify weights
+
A
A
W
W
Wa
Train-inference misalignment
LoRA correction
o: w/ correction
o: during training
to compensate
w/o correction
Figure 3: LoRA correction to compensate model misalignment. To alleviate the misalignment that arises from fine-tuning the lightweight emulator but running inference on the original model, LoRA parameters corrected via feature spaces between the emulator and the original model.
Fine-tuning, Owing to the specific design choices in the emulator, we can apply any standard PEFT fine-tune the significantly smaller emulator, achieving memory-efficient fine-tuning. In particular, if we aim to fine-tune weight matrix W; in the original model, we instead insert a LoRA module next W to fine-tune the emulator as conventional frameworks. This allows us to use any existing training pipeline without modification, making our method plug-and-play replacement for conventional LoRA fine-tuning, while further reducing memory usage.
techniques [1 45
3.3 LoRA Correction for Compression Misalignment
To fully utilize the capacity of the original model during inference, we aim to transfer the LoRA mod- ules fine-tuned on & back to the original model M. However, the emulator construction step that replaces W with its low-rank counterpart W inevitably intro- duces misalignment between these two models. As as result, the base model outputs seen by the LoRA modules different during fine-tuning and infer- ence. The difference is unexpected by the LoRA modules since they were optimized in the context of We To address this issue, we propose a novel LoRA correction algorithm and correct the misalignment explicitly, as shown in Fig.
Algorithm LoRA_ correction
Input: WA. Wp: LoRA weights;
W, W Linear weights from M and E;
X: An hyperparameter for regularization.
Output: W W Corrected LoRA.
1. Apply SVD to WA, obtaining U, E, VT
2. W < U. W ΣVTWB
3. A W (W -WE)
4. W < W
5. W W clamp( AA
return W W
Objective LoRA correction. The goal of our LoRA correction procedure is correct the LoRA module A fine-tuned with the emulator E, and derive new LoRA module A suitable for the original model M. In other word, for each input æ that activates the LoRA, the merged output should remain consistent between training and inference. Formally, we enforce the following condition:
x TW + = =TWF +A) VI VA.
where VA denotes the subspace on which A produces none none-zero zero outputs i.c. the region where is active. is worth noting that enforce Eq. 3 only within VA. that the correction has effeets exclusively when the LoRA module is active. This design prevents unintended side effects in parts of the model unrelated to the LoRA module when doing correction, thereby preserving the integrity of the original model's behavior.
Rewriting Eq.
obtain equivalent condition,
x T(AA - A) T (W- A. Va VA.
This formulation highlights the explicit correction objective. The desired A aims to compensate for the misalignment A,, which exists between outputs of W and W., This can achieved through offsetting the original LoRA module by the correction term A...
5

---

Mitigating compression misalignment. Although this idea is conceptually straightforward, two practical challenges arise. First, the condition Eq. 4 must hold for all inputs x VA. Second, there is no direct way to incorporate the correction term A, into the LoRA weights A = W.WB.
The first challenge is resolved by leveraging two facts: the condition in Eq. 4 is linear, and VA forms a subspace of the input feature space. As result, it suffices to enforce the condition only on basis of VA, denoted by BA
(5) Furthermore, a natural choice for 3x is the set of column vectors of WA. the input projection matrix of the original LoRA module. To address the second challenge, we introduce a preprocessing step that reparameterizes the LoRA module while preserving its functionality. Specifically, we perform SVD on the input projection WA. yielding WA = UVV We then redefine the LoRA factors as
x -AA- - A) = (W- (W- A. Vx E Ba.
W = U, W = WWB
such that the overall transformation remains unchanged A = WaWB W After this reparam- eterization, the columns of W denoted by {ai...., a, }. are orthonormal. This orthogonality makes the behavior of the LoRA module on each basis vector governed directly by the corresponding b,. i-th row of W
a. A = Wa' W/ = - = b;, where e, is the standard basis vector. other words, modifying the output of A on each basis vector a, reduces to adjusting the corresponding row b, in W
With the challenges addressed, now formally describe the LoRA correction algorithm, as presented in Algorithm] [ We begin by preprocessing the original LoRA module A = WaWB B to obtain W and W/ Next, we select the columns of W'1 as the basis BA in Eq. (3. The correction term A is then computed by measuring the output difference between the original weight W and the emulator weight W under this basis
(8) Each row of A represents the correction A.; corresponding a basis vector a; Finally, we adjust W to satisfy the condition in Eq. 5). compensating for the misalignment between W and W
A WT (W-W*)
W W', W = W clamp A, A).
where the clamp function limits the norm of each correction vector A., be no more than multiple A of the corresponding 6;. This hyperparameter À acts as a safeguard against excessive deviation, and we find improves robustness in practice. After applying Algorithm each LoRA module independently, the resulting corrected weights, A W&W n. is thus adapted to account for the misalignment introduced by training on the emulator and inferring on the original model.
4 Experiment
4.1 Experiment Setup
Datasets. In our main experiments, we evaluate EMLoC on seven visual question answering (VQA)
datasets. Four of these are widely used benchmarks: ChartQA 244 Doc (25), InfoVQA (26).
and TextVQA (36). To reflect realistic deployment scenarios that require domain adaptation, we also
include three datasets from specialized domains: PMC-VQA [51] (medical imaging), WebSRC [5]
(web page understanding), and WC-VQA additional analyses are primarily conducted on these three Nomain-speciff datasets. For language modeling, evaluate two challenging mathematical reasoning benchmarks: MathQA [I] and All reported metrics across datasets are based on accuracy.
4 (multicultural knowledge). Ablation studies and
GSM8K 7
Comparison. For the main experiments, we compare EMLoC against four baselines. Original: We directly fine-tune the original large model, InterVVL2.5-SD (6) with LoRA. This baseline incurs significantly higher memory cost compared to other methods and serves as an upper bound in performance. Small: We fine-tune smaller variants of InternVL (4B and 2B) with LoRA, representing the low-resource alternative illustrated in Fig. [a). Offsite: We follow the Offsite-tuninn [43 framework using the variant that omits full-model distillation to ensure fair comparison under the same compute constraints. UPop: A row-pruning-bant -based method 335 is used to construct the emulator, followed by LoRA transfer using the procedure described LORAM 49 Except for the Small bascline, all methods-inccludng including ours perfsorermmmmm inference using the original Intern VL2.5-8B model.

---

Table 1: Performance comparisons of finetuning approaches on VQA with different memory costs. With VL2.5
1.2.5-BB 8B as the original model, and 50% indicates finetuning is conducted on 4B models. Note that direct finetuning on Intern V1.2.-88. (i.e., rows in gray) exceeds the memory budgets of all other finetuning methods and thus is not feasible under the same constraints.
Ratio
Method
CharQQ
DocVQA
InfoVQA
TextVQA
PMC-VQA
WebSRC
WC-VQA
Original
92.2
52.9
87.4
53.4
InteraVL 4B
82.9
90.4
64.6
75.8
50.6
84.8
48.6
50%
Offsite 43
84.3
91.3
69.0
77.7
51.0
76.1
45.9
UPop 35
84.4
92.0
69.7
78.5
50.7
76.4
42.1
EMLoC
84.6
92.3
69.9
78.8
52.3
85.2
48.8
Original
84.5
92.2
69.8
79.6
52.9
87.4
53.4
Intern 2B
78.8
87.4
56.0
73.2
78.1
34.4
25%
Offsite 43
84.0
92.3
69.6
77.2
50.6
76.6
45.4
UPop 35
84.3
92.2
69.6
78.2
50.9
76.6
44.1
EM
84.2
92.3
70.0
79.0
51.6
79.6
46,2
Original
84.8
92.0
68.0
79.5
52.3
84.8
51.9
Intern VL 2B
78.2
85.7
53.5
71.7
44.2
77.4
33.0
25%+
Offsite 4
83.9
91.8
67.0
77.1
75.2
44.2
QLoRA
UPop
35
84.2
91.9
67.1
78.1
50.3
75.8
43.9
EMLoC
84.2
91.9
67.5
79.0
50.9
78.1
44.8
Implementation details. For EMLoC, we use LoRA with rank and train with 500 iterations. The learning rate is set to 10-5 with cosine annealing. The number of calibration data set to 64 and À for LoRA correction is 3. The other baselines use the same hyperparameters setting except we adjust the LoRA rank that the number of trainable parameters of all baselines are the same.
4.2 Main Results
We evaluate EMLoC using IntemmVL..55 as the base model under three realistic memory-constraine fine-tur -unintnngg scenarios. These settings simulate user environments with training memory budgets equivalent to: fine-tuning 4B model (denoted as 50% compression ratio in the table), fine-tuning 2B model (25%). and fine-tuning 2B model using QLoRA [9] quantization (25++QLoRA) As shown in Table I EMLoC consistently outperforms all baselines across both standard and domain-specific VQA benchmarks. The improvements are especially significant on specialized datasets, where adaptation to the target domain is essential. Notably, EMLoC achieves performance closest to upper bound set by full-model fine-tuning.
Main Results.
These results underscore the adaptability of our approach under varying memory constraints, main- taining strong performance even under the tightest budgets. contrast the common strategy of fine-tuning smaller models, EMLoC provides a more effective alternative by better leveraging the capacity the full model. In addition, comparisons with UPop and Offsite-tuning demonstrate the advantages of our downstream aware emulator construction. Unlike row www-pruning or layer- dropping approaches that restrict architectural flexibility, our method tailors the emulator downstream data without altering model structure, enabling more effective and generalizable adaptation. Finally, we note that EMLoC is orthogonal with existing techniques, such as gradient checkpointing and quantization [9]. In our experiments, EMLoC is used together with these techniques, yet it consistently outperforms other baselines. This demonstrates that EMLoC is compatible with existing memory reduction methods, and can be integrated with them to achieve further efficiency gains.
4.3 Analysis
Scalability on Model Size. We further evaluate EMLoC on larger-scale models, including the 26B and 38B variants of Intern VL2.5. For this experiment, we construct 5B-sized emulator, making the fine-tuning cost roughly equivalent training a model. We use LoRA rank for 26B model and rank 4 for the 38B model. As shown in Table EMLoC consistently improves upon the zero-shot performance of these large models, demonstrating its effectiveness even at greater model scales. Importantly, all fine-tuning
conducted with less than 24GB of GPU memory- substatiibb

---

Table Applying EMLoC 26B and 38B models. Note that 5B sized emulator is considered, and all ex- periments are conducted under 24GB memory budget.
Table Applying EMLoC to NLP tasks.
All settings follow LORAM 49 and
LLaMA2 13B is used as the full model.
Size
Method
PMC-VQA
WebSRC
WC-VQA
MATHQA
GSM8K
Zero-shot
49.9
77.1
51.0
FT LORAM-RAND
32.6
24.3
26B
EMLoC
52.5
80.9
52.6
33.8
27.2
Zero-shot
52.5
79.0
53.6
LORAM-STRU
33.8
24.6
38B
EMLoC
57.0
82.1
56.8
EMLoC
33.9
29.8
Table 4: Comparisons of emulator construction settings. The first row corresponding to LO-
RAM [49] setting and the last row corresponding to EMLoC setting. Note that, LORAM requires external data and additional continual pretraining for memory efficient finetuning. We validate that both pruning and SVD) be applied for setring with better performances various tasks using only small subset of downstream data.
Compression Method
Add. Data for compression
Require Add. Cont. Pre-train (GPU-hours)
Overhead (4)
PMC-VQA
WebSRC
WC-VQA
Yes No
Yes No
214
51.0
78.7
43.6
Row-pruning
1.7
51.1
76.6
44.1
Yes
Yes
230
51.3
74.6
42.9
SVD
No
No
0.3
51.6
79.6
46.2
lower than the 95GB typically required for directly fine-tuning the 38B model. This means that users capable of running inference can also perform fine-tuning using our method, without additional hardware requirements. These results underscore the practicality and scalability of EMLoC for eesuureeconstrrained users.
Comparison of Emulator Construction Settings. We compare two emulator construction settings. The first is our proposed downstream-ave aware construction, in which the emulator is derived directly from task-specific data without any additional training. The second follows the approach adopted in LORAM, where the emulator is first constructed using general-purpose data, followed by continual pretraining. For general data, we use Laion-COCC [88] a commonly used dataset for vision- language pretraining and perform continual pretraining using 8 V100 GPUs.
As shown in Table 4 when row pruming is used as the emulator construction method, both settings achieve comparable downstream performance. However, the continual pretraining stage required in LORAM incurs substantial computational overhead, making impractical for individual users. In contrast, when SVD is used for emulator construction, our downstream-awe -ware method not only eliminates the need for continual pretraining but also achieves superior performance while incurring the lowest construction overhead, demonstrating the efficiency and effectiveness of our downstream- aware SVD. Furthermore, the performance gap observed in the last two rows highlights the importance of using downstream data rather than general purpose data as the calibration set De. By employing activation-aware SVD based on downstream activations, the constructed emulator better preserves task-relevant knowledge that facilitates subsequent fine-tuning.
Robustness Across Modalities. To evaluate the generality of our approach beyond vision- vision-language tasks, we apply EMLoC to challenging NLP datasets, MATHQA and GSM8K. We follow the experimental setup of LORAM and use LLaMA 13B (39 as the base model. As shown in Table 3 method achieves comparable performance on MATHQA and outperforms LORAM GSM8K, demonstrating its effectiveness in language-only settings. These results confirm that EMLoC is limited to multimodal models and can be readily extended other modalities. Furthermore. our method retains its core ad advantages minimall memory overhead and no need for continual pretraining-making it more accessible users with limited computational resources.
Effectiveness of Transferring LoRA From Emulator to Original Model. To better understand the role of the emulator and effectiveness of our LoRA correction strategy, we compare model performance before and after transferring the fine-tuned LoRA modules from the emulator to the

---

Table Performance comparisons using finetuned emulators and EMLoC. Note with while both finetuned emulator and EMLoC share the same memory budget, the former does not fully utilize such budget during inference and thus not desirable.
Ratio
Method
ChariQA
Doc
InfoVQA
TexiVQA
PMC-VQA
WebSRC
WC-VQA
Emulator EMLoC
57.1
62.3
29.3
15.0
40.9
42.1
30.7
50%
84.6
92.3
69.9
78.8
52.3
85.2
48.8
Emulator
19.3
6.3
10.8
6.9
31.2
4.0
19.1
25%
EMLoC
84.2
92.3
70.0
79.0
51.6
79.6
46.2
25%+
Emulator 14.8
4.5
10.5
7.3
32.2
4.3
19.0
QLoRA
EMLoC
84.2
91.9
67.5
79.0
50,9
78.1
44.8
Table 6: Ablation study activation-aware SVD and LoRA correction.
Ratio
Activation-Awen SVD
LoRA Correction
PMC-VQL
WebSRC
WC-VQA
51.0
74.4
44.7
x
51.2
74.4
44.8
25%
51.5
79.0
45.8
51.6
79.6
46.2
51.8
83.7
47.9
50%
52.3
84.9
48.8
original model. As shown Table performance on the emulator itself is significantly lower the final model after LoRA transfer. This gap underscores the importance of using the full model for inference and justifies our design choice transferring LoRA back.
Interestingly, despite the poor performance of the emulator, the LoRA modules trained on it improve the original model's s downstream performance once transferred. This indicates that the fine- ennittning process successfully extracts domain-speciff knowledge, which is embedded in the LoRA weights and effectively transferred to the original model. These results highlight the effectiveness of our emulator-base based training.
Ablation Study. We conduct ablation experiments to evaluate the contribution of two key compo- nents in the EMLoC framework: activation aware SVD for downstream-awee aware emulator construction, and the LoRA correction algorithm for mitigating training-inference misalignment. As shown in Table 6 removing either component results a drop performance. This highlights the importance of preserving downstream-relevan knowledge during emulator construction, and the necessity of correcting for discrepancies introduced by training and inference on different models.
By further examining these results, observe that the effectiveness of LoRA correction seems to be closely related to the degree of misalignment between the emulator and the original model. When emulator is constructed without activation-awen of the original model, leading to larger misalignment during fine-tuning and consequently weaker correction effects. Conversely, under milder compression settings where misalignment is reduced, the impact LoRA A correction becomes more pronounced. These findings suggest that while our current of SVD-LLM 40 provides strong foundation for emulator construction, there remains room to explore more adaptive or task-aware compression strategies that could further enhance model alignment and correction effectiveness.
SVD, it fails to retain key downstream behaviors
Applying Diffusion Model Personalization To further assess the generalizability of EMLoC beyond text generation, we apply it to the task of personalizing large text-to- to-image diffusion model. Specifically, we use DreamBooth [3] a well-established personalization method combined with LoRA to fine-tune FLUX.1-dee [77. 12B state-of-the diffusion model. We conduct experiments on 8 subjects from the DreamBooth dataset, fine-tuning each for 500 steps without prior preservation.
For evaluation, we report standard metrics used in prior works: DINO 3 CLIP-I 29 and CLIP-T scores. Due the short fine-mniin duration this experiment, we adopt standard SVD for emulator construction instead of activation-aware SVD, which would be less cost-effective in this setting.

---

46.3
46.2
Accuracy
Table 7: Quantitative results of applying EMLoC to diffusion model. EMLoC achieves comparable, sometimes slightly lower, performance than direct fine- tuning while significantly reducing memory usage.
46.0
45.9
45.7
45.5
Method
Fine- Fine-tuning memory (GB)
DINO
CLIP-I
CLIP-T
80
Figure 4: Sensitivity analysis of À in the LoRA correction algorithm. We plot per- formance on WC-VQA under different A.
w/o EMLoC
35.1
0.652
0.851
0.306
EMLoC
22.9
0.615
0.831
0.321
Quantitative results are presented in Table 7 Note that normal fine-tuning requires over 35GB of memory, whereas EMLoC enables fine-tuning within a 24GB GPU budget. While EMLoC achieves slightly lower DINO and CLIP-I scores, it yiclds higher CLIP-T score-likely due to slower convergence. We also show qualitative examples in Fig. 6 These results demonstrate that EMLoC can be effectively applied to image generation tasks, reinforcing its potential plug-and-play replacement for conventional LoRA fine-tuning with reduced memory requirements.
Impact of Hyperparameter A. We investigate the effect of the hyperparameter A, which constrains the norm of the correction term in our LoRA correction algorithm. As shown in Fig. performance varies with different choices of X. In particular, when no constraint is applied (i.e., performance degrades noticeably. We observe that, in such cases, the correction term may have a significantly larger norm than the original LoRA output. We hypothesize that correction term with large norm will dominate and distort the LoRA modules, ultimately harming downstream performance. In contrast, applying an appropriate constraint on the correction magnitude helps preserve the learned knowledge in the LoRA modules while still compensating for model misalignment.
co),
Conclusion
We present EMLoC,, novel framework that makes fine-tuning large foundation models as memory- efficient as inference. By introducing a downstream- emulator constructed via activation- aware SVD and performing all training on this low-rank approximation, EMLoC enables individual users to fine-tune large models without exceeding the memory budget required for inference. Our LoRA correction algorithm further improve the performance by addressing the mismatch introduced by training and inference diffferent model bases. Extensive experiments across diverse datasets and modalities demonstrate that EMLoC not only outperforms other baselines but also scales to models as large as 38B parameters on commodity hardware. These results open up new opportunities for accessible and scalable model customization, empowering broader adoption of large language models in specialized, constreedn settings.
Acknowledgment This work is supported in part by the National Science and Technology Council via grant NSTC 006, and the financial supports from the Featured Area Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (1 11499900) We also thank the National Center for High-performe Computing (NCHC) for providing computational and storage resources.
NSTC 1 2-21---000-00--MM and NSTC 1 114-2640-E-00
10

---

References
[1] Aida Amini, Sandia Gabriel, Shanchuan Lin, Rik
Yejin Choi, and Hannanch Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- -aasod formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Pupers). pages 2357-2367. Minneapolis, Minnesola, June 2019. Association for Computational Linguistics. doi: 10. 1/N19-111
[2] Shuai Bai. Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv: 13923, 2025.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IFEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[4] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: (el.::6600066744 2016.
[5] Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A dataset for web-based structural reading comprchension. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 4173-4185, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties.
[6] 7he Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110.14100 2021.
[8] DeepSeek-AL Deepseek-v3 v3 technical report, 2024. URL httthtttssxxxi...... abs////ss//22// 19437
[9] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lls.. Advances in neural information processing systems, 16:10088--011 15, 2023.
[10] Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems, 30, 2017.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 24Xi:22407.2183. 2024.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenct classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026-1034, 2015.
[13] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attriyann and Sylvain Gelly. Parameter-effticee transfer learning for alp. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang. Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
[15] Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-cfficient multi-task fine- fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565-576, 2021.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA. USA, May 7-9, 2015, Conference Track Proceedingy. 2015. URL http://arxiv, http://axxxvvvv.o./// absa 1412
6980
[17] Black Forest Labs. Flux. httpps///gttuub.coo////ddfffffes
2024.
11

---

[18] LAION. Laion coco: 600m synthetic captions from laion2b-en. https://laion. aicblog..i/llo..
laion- COCO/
2022.
[19] Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. states. Advances in Neural Information Processing Systems, 36:15136-15171, 2023.
[20] Baohao Liao, Shaomu Tan, and Christof Monz. Make pre-trained model reversible: From parameter memory efficient fine-tuning. Advances in Neural Information Processing Systems, 36:15186-1522 15209, 2023.
[21] Shih- Yang Liu, Chien Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 2024.
[22] Ming Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal generative ai copilot for human pathology. Nature, 634480333:446-473, 2024.
[23] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang. Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration modulc. arXiv preprint arXiv:2311.05535 2023.
[24] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. htt.3//112222..niing...
[25] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200 2209, 2021.
[26] Minesh Mathew, Viraj Bagal, Rubên Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographiecça. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706, 2022.
[27] Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid, and Anurag Arnab. Time-memory-and parameter-efficient visual adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 5536-5545, 2024.
[28] Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jac Lee. Yo'Ilava: Your personalized language and vision assistant. In Advances in Neural Information Processing Systems, 2024.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR. 2021.
[30] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing. Networking. Storage and Analysis, pages 1-16. IEEE, 2020.
[31] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-22510. 2023.
[32] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. In European Conference on Computer Vision, pages 422-438. Springer, 2024.
[33] Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-IIm: A trainable agent for role-playing. In EMNLP, 2023.
[34] Noam Shazcer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604, PMLR, 2018
[35] Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang. Chun Yuan, and Jiaqi Wang. UPop: Unified and progressive pruning for compressing dision-languae transformers. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 31292-311 31311. PMLR, 2023.
12

---

[36] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pages 8317-8326, 2019.
[37] Yi-Lin Sung. Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35:12991-133 133005, 2022.
[38] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology- arXiv preprint arXiv:2403.0829) 2024.
[39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2.XX::3307.09288 2023.
[40] Xin Wang, Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware singular value decom- position for large language model compression. In International Conference on Learning Representations (ICLR), 2025.
[41] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus, Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022.
[42] Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Ncdjma Ousidhoum, Afifa Amriani, et al. Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines. arXiv preprint arXiv:2410. 12705, 2024.
[43] Guangxuan Xiao, Ji Lin, and Song Han. Offsite-tuning: Transfer learning without full model. arXiv preprint arXiv:2302.04433 2023.
[44] Dongshuo Yin. Yiran Yang. Zhechao Wang, Hongfeng Yu, Kaiwen Wei, and Xian Sun. 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 201 16-20126. June 2023.
[45] Dongshuo Yin, Xueting Han, Bin Li, Hao Feng, and Jing Bai. Parameter-efficiet is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1398-1406, 2024.
[46] Shenghin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and Zhongyu Wei. Disc-lawllm: Fine-tuning large language models for intelligent legal services, 2023.
[47] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parfmeter-efficien fine-tuning for transformer-he based masked language-models, arXiv preprint arXiv:2 10199, 2021.
[48] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28, 2020, Proceedings. Part III 16, pages 698 714. Springer, 2020.
[49] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Yang You, Guiming Xie, Xuejian Gong, and Kunlong Zhou. Train small, infer large: Memory- lora training for large language models. arXiv preprint Xiv:2200211353: 2025.
[50] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficiett fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
[51] Xiaoman Zhang, Chanyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc- vqu: Visual instruction tuning for medical visual question answering. arXiv preprint 205. 10415, 2023.
[52] Zhengxin Zhang, Dan Zhao, Xupeng Miao, Gabriele Oliaro, Qing Li, Yong Jiang, and Zhihao Jia. Quantized side tuning: Fast and memory-efficient tuning of quantized large language models. arXiv preprint 15X1:22401.07159, 2024.
[53] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-efficient efficient IIM training by gradient low-rank projection. In International Conference on Machine Learning, Vienna, Austria, 2024. PMLR. URL //wpvvaaaivvvv/va///// org/ abs /403. 03507
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflext the paper's contributions and scope about the memory efficient fine-tuninge.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

For each theoretical result, we provide the full set of assumptions and complete proof in the main text. Additional details and elaborations on the proofs can be found in the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the information necessary to reproduce the main experimental results is included in the main text and the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We plan to fully release the code with reproduction instructions before the camera-ready deadline. However, it is not publicly available the time of submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All the training and test details are specified in the main context and supple- mentary details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to limited computational resources, we were unable to run multiple trials report error bars or statistical significance. We acknowledge this as limitation and plan to include variance estimates in future work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on the computer resources needed is provided in main context and the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp/e//aarrrsssccppp/l//cciee
Answer: [Yes]
Justification: research conducted in this paper carefully follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in the supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We discuss the safeguards in the supplementary materials.
Guidelines: Our paper poses no such risks.
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets (c.g., datasets, code, and models) used in the paper are properly credited. Their licenses and terms of use are explicitly mentioned and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Our work does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hutt::://eerrpp.cc//offffrrreeee cc///oofeeeennes/2225//.... for what should or should not be described.
20

---

EMLoC: Emulator-based Memory-efficien Fine-tuning with LoRA Correction
Appendix
This supplementary document provides additional context, analysis, and results to support the main paper. We begin with discussion of the limitations of approach and potential directions for future work, followed by brief examination of the broader social impact and additional related works. We then present supplementary proofs and derivations justify components of method. Additional implementation details are included to facilitate reproducibility. Finally, we provide extended experimental results that further validate the effectiveness and versatility of EMLoC.
Limitation and Future Work
While our emulator construction method enables significant memory savings and offers flexibility in placing LoRA modules, it relies off-the-shelf SVD methods. These methods primarily designed to preserve the inference-time behavior of the model, such as maintaining output logits, rather than the fine-tuning dynamics. As result, the emulator may not fully capture the training characteristics of the original model, which could introduce suboptimalities during fine-tuning. This limitation is shared by all baselines. A promising direction for future work is develop emulator construction strategies explicitly tailored to preserve fine-tuning behavior, such as gradient directions parameter update trajectories. Such approach could more faithfully mimic the training dynamics of the original model, potentially narrowing the performance gap between EMLoC and directing LoRA fine-tuning the original model.
7 Social Impact
EMLoC lowers the memory barrier for fine-tuning large foundation models, broadening access for users with limited resources and promoting more inclusive participation in AI development. While this democratization has clear benefits, it may also increase the risk of misuse, such as facilitating the generation of harmful biased content. Additionally, although EMLoC reduces memory usage, widespread fine-tuning of large models may still contribute to environmental concerns. We encourage responsible use and transparency in deployment mitigate these risks.
Additional Related Work
Module arithmetic. high level, EMLoC shares some conceptual similarities with module arithmetic (2]. Both approaches involve modifying adjusting model weights. While there is conceptual similarity, our LoRA correction method serves a different purpose. Module arithmetic techniques typically aim to merge multiple modules to combine different capabilities. In contrast, our method focuses on addressing the discrepancy between the compressed emulator used during training and the full model used during inference, by compensating for the misalignment introduced by this difference to further improve the performance.
SVD-related structure. EMLoC may look similar with other SVD-related structure design [44 LoRand can be viewed variant of the Houlsby adapter [3] that uses an SVD-like formulation to parameterize the up- and down-projection matrices. Specifically, it represents adapter weights as
product of summation of two low-rank matrices and a kernel matrix (i.c., W PT KQ). By

---

adjusting the rank and sharing kernel matrices across layers. LoRand further reduces the number of trainable parameters. Similar the distinction between EMLoC and LoRA, LoRand focuses on reducing the size of trainable adapter modules, whereas EMLoC reduces the memory for the pretrained weights in the base model, which improves the efficiency way not considered by LoRand. These approaches are orthogonal and complementary.
Supplementary Proofs and Derivations
9.1 Verification of Corrected LoRA
In the main text, we proposed that the corrected LoRA module A = WA W can be constructed by
W WA: W W3 A.
(10)
where A WT (W- - We). We now verify that substituting A into the original model indeed satisfies the desired condition
Vr VA. Note that we can write any VA linear combination of the columns of W
xT (W + + = xJ (we + A)
(11)
3y g such that W Yr VA.
(12)
We now substitute this into the left-hand side of Eq. 11 and simplify:
x (W + A = (WA7)) (W + A7)
(WA7)) T (W W)WB))
(WA7) ((W WAW,, - WAA)
(WAr)) [(W +A WA ---W WT (W - -WE))
=>'(WA W + WWTTW (W (W-WP (W-W)))
(13)
=7'(WT W W+TH +WTTA- A- IWA (W- -WE))
=7'(WA (WWTWE W +WTAA
= =(WA)) T(we +A)
= =T(EE +A) Vo E VA.
which is exactly the right-hand side of Eq.
(T)).
9.2 Justification for Using SVD in LoRA Correction Preprocessing
In our LoRA correction algorithm, we introduce a preprocessing step where we apply SVD to the LoRA weights to obtain W and W It is important note that, in the subsequent derivation of the correction procedure, the only essential property W is that its columns form an orthonormal basis. Any other pair (W7, WT) that satisfies this orthonormality condition correction algorithm. In this section, we show that although SVD may appear to be an arbitrary and critically, the final corrected LoRA result remains the
also be used in the
choice, it is valid and convenient same for any orthonormal decomposition.
Formally, suppose we have
WA = W = WAWB.
(14)
where both W and W have orthonormal columns. Our goal is to show that the resulting corrected LoRA modules are identical:
W (Wr WxT (W- - W = W (WH - WX (W - -W*)).
(15)
Given Eq.
14 this reduces to prove that
WA WWT...
(16)
Since the columns both W and W are orthonormal and span the same subspace VA. W WTT and WX are identical-they both represent the projection onto VA. Therefore, the resulting corrected LoRA modules are equivalent, regardless of the specific orthonormal basis chosen. This justifies the use of SVD in our preprocessing step: while not strictly necessary, provides one convenient and consistent way to obtain the required orthonormal basis.

---

Table 8: Information of experiments for main results.
Table 9: Information of the experiments for 26B and 38B large models.
Base Model
LoRA Rank
Trainable Parameters
Ratio
Method
Parameters
Intern VL-26B
IntemVLL..5-38D
100%
Original
8.
18.9M
Original Parmaeters
25.5B
38.4B
VI.ernV.. 4B
3.7B
10
18.7M
Offisit
3.8B
64
18.9M
Emulator
50%
UPop
3.8B
18.7M
Parameters
5.0B
5.3B
EMLoC
3.7B
18.9M
LoRA Rank
Intern 2B
2.2B
18.7M
6
Offsite
2.2B
64
18.9M
25%
Trainable
UPop
2.3B
19
18.9M
27.1M
33.5M
EMLoC
2.3B
18.9M
Parameters
Table 10: Performance comparison between EML.C and zero-shot inference. Note that fine- tuning with EMLoC requires less memory than performing inference. This enables users who can run inference to also benefit from fine-tuning, gaining additional performance improvements without increased memory cost.
Quantization Method
CharQA DocVQA
InfoVQA
TextVQA
PMC-VQA WebSRC WC-VQA
Zero-shot
83.8
92.0
69.6
78.5
48.6
76.5
43.1
x
EMLoC 50%
84.6
92.3
69.9
78.8
52.3
85.2
48.8
EMLoC 25%
84.2
92.3
70.0
79.0
51.6
79.6
46.2
Zero-shot
83.7
91.3
67.1
78.6
48.1
67.4
42.0
EMLoC 25%
84.2
91.9
67.5
79.0
50.9
78.1
44.8
10
Additional Implementation Details
10.1
Usage of Existing Assets
In our experiments, make of publicly available datasets, models, and codebases in accordance with their respective licenses. Specifically, we use the InfoVQA and TextVQA datasets, which are released under the CC-BY license; the PMC-VQA and WC-VQA datasets, available under the CC BY-SA license; and the WebSRC and LAION-COCO datasets, released under the MIT license; and the ChartQA data set, release under the GPL-3.0 license. The model backbones and training code are based on Intern VL2.5, which is licensed under MIT. All assets were used strictly for academic, non-commercial research in accordance with their intended terms. We thank the respective authors and communities for making these valuable resources publicly available.
10.2 Implementation Details
Main Results. Detailed information, including the number of base model parameters, LoRA ranks, and the number of trainable parameters, for the experiments in main results (Table I can be found in Table Note that we use different LoRA ranks for each baseline to ensure that the number of trainable parameters is approximately the same across methods.
Large Models. Details of the experiments involving the 26B and 38B models (Table 2 are provided Table Note that for these experiments, we report performance on a subset of the test set due limited computational resources, which made full-set inference infeasible within reasonable time.
11 Additional Results
11.1 Zero-shot Results
In Table TO we report the zero-shot performance the backbone models used in our experiments. As expected, fine-tuning with EMLoC leads to consistent improvements over the zero-shot baselines.

---

Table 11: Computational resource usage and performance comparisons of finetuning approaches. Ratios follow the definitions in Table I Note that EMLoC achieves stronger performance than other baselines while maintaining comparable computation resource.
Fine-tuning memory (GB)
Inference
Throughput (sample/sec)
Ratio
Method
TFLOPs
PMC
WebSRC
WC-VQA
memory (GB)
100%
Original
22.3
16.1
37.4
0.09
52.9
87.4
53.4
Intern 4B
15.4
8.1
18.9
0.17
50.6
84.8
48.6
Offsite
13.8
16.1
16.1
0.20
51.0
76.1
45.9
50%
UPop
14.0
16.5
0.16
50.7
76.4
42.1
EMLoC
14.2
16.1
17.8
0.16
52.3
85.2
48.8
InternVL 2B
10.9
4.9
12,9
0.29
44.6
78.1
34.4
Offsite
10.7
16.1
9.3
0.33
50.6
76.6
45.4
25%
UPop
11.3
16.1
10.3
0.22
50.9
76.6
44.1
EMLoC
11.5
16.1
11.3
0.21
51.6
79.6
46.2
Table 12: Overall time overhead analysis. We compare the time of EMLoC in each stage with LoRA and QLoRA. A. Note that due to our design choice of training free emulator construction and LoRA correction stage, the time overhead introduced by these additional stage is negligible.
Size
Method
Construction (hr)
Fine-tuning (hr)
Correction
#GPU
Overall wall time (hr)
LoRA
11.6
0
11.6
8B
QLoRA
0
12.1
0
1
12.1
EMLoC
0.3
4.7
20 (sec.)
1
5.0
LoRA
0
15.8
0
4
15.8 (estimated)
38B
QLoRA
19.5
0
19.5
EMLoC
1.2
14.2
(sec.)
While surpassing zero-shot performance is not surprising, important to highlight that EMLoC achieves these gains under the same memory requirements inference. This means that users who are able to run inference on these models can also benefit from task-specific fine-tuning- without needing additional memory resources-making model adaptation more accessible in practice.
11.2 Computation Resource
We report the computational resource usage of our main experiments in Table [] including memory consumption during both fine-tuning and inference. Additionally, we present the FLOPs per forward pass and the throughput, measured as the time taken for full forward and backward pass on single training sample. All metrics are evaluated on the same machine with a sequence length of 2048 to ensure fair comparison. Notably, Offsite-tunii fffsite-tuning exhibits slightly higher throughput due to its emulator using significantly fewer layers than other baselines. Despite this, EMLoC achieves superior fine-tuning performance while maintaining comparable computational resource, demonstrating its efficiency and effectiveness.
For a more comprehensive analysis, we include additional wall-clock runtime comparisons Table T2 covering the full EMLoC pipeline including emulator construction, fine-tuning, and LoRA correction- against both LoRA and QLoRA fine-tuning. From these results, we observe that EMLoC achieves lower overall runtime compared the baselines, even when accounting for the overhead of emulator construction and correction. This efficiency arises from the inrivation-aaan SVD construction, which is fast and avoids expensive continual pretraining, as well from the lightweight emulator, which improves throughput during fine-tuning. Moreover, the correction step incurs negligible computational cost relative to other stages while providing additional performance gains. These findings reinforce the practical impact of EMLoC: it not only reduces memory consumption but also remains competitive or superior in wall-clock efficiency, making it well-suited for real-world, resource-constrained deployment scenarios.

---

Table 13: Comparison between EMLoC and QLoRA. Quantization- based methods offer limited
flexibility in memory reduction and can degrade inference performance, while EMLoC enables fine-tuning on consumer-grade 24GB GPUs without affecting the base model's 's inference quality.
Intern VL2.5-26B
Intern VL2.5-38B
Method
Fine- Tine-tuning memory (GB)
WC VQA
Fine-tuning memory (GB)
WC VQA
Zero-shot
53.6
51
LoRA
72.9
94.8
QLoRA
38.1
39.0
43.4
43.6
EMLoC
18.6
56.8
20.1
52.6
Table 14: Compare EMLoC with stronger small model.
Model
Method
Sine-tuning memory
PMC-VQA
WebSRC (val set)
WC-VQA
Intern VI. 2.5 2B
LoRA
10.9 (GB)
44.6
78.2
34.4
Intern VI. 3 2B
LoRA
10.9 (GB)
48.4
79.5
44.8
InternVL 2.5 8B
EMLoC
11.5 (GB)
51.6
80.8
46.2
11.3
Comparison with QLoRA
Although quantization- based methods like QLoRA are orthogonal to our focus focus-they do not re- duce fine-tuning memory relative inference, include comparison to highlight the potential advantages of our emulator-based approach. Results are presented in Table [3] Memory usage is measured in the same way Table [1] with the exception that methods other than EMLoC are across multiple GPUs without distributed data parallelism (DDP), and model weights are split across devices. Due limited computing resources, we omit standard LoRA fine-tuning results for these large models.
By design, quantization methods offer limited flexibility and fixed upper bound memory savings. As shown in Table 13 even with 4-bit quantization, QLoRA cannot support fine-tuning of Intern Intern VL2.5-262 or Intern L2.5-38B
24GB consumer GPU. In contrast, EMLoC enables flexible memory reduction via emulator construction, allowing fine-tuning degrade the original model's performance-even during inference. In our experiments, quantizing V1.2.5 and 38B significantly reduced accuracy, the extent that QLoRA underperformed compared the half- precision zero-shot baseline. These results underscore the practical benefits of emulator-basee fine-tuning, offering greater memory flexibility without compromising inference performance.
Furthermore, while quantization performs well in most cases, too-tuning under memory budgets.
11.4
Using Stronger Small Model
To provide further insight, we conducted an additional experiment using Intern VI.3-BB (the latest version of the Intern VL2.5 models used in main paper) as a stand-in for compact model baseline. The results in Table 14 show that EMLoC applied to Intern VL2.5-8B with a 25% compression ratio still outperforms direct fine-tuning of Inter VL3-2B, This suggests that EMLoC can retain the performance benefits of larger models even under tight memory constraints, offering a compelling alternative to using smaller models.
11.5 Applying EMLoC to Houlsby Adapter
Note that EMLoC's first two stages are fully compatible with other adapters, still enabling memory- efficient training. To demonstrate this, conducted an additional experiment using Houlsby Adapters [33 in place of LoRA. As shown in Table 15] EMLoC still performs better baseline methods in Table highlighting its flexibility beyond LORA. Furthermore, we observe that the LoRA- based variant benefits noticeably from the correction algorithm, achieving higher performance than the importance the correction stage enhancing LoRA's
Houlsby counterpart. This demonstrates effectiveness within the EMLoCC framework. However, the current LoRA correction algorithm is indeed tailored to the linearity of LoRA modules, and we believe there no straightforward extension to other PEFT methods.
5

---

Table 15: Combining EMLoC with Houlsby Adapter. Note that the first stage of EMLoC is compatible with other PEFT methods to achieve significant memory reduction.
Method
PMC-VQA WebSRC (val set)
WC-VQA
EMLoC w/o correction
51.5
79.2
45.8
EMLoC w/ correction
51.6
80.8
46.2
Houlsby Adapter
51.2
79.0
44.5
11.6 Ablation Study the Number of Calibration Data
EMLoC relies on a small calibration set De to perform activation-aware SVD during emula- tor construction. In our main experiments, we
46.2
46.0
64 samples the default size for Dc. To
Accuracy
45.6
evaluate the sensitivity to this choice, we vary
45.4
the number of calibration examples and report
45.2
the results in Fig. 5 As shown, model perfor- mance plateaus after using 32-4 64 calibration samples, indicating that only a small amount of susk-specific data sufficient to construct an ef- fective emulator. This highlights the practicality of EMLoC low-resource scenarios.
45.0
64
number of calibration data
Figure 5: We plot performance on WC-VQA under different number of calibration data.

---

with a city in background
a blue house in background
wearing a santa hat
Reference
in a chef outfit
purple
on top of a wooden floor
wearing pink glasses
in a firefighter outfit
in a purple wizard outfit
Reference
in the snow
on top pink fabric
on top of purple rug a forest
wearing a yellow shirt
wearing a top hat and monocle
Reference
jungle
Eiffel Tower in background
a wheat field In background
floating on top of water
floating in an ocean of milk
Reference
the beach
on a cobble- stone street
mountain background
on top of white rug
Reference
red
cube shaped
Figure 6: Qualitative results of applying EMLoC to diffusion model personalization. DreamBooth with LoRA used to personalize the 12B FLUX. 1-dev diffusion model, illustrating that EMLoC can be effectively extended to generative tasks beyond text.


Reviews:
[{"review_id": "wguFyCNiCa", "reviewer": "Reviewer_5hvp", "summary": "This paper proposes EMLoC, a novel emulator-based fine-tuning framework. EMLoC constructs an emulator by applying low-rank compression to the model parameters. Fine-tuning is then performed on the emulator using LoRA, and the resulting LoRA weights are corrected before being applied back to the original model for inference. This approach significantly reduces both the training time and memory footprint.", "questions": "See above", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths \n+ The paper presents a clear motivation and a well-structured narrative. The proposed method is explained clearly and thoroughly.\n+ The proposed method is practically meaningful and facilitates model fine-tuning under resource-constrained settings.\n+ The method is novel, and the proposed correction technique offers valuable insights.\n\nWeaknesses\n+ Since QLoRA also reduces training costs, would it be possible to provide a direct comparison between EMLoC and QLoRA—both in terms of accuracy and memory consumption—to better demonstrate the advantages of EMLoC?\n+ As a suggestion, I believe that EMLoC shares some conceptual similarities with module arithmetic, and it may be beneficial to include a discussion of this connection in the related work or appendix.\n\n[1] Ziplora: Any subject in any style by effectively merging loras\n[2] Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning\n[3] Lcm-lora: A universal stable-diffusion acceleration module", "was_revised": true, "final_justification": "The author resolved my issue, and I will keep my rating.", "rebuttal": "**Provide a direct comparison between EMLoC and QLoRA in terms of accuracy and memory consumption, better demonstrating the advantages of EMLoC. (W1)**\n\nThanks for the suggestion. In fact, we have included comparisons and discussion with QLoRA, using InternVL2.5 26B and 38B models in Table 11 of the Supplementary Material, where we observe that even with 4-bit quantization, QLoRA still requires 43GB of memory—while EMLoC can fine-tune the 38B model using only 20GB and achieves up to a 9% accuracy gain. Compared to QLoRA, EMLoC offers two main advantages:\n- Greater flexibility in controlling the compression ratio, rather than being constrained by fixed bit-widths in quantization-based method;\n- No degradation in inference performance, since quantization affects the model even during inference, whereas EMLoC retains the original model parameters for deployment\n---\n**EMLoC shares some conceptual similarities with module arithmetic. It may be beneficial to include discussions of this connection in related work or appendix. (W2)**\n\nWe thank the reviewer for pointing out the potential connection to module arithmetic. We agree that, at a high level, both approaches involve modifying or adjusting model weights. While there is a conceptual similarity, our LoRA correction method serves a different purpose. Module arithmetic techniques typically aim to merge multiple modules to combine different capabilities. In contrast, our method focuses on addressing the discrepancy between the compressed emulator used during training and the full model used during inference, by compensating for the misalignment introduced by this difference to further improve the performance. We will include the above discussions into the related work session."}, {"review_id": "bLmK08CzIX", "reviewer": "Reviewer_ZRd4", "summary": "This paper introduces a novel memory-efficient parameter-efficient fine-tuning method, with experimental results suggesting that it can effectively reduce GPU memory requirements during model fine-tuning. The authors claim that, without quantization, their method enables fine-tuning a 38B model on a consumer-grade 24GB GPU.", "questions": "See above", "limitations": "See above", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:​​\n\n(1) The figures are well-organized, and Figure 3 appears particularly interesting.\n\n(2) The experimental results are impressive.\n\n(3) The motivation illustrated in Figure 1 is easy to follow.\n\n​​Weaknesses:​​\n\n(1) The writing in Section 3.1 is confusing. The section title defines a memory-efficient paradigm, yet the content seems to be a formalized description of the LoRA training framework. I did not find any discussion on memory efficiency in this section.\n\n(2) I do not fully understand the role of the \"emulator\" in this paper or how it reduces training memory usage. How does it differ from LoRA? Could the authors provide a more detailed explanation?\n\n(3) In Section 3.2, the authors list three necessary conditions for memory efficiency: (a) fewer parameters, (b) flexibility and tunability, and (c) retention of downstream task knowledge. These are inherently characteristics of LoRA itself. I am particularly curious about how these three points contribute to memory efficiency, especially regarding activation/optimizer states as depicted in Figure 1.\n\n(4) The SVD-based design has already been explored in LoRand. The authors should discuss the differences between their method and LoRand in the related work section.\n\n(5) The authors provide extensive textual descriptions of LoRA modifications, and Figure 3 looks intriguing. However, the results in Table 6 suggest that the impact of LoRA modifications is marginal. Do the authors have further explanations for this observation?\n\n(6) The results in Table 4 are remarkable. However, after reading the paper, I still do not grasp why the proposed method significantly reduces training costs. Could the authors explain this in simpler terms? Additionally, refining the writing could help reduce the readers' comprehension burden.\n\n(7) E3VA [1] (from MSRA) is a representative work in memory-efficient fine-tuning. Have the authors attempted to compare training costs with E3VA?\n\n[1] Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions, ACMMM'24.", "was_revised": true, "final_justification": "The author partially resolved my issue, and I will increase my rating.", "rebuttal": "**What is the difference and relation between EMLoC and previous works like LoRA and E3VA? (W1, W7)**\n\nTo clarify, as explained in Lines 32–40, the total memory cost of fine-tuning includes three main components:\n- Optimizer states — auxiliary information such as momentum and variance in the Adam optimizer for each trainable parameter;\n- Intermediate activations — values retained during the forward pass for use in backpropagation;\n- Model parameters — the full base model itself for forward and backward passes.\n\nLoRA reduces memory usage by lowering the number of trainable parameters, thereby decreasing the optimizer states. E3VA reduces activation memory by avoiding backpropagation through the backbone via architectural changes to module placement. In contrast, EMLoC addresses the often-overlooked bottleneck—the memory required to load and store the overall model parameters—by constructing a lightweight emulator to replace the original model during fine-tuning. Thus, EMLoC targets a fundamentally different source of memory consumption compared to LoRA and E3VA. However, as shown in Table 1, EMLoC combined with QLoRA and gradient checkpointing still consistently outperforms baselines such as row pruning proposed by LoRAM and layer dropping proposed by Offsite-Tuning. These results demonstrate EMLoC’s compatibility with other techniques and effectiveness in reducing model weight memory\n\n---\n**What is the role of the \"emulator\" in EMLoC? How does the emulator-based framework reduce training memory usage? What is the difference between emulator-based framework and the standard LoRA framework? (W1, W2, W3)**\n\nTo reduce the memory usage of model parameters, EMLoC introduces a lightweight surrogate model (i.e., the emulator), to replace the original model during standard LoRA fine-tuning, as described in Section 3.1. This emulator is constructed via activation-aware SVD using a small calibration set with the following three criteria (discussed in Section 3.2):\n\n1. Fewer parameters: This enables memory-efficient fine-tuning since EMLoC no longer needs to load full original model parameters during fine-tuning\n2. Flexible placement of LoRA modules: This makes EMLoC fully compatible with standard LoRA workflows and serves as a plug-and-play replacement.\n3. Retention of downstream task knowledge: This makes the emulator behave closer to the original model during fine-tuning, alleviating the model gap between the emulator and the full model.\n\nThis design differs from standard LoRA, which only reduces the number of trainable parameters but still requires loading the full model into memory. In contrast, EMLoC reduces the memory footprint of the entire model during training.\n\n---\n**The results in Table 4 are remarkable, but it is not clear why the proposed method significantly reduces training costs. (W6)**\n\nThe significant reduction in emulator construction overhead shown in Table 4 comes from our use of activation-aware SVD on a small set of downstream data. Compared to prior methods like LoRAM which relies on continual pretraining over large general corpora and causes substantial overhead and potential downstream-task misalignment, our EMLoC is a lightweight and efficient approach. As a result, our method not only avoids this costly stage, but also produces an emulator more tailored to the target task, leading to both lower construction cost (0.3 vs. 214 GPU-hours) and better downstream performance (up to 2.6% difference in accuracy).\n\n---\n**Missing discussion on a SVD-based design of LoRand. (W4)**\n\nUnfortunately, we were unable to locate a method called “LoRand,” but we believe the reviewer may be referring to RandLoRA [1]. While RandLoRA also utilizes SVD, its goal is fundamentally different from ours. RandLoRA draws inspiration from SVD to reformulate the LoRA module as a summation of multiple bases in order to improve expressiveness. In contrast, our use of SVD is applied to the base model itself, with the objective of compressing model parameters to enable memory-efficient fine-tuning. We will clarify this distinction and cite RandLoRA in the related work section.\n\n[1] Albert, Paul, et al. \"RandLoRA: Full-rank parameter-efficient fine-tuning of large models.\" arXiv preprint arXiv:2502.00987 (2025).\n\n---\n**Table 6 seems to suggest that the impact of LoRA modifications is marginal. (W5)**\n\nWe thank the reviewer for this observation. The marginal improvement is due to our aggressive compression setting, which may lead to relatively large misalignment between the emulator and the original model.\n\nTo better understand the results in Table 6, we conducted an additional ablation experiment using a less aggressive 50% compression ratio during emulator construction. The results support our hypothesis that the effectiveness of LoRA correction is related to the degree of misalignment between the emulator and the original model. When the emulator is constructed without activation-aware SVD, it fails to preserve key downstream behavior of the original model, possibly resulting in greater misalignment during fine-tuning and, consequently, weaker correction effects. Conversely, with less severe compression, the misalignment is reduced, and we observe that the impact of LoRA correction becomes more pronounced.\n\nThese findings suggest that while our current use of SVD-LLM provides a strong baseline for emulator construction, there is room for future exploration of more adaptive or task-aware compression strategies to further enhance alignment and correction effectiveness.\n\n| Ratio | Activation aware SVD | LoRA Correction | PMC-VQA | WebSRC | WC-VQA |\n|:-------:|:----------------------:|:-----------------:|:---------:|:--------:|:--------:|\n| 25%   | x                    | x               | 51.0    | 74.4   | 44.7   |\n| 25%   | x                    | v               | 51.2    | 74.4   | 44.8   |\n| 25%   | v                    | x               | 51.5    | 79.0   | 45.8   |\n| 25%   | v                    | v               | 51.6    | 79.6   | 46.2   |\n| 50%   | v                    | x               | 51.8    | 83.7   | 47.9   |\n| 50%   | v                    | v               | 52.3    | 84.9   | 48.8   |"}, {"review_id": "MEGwL2WAQs", "reviewer": "Reviewer_VfBn", "summary": "EMLoC introduces a three-stage pipeline for fine-tuning large models under the same memory budget as inference. First, it constructs a lightweight task-specific emulator by applying activation-aware SVD to the original model's weight matrices using a small calibration dataset. This yields a low-rank model that retains the same layer shapes but has far fewer parameters. Second, EMLoC fine-tunes only this emulator with standard LoRA adapters using any training pipeline. Finally, to deploy the full model, a novel LoRA correction algorithm adjusts the learned adapters to compensate for differences between the emulator and the original model. Inference then runs on the full model with the corrected LoRA modules. Experiments across vision–language and NLP tasks show that EMLoC matches or exceeds baselines despite having much tighter memory, and EMLoC achieves accuracy close to that of full-model tuning.", "questions": "1.\tThe method uses only 64 calibration examples for SVD. How sensitive are the results to this choice? Have the authors tested larger or differently distributed calibration sets? (Fig.4 shows slight $\\lambda$ sensitivity, but not data-scale sensitivity.)\n\n2.\tDid the authors compare against the recent LoRAM method or evaluate quantization-based fine-tuning? How would EMLoC perform relative to these approaches, and what are the trade-offs?\n\n3.\tThe correction algorithm is defined for linear LoRA modules $\\Lambda = W_A W_B$. Can this idea extend to other adapter forms (e.g. prefix-tuning or Mixture-of-Adapters)? Which layers receive correction, and is $\\lambda$ chosen per-layer or globally?\n\n4.\tWhat is the runtime and memory overhead of building the emulator (many SVDs) and performing LoRA correction? For very large models (38B+), is this overhead negligible compared to full training, or does it pose practical limits?\n\n5.\tThe paper notes EMLoC works without quantization. Could the authors combine EMLoC with model quantization (e.g. 4-bit LoRA) to push memory even lower? Are there reasons this was not explored or any pitfalls?\n\nOther limitations:\n\nMemory Factors Not Addressed: EMLoC focuses on model-weight memory only. Optimizer states and activations can still dominate overall usage. Thus EMLoC alone may still require techniques like sparse Adam or activation checkpointing for very deep networks. In contrast, methods like Quantized Side Tuning tackle these sources as well.\n\nAssumption of Linear Corrections: The LoRA correction enforces consistency of linear merged outputs on a subspace. This assumes the mismatch between full and low-rank models can be fixed by additive linear terms. If the compression-induced error is highly nonlinear, the correction might be insufficient, but the paper offers limited analysis of this.\n\nEvaluation Scope: Most experiments are on VQA and math QA benchmarks. It is unclear how EMLoC performs on large-scale open-domain NLP tasks (e.g. summarization, translation) or on very low-data regimes. Also, diffusion results show somewhat lower image-similarity scores than full training, suggesting possible quality trade-offs not deeply examined.\n\n\nReferences:\n\n[1] Wang, Xin, et al. \"Svd-llm: Truncation-aware singular value decomposition for large language model compression.\" arXiv preprint arXiv:2403.07378 (2024).\n\n[2] Zhang, Jun, et al. \"Train small, infer large: Memory-efficient lora training for large language models.\" arXiv preprint arXiv:2502.13533 (2025).\n\n[3] Dettmers, Tim, et al. \"Qlora: Efficient finetuning of quantized llms.\" Advances in neural information processing systems 36 (2023): 10088-10115.\n\n[4] Xiao, Guangxuan, Ji Lin, and Song Han. \"Offsite-tuning: Transfer learning without full model.\" arXiv preprint arXiv:2302.04870 (2023).\n\n[5] Zhang, Zhengxin, et al. \"Quantized side tuning: Fast and memory-efficient tuning of quantized large language models.\" arXiv preprint arXiv:2401.07159 (2024).\n\n[6] Shi, Dachuan, et al. \"UPop: Unified and progressive pruning for compressing vision-language transformers.\" International Conference on Machine Learning. PMLR, 2023.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nEMLoC directly eliminates the fine-tuning vs inference memory gap by training on a much smaller model. Its emulator preserves task-relevant knowledge via activation-aware SVD, and supports arbitrary LoRA placements. The LoRA correction is a novel, lightweight fix for model mismatch, enabling near upper-bound performance on held-out tasks. The paper is also well-presented with precise figures and ablations.\n\nWeaknesses:\n\nThe core idea of fine-tuning on a compressed model and then transferring to the full model is similar to recent work. EMLoC's novelty lies in using SVD-LLM for the emulator and a specific correction, but it builds on prior concepts of adapter transfer. The emulator relies on a small calibration set, and the efficacy of the SVD approximation without whitening (unlike SVD-LLM) may limit fidelity. Only 64 calibration examples are used. It's unclear how sensitive the results are to this choice.\n\nMeanwhile, LoRA correction involves hyperparameters ($\\lambda$) and a basis choice, but its effect is only briefly analyzed (Fig. 4 suggests that the results are stable). The method only targets model-weight memory, indicating that it does not reduce optimizer state or activation memory; therefore, it remains complementary to approaches like gradient checkpointing or 4-bit quantization. Baselines omit some solid or recent alternatives (e.g., there is no direct comparison to the LoRAM scheme or quantized fine-tuning, such as QLoRA).\nFinally, the algorithmic complexity of performing many SVDs and corrections on large models is not discussed; this might pose a practical overhead.", "was_revised": true, "final_justification": "the author has clarified my doubts.", "rebuttal": "**Novelty lies in using SVD-LLM for the emulator and a specific correction, but builds on prior concepts of adapter transfer. (W1)**\n\nWe’re happy to clarify the novelty of our EMLoC and explain how it differs from prior SVD-based methods. Our EMLoC introduces several distinct innovations:\n1. Flexibility in parameters to update: Unlike prior methods such as LoRAM (row pruning) and Offsite-Tuning (layer dropping), which impose hard constraints on which parameters can be updated, EMLoC adopts SVD to preserve the flexibility of standard fine-tuning, which allows all weights to be tunable.\n2. No additional data / continual pretraining for emulator construction: By leveraging downstream data for activation-aware SVD, EMLoC avoids costly continual pretraining required by earlier approaches such as LoRAM, significantly reducing the overhead of emulator construction.\n3. LoRA correction mechanism: We propose a LoRA correction mechanism that explicitly addresses the misalignment between the emulator used during training and the original model used during inference—a challenge that prior work has overlooked.\n\nAs demonstrated in Table 4, these innovations enable EMLoC to achieve better performance (up to 2.6% in accuracy) with significantly lower overhead (0.3 vs. 214 GPU-hours) compared to prior methods like LoRAM.\n\n---\n**How sensitive are the results to the size or the distribution of the calibration data? (W2, Q1)**\n\nAs shown in Figure 5 of the Supplementary Materials, performance plateaus after 32–64 calibration samples. We therefore use 64 examples as a practical default, indicating that our method is not highly sensitive to calibration set size, and a very small number of downstream samples is sufficient for effective emulator construction.\n\nRegarding data distribution, Table 4 highlights the importance of using downstream data for calibration. Specifically, following LoRAM, we applied compression and continual pretraining using a general-domain dataset. However, this setup yielded worse results—up to 5% lower accuracy—compared to EMLoC where a small calibration set from the downstream-task dataset is used. This supports our design choice of using a small but downstream-aware sample set.\n\n---\n**LoRA correction involves hyperparameters (𝜆) and a basis choice, but its effect is only briefly analyzed. (W3)**\n\nThe impact of the 𝜆 is analyzed in Figure 4. We observe that increasing 𝜆 up to a moderate value (e.g., 5 in Figure 4) improves accuracy by 0.5%, likely due to less distortion in the correction term. However, overly large 𝜆 values degrade performance—even falling below that of uncorrected LoRA. We hypothesize that an overly large correction term may dominate the original LoRA, ultimately degrading performance.\n\nRegarding the basis choice in the correction step, we provide a detailed discussion in Section C.2 of the Supplementary Material. We show that the resulting corrected LoRA weights remain the same, regardless of the choice of basis—demonstrating that the correction is mathematically invariant to this choice.\n\n\n---\n**EMLoC focuses on model-weight memory only, not reducing optimizer state or activation memory. It remains complementary to approaches like gradient checkpointing or 4-bit quantization. (W4, L1)**\n\nIt is correct that EMLoC focuses on reducing memory related to model weights. However, our method can be served as complementary to existing techniques. In fact, EMLoC is designed to be integrated seamlessly with methods like quantization, and activation checkpointing. As shown in Table 1, we combine EMLoC with QLoRA, and gradient checkpointing, and consistently observe better performance than baselines such as row pruning proposed by LoRAM and layer dropping proposed by Offsite-Tuning. These results demonstrate that EMLoC effectively reduces the memory footprint of model weights while remaining fully compatible with other techniques, jointly advancing the frontier of memory-efficient fine-tuning.\n\n---\n**Did the authors compare against the recent LoRAM method or evaluate quantization-based fine-tuning? (W5, Q2)**\n\nWe have included comparisons and discussion with QLoRA, using InternVL2.5 26B and 38B models in Table 11 of the Supplementary Material, where we observe that even with 4-bit quantization, QLoRA still requires 43GB of memory—while EMLoC can fine-tune the 38B model using only 20GB and achieves up to a 9% accuracy gain. Compared to QLoRA, EMLoC offers two main advantages:\n- Greater flexibility in controlling the compression ratio, rather than being constrained by fixed bit-widths in quantization-based method;\n- No degradation in inference performance, since quantization affects the model even during inference, whereas EMLoC retains the original model parameters for deployment\n\nRegarding LoRAM, we clarify that our experiments do include a direct comparison. As shown in Table 4, the first row corresponds to LoRAM, while the last row corresponds to EMLoC. We observe that EMLoC achieves comparable or better performance, while also incurring significantly lower overhead for constructing the emulator.\n\n---\n**Could the authors combine EMLoC with model quantization to push memory even lower? (Q5)**\n\nYes, we have explored combining EMLoC with quantization, and the results are presented in Table 1, specifically in the third block of rows. The results show that EMLoC still outperforms prior baselines—row pruning corresponding to LoRAM, and layer dropping corresponding to Offsite-Tuning—by up to 2.9% in accuracy, suggesting that EMLoC is complementary to quantization for further memory reduction.\n\n---\n**What is the runtime and memory overhead of building the emulator and performing LoRA correction? (W6, Q4)**\n\nWe thank the reviewers for raising such practical concerns. We have provided detailed measurements of memory consumption, throughput, and FLOPs during fine-tuning in Table 10 of the Supplementary Material, and the overall overhead of emulator construction in Table 4. For a more comprehensive analysis, we now include additional wall-clock runtime comparisons covering the full EMLoC pipeline — including emulator construction, fine-tuning, and LoRA correction — against both LoRA and QLoRA fine-tuning.\n\n|Size|Method|Construction (hr)|Fine-tuning (hr)|Correction|#GPU|Overall wall time (hr)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|8B|LoRA|0|11.6|0|1|11.6|\n|8B|QLoRA|0|12.1|0|1|12.1|\n|8B|EMLoC|0.3|4.7|20 sec|1|5.0|\n|38B|LoRA|0|15.8|0|4|15.8 (estimated)|\n|38B|QLoRA|0|19.5|0|2|19.5|\n|38B|EMLoC|1.2|14.2|50 sec|1|15.6|\n\nThe table below details EMLoC's memory usage across different steps. For the 8B model, each step of EMLoC uses less than 12GB, while the 38B model stays under 24GB. This efficiency makes EMLoC accessible to individual researchers, as it avoids the substantial computational and memory overhead typically required by previous methods that rely on continual pretraining.\n|Model size|Construction (GB)|Fine-tuning (GB)|Correction (GB)|\n|:-:|:-:|:-:|:-:|\n|8B|10.8|11.5|0.7|\n|38B|23.8|20.1|0.8|\n\n---\n**Most experiments are on VQA and math QA benchmarks. How does  EMLoC perform on large-scale open-domain NLP tasks or on very low-data regimes? (L3)**\n\nThanks for the suggestion. While recent works such as LoRAM (ICLR 2025) have primarily focused on NLP tasks like QA and code generation, our evaluation goes beyond standard text-generation settings by including vision-language tasks (VQA) and diffusion model customization, demonstrating the broad applicability of EMLoC. Due to time constraints during the rebuttal period, we were unable to include additional experiments in other domains, but we have conducted more extensive ablations and added new baselines to address the review comments and to strengthen our analysis.\n\n---\n**Can EMLoC extend to other adapter forms, considering that the correction algorithm is defined for LoRA? (Q3)**\n\nYes. EMLoC’s first two stages are fully compatible with other adapters, still enabling memory-efficient training. To demonstrate this, we conducted an additional experiment using Houlsby Adapters in place of LoRA. As shown in the table below, EMLoC still performs better baseline methods in Table 1, highlighting its flexibility beyond LoRA. However, the current LoRA correction algorithm is indeed tailored to the linearity of LoRA modules, and we believe there is no straightforward extension to other PEFT methods.\n\n|Method|PMC-VQA|WebSRC (val set)|WC-VQA|\n|:-:|:-:|:-:|:-:|\n|LoRA w/o correction|51.5|79.2|45.8|\n|LoRA w/ correction|51.6|80.8|46.2|\n|Houlsby Adapter|51.2|79.0|44.5|\n\n---\n**Which layers receive correction, and is chosen per-layer or globally? (Q3)**\n\nThe correction algorithm is applied to each LoRA independently, and we will clarify this in the revised paper.\n\n---\n**The LoRA correction assumes linear mismatch between the emulator and the original model on a subspace. What if the mismatch is nonlinear? (L2)**\n\nWe agree with the reviewer that our correction method assumes a linear correction. However, this limitation is shared by all approaches that apply weight-based corrections to LoRA modules without modifying the model architecture. As noted in (W1), EMLoC advances beyond prior work by explicitly addressing the mismatch between the training and inference models—an issue previously overlooked. While nonlinear correction remains an open direction for all LoRA-based methods, our results show that even linear correction consistently improves performance.\n\n---\n**Diffusion results show somewhat lower scores than full training, suggesting possible quality trade-offs not deeply examined. (L4)**\n\nWe totally agree with this observation. All efficient fine-tuning methods, including ours, involve trade-offs between efficiency and performance. While EMLoC achieves significant memory savings, there may be slight drops in performance compared to our upper-bound, full training."}, {"review_id": "Wm21B4B2oD", "reviewer": "Reviewer_XKMB", "summary": "This paper presents EMLoC, a three-stage framework that enables fine-tuning of large pre-trained models under the same GPU memory budget required for inference. First, it constructs a downstream-aware emulator by applying activation-aware SVD to each weight matrix using a small calibration set, yielding a lightweight low-rank model tailored to the target task. Second, it fine-tunes this emulator via LoRA, dramatically reducing memory overhead during training. Third, it introduces a novel LoRA correction algorithm that compensates for the misalignment between the emulator and the original model by adjusting the LoRA factors before merging them back into the full model. Experiments on multimodal VQA benchmarks (ChartQA, DocVQA, InfoVQA, TextVQA, PMC-VQA, WebSRC, WC-VQA) and NLP tasks (MathQA, GSM8K) demonstrate that EMLoC consistently outperforms strong baselines—including Offsite-Tuning, row-pruning (UPop), and LORAM—under 25–50% memory budgets, and scales to 26B and 38B models on a single 24 GB GPU.", "questions": "Formatting: In table 5, the last row reads \"Ours\" while others reads \"EMLoC.\" Perhaps this is a typo.\n\nMore Baselines: It would be great to see more baselines on distillation or smaller foundation models when comparing EMLoC.\n\nTime Costs: Apart from memory cost, it is important to also evaluate the time cost of EMLoC. It would be less practical if the efficiency in memory usage comes at significant cost on runtime.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\nMemory-Efficiency Guarantee: By construction, EMLoC closes the memory gap between inference and fine-tuning without requiring specialized hardware or gradient checkpointing techniques. Users can adapt models as large as 38 B with only 24 GB of GPU RAM.\n\nDownstream-Aware Compression: Activation-aware SVD ensures the emulator retains task-relevant features, outperforming both generic SVD and row-pruning approaches while incurring minimal construction overhead (~0.3 GPU-hours vs. >200 GPU-hours).\n\nCorrection Mechanism: The proposed correction algorithm rigorously derives a basis (the SVD factors of the LoRA input matrix), computes the emulator→model output discrepancy, and clamps corrections to preserve stability—yielding substantial accuracy gains (e.g., +3–4 points over finetuned emulator alone).\n\nComprehensive Evaluation: The paper evaluates across seven VQA datasets (including challenging domain-specific ones) and two math reasoning benchmarks, as well as multiple model scales (8 B→38 B), demonstrating consistent gains and strong robustness to hyperparameters.\n\nWeaknesses:\n\nCompute Overhead Details: Although emulator construction is fast, the paper lacks explicit wall-clock or FLOP comparisons for the fine-tuning and correction stages versus baselines, which would clarify real-world training costs.\n\nTime Efficiency: EMLoC ensures efficient usage of memory at training and inference, but there lacks analysis on the time costs of EMLoC.\n\nOther Distillation Baselines: A more appropriate baseline would be a foundation model of smaller size that is capable to be finetuned within the memory budget. There lacks comparison in this aspect, and the only direct finetuning baseline is from a larger (infeasible) model and serves as an upperbound.", "was_revised": true, "final_justification": "The authors have highlighted the runtime cost and demonstrated the improvements from EMLoC during rebuttal, which I found to be convincing.", "rebuttal": "**The paper lacks explicit analysis of time costs (e.g., wall-clock or FLOP) of EMLoC for the fine-tuning and correction stages versus baselines, which would clarify real-world training costs. (W1, W2, Q3)**\n\nWe thank the reviewers for raising such practical concerns. We have provided detailed measurements of memory consumption, throughput, and FLOPs during fine-tuning in Table 10 of the Supplementary Material, and the overall overhead of emulator construction in Table 4. For a more comprehensive analysis, we now include additional wall-clock runtime comparisons covering the full EMLoC pipeline — including emulator construction, fine-tuning, and LoRA correction — against both LoRA and QLoRA fine-tuning.\n\n|Size|Method|Construction (hr)|Fine-tuning (hr)|Correction|#GPU|Overall wall time (hr)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|8B|LoRA|0|11.6|0|1|11.6|\n|8B|QLoRA|0|12.1|0|1|12.1|\n|8B|EMLoC|0.3|4.7|20 sec|1|5.0|\n|38B|LoRA|0|15.8|0|4|15.8 (estimated)|\n|38B|QLoRA|0|19.5|0|2|19.5|\n|38B|EMLoC|1.2|14.2|50 sec|1|15.6|\n\nFrom these results, we observe that:\n\n- EMLoC achieves lower overall runtime compared to the baselines, even when accounting for the overhead of emulator construction and correction.\n- This efficiency stems from the activation-aware SVD construction, which is fast and avoids expensive continual pretraining, and from the lightweight emulator, which increases throughput during fine-tuning.\n- Furthermore, our correction step is computationally negligible compared to other stages while providing additional performance improvement.\n\nThese findings reinforce the practical impact of EMLoC: it not only reduces memory consumption but also remains competitive or superior in wall-clock efficiency, making it well-suited for real-world, resource-constrained deployment scenarios.\n\n----\n**Lack of comparisons with baselines on distillation or a foundation model of smaller size that is capable of being finetuned within the memory budget. (W3, Q2)**\n\nThanks for the insightful suggestion. In fact, we have already included such comparisons in Table 1, where we report results for InternVL2B and InternVL4B, both of which have comparable training costs with our methods.\n\nRegarding the suggestion to include distilled model baselines, we appreciate this valuable point. Unfortunately, we were unable to find any open-source, well-distilled versions of the InternVL2.5 series for a fair and direct comparison. Additionally, due to resource constraints, we are not in a position to conduct large-scale distillation ourselves. Nevertheless, to provide further insight, we conducted an additional experiment using InternVL3-2B (the latest version of the InternVL2.5 models used in our paper) as a stand-in for a compact model baseline. The results below show that EMLoC applied to InternVL2.5-8B with a 25% compression ratio still outperforms direct fine-tuning of InternVL3-2B. This suggests that EMLoC can retain the performance benefits of larger models even under tight memory constraints, offering a compelling alternative to using smaller or distilled models.\n\n| Model           | Method | Fine-tuning memory (GB) | PMC-VQA | WebSRC (val set) | WC-VQA |\n|:-----------------:|:--------:|:-------------------------:|:---------:|:------------------:|:--------:|\n| InternVL 2.5 2B | LoRA   | 10.9                    | 44.6    | 78.2             | 34.4   |\n| InternVL 3 2B   | LoRA   | 10.9                    | 48.4    | 79.5             | 44.8   |\n| InternVL 2.5 8B | EMLoC  | 11.5                    | 51.6    | 80.8             | 46.2   |\n\n---\n**Formatting: In table 5, the last row reads \"Ours\" while others read \"EMLoC.\" Perhaps this is a typo. (Q1)**\n\nWe thank the reviewer for catching this. Yes, the last row in Table 5 should read \"EMLoC\", and we will correct it in the final version."}]

Author rebuttal:

Meta-review: This paper proposes a novel method called EMLoC for memory-efficient fine-tuning. The authors also provide a novel LoRA correction mechanism which accounts for misalignment between the original and distilled model. The proposed method enables fine-tuning a 38B model on 24GB GPU which is very efficient. All reviewers found experimental results convincing.

The authors provided a very detailed rebuttal addressing the majority of reviewers’ concerns. This includes providing new baseline comparison with QLoRA and verifying compatibility with other parameter-efficient fine-tuning methods. They also provided algorithmic complexity and sensitivity of calibration data size. I request that authors make sure all new experiments will be added in camera ready versions. Given initial positive feedback and detailed rebuttal I accept this paper.
Decision: Accept (poster)