DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models
Authors: Komal Kumar
Rao Muhammad Anwer
Fahad Shahbaz Khan
Salman Khan
Ivan Laptev
Hisham Cholakkal
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Fine-Tuning
LoRA
Unified Models
Personalization
Model Adaptation
Parameter-Efficient Training
Text-to-Image Models
Primary area: deep_learning
TL;DR: An efficient fine-tuning framework that adapts a pre-trained weight matrix by decomposing its update into two components: (1) a projection onto the complement of a low-rank subspace spanned by a low-rank matrix, and (2) a low-rank update.

Abstract: Efficient fine-tuning of pre-trained Text-to-Image (T2I) models involves adjusting the model to suit a particular task or dataset while minimizing computational resources and limiting the number of trainable parameters. However, it often faces challenges in striking a trade-off between aligning with the target distribution: learning a novel concept from a limited image for personalization and retaining the instruction ability needed for unifying multiple tasks, all while maintaining editability (aligning with a variety of prompts or in-context generation). In this work, we introduce DEFT, Decompositional Efficient Fine-Tuning, an efficient fine-tuning framework that adapts a pre-trained weight matrix by decomposing its update into two components with two trainable matrices: (1) a projection onto the complement of a low-rank subspace spanned by a low-rank matrix, and (2) a low-rank update. The single trainable low-rank matrix defines the subspace, while the other trainable low-rank matrix enables parameter adaptation within that subspace. We conducted extensive experiments on the Dreambooth and Dreambench Plus datasets for personalization, the InsDet dataset for object and scene adaptation, and the VisualCloze dataset for a universal image generation framework through visual in-context learning with both Stable Diffusion and a unified model. Our results demonstrated state-of-the-art performance, highlighting the emergent properties of efficient fine-tuning. Our code is available on \href{https://github.com/MAXNORM8650/DEFT}{DEFT}.

DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models
Komal Kumar Rao Muhammad Anwer!
Fahad Shahbaz Khan
Salman Khan' Ivan Laptey! Hisham Cholakkal
Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE (komal kumar, salman, khan, ivan. laptev, hisham. cholakkal) @mbzuai. ac
anwer, fahad. khan,
Instruction
Uniforer
Depth
Canny
Normal
Redux Style
generations
Instrction
Misokc
Instuction
Restace
ML 5I3
Instruction
Open
Instent Style
Conmagi
Concept personaliztio for composition
Office (indoor)
Scene Personalization
Hools
Folleo
Ih Peplenteall
The belt enthoant
Church (Outdcor)
good pala around
Figure 1: This figure illustrates the tasks achieved using Decompositional Efficient Fine-Tuning (DEFT) method across diverse settings. includes in-context learning for a variety of image generation tasks, concept personalization for composition, where these compositions emphasize the personalization of the environment and the interaction between characters and pets, blending various personalization, featuring Church Rock
scenarios. Moreover, it highlights outdoor and indoor and the office. All the tasks fine-tune the baseline OmniGen model (8 using DEFT.
Abstract
Efficient fine-tuning of pre-trained Text-to-Image (T2I) models involves adjusting the model to suit a particular task dataset while minimizing computational
CCoresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

resources and limiting the number of trainable parameters, However, it often faces challenges in striking trade-off between aligning with the target distribution: learning novel concept from a limited image for personalization and retaining the instruction ability needed for unifying multiple tasks, all while maintaining editability (aligning with variety of prompts or in-context generation). In this work, we introduce DEFT, Decompositional Efficient Fine- Tunie, an efficient fine-tuning framework that adapts a pre-trained weight matrix by decomposing its update into two components with two trainable matrices: (1) a projection onto the complement of a low-rank subspace spanned by low-rank matrix, and (2) a low- rank update. The single trainable low-rank matrix defines the subspace, while the other trainable fow-rank matrix enables parameter adaptation within that subspace. We conducted extensive experiments on the Dreambooth and Dreambench Plus datasets for personalization, the InsDet dataset for object and scene adaptation, and the VisualCloze dataset for a universal image generation framework through visual in-context learning with both Stable Diffusion and unified model. Our results demonstrated state-of-the-art performance, highlighting the emergent properties of efficient fine-tuning. Our code is available on DEFT
Introduction
Text-to-image (T2I) models have revolutionized the way we generate images, transforming abstract concepts and textual descriptions or prompts into compelling visual representations. Models such as
Stable Diffusion have gained significant attention for their ability to generate high-quality images across various domains. Despite these impressive capabilities, adapting T2I models to specific
tasks-such as personalization for novel concepts [34 31 or multi-task generalization [25] remains
a challenge. Traditional fine-tuning approaches often require substantial computational resources and large amounts of reference data to train or adapt the model [19], which limits their practicality for many applications. This limitation is particularly evident in tasks requiring personalization, such as adapting a model to generate images of specific subject or scene, where training or reference data is often scarce. For instance, Custom Diffusion [9] tends to overfit when trained on a limited number of reference images, leading to poor generalization across prompts [45] 34).
DreamBooth 35 introduced personalized image generation method in which Low-Rank Adaptation (LoRA) 17 was employed to enable subject-specific tuning by applying learned LoRA modules at inference time, thereby preserving the identity of the subject during image generation. Although DreamBooth and other bosee methods, such as LoraMerge [6] and ComposLoRA [56], improve subject personalization and separation using a limited number of reference images, they still lack fine-grained control over pose, spatial positioning, and lighting. These methods also often struggle with convergence and overfitting, especially when the low-rank updates are unconstrained and do not align well with the pre-trained weights. Furthermore, many existing LoRA-basee approaches suffer from limited instruction following capabilities, particularly in complex scenarios involving multiple subjects scenes (See the Figure I..
In this work, we introduce DEFT (Decompositionall Efficient Fine- Tuning), a novel fine-tuning framework that overcomes these limitations by decomposing the weight matrix update into two components: (1) a projection onto the orthogonal complement of low-rank subspace, and (2) a low-rank update that allows for flexible adaptation within that subspace. This decomposition enables more efficient adaptation of pre-trained models to new tasks without sacrificing the model's ability to generalize to wide range of scenarios. By using two trainable low-rank matrices, DEFT extends the adaptability of the model while maintaining stability during the adaptation process, ensuring that the model doesn't 't suffer from catastrophic forgetting.
We evaluate DEFT on several challenging datasets, including Dreamboot 3 and Dreambench Plus 31 for personalization, the InsDet 377 dataset for object and scene adaptation, and the Visual- Cloze 25 dataset for universal image generation via visual in-context learning. Our experiments demonstrate that DEFT achieves better performance than saute-of-the-ard methods, highlighting its effectiveness and flexibility in fine-tuning models for diverse tasks. In particular, DEFT excels in tasks such multi-concepy composition, expanding model capabilities to universal image generation [25, and reducing overfitting on small datasets. Moreover, we show that DEFT offers significant improvements in personalization, especially when adapting models to multiple subjects complex
2

---

scenes /37, overcoming issues such as blending artifacts and limited control over pose and context. Unlike other methods, DEFT overcomes challenges such as blending artifacts and offers finer control over pose and context. Using its low-rank update in subspace, DEFT maintains a balance between efficient adaptation and generalization, providing greater flexibility in fine-tuning while minimizing the risk of catastrophic forgetting, advantages that are not fully realized by existing methods like DreamBooth or LoRA. The key contribution of the paper can be summarized as follows.
We propose a novel fine-tuning framework, DEFT, which improve adaptability by decomposing weight updates into two components: projection onto low-rank subspace and low-rank adjustment. This structure enables efficient weight editing while preserving prior knowledge.
DEFT supports efficient fine-tuning with minimal data, enabling personalization and adaptation to new tasks without extensive retraining or risk of overfitting.
We validate DEFT with extensive experiments datasets such DreamBooth, Dreambench Plus, VisualCloze, and InsDet, demonstrating its effectiveness in personalization and universal image generation. DEFT shows performance across subjects, scenes, and multi-concept compositions.
2 Related work
Personalized Diffusion Models by tuning: simple yet effective way to adapt concepts in text-to- image models is by selectively tuning subset of parameters, tailoring them to the desired concepts 3 [54]. [33]. [55]. Methods like DreamBooth [35] fine-tune all model weights on few images but are resource- inttuuese-itttiveeeeeee and prone to overfitting. Textual Inversion B is lighter approach, learning a new token embedding to represent a concept without modifying the model. While efficient, it struggles with fine details. Custom Diffusion [19] fine-tunes subset of parameters,
81. B5
learning concepts in minutes, and supports multi-concept training. Perfusion
[4] and AnyHyper 22 10 reduce trainable parameters but still face issues like overfitting and interference. These methods improve the text embedding space [8], use fine-tuning (35]
[4 6 28 or provide adapters (29 53 for personalized control, with some training-fire approaches [5] 10 18, 38, 46).
LoRA 17 and SVDiff [14 decompose matrices in diffusion models, but these focus on adjusting
scale rather than structure. PaRa [4] eliminates components during personalization for more stable, robust model mixing, avoiding the overfitting risks of scale adjustments [26] [3] [5]. Our dynamic
framework adapts fine-tuning methods like LoRA 6 and PaRa [4, which project onto low-rank subspace and have a flexible low-rank update for new concepts or tasks.
models support image generation tasks where text and images are encoded and concatenated, separated tokens and fed to the single transformer models. These models share the same transformer blocks and can utilize objectives such as MaskGIT or (27 for image generation. Unified models also support in-context learning along with prompt. Combining multiple personalized concepts in one image is more complex than generating a single concept. Fine-tuning a model on multiple concepts can lead to interference dominance of one concept over others [19 12]. Early methods inserted multiple learned tokens into prompts, but diffusion models often fail to distinctly generate all concepts, especially if they share attributes or were not jointly trained. Kumari et al. [19] addressed this by fine-tuning individual concept models at low rates or merging them, allowing composition of new concepts with quality trade-offs. Mix-of-Show -how 122 uses fusion LoRA and additional guidance such as sketches, improving composition, but requiring extra input and retraining. Without guidance, it suffers from concept vanishing mixing [12]. AoRA-Composer (56) merges multiple LoRAs inference without retraining, maintaining concept isolation and visibility, demonstrating improved results in 2-3 concept compositions. Wu et al. 47 introduced the LoRA Expert Mixture (MOLE), which combines LoRA through learned gating weights, balancing multiple concepts. These methods show promise, but still require manual tuning or supervised training. and may struggle with many concepts or without guided layouts. In this work, we employ low-rank DEFT adapters to fine-tune models for new tasks, including tasks that were not originally supported by the base models.
Unified Models fine-tuning: Unified 48 2.5 are the single transformer-
3 Methodology
Figure 2 provides an overview of DEFT and compares it to other efficient fine-tuning techniques, namely PaRa 4 and LoRA (6). We begin by considering image generation models such as Stable
3

---

Diffusion (SD) 32 and the unified Omnigen 48 model as our base architectures. SD, latent text- to-image generation model, based on Diffusion Probabilistic Models (DDPM) [T 39 Similar to LoRA's approach in diffusion models [6, we introduce low-rank adapters to the UNet and condition encoders that operate in the latent space [00 In the case of unified models, we apply low-rank
adapters to the linear layers of the transformer.
In this methodology, we first describe the foundational low-rank adaptation mechanism, then build upon this to introduce DEFT, which enhances flexibility while maintaining model generalization. The following sections explain these methods in detail.
aB" A
W QQ'W
PP W
PR
Rd
B
QT
pr
W
Fass
W
P
W
A
P
LoRa
PaRa
DEFT
Figure 2: The figure illustrates the difference between DEFT and other similar efficient fine- tuning techniques: (a) LoRA (Low-Rank Adaptation) modifies the pre-trained weight matrix W by adding low-rank update aB A, where A and B trainable low-rank matrices. (b) PaRa (Parameter-efficient Rank Adaptation) introduces a similar low-rank update, W - QQ' with Q representing the low-rank orthonormal basis. (c) DEFT (Decompositional Efficient Fine-Tuning) further decomposes the update into two components: a projection onto the subspace, W PP'W. and low-rank adjustment, PR where P and R are trainable matrices that enable more flexible adaptation while preserving the model's generalization ability.
3.1 Decompositional Efficient Pine-Tuning
M = [W1, W2,..., W.; W. E Rm pre-trained weights. Instead of updating the full weight matrix W., which can be computationally expensive requiring ((ma) per matrix, the update is decomposed into smaller matrices to reduce the number of trainable parameters. We can assume
W E M E Rmx= pre-trained weight matrix. We aim compute adapted weights W', such that: w' W + AW where AW is low-rank update constrained to rank < min (m, n) so that 0 reduce significantly. DEFT builds the concept of low-rank adaptation, as introduced in techniques like LoRA 6 and Pa While LoRA injects trainable low-rank matrices into specific layers, approach goes further by decomposing the low-rank update into components: the projection onto a subspace and the low-rank adjustment. This decomposition allows us to adapt the model more flexibly to broader range of tasks while preserving its original performance. LoRA injects trainable low-rank matrices into specific layers while freezing the original weights. For a linear layer with input R" the modified forward pass becomes: h Wr AW Wx Ar. A is initialized with random Gaussian weights, and B with zeros, ensuring AW = at initialization.
The efficacy of low-rank updates is grounded in the observation that neural networks reside on low intrinsic manifolds during adaptation ..] For a pre-trained weight W, the update AW amplifies task-specific feature directions not emphasized in W. Let UEV be the SVD of W. The projection of AW onto W's singular vectors reveals that AW primarily modifies directions orthogonal to W's dominant singular vectors.
Inspired by this orthogonal direction, PaRa introduced low rank update in the orthogonal complement of the freeze weight, and they used rank reduction via the following formulation: W Wo QQ" Wo, where Q an orthonormal basis matrix and W, is the pre-trained weight matrix (Please refer to Appendix B for the proof). This operation reduces the rank of the weight matrix

---

Wo, ensuring that the model's 's adaptation is constrained to smaller and more relevant subspace for the target task.
We extend the idea of modifying subspace with the low-rank update to improve the flexibility for
diverse tasks in DEFT. The column space of Wiatt = (7 QQ > + QR extends the subset of
Wo's 's column space by incorporating Q Qfoor the full proof, please refer to Appendix/C), enabling adaptation new tasks which can be described as:
decomposition trainable matrices Appendix D
W = (I - QQ-QQ_)W, QR
By adding QR W reduce. the total column space col (Wioa) becomes:
col(W (ota)) col( roduse) + co)(QR) col(WW) col(Q)-
If col( & col(Wo), the subspace is extended, allowing adaptation to new tasks. That can be as a low-rank update, where low-rank updates inject task-specific directions into the weight matrix..
3.1.1 Decomposition approaches
To generalize the rank-reduction mechanism beyond QR decomposition, we extend our method to support several decomposition techniques for constructing the projection matrix P R used to eliminate subspaces from the original weight matrix Wo- In QR decomposition, B is decomposed as B = QR, where Q is orthonormal, and R is upper triangular. TruncatedSVD [21 uses USVT, where U contains the top r left singular vectors, and is a diagonal matrix of singular values. Low-rank matrix factorization (LRMF) [51] uses a scaled basis Ù = UVS. while non-negative matrix factorization (NMF) 23 decomposes B into non-negative matrices B R WH. Eigen decomposition computes BB" E Raxd and performs eigen decomposition to form the projection matrix P V² In all cases, the projection matrix follows the form Wreduce = Wo PP' Wo + PR, with the linear transformation. The final DEFT update equation becomes:
h Wredce = Wor - PPT
P Wox + PRæ.
(2)
As illustrated in Figure when using PPT the up- date removes components aligned with the entire span of P, producing strong displacements along both positive and negative directions. By contrast, replacing it with eliminates only the non-negative portion of the span, leading to sparser and more struc- tured updates. This connects naturally to NMF, where non-negativity encourages additive feature combinations.
To further stabilize optimization, the framework applies a higher learning rate to R compared P, mirroring the design of LoRA. This modular formulation enables differ- ent structural biases to be introduced in a plug-and and--lay manner during fine-tuning, potentially improving adapt- ability under different data regimes or downstream tasks. To update on the framework works with the increased learning rate for compared to P, so this turns out to follow similar path as LoRA. This modular formulation allows different structural biases to be introduced in plug- and-play fashion during fine-tuning, potentially improving adaptability under different data regimes.
Figure 3: Displacement field visualiza- tion of the DEFT update. Left: subtrac- tion using the full span of P, where both positive and negative entries contribute to the removed subspace. Right: subtrac- tion using only the non- non-negative shadow ReLU((). which restricts removal to additive components and yields weaker, more selective displacements.
Experiments
4.1 Experimental details
To check the effectiveness of DEFT, we conducted extensive experiments. We wrote the library of the DEFT using Torch, which can be adapted to any model. We follow the similar design choice like LoRA [1]. DEFT aim to inject knowledge through low-rank updates while maintaining the
5

---

original model's stability. We use rank (r) equal to 4 for DreamBench Plus [31 benchmarking for all of the methods. For Visual cloze [5] universal image generation, we used r 32. For
evaluation purpose, we conducted experiments
universal image generation from Visualcloze [25]
image personalization on DreamBoot [34] subject (live, no-living, and styles) personalization, and scene personalization. Please see the Figure I for the tasks supported with qualitative results. All experiments were conducted with 4 NVIDIA RTX A6000 50GB GPUs.
Baseline: We extend stable diffusion and stable diffusion XL (SDXL)
[22 image personalization,
along with Omnigen 48), into unified model. This model utilizes a single transformer- based image generation framework, simultaneously modeling text tokens and image tokens for multi- concept composition. The goal is to expand the model's 's capabilities support the DEFT module, enabling subject-wise parameter tuning and controlled scene generation without requiring full model fine-tuning. For training and evaluation, we use the diffuser library (42
Dataset: For evaluation, we use several datasets, including the DreamBooth Benchmark [34] which contains 30 personalized concepts across 15 categories, with 4-6 images per concept and 25 challenging prompts. Additionally, we curate datasets for single-subject and multi-subject personalization, ensuring diverse compositions. Key datasets include VisualCloze [25] (We created 3M instructions for in-context learning for fine-tuning), DreamBooth (30 concepts for subject-driven generation), DreamBench Plus [I (150 concepts for noman-aligned benchmarks), and InsDet [37], the high-resolution dataset for instance detection with 100 objects and 5 scenes. We created
instructions for these 100 objects with an average of 22+ images using Blip2 (24), and for the scene, we used Qwen 32B 550 for the details description.
4.2 Results and Analysis
4.2.1 DEFT improves Instruction following abilities
Table reports CLIP-T scores, measuring image-text alignment across T21 models. Traditional methods (Textual Inversion, DreamBooth) and recent variants (LoRA, BLIP-Diffusinn, PaRa) show varying performance on SD v1.5 and SDXL.. Our DEFT, built on SDXL v1.0, achieves the best score (0.361), surpassing LoRA (0.341) and PaRa (0.354). This improvement stems from DEFT's low-rank injection, which expands the fine-tuning subspace, retaining the original model's instruction-following capabilities, enabling more coherent image generation in personalization tasks.
Table 1: This table illustrates the performance of several T2I models 150 subjects with eight different and diverse prompts from the Dream- bench Plus (31 dataset. We evaluated with their associated CLIP-T scores used for image-text alignment. The results show the ability of each model to integrate visual and textual information across different versions of models, such SD v1.5 and SDXL v1.0. For our DEFT, we used the same setting as DreamBooth LoRA (6, in- cluding the SDXI..
Table 2: Style transfer and conditional genera- tion comparison for quantitative performance on Visualcloze (25 test dataset. In Visualcloze re- sults, I (00 represented as dev and FLUX. SUUX.1-FO 1-Fil-dee as Fill. Both DEFT are evalu- ated on rank = 32 fine-tuned with the OmniGen 48 model. Different scores are used for the two distinct tasks.
Condition
CLIP-Scure
Imagee Cumistency
Image
IMNOVI
DINOY2
Onniienn
95.45
$7.13
Canny
VisuiClous
59.32
Frameworks
T2I Model
CLIP-T
DRF'T (Durs)
95.78
WL37
98.65
UnniCien
92.02
77.39
Textual Inversion
9
SD v1.5 0.302
Depth
VaralClore
DreamBooth
34
SD v1.5
0.323
DEFT (Qurs)
93.18
68.98
85:75
DreamBooth LoRA
6
SDXL v1.0
0.341
Style Type
Method
Test Score!
Image Score"")
F1(1)
BLIP-Diffusion
24
SD v1.5
0.286
Inarmatyla
0.31
0.60
OnniCen
0.27
0.52
0.55
Emu2 (40]
SDXL v1.0
0.310
InstundStyle
VisulClote des
0.30
0.53
IP-Adapter-Plus 52
VIT-H
SDXL
0.282
VaradClore4i1
0.29
0.55
IP-Adapter 52
VIT-G
SDXL vI.1
0.309
DETT (Ours)
0.28
0.69
0.59
OrunCCan [5
0:27
0.58
0.47
PaRa
SDXL v1.0
0.354
Redussttlle
VasnlClstn-de
0.29
0.53
DreamBooth DEFT (Ours)
SDXL v1.0
0.361
0.37
0.55
DEFT (Ours)
0.26
0.69
0.49

---

4.2.2 Universal Image Generation through Adapter
Beyond personalization, assess DEFT's generalization by fine-tuning OmniGen [48] on Visual- Cloze [25]. Tables 2 B that DEFT consistently outperforms OmniGen and VisualCloze in both style transfer and conditional generation.
For image consistency, DEFT yields higher CLIP and DINO scores under Canny Edge and Depth Map conditions, demonstrating stronger alignment between text and visuals. In style transfer, it achieves the best Image Scores (0.69) and Fl metrics across both InstantStyle and ReduxStyle tasks, indicating better preservation of textures and styles.
Table further further highlights DEFT's advantage in controllability and quality. Under Canny Edge, DEFT achieves the highest Fl and SSIM, with competitive FID. Under Depth Map, it maintains strong controllability (F1 0.86) while substantially improving SSIM (0.72). These results confirm that DEFT enhances both structural fidelity and perceptual quality, making it well-suited for universal image generation.
Table 3: Comparison of various methods on controllability and quality metrics across Canny Edge and Depth Map conditions. Our DEFT fine-tune OmniGen
48
Canny Edge
Depth Map
Controllability
Quality
Controllability
Quality
Method
F1
RMSE
FID 1
SSIM 1
F1
RMSE 1
FID
SSIM |
ControlNet [53
0.13
46.06
36.83
OmniControl
49
0.47
29.58
0.61
0.47
21.44
36.23
0.52
OneDiffusion
22
0.39
32.76
0.55
0.39
39.03
39.03
0.49
OmniGen
0.43
4.55
51.58
0.47
0.85
9.83
115.54
0.69
VisualCloze _ev
0.39
30.36
0.61
0.39
25.06
42.14
VisualCloze Fill
0.35
30.6
0.55
0.35
10.31
33,88
0.54
DEFT (Ours)
0.48
4.11
46.62
0.66
0.86
9.38
46.74
0.72
4.2.3 Qualitative comparison
In Figure we compare the consistency of reference images across various tasks from DreamBench Plus 31. The DEFT method outperforms LoRA in generating consistent, high-quality images. DEFT effectively maintains visual coherence and fine details, such as textures and proportions, when adapting to diverse prompts involving cat, horse, and pixel warrior. For example, the horse color is maintained in most of the images compared to LoRA.
In Figure 5 we showcase various objects and live subjects from Dreambench Plus 31 LoRA and our DEFT shows comparable results but there are some details that can seen. For instance, in the Jellyfish Image, DEFT captures delicate transparency and lighting, which PaRa and LoRA fail to replicate. Similarly, in the Taxi Image, DEFT preserves sharp details like streetlights and reflections, while the other methods show blurred elements. The Drum Image demonstrates DEFT's ability to maintain fine textures and lighting, unlike PaRa and LoRA, which produce less-defined results. In the Pokémon Figures Image, DEFT preserves vibrant colors and detailed textures, while PaRa and L produce duller, less detailed images. Lastly, DEFT excels at reproducing fine fur textures and eye clarity in the Cat Image, where PaRa and LoRA struggle with detail preservation.
4.2.4 Emergent properties efficent-finetuiig
The emergent properties of efficient fine-tuning are evident when a model, initially unaware of specific concepts, is trained on a small set of images representing diverse scenarios. In the Figure σ we show the multi-concept personalization by finetuning omnigem, which does not require separate LoRA adapter for each task like LoRAMerge G In the first row, the model is fine-tuned on a few pet-related images, such as Corgi dog and teddy bear, in various environments. It learns to recognize key features of pets, like shape, color, and behavior, despite having no prior knowledge. In the second row, the model demonstrates its ability to generalize these learned features, successfully recognizing pets in new contexts- such as outdoors or with people-unde different lighting and settings. The third row shows the model fine-tuned on a small number of object-related images, including toys and accessories. Here, the model learns to identify objects based on features like shape.
7

---

DEFT
LoRA
DEFT
DEFT
LoRA
Figure 4: Comparison of image generation consistency between our DEFT and LoRA across various categories to check the consistency of the reference. The images show results for different subjects, including cats, horses, and pixel worrier (Zoom for best view).
PaRa
Original
LoRA
Figure 5: This figure compares the performance of LoRA, PaRa, and DEFT models in object-based tasks using the DreambenchPlus dataset. All methods use the same prompt, with the ideal model exhibiting results that closely resemble the original while maintaining diversity.
size, and texture. In the fourth row, the model's generalization ability is further tested as it accurately recognizes these objects in new, unseen conditions, such as backpacks in varied environments like parks, streets, and cityscapes. This illustrates how efficient fine-tuning, with minimal data, empowers the model to not only learn specific concepts but also extend knowledge to handle wide array of tasks and environments without requiring extensive retraining.

---

Testing conditionss
Training
Figure 6: This figure demonstrates the emergent properties of the model, which adapts the training set to generate diverse combinations under varying testing conditions. The model successfully generates new and creative image variations by combining elements from different objects and scenes, showcasing its ability to generalize beyond the original training data and produce novel compositions. All the results on the right side were obtained after fine-tuning OmniGen using DEFT.
4.3
Ablation study
To better understand DEFT's behavior, we include ablations on decomposition methods, training duration, and novel capabilities.
Effect of decomposition. In Sec. 3.1.1 we described the various decomposition methods that can function the eqn eqn[[ootee oother than QR decomposition. In Figure we provide visual example of different decompositions on fine-tuning in the DEFT style. All methods, including LRMF, NMF, QR, TSVD, and Relaxing P. closely mimic the reference image, each capturing key aspects such as lighting, subject clarity, and environmental context. In the default setting of our DEFT, we used it without decomposition because this configuration provides baseline for evaluating the performance of the model with minimal modifications. By avoiding decomposition initially, we ensure that any observed improvements or changes performance can be directly attributed to the fine-tuning process itself, rather than the additional complexity introduced by matrix decompositions.
Decomposition Methods. As shown in Table 4 and Figure 7 alternative decompositions (e.g., NMF, QR, TSVD) achieve competitive performance, with learnable variants (Relaxing P, Pamt) providing stronger prompt control. This highlights that DEFT is flexible to different factorization strategies, though our default setting avoids decomposition for simplicity.
Table 4: Performance comparison of different decomposition methods as discussed in section 3.1.1 Relaxing P and Pafm indicates the learnable matrix which directly adapts the Wo as shown in the E Pafm is non-negative version for adaptation as shown in the Figure 3 All decomposition approaches demonstrate competitive performance, with Paim showing significant control over prompts, highlighting the value of structural non-negativity
fine-tuning bias. All results are evaluated with DreamBooth 34 Dog (See Figure 7 with 8 diverse prompts.
Method
Speed (ms)
CLIP-
CLIP-T
DINO-VI
Aesthetic
Sharpoess
L8ME
29.10
0.827 + 0.008
0.220 = 0.0G1
0.307 + 0.040
0.04 40014:100 0.005
549 + 4144
5.16
0.883 0.03
0.222 0.042
0.357 + 0.068
0.15 0.005
9)
5.38
0,875
0.177 0.041
11330 0.000
0.17
210 106
TSVD
28.72
0.875 0.011
0,223 0.041
0.333 0.002
0.066 0.004
331 134
Reising
5.22
0,879 à: 0.000
0.205 0.001
0,300 4: 0.00S
0.15 0.000
340 150
Releving Peot
03323 0.07
0.200 0.033
01.00
0.06
20

---

Refernoe Image
Inmi
nm!
oF
tovd
Relexing F
Relexing Pv
1. A raalisii photo of sks beach during
clase-up photo sks dog in a kichen with werm lightng.
Figure 7: This figure showcases various rank-reduction techniques for personalization. By extending traditional decomposition methods such as QR, Truncated SVD, LRMF, and NMF, Relaxing enables the projection matrix P to adapt and refine the original weight matrix Wo (See Eq 2 effectively generating new image concepts without any decomposition required.
Training Steps. Table 5 compares DEFT with LoRA at different training horizons. While both improve with more steps, DEFT maintains stable instruction adherence (CLIP-T) and image quality, whereas LoRA suffers degraded alignment at longer training. This makes DEFT more suitable for extended fine-tuning.
Table 5: Effect of training steps on fine-tuning OmniGen with DreamBooth's Dog 34 We compare DEFT (using QR decomposition with rank 8) against LoRA (also with rank 8). DEFT maintains balance between image quality and cistruction-followiii capability across training durations, whereas LoRA suffers a significant drop in instruction adherence, especially at longer training horizons.
Training Steps 2000
Training Steps 8000
Method
CLIP4I
CLIP-T
Aesthetic
Sharpness
CLIP-I
CLIP-T
Aesthctic
Sharppess
DEFT
0.811 a 0.066
0,302 + 0.026
0.015 + 0.005
294 a 261
0.882 1 0.059
0,319 a 0.625
0.015 2: 0.000
320 1 364
LORA
0.836 = 0.009
0.266 = 0.033
0.006 + 0.004
449 # 334
0.876 + 0.024
0.219 + 0.023
0.013 + 0.002
65
Camera-Aware Generation. To test whether fine-tuning can add new abilities beyond style or subject adaptation, evaluate DEFT on 3Daaware generation using SFM data (36) extracted from a drone video a church (see the last row of Figure 6). We trained OmniGen with all 11 intrinsic and extrinsic camera parameters using special token. Table 6 compares pretrained, instruction-onle (DEFTINs) iNs), and camera-augmented (DEFT SFM) inference. DEFT STM shows slight but consistent gains in CLIP-I, CLIP-T, and Sharpness over both baselines, suggesting that DEFT can be extended toward camera-aware generation. A camera position encoding could further improve this capability.
Efficiency. Furthermore, we analyzed training efficiency for the rank-64 configuration with batch size of on the OmniGen 3.762B parameter model. LoRA achieved 12 steps/sec, as it does not re- quire decomposition or matrix multiplication, with a peak memory usage of 8.03 GB and 37.7M trainable parameters. DEFT ran at 11 steps/sec with 7.95 GB peak memory and the same 37.7M trainable parameters;
Table 6: Testing DEFT's camera-aware generation on SFM data: pretrained vs. instruction-only vs. instruc- tion+camera parameters.
with relaxed P (no decomposition), matrix mul- tiplication is still required. PaRa operated at steps/sec with a peak memory usage of 7.86 GB and 25.2M trainable parameters
CP-]
CP-
DO-VI
Sham
Pre
0.475
0201
0.20
1179
DEFTss
0.647
0:10
0.300
DRFT
0.600
0.213
0.343
1852
5 Conclusion
In this work, we have introduced DEFT, novel framework designed to optimize the fine-tuning process of pre-trained models. DEFT improves the model's adaptability by decomposing weight updates into two key components: projection onto a low-rank subspace and flexible low-rank update. This decomposition allows the model to fine-tune efficiently, maintaining high performance while reducing the computational burden. Using DEFT, we have shown that fine-tuning can be efficient without sacrificing flexibility or model generalization. This work highlights the potential of DEFT to overcome the challenges of personalization and adaptability in text-to-image generation models. Additionally, results suggest that DEFT could be a future fine-tuning framework.
10

---

References
[1] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012. 13255,, 2020.
[2] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano, Domain-agnostic uuing--enodee for fast personalization of text-to-image models. In SIGGRAPH Asia 2023 Conference Pupers. pages 1-10, 2023.
[3] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11315-11325, 2022.
[4] Shangyu Chen, Zizheng Pan, Jianfci Cai, and Dinh Phung. Para: Personalizing text-to-image diffusion via parameter rank reduction. In The Thirteenth International Conference on Learning Representations, 2025. [5] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems, 36, 2024.
[6] cloneofsimo. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2022.
[7] Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text-to-image generation via positive-negative prompt-tuning. arXiv preprint 1rXiv:2211.11333 2022.
[8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0 01611,, 2022.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-imaae generation using textual inversion. arXiv preprint arXiv:2208. 01618, 2022.
[10] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder- based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 42(4):1-13, 2023.
[11] Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:2311.2200 2023.
[12] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zähan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 115508199900 2023.
[13] Yiwen Guo, Anbang Yao, and Yurong Chen, Dynamic network surgery for efficient dans. Advances in neural information processing systems, 29, 2016.
[14] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1224-333,,22 2023.
[15] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, volume 28, 2015.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 2020.
[17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. arXiv preprint 0rXiv:2106.09655 2021.
[18] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li. Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang. and Yu-Chuan Su. Taming encoder for zero tine-tuning image customization with text-to-image diffusion models. arXiv preprint 2.34.v:..04.02660
2023.
[19] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Yan Zhu. Multi-concept customization of text-to- text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[20]
Black Forest Labs. Flux. btttps://git t/gbb com/ baoo///lakk-aaea watresttlaassfluu 2024.
[21]
Rasmus Munk Larsen. Lanczos bidiagonalization with partial reorthogonalization. DAIMI Report Series, (537), 1998.
[22] Duong Le. Tuan Pham, Sangho Lee, Christopher Clark, Aniruddha Kembhavi, Stephan Mandt, Ranjay Krishna, and Jiasen Lu. One diffusion to generate them all. arXiv preprint arXiv:24/ 16318, 2024.
[23] Daniel Lee and H Sebastian Seung. Algorithms for non-negative matrix factorization. Advances in neural information processing systems, 13, 2000.
[24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language -aaagge--iageeaee pre- pre-tranning with frozen image encoders and large language models. In International conference on machine learning. PMLR, 2023.
pages

---

[25] Zhong-Yu Yu Li. Ruoyi Du, Juncheng Yan, Le Zhuo, Zhen Li, Peng Gao, Zhanyu Ma. and Ming- Mingg Cheng. Visualcloze: A universal image generation framework via visual in-context learning. arXiv preprint urXiv:2504.0796 2025.
[26] Baoyuan Liu, Min Wang. Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional
neural networks. In Proceedings of the IEEE conferencee on computer vision and pattern recognition, pages
806-814, 2015.
[27] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209. 03000, 2022.
[28] Shyam Marjit. Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, and Pin-Yu Yu Chen. Dif- fusckrona: A parameter efficient fine-tuning method for personalized diffusion models. arXiv preprint MxXiv2402.174412 2024.
[29] Chong Mou, Xintao Wang, Liangbin Xie. Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[30] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pages 8162-8171. 2021.
[31] Yuang Peng, Yuxin Cui, Haomiao Tang, 7ekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned henchmark for personalized image generation. In The Thirteenth International Conference on Learning Representattons, 2025.
[32] Robin Rombach, Andreas Blattmann. Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022.
[33] Aniket Roy, Maiterya Suin, Anshul Shah, Ketul Shah, Jiang Liu, and Rama Chellappa. Diffnat: Improving diffusion image quality using natural image statistics. arXiv preprint arXiv:2311.09755 2023.
[34] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman, Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22500-22510, 2023.
[35] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Ruhinstein, and Kfir Aberman, Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the
IEFF/VVF conference on computer vision and pattern recognition, pages 22500-22550, 2023.
[36] Johannes L Schonberger and Jan-Michael Frahm. Structure- senuctureffrom-aee motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4 11044113 13, 2016.
[37] Qianqian Shen, Yunhan Zhao, Nahyun Kwon, Jeceun Kim, Yanan Li, and Shu Kong. A rishlucoouttion dataset for instance detection with multi-view object capture. Advances in Neural Information Processing Systems, 36:42064 42076. 2023.
[38] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.0341 2023.
[39] Jaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
[40] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398-14400, 2024.
[41] Yoad Tewel. Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-lacked rank one editing for text-to-image personalization. ACM SIGGRAPH 2023 conference proceedings, pages 1-11, 2023.
[42] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https: //gttbb com/ hwwghhugggncccceeeee bttgggginnaaeeeeffueeeee
2022.
[43] Andrey Voynov, Qinghao Chu, Daniel Cohen Or, and Kfir Aberman. p+: Extended textual conditioning in text-to-image generation. arXiv preprint Sttiv:2333.0522 2023.
[44] Haofan Wang, Peng Xing, Renyuan Huang, Hao Ai, Qixun Wang, and Xu Bai. Instantstyle-plus: Style transfer with content-preserving in text-to-image generation. arXiv preprint arXiv:2407.007 007888 2024.
[45] Fanyue Wei, Wei Zeng, Zhenyang Li. Dawei Yin, Lixin Duan, and Wen Li. Powerful and flexible: Personalized text-t-immage generation via reinforcement learning. In European Conference Computer Vision, pages 394-410. Springer, 2024.
[46] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized texi-to-image generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15943-15953, 2023.
[47] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404. 13628, 2024.
[48] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang. Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409. 11340, 2024. Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint any time for human motion generation. arXiv preprint arXiv:2310.085800 2023.
[49]
[50] An Yang. . et al. Qwen2 technical report. arXiv preprint
2024.
12

---

[51]
Jie Yang, Notes on low-rank matrix factorization, arXiv preprint =rXiv:1577.00333 2015.
[52] Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308. 06721, 2023.
[53] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847. 2023.
[54]
Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong- Yoeg-Yee Lec, Oliver Deussen, and Changsheng Xu. Prospect: Prompt spectrum attribute-aware personalization of diffusion models. ACM Transuctions on Gruphics (00)) 1-14, 2023.
[55] Haoyu Zhao. Tianyi Lu, Jiaxi Gu, Xing Zhang. Qingping Zheng, Zuxuan Wu, Hang Xu, and Yu-Gang Jiang. Magdiff: Multi- aligmmtl diffusion for high-fideling video generation editing. European Conference on Computer Vision, pages 205-221. Springer, 2024.
[56] Ming Zhong. Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen, Multi-lora composition for image generation. arXiv preprint arXiv:2402.
2024.
13

---

A Preliminaries and Notation
Let Wo R=x=" denote fixed (pretrained/frooeen) weight matrix. Let Q Rr with < denote a matrix whose columns are orthonormal, i.e., Q Q = Iy. We write Pq= == QQ E mm for the orthogonal projector onto the subspace span C IR" then I Pq is the projector onto sQan
We use col(A) A) for the column space (image) of matrix A. For two subspaces U, V C R., the sum U + y = [u + u: U, V) and dim(U V) = dim U + dim V dim(U n V).
We define the reduced baseline W Wreduce == (1 Po) Wo = (1 - QQT) Wo, and the adapted total weight Wiotal := W reduce + QR (I - QQ W + QR, where E gexa is trainable (low rank through r).
B Proof of Column Space Subset
Claim: The column space (image) of W Weduuee Wi - QQ Wo subset of the column space of Wo.
Let S. col(W)), the column space of Wo and Q be orthogonal matrix from the QR decomposition of P, so QTQ== -.
Proof:
Now any vector col W reduce) can be written
W We (IQQ)
) Wor,
where r is an arbitrary input vector.
To decompose Wor, let Woz. By definition, So- The term QQ is the orthogonal projection of y onto cQl( cQ) C So. then QQ y E So, since projections onto subspaces of So remain in Sn-
First, we will check the Linearity of Sg. Since So is subspace, it closed under addition and scalar multiplication. Therefore, yQQ E So. as both and QQ are in Sg. Every v = (IQQ' Jy lies in Sg. Thus, col(W reduce) S co).
We assumes col(Q) C co.(..). This is enforced during training by:
Initializing to zero, ensuring Q starts within So- Fine-tuning P on task-specific data, which implicitly aligns col(Q) with directions in So relevant to the task.
By subtracting components of Wo along learned orthonormal bases Q, the method reduces the output space within Sy, preserving the original model's expressivity while enabling parameter-efiie efficcient adaptation. The rank reduction is controlled by the dimensionality of Q, which is typically small (e.g.. 4).
The column space of Wreduce is a subset of Wo column space because Wreduce = (TQQ WW and the projection QQ T Wo removes components of Wo only within the subspace spanned by Q. Since Q is learned to lie in col(W)) during training, the difference WoQQ W, remains entirely in col( This ensures dimensionality reduction without exiting the original output space, as required.
C Extension of Column Space with Wiotal
Claim: The column space of W = (IQQT) W + QR extends the subset of W 's column space by incorporating QR, enabling adaptation new tasks.
Proof:
The total weight matrix W Wiotal is given by:
Wrotal
+
QR Low-ronk adaptation
Weoduo
14

---

The term W Weduse projects Wo onto the orthogonal complement of Q's column space, as proven earlier. The term QR adds a trainable low-rank component derived from P.
Based on the previous proof Appendix B we know that:
redduc)) C col(Wo).
This implies that W reduce retains the original model's learned features but in a reduced subspace.
Since Q is orthogonal (from P QR.,. we have the column space of QR; col(Q col(Q). The column space of Q depends on the learned matrix P. P evolves to include directions outside col(Wo), then col(Q) col(W.).
The column space of Winaa is the sum of the column spaces of W Weducee and Q Rt:
col(W = reducc)))))) + col(QR).
Substituting the results from the previous steps:
col ww) C col(W)) + col(Q).
(Q) L co). then the sum (Wl( + col(Q) is strictly larger than co). This occurs when P (and hence Q) is trained on task-specific data to capture directions outside the original subspace of Wo-
LoRA parametrizes weight updates AW = AB, where A and B are low-rank. In this case, QR serves role of AW, with Q acting as orthonormal bases and R as adaptable coefficients. Both methods enable adaptation by expanding the column space with low-rank updates.
Conditions:
B is initialized to zero but fine-tuned on task-specific data.
Gradient updates allow B (and Q) to explore directions beyond col(Wo), breaking the initial assumption that col(Q) S col(W)).
B E Rddr, with < d, ensuring parameter efficiency while still enabling the expansion of the column space.
By adding QR W Weduuee the total column space col(W becomes:
col(Wooal cos + col(QR) col(Wo) col(Q).
col(Q) & col(Wo), the subspace is extended, allowing adaptation to new tasks. This mirrors
LoRA's mechanism, where low-rank updates inject task-specific directions into the weight matrix.
D Decomposing weight matrix
1. Decomposing single column wi.
Let w; 1m be any column of W and let P QQ denote the orthogonal projector span(Q) C mm
1. Parallel part (projection onto span(Q)).
P pu; QQQ w.) e sQan(Q).
Here QT w; R" is the coordinate vector of w; in the basis Q; multiplying by Q lifts those coordinates back to the ambient space R.
2. Orthogonal part (component orthogonal to sQan(Q)).
(I P)wi, with Q P)w; 0.
3. Reconstruction.
wi Pwi + (1 P)w.
15

---

Geometrically, P wy is the foot of the perpendicular dropped from onto SQan(Q); the residual vector (I P)w; completes the decomposition.
Extending the idea to the whole matrix W.
Write W
W1 Wn ]- Applying the column-wise decomposition to every yields
W Pw1 Pw2 Pwn] ] + [(T P)w1 (T P)w2 ... (T - P)wn
P [[]] W2 ... W,, 1 (I P)[w) W1 W2 ... wA
PW + P)W.
Because p= QQ", the first term can be expressed more compactly:
PW PW QQQ w):
so the final matrix decomposition is
W w) + (I QI-QQ W
We replacing this Q' W with trainable low rank matrix R that extends flexibility adapt for new tasks, keeping W weights frozen. Hence, the final equation will become:
W QR + (1 )Q
16

---

Supplementary Material: Emergent Properties of Efficient Fine-Tunee Tane-Tuninggu in Text-to-Image Models
Dream Branch Plus Comparison: 1 http::///aaoyyooossrrrbbbbaaaaaa www.ii...eeetteeee.eee app
% Ommigen-Visualcle http::////aaoyyyooooooo//eee/e/a
% InsT Objects Qualitative Results:
S1 Further Quantitative and qualitative results
S1.0.1 Image generation consistency
In this section, we present a detailed comparison of various fine-tuning methods for different subjects using the CLIP-image score on the Dreambooth dataset, as shown Table The methods include the proposed DEFT, PaRa [4] and LoRA [6] alongside previous approaches such as Texture Inversion and DreamBooth. The table compares performance across three distinct subject categories: SEAR_PLUSIEE
CAT, and DOG8. Our proposed DEFT method, with rank of 4, achieves high CLIP-image scores of 0.8339 for
0.9280 for CAT, and 0.8721 for DOGS. Notably, PaRa with rank 4 shows slightly improved results, especially for DOG8, with score of 0.8838. In contrast, LoRA methods, particularly with ranks 4 and 8, show lower performance scores, especially for BEAR PLUSHIE. Compared to previous methods such as PaRa [4] and SVDIFF [14] proposed methods, DEFT, exhibit competitive or superior results in terms image-text alignment across all subject categories, underlining their effectiveness for multimodal image generation tasks.
"A photo of [V]"
BEAR PLUSHIE
CAT
DOG8
DEFT (rank=8) (Our)
0.8415
0.9504
0,8882
DEFT (rank=4) (Our)
0.8339
0.9280
0.8721
PaRa
4
(rank=4)
0.8271
0.9315
0,8780
PaRa 6 (rank = 8)
0,8051
0.9467
0.8955
LoRA 6 (rank=4)
0.7741
0.8057
0.7773
LoRA
34 (rank = 8)
0.7943
0.8583
0.8295
SVDIFF 14]
0.7818
0.8854
0.8363
DREAMBOOTH 34
0.7921
0.8893
0.8392
TEXTURE INVERSION
9
0.7421
0.8048
0.7432
Table 7: Comparison of Various Methods for Different Subjects Using Clip-Image Score on Dream- booth Dataset: The table presents the comparison of different fine-tuning methods on the Dreambooth dataset, evaluated using the CLIP-image score. It highlights the performance of the proposed DEFT, PaRa, and LoRA methods against previous approaches, including Texture Inversion and DreamBooth, across multiple subject categories like BEAR PLUSHIE, CAT, and DOG8.
$1.0.2 Qualitative comparison
Furthermore, the Figure presents presents qualitative results comparing different fine-tuning strategies and DEFT applied to the Unified Omnigen model. showcases various image generations of a dog across different environments and prompts, including a lush green field, beach, a snowy landscape, a cityscape, a garden, and a forest. Each model--Base, LoRA, PaRa, and DEFT produccss distinct results, emphasizing how these fine-tuning methods affect image generation based specific prompts. The outcomes demonstrate the ability of these techniques to enhance the generalization and adaptivity of the model while maintaining high-quality, realistic results. The comparisons underline the impact of efficient fine-tuning in improving the model's 's ability to generate diverse, accurate images across various scenarios.
Furthermore, the Figure 9 demonstrates the model's impressive ability to generalize across wide range of unseen prompts. The image features four different outputs generated based on distinct themes: abstract, fantasy, futuristic, and historical prompts. Despite the varied nature of the inputs, the model consistently produces high-quality results, showing its adaptability to different styles
17

---

Concept
Prompts
realistic photo
close-up photo of sks dog <ing Lnage_ [1ogg__________0 j></ing in Kitchen with warn Lighting", photo of sks dog <Lnges [Laaee_ je</ing= on a snawy rountain top
<iagee] Lsage, A/s//ag>>
photo sks lodog <ing-s [mage__ Lngg *leating
pool surrounded by patn trees", futuristic
"an rendering
dog <iapes 1nss/g/
night", Lights",
dog <img-c] Image_ pps/tago
with
deserted Island with Clear blue sky and coln see". photo of sks dog <impse laage, 3 inp/ vurrounded by wstuen leaves in a forest",
dog <imppi] image, 3 anpe
"a styllzed photo of sks <ingre] inage (></ing> on rainy day with raindrops visible", dog <ingos] Lmage ///e/ lago
draratic shadows"
Base
Lora
PaRa+Uncod
PaRa
DEFT
Figure 8: Qualitative Results on Unified Omnigen Model Comparing Efficient Fine-Tuning and DEFT: This figure presents qualitative results comparing efficient fine-tuning strategies and DEFT on the Unified Omnigen model. The outcomes demonstrate the capability of these techniques to enhance model generalization and adaptivity while maintaining high-quality results.
and concepts. The figure emphasizes the model's versatility, highlighting its capacity to maintain visual coherence and output quality across diverse scenarios, from abstract landscapes to historical depictions. This illustrates the robustness of the model in handling various types of prompts while ensuring consistency in the final image outputs.
$1.0.3 Qualitative and quantitative differences
The qualitative and quantitative comparison between the methods DEFT and LoRA, as shown in both the table 8 and the images TO on dreambench plus (31) with SDXL. 322 finetuning, reveals distinct strengths for each model in generating images of cats and horses From the quantitative perspective. LoRA consistently achieves higher scores across DINOv1 I and DINOv2 for both the Kitten and HORSE images. For example, LoRA outperforms DEFT in DINOv1 and DINOv2 for the Kitten image (83.5538 vs. 79.9416 for DINOv1, and 72.2653 vs. 65.0358 for DINOv2), and similarly for the HORSE images (83.5538 vs. 77.8501 for DINOv1, and 72.2653 vs. 65.0358 for DINOv2). These results suggest that LoRA is better at capturing complex features and achieving higher-quality representations in these metrics, which might reflect its greater flexibility and artistic adaptability.
In contrast, DEFT demonstrates stronger performance in CLIP-I and CLIP-T for some images, especially for the Kitten images (83.5867 for DEFT vs. 81.3446 for LoRA in CLIP-I), indicating its ability to produce more realistic, detailed representations that preserve the original essence of the
18

---

Unseen prompt abstract
Unseen prompt fantasy
Unseen prompt futuristic Unseen prompt historical
Figure 9: Diverse Prompt Generalization: This figure shows the generalization capabilities of the model across diverse prompts, emphasizing its ability to handle variety of inputs while maintaining consistent output quality.
DEFT
LoRA
Figure 10: Qualitative comparison of the DEFT and LoRA methods for generating images of cats and horses. The first row shows images of cats, while the second row shows horses. DEFT produces realistic, detailed images, while LoRA introduces more artistic and stylized elements, showcasing its flexibility in adapting creative representations.
animals. DEFT's output tends to have clearer textures, sharper details, and more lifelike features, showcasing its strength in realism and faithful reproduction.
The images generated by DEFT are more grounded in reality, with clear textures and natural settings. In the case of the HORSE images, DEFT retains more authentic anatomical features and textures, reflecting a more realistic depiction the animals. On the other hand, LoRA brings more artistic flair to the HORSE images, with creative use of colors, dynamic poses, and abstract elements. While LoRA's outputs are more vibrant and imaginative, they may not always preserve the natural look and feel of the animals consistently as DEFT does.
Despite LoRA's higher scores in DINOv1 and DINOv2, these results do not fully capture its ability to maintain realistic features across all images. LoRA excels in producing creative, artistic rep- resentations, but at the cost of some consistency in realism. DEFT, with its emphasis on realism, demonstrates more stable, high-fidelity outputs, particularly for complex subjects like horses.
This analysis shows that while LoRA excels artistic flexibility and creative interpretation, achieving higher DINOv1 and DINOv2 scores, DEFT remains superior in generating more realistic and detailed images. The choice between the two methods ultimately depends on the desired outcome outthme-whether the goal is to prioritize artistic creativity or to maintain realistic accuracy.
whether
S1.0.4 Scene personalization
Figures and showcase the model's scene personalization capabilities, demonstrating its profi- ciency in generating high-quality visual content with specific environmental characteristics.
19

---

Method
Image
DINOv1
DINOv2
CLIP-I
CLIP-T
DEFT
Kitten
79.9416
73.5553
83.5867
35.5409
DEFT
Stork
67.9085
62.7926
76.3276
36.8799
DEFT
Kitten 2
77.8501
65.0358
77.6003
36.7579
DEFT
HORSE
77.8501
65.0358
77.6003
36.7579
LoRA
Kitten
82.9492
77.9071
86.7823
33.7977
LORA
Stork
70.1220
61.8301
74.9231
36.9636
LoRA Kitten 2
83.5538
72.2653
81.3446
34.1923
LoRA HORSE
83.5538
72.2653
81.3446
34.1923
Table 8: Comparison of DEFT and LoRA across multiple evaluation metrics (DINOvI, DINOv2,
CLIP-I, CLIP-T) for images of cats and horses. The table highlights how LoRA consistently achieves higher scores in DINOv1 and DINOv2, indicating its strength in capturing complex features, while DEFT excels in CLIP-I and CLIP-T, reflecting its focus on realism and detailed preservation of the original subjects.
Figure I1 titled Church Rock Scene Personalization, illustrates how the model adapts the Church Rock scene various settings. These scenes include dynamic backgrounds, such as pool surrounded by palm trees, a futuristic city at night, snowy mountain top, crowded street market, and forest with autumn leaves. Each personalized scene is result of fine-tuning, reflecting how the model can generate diverse visual representations of the same object in unique environments, showcasing its flexibility in handling scene-specific details.
Figure 12 titled Table Scene Personalization, further exemplifies the model's ability to adapt to specific environments. In this case, the model personalizes simple table scene by generating various configurations of objects like bottles and caps, based on detailed prompts. The generated scenes show different bottles, one filled with orange liquid and another empty, both with distinct cap colors. This reflects the model's ' ability to generate high-quality content by adapting to specific setups, maintaining both object clarity and spatial coherence within the scene.
Flouting pool surrounded by palm troes
artistic rendering snowy mountain top A. street Surrounded by auturnn in funuristic city night
market with vibeant leaves in forest colors and lights on a beach during sunset
Figure 11: This figure illustrates the scene personalization capabilities of the model using the church rock scene, showcasing how fine-tuning allows for detailed control over scene- seenesspccific characteristics.
20

---

Standing bottles
Irarapureni plastic botile filled with orange sits upright cap.
transpucnnt plastic boille cupply sits upright with red cap
Figure 12: Table Scene Personalization: The figure demonstrates how the model personalizes a table scene, reflecting the ability to adapt and generate high-quality visual content in specific environments.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the key contributions of the paper, including the introduction of the DEFT framework, its focus on efficient fine-tuning with minimal data, and its ability to handle personalization and multi-task composition.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification: The paper does not explicitly discuss the limitations of the proposed DEFT framework. There is such limitation of DEFT similar lora, but would still like to discuss the supplementary due to limited space.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors
21

---

should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't ' acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The paper includes well-defined theoretical results, supported by clearly stated assumptions, theorems, and formulas. All key equations are numbered and cross-referenced throughout the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our framework is simple and can be written from seratch using PyTorch with using PEFT library. It can be simple used in place of LoRA.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
22

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Everything we be available GitHub and Huggingface online for repro- ducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swi.inosiiippccyc for more details. //ipss.c/pubbll http:///cccggguuueeee////////e// hwcpi///giddsse/eoo Caaeeee//ooSSSmmeieeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
23

---

similar framework as LoRA which make it easier to Answer: [Yes]
Justification: We only used online available benchmarks for the dataset which have these availabilities. For experiments, reproduce.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our work is mainly related image generation. We used which has determin- istic measures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We only used GPUs throughout all the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
24

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp//s/eearrps..ccc/p//////eha
Answer: [Yes]
The research in the paper adheres to the NeurIPS Code of Ethics, ensuring transparency, fairness, and respect for participants' rights. The experiments conducted are in accordance with established ethical standards, and the paper does not involve any unethical practices, such as misrepresentation of results or the use of biased datasets.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Image generation have positive and negative impact, but mainly targeted the fine-tuning approach, which mainly depends onthe problem we are solving.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We do not describe in the main paper but we will certainly release with the supplemntry.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper properly credits the creators and original owners of all assets used, including code, data, and models, evident in the citations for methods such as DreamBooth, LoRA, and OmniGen. The license and terms of use for these assets are explicitly referenced in the relevant sections, ensuring that the work complies with the usage rights of these resources.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets, including the DEFT framework and its associated implementation code. These assets are well-documnnteed, with clear explanations provided alongside the release. Additionally, the paper includes appropriate references and acknowledgements for any third-party resources used.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
26

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We only used the public dataset and Ilm to generate the instructions.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
The core methods in this research do not involve large language models (LLMs) important, original, non-standard component. LLMs were not used in the development the core methodology or in any original research aspects, and their usage is not central to the scientific rigor contributions of the paper. The L.LMs used for writing, editing, or formatting do not impact the overall methodology.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: / httpa////eeeipp..p.coeeeeeeerre ccc/oonferencees/225///i for what should or should not be described.
27


Reviews:
[{"review_id": "OnQDy4PrY1", "reviewer": "Reviewer_wNps", "summary": "This paper introduces a new method called ﻿Decompositional Efficient Fine-Tuning (DEFT) to efficiently fine-tune pre-trained text-to-image models. Following the spirit of the commonly adopted low-rank update approach, DEFT further decomposed the update into two components, namely the projection onto a subspace and the low-rank adjustment. Both quantitative and qualitative experiments are performed to validate the efficacy of the proposed approach in performing highly efficient model fine-tuning that achieves faithful personalization without compromising the original capabilities of the pre-trained model.", "questions": "1. In Section 4.2.2, are ﻿other competing methods (e.g., OmniGen) fine-tuned on the ﻿VisualCloze dataset? If so, which fine-tuning technique is utilized to fine-tune them? This should be provided in detail.\n2. In Section 4.2.4, the authors claim that the proposed method exhibits a certain kind of emergent properties, which is emphasized in the paper title. However, the experimental results are inadequate. More generation results (e.g., handling completely new tasks/data) should be provided to support the claim.\n3. Are there any representative failure cases? Some results and discussions regarding this should be provided for a better understanding of the limitations of the proposed method.", "limitations": "Some representative failure cases are supposed to be given and the authors should provide some discussions on the limitations of the proposed work as well.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. Fine-tuning pre-trained text-to-image models is a challenging task. A wide range of prior works suffers from striking a balance between adaptation to new data and preserving the pre-learned knowledge. To alleviate this, this work explores a novel low-rank update scheme, which I believe could be of great interest to the text-to-image generation community.\n2. The proposed approach is simple yet effective. The authors also provide ﻿rigorous calculation and analysis to ﻿provide theoretical guarantees on the efficacy of the proposed method.\n3. Experimental results are impressive. Extensive evaluations demonstrate that the proposed model obtains better performance compared to previous fine-tuning methods in terms of﻿generation flexibility and model generalization.\n4. The motivation of this work is clearly elaborated and the paper is well-written and easy to follow.\n\n## Weaknesses\n1. For the literature review part, this paper lacks the introduction of another line of work that adopts training-free strategies to adapt the pre-trained model to new datasets or tasks.\n﻿2. As for the ablation study, some quantitative evaluation results should be given for a better investigation of the effect of decomposition.\n3. For the generation results showcased in Figure 3, I personally think that the performance of LoRA and DEFT is quite close. Maybe the authors could conduct a user study for a more comprehensive assessment of different methods.\n4. This paper lacks discussions of the convergence speed of different fine-tuning methods, which I believe is a key factor reflecting the performance.", "was_revised": true, "final_justification": "This is a solid work that exploits a novel method to fine-tune pre-trained text-to-image models. The authors addressed my major concerns in their rebuttal. Hence, I recommend acceptance of this paper.", "rebuttal": "### Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment.\n### **1. Literature Review on Training-Free Strategies**\n\n**Response:** Thank you for this valuable suggestion. We acknowledge that training-free approaches represent an important category in model adaptation. Our current manuscript includes several training-free methods such as Textual Inversion variants, ELITE, and InstantBooth that control generation through text-encoder modifications.\n\nHowever, these training-free methods have significant limitations compared to DEFT. They struggle with extending to new concepts, especially out-of-distribution images, and rely on conditional components that are incompatible with unified models where image and text tokens are processed through unified self-attention. Additionally, many require per-image optimization, resulting in higher inference times. DEFT achieves superior subject fidelity with CLIP-I scores of 83.59 vs 74.21 compared to Textual Inversion variants on the DreamBooth Plus dataset (please see table 1 in the supplementary material).\n\n### **2. Quantitative Ablation Study on Decomposition Effects**\n\n**Response:** We provide a comprehensive quantitative evaluation of different decomposition methods with finetuning omnigen on sks dog from dreambooth dataset for 200 epochs each:\n\n| Method       | CLIP-I            | CLIP-T            | DINO-V1           | Aesthetic         | Sharpness         |\n|--------------|------------------|-------------------|-------------------|-------------------|-------------------|\n| LRMF         | 0.8270 ± 0.0380  | 0.2200 ± 0.0505   | 0.3070 ± 0.0397   | 0.0143 ± 0.0047   | **348.62 ± 414.16**   |\n| NMF          | 0.8834 ± 0.0328  | 0.2224 ± 0.0417   | 0.3569 ± 0.0683   | 0.0154 ± 0.0031   | 206.06 ± 90.52    |\n| QR           | 0.8747 ± 0.0428  | 0.2168 ± 0.0411   | 0.3395 ± 0.0601   | **0.0166 ± 0.0025**   | 215.45 ± 105.98   |\n| TSVD         | 0.8753 ± 0.0305  | 0.2234 ± 0.0408   | 0.3334 ± 0.0617   | 0.0156 ± 0.0035   | 331.29 ± 134.12   |\n| **Relexing P** | **0.9232 ± 0.0370** | **0.2660 ± 0.0332** | **0.4403 ± 0.0960** | 0.0157 ± 0.0025 | 175.22 ± 25.13 |\n\nRelexing P achieves the highest CLIP-I and DINO-V1 scores, while all methods maintain consistent performance levels, demonstrating the robustness and flexibility of our framework. QR maintains quality, strikes a balance between quality and personalization.\n\n### **3. Visual Comparison and User Study**\n\n**Response:** We agree that visual quality appears similar in Figure 3, but DEFT demonstrates superior instruction-following capabilities. We conducted a comprehensive user study with 16 participants comparing PARA, LoRA, and DEFT across different scenarios:\n\n| Method | Average Score | Best Score | Worst Score | Win Rate |\n|--------|---------------|------------|-------------|----------|\n| PARA | 7.64 | 8.75 | 6.65 | 25.0% |\n| LoRA | 7.60 | 8.80 | 6.84 | 12.5% |\n| **DEFT** | **8.20** | **9.50** | 6.78 | **62.5%** |\n\nAdditionally, we conducted a Claude Sonnet evaluation across multiple metrics:\n\n| Metric | LoRA | PARA | DEFT |\n|--------|------|------|------|\n| Task Achievement | 8.3 | 6.8 | **8.7** |\n| Subject Consistency | 8.2 | 6.4 | **8.8** |\n| Completion Quality | 8.4 | 7.1 | **8.6** |\n| Visual Fidelity | 8.1 | 7.3 | **8.5** |\n| Prompt Adherence | 8.3 | 6.9 | **8.6** |\n| **Average** | **8.26** | **6.90** | **8.64** |\n\nDEFT consistently outperforms both methods across all evaluation criteria.\n\n### **4. Convergence Speed Analysis**\n\n**Response:** Thank you for highlighting this important aspect. We conducted a convergence analysis comparing DEFT and LoRA at different training steps on the full Dreambooth (30 concepts) on omnigen:\n\n| **Method** | **Steps** | **CLIP-T**           |\n|------------|-----------|----------------------|\n| **DEFT**   | 2000      | 0.3019 ± 0.0264      |\n|            | 8000      | **0.3187 ± 0.0248**  |\n| **LoRA**   | 2000      | 0.2856 ± 0.0329      |\n|            | 8000      | 0.2194 ± 0.0230      |\n\nDEFT shows consistent improvement across all metrics and achieves better convergence characteristics. Our method modifies the column subspace of the pretrained model, providing advantages for convergence with both large and small image datasets.\n\n**Convergence Characteristics:**\nBecause the baseline model is very large (around 3.8 B), it is difficult to prove the convergence in a small dataset. We found that training loss is very fluctuating for all the cases. Here are some insights:\n| Method | Aesthetic Score | CLIP-I | Best Loss (at epoch) |\n|--------|----------------|--------|---------------------|\n| **DEFT** | 0.0109 ± 0.0093 | 0.3019 ± 0.0264 | **0.0240 (epoch 4)** |\n| **LoRA** | 0.0154 ± 0.0052 | 0.2911 ± 0.0311 | 0.0482 (epoch 27) |\n| **PaRa** | 0.0112 ± 0.0093 | 0.2710 ± 0.0401 | 0.0310 (epoch 12) |\n\nDEFT achieves the lowest training loss (0.0240) significantly faster, at epoch 4, compared to LoRA, which requires 27 epochs, and PaRa, which requires 12 epochs. Note that faster convergence does not guarantee the quality of image generation, and the performance is also not comparable, so scores are calculated after 200 epochs only. The faster convergence stems from our method's ability to modify the column subspace of the pretrained model, providing inherent advantages for both large and small image datasets.\n\n### **5. VisualCloze Fine-tuning Details**\n\n**Response:** Thank you for this clarification request. We used the same base models as VisualCloze for evaluation. OmniGen was not fine-tuned on the full VisualCloze dataset as it leverages in-context learning for generalization. For fair comparison, we fine-tuned both DEFT and LoRA on the depth estimation task using 200k image-instruction pairs:\n\n| Metric | DEFT | LoRA |\n|--------|------|------|\n| **DINOv1 Score** | 89.31 | 89.08 |\n| **DINOv2 Score** | 83.40 | 82.31 |\n| **CLIP-I Score** | 93.70 | 92.93 |\n| **CLIP-T Score** | 27.65 | 27.61 |\n\nDEFT consistently outperforms LoRA across all metrics in this controlled comparison.\n\n### **6. Emergent Properties Evidence**\n\n**Response:** We appreciate this feedback and provide clarification of our emergent properties definition. By emergent properties, we refer to capabilities that arise from DEFT's dual decomposition structure without explicit training for those specific combinations.\n\nOur existing evidence demonstrates multi-concept composition from single-concept training data, as shown in Figure 5 where individual concepts (single dog images, separate human images) combine to create complex spatial reasoning scenarios. The VisualCloze results further demonstrate that a single DEFT adapter handles over 8 distinct tasks with unified capability and emergent transfer between modalities.\n\nWe acknowledge this could be presented more comprehensively and will strengthen this section with additional generation results showing completely new task combinations.\n\n### **7. Failure Cases and Limitations**\n\n**Response:** We appreciate this important question. Our primary limitation occurs with complex scene compositions involving multiple novel elements. For example, prompts like \"generate Einstein + new features + dog + car + tree + house in a kitchen setting\" can produce mixed or blended features when the original model and DEFT training dataset haven't encountered such complex combinations.\n\nThe quality bottleneck often stems from limited instruction dataset coverage. When attempting to compose highly complicated combinations of objects, spatial relationships, and contexts, the method may struggle with feature disambiguation. We will expand this analysis in the main paper and provide a more comprehensive discussion of limitations and societal impacts."}, {"review_id": "m8irY0rAjb", "reviewer": "Reviewer_7bp5", "summary": "This paper introduces Decompositional Efficient Fine-Tuning (DEFT) for the parameter-efficient fine-tuning of text-to-image models. DEFT proposes modifying a pre-trained weight matrix by decomposing its update into two trainable components: (1) a projection that removes components of the original weight matrix along a low-rank subspace, similar to PaRa, and (2) a flexible low-rank update, via a second matrix `R`, that adds new capabilities to the model. The authors argue this formulation provides a better balance between learning new concepts (personalization) and retaining the general knowledge of the pre-trained model, thereby improving adaptability and reducing overfitting.", "questions": "1. Motivation for the `R` matrix: Could you provide a clearer, more intuitive explanation in the main text for why adding the second trainable matrix `R` is necessary? While the mathematical proof is in the appendix, a high-level motivation for how it enables learning \"new tasks\" beyond PaRa's capabilities would improve the paper's clarity.\n2. Comparison to OmniGen is a bit misleading : In Section 4.2.2, DEFT is fine-tuned on the OmniGen model and then compared to a \"base OmniGen\" on the VisualCloze dataset. To ensure a fair comparison, could you clarify if the \"base OmniGen\" baseline was also fine-tuned on the VisualCloze training set, or if it was the pre-trained model evaluated zero-shot?\n3. Explanation for poor FID scores: Could you provide an explanation for why DEFT's FID scores in Table 3 are consistently worse than most other methods? Does the DEFT formulation present a trade-off where it improves in some areas (like SSIM) at the cost of image fidelity as measured by FID?", "limitations": "The authors did not include a dedicated limitations or societal impact section in the main body of the paper. This is a significant omission. A thorough discussion should have been included, acknowledging:\n* The lack of quantitative analysis on computational overhead (inference speed, memory).\n* The potential for the method to be used for malicious purposes, such as generating deepfakes, which is a standard concern for all powerful generative models.\n* A more thorough discussion of failure cases or scenarios where DEFT might not perform as well as other methods.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n* Strong qualitative results: The paper presents compelling qualitative results across a wide range of tasks, including subject personalization, style transfer, and multi-concept composition. The images in the paper and supplementary material effectively showcase the method's ability to produce high-fidelity and contextually plausible images.\n* Extensive experiments: The authors validate DEFT across multiple datasets (DreamBooth, DreamBench Plus, VisualCloze, InsDet), demonstrating its versatility in both personalization and universal generation settings.\n* Methodological clarity and reproducibility: The paper provides the mathematical formulation for DEFT, with proofs included in the appendix. The authors have also provided links to supplementary websites and provided codes in the supplemental materials, which greatly aids in reproducibility, though I haven’t tested the codes by myself.\n\n### Weaknesses\n\n* Unsupported title and central claim: The paper's title, \"Emergent Properties of Efficient Fine-Tuning,\" is a significant overstatement. The paper only includes a small, qualitative subsection (4.2.4) to this topic, which essentially re-labels generalization as an \"emergent property\" without a clear definition or rigorous analysis. The grand claim in the title is not sufficiently supported by the paper's content.\n* Unsubstantiated claims about decomposition methods: In Section 3.1.1, the paper proposes that using different matrix decomposition techniques (e.g., SVD, NMF) could offer unique benefits for different tasks. However, this claim is not backed by experimental evidence. The only comparison in Figure 6 shows all methods produce very similar results on a single task, failing to demonstrate any distinct advantages.\n* Lack of inference time analysis: A core motivation for PEFT methods is efficiency. The paper claims DEFT is \"efficient\" but provides no quantitative analysis of its inference latency or memory overhead compared to LoRA, PaRa, or other baselines. Without this data, the efficiency claims are unsupported.\n* Unexplained poor performance on key metrics: In Table 3, the FID scores for DEFT, which measure image realism, are notably worse than most competing methods like OmniControl and ControlNet. The paper offers no explanation for this underperformance, instead focusing on other metrics where it performs better.\n* Missing comparisons to key literature: The paper fails to discuss or compare its method against other recent and highly relevant fine-tuning techniques. For instance, DoRA [1] and OFT (Orthogonal Finetuning) [2] are sophisticated methods that also aim to improve upon LoRA, making them critical points of comparison that are missing from the related work and experiments.\n\n### References\n\n[1] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen: DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024\n\n[2] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, Bernhard Schölkopf: Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS 2023", "was_revised": true, "final_justification": "The authors' responses addressed my concerns. Therefore, I increased my score from 4 to 5.", "rebuttal": "### Thank you for providing your valuable and constructive feedback!\n### **1. Title and Central Claims**\n\n**Response:** We acknowledge this valid concern and agree the current title overstates our claims relative to the experimental evidence provided. While Section 4.2.4 demonstrates genuine emergent behavior such as multi-concept spatial composition from single-concept training (Figure 5) and single adapter handling 8+ distinct tasks through unified training (VisualCloze results), we recognize this evidence requires more rigorous quantitative analysis to support such prominent title claims.\n\nWe propose revising the title to focus on our core technical contribution: **\"Dual-decomposition Column Space Extension for Fine-tuning of Text-to-Image Models\"**. This better reflects our mathematical foundation where column space extension col(W_total) = col(W_reduce) + col(QR) enables the demonstrated capabilities without overstating the emergent properties aspect.\n\n### **2. Decomposition Methods Analysis**\n\n**Response:** We provide a comprehensive quantitative evaluation demonstrating distinct performance characteristics across decomposition methods:\n\n| Method | CLIP-I | CLIP-T | DINO-V1 | Aesthetic | Sharpness |\n|--------|--------|--------|---------|-----------|-----------|\n| LRMF | 0.8270 ± 0.0380 | 0.2200 ± 0.0505 | 0.3070 ± 0.0397 | 0.0143 ± 0.0047 | 348.62 ± 414.16 |\n| NMF | 0.8834 ± 0.0328 | 0.2224 ± 0.0417 | 0.3569 ± 0.0683 | 0.0154 ± 0.0031 | 206.06 ± 90.52 |\n| QR | 0.8747 ± 0.0428 | 0.2168 ± 0.0411 | 0.3395 ± 0.0601 | 0.0166 ± 0.0025 | 215.45 ± 105.98 |\n| TSVD | 0.8753 ± 0.0305 | 0.2234 ± 0.0408 | 0.3334 ± 0.0617 | 0.0156 ± 0.0035 | 331.29 ± 134.12 |\n| Relexing P | 0.9232 ± 0.0370 | 0.2660 ± 0.0332 | 0.4403 ± 0.0960 | 0.0157 ± 0.0025 | 175.22 ± 25.13 |\n\nRelexing P achieves the highest CLIP-I and DINO-V1 scores, while all methods maintain consistent performance levels, demonstrating the robustness and flexibility of our framework. QR maintains quality, strikes a balance between quality and personalization.\nThe results show NMF excels in image-text alignment and feature representation, while QR provides the best aesthetic quality (0.0166). This validates our claim that different decomposition methods offer distinct advantages for specific tasks.\n\n### **3. Inference Efficiency Analysis**\n**Response:** We acknowledge that comprehensive efficiency claims should include detailed training and inference analysis. We conducted extensive benchmarks comparing DEFT against LoRA and PaRa across different configurations:\n\n**Training Efficiency Analysis:**\n\n**Rank 64 Configuration (Batch size = 2):**\n\n| Method | Steps/Sec | Peak Memory (GB) | Trainable Params | CLIP-I Score | \n|--------|-----------|------------------|------------------|--------------|\n| **LoRA** | **12** | 8.03 | 37,748,736 | 0.8757 | \n| **DEFT** | **11** | **7.95** | 37,748,736 | **0.8834** | \n| **PaRa** | 8 | **7.86** | **25,165,824** | 0.8271 | \n\n**Key Performance Insights:**\n\n1. **Training Speed Hierarchy:** DEFT shows moderate overhead (23% slower than LoRA at rank 8, 8% slower at rank 64)\n\n2. **Memory Efficiency:** \n   - At rank 8: DEFT uses 7.71GB vs LoRA's 7.72GB (marginal improvement)\n   - At rank 64: DEFT uses 7.95GB vs LoRA's 8.03GB (1% improvement)\n3. **Parameter Efficiency Trade-offs:**\n   - DEFT achieves competitive parameter efficiency without sacrificing performance\n\n**Scalability Analysis:**\nThe performance gap between DEFT and LoRA decreases at higher ranks (23% → 8% slowdown), suggesting DEFT's overhead becomes relatively smaller for larger adaptations. This indicates favorable scaling properties for complex fine-tuning scenarios.\n\nThe efficiency analysis reveals DEFT occupies a sweet spot: trading minimal speed for substantial gains in memory efficiency and output quality, making it optimal for deployment scenarios where resource constraints and quality requirements are balanced priorities.\n\nFurthermore, we conducted extensive benchmarks (928 configurations) comparing DEFT against LoRA and other baselines:\n\n**Inference Efficiency Benchmark (Omnigen 3.8B model 1 layer):**\n| Method | Inference Latency (ms) | Parameter Efficiency | CLIP-I Score | DINO-V1 Score |\n|--------|------------------------|---------------------|--------------|---------------|\n| **LoRA** | **13.55** | 5.18% | 0.8757 | 0.8295 | \n| **PaRa** | 16.51 | **2.63%** | 0.8271 | 0.8780 | \n| **DEFT (fused)** | 15.08 | 5.18% | **0.8834** | **0.8882** | \n| **DEFT (cached)** | 14.2 | 5.18% | **0.8834** | **0.8882** | \n\n## **Decomposition Method Performance vs Speed Analysis**\n\n| Method | Speed (ms) | CLIP-I | CLIP-T | DINO-V1 | \n|--------|------------|--------|--------|---------|\n| **QR** | **5.38** | 0.8747 ± 0.0428 | 0.2168 ± 0.0411 | 0.3395 ± 0.0601 | \n| **NMF** | **5.16** | **0.8834 ± 0.0328** | 0.2224 ± 0.0417 | **0.3569 ± 0.0683** | \n| **Eigen** | 10.87 | 0.8738 ± 0.0298 | **0.2345 ± 0.0411** | 0.3304 ± 0.0580 | \n| **TSVD** | 28.72 | 0.8753 ± 0.0305 | 0.2234 ± 0.0408 | 0.3334 ± 0.0617 | \n| **Relaxing P** | **5.22** | 0.8738 ± 0.0298 | **0.2345 ± 0.0411** | 0.3304 ± 0.0580 | \n|--------|------------|--------|\n| Relexing P | \t5.22ms | \t2x faster | \n| orthogonalized mean | \t12.51ms | \tExpensive| \n\n**Key Findings:** DEFT achieves parameter efficiency of 5.18% while maintaining inference speed within 20% of LoRA. With precomputed caching of (I - QQ^T)W during deployment, DEFT's inference cost reduces to within 10-20% of LoRA while preserving superior task performance.\n\n### **4. Missing Literature Comparisons**\n\n**Response:** Thank you for this important suggestion. We have extended our comparisons to include OFT and acknowledge DoRA's relevance:\n\n**Extended Comparison with OFT:**\n\n| Method | BEAR_PLUSHIE | CAT |\n|--------|--------------|-----|\n| **DEFT (rank=8)** (Ours) | **0.8415** | **0.9504** |\n| DEFT (rank=4) (Ours) | 0.8339 | 0.9280 |\n| PaRa (rank=4) | 0.8271 | 0.9315 |\n| LoRA (rank=4) | 0.7741 | 0.8057 |\n| OFT (Block=4) | 0.785 | 0.8559 |\n| SVDIFF | 0.7818 | 0.8854 |\n| DREAMBOOTH | 0.7921 | 0.8893 |\n\nOFT requires additional orthogonality constraints during training (R^T R = RR^T = I, ||R - I|| ≤ ε) which increases computational overhead, whereas DEFT requires no additional constraints. Regarding DoRA, while it decomposes weight updates into magnitude and direction components (W' = W + m * (V * B^T / ||V * B^T||)), it requires tuning magnitude scaling and direction learning rates, unlike DEFT's parameter-free approach. We will incorporate these methods in the expanded Section 2.\n\n### **5. R Matrix Motivation**\n\n**Response:** DEFT's innovation lies in its dual learning approach. The projection term (I - PP^T)W₀ strategically modifies the pre-trained model's representation space, while the R matrix actively learns new task-specific knowledge to populate that modified space.\n\n**Intuitive Explanation:** Consider the original model with col(W₀) = span{v₁, v₂, v₃}, but a new task requires direction v₄.\n\nWithout R: W = (I - PP^T)W₀ → col(W) ⊆ span{v₁, v₂, v₃} (cannot represent v₄)\n\nWith R: If P learns such that col(P) contains v₄, then W_DEFT = (I - PP^T)W₀ + PR enables col(W_DEFT) ⊆ span{v₁, v₂, v₃} + span{v₄}, where R learns to map inputs to the v₄ direction.\n\nThis transforms adaptation from mere 'knowledge rearrangement' to active 'knowledge extension'.\n\n### **6. OmniGen Comparison Fairness**\n\n**Response:** Thank you for seeking this clarification. We used the same base models as VisualCloze for evaluation. OmniGen was not fine-tuned on the full VisualCloze dataset as it leverages in-context learning for generalization. For fair comparison, we fine-tuned both DEFT and LoRA on the depth estimation task using 200k image-instruction pairs:\n\n| Metric | DEFT | LoRA |\n|--------|------|------|\n| **DINOv1 Score** | 89.31 | 89.08 |\n| **DINOv2 Score** | 83.40 | 82.31 |\n| **CLIP-I Score** | 93.70 | 92.93 |\n| **CLIP-T Score** | 27.65 | 27.61 |\n\nThis controlled comparison demonstrates DEFT's consistent advantages even in fair evaluation settings.\n\n### **7. FID Performance Trade-off Explanation**\n\n**Response:** The FID vs SSIM performance pattern reflects a fundamental design trade-off in DEFT's architecture. DEFT employs a conservative adaptation strategy: W = (I - PP^T)W₀ + PR, where (I - PP^T)W₀ actively preserves pre-trained knowledge while PR enables controlled adaptation.\n\nThis design prevents aggressive style deviations while learning new concepts, resulting in more structurally consistent generations that maintain better SSIM scores but may appear less visually striking to FID's perceptual metrics. DEFT explicitly trades raw generation quality (FID) for structural consistency (SSIM), making it particularly suitable for applications where consistency and controllability are prioritized over pure visual novelty.\n\n### **8. Limitations and Societal Impact**\n\n**Response:** We acknowledge this significant omission and will include a dedicated limitations section addressing:\n\nSocietal Impact: Potential misuse for deepfake generation (standard concern for generative models)\nLimitations: Limited instruction dataset coverage affecting novel combinations\n\nWe will have a separate section for limitations and societal Impact in the revised manuscript. \n### **9. Formatting Issues**\n\n**Response:** We acknowledge the formatting concern regarding supplemental content in the main paper. We will relocate the mathematical proofs for PaRa and DEFT to the supplementary material and ensure strict adherence to formatting guidelines in the revision.\n\nWe appreciate the reviewer's thorough analysis and will address all raised concerns through title revision, expanded efficiency analysis, comprehensive literature comparison, dedicated limitations section, and proper formatting in our revision."}, {"review_id": "kYNz2VCUk7", "reviewer": "Reviewer_wP2X", "summary": "The paper proposes DEFT (decompositional efficient fine-tuning) which resolves the two conflicting objectives: Personalization flexibility and parameter-efficiency for T2I diffusion models. DEFT consists of dual pathway (i) low-rank adapter that projects the frozen weight matrix onto the orthogonal complement of a learned low rank subspace and (ii) low-rank adapter that separately adds into that subspace. \nThis is a combination of LoRA and PaRa. \n\nApplied to SD and the unified Omnige, DEFT supports one-shot subject personalization, multi-concept composition, style transfer, conditional generation (Canny, depth, pose, etc.) and even visual in-context learning. All these objectives are performed with a single adapter, not task-specific LoRAs. They conduct experiments on various benchmarks and show impressive gain in text alignment, controllability and enhanced details compared to LoRA and RAPA. The emergent abilities are shown by the model composing multiple concepts and scenes without additional guidance in train time.", "questions": "(i) Provide ablation results on the components of the methodology. Specifically, there should be evaluation report on using PaRa only, LoRA only, and DEFT.\n(ii) What is the core difference between DEFT and mere composition of LoRA and PaRa?", "limitations": "No, not in the main paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed method, DEFT, might look like a simple composition of LoRA and PaRa but in fact, it introduces a dual architecture: (i) a projection that contributes to generalization, and (ii) subsequent low-rank injection that provides expressivity (task specific direction).\n- The evaluation results are impressive.\n- The method is simple yet powerful.\n\nWeaknesses\n- Ablation on using LoRA / PaRa / Both should be shown in the main paper.", "was_revised": true, "final_justification": "My concern about ablation study on baseline models have been addressed. The paper is a technically solid paper, therefore I maintain my original score.", "rebuttal": "### Thank you for providing your valuable feedback!\n\n### Ablation Studies: Component-wise Analysis\n\n**Response:** We provide comprehensive ablation analysis using our actual experimental results:\n\n**Individual Component Performance (DreamBooth Dataset, CLIP-Image scores):**\n\n| Method | BEAR_PLUSHIE | CAT | DOG8 | Parameters |\n|--------|--------------|-----|------|------------|\n| **PaRa only (rank=4)** | 0.8271 | 0.9315 | 0.8780 | 3,145,728 |\n| **LoRA only (rank=4)** | 0.7741 | 0.8057 | 0.7773 | 4,718,592 |\n| **DEFT (rank=4)** | **0.8339** | **0.9280** | **0.8721** | **4,718,592** |\n| **DEFT (rank=8)** | **0.8415** | **0.9504** | **0.8882** | 7,864,320 |\n\n### Core Difference: Coordinated vs Independent Adaptation\n\n- **PaRa only:** W' = W₀ - QQ^T W₀ \n- **LoRA only:** W' = W₀ + AB^T\n- **Composition:** W' = W₀ - QQ^T W₀ + AB^T *(independent Q, A, B)*\n- **DEFT:** W' = W₀ - PP^T W₀ + PR *(shared P matrix)*\n\n**Parameter Count Comparison:**\n\n| Method | Matrix Components | Total Parameters | Memory Overhead |\n|--------|------------------|------------------|-----------------|\n| **PaRa only** | Q matrix | 3,145,728 | Baseline |\n| **LoRA only** | A, B matrices | 4,718,592 | +50% vs PaRa |\n| **Composition** | Q + A + B matrices | **7,864,320** | **+150% vs PaRa** |\n| **DEFT** | P, R matrices | **4,718,592** | **+50% vs PaRa** |\n\n**Key Insight:** DEFT achieves the benefits of composition with **40% fewer parameters** than naive combination (4.7M vs 7.8M parameters).\n\n**Performance Efficiency Analysis:**\n\nBased on our authentic individual results, we can analyze the efficiency gains:\n\n| Metric | PaRa Only | LoRA Only | Composition | DEFT |\n|--------|-----------|-----------|---------------------------|---------------|\n| **BEAR_PLUSHIE** | 0.8271 | 0.7741 | 0.6832 | **0.8339** | \n| **CAT** | 0.9315 | 0.8057 | 0.7986 | **0.9280** |\n| **Parameters** | 3.1M | 4.7M | **7.8M** | **4.7M** |\n\n**Results:**\n\n| Method | Step Time (ms) | Peak Memory (GB) | Parameter Efficiency |\n|--------|----------------|------------------|---------------------|\n| **LoRA (rank=8)** | 7,426 | 7.72 | 4.7M params |\n| **PaRa (rank=8)** | 10,916 | 7.72 | 3.1M params |\n| **Composition** | 18,342 | 8.5 | **7.8M params** |\n| **DEFT (rank=8)** | **9,145.8** | **7.71** | **4.7M params** |\n\n**DEFT Training Advantage:** 50% faster than expected naive composition while using 40% fewer parameters.\n\n**Architectural Coordination Benefits:**\n\n**Composition Problems:**\n```\nW' = (I - QQ^T)W₀ + AB^T\n     ↳ Q removes information from W₀\n     ↳ A,B add unrelated information\n     ↳ Potential interference between operations\n```\n\n**DEFT Coordination:**\n```\nW' = (I - PP^T)W₀ + PR\n     ↳ P defines removal space\n     ↳ R operates within P's coordinate system\n     ↳ Guaranteed alignment between operations\n```\n\n**Decomposition Results:**\n\nOur actual decomposition ablation demonstrates DEFT's flexibility:\n\n| Method | CLIP-I | Huamn recommandation|  Parameters | Efficiency Ratio |\n|--------|--------|------------|------------------| ------------------|\n| **NMF** | 0.8834 ± 0.0328 | (9.0) Good |4.7M | **1.88 × 10⁻⁴** |\n| **QR** | 0.8747 ± 0.0428 | (9.5) Very Good |4.7M | 1.86 × 10⁻⁴ |\n| Composition | 0.7806 ± 0.0393 | (2.0) Very bad |7.8M | 1.03 × 10⁻⁴ |\n\n\n**Summary:** \n\n1. **Parameter Efficiency:** DEFT uses 40% fewer parameters than naive PaRa+LoRA composition\n2. **Performance Superiority:** DEFT outperforms both individual methods and expected composition performance\n3. **Training Efficiency:** 50% faster training than expected composition approach\n4. **Architectural Advantage:** Coordinated subspace adaptation prevents interference between removal and addition operations\n\n**Note:** We acknowledge that direct experimental comparison with naive composition would strengthen this analysis. The theoretical analysis based on our authentic individual results demonstrates DEFT's fundamental optimization advantages over the simple method combination."}, {"review_id": "LTvFFjWydM", "reviewer": "Reviewer_1FhC", "summary": "This paper expands LoRA and PaRa by decomposes its update into two components.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The idea of combining LoRA and PaRa is interesting and demonstrates slight improvements compared to existing baselines.\n\nWeaknesses:\n1. The paper primarily claims improved flexibility, but the justification for this enhancement, both theoretically and empirically, is insufficient. The paper lacks comprehensive visual or quantitative evidence clearly illustrating how flexibility is notably improved, such as through demonstrations involving significant pose alterations or high dynamic range changes.\n2. For quality comparison on Canny Edge and Depth Map, why is FID much higher than other baselines?\n3. The \"Emergent Properties\" described in Section 4.2.4 are neither convincingly justified nor empirically supported. The arguments are unclear and lack concrete evidence, weakening the overall narrative of the paper.\n4. A more rigorous user study could strengthen claims regarding perceptual improvements.\n5. Since the proposed method combines LoRA and PaRA, it should also include a comparison against DreamBooth PaRA to ensure a fair and thorough evaluation.\n\nTypos (This list a few of it, please pay a careful attention on this):\n1. Capitalization: L176 \"dreamboot\"; L173,L231 \"DreamBench plus\"; L262,Fig 6 \"eqn\"; Fig 5 \"omnigen\";\n2. Inconsistent: L175 \"Visualcloze\", L188 \"VisualCloze\", L174 \"Visual cloze\", L248: \"omnigem\"\n3. Fig 6: \"with out\" -> \"without\", \"W_0\" should be in latex ($W_0$).", "was_revised": true, "final_justification": "I appreciate the authors’ thoughtful and detailed rebuttal. The responses have addressed my initial concerns.", "rebuttal": "We sincerely thank the reviewer for providing such thorough and constructive feedback. We greatly appreciate the time and effort invested in reviewing our work, and we acknowledge that the raised concerns are valid and important for strengthening our paper.\n\n### 1. Insufficient Justification for Flexibility Enhancement\n\n**Response:** We  thank the reviewer and the flexibility claims will be supported with additional justification in the final draft.\n\n**Theoretical Foundation:**\nWe would like to clarify that the DEFT's design is motivated by the below limitations of existing methods: (i) PaRa operates on the column subspace of the original pre-trained model, aiming to modify weights while preserving original information, but it cannot simultaneously add new concepts or styles, limiting its flexibility for novel concept adaptation. (ii) Conversely, LoRA can effectively learn new concepts but may lose some capabilities of the original model, which is particularly problematic in text-to-image models where maintaining original capabilities (prompt following, object combination, image editing) while personalizing new concepts is crucial.\n\nDEFT address both these limitations by operating on two complementary subspaces. For a given original weight W₀ with SVD: W₀ = UΣV^T:\n- **LoRA constraint:** Updates AB^T can interfere with existing singular structure\n- **DEFT advantage:** \n  - (I - PP^T) operates in col(P)⊥ (orthogonal complement)\n  - PR operates in col(P)\n\nThis dual control enables: (1) Selective modification through (I - PP^T), allowing preservation or removal of specific existing directions, and (2) Addition of entirely new orthogonal directions through PR.\n\n**Empirical Evidence:**\nThe flexibility of DEFT design can also  also observed  in the quantitative results presented in our paper as copied below: \n\n**Positional Flexibility Analysis:**\nWe provide quantitative results demonstrating DEFT's 3D positional awareness capabilities:\n\n| Metric | Original Model | DEFT (position aware) | DEFT + SFM inference |\n|--------|----------------|----------------------|---------------------|\n| **CLIP-I Score** | 0.4753 ± 0.0342 | 0.6468 ± 0.0589 | **0.6603 ± 0.1092** |\n| **CLIP-T Score** | 0.2010 ± 0.0638 | 0.2098 ± 0.0274 | **0.2132 ± 0.0409** |\n| **DINO-V1 Score** | 0.2197 ± 0.0280 | 0.3501 ± 0.0398 | **0.3427 ± 0.0447** |\n| **Sharpness** | 1179.31 ± 822.56 | 1812.73 ± 677.32 | **1852.48 ± 655.29** |\n\n**Visual Demonstrations:**\n- **Figure 1 - Scene Personalization:** Office scene control showing incremental additions (desk+computer → +books → +coffee → +plant → +papers → +clock), and Church Rock positioning across diverse environments (pool, city, mountain, beach, forest)\n- **Figure 5 - Novel Composition:** Training set adaptation showing diverse combinations under varying testing conditions beyond original training data\n- **Supplementary Figure 4:** Multi-task versatility across realistic photographs, abstract interpretations, segmentation, depth mapping, and edge detection\n\nAs suggested, we will included, additional quantitative analysis to further strengthen our flexibility argument.\n\n### 2. FID Performance Analysis\n\n**Response:** We appreciate the reviewer's attention to this important metric. We acknowledge that our FID performance requires honest explanation.\n\nThe FID versus SSIM performance pattern reflects a fundamental design trade-off in DEFT's architecture. DEFT employs a conservative adaptation strategy: W = (I - PP^T)W₀ + PR, where (I - PP^T)W₀ actively preserves pre-trained knowledge while PR enables controlled adaptation.\n\nThis design intentionally prevents aggressive style deviations while learning new concepts, resulting in more structurally consistent generations that maintain better SSIM scores but may appear less visually striking to FID's perceptual metrics. DEFT explicitly trades raw generation quality (FID) for structural consistency (SSIM), making it particularly suitable for applications where consistency and controllability are prioritized over pure visual novelty.\n\nWe believe this trade-off is valuable for many practical applications, though we acknowledge it may not be optimal for all use cases.\n\n### 3. Emergent Properties Claims\n\n**Response:** We sincerely acknowledge this valid concern and agree that our current title may overstate our claims relative to the experimental evidence provided.\n\nWhile Section 4.2.4 demonstrates what we consider emergent behavior through Figure 5 (multi-concept spatial composition from single-concept training) and VisualCloze results (single adapter handling 8+ distinct tasks), we recognize this evidence requires more rigorous quantitative analysis to support such prominent claims.\n\nIn response to this valuable feedback, we propose revising the title to focus on our core technical contribution: **\"Dual-decomposition Column Space Extension for Fine-tuning of Text-to-Image Models\"**\n\nThis revision better reflects our mathematical foundation while avoiding overstated claims about emergent properties.\n\n### 4. User Study and Perceptual Improvements\n\n**Response:** We appreciate the suggestion for more rigorous evaluation. We conducted both human and automated evaluations:\n\n**Human User Study Results:**\n| Method | Average Score | Best Score | Worst Score | Win Rate |\n|--------|---------------|------------|-------------|----------|\n| PaRa | 7.64 | 8.75 | 6.65 | 25.0% |\n| LoRA | 7.60 | 8.80 | 6.84 | 12.5% |\n| **DEFT** | **8.20** | **9.50** | 6.78 | **62.5%** |\n\n**Automated Evaluation (Claude Sonnet Judge):**\n| Metric | LoRA | PaRa | DEFT |\n|--------|------|------|------|\n| Task Achievement | 8.3 | 6.8 | **8.7** |\n| Subject Consistency | 8.2 | 6.4 | **8.8** |\n| Completion Quality | 8.4 | 7.1 | **8.6** |\n| Visual Fidelity | 8.1 | 7.3 | **8.5** |\n| Prompt Adherence | 8.3 | 6.9 | **8.6** |\n| **Average** | **8.26** | **6.90** | **8.64** |\n\nWe acknowledge that a larger-scale user study would further strengthen these claims.\n\n### 5. DreamBooth PaRa Comparison\n\n**Response:** We thank the reviewer for this important suggestion. We have provided comprehensive comparisons with PaRa in our supplementary results using the DreamBooth dataset with SDXL baselines:\n\n| Method | BEAR_PLUSHIE | CAT |\n|--------|--------------|-----|\n| **DEFT (rank=8)** (Ours) | **0.8415** | **0.9504** |\n| DEFT (rank=4) (Ours) | 0.8339 | 0.9280 |\n| PaRa (rank=4) | 0.8271 | 0.9315 |\n| PaRa (rank=8) | 0.8050 | 0.9467 |\n| LoRA (rank=4) | 0.7741 | 0.8057 |\n| LoRA (rank=8) | 0.7943 | 0.8583 |\n| OFT (Block=4) | 0.785 | 0.8559 |\n| SVDIFF | 0.7818 | 0.8854 |\n| DREAMBOOTH | 0.7921 | 0.8893 |\n| TEXTURE INVERSION | 0.7421 | 0.8048 |\n\n### 6. Typography Corrections\n\n**Response:** We sincerely apologize for these oversights and will carefully address all typography issues:\n\n**Corrections to be made:**\n- L176: \"dreamboot\" → \"DreamBooth\"\n- L173, L231: \"DreamBench plus\" → \"DreamBench Plus\"\n- L262, Fig 6: \"eqn\" → \"Eq.\"\n- Fig 5: \"omnigen\" → \"OmniGen\"\n- Standardize all instances to \"VisualCloze\"\n- L248: \"omnigem\" → \"OmniGen\"\n- Fig 6: \"with out\" → \"without\", \"W_0\" → proper LaTeX formatting\n\nWe deeply appreciate the reviewer's attention to detail and will ensure careful proofreading in our revision."}]

Author rebuttal:

Meta-review: The paper proposes a method for fine-tuning text-to-image models, which can be seen as a combination of two existing methods: LoRA and PaRa. 

The method works well, outperforming relevant baselines both quantitatively and qualitatively. All the reviewers are positive about the paper.

Initial concerns included comparisons to baselines (e.g. LoRA), presentation, and some missing analysis of the method, but these have satisfactorily been addressed in the author-reviewer discussion.

Given all of the above, I recommend acceptance.
Decision: Accept (poster)