Title: DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Efficient Fine-Tuning

LoRA

Unified Models

Personalization

Model Adaptation

Parameter-Efficient Training

Text-to-Image Models
Abstract: Efficient fine-tuning of pre-trained Text-to-Image (T2I) models involves adjusting the model to suit a particular task or dataset while minimizing computational resources and limiting the number of trainable parameters. However, it often faces challenges in striking a trade-off between aligning with the target distribution: learning a novel concept from a limited image for personalization and retaining the instruction ability needed for unifying multiple tasks, all while maintaining editability (aligning with a variety of prompts or in-context generation). In this work, we introduce DEFT, Decompositional Efficient Fine-Tuning, an efficient fine-tuning framework that adapts a pre-trained weight matrix by decomposing its update into two components with two trainable matrices: (1) a projection onto the complement of a low-rank subspace spanned by a low-rank matrix, and (2) a low-rank update. The single trainable low-rank matrix defines the subspace, while the other trainable low-rank matrix enables parameter adaptation within that subspace. We conducted extensive experiments on the Dreambooth and Dreambench Plus datasets for personalization, the InsDet dataset for object and scene adaptation, and the VisualCloze dataset for a universal image generation framework through visual in-context learning with both Stable Diffusion and a unified model. Our results demonstrated state-of-the-art performance, highlighting the emergent properties of efficient fine-tuning. Our code is available on \href{https://github.com/MAXNORM8650/DEFT}{DEFT}.


Peer reviews:
comments: SUMMARY:
This paper expands LoRA and PaRa by decomposes its update into two components.

STRENGTHS AND WEAKNESSES:
Strength:
1. The idea of combining LoRA and PaRa is interesting and demonstrates slight improvements compared to existing baselines.

Weaknesses:
1. The paper primarily claims improved flexibility, but the justification for this enhancement, both theoretically and empirically, is insufficient. The paper lacks comprehensive visual or quantitative evidence clearly illustrating how flexibility is notably improved, such as through demonstrations involving significant pose alterations or high dynamic range changes.
2. For quality comparison on Canny Edge and Depth Map, why is FID much higher than other baselines?
3. The "Emergent Properties" described in Section 4.2.4 are neither convincingly justified nor empirically supported. The arguments are unclear and lack concrete evidence, weakening the overall narrative of the paper.
4. A more rigorous user study could strengthen claims regarding perceptual improvements.
5. Since the proposed method combines LoRA and PaRA, it should also include a comparison against DreamBooth PaRA to ensure a fair and thorough evaluation.

Typos (This list a few of it, please pay a careful attention on this):
1. Capitalization: L176 "dreamboot"; L173,L231 "DreamBench plus"; L262,Fig 6 "eqn"; Fig 5 "omnigen";
2. Inconsistent: L175 "Visualcloze", L188 "VisualCloze", L174 "Visual cloze", L248: "omnigem"
3. Fig 6: "with out" -> "without", "W_0" should be in latex ($W_0$).

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DEFT (decompositional efficient fine-tuning) which resolves the two conflicting objectives: Personalization flexibility and parameter-efficiency for T2I diffusion models. DEFT consists of dual pathway (i) low-rank adapter that projects the frozen weight matrix onto the orthogonal complement of a learned low rank subspace and (ii) low-rank adapter that separately adds into that subspace. \nThis is a combination of LoRA and PaRa. \n\nApplied to SD and the unified Omnige, DEFT supports one-shot subject personalization, multi-concept composition, style transfer, conditional generation (Canny, depth, pose, etc.) and even visual in-context learning. All these objectives are performed with a single adapter, not task-specific LoRAs. They conduct experiments on various benchmarks and show impressive gain in text alignment, controllability and enhanced details compared to LoRA and RAPA. The emergent abilities are shown by the model composing multiple concepts and scenes without additional guidance in train time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed method, DEFT, might look like a simple composition of LoRA and PaRa but in fact, it introduces a dual architecture: (i) a projection that contributes to generalization, and (ii) subsequent low-rank injection that provides expressivity (task specific direction).\n- The evaluation results are impressive.\n- The method is simple yet powerful.\n\nWeaknesses\n- Ablation on using LoRA / PaRa / Both should be shown in the main paper.\n\nQUESTIONS:\n(i) Provide ablation results on the components of the methodology. Specifically, there should be evaluation report on using PaRa only, LoRA only, and DEFT.\n(ii) What is the core difference between DEFT and mere composition of LoRA and PaRa?\n\nLIMITATIONS:\nNo, not in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Decompositional Efficient Fine-Tuning (DEFT) for the parameter-efficient fine-tuning of text-to-image models. DEFT proposes modifying a pre-trained weight matrix by decomposing its update into two trainable components: (1) a projection that removes components of the original weight matrix along a low-rank subspace, similar to PaRa, and (2) a flexible low-rank update, via a second matrix `R`, that adds new capabilities to the model. The authors argue this formulation provides a better balance between learning new concepts (personalization) and retaining the general knowledge of the pre-trained model, thereby improving adaptability and reducing overfitting.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* Strong qualitative results: The paper presents compelling qualitative results across a wide range of tasks, including subject personalization, style transfer, and multi-concept composition. The images in the paper and supplementary material effectively showcase the method's ability to produce high-fidelity and contextually plausible images.\n* Extensive experiments: The authors validate DEFT across multiple datasets (DreamBooth, DreamBench Plus, VisualCloze, InsDet), demonstrating its versatility in both personalization and universal generation settings.\n* Methodological clarity and reproducibility: The paper provides the mathematical formulation for DEFT, with proofs included in the appendix. The authors have also provided links to supplementary websites and provided codes in the supplemental materials, which greatly aids in reproducibility, though I haven’t tested the codes by myself.\n\n### Weaknesses\n\n* Unsupported title and central claim: The paper's title, \"Emergent Properties of Efficient Fine-Tuning,\" is a significant overstatement. The paper only includes a small, qualitative subsection (4.2.4) to this topic, which essentially re-labels generalization as an \"emergent property\" without a clear definition or rigorous analysis. The grand claim in the title is not sufficiently supported by the paper's content.\n* Unsubstantiated claims about decomposition methods: In Section 3.1.1, the paper proposes that using different matrix decomposition techniques (e.g., SVD, NMF) could offer unique benefits for different tasks. However, this claim is not backed by experimental evidence. The only comparison in Figure 6 shows all methods produce very similar results on a single task, failing to demonstrate any distinct advantages.\n* Lack of inference time analysis: A core motivation for PEFT methods is efficiency. The paper claims DEFT is \"efficient\" but provides no quantitative analysis of its inference latency or memory overhead compared to LoRA, PaRa, or other baselines. Without this data, the efficiency claims are unsupported.\n* Unexplained poor performance on key metrics: In Table 3, the FID scores for DEFT, which measure image realism, are notably worse than most competing methods like OmniControl and ControlNet. The paper offers no explanation for this underperformance, instead focusing on other metrics where it performs better.\n* Missing comparisons to key literature: The paper fails to discuss or compare its method against other recent and highly relevant fine-tuning techniques. For instance, DoRA [1] and OFT (Orthogonal Finetuning) [2] are sophisticated methods that also aim to improve upon LoRA, making them critical points of comparison that are missing from the related work and experiments.\n\n### References\n\n[1] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen: DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024\n\n[2] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, Bernhard Schölkopf: Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS 2023\n\nQUESTIONS:\n1. Motivation for the `R` matrix: Could you provide a clearer, more intuitive explanation in the main text for why adding the second trainable matrix `R` is necessary? While the mathematical proof is in the appendix, a high-level motivation for how it enables learning \"new tasks\" beyond PaRa's capabilities would improve the paper's clarity.\n2. Comparison to OmniGen is a bit misleading : In Section 4.2.2, DEFT is fine-tuned on the OmniGen model and then compared to a \"base OmniGen\" on the VisualCloze dataset. To ensure a fair comparison, could you clarify if the \"base OmniGen\" baseline was also fine-tuned on the VisualCloze training set, or if it was the pre-trained model evaluated zero-shot?\n3. Explanation for poor FID scores: Could you provide an explanation for why DEFT's FID scores in Table 3 are consistently worse than most other methods? Does the DEFT formulation present a trade-off where it improves in some areas (like SSIM) at the cost of image fidelity as measured by FID?\n\nLIMITATIONS:\nThe authors did not include a dedicated limitations or societal impact section in the main body of the paper. This is a significant omission. A thorough discussion should have been included, acknowledging:\n* The lack of quantitative analysis on computational overhead (inference speed, memory).\n* The potential for the method to be used for malicious purposes, such as generating deepfakes, which is a standard concern for all powerful generative models.\n* A more thorough discussion of failure cases or scenarios where DEFT might not perform as well as other methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method called ﻿Decompositional Efficient Fine-Tuning (DEFT) to efficiently fine-tune pre-trained text-to-image models. Following the spirit of the commonly adopted low-rank update approach, DEFT further decomposed the update into two components, namely the projection onto a subspace and the low-rank adjustment. Both quantitative and qualitative experiments are performed to validate the efficacy of the proposed approach in performing highly efficient model fine-tuning that achieves faithful personalization without compromising the original capabilities of the pre-trained model.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Fine-tuning pre-trained text-to-image models is a challenging task. A wide range of prior works suffers from striking a balance between adaptation to new data and preserving the pre-learned knowledge. To alleviate this, this work explores a novel low-rank update scheme, which I believe could be of great interest to the text-to-image generation community.\n2. The proposed approach is simple yet effective. The authors also provide ﻿rigorous calculation and analysis to ﻿provide theoretical guarantees on the efficacy of the proposed method.\n3. Experimental results are impressive. Extensive evaluations demonstrate that the proposed model obtains better performance compared to previous fine-tuning methods in terms of﻿generation flexibility and model generalization.\n4. The motivation of this work is clearly elaborated and the paper is well-written and easy to follow.\n\n## Weaknesses\n1. For the literature review part, this paper lacks the introduction of another line of work that adopts training-free strategies to adapt the pre-trained model to new datasets or tasks.\n﻿2. As for the ablation study, some quantitative evaluation results should be given for a better investigation of the effect of decomposition.\n3. For the generation results showcased in Figure 3, I personally think that the performance of LoRA and DEFT is quite close. Maybe the authors could conduct a user study for a more comprehensive assessment of different methods.\n4. This paper lacks discussions of the convergence speed of different fine-tuning methods, which I believe is a key factor reflecting the performance.\n\nQUESTIONS:\n1. In Section 4.2.2, are ﻿other competing methods (e.g., OmniGen) fine-tuned on the ﻿VisualCloze dataset? If so, which fine-tuning technique is utilized to fine-tune them? This should be provided in detail.\n2. In Section 4.2.4, the authors claim that the proposed method exhibits a certain kind of emergent properties, which is emphasized in the paper title. However, the experimental results are inadequate. More generation results (e.g., handling completely new tasks/data) should be provided to support the claim.\n3. Are there any representative failure cases? Some results and discussions regarding this should be provided for a better understanding of the limitations of the proposed method.\n\nLIMITATIONS:\nSome representative failure cases are supposed to be given and the authors should provide some discussions on the limitations of the proposed work as well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}