Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift
Authors: Gihoon Kim
Hyungjin Park
Taesup Kim
Venue: ICLR 2026 Poster (2026)
Keywords: Text-to-Image Diffusion Models
Personalization
Overfitting
Distributional Drift
Regularization
Lipschitz Constraints
Primary area: generative models
TL;DR: We propose a simple yet effective objective grounded in Lipschitz regularization that guarantees preservation of the pretrained distribution, which prior approaches failed to ensure.

Abstract: Personalizing text-to-image diffusion models involves integrating novel visual concepts from a small set of reference images while retaining the model’s original generative capabilities. However, this process often leads to overfitting, where the model ignores the user’s prompt and merely replicates the reference images. We attribute this issue to a fundamental misalignment between the true goals of personalization, which are subject fidelity and text alignment, and the training objectives of existing methods that fail to enforce both objectives simultaneously. Specifically, prior approaches often overlook the need to explicitly preserve the pretrained model’s output distribution, resulting in distributional drift that undermines diversity and coherence. To resolve these challenges, we introduce a Lipschitz-based regularization objective that constrains parameter updates during personalization, ensuring bounded deviation from the original distribution. This promotes consistency with the pretrained model’s behavior while enabling accurate adaptation to new concepts. Furthermore, our method offers a computationally efficient alternative to commonly used, resource-intensive sampling techniques. Through extensive experiments across diverse diffusion model architectures, we demonstrate that our approach achieves superior performance in both quantitative metrics and qualitative evaluations, consistently excelling in visual fidelity and prompt adherence. We further support these findings with comprehensive analyses, including ablation studies and visualizations.

Published as a conference paper ICLR 2026
PRESERVE AND PERSONALIZE: PERSONALIZED TEXT-TO-IMAGE DIFFUSION MODELS WITHOUT DISTRIBUTIONAL DRIFT
Gihoon Kim
Hyungjin Park
Taesup Kim
Graduate School of Data Science
Scoul National University
ABSTRACT
Personalizing text-to-image diffusion models involves integrating novel visual concepts from a small set of reference images while retaining the model's original generative ca- pabilities. However, this process often leads to overfitting, where the model ignores the user's prompt and merely replicates the reference images. We attribute this issue to a funda- mental misalignment between the true goals of personalization, which are subject fidelity and text alignment, and the training objectives of existing methods that fail to enforce both objectives simultaneously. Specifically, prior approaches often overlook the need to explicitly preserve the pretrained model'ss output distribution, resulting in distributional drift that undermines diversity and coherence. To resolve these challenges, we introduce Lipschitz-basee regularization objective that constrains parameter updates during per- sonalization, ensuring bounded deviation from the original distribution. This promotes consistency with the pretrained model's behavior while enabling accurate adaptation to new concepts. Furthermore, our method offers a computationally efficient alternative to commonly used, across diverse diffusion model architectures, we demonstrate that our approach achieves superior performance in both quantitative metrics and qualitative evaluations, consistently excelling in visual fidelity and prompt adherence. We further support these findings with comprehensive analyses, including ablation studies and visualizations.
sampling techniques. Through extensive experiments
INTRODUCTION
Recent advances in diffusion- based image generation models have led to remarkable progress photorealistic and diverse image synthesis (Dhariwal & Nichol 2021
Ho et al. 2020) Song et al. 2020a Nichol & Dhariwal
1021 Ho& Salimans 2022: Song et al. 2020b Karras ct al. 2022). In particular, text-to-iie to-iiage diffusion
models (Rombach et al. 2022 Saharia et al. 2022 Ramesh et al. [002 Podell et al. 2023 Esser et al.) 2024)
have demonstrated the ability to synthesize high-quality images across a wide range of prompts, effectively capturing the semanties of general descriptions such "a dog in the snow".
However, these models struggle when prompts involve user-specific or hard-to-descrine subject details. In real-world scenarios, users often wish to personalize generation with only few reference images of their own subjects, which makes the task fundamentally challenging. This motivates the need for personalization methods that adapt pretrained models to novel, user-specific concepts using only small number of images (Gal et al. [2022] Ruiz et al. (022). The ultimate goal of personalization is to faithfully encode new concepts (e.g.. "my dog") ") while preserving the model's inherent ability follow prompts (c.g.. "in the
*Corresponding author

---

Published as a conference paper at ICLR 2026
snow"), thereby generating diverse and compositionally rich images such "my dog in the snow". Unfortunately, learning new subjects from limited data often leads to overfitting where the model disregards prompt content and collapses to simply reproducing the reference images.
To mitigate this issue, recent methods have attempted to constrain the adaptation process by freezing parts of the network (Hu et al. 2022 Kumari et al. 2023 Tewel et al. 2023) or limiting the magnitude or direction of parameter updates (Qiu et al. 2023 Han et al. (023). While partially effective, these methods rely on heuristic architectural constraints rather than explicilly considering distributional preservation. As result, they offer only indirect control over the preservation of the pretrained distribution. Moreover, such techniques often introduce a trade-off between flexibility and stability; freezing too many parameters can prevent proper adaptation new subjects, while relaxing constraints increases the risk of overfitting and prompt misalignment. These limitations make it difficult to precisely balance the dual objectives of personalization.
In contrast, our approach tackles this challenge at the objective level, providing explicit mechanism to regularize deviation from the pretrained model without relying on architectural heuristics. By formulating the problem through Lipschitz-based regularization, we offer principled and continuous control over model updates, enabling better generalization. Figure illustratee the trends arising from the differences between existing and proposed objectives, which are further discussed in Section 4.3 We also show that this formulation can be expressed as a variant of L2 regularization, making it simple to implement yet theoretically grounded. Moreover, our approach does not require a pre-sampling stage for regularization, substantially reducing training time.
We validate our approach through comprehensive experiments across multiple diffusion backbones and baseline personalization methods. Our approach achieves superior performance in both quantitative metrics and qualitative evaluations, excelling visual fidelity as well as prompt consistency. also reduces training time by more than half, offering substantial efficiency gains in practice. Furthermore, we present intuitive analyses through visualizations and ablation studies, providing empirical support for our theoretical claims. Our contributions summarized follows:
We identify an objective-level gap in existing personalization approaches and provide a thcoretical analysis of its implications.
We propose a Lipschitz- Lipschitz-based regularization objective that guarantees distributional preservation during personalization. The proposed formulation is simple yet theoretically grounded, aligning the training objective with the true goals of personalization.
Our method demonstrates improvements in both subject fidelity and text alignment, together with clear gains in training time efficiency. We also provide intuitive insights through visual analyses and ablation studies that support the theoretical and empirical findings.
2 RELATED WORK
Personalized Texteto-Image Generation. A central question of few-shot personalization has been how to adapt pretrained networks to new concepts with only few subject-specific images. Textual Inversion (Gal
jt al., 2022 Voynov et al. (023) encodes subject-specific information into learned text tokens, while
DreamBooth (Ruiz et al. 2023) finetunes the entire network. However, instability in the training process results in mode collapse and overfitting. Subsequent studies aim to improve stability and efficiency by
updating subsets of parameters (Meng al. 2022; Kumari et al. [023 Tewel et al.] (023) or by applying low-rank adaptation (LoRA) (Hu et al. [2022] Chen et al. (024). In this context, SVDiff (Han et al. 2023) focuses updates the singular values of weight matrices, whereas OFT (Qu et al. 2023) restricts them to angular components. While these enforced constraints help stabilize the training process and preserve generative capacity, the reduced flexibility restricts the model's 's expressiveness for personalized concepts.
2

---

Published as conference paper ICLR 2026
Pretrained
Target
Ideal Case
(b) Naive
(c) Prior Preserv.
(d) Ours
Figure 1: Visualization of training objective differences on the toy dataset for personalization, which are further discussed in Section [4.] The diffusion model is first trained on the pretrained dataset (five colors) and then personalized with the target dataset (yellow). (a) represents the ideal scenario, where the target data learned while the original knowledge preserved. (b) and (c), corresponding to Eqs. [nd 3 fail preserve the distributions of all classes. In contrast, (d), which incorporates the proposed regularization, better maintains the structure of the pretrained distribution and best matches the ideal case in (a).
On the other hand, auxiliary conditioners have been employed without modifying the pretrained model. For instance, BLIP- -i--Difuusion (Li et al. 2023a) leverages pretrained langunle models (Li et al. 2022 2023b), and IP-Adepter (Chen al. 2023 Zhang al. 2023) introduces adaptor network trained on large-scale datasets. These approaches increase versatility, but they show limited performance on individual personalized concepts. Moreover, they require substantial computational and external resources, making them impractical in few-shot personalization scenarios. Whether through selecting trainable components or designing auxiliary conditioners, these methods remain tied specific backbone architectures and therefore fall short of providing a general solution.
Objective Functions for Preserving Distribution. Objective-level formulations for distribution preservation provide principled direction for personalization. Prior Preservation Loss (Ruiz et al., (023) introduces a regularization function via pre-sampling over the generic category of novel subject, as detailed Section Subsequent work extends this idea by constraining novel subjects not only at the generic concept level but also at the image level (Qiao et al., 2024) and text level (Huang al.) 2025). However, preserving category-level information does not necessarily imply the preservation of all other prior knowledge, point we discuss in Section 4.1 Beyond personalization, diffusion-basse
applications (Poole et al. 2022 Hertz et al. 2023 Wang et al. | 2023) introduce objectives to align the output distribution of a learnable generator with that pretrained diffusion model. Such methods incorporate pairwise supervision ( (Yin et al. 2024b)
adversarial objectives (Yin et al. 2024a; Sauer al. 2024). Their dependence on external datasets and extensive optimization, however, restricts their applicability in personalization settings where only a handful of subject images available. The limited guarantees of prior methods, together with the poor applicability of alternative approaches to few-shot personalization, point to the need for goal-aligned formulations.
3 BACKGROUND
Text-to-Image Diffusion Models. Diffusion models are trained to maximize an evidence lower bound
(ELBO) on the data log-likelihood log p(x) due the intractability (Luo 2022 Chan et al. 2024 Nakkiran
et al. 2024). The generative process is formulated over a sequence of latent vanables (Sohl Dickstein et al.
2015
which can be written as:
log pxx) = log p(XOr)) dxi.T 2 E (xx)) log p(xI.)
(I)
g(xIT xo)
3

---

Published as a conference paper ICLR 2026
Rather than optimizing Eq. directly, DDPM (Ho al. 2020) adopts a surrogate loss for transition-noise prediction. Text-to-Iiee Imaaeet-Imaaee diffusion frameworks (Rombach et al. 2022 Saharia et al. 2022 Podell et al. 2023) train the denoiser conditioned on a text prompt c, and the resulting conditional surrogate loss maximizes the joint distribution po(x,c) c In detail, an encoder & maps image x to latent {x)), and noisy latents are generated as 24 v z vI at 6, where e ~ (0, I), ~ Uniform ...... T) and ar denotes the cumulative product of the noise schedule. Training then minimizes the conditional cise-preeiitton loss:
Ldenoise
E,, e, <||e Ea (Z,, t)
(2)
Prior Preservation Loss. The combination of the personalization loss and the prior preservation loss has become conventional objective for personalization. For the personalization loss Epersonaliee- the target prompt Crarget (e.g., "A photo of V dog") incorporates the special token V*, which guides the model to learn subject-specifie features. For the prior preservation loss Lpriors the class prompt Celass (e.g., "A photo of a dog") is used, and 100~200 class-prompt latents z are sampled from the pretrained model. As both the combined formulation is:
objectives follow the conditional diffusion objective in Eq. 2
Lioull Exe, 22, Ciugen Crisa e,t ||c
Cy (Ze, Cuaret t)l + Aprior ||c co (Z/= Celass t)][2
4 METHOD
this section, we investigate why the standard objective fails to enable effective learning in personalization. Next, we explore how to provide a distributional bound tailored to preserving the generative capacity of the pretrained distribution. Let x E X' denote an image and e E C its associated text prompt. The pretraining dataset Drase {z,,, ) is sampled i.id. from distribution Poase (2, that closely approximates the true data distribution p" (2, c). The personalization dataset Dper = {(xgg ((r))())}3 b)) with M << N, has
underlying distribution Pper (z. c). Such assumptions are standard and align with practical settings, ranging
from large-scale pretraining on LAION-400M (Schuhmann et al. 2021 2022) to personalization datasets with only 4~66 images per subject (Ruiz al. 2023).
4.1 MOTIVATION: OBJECTIVE- GOAL MISALIGNMENT
The goal in personalization is not merely to fit the provided data distribution, but to preserve the generalization capabilities of the pretrained model while incorporating novel subject information. We now demonstrate a fundamental misalignment that standard personalization objectives lead to divergence from the original distribution.
Theorem 1. p" (x, e) denote the reference distribution, and let the model parameters Opane have distribution Pe (z, c) satisfying, for any E >> O,
1p" c) -Po (x, e) < E.
Suppose there exists a measurable set D X x C. such that
Padapp (D) Y DD(D)= 8,
> 0. 7
The model is adapted by gradient descent on the denoising loss Laenvire from Eq. trained (I, e) Padaps Oc+1 0₂ oodennniee (Oc),, where 0g Obase.
After sufficient iterations Po, Paddut;; under universal approximation and convergence assumptions. Then
DKI(p' PP < DKi(p" llpo.).
"This follows from the conditional distribution log (x(x c) Dog,o(xx, c)x llog po(x, Zi.r dzy logp(c), where log p(c) is constant with respect to O.
4

---

Published as conference paper ICLR 2026
Algorithm 1 Lipschitz- based Regularization for Personalization
Personalization Time (s)
Require:
Pretrained cfase dataset Peer prompt Gpe- weight
5000
Prior Preserv.
Ensure: Personaized Edger
Ours
Prior-preservaion pre-sampling:
1: for- byrm Nardo
2:
(2 boat Das Day
(n))
3000
22k
end-for
2000
4: fager copy Weights( Cauae Freeze Caess
Personalization Phase:
5: step
6:
a.
a. D E(a)) ce TextFnender( (pos)
7:
Eo Unif(((
Eys= vor v 12 e~.(0.1)
5015-0
50 cp
co
503-aaM
8:
ai
9c, a
I1
+ ||e tlper (Z,, c012 2; 10 0
10:
Update 9pz w.t. VC
Figure 2: Training efficiency of the proposed method versus the baseline.
11: end for
Proof. See Appendix
Remark 1. Personalization based on the standard diffusion objective (Eq, 2 provides no guarantee of
preserving the pretrained distribution and may fail to preserve the generative capacity, which can potentially lead to overfitting or mode collapse.
Let Petass be the sub- distribution of p(z, Cetass) used for prior preservation as described in Section Pper a mixture of Pper and Pelass. still satisfies M < N. Then,
Corollary 1.
(D) 27, p PD(D) <8 ( > 8).
By Theorem
DKI(p" [| < (KL(P" l
Remark 2. Even when trained with pror-preseervation samples based the class prompt, in Eq. the model still offers no guarantee preserving the pretrained distribution.
A fundamental limitation of existing objectives is revealed by Remarks and highlighting the need for new objectives that can achieve both subject adaptation and distributional preservation. It also explains why the unintended distribution shift poses inherent challenges in personalization. This will be examined as illustrated in Figures lb and Ic
4.2 LIPSCHITZ-BASER REGULARIZATION FOR DISTRIBUTION PRESERVATION
Motivated by Section 4.1 we investigate hipschitz-based regularization that provides explicit bound on distributional preservation. The following Theorem 2 provides the justification by showing that the generative distribution of the model is bounded under the Lipschitz continuity of the noise prediction network. The detailed proof is deferred to Appendix G
Theorem 2. If the diffusion model Eg is Lipschitz continuous in 8, for any two parameter sets 01 and 02 there exists constant X > ( such that
DRL (pe, ll pe.) I A 1|0, 02
(4)
Proof sketch.
Given that the composition of Lipschitz functions preserves Lipschitz continuity (Neyshabrr et al.
2015) Asadi et al.] (2018) and that attention mechanisms admit finite Lipschitz constants on compact input domains (Castin et al. 2024), assume Eo (x, is Lipschitz in with finite constant Lg-
5

---

Published as a conference paper ICLR 2026
Table 1: Performance gains across backbone models.
Exampiex
5D-N DD-L
Model
Method
DINO 1
CLIP-T 1
CLIP-11 1
DB
0.6028
0.2793
0,7881
+ Ours
0.6394 (+0.0366)
0.2976 (+0.0183)
0.7948 (H0.0067)
cut with
SD-30
CD
0,5568
0.3154
0,7539
mountall
+ Ours
0.5638 (+0.0070)
0.3158 (+0.0004)
0.7550 (+0.0011)
background
DB-LoRA
0,3108
0,7739
+ Ours
0.6038 (+0.0262)
0.2975 (-0.0133)
0.7915 (+0.0176)
SD-XL
D.5337
0.3239
0.7485
SD-Xt
Ours
0.5493 (+0.0156)
0.3251 (+0.0012)
0.7540 (+0.0055)
DB-LoRA
0.6562
0,3099
0,7970
Ours
0.6819 (+0.0257)
0.0014
0.8103 (+0.0133)
[V] dog worng
SD-30
SD-30
CD
+ Ours
0.6271 (+0.0103)
0,6168
0.2955 (-0.0076)
0,3031
0. 0.8023 (+0.0094)
0,7929
mimborw scarf
DB-LoRA
0,6043
0.3098
0,7823
Figure 3: Visual comparison of the proposed
+ Ours
0.6217 (+0.0174)
0.3103 (+0.0005)
0.7949 (+0.0126)
method and baselines across backbone models.
By Tweedie's 's formula (Efron 2011), the score is so(x, = - E0(œ t)/or, and scalar multiplication by 1/a; preserves Lipschitz continuity.
The probability-loww ODE gives log po)) = logpr(ET) - So- (2 1) das dt (Song & Ermon 2019), and since integration a linear operator, it also preserves the Lipschitz bound.
de
Finally, the triangle inequality implies DKL(pe, || po.) S A 1 02]l,, where cLo for some c >> 0.
Remark 3. For k squaring the inequality yields a form directly connected an L2 regularization term, A- 0 - 22112 - X' ||0] - 12112. where X' accounts for the squared scaling.
Building on Theorem 2 we propose a novel objective that regularizes personalization relative to the pretrained distribution based on Eq. Remark demonstrates that minimizing the 1.2 regularization term serves as a relaxed form of the Lipschitz constraint. Although the 1.2 form is conventional, its use surrogate for Lipschitz continuity the context of personalization has not been utilized before. The full training procedure with the proposed objective is summarized in Algorithm [ the new subject distribution and preserves the pretrained distribution, which the conventional objective unable to ensure.
3
As result, the proposed formulation both learns
The proposed objective also provides practical advantages. First, preserving the distribution through parameter distance climinates the need for additional datasets such prior samples external resources. As illustrated in Figure 2 removing this requirement yields a considerable reduction in training time. Second, the Lipschitz- based regularizer bounds the KL. divergence from the pretrained distribution with single hyperparameter X, thereby quantifying the degree of distributional preservation during training and enabling explicit control as discussed in Section 5.2 By providing simplified yet explicit mechanism for control, our approach contrasts with previous methods that addressed the issue only implicitly through heuristic strategies. The subsequent experiments validate these advantages and highlight the practical impact of our objective.
4.3
TOY EXPERIMENT: METHOD VALIDATION
In this section, we present visual analysis using toy data to validate the claims established in Theorems] | 2 The experiment investigates how the proposed method influences the preservation the pretrained distribution when the model adapts to new target, in comparison to existing approaches. We employ diffusion models trained on 2D data sampled from Gaussians with shared variance and different means. As illustrated in Figure I the leftmost figures show the pretrained distribution of five classes (top) and an introduced class be learned (in yellow, bottom). The other four figures present the outcomes of personalization under different objectives.
6

---

Published as conference paper ICLR 2026
Dreambooth
Textual
Custom
Examples
Prompt
Ours
Dreambooth
SVDiff
IP-Adaplor
-LoRA
Inversion
Diffusion
Diffusion
(M) deg th chef corfs
bookgroand
a I'] cor. top of's. wcoden floor
[VV candie
with the Fiflet
Tower in
badgnuand
[M] dog
waring sasta
Figure 4: Visual comparison with baseline methods. Ours denotes DreamBooth with the proposed method.
These results demonstrate that the proposed regularization preserves the pretrained distribution more effec- tively than conventional objectives. When trained with Eq. [ the pretrained distribution is not preserved. Using Eq. 3 partially preserves the selected class (shown in deep blue), the rest of the pretrained classes are not maintained. In contrast, applying our method as in Algorithm enables the model to better preserve the original distribution while still adapting the new data. This empirical result suggests that, unlike con- ventional methods which primarily preserve the subject class (e.g.. "dog" in "my dog in the snow"), the proposed regularization retains broader contextual generality (e.g., "the snow") during personalization. It further demonstrates that the method maintain the pretrained distribution without requiring explicit access to the original training data, making it well-suited for personalization scenarios with limited data and complex distributions.
5 EXPERIMENTS AND ANALYSIS
We evaluate the proposed method through qualitative and quantitative comparisons with basclines across diverse backbones. We further conduct three ablation studies to examine the practical implications of the proposed formulation in the context of personalized diffusion models. Further analysis, including failure cases, is provided in Appendix L
Implementation Details. We evaluate our method and baselines on the widely used personalization bench- mark (Ruiz al., (023), which includes 30 subjects, each with up to example images. To assess the generality of the proposed method, evaluate it across different backbones, ranging from Stable Diffusion v1.5 (SD-1.5) (Rombach et al., (022) to Stable Diffusion-XL. (SD-XL) (Podell et al. (2023) and Stable Diffusion-3.0 (SD-3.0) (Esser al. 2024). Since both SD-XL and SD-3.0 discourage updating all parameters due to computational inefficiency, we focus our evaluation on parameter-efficient methods. For baseline
comparisons, consider DreamBooth (DB) (Ruiz al. 2023), Textual Inversion (Gal et al. 2022). Custom
7

---

Published as a conference paper ICLR 2026
Table 2: Quantitative comparison of baseline methods.
Examples
Prompe
= 0.0
2 0.5k
A 2k
Method
1 1 CLIP-T CLIP-1 f
Rank
Few-shol personalization methods
Ours
0.6394
0.2976
0.7948
DreamBooth
0.6028
0.2793
0.7881
DreamBooth-oRk,
0,5778
0,3095
0.7731
Textual Inversion
0.5342
0.2601
0.7539
Castom Diffusion
0.5568
0.3154
0.6479
SVDiIT
0.3839
0.3194
0.6886
OFT
0.6320
0.2370
0.7850
Conditioning with external
BLIP- Difflssisi
0,5943
0.2865
0.7935
IP-Adaptor
0.6304
0.8318
Figure 5: Visual comparison by regularization strength.
Diffusion (CD) Kummar et al., 2023), SVDiff (Han et al. 2023), and OFT (Qiu et al. 2023), as well as an extension based LoRA (Hu et al. 2022). While we attempted to reproduce OFT using the official codebase, it did not produce usable outputs under our setting. As result, we report its performance based on the results presented in the original paper. We also include BLIP-Diffusion (Li al. 2023a) and IP-Adapter (Chen
(et al. 2023) for comprehensive comparisons. For more details on the setup and implementation of individual experiments, please refer Appendix H
Evaluation Methods. We evaluate each method based on the images generated from evaluation prompts. For each subject, we use the 25 evaluation prompts provided in the benchmark, generating images per prompt, resulting in 100 images per subject and total of 3,000 images across all 30 subjects. We evaluate model performance using three metries: DINO, CLIP-T, T, and CLIP-I. We use pretrained DINO ViT-S/16 (Caron et al.. (021)) and CLIP V/T-B/32 (Radford et al.) [02T] models to extract image and text embeddings. CLIP-I and DINO measure subject fidelity by computing the cosine similarity between each generated image and the corresponding input reference images. CL.PPTT evaluates text-image alignment based on cosine similarity between the prompt and the generated image. A broader range of results is provided Appendix L
5.1 EXPERIMENTAL RESULTS
Comparisons across Backhone Networks. Table 1 presents quantitative results showing the effect of replacing the baseline objective with the proposed objective across different backbones. The method improves either both image fidelity and text alignment, or improves one without reducing the other. This remarkable since the two objectives are typically in conflict, where gains in one often lead to losses in the other. In SD-1.5 with DB, the DINO score increases substantially, while the CLIP-T score also improves. Moreover, we observe consistent improvements when the proposed objective is combined with other personalization strategies. For example, CD in SD-1.5 and SD-XL, well DB-LoRA in SD-3.0, show gains across all metrics, while the remaining settings still yield meaningful improvements.
A key aspect of the performance improvements that our method substantially enhances image fidelity, while maintaining text alignment at high level (around 0.3). This indicates that the proposed method accelerates the learning of novel concepts while preserving the knowledge encoded in the pretrained model. As shown Figure 3 both baseline and our method produce results that faithfully reflect the input prompts. However, Figure also shows that baseline methods fail to capture the characteristics of the reference image. For example, in the case of CD with SD-XL, the identity of the dog is not preserved by the baseline, while our method achieves a clear improvement. These observations are consistent with the quantitative improvements and highlight that the proposed approach contributes both robustness and versatility.
Comparisons with Baseline Methods. As shown in Figure we compare the proposed method with various baseline approaches. The proposed method demonstrates superior performance in both fidelity and prompt
8

---

Published as a conference paper ICLR 2026
0.31
031
-*-2 -kk
-.-2 (k)
4003
A-0.51
CLIP-T
CLP-T
|Ac
A-11
1000
A=2k
1=5k
1-1.0k
0.29
0.9
0.74
0.75
0.78
0.8
C.2
0.52
0.56
0.5
0.64
0.88
CLIP-1
DINO
|40]
Figure Trade-off curves between image fidelity and text alignment (left and middle) and qualitative output variations (right), both evaluated across different À values.
alignment compared to the baselines. Among the baselines, DB and DB-LoRA show competitive performance. However, DB either fails to incorporate the prompt or exhibits artifacts in the third row. DB-LoRA tends underrepresent subject identity in the first Other methods generally underperform. CD struggles capture fine-grained subject details. Textual Inversion and SVDiff tend to focus more on prompt alignment than subject fidelity. BLIP-Diffusion and IP-Adapter yield results that preserve the overall subject structure but often miss ddentity-speciffi features. In contrast, our method faithfully captures the fine- fine- grained details of the input images while reflecting the text prompts. We conduct the human evaluation Appendix providing a detailed analysis across multiple criteria to assess perceptual aspects well.
These qualitative observations are consistent with the quantitative results summarized in Table 2 When ranking models jointly on fidelity (DINO and CLIP-I) and alignment (CLIP-T), our method is ranked first. This indicates that the proposed approach provides more balanced improvement across all metrics compared the baselines. Since other baselines achieve higher scores on one metric but lower on the other, no baseline simultaneously offers comparable performance on both. For example, while SVDiff achieves the highest CLIP-T score, its overall ranking remains low. These observations reveal a clear trade-off between text-image alignment and subject fidelity. Nonetheless, our method alleviates this trade-off and achieves superior performance in both quantitative and qualitative evaluations.
5.2 ABLATION STUDIES
Balancing Adaptation and Preservation. We analyze how the proposed regularization affects the quality of personalized outputs. Figure 5 shows the effect of varying the regularization weight X. With higher X, the generated images align more closely with the prompt but often miss key subject attributes. With lower X. they capture subject-specific features, but may collapse imitating the input images or introduce visual artifacts. A similar pattern is quantitatively confirmed in Figure These experiments investigate the behavior of image fidelity and text aligmment under varying regularization strengths. A larger increases CLIP-T, indicating stronger text alignment and greater preservation of the pretrained distribution. In contrast, a smaller A leads to higher DINO and CLIP-I scores, indicating closer adaptation to the target subject in terms of visual fidelity. This observation suggests that the proposed objective simultaneously addresses both adaptation and preservation while enabling a controllable balance between them.
Empirical Validation of the Lipschitz Condition. We assess whether diffusion models follow our claim during personalization under the assumptions. To examine whether the Lipschitz continuity holds in diffusion models, measure the parameter deviation AO and the corresponding output difference Ae between the pretrained and personalized models. Figure iillsstrass the effect of applying Lipschitz regularization to the personalized diffusion model. As the regularization strength À increases, both A and A decrease proportionally. This behavior indicates that the change in the output of the model is bounded by the change in parameters. The result provides empirical support for our theoretical claim.
9

---

Published as a conference paper ICLR 2026
Training Efficiency. We measure the training time required for personalization to assess the proposed method compared to prior regularization across different backbones. Figure 2 reports averaged results from 10 trials under the same environment for each method. The results shows that our approach is substantially more efficient than methods relying on pre-sampling. The prior preservation strategy consumes the majority of training time for generating prior samples, whereas our algorithm does not require this bottleneck as outlined in Algorithm| [ The improvement is particularly evident in parameter-efficien baselines. Their shorter training time renders the cost of pre-sampling dominant. In SDXL-CD, the training time is reduced to less than fic-ffith of the baseline. Other basclines also achieve more than twofold reduction. As training efficiency is essential for the feasibility of real-world applications (Cho et al. 2024 Chen et al. (2025). such improvements represent key advantage of the proposed method alongside its performance gains.
6 CONCLUSION
this paper, we demonstrate that existing personalization objectives are misaligned for preventing distri- butional drift when adapting to novel concepts. Therefore, introduce a Lipschitz-based regularization objective that bounds distributional shift under a Lipschitz constraint. The proposed formulation is simple yet theoretically grounded, aligning the training objective with the true goals of personalization. Through extensive experiments across diverse backbones and baselines, we validate consistent improvements in both subject fidelity and text alignment. Ablation and visual analyses further highlight the necessity of the regular- ization, while the method requires no additional time-consuming stages and provides explicit control over the dlaptation-preseevation trade-off. In conclusion, we contribute a principled approach to personalization that integrates new concepts without unintended deviation from the pretrained distribution. We hope our findings provide a practical reference for balancing adaptation and preservation in post- post-training across diverse domains.
LIMITATIONS AND FUTURE WORK
Although our work advances over prior limitations and improves overall performance, it does not guarantee consistent gains across every metric, prompt, or subject. This is because the required degree of adaptation to a novel concept and the level of preservation needed maintain pretrained behavior can vary across subjects and backbones. Future methods should move beyond retaining prior knowledge and instead focus on better integrating novel subject characteristies to enable more effective and generalizable personalization. These challenges highlight the need for adaptive personalization training strategies (Xuhong et al., 2018 Asadi et al. 2022). One potential direction to adaptively weight regularization across parameters. In particular, EWC (Kirkpatrick et al. 2017) can assign more precise adaptive weights using the Fisher information F; for each parameter 0,. However, F; is computed (or approximated) from the log- log-likelihood gradients evaluated on the original training dataset D pretrain and the corresponding pretrained parameters pretrrii Because the original pretraining data are not available in practical personalization scenarios, this approach is difficult to apply in practice. Nevertheless, if Fisher information could be estimated reliably in this setting, it would provide further benefits for adaptive optimization.
ACKNOWLEDGEMENT
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2025-25424412 Controllable Image Generation in Diffusion Models with Distribution Preservation; No. RS-2024-00335809, Research Al Robustness Against Distribution Shift Real- World Scenarios; and No. RS-2023-00222663, Center for Optimizing Hyperscale AI Models and Platforms).
10

---

Published as conference paper ICLR 2026
REFERENCES
Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity mosel- reinforcement learning, In International Conference Machine Learning, pp. 264-273. PMLR, 2018.
Kavosh Asadi, Rasool Fakoor, Omer Gottesman, Taesup Kim, Michael Littman, and Alexander Smola. Faster deep reinforcement learning with slower online network. Advances in Neural Information Processing Systems, 35:19944-19995 19955, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Valérie Castin, Pierre Ablin, and Gabriel Peyré. How smooth is attention? In Proceedings of the 41st International Conference on Machine Learning, pp. 5817-5840, 2024.
Stanley Chan et al. Tutorial on diffusion models for imaging and vision. Foundations and Trends@ in Computer Graphics and Vision, 16(4):322 471, 2024.
Shangyu Chen, Zizheng Pan, Jianfei Cai, and Dinh Phung. Para: Personalizing text-to-image diffusion via parameter rank reduction. arXiv preprint arXiv:2406.0564 2024.
Shangyu Chen, Zizheng Pan, Jianfei Cai, and Dinh Phung. Para: Personalizing text-to-image diffusion via parameter rank reduction. In Proceedings of the 13th International Conference on Learning Reprexentations (ICLR), 2025.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems, 36:30286 30305, 2023.
Wonguk Cho, Seokeon Choi, Debasmit Das, Matthias Reisser, Taesup Kim, Sungrack Yun, and Fatih Porikli. Hollowed net for on-device personalization of text-to-image diffusion models. Advances in Neural Information Processing Systems, 0444058-43079 2024.
TM Cover and JA Thomas. Elements of information theory 2nd ed., 2006.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-878 8794, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 (496):1602-1614, 2011.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the 41st International Conference on Machine Learning, pp. 12606- 12633, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashmik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.016 01618, 2022.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7323-7334, 2023.
11

---

Published as conference paper ICLR 2026
Amir Hertz, Ktir Aberman, and Daniel Cohen-Or. Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2328-2337, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprini arXiv:2207 /2598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)33 2022.
Jiannan Huang, Jun Hao Liew, Hanshu Yan, Yuyang Yin, Yao Zhao, Humphrey Shi, and Yunchao Wei. Classdiffusion: More aligned personalization tuning with explicit class guidance. In The Thirteenth International Conference on Learning Representations, 2025.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565-26535 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Barnnkkkaaaaii al. Overcoming catastrophic
Kieran Milan, John Quan, Tiago Ramalho, Agnieszka forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-355 2017.
Nupur Kumari, Bingliang Zhang, Richard Zhang. Eli Shechtman, and Jun-Yan Zhu. Multi-concept cus- tomization of -inage diffusion. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1931-1941, 2023.
Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text- to-image generation and editing, Advances in Neural Information Processing Systems, 36:30146-300 30166, 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. International conference on machine learning, pp. 12888-12900 12000. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ing -magge---ageeeee pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208. / 1970, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 15:17759-17372. 2022.
Preetum Nakkiran, Arwen Bradley, Hattie Zhou, and Madhu Advani. Step-by-step diffusion: An elementary tutorial. arXiv preprint arXiv:22406. 08929,,400008929, 2024.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on learning theory, pp. 1376-1400 1401. PMLR, 2015.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Frank Nielsen, Deanel Boissonnat, and Richard Nock. Bregman voronoi diagrams: Properties, algorithms and applications. arXiv preprint 1rXiv:0709.2196, 2007.
12

---

Published as a conference paper ICLR 2026
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann. Tim Dockhom, Jonas Müllrr, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Pengchong Qiao, Lei Shang, Chang Liu, Baigui Sun, Xiangyang Ji, and Jie Chen. Facechain-sude: Building derived class to inherit category attributes for one-shot subject-driven driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7215-7224, 2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320-79362, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference muchine learning, pp. 8748-8763. PmLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/255 1(2)33 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution
image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tniig texl-to-image diffusiin models subject-driven generation. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 22500-22510 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35: 36479-36494, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference Computer Vision, pp. 87-103. Springer, 2024.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-fille tip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 1.02/////////4 12,0001114, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 25:25278-25299 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pp. 2266-2265. pmir, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint axX1v=2010.02502, 2020a.
13

---

Published as conference paper ICLR 2026
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 conference proceedings, pp. 1-11, 2023.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.09522, 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36:8406- 1084006-441,,22 8441, 2023.
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International conference on machine learning, pp. 2825-2834, PMLR, 2018.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455 47487, 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6613-6623, 2024b.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023.
14

---

Published as a conference paper ICLR 2026
APPENDIX
A ROLE OF À AND PRACTICAL REGULARIZATION IN PERSONALIZATION
Although À is motivated by the Lipschitz property described in Theorem 2 its practical value in personalization is determined by the characteristics of each subject. As shown in line 9 of Algorithm [ the regularizer coefficient balances the new learning signal and the preservation of the pretrained capacity. Since learning dynamics differ across subjects, the optimal À varies as well. Moreover, computing the Lipschitz constant of requires access to the pretrained data distribution, whereas personalization provides only handful of subject images, making it challenging to recover distributional characteristics necessary for estimating the exact Lipschitz constant. This limitation highlights the need for empirical tuning in practice.
We attempt to estimate the Lipschitz constant using Monte Carlo approximation over and t, combined with a randomized singular-value estimate of the Jacobian for SD1.5. The estimated Lipschitz constant is approximately 184. As shown in Figure this magnitude tends to prioritize image fidelity over text aligmment. Figure @ further shows that choosing an appropriate value of is important for balancing these two aspects. This estimated constant is smaller than the reported X 500. We note that this difference arises because, as shown in Algorithm 1, the coefficient serves not only as a Lipschitzz-motivaee term but also balancing factor between preserving pretrained capacity and learning subject-specie this reason, personalization may require using a larger X, and doing so does not violate Theorem 2 We also noticed that the theoretical Lipschitz and the practical objective coefficient À were used with a bit of overlap in terminology, and we will clarify this distinction. We note that choosing a different À only changes the strength of the regularization and does not violate the bound on distribution preservation established Theorem P
guidance. For
We Finally, we discuss the norm choice related to Remark 3 We adopt the &g norm as clarified in Remark S
observed that using the raw absolute norm |-| leads to unstable updates, whereas using 1-12 consistently yields stable training. This observation suggests that our formulation shares a degree of similarity with KKrrkpatrick] et al. (2017), the squared & norm yields larger gradients for larger parameter differences and thus provides a mild form of parameter-wise adaptivity.
B DISCUSSION
We provide further discussion on the relationship between evaluation metrics and the claims made in diffusion-based personalization models. Our method introduces flexible mechanism to balance image fidelity and text alignment. As result, we observe model variants that score higher in either image fidelity or text alignment. While the main paper structured around results that show improvements across all metrics, we also observed that some variants with slightly lower text-alignment scores nevertheless produced images that appeared qualitatively satisfactory and well aligned with the prompts. Although this observation subjective, it raises an interesting dilemma. Demonstrating superior performance metric does not guarantee improved qualitative alignment, and this misalignment highlights a core challenge in personalization. While the proposed approach successfully manages the trade-off between metries and often produces results that appear qualitatively superior, we emphasized improvements across all quantitative measures to highlight numerical gains. This underscores a broader challenge in diffusion research: the ambiguity and limitations of current metrics. We believe this calls for a deeper examination of how evaluation metrics are used and interpreted in generative modeling.
baselines every
C ETHICS STATEMENT
Although our work takes an analytical approach, the broader field of personalized image generation requires careful consideration due possible misuse. These models could be used to create misleading or harmful
15

---

Published as conference paper ICLR 2026
images, which may negatively affect individuals or society. This highlights the need consider ethical implications alongside technical progress in this area of research.
D REPRODUCIBILITY STATEMENT
We ensured a controlled and transparent experimental environment to support fair comparison and repro- ducibility. For baseline comparisons, we followed the original hyperparameter settings and code configurations whenever possible. We fixed random seeds during training and evaluation to further enhance reproducibility. Additional details are provided in Appendix H We will release the code together with the paper.
LLM USAGE
We employed large language models (LLMs) in supportive role. We used them primarily for translation, literature search and verification, and for prototyping code consistent with our intentions. The underlying ideas and design were conceived and developed by the authors.
16

---

Published as conference paper ICLR 2026
F PROOF OF THEOREM
Theorem 1. Let p" (a, c) denote the reference distribution, and let the model parameters Obsee have distribution Pr (x, c) satisfying, for any
Ip (xx c) - Pooam (x, c) < E.
Suppose there exists a measurable set D C X x C such that
Padapt (D) = 7/+ D'(D)=
8 > 0.
The model is adapted by gradient descent on the denoising loss Ldeuoise from Eq. trained on (x, c) Padapt De+1 = 0: 17 (VeLDenois((())) where 0g Obase.
2
After sufficient iterations t, pa, Padapt: under universal approximation and convergence assumptions.
Then
DKr(p" (1
Dri(p" (/
Proof.
By the three-point property of Bregman divergences Nielsen et al. 2007 we get
DKL (p" p..) = DKL(P'' pP + DKL (pr |l pe.) + (p" Pr log(pe.) log (pe
Applying the
inequality to the rightmost term,
(p" Perase log(pe.) log(pe ) S I|p" Poawell log(pe.) log(perare)) )|
It follows from the convergence assumption that log(ps,) log(Pe )l < M (M < o0).
Then,
||p" Paomeel log(pe.) - log log(perane)) )ll <EM,
the inner- product term is negligible. Hence,
DKL (p" [p.)) so DKL (p" || P + DK (Pobac Po.)
Applying Pinsker's inequality Cover & Thomas on the biased region D obtain DKL (po
ll 2 2(7 6)2.
Then,
DEL (p" po + DK (Pornm Po.) N DEL (p" || POnass) 2 8)2.
Putting everything together, we obtain
(x po.) = DxL(p" Po
+ DDKI (l + (pp Porano Péo Verned
~ DRL (p"
+ DKI (Po || PP.)
DKL (p"
2( 8)2
DKL (p"
Then,
DKr(p" (( < (xi(p" lppe).
claimed.
17

---

Published as conference paper ICLR 2026
G PROOF OF THEOREM 2
Theorem 2. If the diffusion model Eg is Lipschitz continuous in 9, then for any two parameter sets 0o and 02. there exists constant À >> 0 such that
DEL (po, lo.))) = A ||0 02 Ik-
(5)
Proof. Given that the composition Lipschitz functions preserves Lipschitz continuity (Neyshabur et al. 2015 Asadi et al.) 2018) and that attention mechanisms admit finite Lipschitz constants compact input domains (Castin et al. 2024), we assume s (1 t) is Lipschitz in 0 with finite constant L. Hence for all æ, t and any 01, 02,
||oo, (2,, ()
602 (2,, 0
<
L 1|0
02 ll
By Tweedie's formula Efron 2011 Ho et al. 2020); Song et al. (2020b),
E0(3r,
so(2, t)
Scalar multiplication preserves Lipschitz continuity, ||so, (x. t) s0, (x, t)ll L' |0] - 0211k-
It follows that is Lipschitz in with constant L' = L T
By the probability-flow ODE formulation Song al. (2020b)
Song & Ermon 2019)
T
dæz dt. so( t)
Po = lo pr (2r)
dt
Since (rr))(rT) is constant with respect to and integration preserves Lipschitz continuity,
log po, (I) - loogooo (x) S L" 1|01 021la for Vo E X
holds with 0 L'dt = L o, dt.
Applying the triangle inequality to the definition of KL. divergence then gives / po, (x) (log po, (x) log (r)) dx
(Ko(Po, lIpo,) =
po, (Ir) llogpo (x) - lo, po, (z)) dx
< L " |10 - 02||A-
This completes the proof with L"
"In practice, oe is never exactly zero: implementations either avoid the endpoint(s) or enforee small floor Ø; Omin > O for numerical stability. Hence sup, o. < 3o in our setting. FFollowin standard practice, omit the prompt and write e (z, t) and log (x). All statements extend directly the conditional case by replacing c0 t) with ea e, f) and log pe (x) with log (o( c).
18

---

Published as a conference paper ICLR 2026
H ADDITIONAL IMPLEMENTATION DETAILS
Ours. All experiments are conducted on single NVIDIA RTX 3090 GPU. We use the special token sks, newl depending on the baseline. For the proposed Lipschitz-buunde regularization, we compare both the L and L2 norm variants and observe no meaningful difference in performance; accordingly, we adopt the more widely used L2 norm. We also experiment with integrating DreamBooth's prior-preservation loss but do not obtain improvements, suggesting conflict between the objectives.
experiments across different methods, we primarily used DreamBooth on SD 1.5, as this backbone is the most widely adopted and thus provides a fair basis for comparing diverse approaches. The corresponding hyperparameters are listed above, and other methods were also evaluated on the same backbone for consistency. re-implemented the official codebase to allow more detailed experimentation. This
For DreamBooth, ensured fixed random seeds and controlled variables for fair comparison. Finally, to measure the time required for the personalization process, we performed ten runs for each bachbone-method combination and report the average.
In experiments across different backbones, aimed evaluate the effect of the proposed objective under comparable settings using existing codebases. The hyperparameters were as follows. For DreamBooth on SD-1.5, we set the regularization weight to À 500, trained with batch size of 1 and a learning rate of 2 10-6 for 1,000 iterations. For Custom Diffusion on SD)-1.5, we used À = 175 with a learning rate 3 x 10-5 for 250 iterations. For SD-XL. we considered two settings: À
with learning rate of x 10 for 250 iterations, and À 10 with a learning rate of 8 x 10 For SD-3.0, we used learning rate 10-4, À = 10.1, and 500 iterations. For LoRA, the settings were as follows: on SD-1.5, we used learning rate of x 10-4, À x 10-7 and 500 iterations; on SD-XL, a learning rate of 5 x 10-1, ==1x 10 and 500 iterations; and on SD-3.0, a learning rate of x 10 100 = 15, and 680 iterations.
The choice of backbone and personalization method leads to different hyperparameter settings and training behaviors.
We provide additional implementation details for the baselines used in our experiments in Section The proposed method and DreamBooth are implemented and trained based on the Diffusers library For
Baselines.
Textual Inversion Gal al. 2022 Custom Diffusion Kumari et al. 20233 6 SVDiff Han et al. 2023 and
LoRA Hu et al. 2022 we use available implementations and follow their default training settings. OFT Qiu
ct al. 2023 is trained using the official codebase, but we observed poor performance and were unable
fully reproduce the reported results; hence, we cite the numbers from the original paper. BLIP- ILLP-Diifusion Li ct al. 20233 and AP-Adppeer Chen al. (2023 are evaluated using the released model weights on our benchmark through direct inference.
Moreover, experiments with different backbone models are conducted using the Diffusers library. For Stable Diffusion-XLL -L (Podell et al.] 2023), we follow the same Custom Diffusion reference as above. Since no official implementation of Custom Diffusion for Stable Diffusion-3.0 (Esser et al. 2024) is available, we
inversion
http:///cccccclllooooaooo////rer ffusion hrt.hhhhhhh//../..o/o///oo//o/on
19

---

Published as conference paper ICLR 2026
reimplemented this variant ourselves. For the LoRA version of Stable Diffusion-3.0, we followed the official implementation
Ablation Studies. We provide additional implementation details for the ablation studies on our proposed personalization process in Section 5.2 Following the same training configuration as in Section [ we fix the learning rate and number of training iterations, varying only the regularization strength for the experiments measuring changes from the pretrained model (i.e., 20 and Ae). For De, we measure the 1.2 distance between outputs generated using the unconditional class token from the pretrained and personalized models. The
reported value is the accumulated L2 distance over all sampled outputs.
Toy Experiment. We validate our method using the toy 2D conditional diffusion model as described in Section 4.3 The setup generates 1, 1,000 samples per class from 2D Gaussian distributions with a standard deviation of 0.5. Class labels range from 0 4, positioned at the vertices of regular pentagon. A new sixth class (class 5) supports finetuning. The noise schedule follows cosine- based formulation with T 100 timesteps. The model uses a one-hidden-layer MI.P with ReLU activation and class/time embeddings. Initial training runs for 1,000 iterations capture the base distribution effectively, Training for the new class is conducted for 5,000 iterations to ensure convergence within the experimental setup. We adopt a DreamBooth-like setup by jointly training the new class with class (deep blue) data. Considering the problem's 's complexity, we assign a large weight 100 to class 0 in order to clearly observe the effect of different objectives. When using smaller weights, the other classes tend to collapse toward the new data distribution. However, with weight 100, the objective's ' influence on class O becomes apparent. For our proposed method, we apply Lipschitz regularization with respect to the pretrained weights, using a regularization strength of À = 50. Even with relatively small weight (e.g., 1) on the regularization term, its effect is clearly observable this setup as shown in Section
20

---

Published as a conference paper ICLR 2026
I ABLATION ON THE ToY EXPERIMENT
We conduct an ablation on the toy experiment introduced in Section 4.3 to visualize the trend of the proposed method. As shown in Figure 7 the relationship between the newly learned distribution and the original one varies with the regularization strength A. Without Lipschitz regularization, all classes tend to collapse toward the newly introduced distribution. Notably, even with À = 1, the regularization effect becomes evident. In addition, increasing À progressively enforces preservation of the original distribution, and the newly learned class mapped closer one of the existing modes. These results demonstrate that the proposed objective effectively encourages distribution preservation and supports the balancing behavior observed in Section 5.2 Figure provides additional visual comparisons across different values of A. Similar to the earlier analysis, these results highlight how the regularization interpolates between adapting to new concepts and preserving prior distributions.
=
= 1
X = 10
= 1k
À = 10k
Figure 7: Visualization results under different regularization strengths A.
Examples
Prompt
A 0.0
A 0.5k
A 2k
on
[V] dog in the
[V] toy in the jungle
Figure 8: Additional visual comparison by regularization strength.
21

---

Published as conference paper ICLR 2026
J
USER PREFERENCE STUDY
We conducted a user preference study to further support the proposed method.
J.1 SETUP
A total of 35 participants were asked select the most preferred image among the model outputs generated from the given test prompts: 35 for general preference and 15 for the detailed criteria evaluation. We designed 30 questions covering all classes in the dataset Ruiz al. (2023). and randomized the order of the models for each class when presenting them to the participants. We compared four models: our method, DreamBooth (DB) Ruiz al. (2023). ). DreamBooth LoRA (DB LoRA) Hu al. (2022), and the
conditioner-bass IP- Addatoo Chen et al. 20233 chosen based on their competitive quantitative performance.
The evaluation criteria were explained the participants in advance, and they were instructed to consider the
following aspects comprehensively:
Identity Preservation: whether the identity the user-specified object well preserved.
Prompt Coherence: whether the condition described in the prompt is accurately reflected.
Image Quality: whether the image is free of artifacts distortions.
Overall Preference: jointly considers the three criteria above.
J.2 EXPERIMENTAL RESULTS
The results of the user preference study are shown in Table 3 Our method achieved the highest overall preference 45.811 11, while IP-Adaptor recorded the lowest preference 11.53%. This is in contrast to their positions on the DINO score, where both models ranked among the highest in the quantitative experiment of our submission. We attribute this discrepancy to the Adaatoo'''' inability to accurately represent the target identity, as shown in Figure 4 of our manuscript. DB and DB-LoRA received preference scores of 25. % and 17.52%, respectively.
The detailed criteria enable more fine- the-graaied interpretation of the overall preference and reveal clear tendencies of each method. DB shows strong preference in identity preservation but lower prompt adherence. In contrast, DB-LoRA shows lower identity preservation but twice higher prompt adherence. As discussed Section [5] these tendencies are consistent with the known behaviors of the two methods: DB tends to overfit, whereas DB-LoRA tends to underfit. IP-Adaptor produces good image quality but remains weak in personalization. Our method maintains balanced performance across all criteria. This suggests that the proposed objective captures subject-specific information while preserving the pretrained distribution, leading to consistent and stable results.
Table 3: User study results across four criteria (all values in %).
Method
Identity Preservation
Prompt Coherence
Image Quality
Overall Preference
Ours
44.22
43.78
44.44
45.81
DB
33.33
23.11
24.89
25.14
DB-LoRA
11.78
22.67
13.11
17.52
IP-Ataptor
10.67
10.44
17.56
11.53
22

---

Published as a conference paper ICLR 2026
K ADDITIONAL COMPARISON WITH DREAMBOOTH
We conduct an additional experiment using DreamBooth. is the most comparable method since fine-tunes all parameters as method does. To compare the effect of the objective, unify all hyperparameters except the loss. We set the learning rate of DreamBooth to 2 x 10 6 the same as ours, instead of its default x 10-6. The results are shown in Table Lowering the learning rate in DreamBooth leads an imbalanced outcome. improves the CLIP-T score, but lowers both DINO and CLIP-I scores. This suggests reduced image fidelity. As shown in Figure 9 the visual results fail to preserve subject identity in Rows 1 and 3, although this is not always the case, as seen in Row 2. These findings highlight the difference in objectives. Our method achieves better trade-offs and is more suitable for personalization.
Table 4: Additional quantitative comparison of personalization methods. All scores are identical those reported above, except for the last row.
Method
DINO f
CLIP-T
CLIP-I
Ours
0.6394
0,2976
0.7948
DreamBooth
0.6028
0,2793
0.7881
DreamBooth (IR 2 x 10 ))
0.5707
0.3104
0.7695
Examples
Prompt
Ours
Dreambooth
[V] backpack the beach
a [V] dog the snow
[V] toy in the jungle
Figure 9: Additional visual comparison with other baselines.
23

---

Published as a conference paper ICLR 2026
L ADDITIONAL VISUAL COMPARISON AND FAILURE CASES
We present additional qualitative results and failure cases. As shown in Figure I0, our method captures both subject-specific features and prompt semantics. In contrast, DreamBooth often produces artifacts fails to reflect the prompt, while DB-LoRA tends to preserve the prompt but underrepresents subject identity. Other baseline methods generally struggle to produce coherent results. We also highlight representative failure cases. Figure [T] case that struggles with cartoon-style minor concepts. The model learns the subject's color but fails reconstruct its structural identity similar other methods. In Figure [2 the model fails to properly apply the prompt to the subject. This further shows that prompts requiring changes in the subject's ss appearance are difficult handle. These cases indicate that our method also faces challenges in generating certain subject-prompt prompt combinations. As discussed in Section 6 this underscores the importance of incorporating the distributional structure of both the pretrained model and the target subject.
We also present additional results on SD-XL and SD-3.0 in Figure 13 Both backbone models are advanced architectures, and the overall image quality is generally high. For SD-XL, Custom Diffusion fails to preserve subject identity, which is reflected in the low image fidelity scores. While CD+Ours shows improvements, certain cases (e.g., the first row) remain insufficient. Similarly, LoRA exhibits identity shifts in the first row, whereas LoRA+Our successfully preserves the subject identity, consistent with the improved image fidelity scores. Although CLIP-T remains stable even when it drops in Table I. some failure cases can still be observed. For example, in Figure 1 the second row of SD-3.0 with CD+Ours does not reflect the prompt, and the third row shows only partial adherence. A similar pattern appears in SD>XL. LoRS+Ours, where the prompt is partially captured in the third row. These examples illustrate situations where the text alignment weakens during personalization, which aligns with the observed drop in CLIP-T. We note that although our method improves most metries and visual results, careful handling is still required across different models and subjects.
Dreambooth
Textual Inversion
Examples
Prompt
Ours
Dreambooth
-LoRA
[V] on top of pink fubric
Custom Diffusion
BLIP- Diffusion
SVDiff
IP-Adaptor
Figure 10: Additional visual Comparison with other baselines.
24

---

Published as conference paper ICLR 2026
Dreambooth -LoRA
Textual Inversion
Examples
Prompt
Ours
Drcambooth
shaped [V]
Custom Diffusion
BLIP- Diffusion
teupot
SVDiff
IP-Adaptor
Figure 11: Visual comparison failure cases across baselines.
Dreambooth
Textual Inversion
Examples
Prompt
Ours
Dreambooth
-LoRA
78
[V] cartooe with hiue house in the background
Custom Diffusion
BLIP- Diffusion
SVDiff
IP-Adaptor
Figure 12: Visual comparison of failure cases across basclines 2.
25

---

Published as conference paper ICLR 2026
Examples & Prompt
CD
CD+Ours
LoRA
LoRA+OurS
SX-CL
[V] toy in the jungle
SD-3.0
SD-XL
[V] stuffed animal on a cobblestone street
SD-3.0
SD-XL
[V] stuffed animal on top of wooden floor
SD-3.0
Figure 13: Visual comparison of failure across different backbones.
26


Reviews:
[{"review_id": "zYTIUXSXCf", "reviewer": "Reviewer_LdxF", "summary": "The paper targets overfitting and distributional drift in few-shot personalization of text-to-image diffusion models. It argues that existing objectives (plain denoising, prior-preservation via class prompts) are misaligned with the dual goals of subject fidelity and text alignment because they provide no explicit guarantee of preserving the pretrained distribution. The core contribution is a Lipschitz-based regularization objective that bounds the deviation of the personalized model from the pretrained one.", "questions": "1 Human evaluation: Could you expand user studies (more raters/prompts) and include identity verification (face/instance retrieval), and prompt adherence judged by humans, to validate metric conclusions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 Clear objective-level insight: reframes personalization as a distribution-preserving adaptation problem and exposes misalignment of standard/“prior-preservation” losses with distributional stability.\n\n2 Simple, principled, and practical: the Lipschitz argument leads to a tractable L2 parameter-distance regularizer that is backbone-agnostic, easy to implement, and removes pre-sampling overhead.\n\n3 Solid empirical results: consistent quantitative gains on multiple backbones and personalization strategies; qualitative examples show improved subject identity without sacrificing prompt following. The training-time savings are compelling for practice.", "weaknesses": "1 Assumptions and tightness: The Lipschitz continuity of εθ w.r.t. θ and the resulting KL bound are plausible but high-level; constants (λ) and norm choices are not characterized for realistic models. The theory justifies using parameter-distance regularization but provides limited guidance on magnitude selection beyond empirical tuning.\n\n2 Failure analysis: Some failure cases remain (identity structure, prompt compositionality). It would help to characterize when the regularizer helps or hurts (e.g., highly stylized subjects, complex relational prompts).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2z5aZ1bs5H", "reviewer": "Reviewer_xLt7", "summary": "This paper focuses on the personalization task in the field of image generation. The authors proposed a method to introduce a Lipschitz-based regularization objective that constrains parameter updates during personalization. The experiment shows the method can achieve competitive performance for this task.", "questions": "Please see the section of weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The presentation of figures is great and easy to understand.\n- The math notations in this paper are self-contained and well-defined.\n- The paper writing is easy to follow.\n- The introduction section is good and clearly explains the motivation and the main claims of the paper.\n- The proposed method is straightforward.", "weaknesses": "- I have to say that tuning-based personalization (like, DreamBooth, Custom Diffusion) is outdated. Learning-based personalization is the mainstream in the current image generation community.\n- How did you obtain the results of Fig. 2, or is it just an illustration?\n- Could you explain more about the claim \"Remark 1. Personalization based on the standard diffusion objective (Eq. 2) provides no guarantee of preserving the pretrained distribution and may lead to divergence\"? It is a little bit confusing.\n- The evaluation is all based on UNet-based diffusion models. I am curious about the comparison results based on the DiT-based model.\n- The evaluation lacks the image quality metrics that are also important.\n- There are also some previous works exploring the optimization of personalization learning processes. Have you considered comparing the proposed methods with them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DucEwolZ0l", "reviewer": "Reviewer_Bv3p", "summary": "The paper tackles distributional drift in few-shot personalization of text-to-image diffusion models by adding a Lipschitz-motivated parameter-distance regularizer to the personalization loss. The authors argue standard objectives (and even prior-preservation) don’t explicitly preserve the pretrained distribution (Theorem 1), and show that if the denoiser is Lipschitz in parameters, bounding $|\\theta - \\theta_{base}|$ controls KL shift (Theorem 2).", "questions": "Can you quantify Theorem 2’s $\\lambda$ (constant) for a given backbone (e.g., SD-1.5) via empirical Lipschitz estimation, and relate it to your best-performing $\\lambda$ in training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear statement of objective–goal misalignment; neat, didactic Theorem 1 showing why standard diffusion fine-tuning drifts.\n2. Theory sketch linking parameter Lipschitzness → KL control (Theorem 2); connects elegantly to L2 surrogate.\n3. Consistent empirical lift on fidelity (DINO/CLIP-I) with competitive CLIP-T across multiple backbones", "weaknesses": "1. The objective is effectively weight decay to $θ_{base}$; relation to continual-learning regularizers (e.g., EWC) and why your variant is preferable in diffusion personalization should be made explicit.\n2. You note CLIP-T may drop when fidelity improves; more analysis on prompt adherence failures (per failure figs) would help.\n3. In Table 1, SD-3.0 / “+Ours” row shows “CLIP-I +0.0022” but one entry reads “0.0094”. Why is this error proportion so large?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)