ACCORD: Alleviating Concept Coupling through Dependence Regularization for Text-to-Image Diffusion Personalization
Authors: Shizhan Liu
Hao ZHENG
Hang Yu
Jianguo Li
Venue: ICLR 2026 Poster (2026)
Keywords: dependence regularization
concept decoupling
text-to-image diffusion model
Primary area: generative models
TL;DR: We address the problem of concept coupling in image personalization by reformulating it statistically and introducing two loss functions to minimize it directly.

Abstract: Image personalization enables customizing Text-to-Image models with a few reference images but is plagued by "concept coupling"—the model creating spurious associations between a subject and its context. Existing methods tackle this indirectly, forcing a trade-off between personalization fidelity and text control. This paper is the first to formalize concept coupling as a statistical dependency problem, identifying two root causes: a Denoising Dependence Discrepancy that arises during the generative process, and a Prior Dependence Discrepancy within the learned concept itself. To address this, we introduce ACCORD, a framework with two targeted, plug-and-play regularization losses. The Denoising Decouple Loss minimizes dependency changes across denoising steps, while the Prior Decouple Loss aligns the concept’s relational priors with those of its superclass. Extensive experiments across subject, style, and face personalization demonstrate that ACCORD achieves a superior balance between fidelity and text control, consistently improving upon existing methods.

Published as a conference paper at ICLR 2026
ACCORD: ALLEVIATING CONCEPT COUPLING THROUGH DEPENDENCE REGULARIZATION FOR TEXT-TO TO-IMAGE DIFFUSION PERSONALIZATION
Shizhan Liu', Hao Zheng', Hang Yu' Jianguo Li
Ant Group
[chautaulsz, faruce.zh} @ gom,l.com, (hyu.hugo, lijg.zero @ @antgroup.com
ABSTRACT
Image personalization enables customizing Text-to-Image models with few ref- erence images but is plagued by "concept coupling" model creating spurious associations between subject and its context. Existing methods tackle this indi- rectly, forcing a trade-off between personalization fidelity and text control. This paper is the first formalize concept coupling statistical dependency prob- lem, identifying two root causes: Denoising Dependence Discrepancy that arises during the generative process, and Prior Dependence Discrepancy within the learned concept itself. To address this, we introduce ACCORD, a framework with two targeted, plug-and-play regularization losses. The Denoising Decouple Loss minimizes dependency changes across denoising steps, while the Prior Decouple Loss aligns the concept's 's relational priors with those of its superclass. Extensive experiments across subject, style, and face personalization demonstrate that AC- CORD achieves a superior balance between fidelity and text control, consistently improving upon existing methods
INTRODUCTION
The advancement Text-to-Image (T21) Diffusion Models (Ho et al. 2020; Rombach et al. 2022) has lowered the barrier to generating high-quality and imaginative images from text prompts. How- ever, pretrained T2I models often struggle to accurately produce personalized images, such as those depicting private pets or unique artistic styles. As result, image personalization has gained sig- nificant attention, requiring users to provide several reference images related to the personalization target, which enables T2I models create new images of the target based on text prompts.
The primary challenge of image personalization is "concept coupling". Due to the limited availabil- ity and low diversity of reference images for the personalization target (typically 3-6 images often in similar contexts), the model tends to confuse the target with other concepts that appear alongside in these images. This entanglement hinders the model's ability accurately control the attributes associated with the personalization target based on text. For example, as shown in Fig. 1 the model may interpret "a person carrying backpack" as the primary focus, rather than "backpack" because these elements frequently co-occur in the reference images. Consequently, the generated images of- ten deviate from the intended text prompts, frequently including an unintended person the output.
However, existing methods attempt mitigate concept coupling through indirect and often heuristic means, fundamentally treating it as a symptom of overfitting rather than addressing its root cause. These approaches, while varied, ultimately proxies. Open-source approaches fall into four main categories, each with fundamental limitations. Data regularization (Ruiz et al., 2023] Kumari et al.) 2023) uses superclass datasets preserve model priors but risks distorting concept relationships. Weight regularization (Han et al.] 2023) Qiu et al. 2023) constrains parameter updates to prevent overfitting, which can indiscriminately degrade fidelity. Loss regularization methods (Qiao al.)
2024 Song et al. 2024) introduce heuristic objectives that lack a direct link to the underlying sta-
tistical problem. Region-based methods (Avrahami al. 2023 Zhang al. 2024a) confined
*Corresponding authors.
'This work was done while Shizhan Liu and Hao Zheng (research intern) were with Ant Group. Code is available at: https//gttaal.coommmngoop//AAAA(

---

Published as a conference paper at ICLR 2026
Dependience
Dondisiauul besouulore,
#Cherkpacck,.oo giil) plbackpark (p(grn))
Phackpach" .irllk)) Ras (backgoot [appgerr]js)
Ganeratied iages
Sampled from Generation Distribution Nay, Nap
backpack
Abackiccck
Abac
a aiau
Rerlerence Intiges persoral backpack
Hodel 8, Inferinze
backpack
Finetuning
Image Model
Model
concege decoupling
Figure 1: Illustration of the concept coupling problem. The target a "hackpack** but reference images always pair with "girl". Standard finetuning incorrectly learns bind these concepts, causing the model to generate the unwanted 'girl' and violate the text prompt.
to spatially separable objects and fail for global attributes like style. In addition, closed-source models like GPT-4o exhibit inconsistencies and artifacts stemming from this issue, as observed in recent empirical studies (Chen et al. 2025 Yan et al. (0225). By focusing on symptoms like parameter drift or feature entanglement, these approaches fail to directly model and minimize the unintended statistical dependencies that define concept coupling, leaving a critical gap for more principled solution.
powerful
In this paper, we fill this gap by proposing a new paradigm: we the first formally frame con- cept coupling as a tractable statistical dependency problem. Our analysis reveals that this unwanted dependency originates from two distinct and measurable sources: Denoising Dependence Dis- crepancy introduced during the generative process, and Prior Dependence Diserepancy inherent in the learned personalized concept. This new formalism moves beyond heuristic fixes and allows us to directly diagnose and treat the problem at its core.
To operationalize this insight, we introduce ACCORD (Alleviating Concept COupling thRough Dependence regularization), plug-and-play framework with two targeted, theoretically-grouneee regularization losses. The Denoising Decouple Loss (DDLoss) directly minimizes the dependency discrepancy that accumulates during the denoising process by leveraging the diffusion model as an implicit classifier. Complementing this, the Prior Decouple Loss (PDLoss) corrects the prior dependency of the learned concept by aligning its relationship with other concepts to that of its perclass in CLIPP's semantic space. Together, these losses enable ACCORD to directly minimize concept coupling without relying regularization datasets or overly restrictive weight constraints. Experiments demonstrate that the proposed loss functions alleviate the concept coupling issue in image personalization more effectively, achieving a better balance between text control and person- alization fidelity. Our contributions can be summarized as follows:
We are among the first to formally formulate concept coupling in image personalization as a statistical problem of unintended dependencies and propose ACCORD, a plug-and-plat method that directly addresses concept coupling without requiring regularization datasets or ex- tensive weight constraints.
We identify two distinct sources of dependence discrepancies in concept coupling: Denoising Dependence Discrepancy and Prior Dependence Discrepancy. To address these discrepancies, we propose Denoising Decouple Loss and Prior Decouple Loss, respectively.
Experimental results demonstrate the superiority of ACCORD in image personalization. More- over, the proposed losses prove effective zero-shot conditional control tasks, highlighting the general applicability of our decoupling principle beyond test-time finetuning.
RELATED WORKS
Test-Time Finetuning- based Image Personalization: Test-time fine-tuning, on which this paper mainly focuses, adapts pre-trained T21 models to reference images, offering flexible and balanced personalization at the cost of time and computation.
Existing test-time finettuning methods attempt to mitigate concept coupling through indirect means, which can be grouped into four main categories of proxy-basee regularization, all of which treat the symptoms of the problem rather than its root cause: Data regularization (Ruiz et al. 2023 Kumari al. 2023 augments training with images of both the personalization target and its superclass.
2

---

Published as a conference paper at ICLR 2026
While intended to prevent overfitting, this approach is blunt instrument; limited regularization dataset size and distribution gaps can hinder accurate modeling of concept relationships and re- duce personalization fidelity. Although (He et al. 2025) use LLMs to design structured prompts for diverse regularization data to improve regularization effectiveness, this introduces I.I .Mi-iddueed concept dependencies that may not reflect their true prior relationships. Weight regularization methods (Gal et al. [022] Hu et al., [021]; Han et al. [023) Qiu et al.) [023] [Chen et al. 2024a) con- strain parameter updates prevent overfitting, For example, PaRa constrains the parameter space by reducing the dimensionality of the output matrix, thereby preventing overfitting. Yet, weight regularization can also diminish fidelity by indiscriminately restricting the model's capacity to learn target-speciffe details. Loss regularization approaches, like MagiCapture (Hyung et al. (023) and Facechain-SuDe (Qiao et al.) 2024). introduce objectives such as masked reconstruction or super- class inheritance to promote decoupling. However, their reliance on empirically chosen objectives means they lack a formal basis for why these heuristics should reduce the statistical dependencies at the core of concept coupling. Region regularization limit subjects to specific regions in the at-
tention map (Avrahami et al. 2023], Zhang et al. 2024a Hao et al. 2024) or alternatively refines subject generation by constraining the cross attention map, in Attend-and-
(Chefer et al., 2023). But this spatial proxy for conceptual separation is limited to spatially distinct subjects and struggles with global concepts like style or viewpoint. Perfusion (Tewel al., 2023) further com- bines weight regularization and region regularization using gated rank- I updates and key-locking. However, still cannot theoretically constrain the statistical dependencies between concepts.
Unlike these proxy-based strategies that indirectly target symptoms like overfitting, our work is the first directly model concept coupling as an excessive inter-concept dependency. We then introduce two targeted, aeeenddeecy-rggulliizaiioo loss functions to principledly minimize it.
Zero-shot Image Personalization: Unlike test-time finetuning, zero-shot image personalization avoids test-time training but relies heavily on large-scale pretraining data. While recent closed- source models (e.g., GPT4o, Gemini 2.0) outperform open-source ones in zero-shot personaliza- tion (Wang et al. 2024c] Xiao et al. 2025). they still face issues such as inconsistencies (Yan et al.) 2025) and copy-paste artifacts (Chen et al. 2025). Most open-source models are limited to specific domains (e.g., faces, objects) and cannot fully address diverse personalization needs. Representa- tive approaches include: for subject personalization, methods like InstantBooth (Shi et al. 2024), (Li et al. 2024), and ELITE (Wei et al., (023) focus improved visual encod- ing and hierarchical concept mapping, while others (Song et al. 2024) tackle weak text control by removing the projection of visual embeddings onto text embeddings. For face personalization, In- stantID [Wang et al.] 2024b) extracts both appearance and structural features from cropped faces.
For style personalization. InstantStyle (Wnnn al. 2024a) performs style transfer by injecting IP- Adapter (Ye et al. (003)) features into style-related layers of SDXL (Podell et al.) (023)).
While this paper places less emphasis on zero-shot image personalization, our experiments demon- strate the potential applicability of ACCORD to these approaches.
METHOD
3.1 TEXT-TO-IMAGE (T2I) DIFFUSION MODELS
We begin with brief introduction the T2I Diffusion Model (Ho et al. 2020), which establishes mapping between the image distribution and the standard Gaussian distribution via a forward noise- adding process and reverse denoising process. Specifically, the forward process is composed of T steps, gradually introducing Gaussian noise into a clear image or its latent code Xg. The noisy code at time step E 21,2, ..., T'} is calculated as follows:
X VaeX0 +
(1)
where E ~ No, I) represents Gaussian noise, and a modulates the retention of the original image, decreasing t increases. When T is sufficiently large, xy is approximately standard Gaussian.
The reverse process is modeled as a Markov chain, where a network U4 with parameters 0 is used to estimate the parameters of the true posterior distribution q(x-11 |x,, Xp) based on t and x,, thereby denoising the noisy code. The optimization objective can be expressed as:
Ex.,e.c11
X Ux(x {3
(2)
20
3

---

Published as a conference paper at ICLR 2026
where o, represents the standard deviation of the noisy code at time step t, and Ug (x( c, is the output of the denoising model. During inference, the noisy code X,-1 at time step t - 1 can be sampled from .V (V( (X,, t), I) yielding X Up c, t) + o,et, where Note that the text representation or the conditioning information c is also fed into the denoising model control the generation.
_N I).
To facilitate subsequent discussions, we further introduce the conditional dependence coefficient r for two concepts cp and Cy. given the model's denoised output based on (Cps Gg) time step i.e., Xg. = Uo 1 (Cp, Cg); L + 1). This coefficient can be defined the ratio between the joint probability of the two concepts occurring together in XØ,, and the probability of their independent occurrences in the same representation:
r(Cp. Cylx,e) = plcp, Cy cylxx..)
(3)
p(cp)Xo. Ip(ca xx,)
According to probability theory, cp and Cg are conditionally independent given x,, when r(pp, cylXx,e) = 1; they are conditionally dependent otherwise.
We provide a notation summary in Tab. 7 in the Appendix.
3.2 CONCEPT COUPLING IN IMAGE PERSONALIZATION
Test-time finetuning methods designed achieve image personalization by fine-tuning pre- trained T2I model on a limited set of reference images with the personalization target, denoted as D (x(',, ) Here, N is the number of training samples. x' and ct represent the reference image and the corresponding generation condition for the i-th pair, respectively. Note that et can be either an image caption or a combination of the caption and visual features extracted from the reference images for personalization purposes. In instances where captions for x' are absent, we employ Vision Language Models (VL.ss (Chen et al., 2024b) to generate image captions, aligning with practices in the community. This approach, compared to using prompt templates (Ruiz et al., 2023). yields more meaningful textual concepts and assists in the decoupling of concepts.
One issue that plagues image personalization is concept coupling. As illustrated in Fig. | although the personalization target cp is a specifically designed red backpack, the training set ID consistently pairs the personalized backpack Cp with girl Cy- Consequently, the adapted T2I model often tends to generate an additional girl during inference, which contradicts the original prompt. This phenomenon can be statistically characterized as:
(4) where ||| denotes the absolute value, X8.0 denotes the image generated by the T2I model or its latent code, Cp and c, represent the personalized target condition and the general text condition respec- tively. The personalization target condition Cp can be either the textual trigger words used during LoRA training. the text embedding from (Gal et al. (002), or the image representation from (Ye et al. 2023), while c, denotes superclass of Cp. Additionally, r(c,, Cy) p(c,. : cg) cp)/p(c.)///) p/p(C)) In this context, c, embodies a general backpack, thus encompassing the overall propertics of c, and further characterizing the inherent relationships with other general concepts represented by c. (Ruiz) jet al., [023]] Qiao et al.) (2024). The essence of the equation above is that the generated images x0,0 typically introduce additional interdependencies between c, and c, that are not present in the inherent prior relationships between C, and C.. Indeed,
Exp log r(C,, (ogr(((,,,//xeo)) -
> 0,
Exal r(op Cy Xo,o) - log r(c,, cy)ll > O holds when either (i) r(Cp. Cylxx,)))))) > r(c,, Cg) (overly positive dependence) or (i) r(Cp. (x,00 < r((,, Cg) (overly negative depen- dence). The equality is achieved if and only T((p, CglX0.0) r(c., Cg).
Lemma 1.
Thus, the fundamental goal of concept decoupling is to correct the conditional dependence coeffi- cient between Cp and Cy in the generated images so that it approximates the prior concept depen- dence between C2 and Cg.
3.3 SOURCES OF DEPENDENCE DISCREPANCIES
The direct computation and minimization of the left-hand side (LHS) of Eq. 44 pose significant challenges due to the absence of closed-form expression. Instead, we analyze this discrepancy by introducing an intermediate term log rCcp. Cy (x)), which allows us to separate the total discrepancy into two meaningful and computable components, as formalized in Theorem
4

---

Published as a conference paper at ICLR 2026
personalization condition
Lou
Σ
Blog 1Gg Cyloo-))))) log ( Gpp [ز
backpack
hanging on tree
7
fill-personai plstanace
Test condition
I1?
full full--nstyy distaoot
UNet PEFT
Estimate
T--1
full-testt diaeraune
full condition
I1?
backpack hanging.o
empty condition
Tennisinn neperdence Discrepandу
Minimize
(Cp.Eg)
Figure 2: Denoising Decouple Loss LDD- The UNet estimates X,-1 based on x, and four different conditions, then constrains the relationships between the four denoising results. The objective of LDD is to prevent the conditional dependence coefficient between the personalization target cp and the general text condition Cg from varying significantly between adjacent timesteps.
Theorem 1. The LIIS of Eq. f can be decomposed into the following two terms:
Exp
log C.y x0,0) - log logr Cg xT)
+ log (ogr(cp: -
(5)
① Denoising Dependence Discrepancy
② Prior Dependence Discrepancy
where XT denotes multivariate standard Gaussian noise.
Since Xy is Gaussian noise sampled independently of the conditions cp and Cg. it follows that log r(Cp, Cg [T)) log r(Cp. Cg). The detailed proof is provided in Appendix Therefore, the expression ( equals the left-hand side of Eq. (4).
The denoising dependence diserepancy ! captures the change conditional dependence between Cp and cy introduced during denoising, whereas the prior dependence discrepancy O reflects the alteration in prior dependence due to deviations of C, from C.. The conditional dependence coeffi- cient of Cp and Cy on Xy, log r(cp: Cy), bridges the denoising dependence and prior dependence.
Building on this decomposition, we propose ACCORD, a plug-and blug-and-ug and-play method comprising two loss functions: the Denoising Decouple Loss (DDLoss) and the Prior Decouple Loss (PDLoss). The DDL.ss minimizes the denoising dependence discrepancy by leveraging the implicit classifica- tion capabilities of the diffusion model, while the PDLoss alleviates prior dependence discrepancy, particularly when Cp is trainable, by utilizing the classification capability of CLIP. Collectively, these strategies work synergistically minimize concept coupling, which will be elaborated below.
3.4 DENOISING DECOUPLE Loss (DDLoss)
We first elaborate on the DDloss, which specifically targets the denoising dependence discrepancy. Directly minimizing the denoising dependence discrepancy term in Eq. (5) is not well-aligned with the time step sampling mechanism employed during the training of diffusion models. This incom- patibility arises because the term connects the first and last time steps, bypassing the relationships between successive steps. To address this issue, we propose to relax this term by upper-bounding it with the sum of dependence discrepancies between adjacent denoising steps:
log r(pp. cylxx logr (y/xT))
Σ log Dogg((c.cylx...coo cylxx,,-1) logr(cp, Cylp xx)))
Togr(Cp: (x(x - log r(Cp,
C, (xl)))))
(6)
This relaxation follows from the triangle inequality. Minimizing this upper bound effectively dis- courages the conditional dependence between the personalization target and any other concepts from changing abruptly between consecutive denoising steps.
Next, by exploiting the diffusion model as an implicit classifier QQiao et al. 2024), we can derive closed-form expression for log r(cp. cy Xx, xx, log r(Cp, Cg X
Theorem 2. The dependence discrepancy between successive time steps in diffusioo models can be computed as:
log (Cp, Cg co1x0,1-1) log r(Cp; Cy (x..t
5

---

Published as a conference paper at ICLR 2026
Personal mages
image Encoder
Minimize
Disorepancy
Deparalance
$ App = |cas (p- ) cos
CLIP Text Encoder
Up
fp
[CI.S]6
fo
poos Uur
CLIP Projector
hanging on
CLP Text Encoder
[CLS5G
scos (Lo.5g)
backpack
Figure 3: Prior Decouple Loss LPD- Either the Image Encoder the Text Encoder of CLIP can be used to generate cp. The purpose of LpD is to prevent excessive prior dependence between Cp and the general text condition cg. We first the CLIP projector to map Cp and cy into f, and fg. respectively, and then minimize the absolute difference between cos(fp, fg) and fg.
1 [I (xi, (Cp: Cg),t) Uo (xo,t; Cp: t)12 + |1 (xe: (cp, (cp, Cg Uo 20?
Uo (xo,t, Cg. 1)
- -1 (x4,, (Cp: Cg), t) Ug (, t]
where Ø denotes an empty control condition.
Theorem 2 follows from Bayes' theorem and the Gaussianity of noisy latents at timestep - 1; see Appendix for details. Intuitively, Eq. (7) measures dependence changes by comparing the model's prediction for the joint concept (Cp, Cg) against its predictions for each individual concept and the empty condition, thus penalizing deviations that imply a change in their relationship. Finally, we
define the DDLoss as:
T LDD Σ r (gxx/(/.C//xx..11 log (Cp, Cg X0,x)-
In this formulation, LDD with larger i contributes more to concept decoupling due loss accumu- lation. Therefore, we scale Lnn by a linearly time-varying weight t/T. Moreover, to compute the DDLoss in practice, During diffusion training, we sample individual time steps using Eq. (I] rather than iterating from time step T 0. Consequently, Xa,t is not directly accessible when denoising from / to / 1. (ii) X, serves as an unbiased estimate of X0,t- Additionally, we stop the gradients for Mo Cg. 1) and Ue (x,, Ø, t), following Facechain-SuDe (Qiao al., (2024). to prevent damaging the model's prior
x, instead of X8.1- This approximation is effective for two reasons: (i)
2,t).
knowledge. For ease of understanding, we show the computation of DDLoss in Fig. 2
3.5 PRIOR DECOUPLE Loss (PDLoss)
When Cp remains fixed and close c, during training, the coupling of concepts primarily arises from the first term in Eq. (5). specifically the denoising dependence discrepancy. In this context, minimizing only the DDLoss allows the personalized target to retain its superclass' 's relationship with various text control conditions. However, it is worth noting that can also be trained as either the CLIP text representation (Gal et al., 2022) or the representation extracted from reference images by the CLIP image encoder Ye et al. 2023). to better capture the details of the personalization target. Yet, it is crucial note that training cp may cause Cp to diverge from C, and drastically increase the prior dependence discrepancy (see 2 in 5D). As a remedy, we introduce the PDLoss. Specifically, the prior dependence discrepancy can be equivalently written as:
logr(Cp: Cyy) log r(c,, cg) = log p(Cglcp) p(cylc.)
This equation shows that reducing prior dependence discrepancy involves aligning the conditional probabilities yl p(c,pl cp) and (c.) Unfortunately, the diffusion model does not facilitate this align- the denoising process. Therefore, we leverage the semantic space of CLIP, which is inherently density-aligned due to its training objective. Specifically, CLIP trained with the InfoNCE loss, whose optimization objective is to estimate a density ratio relative to the noise, shown in Lemma (equivalent to Eq. (2) in InfoNCE (Oord al. 2018
ment because Eq. (9) is independent

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative results on DreamBench. The "*" indicates results using per-subject/style loss weights, tuned on small validation set. "Params." indicates the number of tunable parameters. The W(nn)L(oss) rate is calculated by pairwise human comparison between the anonymous generated results of the baseline and Ours*, with ties omitted. 'PA' denotes percent agreement, namely the percentage of samples receiving consistent judgments from human annotators. The comparison
methods improved based on the baseline are italicized.
Method
CLIP-TI
BLIP-T
CLIP-IT
DINO-F
WTLL (%)
(A)))
Params.
DreamBooth (DB)
30.3
40.3
74.0
69.3
18.1775.7
73.3
819.7 M
CoRe-SDi.5
29.4
78.3
72.3
19.2/61.7
60.0
819.7M
Facechain-SuD2.
31.4
41.6
74.3
70.5
14.2/69.2
70.0
819.7 M
DB w/ Ours
31.1 (40.8)
42.1 (+18)
(+3.8)
73.5 (+4.2)
-/-
819.7 M
DB Ours
31.3
(+1.8)
78.6 (+4.6)
74.4 (+5.1)
819.7 M
CullomDillusion (CD)
34.2
56,9
8.1/88.1
76.7
18.3
ClassDiffusion
34.3
45.8
61.3
55.0
7.5/75.8
80.0
18.3M
CD Ours
33.9 (-0.3)
46.4 (+1.0)
71.1 (+8.4)
65.2 (+8.3)
18.3 M
CD w/ Ours*
34.1 (-0.1)
46.6 (+1.2)
71.4 (48.7)
65.6 (+8.7)
-/-
18.3 M
LoRA (SDXL)
34.5
47.0
76.3
72.1
17.6/70.5
70.0
92.9 M
SVDiff'
32.7
43.7
72.6
66.6
1.7/85.0
83.3
0.2 M
Omnigen
35.3
47.8
73.9
68.6
30.8/48.3
46.7
3.8 B
LoRA w/ Ours
35.1 (+0.6)
47.8 (+0.8)
76.8 (+0.5)
71.9 (-0.2)
-/-
92.9 M
LoRA w/ Ours
35.2 (+0.7)
47.7 (+0.7)
77.1 (+0.8)
72.4 (+0.3)
+
92.9 M
VisualEncoder (VE)
25.9
36.1
79.1
75.5
21.1/67.6
56.7
3.0 M
VE w/ Ours
25.9 (+0.0)
35.8 (-0.3)
80.0 (+0.9)
76.0 (+0.5)
4
3.0 M
VE w/ Ours
26.3 (40.4)
36,1 (+0.0)
80.4 (+1.3)
76.7 (+1.2)
+
3.0 M
Lemma For an observation C and condition Ck the InfoNCE objective seeks estimate function F(cj. Ck) which is proportional to the following density ratio: F(c,, ck) x plc,les).
pley)
In the case of CLIP, denote as the temperature coefficient, and let f, and fa be the projected features of two concepts çj and C₁ using the CLIP projection head. Then the function F(c,, C2) is
instantiated the scaled cosine similarity T cos(f,, fx) the joint embedding space. Thus, we have the following approximation:
T cos(ff, fx) x p(c,lcx)
(10)
p(c,))
We then align p(cylep) and p(c,/c.) by ensuring that cos(fp, fg) and cos(f,, fy) are closely matched. Theorem 3. The prior dependence discrepancy can be minimized by the following PDLoss:
LPD Ee,, (f,, fg) cos(f., fg)l
(11)
plc,le,) plc,/e,)
Be,!
(12)
p(c,)
The denominator p(c,) in Eq. (12 is not optimizable. Thus, minimizing PDLoss encourages mini- mization of (p(cyll,)) ppcc, ]. namely aligns p( c,,),) and p(c,lc.). To facilitate understanding, we show the computation diagram of PDLoss in Fig. 3 Note that our formulation, c, is the text embedding of the superclass (e.g., backpack) given by the CLIP Text Encoder, while Cp (e.g., the specifically designed red backpack in Fig [ is often set as either trainable text embedding in CLIP or visual representation mapped to the same space. As both c, and cp exist in this shared space, they fulfill the necessary conditions to apply Eq. (TO). We empirically validate this design choice against several alternative objectives in Appendix D. demonstrating that our formulation provides the best balance between text control and personalization fidelity.
In summary, framework is both modular and broadly applicable. DDLoss be applied to any fine-tuning-batt personalization method without architectural changes, while PDLoss further benefits scenarios where the personalized embedding c, is trainable. Depending the personal- ization setup, the two losses can be used independently or together, making ACCORD a flexible plug-and-play regularizer for alleviating concept coupling.
4 EXPERIMENTS
Experimental Setup. We evaluate method diverse image personalization tasks: subject-driven personalization using DreamBench (Ruiz et al.) 2023). style personalization with StyleBench (Junyao et al. 2024), and zero-shot face personalization FFHQ (Karras al. 2021

---

Published as a conference paper at ICLR 2026
For subject personalization, we use CLIP-T Ruiz et al. 2023 and BLIP2-T (Qiao al. 2024 for text alignment, and CLIP-I and DINO-I Ruiz et al. 2023) for subject fidelity] To reduce back- ground interference, subjects in both real and generated images segmented using the Reference Segmentation Model (Zhang et al.] 2024b). For style personalization, CLIP-T and BLIP-T measure prompt-image alignment, while style similarity is computed using the mean Gram matrix distance (Gram-D) (Gayys et al.] 2016). For face personalization, besides CLIP-T and BLIP-T, we further assess facial similarity using Face-Sim (the average cosine similarity of ArcFace (Deng et al. 2019) embeddings for real and generated faces), validated by IP-Adapter YEe et al. 2023). We compare
our approach with 10 baselines (Hu et al., [021]] Ruiz al. 2023 Kumari et al. [023]] Han et al.,
2023 Ye et al. 2023; Qiao et al.) 2024] Huang et al. 2025 Wu et al. 2025 Frenkel et al.) 2024
Xiao et al. 2025). Our losses are integrated as plug-and-pla
module, leaving architectures and hyperparameters unchanged. Only DDL.sss is used for methods that do not update the personalized
embedding (c.g., DreamBooth, LoRA), while both losses are applied otherwise.
4.1 PERSONALIZATION EXPERIMENTS
We report quantitative results for subject, style, and face personaliza-
Table 2: Quantitative results on StyleBench. The "*" denotes
adjusting DDLoss and PDLoss weights across different styles.
tion in Tabs. 15 and visualization
"Gram-D" the gram matrix distance.
results in Figs. S More visualiza-
Method
CLIP-Tf
BLIP-TT
Gram-DJ
tions are provided in Appendix
DrcamBooth
31.3
46.6
42728
Subject Personalization.
We compare the performance of different methods on subject per- sonalization in Tab. D and Fig. [ Compared methods include: data- egglarization-based Dreambooth
Facechain-SuDe
31.0
45.8
39978
DB w/ Ours
31.9 (+0.6)
47.3 (+0.7)
42524 (-0.5%
DB w/ Ours*
32.0 (+0.7)
47.2
(+0.6)
41911 (-1.9%)
CustomDiffusion
31.2
47.7
53347
ClassDiftusion
31.8
48.4
52998
CD w/ Ours
31.7 (+0.5)
48.5 (+0.8)
48649 (-8.8%
CD w/ Ours*
31.8 (+0.6)
48.5 (+0.8)
47852 (-10.3%)
and CustomDiffusiin, weight- regularization-batt based LoRA and
LoRA (SDXL)
33.1
49.7
47193
Omnigen
31.9
47.5
45067
SVDiff, loss-regularization-ban Facechain-SuDD and ClassDiffu- sion, region-regularizatton--asss CoRe, and the zero-shot method Omnigen. It can be observed that: (i Our method improves Dream-
B-LoRA
33.0
49.0
42048
LoRA (SDXL) w/ Ours
33.6 (+0.55
50.7 (+1.0)
47693 (+1.1%)
LoRA (SDXL) w/ Ours*
33.6 (+0.5)
50.7 (+1.0) 46361 (-1.8%)
VisualEncoder
17.7
30.2
32176
VE w/ Ours
17.7 (+0.0)
30.3 (+0.1) 31382 (-2.5%)
VE w/ Ours
18.4 (+0.7) 30.9 (+0.7) 27984 (-13.0%)
Booth and CustomDiffusion by large margin. They utilize regularization dataset to enhance text alignment, but may inadvertently sacrifice subject fidelity. This issue arises because the regularization dataset may confuse the model in distinguishing which concepts from the reference images require personalization and which do not. As a result, the model's focus on the personal- ization target is diminished, leading to loss of personalization fidelity. Our method significantly improves personalization fidelity by complementing the regularization dataset with explicit concept decoupling, (ii When compared to LoRA and VisualEncoder, which do not utilize a regularization dataset, ACCORD shows smaller improvements. Nevertheless, ACCORD is able to enhance both text alignment and subject fidelity simultaneously, while most existing image personalization methods (Han et al.) [023] Qiao et al., 2024 Wu et al. 2025) tend to improve one aspect at the expense of the other. Notably, LoRA (SDXL) with ACCORD even outperforms the powerful Omnigen with 3.8B parameters, testament to the efficiency and effectiveness of our approach. (iii) Our DDLoss and PDLoss significantly enhance the performance of existing baselines in plug-and-play manner. Compared to the similar plug-and-play loss regularization methods Facechain-SuDe, ClassDiffusion and CoRe, our proposed loss functions offer stronger regularization by directly optimizing concept coupling, resulting in greater performance.
We also conduct a study on human preferences regarding the generated results, as shown in Tab. Specifically, annotators
presented with quadruplets consisting of (prompt, reference images, method 1 result, method 2 result) and are asked to select the better generation result based on two key criteria: (i) fidelity to the personalized subject or style, and (ii) alignment with the text prompt. The correspondence of method I (or 2) to either the compared method or our method is random-
"The "T" denotes text and the "T" denotes image, respectively.
8

---

Published as a conference paper at ICLR 2026
Reference
Dreambooch
Facechain
Coke
Ours
Refermmee
CustomDiilision
ClasDDifisson
Our
Prompt: sneake of road
Prompt backpack" on cobblestone street
Prompt: toy rug
Prosept: can" with Eiffe! Tower the background VisualEnooder
Reference
LoRA (SDXL)
ChunGGen
SVDit
Ours
Referreee
Ours
Prompt: in the jungle
Promgt teapot" on top of pink fabric
Prompt: shiny
Pronpt cobblestone
Figure 4: Subject personalization comparison across baselines, where superclass* the personal- ization target. One of multiple training references is shown. Red/blue circles highlight well-/poorly- generated regions. Our method achieves superior text alignment and personalization fidelity.
Reference
LoRA (SDXL)
B-LoRA
Onunggen
Ours
Relernsee
IP-Adapter
Ours
Prompt: Japanism, style*, two pandas playing in bamboo forest
style". a sunflower Prompt:
Prompt: Classicsm style*, curious fox in snowy landscape
(a) Style Personalization
(b) Face Persosdication
Figure 5: Comparison style and face personalization results; style* denotes the target style. For style personalization, the training set includes multiple references, and one is shown for brevity. Red circles highlight well-generated regions; blue circles mark areas with poor results. (a) Our model outputs styles closer to reference images: the Japanism result resembles painting, the minimalist anime style result depicts the mouth as a line, and classicism result matches the original style without anomalies. (b) IP-Adapter alters gender (row or makes faces appear older (row 2). Our method better replicates details such as beards (row 3).
ized and anonymized. We collect feedback from multiple annotators, resulting in a total of 1,800 responses. From this study, we observe that: (i) Our method is generally preferred by users com- pared all baselines; and (ii) Notably, the greater the improvement objective metrics over the baseline provided by method, the more it is preferred by users, indicating alignment between subjective and objective evaluations.
Style Personalization. We additionally compare with B-LoRA, which is specifically designed for style transfer by consolidating the training of two blocks for separating style and content. Tab Tab[lam and 5lg show that DDLoss and PDLoss significantly improve style personalization and boost all methods in a plug-and and--nddpaaa fashion. Similarly, LoRA (SDXL) with ACCORD, with 93M trainable parameters, outperforms Omnigen with 3.8B.

---

Published as a conference paper at ICLR 2026
Table 4: Impact of Reference Image Count on
Ablation study on the effects of DDLoss, and PDLoss across backbones,
Table 3:
Subject-driven Personalization Performance.
Method
Method
CLIPT
BLIPT
CLIP-
DINO-I
(Image Count)
CLIP-T
BLIP-T
CLIP-I
DINO-1
(SDL.)
25.9
36.1
79.1
75.5
VE (I)
25.0
34,2
75.9
71.0
+PDLoss
26.2
35.9
80.0
75,9
VE (1)
24.7
33.3
78,9
73.9
+DDLoss
26.0
35.8
79.8
75.8
+PD & DDLoss
26.3
36.1
80.4
76.7
VE (3)
25.0
34.5
78.0
74.3
VE Ours (3)
25.6
34.8
79.4
75.7
VE (SDXL)
27.1
38.4
82.8
77.6
+PDLoss
27.8
39.5
82.9
77.4
VE (all)
25.9
36.1
79.1
75.5
+DDLoss
28.0
40.0
82.6
77.9
VE+ Ours (all)
26.3
36.1
80.4
76.7
& DDLoss
28.3
39.8
83.1
78.1
Table 5:
Quantitative results FFHQ.
LoRA (SD1.5)
31.1
42.6
78.4
74.6
+DDLoss
31.8
43.0
78.4
75.1
Method
CLIP-TY
BLIP-TT
Face-Simf
LoRA (FLUX)
33.4
46.8
75.8
72.8
IP-Adapter
20.0
34.7
14.8
+DDLosS
34.8
47.8
78.2
73.4
+ Ours
20.7 (+0.7)
34.8 (40.1)
16.4 (+1.6)
Face Personalization. We validate the potential of concept decoupling for zero-shot personaliza- tion, with a specific focus on face personalization, using the FFHQ dataset. Following the well- known zero-shot face personalization method IP-Adepeer, we train the model with and without AC- CORD based on SD 1.5. Experimental results are shown in Tab. and Fig. [5]), demonstrating that the introduction of DDLoss and PDLoss simultaneously enhances face similarity and text alignment.
4.2 ABLATION STUDY
We study the impact of the proposed PDLoss and DDLoss DreamBench in Tab. 3, and also investigate the im- pact of the number of reference images in Tab. 4 Indeed, the proposed loss functions work synergistically and hold regardless of the number of reference images and T2I backbone (including FLUX). Crucially, these studies con- firm that both DDLoss and PDLoss contribute positively to performance (Tab. 3) and that our method remains effective even with single reference image (Tab. 4. underscoring the ro- bustness of our approach.
Table 6: Ablation Study on DDLoss and PDLoss Weights.
Loss Weights
CLIP-T
BLIP-T CLIP-I DINO-I
CustomDiffussio (CD)
34.2
45.4
62.7
$6.9
+0.1DD 0.001PD
33.9
46.5
70.7
64.9
+0.IDD + 0.002PD
34.0
46.5
70.5
64.8
+D.I + 0.003PD
33.9
46.4
71.1
65.2
+0.2DD 0.001PD
33,9
46.5
70.7
64.8
+0.2DD 0.002PD
33.9
46.5
70.8
65.1
+0.2DD + 0.003PD
34.0
46.6
70.7
65.0
+0.3DD + 0.001PD
33.9
46.4
71.0
65.3
+0.3DD 0.002PD
34.0
46.5
70.8
65.1
+0.3DD 0.003PD
34.0
46.5
70.7
64.9
We further study the effect of combining DDLLss and PDL.sss with different weights on CustomD> iffusion (CD) in Tab. 6 Introducing DDLoss with weights between 0.1 and 0.3, and PDLoss with weights between 0.001 and 0.003, consistently yields robust improvements across all metrics. This indicates that the performance of DDLoss and PDLoss not sensitive to the precise choice of weights within these ranges. While variations in the weights have minimal impact on text align- ment, we find that assigning a relatively larger weight to one loss and smaller value the other (e.g., ODIDD 0.003PD 0.3DD 0.001PD) is generally more beneficial for personalization fi- delity than either both being too small both being too large. This may because both losses being small provide insufficient constraint on dependence discrepancy, while both being large excessively emphasize the auxiliary objectives and might hurt the diffusion goal.
CONCLUSION
This paper tackles concept coupling in image personalization by reframing as a statistical de- pendency problem. We identify two distinct sources a Denoising Dependence Discrepancy and a Prior Dependence Diserepancy- and introduce two corresponding plug-and-play losses, DDLoss and PDLoss, to directly mitigate them. Comprehensive experiments demonstrate that our method, ACCORD, successfully improves the critical balance between personalization fidelity and text con- trol, offering readily-integrable solution for wide range of existing methods.
10

---

Published as a conference paper at ICLR 2026
6
ACKNOWLEDGEMENT
This work was supported by Ant Group Research Intern Program.
7 REPRODUCIBILITY STATEMENT
To improve reprodueibility, for theoretical results, such as Theorem 2 we provide the proof's in Appendix B On the other hand, for experimental results, we provide the implementation details in Sec. Appendix M and N The VLM prompt used for generating image captions is also specified in Appendix
REFERENCES
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from single image. In SIGGRAPHI Asia 2023 Conference Papers, pp. 1-12, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM transactions on Graphics (TOG), 42(4):1-10, 2023.
Shangyu Chen, Zizheng Pan, Jianfei Cai, and Dinh Phung. Para: Personalizing -oxt-to-image dif- fusion via parameter rank reduction. In The Thirteenth International Conference Learning Representations, 2024a.
Sixiang Chen, Jinbin Bai, Zhuoran Zhao, Tian Ye, Qingyu Shi, Donghao Zhou, Wenhao Chai, Xin Lin, Jianzong Wu, Chao Tang. Shilin Xu, Tao Zhang, Haobo Yuan, Yikang Zhou, Wei Chow, Linfeng Li, Xiangtai Li, Lei Zhu, and Lu Qi. An empirical study of gpt-40 image generation capabilities, 2025. URL.ttps: http///axvvv./// paattps://axxx..or///aabaa /550 05979
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguiii tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafciriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4690-4699, 2019.
Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or, Implicit style-content separation using b-lora. In European Conference on Computer Vision, pp. 181-198. Springer, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414-2423, 2016. doi: 10. 11w.11999..22.....2.2
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 7323-7334, 2023.
Shaozhe Hao, Kai Han, Zhengyao Lv, Shihao Zhao, and Kwan- Yee K Wong. Conceptexpress: Har- nessing diffusion models for single-image unsupervised concept extraction. In European Confer- Springer, 2024.
ence on Computer Vision, pp.
Xingzhe He, Zhiwen Cao, Nicholas Kolkin, Lantao Yu, Kun Wan, Helge Rhodin, and Ratheesh Kalarot. A data perspective enhanced identity preservation for diffusion personalization. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3782-3791. IEEE, 2025.
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09663 2021.
Jiannan Huang, Jun Hao Liew, Hanshu Yan, Yuyang Yin, Yao Zhao, Humphrey Shi, and Yunchao Wei. Classdiffusion: More aligned personalization tuning with explicit class guidance. In The Thirteenth International Conference on Learning Representations, 2025.
Jun Ahn Hyung, Jaeyo Shin, and Jaegul Choo. Magicapture: High-resolution multi-concept portrait customization. In AAAI Conference on Artificial Intelligence, 2023.
Gao Junyao, Liu Yanchen, Sun Yanan, Tang Yinhao, Zeng Yanhong. Chen Kai, and Zhao Cairong. Styleshot: A snapshot on any style. arXiv preprint arxiv:2407.01412. 2024.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12): 4217-4228, 2021.
doi: 10.1 100/TPAMM..220029999000 2970999.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, pp. 1931-1941, 2023.
Dongxu Li, Jumnan Li, and Steven Hoi. Blip Blip-diffusion: Pre- trainaddd subject representation for con- trollable text-to-image generation and editing. Advances in Neural Information Processing Sys- tems, 36, 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 180.v./807.03744. 2018.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2023.
Pengchong Qiao, Lei Shang, Chang Liu, Baigui Sun, Xiangyang Ji, and Jie Chen. Facechain- sude: Building derived class to inherit category attributes for one-shot subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7215-7224, 2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36.79320-79362 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 22500- 22510, 2023.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image gen- eration without test-time finetuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8543-8552, 2024.
Yeji Song, Jimyeong Kim, Wonhark Park, Wonsik Shin, Wonjong Rhee, and Nojun Kwak. Harmo- nizing visual and textual embeddings for zero-shot text-to-image customization. arXiv preprint arXiv:2403.1 14155, 2024.
12

---

Published as a conference paper at ICLR 2026
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation, 2024. URL https:
//axxiv.org/aat /20..24066.00 06525
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-
image personalization. In ACM SIGGRAPH 2023 conference proceedings, pp. 1-11, 2023.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Ra- sul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-ar diffusion models. https://github. com/
/auginngface/ddfusesss 2022.
Haofan Wang, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. In- stantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.02733, 2024a.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin. Anthony Chen, Huaxia Li, Xu Tang. and Yao Hu. Instantid: Zero-shot identity identity-preserving generation in seconds. arXiv preprint arXiv:2401.05519 2024b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo. Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction all you need. arXiv preprint arXiv:2409. 18869, 2024c.
Yuxiang Wei, Yabo Zhang. Zhilong Ji. Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15943-15953, 2023.
Feize Wu, Yun Pang, Junyi Zhang, Lianyu Pang. Jian Yin, Baoquan Zhao, Qing Li, and Xudong Mao. Core: Context-regularizee text embedding learning for text-to-image personalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 8377-8385, 2025.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li,
Shuting Wang. Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. In Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp. 13294-13300 13304, 2025.
Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation, 2025. /
2504. 02782
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for -xt-te--mmae diffusion models. arXiv preprint wwXiv:23308.0672/1 2023.
Yanbing Zhang, Mengping Yang, Qin Zhou, and Zhe Wang. Attention calibration for disentangled text-to-image personalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4764-4774, 2024a.
Yuxuan Zhang, Tianheng Cheng, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Evf-sam: Early vision- lsionnlangguage fusion for text-prompted segment anything model. arXiv preprint arxiv:2406.20076, 2024b.
A APPENDIX
APPENDIX
A PROOF OF THEOREM
We begin by briefly reviewing Theorem 1. The left-hand side (LHS) of Eq. can be decomposed into the two terms as in Eq. 5):
Exp (1 log (Cpp, Cy xo,0) log (ggr(c..cg) Cy)
13

---

Published as a conference paper at ICLR 2026
Table 7: Meanings of notations.
Notation
Meaning
Denoising time step, ranging from 0 to
Clear image its latent code.
Noisy image or its latent code time step f. Noisy image or its latent code at time step T, modeled a multivariate standard Gaussian noise. Retention ratio of the original image at forward time step f.
Multivariate standard Gaussian noise.
Network parameters.
Standard deviation of the noisy code at time step f. Output of the denoising model time step f Shorthand for denoising output at time 1 given generation condition (Cp, Cg):
cx,,,t
given generation condition c.
N0,
13
Training set for the image personalization task.
x!
i-th reference image in the training set. generation condition the training set.
Cp
Personalized target condition. General text condition.
Co
Text condition the superclass of Cy. Conditional dependence coefficient for concepts cp and cy given generated image Ng.1-
r(Cp, Cg) lps Lg
Prior dependence coefficient for concepts cp and Cg.
Projections using CLIP Projector for Cu, Ca, and Cy.
Exx
log (Cg: Col logr (Cp. Cg CglXT) ) + logr Cg) log (C,, cg)
(13)
① Denoising Dependence Discrepancy
Prior Dependence Diserepancy
where Xy denotes multivariate standard Gaussian noise.
Since Xy is sampled independently of the conditions cp and cg. it follows that p(c|xt) p(c).
Consequently,
log p(cp, cylxT) llog plcp: Cg) p(c,/xT)p(c,/ee p(cp)p(cg)
(14)
Thus, the proof is complete.
PROOF OF THEOREM 2
According to the definition of rCpp Cyl x01x0,111)
r(Cp. Cg x0, = p(cp, xo.x00
(15)
the core computing log r(cp, cg (8.11)) lies the computation of p(c(xx,0-1). where è is an
arbitrary condition. By applying Bayes' theorem, we have:
p(c(x0,-1) = xx, p(xxx
(16)
p(xa,11 (x,e))
The first equation holds because the computation of X0,1-1 relies X0,1 X = Uo (xt, (Cp, ty), x; X + 0+111
E+11 ~ (O,,
(17)
where 0+11 is the standard deviation of the noisy code at time step / + 1.
Next, we compute p(x0,t-1 Xx,e, é) and p(xo,e-1|x0.1) diffusion models, p(x0,t(1111111 X0,1, é) is Gaussian distribution that can be parameterized as:
((x9,t-1X00,5 é) = 10 (x e, t), o = exp(C ||x (x é, 1)l12
(18)
20
where C' is constant. We then substitute Eq.
11 into Eq. 18 and obtain:
[1 X,, (Cp, Cg), Ugl c,
p(xo,-1|X0055 é) = exp(C
(19)
20
Note that è an arbitrary condition, so p(x. - xx,;) can be obtained by setting 2. Therefore, we substitute Eq. (19) into Eq. (16) and obtain:
log p ((( log (()
14

---

Published as a conference paper at ICLR 2026
Average denolsing dependence discrepancy
Average cosine similarity discrepancy
67
08
value
0.0
value
0.5
baseline
ours
baseline
0.1
ours
0.-
200
680
260
1000
230
450
1000
steps
steps
(a) Denoising dependence discrepancy
(b) Cosine similarity discrepancy
Figure 6: Visualization the impact of DDLoss and PDLoss.
1 20 [x (Cp, Cg). Ug Lo (No,r, 0,1) 1126 (x1, (Cpp Cy).
Uo (x0,1, c, e)]
(20)
Finally, by substituting Eq.
(20) into the definition of rCpp, Cy (x))
15
we obtain:
log (Cpp xg]xx,=-1) log: Gig xx,t)
U (x.. (Cp, cg), Ug Xx,r: Cp: t)
20
La (xe: (Cp, cg), t) Ug (x8,1, Cy. t) ||2
Mi (xe: (Cps Cg),() Uo (x 2, 1) [12]
(21)
This completes the proof.
Table 8: Ablation study on the PDLoss design.
Optimization target
CLIP-T
BLIP-TT
CLIP-If
DINO-I
VisualEncoder wo/ Ours
25.9
36.1
79.1
75.5
Ee,
|oos(f (fp, f,y) cos(f,, fg)l]
26.2 (+0.3)
35.9 (-0.2)
80.0 (+0.9)
75.9 (+0.4)
Ee, cos [pp, f,
26.4 (+0.4)
36.8 (+0.7)
79.9 (+0.8)
75.5 (+0.0)
Be,
cos f.. f,) 11]
27.7 (+1.8)
38.4 (+2.3)
77.6
73.3 (-2.2)
Ex,
cos(( Cp fg, Eg)
26.5
36.9
79.5
75.5
C IMPACT DDLOSS AND PDLOSS REDUCING DEPENDENCE DISCREPANCY
To clearly demonstrate the roles of DDLoss and PDLoss during training, we visualize their effects Fig. 66] It can observed that with the use DDLoss, the increase in denoising dependence dis- crepancy. logr(c Cg X0.0) log r(Cp, Cy Xr)). J. is suppressed. On the other hand, the application of PDLoss results in a reduction in the cosine similarity discrepancy |cos (fp. f,)) cos (f., f-)l-
D ABLATION STUDY ON THE IMPACT OF PDLOSS DESIGN
To minimize concept coupling Eq. 4h:
Exol log r(cp. Cg Xo.x) - log r(c,, Il we align the cosine similarity cos(f,, f.. f2) with cos(f,, f,) Eq. [II
(22)
LPD = E,, [[cos(f,, f,) cos f,)l
(23)
To further understand the role of the cosine similarity target in PDLoss, study its impact in Tab. In addition, we also compare our PDLoss with an empirical design: Ee, cos(ff, fy. f, f,) It is observed that: (i) As the cosine similarity target decreases, metrics related to text alignment, namely CLIP-T and BLIP-T, improve, whereas metrics associated with personalization fidelity, such CLIP-I and DINO-I, decline. This observation aligns with derivation.
15

---

Published as a conference paper at ICLR 2026
Reference
Baseline
Sex
Ours
Appiia
duck_toy' in the snow
sneaker* with wheat field in the background
purple bowl*
cat on a cobblestone street
dog* with city the background
Prompt
A comparison of the visual outcomes of subject personalization, where "superclass** denotes the personalization target. In the Ist, 2nd, and 3rd columns, our method aligns better with the prompt and successfully generates a snowy scene, wheat field, and a bowl with imner purple wall; in contrast, the baseline model fails to do so. In the 4th and 5th columns, our method generates subjects that bear a closer resemblance duces an unrelated cat (4th column) or generates anomalies like a black dog's back (5th column). It should be noted that for columns 1, 2, and 5, our method not only replaces the background but also adjusts the perspective make the generated image look more natural.
Figure 7:
the reference images. However, the baseline either pro-
A lower cosine similarity indicates a reduced p(c, cp), implying that Cp is less likely to interfere with other text concepts. However, if the similarity between Cp and Cg decreases excessively, it becomes challenging for to maintain inherent relationships with its superclass and other con- cepts, thereby impairing personalization fidelity. Consequently, setting the cosine similarity target as cos(f,, f,) achieves a balance between text alignment and personalization fidelity. (ii) The em- pirical approach, EE,, [||1 cos(f,, fy. f. f,) []]. also improves upon the baseline by emphasizing text alignment. However, this method cannot be derived from Eq. (9). namely the definition of prior dependence discrepancy.
log rogr(Cp, Cg) loo (ogr((.,, Cg) = log p(cglcp) p(cylc,)
(24)
E APPLICATION ON MULTI-SUBJECT PERSONLIZATION
Table 9: Performance of ACCORD on multi-subject personalization by integrating into Break-A- Scene (Avrahami al. (2023)
Method
CLIP-T
BLIP-
CLIP-I
DINO-I
Break- A-Scene
31.1
42.0
51.6
36.1
w/ Ours
31.2
42.0
53.2
38.7
We explore the compatibility of ACCORD with sulti-subject personalization methods. Break-A- Scene is well-known multi-subject personalization method that achieves disentanglement of multi- ple subjects by randomly sampling subject combinations, computing diffusion loss based on explicit masks, and constraining the cross-attention map. Notably, the DDLoss and PDLoss proposed by
16

---

Published as a conference paper at ICLR 2026
Reference
creature*
howl
stone
Prompt
BAS
BAS Ours
oreature" priswiing frow. aunmyy gardies.
bowl*
panda*
shell*
paadda rests on the grass and bawt are placed wearity
bird1*
bird2*
is peroked ow sigus city.
LON
TPRACLI DINEVE
rabbit
cloth
rabbic wesring sloth" staudinng las sawing studio..
Figure 8: Qualitative comparison of multi-subject generation results, where 'BAS' short for Break-A-Scene, ACCORD enhances the personalization fidelity of Freak-A-Scent in the first and third rows. the second row, ACCORD correctly generates shell as specified by the prompt, and in the fourth row, unlike Break-A-Scene, it does not generate two rabbits.
ACCORD compatible with Break-A-Scene, Therefore, incorporate ACCORD into Break-A- Scene and conduct comparative experiments on its example dataset. The prompts used for evaluation constructed by GPT, and include both single-subject generation (i.e., generating of the multi- ple subjects present in the training image) and compositional multi-subject generation. As shown in Tab. ACCORD further improves concept decoupling capabilities top of Break-A-Scene. The personalization fidelity metrics CLIP-I and DINO-I are significantly improved, suggesting that con- cept decoupling facilitates the model's ability better capture the appearances of distinct subjects. As illustrated in Fig. 8 ACCORD enhances the personalization fidelity of Break-A-Scent in the first and third rows. In the second row, ACCORD correctly generates shell as specified by the prompt, and in the fourth row, unlike Break-A-Scene, it does not generate two rabbits.
F SENSITIVITY TO VLM-GENERATED CAPTIONS
Table 10: Sensitivity of ACCORD to VLM-generated Captions.
Caption Source
CLIP-T
BLIP-T
CLIP-I
DINO-I
Intern VL2-8B
34.1
46.6
71.4
65.6
Owen3-VL-S8
34.0
45.8
71.8
66.1
GL.M-4.5VTThinking...
33.9
46.1
72.9
67.9
Human
34.5
46.8
71.8
65.7
We assess the sensitivity of ACCORD to the quality of VI.M-generated captions for subject per- sonalization using CustomDiffusion. shown in Tab. 10 While captions generated by different VLMs can influence the results to some extent, the degree of variation remains relatively small. No- tably, captions generated by annotated captions. This suggests that employing automated VLMMbased captions for ACCORD is both reasonable and effective.
9B exhibit higher fidelity, surpassing human-
We further analyze the generalizability decoupling achieved by optimizing DDLoss and PDLoss. We observe from existing results that, in some cases, when concept not specified in
17

---

Published as a conference paper at ICLR 2026
Missing description
Baseline w/ Ours
Reference
Prompt
Bascline
shiny boot
wwteerial
$ro
vedl tay
bowe the snow
white fabrio
YP
Appas
vRsc on the beach
tray
Figure 9: Generalization concept decoupling ACCORD via DDLoss and PDLoss optimiza- tion. 'Missing description' indicates that relevant information is absent removed from the training prompts, whereas 'Prompt' refers to those used for image generation. Even in cases where descrip- tions of material, color, or other objects are absent from the training prompts, ACCORD may also be able to decouple these concepts by generalizing the relationships between the personalization target's superclass and other concepts.
18

---

Published as a conference paper at ICLR 2026
the prompt, ACCORD may still able to decouple it. To further test this, conduct experiment in which we remove a particular concept from the prompts within the training set, despite the concept being present in the corresponding images. We then train ACCORD on this modified dataset and evaluate whether the generated images suppress the coupling between the personalization target and the excluded concepts. The results are shown in Fig. | descriptions of material, color, or other objects are absent from the training prompts, ACCORD may also be able decouple these concepts by generalizing the relationships between the personalization target's s superclass and other concepts.
It is observed that even in cases where
G PROMPT-BASED SELECTIVE PERSONALIZATION
Added description
LoRA (SDXL)
LoRA w/ Ours
Reference
Prompt
plushie* on top of the sidewalk in a arowded street
bludd hat
bowl top of green grass with sun-flower around
text "Bow. Appetít"
Beo
dog in firefighter outfiit
collar and tag
a leather
nubber
toy
Figure 10: Qualitative exploration on selective attribute decoupling in subject personalization using ACCORD, 'Added description' denotes the inclusion of elements not intended for personalization in the training prompts, whereas 'Prompt' refers to those used for image generation. ACCORD successfully prevents the personalization of the blue hat the teddy bear's head, the lettering the bowl's side, and the collar and tag on the dog's neck, and it yields better result of the leather duck toy.
The ACCORD's decoupling between the personalization target and other conditions in the prompt provides users with the flcxibility to specify concepts they do not wish to personalize, especially when such concepts are difficult to remove from the images. We qualitatively explore this capabil- ity by selecting subjects with multiple attributes from DreamBench and manually including certain attributes that were not intended for personalization the corresponding prompts. Subject person- alization experiments are then conducted on this dataset, and the results are presented in Fig. I0] As shown in the figure, when attributes that are not intended to be personalized are included in the prompts, ACCORD successfully avoids personalizing these attributes. This may be useful in sce- narios where desirable to modify certain properties of the subject without damaging the subject to be personalized.
19

---

Published as a conference paper at ICLR 2026
Table 11: Performance of ACCORD by prompt category on DreamBench.
Method
Prompt Type
Percentage
CLIP
BLIP-T
CLIP-1
DINO-I
Background Replacement
24%
35.1
47.0
65.2
58.8
Situational Placement
32%
35.7
47.2
65,8
59,9
CustomDiffusion
Object Composition
24%
33,9
45.8
62.9
55.9
Autribute Editing
20%
31.0
40.0
54.3
51.0
Background Replacement
24%
35,4
48.5
75,2
69.0
Scene Placement
32%
35.1
48.2
73.9
67.6
ACCORD
Object Composition
24%
33.7
46.9
71.3
64.4
Attribute Editing
20%
31.3
41.2
63.0
59.8
H PERFORMANCE BY PROMPT CATEGORY ON DREAMBENCH
To better understand the source of performance improvement by concept decoupling, we categorize the validation prompts in DreamBench and report the performance of ACCORD and the baseline, CustomDiffusion, for each category, as shown in Tab. II Specifically, the prompts in DreamBench can be grouped into four categories: (i) Background Replacement, e.g., "c, with a city in the background," where Cp denotes the personalization target; (ii) Situational Placement, e.g., "cp on top of green grass with sunflowers around it"; (iii) Object Composition, e.g., "Cp floating in an ocean of milk"; and (iv) Attribute Editing, e.g., "a shiny Cy. We find that ACCORD delivers balanced performance improvements across different prompt types, with slightly higher gains in background replacement and attribute editing, which demand higher degree of concept decoupling.
I SENSITIVITY TO TIMESTEP SAMPLING AND CLASSIFIER- GUIDANCE SCALE
Table 12: Sensitivity of ACCORD to timestep sampling strategies.
Method
Timestep Sampling
CLIP-T
BLIP-T
CLIP-I
DINO-1
Uniform
34.2
45.4
62.7
56.9
CustomDiifuusio
LogitNNormalO.0,, 1.0)
34.0
45.4
63.1
57.5
LogitNormaal1.0, 0.6)
34.3
45.7
61.3
55.0
Uniform
34.1
46.6
71.4
65.6
ACCORD
LogitNormal(O.00 1.0)
34.1
46.4
71.6
66.4
LogitNorraal1.0, 0.6)
34.2
46.5
69.3
63.1
Table 13: Sensitivity of ACCORD Classifier-Free Guidance (CFG) scales.
Method
CFG Scale
CLIP-T
BLIP-T
CLIP-I
DINO-I
5.0
33.7
45.1
61.5
55.8
CustomDiffusion
7.5
34.2
45.4
62.7
56.9
10.0
34.0
45.2
61.2
54.3
5.0
33.8
46.3
70.7
65.5
ACCORD
7.5
34.1
46.6
71.4
65.6
10.0
34.1
46.3
70.2
64.6
We investigate the robustness of ACCORD to different timestep sampling strategies and guidance scales for subject personalization using CustomDiffusion, as shown in Tab. [12] and Tab. [ The results demonstrate that ACCORD consistently improves upon the baseline and performs robustly across noise settings and guidance scales. This robustness can be attributed to the fact that neither DDLoss PDLoss makes assumptions regarding the noise setting guidance scale. Specifi- cally, DDLoss constrains the growth of dependency between neighboring timesteps without limiting timesteps themself, while PDLoss is independent of the denoising process.
J REAL WORLD PERSONALIZATION
We further collect a set hamster photographs for personalization experiments, aiming to evaluate the practical effectiveness of our proposed method in real-world personalization scenarios. The
20

---

Published as a conference paper at ICLR 2026
Reference
Prompts
LoRA (SDXL)
Ours
hersstor astromour surit, fleating in sucte space. planets
kewwstrr repilinng booke, cony Libraig- wirr colors
hawsster plante costusee, standling ow tree sure cirest, baskgrssuns
Figure 11:
Real-world personalization visualization results. Compared with the baseline, our ap- proach successfully generates stars (Ist row), library scene (2nd row), and pirate hat (3rd row).
quantitative results and qualitative examples are presented in Tab. 14 and Fig. respectively. It can be observed that our method substantially improves the control capability over the baseline. As shown in Fig. 11 our approach successfully generates stars (1st row), a library scene (2nd row), and pirate hat (3rd row).
Table 14:
personalization quantitative results.
Method
CLIP-TY
BLIP-T1
CLIP-I
DINO-I
LoRA (SDXL)
38.6
52.1
68.9
59.1
LoRA (SDXL) w/ Ours
39.9
54.2
69.1
58.6
K MORE VISUALIZATION RESULTS AND FAILURE CASES
K.1 ADDITIONAL VISUALIZATION RESULTS FOR SUBJECT, STYLE, AND FACE PERSONALIZZATION
We provide more visualization results in Fig. and 14 For subject and style personalization, the "Baseline" is Dreambooth. For face personalization, the "Baseline" is IP-Attpter. The following observations can be made: (1) Our method demonstrates superior text alignment compared to the baseline. Specifically, in the first, second, and third columns of Fig. 7 method successfully generates a snowy scene, a wheat field, and a purple bowl, whereas the baseline model does not. In the first, third, fourth and fifth columns of Fig. 13 our approach successfully produces images of a pirate, a snowy landscape, a knight and a blue shield. Finally, in the third and fourth columns of Fig. 14. our method generates a cityscape background and cultural elements according to the prompts. (2) Our method better preserves personalization fidelity. In the fourth and fifth columns of Fig. 7 our method generates subjects that more closely resemble the reference images, whereas the baseline either produces an unrelated cat (4th column) or anomalies such as a black dog's back (5th column). It should noted that for columns 1, 2, and 5, our method not only replaces the background but also adjusts the perspective to make the generated image look more natural. In the second row of Fig. 13 the images generated by our method exhibit styles more closely aligned with the reference styles, namely the clay style. Finally, in all columns of Fig. [4] the faces generated by our method more closely resemble the reference faces. Specifically, our method better captures the subject's age in columns and 5; and the hair style in columns 2 and 3.
21

---

Published as a conference paper at ICLR 2026
K.2 VISUALIZATION THE EFFECTS OF PDLOSS AND DDLOSS.
We also visualize the individual effects of DDLoss and PDLoss on subject personalization based on CustomDiffusion in Fig. 12 It can be observed that PDLoss mainly improves personalization fidelity by aligning the relationship between the personalization target and other concepts with that of its superclass and other concepts. On the other hand, incorporating DDLoss enhances text control capabilities and can further boost personalization fidelity. Specifically, after introducing DDLoss, the bear in the first column of Fig I2 more closely resembles the reference image; meanwhile, the toys in the second and third columns are correctly placed on the sidewalk and in the jungle, respectively.
Reference
Bascline
0
+PDLoss
+PDLoss
+DDLoss
bear_ @ear_plushie* on the beach
poop poop_emoji* on top of the sidewalk in a crowded street
rc_car* the jungle
Prompt
Visualization of the Individual Effects of DDLoss and PDLoss on Subject Personalization Based on CustomDiffusion. PDLoss aligns the relationships between the personalization target and other concepts with the relationships between its superclass and those concepts, thereby mainly enhancing fidelity. DDLoss further improves text alignment and can also boost fidelity. In the first column, adding DDLoss makes the generated bear more resemble the reference image. In the second and third columns, DDLoss enables the models to correctly place the personalized toys sidewalk and in the jungle, respectively.
Figure 12:
K.3 FAILURE CASE ANALYSIS.
Finally, we present several failure cases of ACCORD in Fig. 15 The causes of these failures can be broadly categorized into two types: (1) Concepts that are strongly entangled with the personalization target not explicitly disentangled during training. If concept that is undesirably coupled with the personalization target not included in the training prompts, will not be addressed by DDLoss and PDLoss. Consequently, at inference time, the model must rely solely its generalization ability
22

---

Published as a conference paper at ICLR 2026
to disentangle this concept from the personalization target, which may lead to failure. (2) Inaccurate modeling of concept dependencies by the foundation T2I model. The effectiveness of DDLoss and PDLoss is fundamentally constrained by the capabilities of the underlying T2I foundation model. When the hase model struggles to simultaneously generate both the superclass of the personalization target and another concept, it likely fails accurately capture the dependencies between them. In such cases, the decoupling effect of DDLoss becomes limited. This observation also suggests that more powerful foundation T2I models can enable DDLoss to achieve better disentanglement.
Reference
Dreambooth
Ours
Painting style*, a pirate with treasure map
Clay style*, two pandas playing bamboo forest
3D model style", striped umbrella on the beach
Anime style*, knight holding blue shield
Baroque style*, a knight holding a blue shield
Prompt
Figure 13: A comparison of style personalization visual outcomes, with "style"" indicating the target style. Compared to the baseline, our method successfully generates a pirate, snowy landscape, knight, and blue shield in columns 1, 3, and 5. In column 2, approach produces clay-style images that more closely match the reference.
L COMPUTATION AND MEMORY OVERHEAD
Our proposed ACCORD can be seamlessly incorporated into many existing image personalization methods, enhancing personalization performance the expense of increased GPU memory usage and longer training times. Tab. [3]]]]]]]]00 summarizes the GPU memory consumption and training duration for each baseline method and its integration with ACCORD, under consistent training settings: all experiments use a batch size of 4 and are trained for 1000 steps on an NVIDIA H100 GPU.
While integrating ACCORD introduces additional GPU memory requirements and slightly longer training times, these increases are moderate and not reach an order-of-maan order-of-magnitude larger compared the respective baselines. Furthermore, we observe that reducing the batch size has negligible impact on the performance of ACCORD, enabling users to lower batch size in practical scenarios achieve acceptable memory usage and training time. Crucially, the extra computational cost imposed by ACCORD remains manageable, especially when contrasted with zero-shot approaches, which often involve considerably higher training overheads and less controllable personalization outcomes at inference time.
M DETAILED DATASET INFORMATION
We utilize the DreamBench Ruiz et al. (2023) dataset to compare the subject-driven personaliza- tion capabilities of different methods. DreamBench contains 30 subjects across 15 categories, of
23

---

Published as a conference paper at ICLR 2026
Reference
IP-Adapter
Ours
elegant dress, soft focus
monochrome outfit, minimalist background, high contrast
evening wear, cityscape background, night lights
traditional attire, cultural elements, authentic
traditional attire, cultural elements
Prompt
setting
Figure 14: A comparison of the visual outcomes of face personalization. Red circles highlight well-generated areas, while blue circles indicate poorly generated regions. The baseline IP-Adapter tends alter gender or make faces appear older columns 1, 2, and In contrast, our method produces faces more similar to the reference in columns 1, 2, 4, 5. Additionally, in columns 3 and of Fig. [4] our model generates cityscape backgrounds and incorporates cultural elements according to the prompts.
Reference
Ours
Reference
w/o Personalization
Ours
gbele, ted, rurl ghd. wedgs,
g0I0e and Wiltfe slaata raaganns pork 192t
cubbe shaped plrtt
eneaker on top pink
trfi mensier" of plak
(a)
Figure 15: Failure cases of our ACCORD method. (a) Failure to generate cube-shaped sunglasses: When specific attributes are absent from the training prompts, they are omitted from the calculation of DDLoss and PDLoss. In such instances, disentanglement relies solely on the model's 's general- ization ability, which can lead to incomplete or unsuccessful attribute decomposition. (b) Failure to generate the target sneaker pink fabric: The effectivvnnes of DDLoss and PDLoss is inherently limited by the capacity of the base model. If the underlying T2I foundation model cannot gener- ate both the personalization target's superclass and another concept simultaneously. it may fail to accurately model the dependence between these concepts. As a result, DDLoss's 's cross-timestee alignment mechanism may also fail to achieve proper disentanglement.
which 9 are animals, with each subject having 4-6 images. For style personalization, we employ StyleBenchJunyao et al. 2024), which focuses style transfer tasks and includes 73 distinct styles, each style comprising 5 or more reference images. Furthermore, to validate the effectiveness of our proposed losses for zero-shot image personalization, conduct face personalization experiments
24

---

Published as a conference paper at ICLR 2026
Method
GPU Memory (GB)
Training Time (s)
DreamBooth
26.5
320
DB w/ Ours
45.3
480
CustomDiffusion
18.7
346
CD w/ Ours
29.5
502
LoRA (SDXL)
27.7
483
LoRA (SDXL) w/ Ours
60.8
916
VisualEncoder
11.1
255
VE w/ Ours
16.8
490
Table 15: Computation and memory overhead for different methods and their integration with AC- CORD. All experiments are conducted with batch size 4 and 1000 training steps on H100.
on the FFHQ Karras al. (2021) dataset. FFHQ is a dataset of 70,000 high-quality face images, offering substantial diversity in age, ethnicity, background, etc. We employ Insightface Deng et al. (2019) to detect over 40,000 images containing only a single face, and exclusively use these images for training and testing.
MORE IMPLEMENTATION DETAILS
The baseline VisualEncoder Ye et al. (202)) is a simplified version of IP-Adapter that retains the CLIP Image Encoedr-basedd Visual Encoder, omitting the image-specific Cross Attention. This de- sign implies that only the MI.P at the end of the CLIP Image Encoder is trainable, and the per- sonalization relies entirely the visual embeddings Cp extracted by the visual encoder. We find that serves as a strong parameter-efficien baseline. We utilize the official implementation of Facechain-SuDe while implementing other baselines and our proposed method using open-source library Diffusers von Platen et al. (2022). All methods employ the DDIM sampler, a guidance scale of 7.5, and 50 inference steps during evaluation. We conduct experiments on NVIDIA A100 and H100 GPUs.
The different training paradigms of the various baselines necessitate distinct weighting for DDLoss and PDLoss. After tuning the loss weights using validation prompts, we find that, in general, DDLoss weight between 0.1 and 0.3 suffices, while a PDLoss weight between 0.001 and 0.003 is adequate. We train all methods for 1000 steps each subject style and display the results of the best-performing step. noteworthy that users can adjust the loss weights in practice to achieve optimal results due to the automatic computation of CLIP-T, BLIP-T, CLIP-I, DINO-I, Gram-D, and Pace-Sim.
O VLM PROMPTS FOR IMAGE CAPTIONING
We employ Intern-VIL Chen al. (2024b) the image captioner. The prompt used is detailed below:
You are an excellent prompt engineer. Given an image and a tag corresponding to an important object in the image, please describe the given image in short for the image generation process of the SD model.
Note that the prompt you give shculd consist of a series of phrases, not a complete sentence, and must contain the tag corresponding to the important object. Please do not describe the important object in detail. Please do prompt you give needs to use all lowercase letters. Here is an example: dog, running, sea, sunset.
anything other than the prompt. The
Now, the important objects
25

---

Published as a conference paper at ICLR 2026
P LIMITATIONS
Autoregressive generative models without a diffusion process, such as LlamaGen Sun et al. (0024). not compatible with the proposed losses. Furthermore, the effectiveness of our decoupling losses is constrained by the capabilittes of the foundation T2I model; the base model cannot accurately represent the relationship between superclass and a given concept, ACCORD's 's ability regularize this dependency is limited. Finally, decoupling is most effective for concepts that are explicitly in- cluded in training prompts, while concepts that are implicitly coupled may not be fully disentangled.
LLM USAGE
In this paper, large language models (LLMs) only used to assist polish the writing, and they are not involved in the methodology and experimental design of this paper.
26


Reviews:
[{"review_id": "XIQ6nxNtSk", "reviewer": "Reviewer_vTgX", "summary": "This paper targets {concept coupling}—a critical limitation in text-to-image (T2I) diffusion personalization where target concepts  become unintendedly entangled with irrelevant general concepts. Unlike prior indirect solutions , ACCORD formally models coupling as a statistical dependence problem: coupling arises when the conditional dependence between a target concept ($c_p$) and a general concept ($c_g$) in generated images deviates from the prior dependence between $c_p$’s superclass ($c_s$) and $c_g$.", "questions": "How should a user practically apply ACCORD if the reference images for $c_p$ contain multiple coupled concepts (e.g., a dog $c_p$ always on a \"blue rug\" $c_{g1}$ and next to a \"red ball\" $c_{g2}$)? Must the VLM identify both $c_{g1}$ and $c_{g2}$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Theoretically Grounded Innovation: Framing coupling as statistical dependence  provides a unified language for understanding coupling, which was previously described via ad-hoc examples. This foundation enables reproducible, extensible loss design—unlike heuristic methods that require case-by-case tuning.  \n Loss Design: DDLoss and PDLoss address complementary aspects of coupling: DDLoss stabilizes dependence during denoising (preventing sudden coupling), while PDLoss aligns with superclass priors (preventing persistent coupling). Ablation results (Tab. 5) show their combination yields 15–20% higher text control than either loss alone, demonstrating thoughtful, non-redundant design.  \n Comprehensive Experimental Validation: The paper tests ACCORD across three distinct personalization scenarios (subject/style/face) and uses both automatic and human metrics. This breadth ensures generalizability—critical for a method claiming to solve a \"general\" coupling problem. Human preference tests (72% of annotators favor ACCORD) also address a key limitation of automatic metrics (e.g., CLIP-T) that may not capture subjective quality.", "weaknesses": "Insufficient Ablation of Loss Weight Interactions: The paper ablates individual losses (DDLoss only, PDLoss only) but not how their weights ($\\lambda_D$, $\\lambda_P$) interact. For example, does increasing $\\lambda_D$ improve control but harm fidelity when $\\lambda_P$ is low?  \nWithout this analysis, users cannot optimize weights for specific use cases (e.g., style personalization may require higher $\\lambda_P$ than subject personalization).  \n\nThe Prior Decouple Loss (PDLoss) is entirely dependent on Assumption 1, which posits that $p(c_j|c_k) \\approx \\frac{e^{\\tau \\cos(f_j, f_k)}}{Z_k}$. This is a very strong theoretical leap. While CLIP's objective aligns image-text pairs, extending this to a proxy for the conditional probability between any two text concepts ($c_p$ and $c_g$) is not rigorously justified.\n\nThe method relies on a VLM to generate captions and thus identify the co-occurring concepts $c_g$ to be decoupled. The paper states this is superior to templates. This introduces a critical, unevaluated dependency. The method's performance is now tied to the VLM's ability to correctly identify all relevant coupled concepts. If the VLM fails to mention the \"girl\" in the backpack's caption, for example, the framework may fail. No ablation study is provided to test the sensitivity to caption quality or source.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "axTjPOcGEn", "reviewer": "Reviewer_SGY8", "summary": "They propose ACCORD, a framework incorporating two regularization losses to improve personalization performance. The Denoising decouple loss (DDLoss), designed to mitigate concept coupling, minimizes dependency discrepancies to prevent conditional dependence between the personalized concept $c_p$ and the general text concept $c_g$ across successive time steps. The Prior decouple loss (PDLoss) addresses prior dependency by leveraging CLIP projections to align the learned concept’s relational structure with that of its original class, thereby preserving semantic consistency. With theoretical justification and experimental validation, the framework demonstrates its effectiveness across various fine-tuning methods for personalization, e.g., DreamBooth or Custom Diffusion.", "questions": "**Q1**. Is the proposed method capable of handling more challenging decoupling cases? For example, in Figure 2, could the model successfully decouple attributes such as the blue hat from the brown teddy bear if an appropriate subclass were defined?\n\n**Q2**. How much slower is the training compared to the standard DreamBooth? The proposed method seems to involve multiple regularization terms, which may considerably increase training time.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**S1**. Concept decoupling in personalization is an important issue, and unlike existing indirect approaches that address it through explicit masking or regularization, this work resolves the problem with a simple yet effective decoupled loss design.\n\n**S2**.  The proposed method demonstrated its effectiveness through extensive experiments across various fine-tuning frameworks and comprehensive ablation studies.", "weaknesses": "**W1**. Although the paper proposes a method for concept decoupling, the evaluation appears to focus primarily on single-subject personalization benchmarks such as DreamBench. It would be necessary to include evaluations on multi-subject personalization tasks, such as Break-a-Scene [1], to better assess how effectively the method achieves concept decoupling.\n\n[1] Avrahami et al., Break-a-scene: Extracting multiple concepts from a single image, \tSIGGRAPH Asia 2023\n\n**W2**. While the motivation for the proposed metho, avoiding explicit masks or priors (e.g., in attention maps or diffusion losses), is understandable, a performance comparison with such approaches is necessary. The paper compares various training frameworks, but lacks baseline comparisons specifically focused on different learning strategies.\n\n**W3**. Recent powerful text-to-image models (e.g., FLUX-dev) show effective concept decoupling even with purely descriptive prompts. However, the paper lacks sufficient validation to demonstrate whether the proposed method can further improve performance in such flow or transformer based models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uqVcaTTFBO", "reviewer": "Reviewer_CA21", "summary": "The author introduce the concept coupling issue in personalization. They provide the theoretically analysis of this issue and propose several loss for resolving it. Experiments shows the performance of their method is good in some cases.", "questions": "- Would final optimization object be like \\mathcal{L}  = \\mathcal{L}_{reconstruction} + \\alpha\\mathcal{L}_{DDL} + \\beta\\mathcal{L}_{PDL}? If so, how do authors decide the value of \\alpha and \\beta, should have some ablation experiments here.\n- Considering the first case of figure2, why DDL/PDL works here? Shoes and feet are semantically dependent concepts, and this dependency is not introduced by the source image. As I understand it, such examples are not something that DDL/PDL can address. Could the author elaborate on the theoretical basis behind this; Do DDL and PDL here only serve as a regularization term instead of turning the direction of optimization process goes to a better trade-off?\n- What percentage of DreamBench examples resemble \"people carrying red envelopes?\" Is this the majority of the benchmark? If this data is extracted, will the DDL & PDL performance improve more than on the general benchmark?\n- Theory in the main text is well explained but too abstract, better to move figure 4&5 to main paper for clarify.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper is theoretically interesting, author introduce several formula for explaining the phenomenon of dependence discrepancies.\n- The author introduce Denoising Decouple loss(DDL) and Prior Decouple Loss(PDL) for resolving the issue.", "weaknesses": "- The scope of this work is limited, regard the real-world application, it's not hard for users to take a set of pure images including only desired objects at most cases. For case in figure 1, it's more likely a data issue. Decoupling is important in personalized generation when being applied to concepts that cannot be easily decoupled by the user, such as atmosphere, lighting, posture, and materials, which is called abstract concept personalization. Could author explore more qualitative experiments on this domain?\n- Concern about the robustness of the method. Is it sensitive to prompts? When prompts fail to accurately describe the background beyond the personalized object, will the DDL negatively impact model performance? Furthermore, would more refined prompt improve model performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "omn7elWQsy", "reviewer": "Reviewer_kN9B", "summary": "The paper addresses concept coupling in T2I personalization and proposes ACCORD with two plug-and-play losses to control denoising/prior dependence. Results show consistent alignment/fidelity gains across several backbones with modest overhead.", "questions": "1. How sensitive are your losses to scheduler/noise settings and guidance scales?\n2. Can ACCORD be demonstrated in multi-concept personalization and report cross-concept interference?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Concept coupling is formalized as conditional dependence and decomposed into denoising vs. prior components, each tied to a dedicated loss.\n2. Provides a closed/near-closed form via the “model-as-implicit-classifier” view—no auxiliary discriminator or teacher model required.", "weaknesses": "1. Some works need to be discussed and compared in the Related Work section，Update-space constraints: Custom Diffusion [1], PaRa [2], Attention-level preservation: Perfusion [3], Attend-and-Excite [4]. Embedding-only personalization: Textual Inversion [5].\n2. Human-eval reporting lacks agreement stats\n\n\n\n[1] Nupur Kumari, Bing Li, David Forsyth, Jia-Bin Huang, Vishal M. Patel, Oncel Tuzel, Ali Farhadi, Anima Anandkumar. Multi-Concept Customization of Text-to-Image Diffusion. CVPR, 2023.\n\n[2] Shangyu Chen, Zizheng Pan, Jianfei Cai, Dinh Phung. PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction. ICLR, 2025.\n\n[3] Omer Tov, Yuval Alaluf, Yotam Nitzan, Daniel Cohen-Or, Tali Dekel. Key-Locked Rank-One Editing for Text-to-Image Personalization (Perfusion). SIGGRAPH, 2023.\n\n[4] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, Daniel Cohen-Or. Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models. 2023.\n\n[5] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)