Conditional Image Synthesis with Diffusion Models: A Survey
Authors: Zheyuan Zhan
Defang Chen
Jian-Ping Mei
Zhenghe Zhao
Jiawei Chen
Chun Chen
Siwei Lyu
Can Wang
Venue: Accepted by TMLR ()

Abstract: Conditional image synthesis based on user-specified requirements is a key component in creating complex visual content. In recent years, diffusion-based generative modeling has become a highly effective way for conditional image synthesis, leading to exponential growth in the literature. However, the complexity of diffusion-based modeling, the wide range of image synthesis tasks, and the diversity of conditioning mechanisms present significant challenges for researchers to keep up with rapid developments and to understand the core concepts on this topic. In this survey, we categorize existing works based on how conditions are integrated into the two fundamental components of diffusion-based modeling, i.e., the denoising network and the sampling process. We specifically highlight the underlying principles, advantages, and potential challenges of various conditioning approaches during the training, re-purposing, and specialization stages to construct a desired denoising network. We also summarize six mainstream conditioning mechanisms in the sampling process. All discussions are centered around popular applications. Finally, we pinpoint several critical yet still unsolved problems and suggest some possible solutions for future research.

Published in Transactions on Machine Learning Research (05/2025)
Conditional Image Synthesis with Diffusion Models: A Survey
Zheyuan Zhan
Chenaheeuonn:jjueddu
State Key Loboratory of Blockchaän and Data Security, Zhefiong University College of Computer Science, Zhejiang University
Defang Chen
defangch&@buffaao.com com
University at Buffalo, State University of New York
Jian-Ping Mei
jpmeittajut.com edu.con cn
College of Computer Science, Zhejiang University of Technology
Zhenghe Zhao
shaozhenghe Azju. edu.cn
College of Computer Science, Zhejiang University
Jiawei Chen
sleepyhunt@:rju.edu.cr
State Key Laboratory of Blockchain and Data Security, Zhejiang Universily Hangzhou High- Tech Zone (Binjiang) Institute of Blockchain and Data Security
Chun Chen
chenc chenc@zju.eddu edu.cc
College of Computer Science, Zhejiang University
Siwei Lyu
siweityu@buffoo..com com
University at Buffalo, State University of New York
Can Wang
wcon@zju..edu edu..u.educccmm cr
State Key Laboratory of Blockchain and Data Security, Zhejiong University
Hangzhou High- Tech Zone (Binjiang) Institute of Blockchain and Dato Security
Reviewed on OpenReview: https: // openreview. net/ forum? id= ewuNHuh66K
Abstract
Conditional image synthesis based on user-specified requirements is a key component in cre- ating complex visual content. In recent years, diffusion- based generative modeling has be- come a highly effective way for conditional image syuthesis, leading to exponential growth in the literature. However, the complexity of diffusion-based modeling, the wide range of image synthesis tasks, and the diversity of conditioning mechanisms present significant challenges for researchers to keep up with rapid developments and to understand the core concepts on this topic. In this survey, we categorize existing works based on how conditions are inte- grated into the two fundamental components of diffusion-based modeling, f.c., the denoising network and the sampling process. We specifically highlight the underlying principles, ad- vantages, and potential challenges of various conditioning approaches during the training, re-purposing, and specialization stages to construct desired denoising network. We also summarize six mainstream conditioning mechanisms in the sampling process. All discussions are centered around popular applications. Finally, we pinpoint several critical yet still un- solved problems and suggest some possible solutions for future research. Our reviewed works are itemized at /ttp::////ttt...../ www.sccmmi.iiiiieee
conditional-- wwwttoionnnifffuue -Diffusion -Model
'Corresponding Author.

---

Published in Transactions on Machine Learning Research (05/2025)
TeText -image
B:lmage restoration
C:Visual signal to image
Dilmage editing
"A zombic style of Picarso"
"A bird spreading wings"
E:Customization
Filmage composition
G:Layout control
holding bas, another person is talking on cell phone
"In the Acropolis"
Figure 1: Seven representative conditional image synthesis tasks with their corresponding inputs and outputs.
Figures are cited from the following papers: (A) Stable Diffusion (Rombach et al.
2022); (B) SR3 (Saharia
et al. 2022ch; ; (C) ControlNet (Zhang et al. 2023b); (D) Imagic (Kawar et al. 2023
(E) DreamBooth (Ruiz
et al. 2023 (F) PbE Yang el al. 2023a (G) Interact (Hoe et al.
2023
1
Introduction
Image synthesis is an essential task in generative artificial intelligence. It is particularly useful when user- provided conditional inputs guide the generation process, enabling precise control to meet diverse needs. Early works have achieved significant breakthroughs in various conditional image synthesis tasks, such as
text-to-image gencration (Reed al. 2016 Zhang al.] 2017 Ding et 2021
Ramesh et al.) 2021p. image
restoration (Ledig et. al. 2017 Wang al. 2021 Maalde el al. 2019 Lee el. al.
2022), and image editing
[Brock et al.) 2017 Ling et al. 2021 Abdal et al. 2020). However, early deep
learning- based generative
models, such as generative adversarial networks (GANs) [Goodfellow et al. 2014 Mirza & Osindero 2014
variational auto-encoders (VAEs) (Kingma & WWeling 2014 Sohn et al. 2015), and auto-regressive models
(ARMs) (Van Den Oord et al. 2016 Van den Oord et al. 2016) face in inherent limitations. GANs are
susceptible to mode collapse and unstable training (Goodfellow et al. 2014; VAËs often produce blurry images (Kigmma & Welling 2014); and ARMs suffer from sequential error accumulation and significant time
delays Van Den Oord al. 2016
In recent years, diffusion models (DMs) have emerged as the stato-of-the-art framework for image generation due to their strong generalive capabilities and versatility (Sohl-Dickstein et al.] 2015 Ho et al. 2020 Song et al. 2021b; Karras et al., 2022; Chen et al. 2024a). In DMs, images are generally synthesized from Gaussian noise through itcrative denoising steps guided by the predictions of denoising network. In practice, DMs achieve remarkable performance in diverse generative lasks, characterized by stable training process, diverse outputs, and exceptional sample quality. Furthermore, compared one-step generative models, the distinctive multi-step sampling process offers DMs a unique advantage in facilitating conditional integration. These benefits have made DMs preferred tool for conditional image synthesis, leading to rapid growth in Diffusion--usse based Conditional Image Synlhesis (DCTS) research over the past few years (Rombach et al. 2022
Saharia et al. 2022b Lu et al. 2023 Choi et al.) 2021 Sa Saharia et al. 2022c Kawar et al. 2023 Hertz et al,
2023 Zhang et al. 2023e: Gal et al. 2023a Zhang et al. 2023b Wang et al. 2024b Fig. iillsstrats seven
popular DCIS tasks with different modalities conditional inputs.
With the rapid growth of rescarch in this area, coupled with the wide variability model architectures, training paradigms, and sampling strategies, as well as the broad scope of potential conditional synthesis tasks, it has become increasingly challenging for newcomers develop comprehensive understanding of the landscape of DCIS. This underscores the need for systematic survey that offers coherent and structured synthesis of current advances in this rapidly evolving field.
Several surveys have focused on the application diffusion models in specific conditional image synthesis
tasks, such as image restoration (Li et al.) 2023g Daras et al. 2024). text-to-image (Zhang al.) 2023a and image editing (Huang et al. (024b), or have categorized diffusion-basee works in the field computer vision according to their target conditional synthesis tasks (Croitoru al. 2023 Po et. al. 2023
While
2

---

Published in Transactions on Machine Learning Research (05/2025)
these task-oriented surveys provide valuable insights into approaches tailored for different tasks, they do not explore the commonalities in model frameworks across different conditional synthesis tasks, particularly in terms of model architectures and conditioning mechanisms.
Two recent surveys (Shuai et al. 2024 Cao et al., 2024) provide an overview of DM-based works across a broad range of conditional image synthesis tasks. However, their soope remains limited, as they primarily focus on DCIS methods built on text-to-image (T22) backbones, neglecting earlier works that integrate conditional inputs into unconditional denoising networks train task-specific conditional denoising networks from scratch. These earlier efforts are foundational to the current advancements in DCIS and are still widely applied in low-level tasks such as image restoration.
In contrast, this survey aims to provide comprehensive and structured framework that covers a wide range of contemporary DCIS works. We present a taxonomy based on the mainstream techniques for condition integration, offering a clear and systematic breakdown of the key components and design choices involved in constructing a DCIS framework. Specifically, we review and calegorize existing DCIS methods by examining how conditions are integrated into the two fundamental components of diffusion modeling: the denoising network and the sampling process. For the denoising network, we delineate the process of establishing a conditional denoising network into three stages. For the sampling process, we categorize six mainstream in-sampling conditioning mechanisms, detailing how control signals are integrated into various components of the sampling process. Our objective to provide readers with a high-level and accessible overview of existing DCIS works across diverse tasks, equipping them with the knowledge to design conditional synthesis frameworks for their own applications, including novel tasks that have yet to be explored. In practice, as image synthesis is fundamental task in computer vision, many more complex visual computing and synthesis tasks are built upon its extensions. Therefore, the methods for image synthesis introduced in this paper be readily extended to more complex visual tasks, such as video synthesis (Wang et al.) 2023c Esser et al.
2023), 3D scene generation (Haque et al. 2023 Höllein et al., 2023). motion generation (Krunratanakul
ct al. 2023 Kulkarni et al. 2024).
The remainder of this survey is organized as follows: We first introduce the background of diffusion models and conditional image synthesis in Sec. [ Next, we summarize methods for condition integration within the denoising network in Sec. 3 and for the sampling process in Sec. 4 Finally, outlines potential future directions in Sec.
2 Backgrounds
Diffusion-based generative modeling adopts a forward diffusion process of gradually adding noise into clean data and learns a denoising network to predict the added noise. In the sampling process, the data is synthesized by reversing the forward process from Gaussian noise basod on the prediction of a denoising network. Currently, branch of conditional synthesis research (Esser et al. 2024 Tewel al. 2024 Wang
et al. 2024a Rout et al. 2024a) leverages the flow matching framework (Lipman et al. 2023 Liu et al. 2023c Heitz et al.] 2023) to model the mapping from a prior distribution to the real data distribution. In practice, most of these works employ a special case of flow matching, where the prior distribution used in flow matching corresponds to a Gaussian distribution, resulting in the same training and sampling algorithm as the original diffusion framework in practice (Gao et al.) 2024h. Therefore, we primarily focus on the gencrative process based on the original diffusion framework, unless otherwise specified. We first introduce the core concepts of discrete-time and contimuous-tin time diffusion modeling in Sec. 2.1 Then, we discuss the model architecture in Sec. 2.2 and highlight representative DCIS tasks in Sec. 2.3 Finally, in Sec. 2.4 we introduced the classic condition strengthening approaches widely employed across various DCIS works.
2.1
The Formulation of Diffusion Modeling
2.1.1
Discrete-Tine Time Formulation
The discrete-time diffusion model was initially proposed (Sohl-Dickstein et al., 2015). It constructs a forward Markov chain to transform clean data into noise by progressively adding small amounts of Gaussian noise that parameterized denoising network can be learned to predict the added noise in each forward
3

---

Published in Transactions on Machine Learning Research (05/2025)
step. Once the denoising network is trained, images can be generated from Gaussian noise by reversing the diffusion process. This idea gained popularity through an important follow-up work known as denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020). This work led to a substantial improvement in the quality of synthesized images in increased resolutions, from 32 x 32 Soohl-Dickstein et al. 2015 to 256 x 256, sparking a rapidly growth of interest diffusion models. Nexl, we adopt the notation from DDPM (Ho 2020), which is widely employed in the literature describe discrete-time diffusion models (Song
et al.
et al. 2021a Rombach al., 2022 Kawar et al. 2023
The forward Markov chain is parameterized based on pre-defined schedule 31..... Sr, where 3e is the noise variance in each step and the total number of steps T is usually large, e.g., 1,000. Given the clean data sam- pled from the training dataset Xg ~ Pdiata (x); the transition kernel is (x, xt-1) - N (xci /xivv= 3,x,- B,1), or, q (xt xo) -V (xt: Voyxo; (1 - -x4 I), where x1:-..;XT XT are latent variables, or = 1 Be, ā. = II: Og: and âT + 0. By progressively adding Gaussian noise to the clean data, this Markov chain transforms the data distribution to an approximate normal distribution, i.e,, J (xxx<x())pxall Xo) (xo)dxo ~ NO( I).
In the training phase, DDPM Ho et al. 2020 learns a denoising network with parameter 0 by minimizing
the KL divergence between the transition kernel po (x(-1x/))) and the posterior distribution (x:-1 X,). In practice, DDPM (Ho et al.] (020) trained on the following re-parameterized loss function to improve the training stability and sample quality:
xx,. |||E |||--ca co (Vārxo + vT vT- 0,e,,)
where ex(x,,t) is prise-prediction network to estimate the added noise e Vava in each step. For the conditional generation that performs denoising steps conditioned on control signals c, the conditional denoising network Eg (x,,t, i, c) can be trained on a loss function similar to Eq. 1
VT-ag
Ex.exxo-pxxole).c [||e Eo (Vaixo + vI aa,,,,,
(2)
In the sampling process, DDPM gradually generates clean data from Gaussian noise by computing the reverse transition kernel pe with the learned network Eq: i.e.,
1
ag-1 BrEr;
X,-1
X
at ED
1 Ct.
where 6, ~ ~) et al. 2021a) proposed family of sampling processes sharing the same marginal distribution p(x,) with the above sampling process, which are written as
I) the standard Gaussian noise independent of x,. The following work DDIM (Song
x, = V + V1 11
ōt- - of 60 + azEt;
(4)
where fo (x:) xi-ve-aice Tice denotes the predicted Xo time step t. For simplicity, we will refer Vā:
fe (xx) as the intermediate denoising output Xojt hereafter. Each choice of o, represents specific sam- pling process in DDIM (Song et al. 2021a)). It is identical to the DDPM generative process in Eq. when o, ((1 - ât-1) (1 az) /1 - a and becomes a deterministic process when o, = 0.
2.1.2
Continuous- Time Formulation
Song et al. (2021b) proposed to formulate a diffusion process (x: Pr with the continuous time variable t E [ as the solution of an Itô stochastic differential equation (SDE) dx = f(x, t)dt + g(t)dw, where W. denotes the slandard Wiener process, and f(x, () and g(() are drift and diffusion coefficients, respectively (Oksendal 2013 Chen et al., 2024a). This diffusion process smoothly transforms a data distri- bution into an approximate noise distribution Pr, and the forward process of DDPM (Ho et al.) 2020) can be regarded as a specific discretization of it. There exists a probability flow ordinary differential equation (PF-ODE) dx [f(x,1) !) =2(()) Vx lx)) dt, sharing the same marginal distribution with the reverse
4

---

Published in Transactions on Machine Learning Research (05/2025)
SDE dx
f t
(()) V px dt g()dd (Song et al. 2021b Karras et al. 2022
Zhang & Chen
2023
Chen et al.
2024a
We can learn a time-dependent score-based denoising network [x, /) to esti-
mate the score function Vx, log p (xx) with a sum of denoising score matching (Vincent 2011 Lyu 2009
objectives weighted by A(t):
Ex A(t) [1ls| (x,, t) Vx logp (x4 xo)ll
(5)
When the score- based denoising network so (xx,t) t) is trained, we can employ general-purpose numerical methods such as Euler-Maruyama and Runge-Kutta methods solve the reverse SDE or PF-ODE and recover clean data Xo from X7.
In practice, learning score- denoising network se soise-prediction network Eæ essentially equivalent(Tt can be proven that co approximates scaled score function -vI- V. logp(x))). The DDPM sampling process described in Eq. 3 can be regarded as a first-order numerical solution for the reverse SDE. Therefore, in the following sections, unless otherwise specified, we will use the notation Eg to represent the denoising network.
2.2 Architecture of the Denoising Network
architectures as the backbone of denoising networks (o et al., 2020 Song et al. 2021a Song & Ermon 2019 2020 As illustrated in Fig 2 the denoising network employed in DDPM (Ho et al. 2020) follows the U-shaped structure downsampling and upsampling blocks in the basic U-Net. At cach resolution level, features from the downsampling blocks are directly passed the corresponding upsampling blocks through skip connections, which helps relain high-resolution local information and prevent the loss of details during the upsampling process. To help the denoising network to better capture visual features and the pixel correlations, the denoising network also replaces the simple convolution layers in the original U-Net with residual convolution layers ZZgoruyko 2016 and self-attention layers.
Pioneering works adopt U-Net [Ronneberger et al.
2015
The U-Net architecture particularly well-suited for diffusion models due to its ability to per- form superior feature extraction, contextual under- standing, precisc segmentation, and dimensionality preservation. These altributes enable it to accu- rately model complex data distributions and gener- ate high-quality images. Building on this founda- tion, many followed-up works have developed more advanced U-Net-based denoising networks by incor- porating multi-head attention (Song al. 2021b
Denolsing U-Net
H
Figure 2: An illustration the DDPM denoising net- work (Ho al. 2020), which predicts the noise basod on the given latent variable x, and time step The timestep f is firstly converted to high-dimensional rep- resentations via time encoder(e.g, (iinuurre.g.siiuusoddll sinusoidal dings and MLPs) and subsequently added to interme- diate feature maps.
Dhariwal & Nichol 2021
Nichol & Dhariwal 2021
normal liaaioizaiion (Ho et al.
2020
Dhariwal & Nichol
2021
Nichol & Dhariwal
2021
,, and cross-attention
embed-
layers [Robaach et al. 2022 Saharia et al. 2022b).
Transformers, known for their scalability, robust-
ness, and efficiency, have also emerged as popular choice for the model architecture in a variety of computer vision tasks. Researchers have attempted to employ transformer architectures the backbone of denoising networks in various conditional synthesis tasks (Yang
et al. 2022b Tang et al. 2022 Gu et al. 2022 Li et al. 2023d). However, these initial efforts have yet to rival the dominance of U-Net architectures in diffusion models. Notably, the recent groundbreaking Diffusion Transformers (DiTs) (Peebles & Xie 2023), built on the Vision Transformer (ViT) (Dosovitskiy el. al. 2020 architecture, first convert spatial input into a sequence tokens and then process them through a series of transformer blocks. The timestep and class label in DiTs are integrated via adaptive layer normalization. In practice, DiTs achieve seate-of-thheart sample quality, surpassing all previous diffusion models comparable computational costs.
5

---

Published in Transactions on Machine Learning Research (05/2025)
Despite the impressive generative performance of transformer-basee model architectures, most DCIS works still adopt U-Net as the model structure. Therefore, in the following sections, unless explicitly stated otherwise, we assume that the denoising network follows a U-Net structure.
2.3 Conditional Image Synthesis Tasks
A conditional image synthesis task T generates target image x by sampling from a conditional distribution:
x x~ CE DT.
(6)
where Dy is the domain of conditional inputs c, and pr is the conditional distribution defined by the task T. Based on the form of conditional inputs and the correlation between the conditional inpuls and the target image formulated conditional distribution pr (xlc), we classify representative conditional image synthesis tasks into seven categories as shown in Fig. 1 (a) Tezt-to-image synthesizes images in accordance with text prompts, (b) Image restoration recovers clean images from their degraded counterparts, (c) visual signal to image converts given visual signals such as sketch, depth and human pose into corresponding images, (d) Image editing edits the given source images with provided semantic, structure or style information, (e) Customization creates different editing renditions for personal objects specified by given images, (f) Image composilion composes the objects and the background specified in different images into single image, and (g) Layout control controls the layout grounding of synthesized images with provided spatial information of foreground objects and background. Further qualitative comparisons between classic DCIS works for text-to-iman tonttteeimaee, image restoration, visual signal to image, image editing and customization are provided in Fig. 11 12 13
15 in the Appendix. For image composition and layout control, due to the varying formats of conditional inputs across different works, a direct comparison is not feasible. Therefore, we present only representative outputs from popular works in Fig. [6 17 Besides, we have sorted out the associations between the conditional synthesis tasks and the representative conditioning mechanisms in Tab. !
2.4 Condition Strengthening in the Sampling Process
Currently, in order to strengthen the influence of the given conditional inputs с in the synthesized image, numerous DCIS works attempt to sample from the conditional strengthened distribution p(x(c))(c(x)) rather than the original conditional distribution p(x[c). In this formula, the parameter controls the strength of conditional imputs which leads to a trade-off between sample quality and diversity. In practice, setting a large scaling factor w can significantly enhance the sample quality and the consistency with the conditional
inputs с at the cost of sample diversity (Dhariwal & Nichol 2021 Ho & Salimans 2022
Classifier Guidance (Dhariwal & Nichol 2021 trains an auxiliary classifier Pe (c|x,) to approximate the likelihood term (c Xt) in label-conditioned image synthesis. However, training an accurate clas- sifier in most of the conditional synthesis tasks is challenging. Classifier-free guidance (Ho & Salimans, 2022) paves a training-free pathway to approximate (c x+) о pxx c) (x.) with the access to con- ditional noise prediction Eg (Xt, c) = -V1 āj Vx, log (xr(c) and the unconditional noise prediction 6g (xx) =- -ā, logp(x)) The proxy noise prediction ég can be expressed as:
Eo (xx, c) = (1 w)en (xx, wep (x+).
Due to its convenience and effectiveness, classifier-free guidance has become the mainstream condition strengthening approach in various DCIS works. To alleviate the potential negative impact of large guid- ance scales on sample diversity, subsequent works (Sadat et al.] 2024 Kynkäänniemi et al. 2024 propose dynamic classifier-free guidance, in which the guidance scaling factor is reduced during the denoising process with high noise levels.
In practice, classifier guidance and classifier-free guidance can also be employod as conditioning mechanisms to inject conditional inputs into the diffusion- based image synthesis framework. Therefore, we incorporate them into our DCTS framework and provide more detailed discussion in Sec 4.5 and Sec 4.3
6

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Stack of conditioning mechanisms applied denoising network and sampling process, for main- stream conditional synthesis tasks Conditioning encoder indicates the module to convert conditional in- puts into task-related feature embedding, where indicates that the encoder is determined by the specific restoration task.
denote the four condition injection methods employed in re-purposing stage described in Sec. 3.2.2 Due to page width limitations, we have placed the DCIS works performing condition integration via the presented stacks of conditioning mechanisms in the row identified by corresponding scrial numbers in Tab. in Appendix.
Stack of conditioning mechanisms for denoising network
Training (backbone)
Conditional encoder
Condition Injection
Backbone fine-tuning
Specialization Serial Number
Text-to-image
/
CLIP, BERT, LLMs
X
DNI
Non.
DN2
Image restoration
X
DN3
(T2I DM)
LLMs-bused
X
DN4
Image editing
(T21 DM)
Nou.
DN5
(T22 DM)
Non./BLIP
/
DN6
(T2I DM)
ViT (CLIP)-basod
Optional
DN7
Customization
(121 DM)
Non.
DN8
(T21 DM)
Convolation-based
x
X
DN9
Visual to image
(T2I DM)
ViT-based
X
DN10
(121 DM)
Convolution-based
DN11
Image composition
(T2I DM)
VIT (CLIP)-based
0,
0
X
DN12
Layout control
(121 DM)
VII (CLIP)-based
DN13
Stack of conditioning mechanisms for sampling process
Task
Backbone model
Conditioning mechanism
Serial Number
Text-to-image
Uncond DM
Guidance
SP1
Conditional restoration DM
Revising Diffusion Process
SP2
Uncond DM
Revising Diffusion Process
SP3
Image restoration
Uncond DM
Guidance
SP4
Uncond DM
Conditional Correction
SP5
Uncond DM T21 DM
Inversion
SP6
T2I DM
Inversion, Conditional Correction
SP7
Image editing
T2I DM
Inversion, Attention Manipulation
SP8
T21 DM
Inversion, Attention Manipulation, Guidance
SP9
Visual to image
'21 DM
Guidance
SP10
Image composition
Uncond DM
Noise Blending
SP11
T2I DM
Attention Manipulation
SP12
Layout control
T2I DM
Attention Manipulation, Guidance
SP13
Unspecified
Noise Composition
SP14
General purpose
Unspecified
Classifier-free Guidance
SP15
Unspecified
Universal Guidance Framework
SP16
3 Condition integration in denoising networks
The denoising network is the crucial component in the diffusion model synthesis framework, which estimates the noise added in each forward step to reverse the Gaussian distribution back into the data

---

Published in Transactions on Machine Learning Research (05/2025)
Text/image pairs
Visual signal/image pairs
"Killen"
"Suit Case "Steel Scren"
Visual signal image Denoising Network
Un-trained Backbone
T21 Denoising Network
Training
Re-purposing
Specialization
Specialization
zombbie style Picauso"
I-In Acrnpolls"
Text-to-lmage
Customization
Sampling
Visual signal image
Customization
Training
Conditional
Condition Integration in Denoising Network
Condition Integration in Sampling Process
Dataset
Inputs
Figure 3: An example of the workflow to build denoising network via training, re-purposing and specialization stages for target conditional synthesis tasks. In this framework, text-to-image(T2)) denoising network is firstly obtained via supervised learning on text/image pairs in training stage. Subsequently, this T2I denoising werw is fine-tuned on visual signal/image pairs for visual signal to image task in re-purposing stage. Next, both T21 and visual signal to image denoising networks can be further fine-tuned on given images in specialization stage to perform customization on the uscr-specified personal object. Figures are cited from (Rombach et al. 2022 Zhang al. 2023b Ruiz al. 2023 Li al. 2023a
Table Comparison of the characteristies of the three stages to perform condition integration in denoising network. The "Training Cost." column reflects the compulational cost involved in establishing denoising network for the target task, while the "Inference Cost" column represents the computational cost required to customize the denoising network for uscr-specified conditional inputs. We further present the guarantees of synthesis quality and the commonly applicable task scope( with capital letters indicating the tasks shown in Fig in this table.
Comparison of conditioning mechanisms for denoising network
Stage
Training Cost
Inference Cost
Guarantees
Applied scope
Training
High
No additional cost
High
A,B
Re-purposing
Medium
No additional cost
Medium
C.D.E,F,G
Specialization
No additional cost A relatively high fine-tuning cost
High
D,E
distribution. In practice, the most straightforward way to achieve conditional control in DM-based synthesis framework is incorporaling the conditional inpuls into the denoising network. In this section, we divide the condition integration in denoising network into three stages: (a) training stage: training a denoising network on paired conditional input and target image from scratch, (b) re-purposing stage: re-purposing a pro-trained denoising network conditional synthesis scenarios beyond the task it was trained on, (c) specialization stage: performing testing-time adjustments denoising network based user-specified conditional input.

---

Published in Transactions on Machine Learning Research (05/2025)
Stable Diffusion RRobacc et 2029.
Imagen
Saharia et al.
Text-to-image
2022b
Nichol aL. [2022] Itanesh al.
2022
Balaji al.
[2022]
Ga of al
2022
SR3 Sabaria
2022c
CDM (Ho
2022b
Training (Sec. 3.1
Palette (Sabaria cf al
2022a
Li et al.
12122a | Sahak et all
Image restoration
2023
Shang et al.
[2024]
Izhang et (20124
Jiang
2023a
Xue et al.
(2024
Zhao al.
(2224a
Preschakul ct al. [2022]
Wang ct al.
[022b]:
Zhang ct al.
Other synthesis
2022
Ld al.] [20231 Lin et (2023 Moghadam et al.
scenarios
2023
Meng et al.
2022b
Yang et al
(0222a) Gralkos et al.
T2I-Adaper Mouu et al. 2024c]. ControlNet (Zhang al.
2023b
PITI (Wang et al. [20224). BLIP diffusion [L et al
2025a
MCIE (Fu et (023) Kocsis et al.
20244 Zhang
Re- purposed conditional encoders
et al 2023d Goel et al (2023 : Yang et &
(021b) Xu et
al.
20211 Xiao
(2023)
(2024);
(2023b
Jia ct al. (2023] Li et al. 20233 1:
Lu ct al. (20244 Shi ct al.
2224a y Shhohhaaa
20124 Fong
2022 uaan
20236 Li et 2023e
Denoising Networks (Sec. 3
IP-adapter Ye ct al.
2023).
GLIGEN [[. et al. 2023i Dragon-
Re-purposing (Sec. 3.2)
Condition injection
Diffusion Mou
20244
Wei al. (2023 i
al.
(023
Wang et al.
(221b]] Q et al (2024 a Gu et al
2024
Instructpix2pix (Brooks
2023 PhE
(Yang
2023a
Yildirim et all (2023) Wel ef Al. M0023a
Zhang al:
202-10
Backbone fine-tuning
Geng et al (023) [:23:: SShevnin et al. (2028)
Zhang et 8.
2024c
Wang 2023b Chen et al.
(2023b Xie al.
(20 b [Sng al.
20:2002200 2023d
Kim et al.
2024cc
Zhang et 8l.
2024e
Imagic
ct. pl. 2023).
Textual inversion Gal
Conditional
2023
Wu et al. [2026 |
Mahajan et al. 2024 Revi al.
projection
2023
Zhang et al. (2023c
Bodur el al. (022
Specialization (Sec. 3.3)
Imagic Kawar ml.
2023 DreamBooth Ruuiz et
2023
Testing-time
Valevski et sl. [202
Li et al. 2023c
Zhang et a..
2023
model fine-tuning
Kumari et sL. [2223]:
Gal et (2023
C el 8l
20233
Liu
ct al.
2023d Gu et
al.
2024
Han et a.. 2023
Figure 4: The proposed laxonomy of DCIS works performing condition integration in denoising network.
In practice, the training stage is often employed for condition integration in fundamental conditional im- age synthesis tasks such as image restoration (Saharia et al. 2022c Li et al. 2022a) and text-to-image
[Rombach et al. 2022; Ho al.) 2022a Peebles & Xie 2023 This stage establishes reliable relation- ship between conditional inputs and targel images albeit at high computational cost due to the need for training from scratch. Given the substantial training cost, branch of works (Zhang et al. 2023bb Li et al.
20231 Zhang et al. 2023d Li et al. 2023a) opt to finc-tune a pre-trained tcxt-to-image denoising network to more complicated conditional synthesis tasks via a re-purposing slage. This strategy skips the training process and significantly reduces computational cost. However, the relationship between novel conditional inputs and target images re-established during the re-purposing stage is generally less reliable compared to training from scratch. In highly personalized tasks such customization (Lin et al.) 2024a Gal et al, 2023a) and image editing (Kawar et al. 2023) the lask-oriented denoising net works established through the training and re-purposing stages often fail to accurately reproduce fine-grained features from the given conditional inputs. In these cases, the specialization slage introduces time-consuming fine-tuning during inference time to align user-specific conditional inputs with the prior knowledge embedded in the denoising network, thereby ensuring detailed consistency between the synthesized image and the provided conditional inputs. We provide a high-level comparison of the pros and cons of performing conditional integration into the denoising network at each stage in Tab. 2
Fig. provides an examplar workflow to build desired denoising network for conditional synthesis tasks including text-to-image, visual signal to image and customization via these three condition integration stages.
Next, we first review the fundamental conditional DMs modeled in the training stage in Sec. 3.1 We then summarize the architocture design choices and conditioning mechanisms for the re-purposing stage Soc. 3.2
Finally, we introduce the works performing condition integration in the specialization stage in Sec. 3.3 Fig. illustrates the taxonomy proposed in this section.
9

---

Published in Transactions on Machine Learning Research (05/2025)
3.1 Condition Integration in the Training Stage
The most straightforward way to integrate the conditional control signal c into the denoising net is performing supervised training from scratch with the following loss function:
px |ll- co (xx, c)
(8)
where с and x denote the paired conditional inputs and target image. Thereby, the learned conditional denoising network co (x,,t, c) can be employed to sample from p(x[c).
Next, we introduce the existing conditional denoising networks trained from scratch, focusing on their model architectures and conditioning mechanisms, which are crucial for creating the connection between the con- ditional imputs and its corresponding image. Because conditioning architectures and mechanisms are always designed based the target conditional synthesis scenarios, categorize these works based on their appli- cations, represented by text-to-inage and image restoration.
3.1.1 Conditional Models for Text-to-Image (T2I)
Text-to-image is a fundamental task in the field of conditional image synthesis, which establishes the con- nection between images and the semantic space of text descriptions. Because of the expressiveness of text prompts, text-to-image DMs always serve as the backbone for more complicated conditional synthesis tasks including image editing (Kawar et al. 2023 Hertz et al.
2023 Brooks et al. 2023), customization (Gal et al.
2023a; Ruiz et. al., 2023). visual signal to image (Mou et al. 2024c Zhang el. al. 2023b). image composition
(Yang et al. 2023a and layout control [Wang et al. 2024b Li et al. 2023i).
The main challenge in modeling an effective text-to-image framework lies in (a) precisely capturing the users' intention described in text prompts and (b) building the connection between text prompts and images acceptable computational cost. In practice, DM-based text-to-image works design different text encoders
based on Transformer (Nichol et al. 2022 Rombach et al. 2022), CLIP (Ramesh et al. 2022 Balaji et al.
2022
Gu et al. 2022) or more powerful large language models (Saharia el al. 2022b Balaji et al. 2022) Lo extract the features from user provided text prompts. For computational efficiency. these works often train the DMs on a low-dimension space, e.g. compressed latent space (Rombach et al. 2022 Gu et al. 2022
or low-resolution pixel space (Nichol et al.] 2022] SSaharia et al.] [202cc Balaji al., 2022 Ramesh et al. 2022), and subsequently convert the synthesized results into desired images via auto-encoders upsampling diffusion models.
Next, we introduce two representative text-to-image models: Stable Diffusion (Rombach et al. 2022 and Imagen (Saharia et al. 2022b), which serve as the T21 backbone for various conditional synthesis tasks.
Similar to VQ-VAE Van Den Oord el al. 2017 and VQ-GAN Esser et al. 2021 , Stable Diffusion (Rombach et al. 2022) employs a pre-trained autoencoder to compress the generative space into a low-dimensional latent space for computational efficiency. In the training stage, the text-conditioned diffusion model Eø t, t, is trained on this latent space to approximate the conditional distribution of the latent representations. In sampling process, the latent representation aligned with given text prompts is firstly generated by the conditional diffusion model on latent space, and then fed into the decoder to recover its corresponding high-quality image.
For conditional control, Stable Diffusion introduces a transformer text encoder to interpret the text prompts and convert into the text embedding. Subsequently text embodding is fused with the fcatures in U-Net architecture of denoising network (Rombach et al. 2022) via cross-attention layers. In practice, the encoder can be different domain-specifi experts other than the text encoder. Thereby, Stable Diffusion can be employed into various conditional synthesis scenarios beyond text-to-image.
Following up the pioneer DM-based text-to-image framework GLIDE (Nichol et al. 2022 and Imagen (Saharia et al., (022b) prefer to train the conditional denoising network on a low-resolution image space and subsequently upsample the synthesized now-resolution image. In order effectively interpret the complex text prompts, Imagen employs pre-trained large language models (c.g. BERT (Kenton Toutanova 2019). GPT (Radford et al. 2021). T5 (Raffel et al.] (020)) powerful text-encoders. For condition injection,
10

---

Published in Transactions on Machine Learning Research (05/2025)
Imagen (Saharia et al. 2022b concatenates the encoded text embedding to the key-value pairs of the self- attention layers in denoising network. In Imagen, the basic 64 x 64 texi-to-inage diffusion model is followed by two cascaded super-resolution diffusion models designed to enlarge the resolution of syuthesized image from 64 x 64 to 1024 x 1024.
Recently, the DiT architecture (Peebles & Xie 2023) has achieved unprecedented sample quality in diffusion- based image synthesis, making it a popular backbonn for many cutting-edge
diffusion- text-to-image 2024]. Building on the con-
(T2I) synthesis models (Chen et al.) 2023 Esser et al.,
2024
Black- Forest
Chen et al. 2023) integrates 2024b introduces a novel and scalable DiT-based architecture for T2I synthesis. In contrast to earlier T2I models (Rombach et al. 2022; Ho et al., 2022a Chen et al. 2023). which rely on consecutive cross-attention and self-attention mecha- nisms to manage interactions between text prompts and images, MMDiT utilizes unified self-attention mechanism for bidirectional mixing between text and image tokens. Furthermore, MMDiT (Esscr et al., 2024) employs the rectified flow framework to model the transition process from Gaussian distribution to clean images. Leveraging the MMDIT (Esser et al.) 2024) framework, Flux (Black-Foree) 2024) achieves thate-of-hhe-ar T2I generation performance, excelling in tasks such as handling long sentences and capturing complex multi-object relationships, which remains challenging for Stable Diffusion.
ditioning mechanism of Stable Diffusion (Rombach et al. 2022
Cross- DiT
text conditions into DiT via cross-attention modules. MMDiT (Esser et al.
3.1.2 Conditional Models for Image Restoration
DM-based conditional training is also widely employed to recover the high-quality clean image x from a given degraded image с (Saharia et al. 2022c a Ho et al.) 2022b Shang et al. 2024 Zhao et al., 2024a). These studics primarily focus on extracting task-relevant features from degraded images to serve as conditional inputs for supervised training, enabling the model to reconst.uct clean images based on these essential representations.
2.1) Conditioning on degraded images. The most straightforward modeling approach involves directly con- ditioning the diffusion model on the degraded image through channel-wise concalenation. A pioneering diffusion-based super-resolution method, SR3 (Saharia et al. 2022c). implements this strategy by concate- nating the low-quality reference image with the latent variable in the channel dimension of the U-Net ar- chitecture. This straightforward conditioning mechanism allows the U-Net architecture comprehensively capture the informative content of the low-resolution image. Concurrent work SRDiff (Li et al.] 2022a) shifts the generative space of SR3 to the residual space, modeling the difference between pairod high- and low-resolution images to avoid reconstructing structures already present in the low-resolution inpul. As a result, SRDiff achieves performance comparable to SR3 while requiring significantly less computational cost. To adapt SR3 for real-world restoration scenarios, SR3+ (Sahak et al. 2023) introduces socond-order degradation simulation to construct more realistic clean/ /egganddd image pairs, thereby enriching the training dataset. Building upon SR3 (Saharia et al. 2022c CDM (Ho et al., 2022bb proposes a cascaded framework of super-resolution diffusion models progressively upscale image resolution, while Palette (Saharia et al. 2022al) extends the SR3 framework to broader range of image restoration tasks through supervised training task-specific paired datasets.
2.2) Conditioning on pre-processed features. However, directly concatenating the degraded image in the channel space imposes a burden on the denoising network, which must implicitly extract task-relevant features from the unprocessed input. To allocate the majority of modeling capacity task-relevant. features, a line of restoration studies (Shang al. 2024 Zhao et al., 20244] Jiang et al., 2023a Xue et al. 2024 Zhang eet al. 2024d) advocates first extracting these features from the degraded image and then conditioning the diffusion model on them.
The statc-of-the-art super-resolution framework ResDiff (Shang et al. 2024 utilizes a pro-trained Convolu- tional Neural Network (CNN) lo generate a high-quality intermediate image from the initial degraded input. It then conditions the denoising network on this intermediate image, along with its high-frequency compo- nents, model the residual between the intermodiate and the clean image. For more challenging restoration tasks, such as underwater image restoration (Zhao al.) 2024a) and low-light image enhancement (Jiang et al. 2023a Xue al.] 2024), where the input images suffer from severe degradation, branch of works
11

---

Published in Transactions on Machine Learning Research (05/2025)
ViT-based encader
encoder
Text
Layout
Tunable
Freeze
Reference image
Denoksing U-Net
x(T
1)
Crasi
Designed image
Cross
XT
X1
Back-hone Fine-tunning
X00
Sketch
Convelution- based encoder
Figure 5: An illustration of the purposeed denoising network based the text-to-image backbone, where < denotes condition integration via channel-wise concatenation, T2I attention layers, addition and developed attention modules respectively as described in Sec. 3.2.2
prefer condition the model on frequency information extracted by discrete wavelet transformations. To restore real-world lext images under degradation, DifTSR (Zhang el al. 2024d) conducts parallel dir- fusion processes consisting of image diffusion model for image restoration and text diffusion model for text recognition. A multi-modal interaction module is further employed to facilitate information exchange belween the two diffusion streams.
3.1.3 Conditional Models for Other Synthesis Scenarios
While most diffusion model base) frameworks for complex conditional synthesis scenarios are built by re-purposing text-to-image backbones, a number of studies instead adopt supervised training from scratch for various conditional synthesis tasks. Some of these early efforts preceding the widespread adoption of
DM-based text-to-image models, targeting tasks such as image editing (Preechakul et al. 2022 and visual
signal to image synthesis (Wang et al. 2022b Zhang et al.) 2022). Others are developed for novel or highly specialized applications, including medical image synthesis (Li et al.
2023j Liu et al. 2023a Moghadam et al. 2023 Meng et al. 2022b), graph-to-image generation (Yang et al.
2022a and satellite image synthesis
Graaiko et al. 2023). where the conditional control signals are difficult to align with the semantic space of text-to-imeg text-to-image backbones.
3.2 Condition Integration in the Re-purposing Stage
Currently, diffusion modelssDDMs are employed in increasingly diverse and complex conditional synthesis
scenarios Ye et al. 2023 Zhang et al. 2023b Li et al. 2023e Zhang et al. 2023d Li et al. 2023i Wang let al. 2024b Shi et al.] 2024b). Simply training denoising networks from scratch for each conditional synthesis scenario would place a heavy burden on computational resources. Fortunately, pre-trained text- to-image(T2I)) DMs effectively associate text embedding with its corresponding image, which serves as a semantic powerful backbone for wide range of conditional synthesis tasks beyond the T21. Studies design task-specific denoising network based T2I backbone and performing fine-tuning pairs of conditional inputs and image to re-purpose the T21 denoising network to the target task. In practice, the re-purposed denoising network can be divided into three key modules: (a) Conditional encoder: The module to encode the lask-specific conditional inputs into feature embeddings, (b) Condition injection: The module to inject taskerelated feature embeddings into the T21 backbone, (c) Backbone: The T21 backbone that can stay frozen be fine-tuned during the re-purposing stage. In the re-purposing stage, conditional fine-tuning can be performed each of these components for condition integration. this section, we summarize the design choices for these modules adopted by existing works for condition integration during the re-purposing stage.
12

---

Published in Transactions on Machine Learning Research (05/2025)
3.2.1 Re- re-puuposed Conditional Encoders
In T21 diffusson models, text embeddings are extracted from input prompts via a text encoder and integrated into the U-Net architecture through cross-attention layers. To adapt the T2I backbone for tasks beyond text- to-image generation, a range of task-specific conditional encoders have been developed to extract features from alternative control signals beyond textual inputs.
1.1) Convolulion loyer-bused encoder for visual signals. For visual signals, conditional encoders are mainly designed based convolution downsample blocks to extract multi-scale structure features.
Pioneer work T2I-Adapter (Mou et al. 2024cb employs a four-layer convolutional network as a lightweight adapter to encode the visual signals into a set of multiscale features. ControlNet (Zhang et al. 2023b) provides a more powerful architecture as the encoder for visual signals, which clones the deep encoding layers in the U-Net architocture of Stable Diffusion. This ControlNet encoder inherits wealth of prior knowlodge in the Stable Diffusion backbone and serves as a deep, robust, and strong architecture for diverse visual signals. Currently, ControlNet delivers state-of-the-art results in diverse visual signal to image tasks and becomes widely-employed conditional encoder various more complicated conditional synthesis scenarios including explicit lighting control [oossi et al.) 2024), image composition (Zhang et al. 2023d), image editing (Goel
et al. 2023 Zhang et. al. 2024e) and virtual try-on (Kim et al., 2024 Zeng el al. 2024).
1.2) ViT-bused encoder for images. In practice, Vision Transformer (ViT)-based encoders are widely utilized to extract features from conditional control signals represented in the form of images. Since most visual signals can be naturally represented as images, pioneering work such as PITI (Wang et al.) 2022a) introduces ViT-based encoder to project visual inputs into corresponding text embeddings for use in T2I backbones. Similarly, ImageBrush (Yang et al. 2024b) adopts a ViT-based encoder to capture visual editing instructions from paired images before and after editing. Prompt-free Diffusion (Xu et al. 2024) further enhances visual encoding by employing a more powerful context encoder (SeeCoder) built on Swin-I. (Liu et al., 2021) to convert given images into meaningful visual embedding.. For customization, a branch of works (Xiao et al.
2023 Ma et al. 2024 Shi el al.) 2024aa Gal et al.) [0233 Jia et al., 2023 Li et al. 2023h Lu et al. 2024
et al. 2023a Shiohara & Yamasaki 2024] project uper-specific objects into features on the textual embedding
space via image encoders built upon different ViT-based frameworks such CLIP (Radford et al. 2021
Swin (Liu el al. 2021 BLIP (Li el al. 2023b), ArcFace (Deng el al. 2019
1.3) LLMs-based encoder for image editing. In recent years, the rapid development of Multimodal Large Language Models (MLLMs) [Lu et al., 2024b), which are capable of jointly understanding semantic infor- mation and associated visual content, has led many recent works to adopt MLLMs as powerful conditional encodors for image oditing tasks that require integrating semantic text with the given image for precise manipulation. Fu et al. (2023) Huang al. (2023c Li et al. (2023e) leverage trainable Multimodal Large Language Models (MLLMs) (Lu et al. 2024b) module the encoder for the given source image and editing instruction. Ranni (Feng et al.
2023 used MLLMs to convert description editing prompts into a semantic panel, which serves as an intermediate representation that contains rich structure and semantic information.
3.2.2 Condition Injection
In order to more effectively incorporate information in conditional inputs into the denoising network during the re-purposing stage across various conditional synthesis scenarios, studies in this field have designed different task-specific condition injection approaches to handle different types of conditional control signals. Here, we categorize these methods into the following four categories.
2.1) Condition injection via concatenation For conditional inputs in the form of image, direct condition injection approach is following the concatenation strategy proposed by SR3 (Saharia et al.) 2022c), which concalenates the image form conditional inputs to the latent variable in the channel space of the U-Net architecture. In practice, this conditioning strategy is usually performed with backbone fine-tuning to handle conditional synthesis tasks that involve complex conditional inputs composed of multimodal components, including cnaruction-basedd editing (Brooks al.] 2023 Sheynin al. 2024 Geng et al. 2023) and image
composition (Zhan et al. 2023d Song et al. 2023d Xie et al.] 2023a
13

---

Published in Transactions on Machine Learning Research (05/2025)
2.2) Condition injection vio T2I ottention layers V In the T2I backbone, the cross-attention layers serve the conditioning module to inject text embedding into the U-Net architecture. Currently, a branch of works also employs the cross-attention layers in the T21 backbone to inject the features extracted from task-specific
conditional encoders (Wang et al., 2022a Yang et al.) 2024b Xu et al.) 2024 Xiao et al. 2023 Gal ct al.
2023b;
Jia et al. 2023 Li et al. 2023a Shiohara
Yamasaki
2024
Zeng et al. 2024).
2.3) Condition injection via addition Because of the alignment between the architecture of conditional encoder and the U-Net encoder in T2I backbone, for convolutional layer-based encoders (Mou et al., 2024c;
Zhang et al., 2023b,, the extracted features are injected via directly adding these features to the corresponding intermediates layers in the U-Net architecture of T21 backbone.
2.4) Condilion injection via developed allenlion modules 4. To achieve more fine-grained control over the syuthesized image, some works developed task-specific attention modules for condition injection in target
conditional synthesis scenarios (Ye et al. 2023 Li ct al.) 2023i Wei ct al. 2023b Wang al. 2024b Mou
et al. 2024a).
A branch of works prefers to incorporate extra attention module into the T21 backbone to inject the task-
specific conditional control signals Ye et al 2023 Wei ct al., 2023b Li ct al.
2023i Hoc et al. 2023 Wang
ct al. 2024b). IP-adapter eYe al. 2023) employs additional image cross atoss-tttention layers to inject the image
embedding into the T2T backbone. For customization, ELITE (Wei et al. 2023b) leverages two parallel cross-
attention layers to inject extracted global and local information of the personal object separately. In T21 backbone, attention layers control the structure and layout information of the synthesized image. To exert accurate object-level layout control, branch of works prefer to add trainable attention module ween
self-attention and cross-attcntion layers (Li ct al. 20231; Ma et al., 2024 Shi et al. 2024a Hoe ct al. 2023; Wang et al. 2024b). GLIGEN (Li et al. 2023ih adds gated self-atlention layer to the U-Net archilecture to inject provided layout information. This conditioning strategy is further employed in customization works (Ma et al. 2024 Shi et al., 2024a) to integrate patch features extracted from personal object images. To perform more detailed layout control, Interact (Hoe et al. 2023) designs an attention-based Human- Object Interaction module to inject the interactions between objects. InstaaneeDiffusion (Wang et al. 2024b) projects different forms of object-level control signals including single points, scribbles, bounding boxes intricate instance segmentation masks into the feature space through MLP tokenizers, and fuses these features with visual lokens from the lext-lo-image backbone via gated self-attention layers.
Another line of works modifies the cross-attention mechanism in T21 backbone to achieve more precise control OQi et al.) 2024 Mou et al. 2024a Lu et al. 2024 Gu et al. 2024h. Different from IP-adapter (Ye el al.] (023). ; DEADiT (Qi et al.
2024] concatenates the key and value allention features derived from image and text embedding respectively and performs a single fused cross-attention mechanism to achieve multimodal conditional control. In practice, performing fused attention mechanism to inject multimodal control signals along with text embedding is also employed in instruct-based editing (Li et al.) 2023f) and pose-guided person image synthesis (Lu et al. 2024). To perform local control based on multiple regional prompts, Mix-and-show (Gu et al. 2024) proposes an attention localization strategy in the re-purposing stage, which substitutes the attention map in specified regions with the al alntion map generated based on the regional prompts.
3.2.3 Backbone Fine-tuning
Currently, most of the re-purposing works confine the fine-tuning only on conditional encoders and condition injection modules to case the computational burden. However, for conditional inputs that contain multimodal components or intricate semantics, performing fine-tuning while freezing the parameters in T2I backbone often fails to fully understand intrinsic connections between the conditional input and target image. In these sconarios, fine-tuning the T2I backbone together with encoders and condition injection modules is a more preferable choice. Based on the fine-tuning strategy, we categorize these works into two types: (a) Fully supervised fine-tuning on annolated datasets, and (b) Self-superviser fine-tuning on bare image dataset.
3.1) Fully superoised fine-tuning on the annotated dataset. In practice, we can re-purpose the T2I backbone on the annotated dataset of paired conditional input and image in accordaacce with the specific task via fully supervised fine-tuning. However, for some synthesis tasks involving complex conditional inputs, major
14

---

Published in Transactions on Machine Learning Research (05/2025)
difficulty lies in collecting sufficient training data to fine-tune the model (Brooks et al. 2023 Zhang et al, 2023d). For instruct-based editing task, which refers to using instructions instead of text descriptions to guide the editing process, Instructpix2pix (Brooks et al., 2023) provides an effective approach for automatically synthesizing training datasets. Firstly, InstructPix2Pis employs a fine-tuned GPT-3 (Brown et al.) 2020) to synthesize editing triplets composed of the input caption, edit instruction and output caption. Subsequently, Instructpix2pix leverages Prompt-to-Promp (Hertz et al., 2023) to synthesize paired images corresponding to the input captions and output captions, which serves as the paired images refore/after editing. This contribution leads to line of works on DM-based instruction editing. A branch of follow-up works attempts to enhance the T2T backbone in some specific tasks by augmenting the training dataset for target scenario including object removal and inpainting (Yildirim et al. 2023), global editing (Li et al. 2023f), dialog-base editing (Wei et al. 2023a) and cont inuous editing (Zhang el al. 2024b).
(Geng el al.) 2023)
and Emu-edit (Sheynin et al. 2024] fine-tune the T21 backbone on larger and more comprehensive synthesized datascts for wide range of vision tasks including image oditing, segmentation, keypoint estimation, detection and low-level vision. To achieve more accurate editing, [uu et al. (2023); Huang al. (2023e); Li et al. (2023e)
fine-tune the T2I backbone with a more powerful MLLMs-based conditional encoder to enhance the editing prompts. Based on reinforcement learning, HIVE (Zhang et al. 2024c) fine-tunes the instruct-based editing model with reward model reflecting the human feedback for editing performance.
3.2) Self-superoised fine-tuning on bore image dataset. In non-general conditional synthesis scenarios in- volving image composition or mask-based editing, the form of conditional inputs may be complicated. For example, a classic image composition task aims to fuse a foreground reference image into the background main image within the mask region. In these tasks, collecting annotated training data pairs is almost impossible. A feasible approach is to create paired data based on the target scenario through cropping on bare image dataset, and thereby fine-tune the T2I backbone in self-supervised manner. For image composition task, РъЕ (Yang et al. 2023ab randomly crops the foreground objects from the source image as the reference image and the corresponding mask, while the remaining background as the background main image. Subsequently, PbЕ (Yang et al.) 2023a) fine-tunes the T2I backbone with the cropped reference image and main image. In practice, such a strategy is widely employed in conditional synthesis scenarios involving inpainting (Wang et al. 2023b Xie et al. (023a)) and composition (Song et al.) 2023d Kim et al. 2023b Zhang et al.) 2023d: Xic et al. 2023b Chen et al.] 2024c). To generate reasonable masks for text-based inpainting, Imagen Editor (Wang et al.) 2023b) employs an off-the-shelf object delector to generate masks on the image in captioned image datasets, which covers a region relevant to the text caption of image. SmartBrush (Xie et al. 2023a)
randomly augments the cropped training masks to create accurate instance masks, which facilitates the T21 backbone to follow the shape of the input mask in testing-time.
For image composition, the greatest challenge faced by the self-supervised fine-tuning strategy is how to avoid the trivial copy-and-paste solution caused by the training data cropped from & single image (Yang) et al. 2023a Xie et al.) 2023b Zhang et al. 2024eh. Currently, image composition frameworks resort to compress the information in the conditional inputs into an information bottleneck. This, in turn, forces the T2I backbonc to interpret the intrinsic connections between the conditional input and the desired image, thereby effectively avoiding the copy-and-paste solution. РЫE (Yang el al. 2023a) and Dreaminpainter {Xie) et al.) 2023b) select a part of the image tokens for condition injection to create an information bottleneck. ObjectStitch (Song et al., (023d) employs a two-stage fino-tuning strategy to decouple the fine-tuning stages of the conditional encoder and the T2I backbone. ZZhang et al. (2024e]; Chen et al. (2024c); ZZang et al. (023d) prefer to remove mask out the information such as colors, textures or background in the source image to prevent identical mapping.
3.3 Condition Integration in the Specialization Stage
Although, in theory, any form of conditional input e can be incorporated into the denoising network (x(x,, c), in practice, integrating complex control signals into the conditional space of the network during training and re-purposing presents significant challenges. These challenges primarily stem from the difficulty of collecting annotated training data and modeling the intricate relationships between conditional inputs and the desired outputs, thereby limiting the model's capacity to generalize zero-shot or few-shot conditional inputs.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Visual subject
Conditional
Projection
of
Denolsing U-Net
x (T
1)
Cruss attention
XT
X1
Testing-time Fine-tuning
X0
Figure 6: The specialization process to align a given personal object (the clock) with a pesudo-word S* in the conditional space of a text-to-image backbone. The clock image cited from Textual Inversion (Gal et al. 2023a).
A straightforward idea to remedy these issues is to align the user-specified conditional inputs with the condi- tional space of a general '21 backbone through a specialization stage. As shown in Fig. [ the specialization for given specific conditional inputs is typically achicvod by (a) conditional projection, which projects the given conditional inputs onto the conditional space of the T2I backbone via embedding optimization Kawar
et al. 2023 Gal et al. 2023a), or Vision-Language Pre-training (VLP) framework (Li et al.) 2022b 2023b).
(b) testing-time model fine-tuning, which fine-tunes the denoising network to insert the conditional inputs into the prior of the T2I backbone. In practice, works perform condition integration in the specialization stage are mainly targeted to image editing and customization tasks to achieve desired edits on user-specified
visual subjects including source imago(image oditing) and personal objocts(eustomization) while preserving
the characteristics and details in these visual subjects (Kawar al. 2023 Ruiz el. al. 2023 Gal el al.
2023a).
3.3.1 Conditional Projection
A widely used approach for editing or customization tasks involves projecting the given visual subject into a corresponding lext representation within the conditional space of a text-to-image model.
1.1) Conditional embedding optimization. In order to find proper text embedding for given visual subject, branch of works directly searches the optimal embedding for the user-specified conditional inputs by optimizing the following objective function:
[||e en (xr, v)12
= arg minE
(9)
where v denotes the optimized text embedding for the user-specified visual subject cı, and eœ denotes the T2I denoising nork The embedding v. serves as pseudo-word S* for the visual subject and can be further composed into various natural language prompts to create different editing renditions for the given visual subject (Kawar et al. 2023 Gal et al. 2023a).
2023 optimizes the embedding for the source image. Subse- quently, Imagic performs interpolation between optimized source embodding and target embedding Vigt to obtain v 17 Vegt (1 - 17) vv", which serves the conditional input for denoising process. Diflusion Disentanglement (Wu al. 2023) optimizes the time-dependent combination weights of the source and target text embeddings along the sampling process instead of interpolation to retrieve time-adaptable em- bedding for editing. To reduce the computational cost of the optimization process, (Zhang et al. 2023c Mou et al. 2024b) first. employed a image encoder to generale a coarse embedding of the given visual subject, and subsequently fine-tuning the coarse embedding via optimization.
For image editing, Imagic (Kawar et al.
Pioneer customization work Textual Inversion Gaal al. 2023a) performs optimization discover the text embedding v for personal object described by few reference images (typically to 5). This optimized
16

---

Published in Transactions on Machine Learning Research (05/2025)
embedding v serves the pseudo-pronoun S* for the personal object in the further conditional sampling process. To provide human-readable text description instead of text embedding for given personal object, PH2P (Mahajan et al. 2024) employ quasi-newton L-BFGS (Shanno 1970) to directly optimize discrete tokens from an existing pro-specified vocabulary for the target image.
1.2) Employing VIPP models. However, performing time-consuming optimization for each new visual subject hinders the deployment of these methods in practical applications. Therefore, a branch of works prefers to employ Vision-Language Pre-training (VLP) models to directly generate the embedding for given visual
subjects [hhang et al. 2023c Li et al. 2023a
BLIP (Li et al. 2022b) is strong VLP framework to synthesize captions for given images, which is widely employed in image editing tasks to generate an initial lext. prompt to describe the uncaptioned source image (Zhang et al., 2023c Li et al. 2023a Bodur et al. 2024 Parmar et al., 2023). BLIP can also be used to enhance uscr-provided prompts for climinating editing failure caused by missing contexts in the coarse input prompls (Kim et al.
2023c). , Besides, PRedIOOR (Ravi et al.] (023) prefer to leverage DALL-E2 [Ramesh et al.] 2022 to fuse the source image with the target prompt by performing SDEdit (Meng et al.
2022al process on the CLIP embedding space.
3.3.2 Testing-time Model Fine-Tuning
In editing and customization tasks, directly using the denoising network built through scenario-oriented training and re-purposing often fails to preserve the characteristics and fine details of the user-specified visual subject, primarily due to the absence of subjoct-specific prior knowledge (Kumari ct al., 2023)
To customize the T2T backbone for user-specified conditional inputs, approaches in this category resort to performing testing-time fine-tuning the T21 backbone to insert the given visual subjects into the denoising
network Ruiz et al. 2023: Kumari et al. 2023)
To better preserve the outlook of the source image in editing tasks, a branch of works Kawar et. al. 2023 Valevski et al.) 2023 Zhang et al.] 2023e represented by Imagic (Kawar et al. 2023) fine-tune the T21 backbonn to bind the source image with its corresponding text description Care in the conditional space of T2I backbone. In order to simultaneously edit the foreground and background in the source image, LayerDiffusion (Li et al., 2023c) employs Segment Anything Model (SAM) (Kirillov al.] 2023) to create masks for foreground objects. Subsequently, LayerDifflusin (Li et al. 2023c) fine-tunes the T21 backbone with a designed loss composed of the diffusion loss in both foreground and background region to edit the foreground object and background independently. SINE (Zhang et al.] 2023f) introduces a patch-based fino-tuning stratogy which incorporates the positional embedding into conditional T2I space to synthesize arbitrary-rrsolutionn edited image.
For the customization task, BreamBooth (Ruiz et al. 2023 fine-tunes the T21 backbone entangle fixed unique identifier with the semantic mcaning of the personal object. To alleviate the computational burden in the testing-Line time fine-tuning, followed up works (Kumari et al. 2023 Gal et al. 2023b Choi et al. 2023 Liu
et al. 2023d Gu et al. 2024 Han et al. 2023) prefer to only fine-tune a specific part of model parameters.
CustomDiffusion (Kumari et al. 2023) finc-tunes only the cross-attention layers. E4T (Gal ct al. 2023b)
optimizes low-rank adaptations (LoRA) Hu et al., (222]) of weight residuals in cross- and self-attention
layers to further reduce computational cost. Cones (Liu et al. 2023d) fine-tunes the attention layer concept neurons highly-related to the given visual subject. Cones2 (Liu et al.) (022e)) and Mix-and-show {Gu et al.,
2024h resort to fine-tune the text encoder in T2I backbone. SVDiff (Han et al. 2023) fine-tunes the singular values of the decomposed convolution kernels.
4 Condition integration in the sampling process
In DM-based image synthesis frameworks, the sampling process iteratively reverse noisy latent variable into desired image with the prediction of the denoising network. As mentioned Sec. B integrating the conditional control signalss into the denoising network always requires time-consuming training, fine-tuning or optimization. To ease the burden for conditioning the denoising network, numerous works perform condition
17

---

Published in Transactions on Machine Learning Research (05/2025)
4.1: Inversion
x
x
DDEM
10
XL
x1T -1-1
Inversed DDIM
Cunditional
Zabva"
exx, Zebra")
x
(
بلي Ae i
Denolsing
Allention Manipulatian
Noise Blending
Step
(x...
xit 1
X 4)
Conditional
X:-1 = Mike1 + [1 Mfx-11
TegrreerR "Zemmn'
laversed PDIM Stay
Densising Step
4.5: Guidance
X--1
"A Horse"
Zebva"
Conditional Denoising Step
Source inndge
Target
Figure 7: An example of the conditional sampling process for image editing, in which we incorporate all six mainstream in-sampling conditioning mechanisms for diffusion sampling process to provide a comprehensive overview the content in this section. The sample images are from Diffedit (Couairon et al. 2023
Table 3: A Comparison of the conditioning mechanisms for sampling process. Tn the "Inference Cost.' col- umn, we present the additional computation cost for performing the corresponding conditioning mechanism in sampling process (where denotes the lotal number of sampling steps). The 'Guarantee" column illus- trates the synthesis quality guarantees of conditioning mechanism. In this column, 'theoretical guarantee" indicates the conditioning mechanism is thcoretically supported to sample from the corresponding conditional distribution, while 'empirical results only" means the method is developed based successful experimental results, at the cost of disrupting the structure of the standard sampling process.
Comparison of conditioning mechanisms for sampling process
Conditioning Mechanism
Inference Cost
Guarantees
Applied scopes
Inversion
T NFEs for denoising network
Thcoretical guarantce
D,E.F
Attention Manipulation
NFFs for denoising network
Empirical results only
Noise Blending
T NFEs for denoising network
Theoretical guarantee
ALL
Revising Diffusion steps
additional cost
Thcoretical guarantee
Guidance
T NFFs for guidance loss function
Depends on the guidance loss
ALL
Conditional Correction
No additional
Empirical results only
B.D,F
integration in the sampling process to ensure the consistency between synthesized image and given conditional
input without computational intensive supervised-traiie training or fine-tuning (Su et al. 2023 Hertz et al.] 2023
Liu et al. 2022: Kawar al. 2022 Dhariwal & Nichol 2021 Choi al. 2021).
Based on how the conditional control signals are incorporated into the sampling process, divide main- stream in-sampling conditioning mechanisms into six categories: (a) inversion, (b) attention manipulation.
(c) noise blending, (d) revising diffusion process, (e) guidance and (f) conditional correction. In Tab. we provides a comparison of the characteristics of all the six conditioning mechanisms for sampling process.
We illustrate these conditioning mechanisms with an exemplary image editing process in Fig. In this section, we will introduce the core idea of these conditioning mechanisms and summarize the corresponding representative works as illustrated in Fig. g
18

---

Published in Transactions on Machine Learning Research (05/2025)
SDEdit (Meng et al.
20224
DDIB (Su et sl 2023 L Null-text luver-
sion Mokady
2023),
Cycledifusion (Wu & De la 'Torre 2023
DDPM inversion HubermannSSpa Spiegelg et al. [2023]. Dong ct al. (008):
Inversion (Sec. 4.1
Huang
2023a Wang
(2023d) Wallacs
al. [2023] Iaae
et sl.
2023
Ju et al (2028) i Meiri et al. [0223 Pan et al. (2023 Lu
al. (2024) 1 Nie al. (2023)) Zhang et a..
Chung ct al
2224 Shi ot al. [2024b]
Proupt-to-Promp (Hertz et al. 2023). PnP (Tumanyan et al. 2023 Mas-
actrl (Cao al. (202]] Ddiff-i [Balaji et al. 2022). CCenn Lat herr
Attention
2023 Choi ct al. (2023] Wang of al. (2023d E Yang ct al. (20244 F Lam
manipulation (Ser.
4.2
2024a]; Claung el al.] (2024)
Shi SL. 2024b Mou
(024m)
Lu ct al (2023) ; Liu et al.
(2023e
Guo & Lin [2022
Composable DMs [Luu et al. 2022 Classifter-fres guidance (Ho & Sali-
mans 2022 [Brack aL [2024]
Zhno al. (2023a); [Shirakawu & U Chida
Noise blending (Sec.
[2024
Bar- Tal ot al. (2023 Pan ct al. (2028 Zhang ct al. (023) Goel
2023
Sampling (Sec. 4
IR-SDE [Luo et al. 2023
SNIPS
(Kawar at al. 2021 DDRM
Kawar
Revising process (Sec.
ct al. 2022), Wolker al. [2024
Yue et al. (2024);
Wang et al.
2024d :
Delbracio & Milantar (2028 Wang et al. (2024g
Classifier Guidance DDhriwal & Nichol 2021). J, MCG [Chung et al.
2022h
DPS (Chung
(023b) Blend Ditrusion [Avrahami
(222))
Skeeth
guided DM [Voynov et al. 2023), ) Pix2Pix-Zero [Parmar et al 2023) Free-
DoM Yuu al. 2023 Universal Guidance [Bansal all 2023, /Sog et
Guidance (Sec.
4.5
2023e Rout al (2024b) Chung et al.
(2023a Fel et all (2026) : Liu
el sL.
20233) y Kwon & Ye (2023); [Singh el all [2023] 3 Luo el all 2021: Mo
ct
20244 Lin et al, (2023) Park al. (2024) Chen
(20244)
stein ct
al. (204 y Mou ef al. (0024baa)
SDE [Song
2021b). Repaint (Lugmayr
2022). ILVR Choi
Conditional
ct al. 2021), Diffedit
(Couairon et al. 2023). CCDF (Ching et al 2022b)
correction (Sec.
MCG
[Chung et. 2022n). Patashnik al. 2023
Wang et. 2023a
Lin et sl. [20245] Huang ot al. (223b
Figure 8: The proposed taxonomy of DCIS works performing condition integration in sampling process.
4.1 Inversion
In diffusion model (DM)-based image synthesis, the starting latent variable controls the spatial structure and semanties of synthesized result. Inversion process provides effective way to encode the given source image back into its corresponding starting latent variable and effectively preserve the image structure and semantics for further editing. In this section, we firstly summarize the inversion approaches in Sec. 4.1.1 Next, we will discuss the applications of inversion in various conditional synthesis scenarios in Sec. 4.1.2
4.1.1 Inversion Approaches
Mainstream inversion approaches perform inversion based on the forward diffusion process, deterministic sampling process and stochastic sampling process. We denole these three basic inversion pathways as noise- adding inversion, deterministic inversion, and stochastic inversion, respectively.
1.1) Noise-adding inversion. Noisc-Adding Inversion performs standard forward diffusion process to invert the source image to certain noise step T', i.e., q (XT" xo) = - (XT:; Vor, (1 - T), where the latent variable Xy is a mixture of source image and Gaussian noise.
1.2) Deterministic inversion. In practice, noise-adding inversion may smooth out details in the source image. To more precisely preserve image features, deterministic inversion is proposed to encode the source image Xo into its corresponding latent variable Xy based on the discretization form of diffusion ODEs such as DDIM (Song et al., 2021a)). Theoretically, with sufficiently large difflusion step T, DDIM inversion can guarantee perfect reconstruction, which ensures the latent variable xy obtained from DDIM inversion to be meaningful diffusion starting point encapsulating all features pertaining to the source image Xp-
1.3) Stochaslic inversion. Different from deterministic inversion approaches performing inversion based on deterministic sampling process, a branch of works prefers to invert the stochastic sampling process in Eq. Unlike the deterministic sampling process, which is determined by the starting point latent variable XT. the stochastic sampling process involves the noise vector e, added in each reverse transition kernel. Therefore, we have to memorize each noise vector e, along the inversion process to ensure the reconstruction
19

---

Published in Transactions on Machine Learning Research (05/2025)
property. Despite the additional memory requirements, stochastic inversion alleviates, to some extent, the reconstruction failures caused by accumplated errors in deterministic inversion.
1.4) Enhanced inversion approaches. In conditional synthesis, the classifier-free guidance for condition strengthening significantly magnified the accumulated error in inversion process, which leads to poor re- construction and edit performance. Therefore, a series of inversion methods are developed to ensure the inversion performance under classifier-free guidance.
For deterministic guided sampling process to reduce the reconstruction error, including optimizing the null-text. embed- ding (Mokady et al., 2023), text embedding for the source image (Dong et al. 2023), key and value matrix in the sell-allention layers (Huang el al. 2023a),, and the prompl embedding for cross-attention layers (Wang et al.) 2023d). To get rid of the computational burden for fine-tunig,, branch of works has developed
inversion, some approaches prefer to fine-tune relevant parameters in the classifier-fre ffree
tuning-free approaches for perfoct reconstruction (Wallace al.) 2023 Han al. 2024 Miyake et al.) 2023;
Ju et al. )023). EDICT (Wallace al., 2023) achieves precise DDIM inversion by utilizing an equiva- lent reversible process consisting of two coupled noise vectors. Negative-prompt Inversion (Miyake et al.
2023) demonstrates the prompt of the source image can serve training-free free substitute for null-text emn bedding. Proxedit (Han et al. 2024) further enhance the reconstruction performance of Negative-promp prompt Inversion (Miyake et al., 2023 by incorporating a regularization term in classifier-free guidance to prevent over-amplifying the editing direction in sampling process. fixed-point Inversion (Meiri et al.) 2023) and AIDI (Pan et al. 2023) perform fixed-point iterations in each step of DDIM inversion to reduce the accumulation errors due to the discrete DDIM process. Besides, Fixed-point Inversion (Meiri et al., 2023) provides a brief cycle of fixed-point iterations for the VAF-encoded latent representation of the source image to eliminate the misfit between latent representation and the given text prompts in latent diffusion models. TF-ICON (Lu et al. 2023) and LEDITS++ (Brack et al. 2024 perform inversion based on high-order diffusion differential equation solvers (Lu et al. 2022a b) which significantly accelerates the inversion process and improves the accuracy of inversion.
In theory, for stochastic inversion, any sampling sequence initialized from the source image can serve as the sequence of latent variables in the stochastic inversion process. However, arbitrary sampling trajectories may deviate from the prior marginal distribution of latent variables, thereby compromising the model's editing capability during the reccton process. To construct a reasonable sampling sequence, pioneer work Cyclediffusion (Wu & De la Torre 2023) samples a Xy ~ (O, and subsequently denoises it based on the source image Xo to recover the sampling sequence. DDPM inversion ([Hubernau--Spieeelllee et al. 2023 constructs an fliting-friendl sampling sequence by independently sampling each intermediate latent variable x, based on the source image Xo. This approach enables reconstruction of the source image with the noise precisions, effectively mitigating error accumulation during the inversion process. SDE-Drag (Nie) et al. 2023) provides a theoretical foundation for the superior editing performance of stochastic inversion compared to its deterministic counterpart. It shows that, under stochastic inversion, the KL divergence between the distribution of the edited image and the prior data distribution decreases, whereas it remains unchanged in widely adopted deterministic inversion methods.
4.1.2 Applications of Inversion in Conditional Synthesis
Inversion process converts the provided source image into its corresponding latent variable. In practice, this latent variable can serve as the starting point for the sampling process to perform basic image-to-image translation, text-based image editing or be further manipulated for more complicated tasks.
Image-to-image translation aims to translate the content of given source image to desired target ap- pearance, which serves as fundamental basis for image editing. The pioneering work SDEdit (Meng et all 2022a) performs this translation by denoising a noise-perturbed version of the source image with the de- noising network trained on the target domain. This process effectively preserves the structural content of the source image while imparting with the visual characteristics of the target domain. Li et al. (202bb) adopt the image-to-image translation strategy introduced in SDEdit to solve the subproblem associated with the prior term in the Half- Salf-Quarratic Splitting (HQS) framework |Geman & Yang 1995) for diffusion- basod image restoration.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Based on deterministic inversion, DDIB (Su et al., 2023) proposes highly flexible framework for image-to- image translation between two data domains, a and 13, via simple transformation x* = (x(E(())) Here, x and x" represent the source and target images in domains a and 3, respectively, while &, and Da denote the deterministic inversion and sampling processes performed using diffusion models specific to each domain. The DDIB framework can be applied using either two independently trained diffusion models or a single model conditioned on different control signals.
In practice, text-based image editing, which aims to modify a source image C, described by source prompt Care to align with the given target prompt Ctyt can be achieved by performing the DDIB image-to-image translation process: x* = De.ge (Ec..o (cr)). Here, e₁ and x" represent the paired source and edited images, while Eey and Degy denote the inversion and sampling processes conditioned on the source and target prompts, respectively. However, this editing process can only coarsely preserve semantic consistency and overall structure, often failing to retain the fine-grained details of the source image.
To more effectively preserve intricate details during editing, the inversion process is often augmented with additional conditioning mechanisms. One widely adopted approach involves applying conditional corrections
using spatial masks to protect regions that do not require modification (Couairon et al. 2023: Li et al. 2023c;
Patashnik et al. 2023 Yang et al.) 2024a Wang et al., 2023a Lin et al.
2024b Huang et al. 2023b). Another
line of work focuses on manipulating attention features during the editing process to explicitly incorporate
the visual appearance of the source image, as discussed in Sec 4.2 (Hertz et al.
2023
Tumanyan et al. 2023
Additionally, some methods enhance source-image fidelity by fine-tuning the model or applying conditional
projection techniques during the specialization stage, as outlined in Sec 2023c
Kawarr et al. 2023: Zhang et al.
Moreover, by leveraging lask-specific conditional encoders that convert multimodal control inputs into text embeddings, the inversion- based editing framework can be extended to conditional synthesis tasks beyond text-driven editing. For instance, InST (Zhang et al.] 2023c) performs style transfer by denoising a noise- perturbed reference image obtained via inversion with a denoising network conditioned on embedding vectors extracted from a style image.
For more complex conditional synthesis scenarios, the latent variables obtained through inversion can be further manipulated to incorporate additional information beyond the original source image. In image composition tasks, several methods propose fusing latent representation atio derived from multiple source images (Chung et al.) 2024 Lu et al. 2023 For example, Style Injection in Diffusson (Chung et al. 2024h combines the latent variables of both style and content images obtained via DDIM inversion to achieve style transfer.
Similarly, TF-ICON (T.uet et al. 2023) performs image composition by integrating the inverted representations of main and reference images. In drag-based editing, the latent variable can be spatially adjusted according to user-provided drag instructions. DragDiffusion (Shi et al. 2024b) refines the latent variable using motion supervision loss tailored for drag-style manipulation. In contrast, the stochastic inversion- mnversion-based method SDE- Drag (Nee et al. 2023) employs copy-and-pase strategy to manipulate latent variables without relying on latent-space optimization.
4.2 Attention Manipulation
After determining the starting point via randomly sampling from a Gaussian distribution or inversion meth- ods, the sampling process is performed by iterative denoising steps. As pointed out in E4T (Gal et al, 2023b), the attention layers in the denoising network have the greatest impact on the predicted noise and control the structure and layout of the synthesized image. Therefore, branch of works resorts to design task-specific manipulation to the attention layers in the denoising network to achieve more accurate control over the spatial layout and geometry of synthesized image (Hertz et al., 2023 Tumanyan et al. 2023 Lu ct al. 2023 Patashnik et al., (023). Different from the works performing fine-tuning on modified attention module re-purposing stage (Li et al. 20233 Ye et al., (203)), approaches in this category manipulate the attention layers via. tuning-free replacement localization during the sampling process.
21

---

Published in Transactions on Machine Learning Research (05/2025)
4.2.1 Replacement Manipulation
Pioneer attention manipulation works are designed to preserve the structure of the source image during the inversion-based image editing process. Prompt-to-Prompt (Hertz et al., 2023 performs parallel sam- pling processes for the inverted source image separately conditioned source and target prompts. During the sampling process, Prompt-lo-Prompe replaces the cross-attention maps in the editing branch with its counterpart in the reconstruction branch to preserve the structure of the source image during the editing process. This replacement strategy is further employed in follow-up works for face aging editing (Chen & Lathuilière 2023 and customization based editing (Choi et al., 2023 P2Plus (Wang et al.) 2023d) further performs attention replacement when predicting unconditional noise term in Eq. 7 to achieve more accurate editing under classifier-fres free guidance. To prevent undesired changes caused by cross-attention leakage, DPL (Yang et al., 2024a) optimizes the word embedding corresponding to the noun words in the source prompt to produce more suitable aross-attention maps for attention replacement.
PnP
2023 points out that more detailed spatial features are restored in self-attention layers compared to the cross-attention maps. Therefore, a branch of editing works (Tumanyan et al. 2023
Tumanyan et al.
Liu et al. 2024a Cao et al.,
2023) prefers to replace query and key features in self-attention layers to achieve better structure proservation. This replacement strategy followed by works designed for drag-based based editing (Shi et. al. b024b Mou et al. (0244) and style transfer (Chung al. 2024) to ensure the consisteney ween the synthesized result and the provided source image. However, performing replacement manipulation
attention maps locks the spatial layout of the generated image to that of the source image. To support more complex image editing scenarios, a series of works (Cao et al., 2023 Huang et al. 2024a) prefer to perform attention replacement on the key and value features within the attention layers, enabling the model to handle structural changes in non-rigid editing tasks.
In practice, the effectiveness of editing highly depends on the capability of the underlying text-to-image model. Currently, DiT-based text-to-image models (Peebles & Xie 2023: Chen et al. 2023 Esser et al.,
2024 Black-Forest 2024] demonstrate significantly stronger language understanding and image generation
capabilities compared to traditional text-to-image models (Rombach el al. 2022 Ho el. al.
2022a). As a
result, a series of recent works (Wang et al. 2024a] Tewel et al. 2024 have chosen to apply the classic
attention manipulation strategics to these models (Esser et al.] 2024 Black-Forcs, state-of-lhe-art performance in image editing.
2024 which achieves
4.2.2 Localization Manipulation
In order to enable more precise layout control in the synthesized image, a branch of works manipulates [Patashnik et al. 2023 Lu et al. 2023 Balaji et al. 2022) the attention layers with masks or segmentation maps that specily object locations.
Some of these works propose localized self-attention mechanisms to address different regions separately and locate the contents into desired regions. Masactrl (Cao et al., 2023) and Object-Shape Variation (Patashnik) et al. 2023) firstly extract the regions with attention value above threshold in the cross-attention maps for object text tokens as foreground masks. Subsequently, Masactrl performs self-attention for foreground and background separately to prevent mixing the foreground objects and the background. Object-Shape Variation (Patashnik et al. 2023) restricts the region for attention replacement on the background instead of injecting the full sell-attention maps in every denoising step. For image composition, TF-ICON (Lu et al., 2023) fuses the attention features extracted from the reconstruction process of both main and reference images via a cross-allention mechanism create a composite self-attention map seamlessly blending the two images.
Another line of work incorporates an increment into the cross-attention map to adjust the attention values in the region for designated objects and thereby achieve layout control for the synthesized image. Pioneer text-to-image work Ediff-i [Balaji et al.) 2022) successfully guides the object described by the nouns in the text prompt to the specified area by enhancing the its attention values in the corresponding region. Similarly, Cones2 (Lu et al., 2023c) increases the attention values in the region corresponding to desired objects while reducing the attention values in irrelevant regions to perform layout control. For image editing, Fol (Guo &
22

---

Published in Transactions on Machine Learning Research (05/2025)
amplifies the attention value in the region of the foreground object to be edited to achieve more precisely control for the objects in accordance with editing instructions.
Lin 2023
4.3
Noise Blending
The noise blending process integrates noise predictions from multiple (conditional) diffusion models to enable unified sampling process guided by multiple control signals. Based on the corresponding application scenarios, existing noise blending methods can be categorized into Noise Composition and Classifier Free Guidance.
4.3.1 Noise Composition
In conditional synthesis scenarios that require generating images based on multiple control signals, directly training denoising network to simultaneously handle all conditional inputs often results in prohibitively high training costs. A widely adopted approach for handling multi-conditional synthesis involves independently predicting the noise e, for each conditional component C, and subsequently composing them into a proxy noise E that reflects the influence of all control signals. Composable Dilfusion Models (Liu et al. 2022 propose noise composition method based on Bayes' rule to enable multi-conditional image synthosis, formulated as follows::
غ = Eo (x,,t) + Σ w; (Eo (x,,t, c;) Eo (x,,
(10)
where the unconditional denoising network eg (x,, t) can be trained along with the conditional model by substituting the conditional inputs с with emptyset 0. The noise composition can be performed based on masks or layouts to locate the objects in provided conditional inputs into desired regions. To perform im- age editing on multiple instructions, LEDITS++ (Brack et al.) 2024) calculates the mask for the region related to each instruction with the grounding information in cross-attention layers and noise estimations. Subsequently LEDITS++ (Brack et al.) 2024) performs noise composition based on the formula. of Eq 10 while restricting effect of the conditional term Eg (X,, f, q.) сa (x,, of each editing instruction C₁ in its corresponding mask region. In order to fuse the synthosized results of two diffusion models, MagicFusion [Zhao et al. 2023a firstly generates a mask by contrasting the saliency map the two diffusion models to differentiate the region controlled by each model. Subsequently, MagicFusion (Zhao et al., 2023a) settles the noise into the region controlled by its corresponding diffusion model. Similarly, NoiseCollage (Shirakawa) & Uchida 2024) independently estimates the noises for each individual object and then merges them with crop-and-merge operation based on the provided layouts. In order to perform more seamless noise com- position, Multi-diffusion (Bar-Tal et al., 2023) blends the noise by solving an optimization objective with closed-form optimal solution, which ensure the consistency in composed noise map é.
4.3.2 Classifier Free Guidance
performs extrapolation noise blending between the conditional noise prediction and the unconditional noise prediction éy (xe, c) = (1+w)eg (x,, c)- we& (x,) to balance the quality and diversity of synthesized samples. In practice, some works also propose variations of classífier-free guidance for condition incorporation in different conditional synthesis scenarios. Instructpix2pix [Brooks al.) 2202)) and Pairdiffusion (Goel et al. 2023) develop the classifier-free guidance to adjust the conditioning strength for each component in multiple conditional inputs by decomposing the multi--ondiitonall score function. For customization tasks, SINE (Zhang et al. 2023th interpolates the noise prediction on specialized and pre-trained model to obtain conditional noise prediction in classifier- free guidance, which alleviates the overfitting in the specialized model. Null-text Guidance perturbs the classifier-fi free guidance by altering the noise-level in unconditional prediction to smooth out some realistic details and create cartoon-style images. For inversion-basod editing, AIDI (Pan et al. 2023) proposes a blended classifier-fre guidance based on the positive/negative masks indicating the area be edited or preserved, which enables larger guidance scales and ensures more accurate editing results.
As described in Sec 2.4 classifier-free guidance (Ho & Salimans 2022)
23

---

Published in Transactions on Machine Learning Research (05/2025)
4.4 Revising Diffusion Process
Most of in-sampling conditioning mechanisms such as Guidance, Conditional Correction and Attention Ma- nipulation performs modification the standard formulation of the denoising step, which leads to devia- tions from the predetermined sampling trajectory and results in artifacts in synthesized images. Therefore, branch of works prefer to incorporate the condition control signals into the denoising step via revising the formulation of standard diffusion process to adapt the conditional synthesis task (Luo et al. 2023 Yue et al. Kawar et al., 2022 Wang et al. 2024c). Thereby, the conditional control signals can be incorporated into the corresponding reverse diflusion step of the revised diffusion process without deviations from the diffusion formulation.
2024
Based the revision on diffusion process, these works can be divided into two categories: (a) mean- reverling SDEs, which revise the diffusion process to preserve the information in condit ional inputs for image restoration, (b) decomposition- based noise redefinition, which incorporate sequence of additive noises in the sampling process on the spectral space to revise the noise-level mismatch in noisy linear problems.
4.4.1 Mean-Reverting SDEs
In numerous restoration tasks, most structure and semantic features of the target image are provided by the degraded image c. To avoid consuming part of the model capability regenerating these features from pure Gaussian noise, numerous studies design novel diffusion processes in which the diffused output Xy approximales a noisy version of degraded image с instead of pure Gaussian noise (Welker et al. 2024 Luo et al. 2023 Yue et al.) 2024 Wang et al., 2024d Delbracio & Milanfar 2023). IR-SDE (Luo et al. 2023) constructs a set of mean-reverting SDEs identified by degraded image c, which models the diffusion process from clean image x to a Gaussian distribution averaged on degraded image. Subsequently, IR-SDE trains a conditional denoising network to predict the score function in the reversed mean-reverting SDEs to recover the clean image from the noisy degraded image. Similarly, ResShift (Yue et al. 2024) and DriftRec (Welker ct al. 2024 construct an itcrative degradation process from high-resolution image to its corresponding low- reso uutonution image as the diffusion process and train a conditional demoising net work to reverse the degradation
process for super-resolution SinSR (Wang et al. 2024d distills the sampling process of ResShift (Yue et al.
2024), thereby achieving one-step DM-based super-resolution. InDI (Delbracio & Milanfar 2023 constructs a continuous forward degradation process derived from interpolation: x; = (i-t)x+ttc and trains a denoising
network on paired clean /ann/degnn degraded images to predict the clean image xo- Subsequently, image restoration can be performed by reversing the interpolation-basell degradation process with the prediction of this denoising network.
4.4.2 Decomposition- Based Noise Redefinition
Methods in this category construct novel diffusion processes to recover image x from its partial measurement in the noisy linear inverse problems as follows c Hx n, where H is known linear degradation matrix, n A (0, I2I) is an i.i.d. additive Gaussian noise with known variance. In practice, numerous restoration tasks including inpainting, super-resolution and colorization can be written in the form of this noisy linear inverse problems. SVD Decomposition-basee methods firstly perform SVD decomposition on the linear degradation matrix H to decouple the components in the measurement c. Thereby, the components in measurement с on spectral space can be viewed as a noisy version of their counterparts derived from clean image x. order to incorporate the measurement into the diffusion process while preventing the mismatch in noise-level caused by the noise in measurement c, decomposition-based methods design proper noise sequence to link the noise in the measurement with the noise added in the standard diffusion process. It can be proven that the optimized unconditional denoising network pre- trained the prior of the clean image x is also the optimal solution for the variational objective of the designed novel diffusion process. Thereby, we can perform the sampling process in the spectral space to recover clean image x from its noisy counterpart c based on a pre-trained unconditional denoising network. SNIPS (Kawar et al. 2021 and DDRM (Kawar) ct al. p construct SVD decomposition-basse novel diffusion process in the spectral space based on the annealed Langevin dynamics franework provided by NCSN (Song & Ermon 2019) and the Markov chain diffusion process provided by DDPM (Ho et al. 2020) respectively.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Different from SNIPS and DDRM, DDNM Wang et al. 2024c) construct a general solution & based range-null space decomposition which holds H& = c. In each denoising step, DDNM (Wang et al. 2024ch projects the denoising output Xuj: onto the general solution to guarantee the consistency between denoising output Xojz and given mcasurement c. For noisy linear inverse problem y Hx + n, DDNM (Wang et al. 2024cb incorporates a scaling factor into the iormulation of the general solution and designs a noise sequence corresponding to the scaling factor during sampling process to ensure the noise level in x;-1 aligned with the definition of q (x,-1 Xo) for the pre-trained unconditional denoising network.
4.5 Guidance
Sampling from the conditional distribution p(x(c) with diffusion models requires approximating the con- ditional score function Vx, lo (x, c) with a conditional denoising network Ex f, c). sssin practice, guidance provides another pathway to approximate the conditional score function without time-consuming conditional training, since the conditional score function can be decomposed into an unconditional score function and the gradient of log likelihood as follows:
Xx, log (xe c) Vx, log (c x2) + Vx, log (xt),
(11)
where the score function Vx, log (x+) can be estimated by an unconditional denoising network co(x,, t). Guidance-based methods design lask-specific guidance losses to reflect the alignment. between interme- diate latent variable X₁ and conditional inputs с at each time step, which approximates the log likeli- hood logpr (c xt). For multiple conditional inputs, guidance can also be employed to perform condi- tional control for part of the conditional inputs. We can split the conditional inputs с into components co and с₁ which are incorporated into the diffusion synthesis framework with conditional denoising net- work and guidance. In this case, the conditional score function can be written as Vx, log Pt (x, Co, C1) vx, log (ci x,, co) V log (x, co), where Vx, log (x, co) can be estimated by denoising network conditioned on co and log Pa (c1 X,, co) can be estimated by the guidance loss.
Currently, by designing different task-specific guidance losses, guidance- based methods are widely adopted across various conditional synthesis scenarios. Apart from the simplest case discussed in Sec where a classifier is trained conditioned on class Labels(Dharive) & Nichol, 2021). training an accurate classifier becomes challenging when dealing with more complex control signals. To address this, subsoquent works have proposed more flexible guidance losses that do not require explicit training optimization. In the following, we categorize these approaches based on their target applications.
4.5.1 Guidance for Inverse Problems
As mentioned in Sec. 1.4.2 wide range of restoration tasks can be expressed by recovering a clean image x from given partial measure- ment с in the form of noisy inverse problems: c = Ax) + n, n ~ (0;;oā where A a known degradation function and n denotes the additive noise. In practice, approximating the likelihood (c|x;) and performing guidance on the sampling process is a widely employed strat- egy to solve the noisy inverse problem. Fig. g provides an illustration of the sampling process with guidance for the inverse problem.
M2
M2
c= A(x)
M2
M₁
M,
Z0
Reverse diffusion step
Guidance
MCG (Chung al. 2022a) and DPS (Chung et al. 2023b) approximate the gradient of likeli-
Figure 9: An illustration of the guided sampling process for inverse problems. The curve M, denotes the data man- ifold of the intermediate diffuued output x,- The guidance process (red arrow) moves x, towards the data manifold satisfying the constraint c = A (x) (purple line).
hood as: Vx. pe (c) x.) & Vx, p(g(())))))
1 Vx. x. ||c A (xx::) 1|:- The estimation error ob
can be proven to converge to as de - oc in
25

---

Published in Transactions on Machine Learning Research (05/2025)
most inverse problems. IIGDM (Song et al. 2023a provides a more accurate estimation for the likelihood by approximating Ps (Xo xt) with a Gaussian distribution averaged on Xoje-
To perform these guidance approaches for inverse problems on latent diffusion models (Rombach et al, 2022), PSLD (Rout et al.) 2024b) adds an additional guidance term measuring the reconstruction ability of the intermediate denoising output Zoje to avoid guiding the sampling trajectory towards latent variable zo away from the manifold of real data. Resample (Song et al. 2024) introduces stochastic resampling schema that reliably maps the mcasurement-guieee the subsequent sampling step, effectively preventing noisy image reconstructions in latent diffusion models.
intermediate diffuse output Xojo to the latent variable x,-1 1 for
However, these guidance approaches can only estimate the likelihood term in inverse problems with known a concrete form of the degradation operator A(-). This hinders the deployment of these approaches for unknown real-world degradations. BlindDPS (Chung et al., 2023a) explores the applicability of DPS to blind inverse problems, in which the degradation operator Ap(.) (-) paramcterized with unknown paramcter
g.. To identify the degradation parameter along with the sampling process for the desired image, BlindDPS trains diffusion model for the parameters &p in the degradation operator. In the sampling process, BlindDPS employed similar approximation strategy DPS (Chung et al. 2023bb to estimate the likelihood term follows:
Pt (c Xx: 4p)) R p (c Xoje; Poje)
(12)
Subsequently, BlindDPS performs a parallel sampling process to simultaneously recover the clean image x and the unknown degradation parameters yp from the conditional distribution splc) with the estimated likelihood in Eq. 12 GDP (Fe et al. 2023) offers heuristic approximation for the likelihood term, which consists of distance metric measuring the consistency to conditional inputs and an optional quality enhancement loss to control some desired properties in synthesized results. GDP can also be employed in blind inverse problems by optimizing the degradation parameters in the degradation function A with the distance metric during sampling process.
4.5.2 Guidance for Semantic Control
Guidance can also be employed to ensure the consistency to provided semantic control signals, such as text prompts or semantic images, without time-consuming fine-tuning or training. Semantic guidance losses are usually designed based on pre-trained CLIP models which have a rich shared image-text embedding space.
Blend Diffusion (Avrahami et al. 2022. aims to inpaint the masked region cm in source image Ci based on the provided text description Cd. It designs CLIP loss for the conditional inputs c (Cm, Cr Ca) as follows:
I(x,cc c) = DeLir (Noje. c) + XDbg (xooe, c),
(13)
where DELIP measures the CLIP distance between the intermediate denoising output Xojz and text descrip- tion Ca in mask region for semantic alignment, and Dog calculates the MSE and LPIPS similarity between Xoje and source image Cr in unmasked region for the faithfulness to source image. To jointly control the sampling process using both a text prompt and a style reference image, SDG (Liu al.) (023b) employs a linear combination of the CLIP distances between the current denoising output and the embeddings of both the text and the reference image as the guidance loss. In addition, DiffuselT (Kwon & Yel 2023) introduces an auxiliary structure loss computed from the self-attention features of the source image extracted using a Vision Transformer (VIT) (Dosovitskiy et al. 2020), in order to better preserve the structural integrity of the source image.
4.5.3 Guidance for Visual Signals
In practice, branch of works employs guidance to ensure the consistency between the diffused output and the given visual signal. In order to measure the consistency between intermediate diffused output and the provided visual signal, some works train neural networks to project the intermediate diffused output x, onto its corresponding visual signal and leverage distance metric as the guidance loss for sketch-to- image (Vynnov et al., (2023) and stroko-to-image (Singh et al. (002).. Readout Guidance (Luo et al.) (024) provide unified guidance-based framework for diverse visual signals to image task by training various
26

---

Published in Transactions on Machine Learning Research (05/2025)
readout heads to synthesize different task-specific visual feature maps reflecting the spatial layout or inherent correspondence in images to perform guidance. Different from these works, FreeControl (Mo et al. 2024) prefers to impose guidance loss on the difference in the space of PCA components of self-attention map between the intermediate diffused output and visual signal.
4.5.4 Guidance for Attention Layers
In DM-based conditional image synthesis, the attention layers in denoising network effectively control the layout, structure and semantics of synthesized image. However, directly manipulating the attention layers through replacement or localization as described in Section 4.2 introduces artificial modifications to the internal parameters of the denoising network and may impair its modeling capability. Therefore, branch of works employ guidance to achieve soft control for attention layers.
For image editing, attention guidance is performed as a substitution of attention replacement to softly control the consistency between source image and edited result. Pix2Pix- ZexoPix-Zero Zero (Parmar et al.
2023 employs a guid-
ance loss measuring the L2 distance between the cross-attention maps in editing branch and reconstruction branch instcad of the replacement manipulation in Prompt-to-prompp (Hertz et al.
2023 In order to find a
more expressive attention map as a guidance reference, Redilfuser {Lin el al.) 2023 employs a sliding fusion strategy to fuse the auss-attention maps obtained from sampling branches conditioned on source prompt, target prompt and an intermediate representation. EBMs (Park al. 2024) employs an energy function to guide the integration of the semantic information in editorial prompts with the structure and layout of source image restored in cross-attention layers.
Attention guidance can also be employed to perform attention localization. For object-level layout control, Chen et al. (2024b) employs guidance to control the cross-attention map, which locates the objects in text prompts into their desired bounding boxes. Self-guidance (Epstein et al., (024) extracts various characteris- ties including position, size, shape and appearance of the desired object from the intermediate activations and attention maps. Subsequently, Self-guidance places constraints these characteristics with guidance loss measuring their consistency to desired conditional control signal. For based editing tasks which target to move certain foreground contents in source image into target region, Dragondiffusion (Mou et al. 2024a) designs energy functions based on the cosine distance between intermediate features in the U-Net decoder as guidance to ensure correspondence between the original content region and the target dragging region. DiffEditor (Mou et al., (024bb develops the guidance framework of DragonDiffusion (Mou et al. (024a) by introducing SDE-based sampling process on the masked region instead of ODEs improve editing flexibility.
4.5.5 Enhanced Guidance Framework
In some complicated conditional synthesis scenarios, simply incorporating the gradient of guidance loss each denoising step may lead to artifacts and strange behaviors because of the failure in balancing the realness and guidance constraint satisfaction in guided sampling process. Therefore, some state-of-the-art guidance works provide enhanced guidance frameworks to more effectively fuse the prior knowledge in pre-trained models and the information in conditional control signals. FreeDoM (Yu et al. 2023 employs a tive-travel strategy that rolls back the intermediate latent variable x; to a certain previous time step Xt+j and resamples it to time step t again. This strategy inserts additional steps into the guided sampling process, allowing for a more seamless integration of the information from the pre-trained model and the conditional control signals.
In order to enhance the sample consistency conditional control signals, a branch of works (Bansal et al,, 2023 Zhu et al.) 2023 Song et al. 2024 Zhang et al.) 2024a) performs multi-step gradient descent op- timization process instead of the traditional one-step gradient guidance to find the point with minimum guidance loss in the vicinity of the intermediate denoising output Xoj= , and adopts this point to infer the next latent variable X,-1- Universal Guidance (Bansal et al. (023) refers to this enhanced guidance frame- work as backward guidance, and it has successfully generated quality images in tasks such as segmentation, face recognition, object detection, and classifier signals. For inverse problems, DAPS (Zhang et al. 2024a) guides the intermediate denoising output Xoyz toward the conditional distribution p (Xp x,, c) through multi- step MCMC sampling methods (Welling & Teh 2011). DifPPR (Zhu et. al.] 2023 utilizes a Half-Quadratic
27

---

Published in Transactions on Machine Learning Research (05/2025)
Splitting (Geman & Yang 1995 (HQS) optimization process optimization-basel guidance ensure the consistency to the partial measurement c.
Ye et al.) 2024] integrates several classic diffusion guidance approaches (Chung et al.
2023b Yu et al.
2023 Bansal et al.) 2023 He et al.y 2024 Song et al. 2023b) into a unified framework and optimizes the
associated hyper-parameters via an efficient beam search stralegy, leading to enhanced performance in diverse conditional synthesis tasks.
4.6
Conditional Correction
In some conditional synthesis scenarios, the syn- thesized images are controlled by the constraints specified by conditional inputs с (such as the for- mulation of inverse problems). To ensure the synthesized result be consistent to the inputs c, conditional correction-based methods perform correction operator on the intermediate dif- fused output x, (or Xoje), which projects the cur- rent diffused output onto the data manifold satis- fying the constraint imposed by given conditional control signal c. Subsequently, this corrected la- tent variable will be passod into next denoising step, as shown Fig. I0
A(x)
M4
My
M
M
Mo
Reverse siffusionstep
Conditionalcorreccion correction
Figure 10: An illustration of the sampling process with conditional correction operator for inverse problem. The conditional correction process (cyan arrow) projects x; onto the data manifold satisfying the constraint A
Currently, conditional correction are widely em- ployed in image inpainting tasks, which involves synthesizing content for the masked region Cm in incomplete reference image Cy. The constraint
in inpainting tasks can be expressed as: C, = (1 - cm) o x. Pioneer diffusion work SDE (Song et al, 2021b) performs inpainting based on conditional correction by replacing the unmask region in denoising output Xpjt with its counterpart in reference image Cy to ensure the faithfulness to the content in unmasked region. Different from SDE (Song al. 2021b), Repaint (Lugmayr et al. 2022) prefers to perform replace- ment correction latent variable Xt. Besides, Repaint rolls back the intermediate latent variable x, to the previous time step and resamples it back to time step t several times to eliminate the artifacts caused by conditional correction. The constraint in super-resolution task can be written as: c = ΦNX, where denotes the low-rcsolution image of x downsampled by degradation matrix Φx with factor N. ILVR (Choi et al. 2021) performs conditional correction by substituting the low-frequency components in the latent variable with its counterpart noisy low-resolution image to ensure the consistency between degraded latent variable and its counterpart noisy reference low-resolution image. Conditional correction operators are also widely employed in image editing tasks to preserve the background not requiring editing (Couairon et al. 2023
Patashnik et al. 2023 Wang et al. 2023a Lin et al. 2024b Huang et al. 2023b).
Given a background mask in the source image, text-based image editing can be formulated as an image inpainting task, where the masked foreground region is synthesized based on the provided text prompt. However, such foreground or background masks are often unavailable real-world editing scenarios. To address this, line of work proposes automatic mask or segmentation generation by inferring plausible layouts for the user-desired edited image, conditioned on both the source image and the text prompt. DitfEdit (Couairon et al.) 2023) identifies background masks by comparing the denoising outputs of noisy source image when conditioned on the source prompt versus the target prompt. Object-Shape Variation (Patashnik eet al. 2023) segments the source image by clustering attention maps into semantically coherent regions and aligning them with noun tokens in the prompt based on similarity with their corresponding cross-attention maps. Additionally, several approaches (Wang et al.) 2023a] Lin al. (024bb Huang al. (023b) utilize pre-trained image segmentation models automatically generate masks or segmentations, leveraging the structural information from both the source image and the text prompt.
28

---

Published in Transactions on Machine Learning Research (05/2025)
CCDF
(Chung et al. 2022b) proposes general conditional correction formula for constraints in the form of
general noisy linear inverse problem. In practice, the conditional correction operator in (Song et al. 2021b; Lugmayr et al. 2022 Choi et al., 2021) can be expressed in the general form provided by CCDF. Besides, CCDF provides a thcoretical basis for the faithfulness of this corrected sampling trajectory to original sampling process. CCDF proves when the linear degradation operator H is a non-expansive mapping, the upper bound of the deviation in final output Xo will converge to constant as the total diffusion step T + co. MCG (Chung et al.] 2022a) further performs guidance on conditional correction framework provided by CCDF, which alleviates the deviation from original sampling process caused by conditional correction.
5 Challenges and Future Directions
Although DM-based conditional image synthesis has made remarkable progresses in generating high-quality images aligned with various user-provided conditions, there remains a significant disparity between academic advancements and practical needs for conditional image synthesis. In this section, we summarize several main challenges in this field and identify potential solutions to address them in the future.
5.1 Sampling Acceleration
The time-consuming sampling process often creates bottleneck of diffusion based image synthesis, and its acceleration will facilitate the model deployment in practice (Li et al. 2024c Zhao et al.) (023b)). Early works on sampling acceleration are devoted to reducing the number of sampling steps with better numerical solvers (Song et. al. 2021a Lu et al.] 2022a b Zhou et al., 2024a Chen el al.] 2024ab or distilling pre- trained diffusion models to build short-cuts that enable faster sampling (Salimans Ho 2022 Meng et al. 2023 Song et al.) 2023c Zhou et al.] 2024b). However, too few denoising steps in the distiledd models may compromise the ellectiveness of in-sampling condition integration. One feasible solution to first train model to approximate the conditional denoising outputs along the sampling process equipped with in-sampling conditioning mechanisms, and then perform distillation this model (Meng et al. 2023) Another important line of existing works reduces the computational cost of each denoising step by decreasing model parameters using techniques such as knowledge distillation (Chen et al. 2021 2022) and architecture search (Li et al.) 2024c Kim et al. 2023a Zhao et al.) 2023b). Most based parameter compression approaches are currently tailored for text-to-image models. Analyzing whether the parameter redundancy also exists for models of other conditional synthesis tasks, similar to those in text-to-image models, and extending these model compression methods to more complicated downstream tasks, is another promising future direction.
5.2 Artifacts Caused by In-sampling Conditioning Mechanisms
In-sampling conditioning mechanisms summarized in Sec. allow for flexible condition integration in DM- based image synthesis without performing time-consuming condition integration for the denoising network. However, these conditioning mechanisms introduce modification to the standard sampling process in diffusion framework and lead to deviations from the modeled data distribution, which results in artifacts in synthesized
images (Parmar et al. 2023 Lugmayr et al. 2022 Bansal et al. 2023 Yu et al., 2023). The vast majority of works resort to complex adjustment mechanisms to address the artifact issue caused by in-sampling condition integration. This includes time-step rolling back for guidance (Yu et al. 2023) localization for attention maps (Cao et al.) 2023 Lu et al. 2023) and diffusion process revision for restoration tasks (Luo et al.) 2023 Kawar et al. 2022). However, these methods are highly customized based on specific application scenarios. A feasible future direction for developing more generic solution is to perform lightweight. fine-tuning on the denoising network with the diffusion loss based on the intermediate latent variables in the sampling process equipped with in-sampling conditioning mechanisms. This tends to smooth out artifacts under in- sampling conditioning mechanisms and synthesize desired images in lower computational cost compared to performing condition integration in the denoising network.
29

---

Published in Transactions on Machine Learning Research (05/2025)
5.3 Training Datasets
Among the various conditioning mechanisms, the most fundamental and effective pathway for condition integration is still the supervised learning on pairs of conditional input and image. Although training datasets are relatively sufficient for conditional syuthesis tasks involving single-modality conditional inputs, such as text-to-image (Schuhmann et al. 2021 2022), restoration (Agustsson & Timofte
2017 Nah et al.
2017
Karras et. al., 2019), and visual signal to image (Lin et al. 2014 Caesar et al., 2018 Zhou et al. gathering enough data for tasks with complex, multi-modal conditional inputs like image editing,
2017
customization, and composition remains challenging. With the advancement. of training and efficient fine- tuning techniques for large language models, various types of large models are constantly being developed with powerful multi-modal representation learning (Brown et al. 2020 Li et al.) 2022b 2023b) and content generation abilities (Hertz el al. 2023; Tumanyan et al. 2023), making it possible to leverage these pre- trained models to automatically produce desired training datasets. We may also consider self-supervised or weakly supervised learning reduce the demand for large amount of high-quality training data (Zhang
ct al. 2023d
Xie et al. 2023b Zhang et al. 2024c).
5.4
Robustness
Due to the lack of objective task-specific evaluation datasets and metrics some complex tasks, studies for these lasks prefer to compare models based on a set of self-efineed conditional inpuls, making the performance appear overly optimistic. In fact, many renowned text-to-image models (Ramesh al., 2022. Saharia et al. 2022b; Rombach et al., 2022) have been found to produce unsatisfactory synthesized results for certain specilic calegories of text prompts, as demonstrated by the shortcomings of Imagen (Saharia et al. 2022b in generating facial images. Here we point out some pathways to address issues of robustness. First, for conditional inputs where the model performs poorly, augmenting the training dataset is a direct approach. Second, the difficulties in handling conditional inputs in a certain category may be due to the insufficient capability or unsuitability of the conditional encoder for this category of data. In this case, incorporating encoder architectures tailored for this data category into the conditional encoder, or designing more capable compound conditional encoders, becomes preferable choice. Besides, performing specialization for given conditional imputs is also an effective pathway provide robust results at the cost of time-consuming fine- tuning optimization. Finally, sampling process conditioning mechanisms, such as guidance, conditional correction, and attention manipulation, can also be employed to achieve more detailed control and prevent undesired synthesis results.
5.5 Ethic considerations
The developments in Al-generated content (AIGC) propelled by the superior performance of liffusion-baser conditional synthesis and their downstream applications lead to severe ethic considerations in aspects of bias and fairness, copyright, and the risk of exposure to harmful content. Safety-oriented DM-based conditional image synthesis is dedicated to mitigating these issues by embedding watermarks that are easily reproducible in -Mggenarteed images to detect copyright infringement (Yuan et al. 2024 Cui et al. 2023 Wen et al. 2023), and reducing bias by increasing model's orientation towards minority groups in basic unconditional or text-conditioned synthesis via classic conditioning mechanisms, such as fine-tuning (Shen et al.) 2023)
guidance (Um al.) 2024), and conditional correction (Li et al., 2024a). Efforts have also been made in preventing harmful contents in the text-to-image to-image task via harmful prompt detection (Rombach et al. 2022 prompt engineering (Li ct al. 2024a) and safety guidance (Schramowski et al., 2023). The current safety- locused eflorts mainly concentrale on basic unconditional or fext-conditioned synthesis. We believe that for more complex conditional synthesis scenarios, safety-oriented efforts in this area can be focused on four main aspocts: (a) detecting harmful conditional inputs, (b) filtering and removing bias from the training dataset, (c) providing focuse guidance for the sampling process, and (d) implementing safety-focused fine-tuning of the denoising network.
30

---

Published in Transactions on Machine Learning Research (05/2025)
5.6 Unified conditional synthesis framework
While current. research has developed diverse diffusion-based frameworks (Saharia al. 2022c Rombach et al. 2022, Hertz et al. 2023 Li et al. 2023i) specialized for specific conditional synthesis tasks, this often formalizes complex user intents into strict categories, which limits flexibility and case of use amidst the rapidly growing mumber of conditional image synthesis models and tasks. A highly appealing future direction is developing a unified framework that allows users to specify tasks, conditional inputs, and desired outputs flexibly. The recent advent of powerful multimodal large language models (MLLMs) represented by GPT-46 and Gemini 2.0 Flast offers a promising path. Leveraging their strong language understanding and integrated image processing capabilitiess MLLLSs interpret various user intents and potentially unify diverse synthesis tasks within single framework.
However, current MLLMs still exhibit limitations in image generation quality and control, represented by detail inaccuracies in restoration and editing, and difficulties with complex scenes. While the closed nature of SOTA MILMs hinders detailed analysis, we hypothesize these limitations might stem from processing input images by aligning them to a semantic space, polentially neglecting fine-grained local details. Based on the impressive ability of current state-of-the-an MILMs understand user intent and synthesize corresponding images, we believe there is an urgent need to develop powerful open-source MLLM in this field. Additionally, integrating more local details in user-provided images during conversations into the image generation modules of large models is also promising future direction.
6 Conclusion
This survey presents a thorough investigation of DM-based conditional image synthesis, focusing on framework-level construction and common design choices behind various conditional image synthesis prob- lems across seven reprosentative categorics of tasks. Despite the progress made, efforts are still needed in the future to handle challenges in practical applications. Future research should focus on collecting and con- structing high-quality, unbiased, and task-specific datasets, as well as designing effective conditional encoder architectures and in-sampling conditioning mechanisms to enable robust and accurate conditional modeling for generating stable and high-fidelity results. The trade-off between fast sampling and synthesis quality also remains critical challenge for real- world deployment. In addition, exploring unified conditional synthesis frameworks built upon state-of-the-art Multimodal Large Language Models (MLLMs) offers a promising direction, as these models can seamlessly integrate diverse control signals and provide generalizable condi- tioning across a wide range of tasks. Finally, as a popular AIGC technology, it is necessary to fully consider the safety issues and legitimacy brings.
Acknowledgement
Zheyuan Zhan and Can Wang are supported by the National Natural Science Foundation China (No. 62476244), ZJU-China Unicom Digital Security Joint Laboratory and the advanced computing resources provided by the Supercomputing Center of Hangzhou City University. Jian-Ping Mei is supported by the National Natural Science Foundation of China (Grant No. 62276234), and Zhejiang Provincial Natural Science Foundation (Grant No: ZCI.724F0202).
References
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++ How to edit the embedded images? In CVPR, 2020.
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Datasel and study, In CVPR, 2017.
http:///peaal.cm//nee///tttttt careee/mc
https://aistudio.. wow.::uwwww..oooo...o.oooooooee. www.gogggo..ooo//ooop////sttttta
31

---

Published in Transactions on Machine Learning Research (05/2025)
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine; Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXii:2211.01994, 2022.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In CVPR, 2023.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for con- trolled image generation. In ICML, 2023.
Black-Forest.
Flux: Diffusion models for layered image generation., 2024. URL https://githut com/
Dlack-foresttllas/Ehns
Rumeysa Bodur, Erhan Gundogdu, Binod Bhattarai, Tae-Kyun Kim, Michael Donoser, and Loris Bazzani. iedit: Localised text-guided image editing with weak supervision. In CVPR, 2024.
Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinário Passos. Ledits++: Limitless image editing using text-to-image models. Tn CVPR, 2024.
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. In ICLR, 2017.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NewrIPS, 2020.
Holger Cacsar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yihun Xu, Guangyong Chen, Pheng-Ana Ann Heng, and Stan Z Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering, 2024.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng, Masactrl: Tuning- free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. In AAAI, 2021.
Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In CVPR, 2022.
Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of ode-based diffusion sampling. In ICML, 2024a.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2023.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-altention guidance. In WACV, 2024b.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao Anydoor: Zero-shot object-level image customization. In CVPR, 2024c.
Xiangyi Chen and Stéphane Lathuilière. Face aging via diffusion-based editing. Tn BMVC, 2023.
32

---

Published in Transactions on Machine Learning Research (05/2025)
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon. Custom-edit: Text-guided image editing with customized diffusion models. arXi::2355.15779, 2023.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In NewrIPS, 2022a.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come- wwwf-dd--ffffeeeeeee Accelerating conditional diffusion models for inverse problems through stochastic contraction. In CVPR, 2022b.
Hyungjin Chung, Jeongsol Kim, Sehui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems. In CVPR, 2023a.
Hyungjin Chung, Jeongsol Kim, Michael Thompson McCann, Mare Louis Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. In ICLR, 2023b.
Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo. Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer. In CVPR, 2024.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In ICLR, 2023.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffussio models in vision: A survey. IEEE TPAMI, 2023.
Yingqian Cui, Jie Ren, Han Xu, Pengfei He, Hui Liu, Lichao Sun, Yue Xing, and Jiliang Tang. Diffusionshield: A watermark for copyright protection against generative diffusion models. 0ttt:2300.04444. 2023.
Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Jong Chul Ye, Peyman Milanfar, Alexandros G. Dimakis, and Mauricio Delbracio. A survey on diffusion models for inverse problems. arXiv:2410.0000 2024.
Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusson for image restoration. TMLR, 2023.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeirion. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis. In NewrIPS, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-lo-image generation via transformers. In NewrIPS, 2021.
Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han. Prompt luning inversion for lext-driven image editing using diffusion models. In ICCV, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv:2010.1190 2020.
Dave Epstein, Allan Jabri, Ben Poole, Alexci Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In NewrIPS, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-rcsolution image synthesis. In CVPR, 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In ICCV, 2023.
33

---

Published in Transactions on Machine Learning Research (05/2025)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In 1CML, 2024.
Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyuc Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unilied image restoration and enhancement. In CVPR, 2023.
Yutong Feng, Biao Gong, Di Chen, Yujum Shen, Yu Liu, and Jingren Zhou, Ranni: Taming text-to-image diffusion for accurate instruction following. 2xXx:2311.17000 2023.
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction- based image editing via multimodal large language models. arXiv:2309.199 110000.111100 17102, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or An image is worth one word: Personalizing to-t-e generation using textual inversion. In ICLR, 2023a.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain Luning for fast personalization of text-lo-image models. ACM TOG, 2023b.
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin. 2024. URL https:///affuussooollo www............................. gitb...
io/
Donald Geman and Chengda Yang. Nonlincar image recovery with half-quadratic regularization. IEEE TIP, 1995.
Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, et al. Instructdiffusion: A generalist modeling interface for vision tasks. arXiv:2309.0365 2023.
Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with strucurreand-appeaaane paired diffusion models. arXiv:2303. 17546, 2023.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Alexandros Graikos, Srikar Yellapragada, Minh-Quan Le, Saarthak Kapse, Prateek Prasanna, Joel Saltz, and Dimitris Samaras. Learned representation-guuee diffusion models for large-image generation. arXiv:2312.073900 2023.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In NearIPS, 2024.
Qin Guo and Tianwei Lin. Focus your instruction: Fine-grained and multi-instruction image editing by allention modulation. Xiv:2331 10113, 2023.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In ICCV, 2023.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpong Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuming- free real image editing with proximal guidance. In WACV, 2024.
34

---

Published in Transactions on Machine Learning Research (05/2025)
Ayaan Haque, Matthew Tancik, Alexci A. Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In ICCV, 2023.
Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon. Manifold preserving guided diffusion. In ICLR, 2024.
Eric Heitz, Laurent Belcour, and Thomas Chambon. Iterative a-(de) blending: minimalist deterministic diffusion model. In SIGGRAPH, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross-altention control. In ICLR, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207. 200:2207.1298, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistie models. In NewrIPS, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.0200 02303, 2022a.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 2022b.
Jiun Tian Hoe, Xudong Jiang, Chee Seng Chan, Yap-Peng Tan, and Weipeng Hu. Interactdiffusion: Inter- action control in text-to-inage dilfusion models. or arXiv:2312.05849, 2023.
Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In ICCV, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv:2106.0968 2021.
Jiancheng Huang, Yifan Liu, Jin Qin, and Shifeng Chen. Kv inversion: Kv embeddings learning for text- conditioned real image action editing, In PRCV, 2023a.
Mingzhen Huang, Jialing Cai, Shan Jia, Vishnu Lokhande, and Siwei Lyu. Paralleledits: Efficient multi- aspect text-driven image editing with attention grouping. Advonces in Neural Information Processing Systems, 37:22569-225 225955, 2024a.
Nisha Huang, Fan Tang, Weiming Dong, Tong- Yee Lee, and Changsheng Xu. Region-aware diffusion for zero-shot text-driven image editing. arXim:2302.1 1 1797, 2023b.
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-based image editing: A survey, arXiv:2/02. 17225, 2024b.
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, al. Smartedit: Exploring complex instruction-basee image editing with multimodal large language models. arXi::231.006 2023c.
Inbar Hubernaan-Speeglllls,, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv:2304.061. 2023.
Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fino-tuning image customization with text-to-image diffusion models. arXio:2304.0264. 2023.
Hai Jiang, Ao Luo, Haoqiang Fan, Songchen Han, and Shuaicheng Liu. Low-light image enhancement with wavelet-based diffusion models. ACM TOG, 2023a.
35

---

Published in Transactions on Machine Learning Research (05/2025)
Zeyinzi Jiang, Chaojic Mao, Yulin Pan, Zhen Han, and Jingteng Zhang. Scedit: Efficient and controllable image diffusion generation via skip connection editing. arXiv:2312. 1 1392, 2023b.
Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusion-based editing with 3 lines of code. (xXx:2310.01500 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basod generative models. In NewrIPS, 2022.
Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. In ICCV, 2023.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. In NeurIPS, 2021.
Bahjat Kawar, Michael Flad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. NewrIPS, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HHI,T, 2019.
Bo-Kyeong Kim, Hyoung- Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion. arXin:2305. 15798, 2023a.
Jeongho Kim, Guojung Gu, Minho Park, Sunghyun Park, and Jaegul Choo. Stableviton: Learning semantic correspondence with latent diffusion model for virtual try-on. In CVPR, 2024.
Kangycol Kim, Sunghyun Park, Junsoo Lee, and Jacgul Choo. Reference-based image composition with sketch via structure-aware diffusion model. 00X0:2204.09948, 2023b.
Sunwoo Kim, Wooscok Jang, Hyunsu Kim, Junho Kim, Yunjey Choi, Seungryong Kim, and Gaycong Lec. User-friendly image editing with minimal text input: Leveraging captioning and injection techniques. arXiv:2306.0m 022.:7706.2777,, 2023c.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchcad, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023.
Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias NicBner, and Yannick Hold-Gcoffroy. Illumination modeling and control for diffusion models. In CVPR, 2024.
Lightit:
Nilesh Kulkarni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Leonidas Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. In CVPR, 2024.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept cus- tomization of text-to-image diffusion. In CVPR, 2023.
Gihyun Kwon and Jong Chul Ye. Diffusion-baas image translation using disentangled style and content representation. In ICLR, 2023.
Tuomas Kynkäänniemi, Mila Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arx1::2/4.07724, 2024.
36

---

Published in Transactions on Machine Learning Research (05/2025)
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, An- drew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using a generative adversarial network. In CVPR, 2017.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shit Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
Hyunsoo Lee, Minsoo Kang, and Bohyung Han. Conditional score guidance for text-driven imago-to-imago translation. In NewrIPS, 2024.
Dongxu Li, Jumnan Li, and Steven CH Hoi. Blip-diffusion pre-trained subject representation for controllable text-to-image generation and editing. In NeurIPS, 2023a.
Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu. Sell-discovering interpretable diffusson latent directions for responsible text-to-image generation. In CVPR, 2024a.
Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 2022a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML, 2023b.
Pengzhi Li, Qinxuan Huang, Yikang Ding, and Zhiheng Li. Layerdiffusion: Layered controlled image editing with diffusion models. In SIGGRAPH Asia 2023c.
Ruijun Li, Weihua Li, Yi Yang, Hanyu Wei, Jianhua Jiang, and Quan Bai. Swinv2-image Hierarchical vision transformer diffusion models for text-to-image generation. NCA, 2023d.
Shufan Li, Harkanwar Singh, and Aditya Grover. Instructany2pix: Flexible visual editing via multimodal instruction following. arXiv:2312.067 06738, 2023e.
Sijia Li, Chen Chen, and Haonan Lu. Moecontroller: instrucioonbassd arbitrary image manipulation with mixture-of-xxpert controllers. orXiv:2309.00:: 04337, 2023f.
Xiang Li, Soo Min Kwon, Ismail R. Alkhouri, Saiprasad Ravishankar, and Qing Qu. Decoupled data consistency with diffusion purification for image restoration. orXiv:2403.0605, 2024b.
Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, and Zhibo Chen. Dif- fusion models for image restoration and enhancement-a comprehensive survey. arXiv:2308.0930 2023g.
Xiu Li, Michael Kampffmeyer, Xin Dong, Zhenyu Xie, Feida Zhu, Haoye Dong, Xiaodan Liang, al. Warpdiffusion: Efficient diffusion model for high-fidelity virtual try-on. arXiv:231..03667, 2023h.
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: text-to--mage diffusson model on mobile devices within two seconds. NeurIPS, 2024c.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: open-set grounded text-to-image generation. In CVPR, 2023i.
Yunxiang Li, Hua-Chieh Shao, Xiao Liang, Liyuan Chen, Ruiqi Li, Steve Jiang, Jing Wang, and You Zhang, Zero-shot medical image translation via frequency-guided diffusion models. IEEE Trans. Med. Imaging, 2023j.
Haonan Lin, Mengmeng Wang. Yan Chen, Wenbin An, Yuzhe Yao, Guang Dai, Qianying Wang, Yong Liu, and Jingdong Wang. Dreamsalon: A staged diffusion framework for preserving identity-context in editable face generation. arXiv:2403. 19235, 2024a.
37

---

Published in Transactions on Machine Learning Research (05/2025)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrencce Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang, and Ming-Hsuan Yang. Text-driven image editing via learnable regions. In CVPR, 2024b.
Yupei Lin, Sen Zhang, Xiaojun Yang, Xiao Wang, and Yukai Shi. Regeneration learning of diffusion models with rich prompts for zero-shot image translation. wrXiv:2305046511 2023.
Huan Ling, Karsten Kreis, Daiqing Li, Soung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. In NewrIPS, 2021.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Han-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023.
Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing. In CVPR, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrIPS, 2024b.
Jiaming Liu, Rushil Anirudh, Jayaraman J Thiagarajan, Stewart He, K Aditya Mohan, Ulugbek Kamilov, and Hyojin Kim. Dolce: A model-based probabílistic diffusion framework for limited-angle ct reconstruc- tion. In ICCV, 2023a.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In WACV, 2023b.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023c.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Läu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: concept neurons in diffusion models for customized generation. In ICML, 2023d.
Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng. Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones2 2: Customizable image synthesis with multiple subjects. In NewrIPS, 2023e.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv:2211.01000 2022b.
Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Ti-icon: Diflusion-based training- free cross-domain image composition. In ICCV, 2023.
Yanzuo Lu, Manlin Zhang, Andy J Ma, Xiaohua Xie, and Jianhuang Lai. Coarso-to-fine latent diffusion for pose-guided person image synthesis. In CVPR, 2024.
Andrcas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, and Aleksander Holynski. Readout guidance: Learning control from diffusion features. Tn CVPR, 2024.
38

---

Published in Transactions on Machine Learning Research (05/2025)
Ziwei Luo, Fredrik K. Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B. Schön. Image restoration with mean-reverting stochastic differential equations. In ICML, 2023.
Siwei Lyu. Interpretation and generalization of score matching. In UAI, 2009.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to- image generation without test-time fine-tuning. In ACM SIGGRAPH, 2024.
Lars Maalee, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. NeurIPS, 2019.
Shweta Mahajan, Tanzila Rahman, Kwang Moo Yi, and Leonid Sigal. Prompting hard or hardly prompting: Prompt inversion for text-to-image diffusion models. In CVPR, 2024.
Barak Mciri, Dvir Samuel, Nir Darshan, Gal Chochik, Shai Avidan, and Rami Ben-Ari. Fixod- sooed-point inversion for lext-lo-image diffusion models. arXiv:2312. 12540, 2023.
Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022a.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In CVPR, 2023.
Xiangxi Meng, Yuming Gu, Yongsheng Pan, Nizhuan Wang, Peng Xue, Mengkang Lu, Xuming He, Yiqiang Zhan, and Dinggang Shen. A novel unified conditional score-based generative framework for multi-modal medical image completion. wrXiw:2207.0349, 2022b.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXix:1411.178, 2014.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Regative-prompp inversion: Fast image inversion for editing with text-guided diffusion models. N5i:2200516800 2023.
Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. In CVPR, 2024.
Puria Azadi Moghadam, Sanne Van Dalen, Karina C Martin, Jochen Lennerz, Stephen Yip, Hossein Fara- hani, and Ali Bashashati. A morphology focused diffusion probabilistic model for synthesis of histopathol- ogy images. In WACV, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation diffusion models. In ICLR, 2024a.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Diffeditor: Boosting accuracy and ficxibility on diffusion- based image editing. In CVPR, 2024b.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-lo-image dilfusion models. In AAAT, 2024c.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkinn Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text- guided diffusion models. In ICML, 2022.
39

---

Published in Transactions on Machine Learning Research (05/2025)
Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan Li. The blessing of randomness: Sde beats ode in general diffusion-based image editing. In TCLR, 2023.
Bernt Oksendal. Slochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Zhihong Pan, Riccardo Gherardi, Xiufeng Xie, and Stephen Huang. Effective real image editing with accel- erated iterative diffusion inversion. In 1CCV, 2023.
Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye. snergy-based cross attention for bayesian context update text-to-image diffusion models. In NewrIPS, 2024.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH, 2023.
Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Flor, and Daniel Cohen-Or. Localizing object-level shape variations with text-to-image diffusion models. In ICCV, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Ryan Po, Wang Yifan, Vladislav Golyaník, Kfir Aberman, Jonathan T Barron, Amit H Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al. State of the art diffusion models for visual computing, arXim:231000700, 2023.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward meaningful and decodable representation. In CVPR, 2022.
Tianhao Qi, Shancheng Fang, Yanze Wu, Hongtao Xie, Jiawei Liu, Lang Chen, Qian Hc, and Yongdong Zhang. Deadiff: An efficient stylization diffusion model with disentangled representations, In CVPR, 2024.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: unified diffusson model for controllable visual gener- ation in the wild. In NewrIPS, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin RafTel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with CLIP latents. (xxi:2224400122 2022.
Lingmin Ran, Xiaodong Cun, Jia- Wei Liu, Rui Zhao, Song Zijic, Xintao Wang, Jussi Keppo, and Mike Zheng Shou. X-adapter: Adding universal compatibility of plugins for upgraded diffusson model. In CVPR, 2024.
Hareesh Ravi, Sachin Kelkar, Midhun Harikumar, and Ajinkya Kale. Preditor: Text guided image editing
with diffusion prior. arXiv:2302.0700 2023.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lec. Generative adversarial text to image synthesis. In JCML, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Tn CVPR, 2022.
40

---

Published in Transactions on Machine Learning Research (05/2025)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAT, 2015.
Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Semantic image inversion and editing using rectified stochastic diffcrential equations. wrXiv:2410.10990 2024a.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. In NearIPS, 2024b.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine luning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS; unlcashing the diversity of diffusion models through condition-annealed sampling. In ICLR, 2024.
Hshmat Sahak, Daniel Watson, Chitwan Saharia, and David Fleet. Denoising diffusion probabilistic models for robust image super-resolution in the wild. Xiv:223002.00 07864, 2023.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. In NewrIPS, 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-rosolution via itcrative refinement. IEEE TPAMI, 2022c.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR, 2022.
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting, Safe latent diffusion: Mitigat- ing inappropriate degencration in diffusion models. In CVPR, 2023.
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXiv:2111.022 1112111.111..111 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NewrIPS, 2022.
Shuyao Shang, Zhengyang Shan, Guangxing Liu, LunQian Wang, XingHua Wang, Zekai Zhang, and Jinglin Zhang. Resdiff: Combining cnn and diffusion model for image super-resolution. In AAAT, 2024.
David F Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of compu- tation, 1970.
Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. Tn ICLR, 2023.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu odit: Precise image oditing via recognition and generation tasks. In CVPR, 2024.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. In CVPR, 2024a.
41

---

Published in Transactions on Machine Learning Research (05/2025)
Yujum Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. In CVPR, 2024b.
Kaede Shiohara and Toshihiko Yamasaki. Face2diffusion for fast and editable face personalization. In CVPR, 2024.
Takahiro Shirakawa and Seiichi Uchida. Noisecollage: A layout-aware text-to-image diffusion model based noise cropping and merging. In CVPR, 2024.
Xincheng Shuai, Henghui Ding, Xingjun Ma, Rongcheng Tu, Yu-Gang Jiang, and Dacheng Tao. A survey of multimodal-guided image editing with text-to-image diffusion models. arXiv:2406. 14555. 2024.
Jaskirat Singh, Stephen Gould, and Liang Zheng. High-fidelity guided image synthesis with latent diffusion models. In CVPR, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using noncquilibrium thermodynamics. In ICML, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep condi- tional generative models. In NewrIPS, 2015.
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In ICLR, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. In ICLR, 2021a.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Preudoinverse-guidee diffusion models for inverse problems. In ICLR, 2023a.
Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In 1CML, 2023b.
Yang Song and Stefano Ermon. Generative modcling by estimating gradients of the data distribution. In NewrIPS, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. In NewrIPS, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In ICML, 2023c.
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In CVPR, 2023d.
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to- image translation. In ICLR, 2023.
Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv:2205.16009 2022.
Yoad Tewel, Rinon Gal, Dvir Samuel, Yuval Atzmon, Lior Wolf, and Gal Chechik. Add-it: Training-free object insertion in images with pretrained diffusion models. ar Xxi::2411.07255 2024.
Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Monzalez-France Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In CVPR, 2024.
Narck Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and flug-and-play diffusion features for text-driven image-to-image translation. In CVPR, 2023.
42

---

Published in Transactions on Machine Learning Research (05/2025)
Soobin Um, Suhyeon Lee, and Jong Chul Ye. Don't play favorites: Minority guidance for diffusion models. In TCLR, 2024.
Dani Valevski, Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan. Unitune: Text-driven image editing by fine tuning a diffusion model on single image. ACM TOG, 2023.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In NeurIPS, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NewrIPS, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Newral Computation, 2011.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. Tn ACM SIGGRAPH, 2023.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In CVPR, 2023.
Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan. Taming rectified flow for inversion and editing. arXiv:2411.04744, 2024a.
Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka. Instructedit: Improving automatic masks for diflusion-based image editing with user instructions. arXi::2305.88447 2023a.
Su Wang, Chitwan Saharia, Ceslce Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David Fleet, Radu Soricut, al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In CVPR, 2023b.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. 2rXiv:2205.19953 2022a.
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models. arXio:2207. 00050, 2022b.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. In NewrIPS, 2023c.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. In ICCV, pp. 1905-1914, 2021.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit. Girdhar, and Ishan Misra. Instancediiffusion: Instance-level control for image generation. In CVPR, 2024b.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2024c.
Yufei Wang, Wenhan Yang, Xinyuan Chen, Yaohui Wang, Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C Kot, and Bihan Wen. Sinsr: diffusion-based image super-resolutin in a single step. In CVPR, 2024d.
Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In ICCV, 2023d.
Jingxuan Wei, Shiyu Wu, Xin Jiang, and Yequan Wang. Dialogpaint: A dialog- dialog-based image editing model. XXxx:2303.1001 2023a.
43

---

Published in Transactions on Machine Learning Research (05/2025)
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In CVPR, 2023b.
Simon Welker, Henry N Chapman, and Timo Gerkmann. Driftrec: Adapting diffusion models to blind jpeg restoration. IEEE TIP, 2024.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Lise Getoor and Tobias Scheffer (eds.), ICML, 2011.
Yuxin Wen, John Kirchenbaner, Jonas Geiping, and Tom Goldstein. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. arXi::23552000% 2023.
Chen Henry Wu and Fernando De la Torre A latent space of stochastic diffusion models for zero-shot image editing and guidance. In ICCV, 2023.
Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In CVPR, 2023.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcomposer: Tuning- free multi-subject image generation with localized attention. arXio:2305.1043 10431, 2023.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusson model. In CVPR, 2023a.
Shaoan Xie, Yang Zhao, Zhisheng Xiao, Kelvin CK Chan, Yandong Li, Yanwu Xu, Kun Zhang, and Tingbo Hou. Dreaminpainter: Text-guided subject-driven image inpainting with diffusion models. (XXiv:231.003771 2023b.
Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi. Prompt-free diffusion: Taking' text" out of text-to-image dilfusion models. In CVPR, 2024.
Minglong Xue, Jinhong He, Yanyi He, Zhipu Liu, Wenhai Wang, and Mingliang Zhou. Low-light image enhancement via clir-fourier guided wavelet diffusion. arXiv:2401.03788 2024.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar- based image editing with diffusion models. In CVPR, 2023a.
Fci Yang, Shiqi Yang, Muhammad Atif Butt, Joost van de Weijer, et al. Dynamic prompt learning: Ad- dressing cross-altention leakage for text-based image editing, In NewrIPS, 2024a.
Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming- Hsuan Yang. Diffusion-based scene graph to image generation with masked contrastive pre- training. ar 1xXx:2211..1133 2022a.
Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In ACM MM, 2023b.
Xiulong Yang, Sheng-Min Shih, Yinlin Fu, Xiaoting Zhao, and Shihao Ji. Your vit is secretly a hybrid diserininative-geeeriivee diffusion model. arXiv:2008.0700 2022b.
Yifan Yang, Houwen Peng, Yifei Shen, Yuging Yang. Han Hu, Lili Qiu, Hideki Koike, et al. Imagebrush: Learning visual in-context instructions for exemplar-bssed image manipulation. In NewrIPS, 2024b.
Zhen Yang, Ganggui Ding, Wen Wang, Hao Chen, Bohan Zhuang, and Chunhua Shen. Object-aware inversion and reassembly for image editing. In ICLR, 2023c.
Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stefano Ermon. Tfg: Unified training-free guidance for diffusion models. In NeurIPS, 2024.
44

---

Published in Transactions on Machine Learning Research (05/2025)
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXim:2308.067 2023.
Ahmet Burak Yildirim, Vedat Baday, Erkut Erdem, Aykut Erdem, and Aysegul Dundar. Inst-inpaint:
Instructing to remove objects with diffusion models. arXiv:2904.0220 03246, 2023.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training energy-
guided conditional diffusion model. In ICCV, 2023.
Zihan Yuan, Li Li, Zichi Wang, and Xinpeng Zhang. Watermarking for stable diffusion models. 1EEE Internet of Things Journal, 2024.
Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super- resolution by residual shifting. In NewrIPS, 2024.
Sergey Zagoruyko. Wide residual networks. arXiv preprint arXiv:1605.071/6 2016.
Jianhao Zeng, Dan Song, Weizhi Nie, Hongshuo Tian, Tongtong Wang, and An-An Liu. Cat-dm: Controllable accelerated virtual try-on with diffusion model. In CVPR, 2024.
Bingliang Zhang, Wenda Chu, Julius Berner, Chenlin Meng, Anima Anandkumar, and Yang Song. Improving diffusion inverse problem solving with decoupled noise annealing. urXiv:2407.01521, 2024a.
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kwoon. Text-to-image diffusion models in generalive ai: A survey. arXiv:2303.0790 07909, 2023a.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. In NearIPS, 2024b.
Kaiduo Zhang, Muyi Sun, Jianxin Sun, Binghao Zhao, Kunbo Zhang, Zhenan Sun, and Tieniu Tan. Hu- mandiffusion: a coarse-to-fine alignment diffusion framework for controllable text-driven person image generation. arXiv:2211.0625: 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusson models. In ICCV, 2023b.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In ICLR, 2023.
Shiwen Zhang, Shuai Xiao, and Weilin Huang. Forgedit: Text guided image editing via learning and forget- ting. arXit:2309.1053: 2023c.
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. In CVPR, 2024c.
Xin Zhang, Jiaxian Guo, Paul Yoo, Yutaka Matsuo, and Yusuke Iwasawa. Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model. urXie:2306.07500 2023d.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Inversion-based style transfer with diffusion models. In CVPR, 2023e.
Yuzhe Zhang, Jiawei Zhang, Hao Li, Zhouxia Wang, Luwei Hou, Dongqing Zou, and Liheng Bian. Diffusion- based blind text image super-resolution. In CVPR, 2024d.
Zhixing Zhang, Ligong Han, Amab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In CVPR, 2023f.
45

---

Published in Transactions on Machine Learning Research (05/2025)
Zhongping Zhang, Jian Zheng, Zhiyuan Fang, and Bryan A Plummer. Text-to-image editing by image information removal. In WACV, 2024e.
Chen Zhao, Weiling Cai, Chenyu Dong, and Chengwei Hu. Wavelet-basse fourier information interaction with frequency diffusion adjustment for underwater image restoration. In CVPR, 2024a.
Jing Zhao, Heliang Zheng, Chaoyue Wang, Long Lan, and Wenjing Yang. Magicfusion: Boosting text-to-
image generation performance by fusing diffusion models. In ICCV, 2023a.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K
Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. In NewrIPS, 2024b.
Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffussin: Subsecond text-to-image genera- tion mobile devices. arXi::2311116677, 2023b.
Dian Zheng, Xiao-Ming Wu, Shuzhou Yang, Jian Zhang, Jian-Fang Hu, and Wei-Shi Zheng. Selective hourglass mapping for universal image restoration based on diffusion model. In CVPR, 2024a.
Qingping Zheng, Ling Zheng, Yuanfan Guo, Ying Li, Songeen Xu, Jiankang Deng, and Hang Xu. Self- adaptive reality-guided diffusion for artifact-free super-resolution. In CVPR, 2024b.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models around 5 steps. In CVPR, 2024a.
Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, and Siwei Lyu. Simple and fast distillation of diffusion models. In NeurIPS, 2024b.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Lue Van Gool. Denoising diffusion models for plug-and-pla, image restoration. In CVPR Workshops, 2023.
A Appendix
46

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt:
"A photo of Shibo Inu dog with backpack riding bike. It is wearing sunglasses and beach hat"
GLIDE
SD-1.5
SD-2.1
Imagen-3
DALLE-3
Flux
Prompt: "A blue coloured pizza"
CLIDE
SD-1.5
SD-2.1
Imagen-3
DALLE-3
Flux
Prompt: "A photo of confused grizzly bear in calcalus class"
B-S
GLIDE
SD-1.5
SD-2.1
Imagen-3
DALLE-3
Flux
Prompt: AA crayon drawing of space elevator" "
GLIDE
SD-1.5
SD-2.1
Imagen-3
DALLE-3
Flux
Prompt: "A slained glass window of panda ealing bambooo
GLIDE
SD-1.5
SD-2.1
Imagen-3
DALLE-3
Flux
Prompt: "A fog rolling inlo new york"
GLIDE
SD-15
SD-2.1
Imagen-3
DALLE-3
Flux
Figure 11: Visual comparison of the classic diffusion-based works for text-to-image. including GLIDE NNichol
et al. 2022). Stable Diffusion (Rombach et al. 2022 Imagen (Ho et al.) 2022a). DALLE (Ramesh et al.
2022
and Flux (Black-Forest 2024
47

---

Published in Transactions on Machine Learning Research (05/2025)
Sourse
SR3
SRDiff
DDRM
DPS
Sourse
SR3
SRDif
DDRM
DPS
Sourse
SR3
SRDiE
DDRM
DPS
Sourse
SR3
SRDif
DDRM
DPS
Sourse
SRDif
DDRM
DPS
Sourse
SR3
SRDIff
DDRM
DPS
Figure 12: Visual comparison of the classic diffusion- works for super-resolution restoration, including
SR3 (Saharia et al. 2022cl), SRDiff [Li et al. 2022a DDRM Kawar et al. 2022 and DPS (Chung et al. 2023b).
48

---

Published in Transactions on Machine Learning Research (05/2025)
Edit: "Add a cot"
Source
Prompt-to-prome prompt
Plug-and-Play
Masactrl
Imagic
RF-edit
Edit:
"Change black chothes to white clothes"
Source
trompt-to-prompt
Plug-and-
Masactrl
Imagic
RF-edit
Edit: "Add red sun in the sky
Source
trompt-to-prompt
Plug-and-Play
Masactrl
Imagic
RF-edit
Edit: "Change sitting to running" "
Source
toompttto-prompt
Plug-and-Play
Masactrl
Imagic
RF-eddi
Edit: "Change horse to Wooden horse"
Source
Prompttto-prompt
Plug-aud-Play
Masactrl
Imagic
RF-cdit
Edit: "Hold an opple" "
Source
Prompttto-prompt
Plug-and-Play
Masactrl
Imagic
RF-odit
Figure 13: Visual comparison of the classic diffusion- based works image editing, including Prompt-to-
prompt (Hertz al. 2023), ; Plug-and-play (Tumanyan et al. 2023), Masactrl (Cao et al., 2023). Imagi
Kawar al. 2023), and RF-edit (Wang et al. 2024a).
49

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt:
"A cute cat, high quality, extremely detailed"
Visual Signal
Readout Guidance
121-Adapter
X-adapter
ControlNet
Prompt:
"Flower, best qualily, extremely "alailed" xM
Source
Visual Signal
Readout Guidance
121-Adapter
X-adapter
ControlNet
Prompt: "Professional headshol of a woman with the eiffel lower in the background"
Source
Visual Signal
Readout Guidance
T21 121-Aer
X-adapter
ControlNet
Prompt: "A beautiful living room
Visual Signal
Readout Guidance
T2I-Adapter
X-adapter
ControlNet
Figure 14: Visual comparison of the classic diffusion-based works for depth to image (visual signal-to-image) including Readout Guidance (Luo et al. 2024), T2I-Adapter (Mou et al. 2024ch, X-adapter [Ran et al. 2024
and ControlNet (Zhang et al. 2023b).
Prompt: "A teddybear wearing a fedora cap"
Source
Textual inversion
Dreambooth
Custom Diffusion
BLIP Diffusion
Prompt:
"A dog in a swimming pool"
Source
Textual inversion
Dreambooth
Custom Diffusion
BLIP Diffusion
Figure Visual comparison of the classic diffusion-based works for Customization, including Textual
Inversion Gaal al. 2023a). DreamBooth (Ruiz al.) 2023), Custom Diffusion (Kumari al. 2023). and
BLIP Diffusion (Li al.] 2023a
50

---

Published in Transactions on Machine Learning Research (05/2025)
FILA
FILA
2
3
i prodoomnnt phoogngh of AgM
prodouiial photgreph c300
of paawing o' hamberger Fax cal lying down
ol passer of wile, Fax Cogat cal on skateboard
cal riding bike
4
Figure 16: Visual comparison of the classic diffusion-baace based works for Customization, including 1. PЫE Zhang et al. 2023d), DreamInpainter (Xie et al. 2023b 3. TF-ICON [Lu et al. 2023 Anydoor
Chen et al. 2024ch.
51

---

Published in Transactions on Machine Learning Research (05/2025)
BAIGEN-sample
GLIGEN-sample2
GLIGEN-sample3
Caption: "Michael Jackson in a black cloth is singing into microphone" Grounded text: Michael Jackson, black cloth, microphone
Interaction Input
Generated Image
Interaction Input
Generated Image
holding
Cullking
2
Feeding
person
another
person
person
Caption: person is holding bag, another person is talking
Caption: A person is holding a bag, another person is talking on phone.
Figure 17: Visual comparison of the classic diffusion-basse works for Customization, including 1. GLIGEN
et al. 2023i 2. InteractDiffusion (Hoe et al. 2023).
52

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: The corresponding works for the various stacks of conditioning mechanisms shown in Tab
Stack of conditioning mechanisms for denoising network
Serial Number
Model
DN1
Rombach al.
(202
Saharia
2022b
Nichol
(2022)
Ramesh al.
2022
[222g
Balaji
[022
DN2
Saharia al.
2022c Ho (2022b):
al.
2022a
DN3
hang et al.
(0224)
Zhao et al.
(2024a)
Jiang et al. (2023a
Zheng et al.
2024a b
Zhang al.
(2024d))
Xue et al.
2024
DN4
Fu et al.
(2023)
Huang et al.
(2033 Feng et al.
(003)
Ti et. al.
(003ee
Brooks et al.
(2023: Zhang et al.
[224cc): Vildirim et al.
[2023
Zhang al. (002hh
DN5
Gong et al [2023]
Sheymin et al. [2024]]
ot al
20231
DN6
Kawar et. al.
2023]: Wu et al
[2023]: Zhang et al.
20/3cc
Mou et al.
(2024))
Mahajan et al.
2024 :
Favi el al. (2025)
Bodur et al. 22244]
Li al.
2023c
Zhang et al. 20230
DN7
iao cl.
(2023 Ma et al.
(20024) Shi et (2024a) Gal al. (2023b)
Jia cl
(2023]
Lu al. (2024) [Shiohara & Yammssa (2024
DN8
Ruiz al.
(2023): Gal
(2023 Kumari
(2023
Cu ct al.
2024
: Liu ot al.
2023d cf Han al. [2023
DN9
Mou et: al.
2024c)
Zhang et al.
(203h
d Goel et al.
(0223
Qin et
2023): : Yang et ml.
20236
Zhha el al.
2024b
g Ran et ml.
2024
Jiag el
2023b
DN10
Wang ot al (202aa
Xu al.
[2044]
DN11
T.i et al.
(2023hg
Zeng al.
2024];
Kim et al.
1202
DN12
Yang et al.
20223 8
Xie ct al.
(2028a
Wang ct al. (2023b)
Xic et al
2023b
Song et al.
(202d
Kim et al
2023b
Chen et al
2024cc
DN13
1. et al.
2023
Hoe et al.
2023 Wang et al.
2024b)
Stack of conditioning mechanisms for sampling process
Serial Number
Model
SP1
Tian et al.
2024 Liu et al. (2023))
SP2
Luo al.
2023
Yue al. 2024
Welker et al.
2024
Wang al.
2024d
Delbracio & Milanfar
(003))
SP3
Kawar et
2021
2022;;
Wang et
2024c
SP4
Avrahami
al.
2022
Chung al
2022a bs
SSng et al.
(023a))
Chung el.
2023a
Fei et
2023
Rout et
2024b)
SP5
Lugmayr et al.
2022
Choi et al.
2021
Chung al.
2022b
et al.
(2023) Meng et al. [2022a] : Mokady at al.
2023
Dong al.
(2023);
Wang al.
(202d
SP6
Wallace el al.
2023
Miyake ef al. (2023): Ju et al.
(2023]
Meiri et al.
(2023)
Brack et al.
2024 Wu
De la Torre (2023)
Huberman-Sy S-eernann-Sppiieeee glas et al.
22023
| Nie et al.
(2023]
Zhang el al.
2023e
SP7
Conatron et 2023
Yang
2073c
Patashhik
2023
Wang
(2223
Lee ct
2024
H
Huang
[2033b]
Hertz
2023
Tumanyan al.
(2023
Cao al. (21233
SP8
2023
Ching
2024
& Lin
2023
SP9
al.
2023
: Mou al. 20244b Lin
2023):
Park
2024
SP10
Voynov et al.
10023
Singh et al. (2023) Luo et al.
2024)
Mo et al.
2024
SP11
Zhao et
2023a
Shirakawa & Uchida (2024
Bar-Th Tal al. 2023
SP12
Cao et all
(203
Patashnik et al.
2023
Lu et al. 2023
Balaji et. al.
2022
g Lai et al. 200
Guo & Lim 2023)
SP13
Chen et al.
2024b
Epstein et al.
(2024)
Mou et al. 2024a
SP14
Liu
2022
SP15
Ho & Salimans
2022
Sadat et al.
(2024)
Kynkäänniemi et
2024
SP16
Yu al.
(2023
Bansal al.
2023)
53


Reviews:
[{"review_id": "86ZeLNwIk1", "reviewer": "Reviewer_RGBk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The survey is very comprehensive and covers a large number of methods categorized into denoising network-based or sampling-based.\n- The taxonomy choice is clear and concise and describes well the different types of approaches for conditional image synthesis using diffusion models\n- The challenges and future directions section is insightful and concretely presents several fruitful research directions.\n- The paper is generally easy to follow and is well-written and organized.\n- Table 1 is very well formatted and makes clear the conditional mechanisms within each category.\n- Rather than just focusing on a subset of conditional models such as text-to-image diffusion, this survey covers general conditioning mechanisms.\n\n## Weaknesses\n- Although the categorization is done well, given the large number of methods discussed, there is no high-level discussion about the comparisons between the methods and the pros/cons of various methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper surveys the field of conditional image synthesis using diffusion models. The survey categorizes methods into those that condition using the denoising network and those that condition at sampling time. Within each category, the authors subcategorize methods based on training, repurposing, and specialization for denoising network and different condition integration mechanisms for sampling methods. Finally, the authors discuss challenges and future directions in the field including improving efficiency, robustness, and safety in conditional image generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be beneficial to include a high-level discussion of the pros/cons of the subcategories of methods.\n- The citation format is inconsistent throughout the paper. Please use the commands \\citet and \\citep properly, e.g. \"\\citet{} proposes X\" and \"Recent works have studied conditional image synthesis using diffusion models \\citep{}\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JWuvMH2eUA", "reviewer": "Reviewer_U34P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. **Taxonomy:** The paper is generally well-structured, covers a wide range of relevant papers, and categorizes them into clear taxonomies, which allows the reader to quickly navigate the growing literature. Moreover, the figures effectively illustrate different conditioning mechanisms. \n2. **Applications:** Different from existing surveys, the material goes beyond text-to-image generation and covers a wide range of image synthesis tasks such as image restoration, image composition, and layout control. \n3. **Future directions:** The survey provides an insightful outlook on open challenges and future directions in the field of conditional image synthesis with diffusion models. \n\n### Weaknesses\n\n1. **Comparisons:** The paper provides a great overview and *taxonomy* of methods. However, for a survey, it would also be good to provide further *comparisons*:\n    - High-level (e.g., tabular) comparison of different approaches to understand their characteristics, e.g., advantages & drawbacks, applicability, computational costs, guarantees, etc.\n    - While the main scope of the paper is conditional image synthesis, it would still be interesting to categorize which methods are tailored to image domains and which ones are more generally applicable.\n    - Comparison of performance and effectiveness for different (real-world) applications. While I think that an empirical evaluation is out of scope for the current paper, one could still provide a more qualitative comparison.\n2. **Related works:** Please find below some (recent) works which could be added and discussed:\n    - TFG: Unified Training-Free Guidance for Diffusion Models (https://arxiv.org/pdf/2409.15761)\n    - A Survey on Diffusion Models for Inverse Problems (https://arxiv.org/pdf/2410.00083)\n    - Extending RePaint, it would be interesting to mention further resampling-based methods; see, e.g., https://arxiv.org/abs/2305.08995, https://arxiv.org/abs/2307.08123, https://arxiv.org/abs/2403.06054, and https://arxiv.org/abs/2407.01521.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive survey of diffusion models for conditional image synthesis. After giving a short background on diffusion models, the paper provides a taxonomy of techniques based on the employed conditioning mechanisms, breaking it down into two fundamental categories:\n\n1. **Integration in the network:** Methods that incorporate conditioning into the denoising network during training, re-purposing, or specialization.\n2. **Integration in the sampling:** Methods that incorporate conditioning into the sampling process, including inversion, guidance, or attention manipulation.\n\nRepresentative approaches are described in more detail, allowing to categorize and compare the rapidly increasing number of methods that have been developed in recent years. The paper concludes by highlighting open challenges and future directions in the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in “weaknesses” above, my main suggestion would be to add further comparisons (beyond the taxonomy) to help the reader navigate the existing literature.\n\n**Minor:**\n\n- For most citations, `\\citep` should be used instead of `\\citet`\n- `T2I` should be defined on p. 2\n- It should be made clear that $\\epsilon_\\theta$ is defined to be the “noise-prediction” network and not the “denoising network” (i.e., predicting $x_0$) as mentioned on p. 4.\n\n**Typos:**\n\n- “ as describes in Sec. 3.2.2” → described\n- “as tje encoder”\n- \"sampling process iteratively reserve noisy latent variable into desired image”\n- “Noise-ddding inversion”\n- “error accumulation .” (whitespace)", "broader_impact_concerns": "One could add a paragraph on ethical considerations (e.g., bias mitigation strategies, safeguards against misuse, and potential risks of AI-generated content).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MOjG2rw3ee", "reviewer": "Reviewer_UXiJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* Overall, the motivation for this survey is clear and it presents an organized taxonomy while distinguishing itself from previous surveys.\n* Covers many recent works which could serve as reference for future work on conditional image synthesis.\n\n**Weaknesses**:\n* In general, diffusion models work with various deformations, not only Gaussian noise ([1], [2]) as mentioned throughout the manuscript (e.g., page 1 - second paragraph, Section 2 - first paragraph). It is worth noting that while most diffusion models indeed utilize Gaussian noise, it is possible to learn a mapping from *some* distribution to the real data distribution with a diffusion process.\n* Figure 2 is not entirely clear – I understand what the authors try to convey, but I don’t think the figure is successful in doing that. Perhaps more explanation/legend on the colors of the arrows and what they mean. I believe this figure needs to be reworked.\n* Something is weird with the citation format, e.g., in Figure 1 (there is only a space between the method’s name and the authors) and page 2 - second paragraph. I think `\\citep’ is not defined or something like that. Another example is the beginning of Section 2.1.2. Page 4 - “or cross-attention layersRombach et al. (2022)”.\n* Writing could be improved (proofreading, see “Minor”). It really feels like the authors didn’t perform a final pass over the writing after the initial version.\n* In page 8, you mention “and subsequently enlarge the resolution of the synthesized results.”. I believe you can  spare a few words on how this resolution enlargement happens (e.g., super-resolution pre-trained networks).\n* I think the survey is a bit outdated, architecture-wise, as many modern models shifted to Diffusion Transformers (DiTs) and Adaptive Layer Normalization (AdaLN) conditioning.\n* Section 4.1.1 writing: I found this section particularly hard to read; repeated usage of the word “However” (even to start sentences), arbitrary claims such as “which always leads to unsatisfied results especially under classifier-free guidance.” (“always”? I think it is a bit strong), and repeated usage of the term “classifier-free guidance”, which is common for those familiar with the literature, but not defined up to this point (only described later in Section 4.3.2). The differences between deterministic and stochastic inversion are not clear.\n* Section 4.2 requires proofreading.\n* Section 4.3.2 refers the reader to Section 4.5, which comes later in the paper, to understand “traditional guidance”. Given my comment above regarding guidance, perhaps a reordering of the sections is due.\n\n[1] Bansal, Arpit, et al. \"Cold diffusion: Inverting arbitrary image transforms without noise.\" Advances in Neural Information Processing Systems 36 (2024). (https://arxiv.org/abs/2208.09392)\n\n[2] Heitz, Eric, Laurent Belcour, and Thomas Chambon. \"Iterative α-(de) blending: A minimalist deterministic diffusion model.\" ACM SIGGRAPH 2023 Conference Proceedings. 2023. (https://arxiv.org/abs/2305.03486)\n\n[3] Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. (https://arxiv.org/abs/2212.09748 )\n\n**Minor**:\n* Page 4, first paragraph, just before Section 2.1.2, for $\\sigma_t$, is the relationship between the two square-roots multiplication?\n* Since a large portion of the text is focused on the U-Net architecture, and this is an introductory survey, perhaps it will benefit from a figure of this architecture.\n* Page 7, “The main challenge in modeling a effective text-to-image” -> “an effective”, “in (a) precisely capture the users’ intention described in text prompts and (b) build “ -> “capturing”, “building”. “in acceptable computational cost” -> “at acceptable computational costs”. “base on Transformer encoder” -> “based”.\n* Page 10, “ as tje encoder for visual signals,” -> “the”.\n* Page 11, “ some works design developed task-specific attention modules for condition injection” -> choose between “designed” and “developed”.\n* Page 11, what is “a UniFusion block”?\n* Page 12, “However, for conditional inputs contain multimodal components or intricate semantics” -> “inputs that contain”.\n* Page 12, “Base on the fine-tuning strategy,” -> “Based”.\n* Page 19, “ subsequently composing these noise to” -> “noises”.\n* Page 21, “aligned with the definiation of” -> “definition”\n* Page 22, “which are incorporate into “ -> “incorporated”, “and perform” -> “performing”\n* Page 23, fix the sentence: “In practice, a branch of works employ guidance to control the consistency between diffuse output and given visual signal” (“employs”, “diffused output/diffusion outputs” [what is “diffuse output”?], “a/the given visual signal”). “diverse visual signal” -> “signals”.\n* Page 24, “attention map as guide reference” -> “as a guidance”, “variable will be pass” -> “passed”.\n* Page 25, “The constrain “ -> “constraint”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a survey paper encompassing early and recent works on conditional image generation with diffusion models. The survey categorizes, organizes the taxonomy and overviews the different conditioning mechanisms, with two main branches– the network itself and the sampling process. The survey also covers the various applications of these conditioning mechanisms and highlights open questions for future research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please proofread and fix all the grammatical errors in the text. I pointed out some of them under “Minor”. You can try pasting the text in Word or Google Docs to quickly locate obvious mistakes, but some mistakes require actually reading the paper.\n* See “Weaknesses” and “Minor” above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is a survey of conditional image synthesis with diffusion models. It introduces a taxonomy on such models. The reviewers agree that the paper is well-written, clearly organized and comprehensive.

The paper certifies as a Survey, as it surveys conditional image synthesis with diffusion models in a structured and comprehensive way. The authors should submit a camera ready version where changes are not indicated by color.
Decision: Accept (poster)