Cross-Modal Contextualized Diffusion Models for Text-Guided Visual Generation and Editing
Authors: Ling Yang
Zhilong Zhang
Zhaochen Yu
Jingwei Liu
Minkai Xu
Stefano Ermon
Bin CUI
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Model
Text-to-Image Generation
Text-to-Video Editing
Primary area: generative models
TL;DR: We propose a new conditional diffusion model to achieve SOTA performance on both text-guided image generation and video editing.

Abstract: Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. Nevertheless, prevailing text-guided visual diffusion models primarily focus on incorporating text-visual relationships exclusively into the reverse process, often disregarding their relevance in the forward process. This inconsistency between forward and reverse processes may
limit the precise conveyance of textual semantics in visual synthesis results. To address this issue, we propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample into forward and reverse processes. We propagate this context to all timesteps in the two processes to adapt their trajectories, thereby facilitating cross-modal conditional modeling. We generalize our contextualized diffusion to both DDPMs and DDIMs with theoretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing. In each task, our ContextDiff achieves new state-of-the-art performance, significantly enhancing the semantic alignment between text condition and generated samples, as evidenced by quantitative and qualitative evaluations. Our code is available at https://github.com/YangLing0818/ContextDiff

Published as a conference paper at ICLR 2024
CROSS-MODAL CONTEXTUALIZED DIFFUSION MOD- ELS FOR TEXT-GUIDED VISUAL GENERATION AND EDITING
Ling Yang!+ Zhilong Zhang Stefano Ermon² Bin Cui
Zhaochen Yu-
Jingwei Liu
Minkai Xu²
!Peking University Stanford University
(yangling0818, bityzcedu, jingweiliu1996)e1s com
(minkai, ermon]êcs ammon)@cssstanfordd edu, fazl2012maaaa bin. cui edu
ABSTRACT
Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. Nevertheless, prevailing text text-guided vi- sual diffusion models primarily focus on incorporating text-visual relationships exclusively into the reverse process, often disregarding their relevance in the for- ward process. This inconsistency between forward and reverse processes may limit the precise conveyance of textual semanties in visual synthesis results. To address this issue, we propose a novel and general contextualized diffusion model (CONTEXTDuFF by incorporating the moosl context encompassing interac- tions and alignments between text condition and visual sample into forward and reverse processes. We propagate this context all timesteps in the processes to adapt their trajectories, thereby facilitating cross-modal conditional modeling. We generalize our contextualized diffusion both DDPMs and DDIMs with the- oretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing. In each task, our CONTEXTDIFF achieves new state-of-the-an ahe performance, signif- icantly enhancing the semantic alignment between text condition and generated samples, as evidenced by quantitative and qualitative evaluations. Our code is available htts://github..om// Yangl Ling08 8/ContextDDif
INTRODUCTION
Diffusion models (Yang al., 2023b) have made remarkable progress in visual generation and editing. They are first introduced by Sohl-Dickstein et al. (2015) and then improved by Song & Ermon (2019) and Ho et al. (2020), and can now generate samples with unprecedented quality and diversity (Rombach et al., 2022; Yang et al., 2023a: Podell et al., 2023; Yang et al., 2024a). As data, CLIP latent space (Radford al., 2021) is
a powerful representation space for widely used by diffusion models to semantically modify images/videos by moving in the direction of any encoded text condition for controllable text-guided visual synthesis (Yang el., 2024b; Zhang et al., 2024; Ramesh et al., 2022: Saharia et al., 2022b; Wu et al., 2022; Khachatryan et al., 2023).
Generally, text-guided visual diffusion models gradually disrupt visual input by adding noise through fixed forward process, and learn its reverse process generate samples from noise in a denoising way by incorporating clip text embedding. For example, text-to-image diffusion models usually estimate the similarity between text and noisy data to guide pretrained unconditional DDPMs (Dhariwal Nichol, 2021; Nichol et al., 2022a), or directly train a conditional DDPM from scratch by incorporating text into the function approximator of the reverse process (Rombach et al., 2022; Ramesh et al., 2022). Text-to-video diffusion models mainly build upon pretrained DDPMs, and ex- tend them with designed temporal modules (e.g., spatio-temporal attention) and DDIM Song et al. (2020a) inversion for both temporal and structural consistency (Wu et al., 2022; Qi et al., 2023).
"Contributed equally. 'Coresponding authors.

---

Published as a conference paper at ICLR 2024
Forward Process
Forward Process
Xg text
toxt
Xt-1
X
Xt+1
Xt-1
Xt
x++1
xr+1
Xt-1
Rt
k++
+text
+text
Aoicet Heed
Reverse Process
Roje
Rojs+1
Reverse Process
(a) Conventional forward and reverse diffusion processes (b) Contextualized forward and reverse diftusion processes
simplified illustration of text-guides visual diffusion models with (a) conventional for- ward and reverse diffusion processes, (b) our contextualized forward and reverse diffusion processes. Top denotes the estimation of visual sample by the denoising network each timestep.
Figure I:
Despite all this progress, there are common limitations in the majority of existing text-guided vi- sual diffusion models. They typically employ an unconditional forward process but rely a text- conditional reverse process for denoising and sample generation. This inconsistency in the uti- lization of text condition between forward and reverse processes would constrain the potential of conditional diffusion models. Furthermore, they usually neglect the cross-modal context, which encompasses the interaction and aligmment between textual and visual modalities in the diffusion process, which may limit the precise expression of textual semantics visual synthesis results.
To address these limitations, propose novel and general cross-modal contextualized diffusion model (CONTEXTDIFF) that harnesses cross-modal context to facilitate the learning capacity of cross-modal diffusion models. As illustrated in Figure 1, compare our contextualized diffusion models with conventional text-guided diffusion models. We incorporate the cross- interactions between text condition and image/video sample into the forward process, serving as a context-aware adapter to optimize diffusion trajectories. Furthermore, to facilitate the conditional modeling in the reverse process and align it with the adapted forward process, we also use the context-aware adapter to adapt the sampling trajectories. In contrast to traditional textual guidance employed for visual sampling process (Rombach et al., 2022; Saharia al., 2022h), our CONTEXTDIFF offers a dis- tinct approach by providing enhanced and contextually informed guidance for visual sampling. We generalize our contextualized diffusion to both DDPMs and DDIMs for benefiting both cross-modal generation and editing tasks, and provide detailed theoretical derivations. We demonstrate the effec- tiveness of our CONTEXTDIFF generation and text-to- text-to-video editing. Empirical results reveal that our contextualized diffusion mod- els can consistently improve the semantic alignment between text conditions and synthesis results over existing diffusion models in both tasks.
two challenging text-guided visual synthesis tasks: text-to-video
To summarize, we have made the following contributions: (i) To the best of our knowledge, We for the first time propose CONTEXTDIFF to consider cross-modal interactions context-aware trajec- tory adapter to contextualize both forward and sampling processes in text-guided visual diffusion models. (ii) We generalize our contextualized diffusion to DDPMs and DDIMs with thereotical derivations for benefiting both cross-modal visual generation and editing tasks. (iii) Our CoN- TEXTDIFF achieves new state-of-the of-the-art performance on text-to-image generation and text-to-video editing tasks, consistently demonstrating the superiority of our CONTEXTDIFF over existing diffu- sion models with both quantitative and qualitative comparisons.
RELATED WORK
Text-Guided Visual Diffiusion Models Text-to-image diffusion models (Yang et al., 2023a; Podell al., 2023) mainly incorporate the text semantics into the image sampling process (Nichol al., 2022a) for cross-modal comprehension. Latent Diffusion Models (LDMs) (Rombach et al., 2022) apply diffusion models on the latent space powerful pretrained autoencoders for high-resolution synthesis. RPG (Yang et al., 2024b) proposes a LLM-grounded prompt decomposition and utilizes the multimodal chain -fouuht reasoning ability of MLLMS to enable complex/compositionn im- age generation. Regarding text-to- text-to-video diffusion models, recent methods mainly leverage the pre- trained text-to-ing to-image diffusion models in zero-shot (Qi et al., 2023; Wang et al., 2023b) and ohe-shot (Wu et al., 2022; Liu et al., 2023) methodologies for text-to-video editing. For example, Tune-A-
2

---

Published as a conference paper at ICLR 2024
Video (Wu et al., 2022) employs DDIM (Song al., 2020a) inversion to provide structural guidance for sampling, and proposes efficient attention tuning for improving temporal consistency. FateZero (Qi et al., 2023) fuses the attention maps in the inversion process and generation process preserve the motion and structure consistency during editing. In this work, we for the first time improve both text-to-image and text-to-video diffusion models with general context-aware trajectory adapter.
Diffusion Trajectory Optimization Our work focuses on optimizing the diffusion trajectories that denotes the distribution of the entire diffusion process. Some methods modify the forward process with a carefully-designe
transition kernel or a new data-dependent initialization distribution (Liu al., 2022: Dockhorn et al., 2021; Lee et al., 2021; Karras al., 2022). For example, Rectified Flow (Liu et al.. 2022) learns a straight path connecting the data distribution and prior distribution. Grad- TTS (Popov et al., 2021) and PriorGrad (Lee et al., 2021) introduce conditional forward process with data-dependent priors for audio diffusion models. Other methods mainly parameterize the forward process with additional neural networks (Zhang & Chen, 2021; Kim et al., 2022: Kingma et al., 2021). VDM (Kingma et al., 2021) parameterizes the noise schedule with monotonic neural network, which is jointly trained with the denoising network. However, these methods only utilize unimodal information in forward process (Yang et al., 2024a), and thus are inadequate for handling complex multimodal synthesis tasks. contrast, our CONTEXTDIFF for the first time incorporates cross-modal context into the diffusion process for improving text-guided visual synthesis, which is more informative and contextual guidance compared to text guidance.
3 PRELIMINARIES
Denoising Diffusson Probabilistic Models Diffusion models (Ho al., 2020; Song et al., 2020b)
consider an unconditional forward process that gradually disturb the data distribution g(0o) into a tractable prior (0, I) with a gaussian kernel defined by (2..,,........ Br) ((-1) = N(1(1- 1,,1), which admits a close form of conditional distribution of æ; given Øn: q 20)) = (Vaaao. a), where ā, [1 - 3:). Then a parameterized Markov chain ( (x:)) )T is trained match the distribution of the reversal of the forward process. The training objective variational bound of the negative log likelihood of the data distribution g(x)):
( Du.
L = Eyllog
pa [oo:T 2 Eg log pe
(3;;-1 (,,,, mo) admits closed form gaussian distribution with the mean determined by xg and wy. then pe (2 (x)) can be parameterized to gaussian kernel which mean is predict by æt.
Denoising Diffusion Implicit Models DDIMs generalize the forward process of DDPMs non- Markovian process with an equivalent objective for training. Deterministic DDIM sampling (Song al., 2020a) is one of ODE-based sampling methods (Lu al., 2022; Song et al., 2020b) generate samples starting from æT ~ NO I) via the following iteration rule:
It-1 / Ct VI - area (Zt, t) + 11-
(2)
Vor
DDIM inversion (Song et al., 2020a) can convert real image zo related inversion noise by reversing the above process, which can be reconstructed by DDIM sampling. It is usually adopted in editing task (Hertz et al., 2023; Mokady et al., 2022: Tumanyan et al., 2022; Qi et al., 2023).
4 METHOD
4.1
CROSS-MODAL CONTEXTUALIZED DIFFUSION
We aim incorporate cross-modal context of each text-image(video) pair (g)) into the diffusion process as in Figure 2. We use clip encoders to extract the embeddings of each pair, and adopt an relational network (e.g., cross attention) model the interactions and alignments between the two modalities as cross- modal context. This context is then propagated to all timesteps of the diffusion process as a bias term (we highlight the critical parts of our CONTEXTDIFF in brown):
go (2x mo0 e) -Nix, Voraro + ( e, (), (1 r).
(3)
3

---

Published as a conference paper at ICLR 2024
C
Prompt: blooning yellow sunflower.
$
Contextualized Diffusion
+
Cross-Modal Context
Relational Network
X t-1
KE
X₁
XT
Video
Adaptation
predict
OR
Cross-Modal Context
Image
Lo
Forward Process
Reverse Process
t
T
Figure 2: Illustration of our CONTEXTDUFF:
where scalar k, control the magnitude of the bias term, and we the kt vār- the relational network with trainable parameters é, it takes the visual sample Do and text condition as inputs and produces the bias with the same dimension as Tn-
ro(-) is
Concretely, the forward process defined as 4s 1 F2 ..., 2r c) = II to (r, 1 To c). Given cross-modal context rs t), the forward transition kernel depends on 24-1, Vo. and
46 (1:1, Zo. c) = (( + o t) varke-17 c, 1), (4)
where 3. = - (tg- This transition kernel gives marginal distribution as Equation (3) (proof in Appendix C. At each timestep we add noise that explicitly biased by the cross-modal context. With Equation (3) and Equation (4), we can derive the posterior distribution of the forward process fort (proof in Appendix C.I):
(0(x:-1 Dr, wo, C) N/( V 1 Do Vae(1 ār-1) (x. bi bi c)) b. b c)2 (1 - 1) I),
(5)
where (300, c) is abbreviation form of krra( C, l), and tion (5), we can simplify the training objective which will be described latter. In this way, we con- textualize the entire diffusion process with a context-aware trajectory adapter. In CONTEXTDIFE, we also utilize our context-aware context to adapt the reverse process of diffusion models, which en- courages to align with the adapted forward process, and facilitates the precise expression of textual semantics in visual sampling process.
for simplicity. With Equa-
4.2 ADAPTING REVERSE PROCESS
We aim to learn a contextualized reverse process (po (at (1,, c) which minimizes varia- tional upper bound of the negative log likelihood, in Equation (1). (0 is gaussian kernel with learnable mean and pre-defined variance. Allowing the forward transition kernel to de- pend on Zy and the objective function Co.s of our CONTEXTDIFF can be formulated as (proof in Appendix C.2y:
Lo.s EEg Eosser (mrrr) mo)r) D (ttleaae)lappppppp//// c)(p((=)(c) - log pe (2ol 1x1, c)
(6)
Σ DKL (1.1111 To c) cl)
where 0 denotes the learnable parameters of denoising network in reverse process. Equation (6) uses KI. divergence directly compare po p0 |at, c) against the adapted forward process posteriors, which are tractable when conditioned on Do and c. If Ts is identically zero, the objective can be viewed as the original DDPMs. Thus CONTEXTDIFF theoretically capable of achieving better likelihood compared to original DDPMs.
Kindly note that optimizing L, Eg. matching the means for (2 |x,,, No, e) and po(a; la,,, c), they gaussian distributions
To. (2 1|,, e)) is equivalent to

---

Published as a conference paper at ICLR 2024
with the same variance. According Equation (5), directly matching the means requires to param- eterize a neural network μsl that not only predicting 2u, but also matching the complex cross-mood cross-modal context information in the forward process, i.c.,
va--18.
Vail xx-1) (xt - bi( c)) b (a0 e)
C.. = (1 - 1 - á, 20
(7)
Simplified Training Objective Directly optimizing this objective is inefficient in practice because it needs to compute the bias twice at each timestep. To simplify the training process, we employ a denoising network fo c, to directly predict zy from øy at each time step t, and insert the predicted Z. in Equation (5), i.e, Pe.s ( c) (0 &o c). Under mild condition, we can derive that the reconstruction objective Eel.ee mol1 is an upper bound of Lr, and thus an upper bound of negative log likehood (proof Appendix C.2). Our simplified training objective is:
Co.s Ez (2 1) m00122
(8)
where A is a weighting scalar. We set KT = O and there is no learnable parameters in ( æo; c)]lp(zr(c)), which can be ignored. To adapt the reverse process at each timestep, we can efficiently sample a noisy sample H, according to Equation (3) using re-parameterization trick, which has included parameterized cross-modal context ( e, (). and then passes æ; into the denoising network. The gradients will be propagated to rs from the denoising network, and our context-aware adapter and denoising network are jointly optimized in training.
Context-Aware Sampling During sampling, we use the denoising network to predict 6o. and the predicted context-aware adaptation ng (#bo C, t.) is then used to contextualize the sampling trajectory. Hence the gaussian kernel po (x,-11 [2;; e) has mean:
v /at-15 2o
Varl ā(-1)
(xt (00 c)) + 1 c)2
where bz c) krro(ăo, c, t), and variance (1-6,-1) I. In this way, our CONTEXTDIFF can 1-a;
effectively adapt sampling process with cross-modal context, which is more informative and con- textual guided compared to traditional text guidance (Rombach et al., 2022: Saharia et al., 2022b). Next, we will introduce how to generalize our contextualized diffusion to DDIMs for fast sampling.
4.3 GENERALIZING CONTEXTUALIZES DIFFUSION TO DDIMS
DDIMs (Song et al., 2020a) accelerate the reverse process of pretrained DDPMs, which are also faced with the inconsistency problem that exists in DDPMs. Therefore, we address this problem by generalizing our contextualized diffusion to DDIMs. Specifically, we define a posterior distribution 9o (x [t, Lo, e) for each timestep, thus the forward diffusion process has the desired distribution:
(x x c) = Nx Vārzo + be c), (1 -a,)
(10)
If the posterior distribution is defined as:
D4 - Vareo ,T
[ To. c) N V(Vā-1100 + V u -o
(11)
v1 Oz
then the mean of qo (3 2o c) (proof Appendix C.I):
V Va-1 + (x c) * v ū1-1 o7
(12)
v1 Cz
V1-a.-1-00
To match the forward diffusion, we need to replace the adaptation karo(Zn, e, () Ir (Zn, C, t 1). Given of = 0, the sampling process becomes deterministic:
with
VI-ac
2t Vā
H;-1 vā + /1 āi-1
V1 Ct
(13)
01-1 + b.
F ã by (wo c)
VI- Ci.
In this way, DDIMs can better convey textual semantics in generated samples when accelerating the sampling of pretrained DDPMs, which will be evaluated in later text-to-video editing task.
5

---

Published as a conference paper at ICLR 2024
LDM
Imagen
Ours
dancing with a white background featuring graffiti of lion."
"A cozy fireplace with crackling flames, bearth and stack logs nearby.'
Italian espresso scientist in latte art in the shape of laboratory, surrounded accompanied by saucer and spoon."
waterfall lush rainforest teeming with vibrant vegetation."
equipment
Figure 3: Qualitative comparison in text-to-image generation. Our model can better express the semantics of the texts marked in blue. We use red boxes highlight critical fine-grained parts where LDM and Imagen fail to align with texts.
critical
EXPERIMENTS
We conduct experiments on two main text-guided visual synthesis tasks: text-to-image generation (Sec. 5.1) and vi--oooviido editing (Sec. 5.2). We also extend our CONTEXTDIFF other condi- tional generation scenarios: class-to-image and layout-to-image (in Appendix B), to demonstrate the generalization ability. For better understanding and explanation of our proposed contextualized diffusion, we further provide some qualitative analysis on FID-CLIP trade-off (Sec. 5.3), model convergence (Sec. 5.3) and heatmap visualization (Appendix A).
5.1 TEXT-TO-IMAGE GENERATION
Datasets and Metrics. Following Rombach et al. (2022); Saharia et al. (2022b), we use public LAION-400M (Schuhmann al., 2021), dataset with filtefiltered 400 million image-text pairs for training CONTEXTDIFF, We conduct evaluations with FID and CL.P score (Hessel et al., 2021; Radford et al., 2021), which aim to assess the generation quality and resulting image-text alignment.
Implementation Details. For our context-aware adapter, use text CLIP and image CLIP (Rad- ford et al., 2021) (VIT-B/32) to encode text and image inputs, and adopt multi-hee head cross attention (Vaswani et al., 2017) to model cross-modal interactions with 8 parallel attention layers. For the diffusion backbone, we mainly follow Imagen (Saharia et al., 2022b) using a 64 x 64 base diffusion model (Nichol & Dhariwal, 2021; Saharia al., 2022a) and super-resolution diffusion models to upsample 64 x 64 generated image into 256 x 256 image. For 64 x 64 + 256 x 256 super-resolution, we use the efficient U-Net model in Imagen for improving memory efficiency. We condition on the entire sequence of text embeddings (Raffel et al., 2020) by adding cross atten- tion (Ramesh et al., 2022) over the text embeddings multiple resolutions. More details about the anper-parametery can be found in Appendix E.
Quantitative and Qualitative Results Following previous works (Rombach et al., 2022: Ramesh et al., 2022; Saharia et al., 2022b), make quantitative evaluations CONTEXTDIFF on the MS- COCO dataset using zero-shot FID score, which measures the quality and diversity generated images. Similar to Rombach al. (2022); Ramesh al. (2022); Saharia et al. (2022b). 30,000 images are randomly selected from the validation set for evaluation. As demonstrated Tab. 1, our CONTEXTDIFF achieves state-of-the-art performance on text-to-image generation task with

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative results in text-to-image generation with FID score on MS-COCO dataset for 256 256 image resolution.
Zero-shot FID-30K
Approach
Model Type
FID-30K
DF-GAN (Tao et al., 2022)
GAN
21.42
DM-GAN CL (Ye al.. 2021)
GAN
20.79
LAFITE (Zhou et al., 2022)
GAN
8.12
Make-A-Seee A-Seene (Gafni ct al., 2022)
Autoregressive
7,55
DALL-E (Ramesh et al., 2021)
Autoregressive
17.89
Stable Diffusion (Rombach et al., 2022)
Continuous Diffusion
12.63
GLIDE (Nichol et al., 2022b)
Continuous Diffusion
12.24
DALL-E 2 (Ramesh al., 2022)
Continuous Diffusion
10.39
Improved VQ-Diffusion (Tang al., 2022)
Discrete Diffusion
8.44
Simple Diffusion (Hoogeboom et al., 2023)
Continuous Diffusion
8.32
Imagen (Saharia et al., 2022b)
Continuous Diffusion
7.27
Parti (Yu al., 2022)
Autoregressive
7.23
Muse (Chang et al., 2023)
Autorrgrrreeeive
7.88
eDiff- (Balaji et al., 2022)
Continuous Diffusion
6,95
ERNIE-ViLA 2.0 (Feng et al., 2023)
Continuous Diffusion
6.75
RAPHAEL (Xue et al., 2023)
Continuous Diffusion
6.61
CONTEXTDIFF
Continuous Diffusion
6.48
Table 2: Quantitative results in text-to-video editing. Text. and Temp. denote CLIP-text and CLIP-temp, respectively. User study shows the preference rate of CONTEXTDIFF against baselines via human evaluation.
Metric
User Study
Method
Text.
Temp.t
Text. (%)*
Temp
Tune-A- Video (Wu et 2022)
0.260
0.934
FateZero (Qi et al., 2023)
0.252
0,954
84
75
Control (Zhao et al., 2023)
0.258
0.961
81
73
CONTEXTDIFF
0.274
0.970
6.48 zero-shot FID score, outperforming previous dominant diffusion models such as Stable Diffu- sion (Rombach et al., 2022), DALL-E (Ramesh al., 2022), and Imagen (Saharia al., 2022b). achieve
We also make qualitative comparisons in Figure 3, and find that our CONTEXTDIFF more precise semantic alignment between text prompt and generated image than previous methods, demonstrating the effectiveness of incorporating cross-modal context into diffusion models. We visualize more qualitative results in Appendix F.I.
5.2 TEXT-TO- VIDEO EDITING
Datasets and Metrics To demonstrate the strength of our CONTEXTDIFF for text-to-video edting, we use 42 representative videos taken from DAVIS dataset (Pont-Tused et al., 2017) and other in in-the- wild videos following previous works (Wu al., 2022; Qi et al., 2023; Bar-Tal et al., 2022; Esser et al., 2023). These videos cover range of categories including animals, vehicles, and humans. To obtain video footage, we use BLIP-2 (Li et al., 2023) for automated captions. We also use their designed prompts for each video. including object editing, background changes, and style transfers. To measure textual alignment, we compute average CLIP score between frames of output videos and corresponding edited prompts. For temporal consistency, 2021) image embeddings on all frames of output videos and report the average cosine similarity between all pairs of video frames. Moreover, We perform and temporal consistency by pairwise comparisons between the baselines and our CONTEXTDIFF. total of 10 subjects participated in this user study. Taking text alignment as an example, given source video, the participants are instructed to select which edited video is more aligned with the text prompt in the pairwise comparisons between the baselines and CONTEXTDIHE.
compute CLIP (Radford et al.,
study quantify text alignment,
Implementation Details order to reproduce the baselines of Tune-A- Video (Wu et al., 2022),
FateZero (Qi al., 2023), and Control (Zhao et al., 2023), their official repositories

---

Published as a conference paper at ICLR 2024
Source
Videos
Tune-A-
Video
FateZero
Control
Video
Ours
A pure black swan is swimming in the water.
A Siberian Husky lying on the beach.
Source
Videos
Tune-A-
Video
Ross
FateZero
Control Video
Ours
man walking on mountaintop covered with dry grass.
red rose.
Figure 4: Qualitative comparison in text-to-video editing, edited prompt is denoted in color. Our CONTEXTDIFE achieves best semantic alignment, image fidelity, and editing quality.
for one-shot video tuning. Following FateZero, we use the trained Stable Diffusion v1.4 (Rom- bach et al., 2022) as the base text-to-image diffusion model, and fuse the attention maps DDIM inversion (Song et al., 2020a) and sampling processes for retaining both structural and motion in- formation. We fuse the attentions in the interval of t E [0.5 T, T,T] of the DDIM step with total timestep T 20. For context-awand adapter, use the same encoders and cross attention in text- to-image generation. We additionally incorporate ppatio-temport attention, which includes spatial self-attention and temporal causal attention, into our context-aware adapter for capturing spatio- temporal consistency. For each source video, we tune our adapter using source text prompt for learning both context-aware structural and motion information, and use the learned adapter con- duct video editing with edited text prompt. Details about the
are in Appendix E.
Quantitative and Qualitative Results We report our quantitative and qualitative results Tab. and Figure 4. Extensive results demonstrate that CONTEXTDIFF substantially outperforms all these baselines in both textual alignment and temporal consistency. Notably, in the textual alignment in user study, outperform the baseline by significant margin (over 80%), demonstrating the
8

---

Published as a conference paper at ICLR 2024
SSuucee video] jenp car moving on
[Sorce Vidao] woman running.
[Tune-A-Vided] jeep moving the
[Tuna-A. (inna-AViidn) An astronaut running the
[Tuna-A- A-Video+ Context-Awwe Aware Adapter] leep moving the
[Tuna-A Aonn----ideo---- Adapter] running the
Figure 5: Generalizing our contexr-aware adapter to Tune-A-Video (Wu et al., 2022).
perior cross cross-modal understanding of our contextualized diffusion. In qualitative comparisons, observe that CONTEXTDIFF not only achieves better semantic alignment, but also preserves the structure information in source video. Besides, the context-awand adapter in contextualized dif- fusion can be generalized to previous methods, which substantially improves the generation quality in Figure 5. More results demonstrating our generalization ability be found in Appendix F.2.
5.3 ABLATION STUDY
Guidance Scale vs. FID Given the significance of classifier-free guidance weight in controlling image quality and text alignment, in Figure 6, we conduct ablation study on the trade-off between CLIP and FID scores across range of guidance weights, specifically 1.5, 3.0, 4.5, 6.0, 7.5. and 9.0. The results indicate that our context-aware adapter contribute effectively. At the same guidance weight, our context-award adapter considerably and consistently reduces the FID, resulting in significant improvement in image quality.
Training Convergence We evaluate CONTEXTDIFF regarding our contribution to the model con- vergence. The comparison Figure demonstrates that our context-aware adapter can significantly accelerate the training convergence and improve the semantic alignment between text and generated video. This observation also reveals the generalization ability of our contextualized diffusion.
26
LDM LDW with Contect-furire Adietter
35.0
24
24.5
FID.Sk
22.
34.0
20
51.5
16
50
15
100
1.5
150
175
2011
Guidance Scale
Training Steps
Figure 6: The trade-off between FID and CLIP scores for LDM and LDM with context-aware adapter.
Figure 7: The comparison of model conver- gence between Lune-A-Video and Tune-A- Video our context-aware adapter.
CONCLUSION
In this paper, we propose novel and general conditional diffusion model (CONTEXTDIFF) by propagating cross-modal context to timesteps in both diffusion and reverse processes, and adapt their trajectories for facilitating the model capacity of cross-modal synthesis. We generalize our contextualized trajectory adapter to DDPMs and DDIMs with theoretical derivation, and consistently achieve of-te-of-theann performance in two challenging tasks: text-to-image generation, and text- to-video editing. Extensive quantitative and qualitative results on the two tasks demonstrate the effectiveness and superiority of our proposed cross-modal contextualized diffusion models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (No. N2..0232004 and U22B2037).
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: -ixttto-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In European Conference on Computer Vision, pp. 707- 723. Springer, 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
Huiwen Chang, Han Zhang, Jamred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint 20X1::2301.00004 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Ad-
vances in Neural Information Processing Systems, volume 34, pp. 8780-8794 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In International Conference Learning Representations, 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.030// 2023.
Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In The AAAI Con- ference on Artificial Intelligence, volume 37, pp. 579-587, 2023.
Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced -ixttureof--denoiiin-eexxe
In IEEE Conference on Computer Vi-
sion and Pattern Recognition, pp. 10135-1015,, 2023.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniy Taigman. Makeea-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13133 131311, 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- Prompt-to-prompt image editing with cross attention control. International Conference on Learn- ing Representations, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851 2020.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint 2rXiv:2301.110093 2023.
Manuel Jahn, Robin Rombach, and Bjöm Ommer. High-resolution complex scene synthesis with transformers. arXiv preprint arXiv: 105.06458, 2021.
10

---

Published as a conference paper at ICLR 2024
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-2666 26577, 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-t--image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303. STtt::233..13399, 2023.
Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-chul Moon. Max- imum likelihood training of implicit nonlinear diffusion model. Advances in Neural Information Processing Systems, 35:32270 -22844, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, volume 34, pp. 21696-21707, 2021.
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. In International Conference Learning Representations, 2021.
Zhi Lei. Guixian Zhang, Lijuan Wu, Kui Zhang, and Rongjaao Liang. multi-level mesh mutual attention model for visual question answering. Data Science and Engineering, 7(4):339-353, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023.
Bootstrapping language-image
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.0477 047611 2023.
Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927. 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.0999. 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 2172-8171, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022a.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, llya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp-
16784-16804, 2022b.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Honne Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 1004/700.0077 2017.

---

Published as a conference paper at ICLR 2024
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning. pp. 8599 8608. PMLR, 2021.
Chenyang Qi. Xiaodong Cun, Yong Zhang, Chenyang Lei. Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In IEEE International Conference on Computer Vision, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Rex., 21(140):1-67, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18-24 Jul 2021. URL
https://proceedie wmw..ng.mm...................... mmrr..r...... www..pppess/eeee siress/vi33e v11///119////////////// wwweeaala..aee......aaa.a
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 10684-10699, 2022.
Chitwan Saharia, William Chan, Huiwen Chang. Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-imen diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- lion Processing Systems, 35:36479-36494, 2022b.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:211 2ff.:21111..21144 2021.
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriun thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32, 2019.
Yang Song, Jascha Sohl-Dickstein. Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020b.
Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv preprint a20i 16007, 2022.
Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing- Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-imen -magt----eeeeaee synthesis. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 16515-16525, 2022.
12

---

Published as a conference paper at ICLR 2024
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-ples diffusion features for text-driven image-to-ing -oagge translation. arXiv preprint arXi::2211.155222200 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Meng Wang, Yinghui Shi, Han Yang, Ziheng Zhang, Zhenxi Lin, and Yefeng Zheng. Probing the impacts of visual context in multimodal entity alignment. Data Science and Engineering, 8(2): 124-134, 2023a.
Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero- shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303. 17599, 2023b.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:22/2. 2 2022.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. arXiv preprint 23X.:23305.18295 2023.
Ling Yang, Jingwei Liu, Shenda Hong, Zhilong Zhang, Zhilin Huang, Zheming Cai, Wentao Zhang, and CUI Bin. Improving diffusion-based image synthesis with context prediction. In Advances in Neural Information Processing Systems, 2023a.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023b.
Ling Yang, Haotian Qian, Zhilong Zhang. Jingwei Liu, and Bin Cui. Structure Structure-guided adversarial training of diffusion models. IEEE Conference on Computer Vision and Pattern Recognition, 2024a.
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal llms. arXiv preprint arXiv:2401.11700 11708, 2024b.
Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to- image synthesis using contrastive learning. arXiv preprint arXiv:2107.02423 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Qinsheng Zhang and Yongxin Chen. Diffussinn normalizing flow. In Advances in Neural Information Processing Systems, volume 34, pp. 16280-16291, 2021.
Xinchen Zhang, Ling Yang. Yaqi Cai, Zhaochen Yu, Jiake Xie, Ye Tian, Minkai Xu, Yong Tang, Yu- jiu Yang. and Bin Cui. Realcompo: Dynamic equilibrium between realism and compositionality improves text-to-image diffusion models. arXiv preprint arXiv:2402. 2024.
Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu. Controlvideo: Adding condi- tional control for one shot text-to-video editing. arXiv preprint arXiv:2305. /7098, 2023.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language- free training for text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17907- 17917, 2022.
13

---

Published as a conference paper at ICLR 2024
A VISUAL ANALYSIS ON CONTEXT AWARENESS OF OUR MODEL
We conduct visual analysis to investigate how our context-aware adapter works in text-guided vi- sual synthesis. As illustrated in Figure 8 and Figure 9, we visualize the heatmaps of text-image cross-attention module the sampling process each frame image. We find that our context-aware adapter enable the model to better focus the fine-grained semantics in text prompt (Lei al., 2022: Wang et al., 2023a) and sufficiently convey them in final generation results. Because incorpo- rating textual information into diffusion process of the image benefits the rross-modal understanding for image diffusion models.
Larger Value
Source Prompt
Edited Prompt
yellow sunflower blooms
white camation blooms
Carnation
Source Video
Heatmaps in Text- Text-lmage Fross-Attention
Camation Generation Fails
FateZero
Heatmaps in Text- Text-Image Crass-Altention
Carnation Generation Succeeds
FateZero
Context- Sontext-Aware Adapter
Shape Awareness
Figure 8: Our context-aware adapter improves the shape awareness of diffusion models in text-
guided video editing.
Source Prompt
Edited Prompt
A silver jeep driving
silver jeep driving down curvy road in the countryside
Larger Value
down curvy road in the countryside
covered with snow
Source Video
Heatmaps Text- Image Cross Attention
Partially Covered with Snow
FateZero
Heatmaps in Text- Image Cross-Attention
Completely Covered with Snow
FateZero with Context- Aware Adapter
Background Awareness
Figure 9; Our context-aware adapter can improve the background awareness of diffusion models in text-guided video editing.
14

---

Published as a conference paper at ICLR 2024
B GENERALIZING TO CLASS AND LAYOUT CONDITIONAL GENERATION TASKS
We generalize our context-aware adapter into class and layout conditional generation tasks. We re- place the text encoder in original adapter with ResNet blocks for embedding classes layouts, and keep the original image encoder and cross-attention module for obtaining cross-modal context infor- mation. We put both quantitative and qualitative results Tabs. and 4 and Figures 10 and 11. From the results, we conclude that our context-ave with different condition modalities and enable more realistic and precise generation consistent with input conditions, demonstrating the satisfying generalization ability of our method.
adapter can benefit the conditional diffusion models
Table 3: Performance comparison in class-to-image generation on ImageNet 256x 256.
Method
FID
IS
Precision
Recall f
BigGAN (Brock et al., 2018)
6.95
203.63
0.87
0.28
ADM-G (Dhariwal & Nichol, 2021)
4.59
186.70
0.82
0.52
LDM (Rombach et al., 2022)
3.60
247.67
0.87
0.48
Aoaee Adapter
2.97
273.04
0.89
0.55
Table 4: FID performance comparison in layout-to-image generation MS-COCO 256x256.
Method
FID
VQGAN+T (Jahn al., 2021)
56.58
Frido (Fan et al., 2023)
37.14
LDM (Rombach et al., 2022)
40.91
L.DM+Context-Wware Adapter
34.58
Class-to-imape -image generation
Ferret
Catamount
Tup
Egyptian Cat
LDM
LDM Context-Aware Adapter
Figure 10: Qualitative results in class- caass-o-inne to-image generation on ImageNet 256x256. Our context-
aware adapter improves the generation quality of LDM.
15

---

Published as a conference paper at ICLR 2024
Layout-to-image generation
Layout
LDM
Context-Aware Adapter
LDM with
Figure 11: Qualitative results in layout-to-image generation on MS-COCO 256x 256. Our context- aware adapter improves both fidelity and precision the generation results of LDM. We use red boxes to highlight critical fine-grained parts where original LDM fails to align with conditional layout. Our method substantially improves both quality and precision of the generation results.
C THEORETICAL DERIVATIONS
C.1 THE DISTRIBUTIONS THE FORWARD PROCESS
First, we derive the explicit expressions for ( .y. c) and g( 1,,, Tps c). based
cross-modal contextualized diffusion defined by Equation (3).
lemma For the forward process g( D2. Fr Lr, c) - II= g(1z|=1-11 To. c). if the transi- tion kernel ( 26. c) is defined as Equation (4), then the conditional distribution q(x; | c) has the desired distribution as Equation i).i Varazo + ker c, c, I).
Proof 1 We prove the lemma by induction. Suppose at time we have q(z)|æ111, To. c) and q(xt-1/Z0 e) admit the desired distributions as in Equations (3) and (4), respectively, then we need to prove that q( Eo, c) = N(r,, Voraro he conditional distributions of s. given (It-1, Fo, c) and xt-1 given (æro c) with the following equations:
(), (1 T)) We can re-write
C1 -Vo0 krro c,
e, 1) VBte,
(14)
0t-1 var-1 1 (wo, C, - 11-az-
(15)
where E1, are two independent standard gaussian random variables. Replacing Φt-1 in Equa- tion (14) with Equation (15), we have:
It = + k (x0, c,t)
- -Bie Voi* (1 - 662
iro c,1-1)
(16)
Varz0 + kr k C, 1) + Bee + aa (1- (1-ac- 0c-1)
-0-1))e) x
16

---

Published as a conference paper at ICLR 2024
As result, the distribution of Ii given (Zo, c) is gaussian distribution with mean Vāyzu + k c,t) and variance a₁ - (1-āt-1) + Pr = āG which admits the desired distribution.
Proposition 1 Suppose the distribution forward process is defined by Equations (3) and (4), then at each time 1, the posterior distribution g( 1 Dt, Lo, c) is described by Equation (5)
Proof 2 By the Bayes rule, g( Co c) a( (Do () (or0la//2-11110 1,,,) By Equations (3)
q( (Focc)
and (4), the numerator and denominator are both gaussian then the posterior distribution is also gaussian and we can proceed to calculate its mean and variance:
Na Vor--120 + b (x0o -1- a(-1)
q(xt-11 De. vo. c)
N( v 00 + (00 c)2 (1 - I))
(17)
NN V /a + (z c) Varbi-1 c), 3,1)
where (a c) is an abbreviation form of (aro e, t). Dropping the constants which are unre- lated to vo, Lts #t-1 and c, we have:
g(x;-- 120,, vo, c) exp {
(It-1 vā b, -20, + (æt vā - (x0. c))²
2(1 ((--1)
2(1 (+.)
(at,
Va
bi C) + varbi-1 c))2
2
eap
Cl We c)
//2
1
+
ag
) 2ن 0z-1*
Ūt-1
B.
1a-11u + 11-1 c))
(Bt - bi c) + V by (20, c))
+ Vor
az-1
Be
exp
20000 c)
-/// δc-1
1
+
al
x + xt-1#
Br
(V0t-1
Vau
1
Dy
(ar
be c))
* b c)
Ōx-1 Br
ât-1
3r
(18)
where C(ro, Fe. c) is constant term with respect to Rt-1- Note that (1 1-û,
(1-6e-1)[1 100 and with some algebraic derivation, we can show that the gaussian distribution
((x;-1 1 Fc, Do. c) has:
variance (1 -1- ūt 1) ass)
(19)
Va Br
/ai(1 - ax-1))
mean
xo +
(x, be(20, c)) + b.-1 c)
1 ae
Similarly, we can derive the distribution of DDIMs.
lemma 2 Suppose that at each time the posterior distribution is defined by a gaussin distribution with
Mean: Var-1a0 + V1 āt-1 - 03
Et / Va
v1 āe
-krg(au, v1 :1-1 + +kr (o
(20)
C,
VI αe
Variance I.I,
then the marginal distribution qo(2 c) has the desired distribution Equation (3)
Proof 3 We prove by induction. Suppose that at time t. posterior and marginal distributions admit the desired distributions, then we need to prove that at time t.- go (zr--1 Do, c) also has the desired
17

---

Published as a conference paper at ICLR 2024
distribution. Rewrite the posterior and marginal distribution the following:
3t-1 - + V ix-1 02
It
Vā
vI V1-āi ōt
(21)
kar c,t) v1 -0-0001--- -
hi-1 ( + or61
VI- a+
2t = Varao + heroloo c, + v1 0452,
(22)
where EL, E2 are standard gaussian noises. Plugging in Wi, have:
2t-1 = vā
(r, v1 0 kar c,l) V1- 1
āt
at
(23)
+ 1r e, + 0=61 + / - -0-1- - -0/12
=v + 1r 1)) + 0161 + 1-
:i 1211-07- o,
Since the variance o,e + /I 1111--1--1 a o/ is d we have the desired distribution.
C.2 UPPER BOUND OF THE LIKELIHOOD
Here show with our parameterization, the objective function Co Equation (8) is upper bound of the negative log likelihood of the data distribution.
lemma 3 Based on the non-Markovian forward process g( U2 Br ( c) vo. c) and the conditional reverse process pe pa I., 22 .... xT xT|C) II (
po(ørle) [ 1 c), the objective function Equation (6) is an upper bound of the negative log likelihood.
Proof 4
Po po Cu. c) q xo, c)
log (xolc) < -log Eg((r mo.c)
-log log
Po (o:rl (c)
log
q( æo; C)
pe( ()) II Po x2, c)
--Bg(an.rao.el
log
I g( x1- 20, c)
Po [e, c) log po (xT)(c) log (xx; 2 20, c)
log po c)
-- cw
q(<0) xo, c)
log (c) + log (20 x1, c)
=-Ry(e1:: (x..e)
g xo. c)
pa(a;- 81, c)
g( 100, C)
log ( Fe, Fo, c)
( 2o, c)
po
Po (c | c)
= --E (
log q( xo. c)
+ log xz, c) + log
g ( Ly, c)
t>1
= DKL (go ww c)) ( log po [21, c)
+ Σ Ey( Dus |r, ao, c))lpo |2 c))
(24)
18

---

Published as a conference paper at ICLR 2024
lemma 4 Assuming the relational network ra c, is Lipschitz contimuous, iee. Vtt 3a positive real number Ct s.t. [|Ta(To, c,t) | oz e, t)ll I C. Cillæ0 =gll, then ( c, zol2 is an upper bound of DKL (oo (e zo. lp (-t-1 [2t, e)) after scaling.
Proof 5 From the main text, we know that
PKL(go(ar-- ( Fo. (2) c))
(ul(a,,c,t) vāc-1 x0 var(1 ( (x) be(200, c)) b e -
(25)
die
where palar, e, is the mean of qo go( (2,, c). Tn Re. e. With our parameterization,
Here we discard a constant with respect to fo( c,t). t). Thus the objective fanction can be simplified as: where 20
(o(x,, = var-1 1 to 1 de.
vor(1 111-----111 (2--1) (o, bi c)) 1 c),
(26)
11-1 13
zu) + Vor(1 āz-1)
(b: c) be c)) (bi- C) b1-1(x0, c))l12
1 - át
02 14-1
0 - a1-1) /1b, 11b - bi c)lla + 1 c) b c)ll
|1000 Boll2 +
Og
0ft
04
13g
v Vai(1 a(-1)
|000 - 2 +
kg k2 xoll2 + hi xol12
(ze: zall2
(27)
Similar results can be proved for DDIMs by replacing the mean of posterior in DDPMs with DDIMs, defined by Equation (20), in Equation (25).
Assume that the total diffusion step T is big enough and only a neglegible amount of noise is added the data the first diffusion step, then the term Big(z//uudd e) log pe (0o| c) is approximately zero. Now combining Lemmas 3 and 4, we have the following proposition:
(no (xrr(e))))))
Proposition 2 The functionn defined in Equation (8) an upper bound of the negative log likelihood.
C.3 ACHIEVING BETTER LIKELIHOOD WITH CONTEXTDIFF
Next, we show that CONTEXTDIFF is theoretically capable achieving better likelihood compared to original DDPMs. As the exact likelihood is intractable, aim to compare the optimal variational bounds for negative log likelihoods. The objective function of CONTEXTDIFF at time step is Egg Dx (q It, To. (I and its optimal solution is
min E,, DKL ( (x 12,, wo, co. c)) 0.0
= min miingE... D (go (2: R To, c) (2 c))
(28)
I mingly, y 1
vo. (
c)),
0 denotes setting the adapter network identical to 0. and thus mingEeaan DEL z (qs=u(2;-1 Fo, c) (2;-) / c)) is the optimal loss of origianl DDPMs objective at time Similar inequality can be obtained for t=1:
where
min Eg, log æ1, 6.0
(29)
S min Egozo log.o(Zu)21, 21,
19

---

Published as a conference paper at ICLR 2024
As result, have the following inequality by summing up the objectives at all time step:
Ry(wo) log po
S mins,
Bag ( æo. c)llpo (2) (2) c)) + Egg Loggo c)
< ming
E KL D c c)) Bye= logpo 210
(30)
where C EDLL (1s (ar ao, c) po (Tr) is a constant defined by vār. Hence, CONTEXTI>- IFF has tighter bound for the NLL,, and thus theoretically capable of achieving better likelihood, compared with the original DDPMs.
C.4 BETTER EXPRESSION OF CROSS-MODAY SEMANTICS
We provide an in-depth analysis on why CONTEXTDIFF can better express the cross-modal se- manties. Our analysis focuses on the case of optimal estimation, as the theoretical analysis of convergence requires understanding the non-convex optimization of neural network, which is be- yond the scope of this paper. Based on objective function in Equation (8), the optimal solution of CINTEXTDIFF at time can be expressed as
arg min (ole))l - fs( c)1 0.8
arg min arg min R
fo( c)ll
(31)
arg min (asjo)) E pasc -
,-, rr
since the best estimator under L2 loss is the conditional expectation. As a result, the optimal esti- mator of CONTEXTDIFF for an is
E|zo kyTg' (#o. c) + Varzo + vI āge, c),
(32)
while existing methods that did not incorporate cross- modal contextual information in the forward process have the following optimal estimator:
E ool / /a + vI āec, c).
(33)
Compared with existing methods, CONTEXTDIFE can explicitly utilize the cross-modal context Ts- (Io, e) to optimally recover the ground truth sample, and thus achieve better multimodal se- mantic coherence.
Furthermore, we analyze toy example to show that CONTEXTDIFF can indeed utilize the cross- modal relations better recover the ground truth sample. We consider the image embedding zo and text embedding e that were generated with the following mechanism:
To = u(c) + (c)e,
(34)
where is an independent standard gaussain, pcc) and (²(c) are the mean and variance of To ditioned on c. We believe this simple model can capture the multimodal relationships in the em- bedding space, where the relevant images and text embeddings
closely aligned with each other. Then H, = Va,io /1 ā,é is the noisy image embedding in original diffusion model. We aim to calculate and compare the optimal estimation error at time step in original diffusion model and in CONTEXTDIFF:
min | - fo(zo, c)
(35)
E |00 E| (r,, <]ll
The conditional expectation as the optimal estimator of DDPMs can be calculated as:
Eizolan xr, c| = p(c) - Cou(Lo, xy/c) Var(zzle)
- xx)
Vaja(c)2
= p(c) (.J(c)² +1 (Vāsu(c) Iz)
20

---

Published as a conference paper at ICLR 2024
As result, calculate the estimation enror of DDPMs:
x0 Elzo xr, cll
(a (Vāro(c)e+ vI- aye )l2
Ello(c)e a(0(c)2 + -1-11 āe
=d (1 - (L-a))ar(c)) + (c) -ā²)
(36)
(o) +1 ū,)²
at d [(c (c)
Now we use CONTEXTDIFF with parameterized adapter: Voraag -1 age +r(o, c, 1) where r(ø, c, zo is the adapter. We can similarly calculate the conditional mean as the optimal estimator of CONTEXTDIFF:
d (c)(((() Vā.) āt + (r(o,c,t) e,t + vā,) ((r(0,,,t) Vojuu(() xx)
Eg/2o|2, c p(c)
And the estimation error for given in CONTEXTDIEF is:
E 120 Lo <il1
(c)(((()) (r)
El(a(()e 1- āt + (r(0, c, /0:))22200 ((1(0,,,()+ vaija(c)e + v1 ā,6 )|2
= dc) 1 āi (r(0, c, 2o.)
(37)
Comparing the denominators of two estimation errors (Equations (36) and (37)), we can see that using a non-negative adapter will always reduce the estimation error.
D MORE MODEL ANALYSIS
Comparison on Computational Costs We compare our method with LDM and Imagen regard- ing parameters, training time, and testing time in Tab. 5. We find that our context-ane aware adapter (188M) only introduces few additional parameters and computational costs to the diffusion back- bone (3000M), and substantially improves the generation performance, achieving better trade-off than previous diffusion models.
Table 5: Comparison on Computational Costs.
Method
#Params
#Training Cost
Inference Cost
LDM SDXL
1.4B
0.39 s/lmg
3.4 s/Img
12.64
10.3B
0.71 s/Img
9.7 s/Img
8.32
DALL-E
6.5B
2.38 s/Img
21.9 s/Img
10.39
Imagen
3B
0.79 s/Img
13.2 s/Img
7.27
Our CONTEXTDIFF
3B+1
188M
0.83 s/Img
13.4 s/Img
6.48
Contributing to Faster Convergence In the Figure 7 of main text, we generalize our context- aware adapter to other video diffusion model Tune-A-Vided and make a faster and better model convergence. Here, we additionally generalize the adapter to image diffusion model LDM and plot partial training curve on the subset LAION dataset in Figure 12. Similar to video domain, context-aware adapter also substantially improve the model convergence for image diffusion models, demonstrating the effectiveness and generalization ability our method.
Quantitative Results on Likelihood We follow (Kim et al., 2022) to compute NLL/NELBO (Ne- gaitve Log- LikelihooodNeggaive Evidence Lower Bound) for performances of density estimation with Bits Per Dimension (BPD). We train context-aware adapter CIFAR-10 10 and compute NLL with the uniform dequantization. As the results Tab. 6, we conclude that our method is empirically capable of achieving better likelihood compared to original DDPMs.
21

---

Published as a conference paper at ICLR 2024
FID training progress
36
34
Adapter
32
30
28
26
24
22
10
20
train steps
Figure 12: Our context-awand adapter can enable faster model convergence (partial training curve)
Table 6: NLL comparison on CIFAR- 10.
Method
NLL +
DDPM (Jahn et al., 2021)
3.01
DDPM + Context-Aware Adapter
2.63
E HYPER-PARAMETIE IN CONTEXTDIFF
We provide detailed parraepprameters training CONTEXTDUP for text-to-image generation (in
Tab. 7) and text-to- tovvieoo editing (in Tab. 8).
Table 7: Hyper- prpee--prraettrs in training CONTEXTDIFF for text-to-image generation.
Configs/Hyper paraaeete
Values
T
1000
Noise schedule
cosine
Number of transformer blocks for cross-modal interactions
4
Betas of Adan (Loshchilov Hutter, 2018)
(0.9, 0.999)
Weight decay
0.0
Learning rate
le-4
Linear warmup steps
10000
Batch size
1024
F MORE QUALITATIVE COMPARISONS
F.1
MORE QUALITATIVE COMPARISONS ON TEXT-TO-IMAGE GENERATION
In order fully demonstrate the effectiveness of our proposed contextualized diffusion, visualize more qualitative comparison results in Figure 13. The results sufficiently demonstrate the superior cross-modal understanding in generated images of our CONTEXTDUF over other models.
F.2 GENERALIZING TO OTHER TEXT-GUIDED VIDEO DIFFUSION MODELS
Qualitative Results In order to fully demonstrate the generalization ability of the context-aware adapter in our contextualized diffusion, we visualize more qualitative comparison results, where we utilize context-aware adapter to improve Tunc-A- A-Video (Wu et al., 2022) (in Figure 14) and FateZero (Qi et al., 2023) (in Figure and Figure 16). From the results, we observe that our context-aware adapter can effectiveness promote the performance of text-to-video editing, significantly enhancing the semantic aligmment while maintaining structural information in source videos. All video exam- ples also provided in the supplementary material, and we are committed open sourcing the Pain/inference code upon paper acceptance.
22

---

Published as a conference paper at ICLR 2024
Table 8: Hyper-parameters in training CONTEXTDIFF for text-to-video editing.
Configs/llyee--prameee
Values
T
20
Noise schedule
lincar
(Sstort; 9end)
(0.00085, 0.012)
Number of transformer blocks for cross-modal interactions
Frames for causal attention
Betas of AdamW (Loshchilov & Hutter, 2018)
(0.9.0 0.999)
Weight decay
le-2
Learning rate
le-5
Warmup steps
Use checkpoint
Truc
Batch size
1
Number of frames
8-244
Sampling rate
bustling city street during rainy day with reflections wet pavement."
"Thee is an old typewriter and sheet of paper with four letters 'LOVE' on the table."
Ho08 fox
LOVE
RH
"A film camera, with a roll of film partially exposed."
A cup of coffee is table in coffee shop with arched ceiling, vintage furniture."
A market with textiles, aromatic spices under the sun."
Imagen
Ours
Figure 13: Synthesis examples demonstrating text-to-image capabilities of for various text prompts with LDM, Imagen, and CONTEXTDIFF (Ours). Our model can better express the semantics of the texts marked in blue. We use red boxes to highlight critical fine-gre fine-grained parts where LDM and Imagen fail to align with texts. For example, second row, only our method successfully generates the four letters spelling "LOVE". In third row, we generate the specific detail of a film roll, while other methods lose this detail.
23

---

Published as a conference paper at ICLR 2024
|Souree vidco] A rabbit is cating watermelon.
[Tune-A Video] A puppy eating an orange.
[Tune-A- Vineo-Contstt-Awar Adapter| A puppy is cating orange.
[Source video] A man is surfing.
[Tune-A- A-Vido] A raccoon surfing, cartoon style.
(Tune-A- Tune-A-Video+ Contoo--CotettAwrree Adapter] A raccoon is surfing, cartoon style.
[Source video] A man skiing
(Tune-A-Video) Wonder wearing cowboy hat skiing.
[Tunc-A- TTnc-A-Videon A-Video++ ConlextConttt-AAaara Adapter] Wonder woman, wearing cowboy hat is skiing.
Figure 14: Generalizing our context-aware adapter to Tune-A- A-Video (Wu et al., 2022).
24

---

Published as a conference paper at ICLR 2024
[Source Video] Two sharks are swimming in the ocean.
[HateZero] Two goldfishes are swimming the ocean.
FFateZero Context-Aware Adapter Two goldfishes swimming the ocean.
[Source Video] A yellow cat.
[FateZero] A yellow dog.
[FateZero Context-Aware Adapter] yellow dog.
Figure 15: Generalizing our context-aware adapter to FateZero (Qi et al., 2023).
25

---

Published as a conference paper at ICLR 2024
[Source Video] A person wearing a helmet on motorcycle does burnout on the road.
[Fate/ero] person wearing a helmet on motorcycle does a hurnout on frozen lake
|FateZero Context-Aware Adapter person wearing helmet motorcycle does burnout frozen lake
[Source Video] A brown bird sits a feeder that is being hung by red string.
[FateZero] A brown squirrel sits on feeder that being hung by red string.
[FateZero + Context-Aware Adapter] A brown squirrel sits on feeder that being hung by string.
Figure 16: Generalizing our context-aware adapter to FateZero (Qi et al., 2023).
26


Reviews:
[{"review_id": "wUHSQM4V9Z", "reviewer": "Reviewer_DNxF", "summary": "This paper introduces the concept of contextualized forward and reverse diffusion processes, which is interesting. They have made modifications to traditional models and proposed a method that significantly improves semantic alignment. The experimental results section effectively demonstrates the efficacy of this approach. The authors provide detailed theoretical and empirical evidence, which lends strong support to this article.", "questions": "The questions I would like the authors to address have already been raised in the \"weakness\" section. I hope the authors can provide more information on these aspects.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The novelty of this article lies in the introduction of a new contextualized forward and reverse diffusion processes. They have made improvements upon existing methods and provided theoretical support.\n- The results presented in this paper have shown promising performance when compared to existing models, and the visualization section further supports the efficacy of this meth", "weaknesses": "- Regarding evaluation metrics, while existing metrics are commonly used, aligning text, especially fine-grained text content, with images requires more refined evaluation criteria. I would like to hear the authors' opinions on the need for improved evaluation metrics for more fine-grained, context-aware image and video generation. Additionally, why not incorporate human evaluation of the generated data in this context?\n\n-  The authors have introduced some additional controls, and it would be beneficial to discuss the associated costs, such as extra parameters, training time, and testing time, to aid in understanding their proposed method.\n\n- The authors have achieved promising results in natural images or videos, but there is a need for further discussion regarding more fine-grained context-awareness. For instance, it would be interesting to explore whether the method remains effective when dealing with specific text from within an image, as generating precise text remains a challenge for most methods. Similarly, what happens when modifying specific parts of an image? I would like to see the authors' insights on these issues concerning their context-aware adapter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EtHEMvSDEK", "reviewer": "Reviewer_D2VS", "summary": "This paper proposes a  general cross-modal contextualized diffusion model (CONTEXTDIFF) that harnesses cross-modal context to facilitate the learning capacity of cross-modal diffusion models. The cross-modal interactions between text condition and image/video sample are incorperated into the forward process, serving as a context-aware adapter to optimize diffusion trajectories. The context-aware adapter to adapt the sampling trajectories, which facilitates the conditional modeling in the reverse process and aligns it with the adapted forward process. A series of experimental results and mathematical proofs are presented.", "questions": "1. In Figure 6, the author claims that they conduct ablation study on the trade-off between CLIP and FID scores across a range of guidance weights, however, only FID scores are provided in the figure.\n\n2.  Analysis on why adding textual information during the forward process enhances multimodal semantic relevance.\n\n3. Whether this method can be incrementally trained on other pretrained generative models (e.g. Stable Diffusion), and if doing so would result in improved generation performance and faster convergence, is not discussed in the paper.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a method to enhance the multimodal relevance by incorporating multimodal contextual information during the forward process of the diffusion model.\n\n2. Adequate mathematical proofs are provided for both the forward and backward processes.\n\n3. The experimental results to some extent demonstrate that this method yields a high semantic correlation between the generated images and text.", "weaknesses": "1. Compared to the proposed method, existing methods also utilize cross-modal attention during the forward process to control the generated content of images based on information from different modalities.\n\n2. The article does not provide sufficient analysis for why adding textual information during the forward process enhances multimodal semantic relevance. It is based on intuitive reasoning rather than an in-depth analysis.\n\n3. In the provided experimental results, the method proposed in this paper shows limited improvements compared to existing methods (e.g. Imagen) in image generation tasks.\n\n4. The paper does not analyze the limitations of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T1QGxCSZ4Y", "reviewer": "Reviewer_zGd6", "summary": "This work proposes a novel conditional diffusion model, ContextDiff. Rather than only modeling the cross-modal context in the backward process, ContextDiff propagates the context information to all timesteps in both forward and backward process to adapt the trajectories for facilitating cross-modal conditional generation. The proposed method can be generalized to DDPMs and DDIMs and achieves better results in text-to-image generation and text-to-video editing.", "questions": "* How does the method compare to the baseline when integrated into latent diffusion (or Stable Diffusion)?\n* How does the method compare to the baseline in terms of inference latency?\n* What is the Stable Diffusion version used in Table 1? How does the method compare with different versions of Stable Diffusion?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The idea of modeling the cross-modal context in the forward process is interesting, as it differs from previous works that only consider conditional modeling in the backward process.\n* The method has sound theoretical foundations. Furthermore, the generalization to DDIMs is a clear strength that allows fast sampling.\n* The writing of the method section is clear. The adaptation to the previous diffusion process with a bias term is straightforward to understand.\n* The evaluation results show that the model performs better in terms of qualitative metrics of both automated evaluation and user study.", "weaknesses": "* Experiments on latent diffusion: the method uses an Imagen-based framework, which generates a low-res image and then performs super-resolution. However, the author does not evaluate the proposed method on latent diffusion architecture. There are mentions of LDM in Sec 5.3 (ablations), but the setting is not clearly described, and comparisons with other works (rather than the baseline) are not offered.\n* The author does not offer an inference latency evaluation. Does the method slow inference down compared to baseline diffusion methods that do not have context-aware adapters?\n* A small typo (which does not affect the rating): \"A red ross\" -> \"A red rose\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ABuKQfRop", "reviewer": "Reviewer_YRdg", "summary": "This paper proposes a contextualized diffusion model ContextDiff to facilitate the learning capacity of cross-modal diffusion models. It incorporates the cross-modal interactions between text condition and visual sample into both forward and reverse processes, serving as a context-aware adapter to optimize diffusion trajectories. It is also generalized to both DDPMs and DDIMs for benefiting both cross-modal generation and editing tasks with detailed theoretical derivations. Experiments in text-to-image generation and text-to-video editing tasks show its effectiveness. Empirical results reveal that it can successfully improve the semantic alignment between text conditions and synthesis results.", "questions": "1) Are there any more experiments that can prove the text-based editing ability of this approach, for example, conducting T2I editing or T2V editing based on the well-trained T2I generation model of your first experiment?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper for the first time proposes the text-guided visual diffusion model to consider cross-modal interactions in both forwarding and sampling processes.\n\n2) The authors propose their contextualized diffusion model (ContextDiff) and generalize it to DDPMs and DDIMs through the derivations of theoretical formulas.\n\n3) The experiments show that the proposed method has improvements on two tasks: T2I generation and T2V editing, compared with other state-of-the-art methods.", "weaknesses": "1) This paper claims the problem that neglecting the cross-model context in the forward process may limit the expression of textual semantics in synthesis results, but there is no clear explanation of the specific reasons, and there is no intuitive and theoretical analysis of the necessity of adding cross-model to the forward process.\n\n2) As claimed by the authors: “Thus CONTEXTDIFF is theoretically capable of achieving better likelihood compared to original DDPMs”. Please provide the quantitative results on ELBO/ likelihood compared to the baseline.\n\n3) Since this paper is a general improvement on the conditional diffusion model, more results on different conditional generation tasks, such as class-to-image/layout-to-image…should be provided. \n\n4) Some configurations in T2V editing experiments are confusing, as the experiments based on pre-trained Stable Diffusion v1.4 are not enough to prove that the approach of this paper can enable diffusion models better editing ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)