Improving Diffusion-Based Image Synthesis with Context Prediction
Authors: Ling Yang
Jingwei Liu
Shenda Hong
Zhilong Zhang
Zhilin Huang
Zheming Cai
Wentao Zhang
Bin CUI
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model
Image Generation
TL;DR: We propose to explicitly and fully preserve neighborhood context for improving diffusion-based image synthesis.

Abstract: Diffusion models are a new class of generative models, and have dramatically promoted image generation with unprecedented quality and diversity. Existing diffusion models mainly try to reconstruct input image from a corrupted one with a pixel-wise or feature-wise constraint along spatial axes. However, such point-based reconstruction may fail to make each predicted pixel/feature fully preserve its neighborhood context, impairing diffusion-based image synthesis. As a powerful source of automatic supervisory signal, context has been well studied for learning representations. Inspired by this, we for the first time propose ConPreDiff to improve diffusion-based image synthesis with context prediction. We explicitly reinforce each point to predict its neighborhood context (i.e., multi-stride pixels/features) with a context decoder at the end of diffusion denoising blocks in training stage, and remove the decoder for inference. In this way, each point can better reconstruct itself by preserving its semantic connections with neighborhood context. This new paradigm of ConPreDiff can generalize to arbitrary discrete and continuous diffusion backbones without introducing extra parameters in sampling procedure. Extensive experiments are conducted on unconditional image generation, text-to-image generation and image inpainting tasks. Our ConPreDiff consistently outperforms previous methods and achieves new SOTA text-to-image generation results on MS-COCO, with a zero-shot FID score of 6.21.

Improving Diffusion-Based Image Synthesis with Context Prediction
Ling Yang'
Jingwei Liu't
Shenda Hong'
Zhilong Zhang'
Zhilin Huang?
Zheming Cai
Wentao Zhang' Bin Cui
Peking University 2 "Tsinghua University
yangl1ng08180168 com, jingwe1liu19966133 com, zhilong zhangdbjnnd edu. (hongshenda, wentao. zhang, bin. cin.cu. eduiee.dd.co
Abstract
Diffusion models are a new class of generative models, and have dramatically promoted image generation with unprecedented quality and diversity. Existing diffusion models mainly try to reconstruct input image from a corrupted one with a pixel-wise feature- wiie constraint along spatial axes. However, such point-based reconstruction may fail to make each predicted pixel/feature fully preserve its neighborhood context, impairing diffusion-based image synthesis. As powerful source of automatic supervisory signal, context has been well studied for learning representations. Inspired by this, for the first time propose CONPREDIFF improve diffusion- based image synthesis with context prediction. We explicitly reinforce each point predict its neighborhood context (i.e., multi-stride fea- tures/tokens/pixels)) with context decoder at the end of diffusion denoising blocks in training stage, and remove the decoder for inference. In this way, each point can better reconstruct itself by preserving its semantic connections with neighbor- hood context. This new paradigm of CONPREDIFF can generalize arbitrary discrete and continuous diffusion backbones without introducing extra parameters in sampling procedure. Extensive experiments are conducted on unconditional image generation, text-to-image generation and image inpainting tasks. Our CON- PREDIFF consistently outperforms previous methods and achieves a new SOTA text-to-inn coxt-to-image generation results on MS MS-COCO, with zero-shot FID score of 6.21.
Introduction
Recent diffusion models [99, 5, 63, 4, 47, 10, 22] have made remarkable progress in image generation. They are first introduced by Sohl-Dickstein et al. [76] and then improved by Song & Ermon [78] and Ho et al. [28], and can now generate image samples with unprecedented quality and diversity [24, 68, 67]. Numerous methods have been proposed to develop diffusion models by improving their empirical generation results [53, 77, 79] extending the capacity of diffusion models from theoretical perspective [80, 81, 46, 108]. We revisit existing diffusion models for image generation and break them into categories, pixel- and latent-based diffusion models, according to their diffusing spaces. Pixel-based diffusion models directly conduct continuous diffusion process in the pixel space, they incorporate various conditions (e.g., class, text, image, and semantic map) [29, 70, 51, 2, 66] or auxiliary classifiers [80, 14, 27, 54, 40] for conditional image generation.
On the other hand, latent-based diffusion models [65] conduct continuous discrete diffusion process [87, 30, 1] on the semantic latent space. Such diffusion paradigm not only significantly reduces the
"Contact: Ling Yang, yangling08 @8@ 163.com.
|Contributed equally.
*Corresponding Authors: Wentao Zhang, Bin Cui.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

computational complexity for both training and inference, but also facilitates the conditional image generation in complex semantic space [62, 38, 58, 19, 98]. Some of them choose to pre-train an autoencoder [41, 64] to map the input from image space to the continuous latent space for continuous diffusion, while others utilize a vector quantized variational autoencoder [88, 17] to induce the token-based latent space for discrete diffusion [24, 75, 1 114, 85].
Despite all these progress of pixel- and latent-based diffusion models in image generation, both of them mainly focus on utilizing a point-based reconstruction objective over the spatial axes to recover the entire image in diffusion training process. This point-wise reconstruction neglects fully preserve local context and semantic distribution of each predicted pixel/feature, which may impair the fidelity of generated images. Traditional non non-diffusion studies [15, 45, 32, 50, 110, 8] have designed different context-presete
terms for ad advancing image representation learning, but few researches have been done to constrain on context for diffusion-base image synthesis.
In this paper, we propose CONPREDIFF to explicitly force each pixel/feature/token to predict its local neighborhood context (i.e., multi-stride fcatures/ookens/pixlss)) in image diffusion generation with an extra context decoder near the end of diffusion denoising blocks. This explicit context prediction can be extended to existing discrete and continuous diffusion backbones without introducing additional parameters in inference stage. We further characterize the neighborhood context as probability distribution defined over multi-stride neighbors for efficiently decoding large context, and adopt an optimal transsoor loss based on Wasserstein distance [21] to impose structural constraint between the decoded distribution and the ground truth. We evaluate the proposed CONPREDIFF with the extensive experiments on three major visual tasks, unconditional image generation, text-to-image generation, and image inpainting. Notably, our CONPREDDIF consistently outperforms previous diffusion models by large margin regarding generation quality and diversity.
Our main contributions are summarized as follows: (i): To the best of our knowledge, we for the first time propose CONPREDIFF to improve diffusion-basse image generation with context prediction; (ii): We further propose an efficient approach to decode large context with an optimal-transpon loss based on Wasserstein distance; (iii): CONPREDIFF substantially outperforms existing diffusion models and achieves new SOTA image generation results, and we can generalize our model to existing discrete and continuous diffusion backbones, consistently improving their performance.
2 Related Work
Diffusion Models for Image Generation Diffusion models [99, 76, 78, 28] are a new class of probabilistic generative models that progressively destruct data by injecting noise, then learn to reverse this process for sample generation. They can generate image samples with unprecedented quality and diversity [24, 68, 67], and have been applied in various applications [99, 9, 6]. Existing
pixel- and iitent-based diffusion models mainly utilize the discrete diffusion [30, 1, 24] or continuous
diffusion [87, 65] for unconditional or conditional image generation [80, 14, 27, 54, 40, 68). Discrete
diffusion models were also first described in [76], and then applied to text generation in Argmax Flow [30]. D3PMs [1] applies discrete diffusion to image generation. VQ-Diffusion [24] moves discrete diffusion from image pixel space to latent space with the discrete image tokens acquired from VQ- [88]. Latent Diffusion Models (LDMs) [87, 65] reduce the training cost for high resolution images by conducting continuous diffusion process in Now-dimensiona latent space. They also incorporate conditional information into the sampling process via cross attention [89]. Similar techniques employed in DALLE-2 [62] for image generation from text, where the continuous diffusion model is conditioned on text embeddings obtained from CLIP latent codes [59]. Imagen [68] implements text-to-image generation by conditioning
text embeddings acquired from large language models (e.g.. T5 [60]). Despite all this progress, existing diffusion models neglect to exploit rich neighborhood context in the generation process, which is critical in many vision tasks for maintaining the local semantic continuity in image representations [111, 1111,45, 45, 32, 50]. In this paper, we firstly propose explicitly preserve local neighhorhood context for diffusion-based image generation.
Representation Learning Context has been well studied in learning representa- tions, and is widely proved to be a powerful automatic supervisory signal in many tasks. For example, language models [52, 13] learn word embeddings by predieting their context, iee, a few words before and/or after. More utilization of contextual information happens in visual tasks, where spatial context

---

Pu(xe-alike)
Neighborhood Context Prediction
Nr
min
For
prod
Sample Neighbors
ared *(UMRU
Time Embedding
(x-1)
a-Net Transformer
Conditioning
Paler Dewsldig
FNN,
FNN
Decoding
S1. Eu S4
Causstannlu,E.
Neighborhood Distribution
Figure 1: In training stage, CONPREDIFT first performs self-denoising standard diffusion models, then it conducts neighborhood context prediction based denoised point 2/- 1.- In inference stage, CONPREDIFF only uses its self-denoising network for sampling.
is vital for image domain. Many studies [15, 111, 45, 32, 50, 110, 8, 106, 95, 94, 44] propose to leverage context for enriching learned image representations. Doersch al. [15] and Zhang al. [110] make predictions from visible patches to masked patches to enhance the self-supervised image representation learning. Hu al. [32] designs local relation layer to model the context of local pixel pairs for image classification, while Liu et al. [45] preserves contextual structure to guarantee the local feature/pixel continuity for image inpainting. Inspired by these studies, this work, we propose to incorporate neighborhood context prediction for improving diffusion- generative modeling.
3 Preliminary
Discrete Diffusion We briefly review classical discrete diffusion model, namely Vector Quantized Diffusion (VQ-Diffusion) [24]. VQ-Diffusion utilizes a VQ-VAE to convert images x to discrete tokens 2g E 21.2.... K, K + 1). is the size of codebook, and K denotes the [MASK] token. Then the forward process of VQ-Diffusion is given by:
g(xx(x=-1) x" (m=)Q.((ze11)
where vxx) is a one-hot column vector with entry 1 at index æ. And Q₂ is the probability transition matrix from æ;-1 2, with the and--aeeeee VQ-Diffusion strategy. In the reverse process, VQ- Diffusion trains denoising network pe (0(x;=11xz) that predicts noiseless token distribution pe (Bo (x:) each step:
pe (a (.))
Σ q Zo) (x0 (x:),
(2)
#p=1
which is optimized by minimizing the following variational lower bound (VLB) [76]:
C Dww((((gxxxabbb xb))
Continuous Diffusion A continuous diffusion model progressively perturbs input image or feature map Do by injecting noise, then learn reverse this process starting from ør for image generation. The forward process can be formulated as a Gaussian process with Markovian structure:
q( (x)))=;-1) N(a -- 1,
q( (x)) = (W Vo (1 Rz)1),
where Bi,..., Br denotes fixed variance schedule with on := B, and 04 II a,- This forward process progressively injects noise data until all structures are lost, which is well approxi- mated by (0, I). The reverse diffusion process learns model pa( (o(x;-1le lx,) that approximates the

---

true posterior:
Po P0 x.)) == Mi Ex))),,
(5)
Fixing Σg to be untrained time dependent constants I, Ho et al. [28] improve the diffusion training
process by optimizing following objectíve:
con
E
(2). 1) f0 mo)ll + C,
(6)
207
where C is constant that does not depend on 0. ( ma) is the mean of the posterior ((2; 1 Fo, Ft), and so (mr, i) is the predicted mean of po ( mr) computed by neural networks.
4 The Proposed CONPREDIFF
In this section, we elucidate the proposed CONPRE as in Figure 1. In Sec. 4.1, we introduce our proposed context prediction term for explicitly preserving local neighborhood context in diffusion- based image generation. To efficiently decode large context in training process, we characterize the neighborhood information as the probability distribution defined over multi-stride neighbors in Sec. 4.2, and theoretically derive optimal-transport loss function based on Wasserstein distance to optimize the decoding procedure. In Sec. 4.3, we generalize our CONPREDIFF to both existing discrete and continuous diffusion models, and provide optimization objectives.
4.1 Neighborhood Context Prediction in Diffusion Generation
We use unconditional image generation illustrate our method for simplicity. Let -زا E Rd to denote i-th pixel of the predicted image, i-th feature point of the predicted feature map, or i-th image token of the predicted token map in spatial axes. Let NV denote the s-stride neighborhoods of æ and K denotes the total number of N For example, the number of 1-stride neighborhoods is K 8, 24.
and the number of 2-stride ones is
S-Stride Neighborhood Reconstruction Previous diffusion models make point-wise reconstruction, ie., reconstructing each pixel, thus their reverse learning processes can be formulated by po po(ar)_ lar)) In contrast, our context prediction aims to reconstruct x;-1 and further predict its s-stride neighbor- hood contextual representations HAy based on 1; po(x; HAy [zx)), where py is parameterized by two reconstruction networks (t)..)n). Up is designed for the point-wise denoising of xi-1 in ot. and thn is designed for decoding HAY from x;- 1- For denoising i-th point in Tt, we have:
- = (x),,))
where t is the time embedding and 1ip is parameterized by U-Net or transformer with encoder- decoder architecture. For reconstructing the entire neighborhood information HAY around each point xi-1 1- we have:
HN: vin (x)_111)) 4) = the (v)p(,)) ()),
(8)
where vin Rd is the neighborhood decoder. Based on Equation (7) and Equation (8), unify the point- and neighborhood- based reconstruction to form the overall training objective:
xxy Σ
LCONPREDIFF
Mp(ar_1) at) +
Mn (H,,, H
point denoising
content prediction
where x, y are the width and height on spatial axes. a² (if) and H are ground truths. M, and Mn can be Fuclidean distance. In this way, CONPREDIFF is able to maximally preserve local context for better reconstructing each pixel/feature/token..
Interpreting Context Prediction in Maximizing ELBO Mn (H,;, Hx;) Ejex (a and n
We let Mp. be square loss, . y2, where 26. is the j-th neighbor in the context of at iJ is the prediction of a from denoising neural network. Thus we have:
z
5 s/n (v,, (1,, ())))).
(10)

---

Compactly, we can write the denoising network as:
Vin (1p ())()) jEN, (22, (xr,,))(i.)) = ( (1),, j=i.
(11)
We will show that the DDPM loss is upper bounded by ConPreDiff loss. by reparameterizing zy (zt, t). Specifically, for each unit i in the feature map, we use the mean of predicted value in its neighborhood the final prediction:
To (o,, ()( = 1/(WV) + 1)
Σ Va (i.
(12)
JENUU()
Now we can show the connection between the DDPM loss and ConPreDiff loss:
|
2o ( 1)122
Σ
( wo (t: ())))
(
E
(ar () 1))²
JENSUS
E
(V(Z), (((,)) -))) + +1)²,
(13)
JEMUUH
(Cauchy Inequality)
Σ
E (Va),, /623/(1V11 + +1),
JENSUS)
= 1//(WW + +1) Σ (1)))
Σ C - m )
JEN,
In the last equality, we assume that the feature is padded so that each unit has the same number of neighbors I. As result, the ConPreDiff loss is upper bound of the negative log likelihood.
Complexity Problem We note that directly optimizing the Equation (9) has a complexity problem and it will substantially lower the efficiency of CONPREDIFF in training stage. Because the network d/ _2 + R Kd in Equation (8) needs to expand the channel dimension by times for large-context neighborhood reconstruction, it significantly increases the parameter complexity of the model. Hence, we seek for another way that is efficient for reconstructing neighborhood information.
We solve the challenging problem by changing the direct prediction of entire neighborhoods to the prediction of neighborhood distribution. Specifically, for each æi_1. the neighborhood infor- mation is represented as an empirical realization of i.i.d. sampling Q elements from PNy. where
PN; A 1 EwEN; 8n., Based on this view, we are able to transform the neighborhood prediction Mn into the neighborhood distribution prediction. However, such sampling-basee measurement loses original spatial orders of neighborhoods, and thus (Wasserstein distance) for optimization. Wasserstein distance [23, 21] is an effective metric for mea- suring structural similarity between distributions, which is especially suitable for our neighborhood distribution prediction. And we rewrite the Equation (9) as:
permutation invariant loss
BXH Σ
LCOSPREDIFE
Mp(ai-1, &")
W ()) neighborhood distribution prediction
(14)
point denoising
where via ( t) is designed to decode neighborhood distribution parameterized by feedforward neural networks (FNNs), and (2(..)) is the 2- Wasserstein distance as defined below. We provide a more explicit formulation of W ( (x;_ t), PAçe) in Sec. 4.2.
Definition 4.1. Let P, Q denote two probability distributions with finite second moment defined on
C _" The 2-Wasserstein distance between and defined on Z, Z' C 1 is the solution to the optimal mass transportation problem with & transport cost [90]:
W2PP, Q)
inf (ET(P
/
1 - z' 1'2'' 27)
1/2
(15)
zxz'
where I'(P, Q) contains all joint distributions of (Z, Zz) with marginals P. and Q respectively.
5

---

4.2 Efficient Large Context Decoding
Our CONPREDIFF essentially represents the node neighborhood HN; as a distribution of neighbors' representations Pv: (Equation (14)). We adopt Wasserstein distance characterize the distribution reconstruction loss because such has atomic non-zero measure supports in a continuous space, where as KL--divergence cannot be applied. Maximum mean discrepancy
the family of f-divergences may be applied but it needs to choose specific kernel function.
We define the decoded distribution 1i,, (.)_ t) distribution parameterized by wt_1 and The reason for choosing this setting stems from the fact that the universal approximation capability of FNNs allows to (approximately) reconstruct any distributions in I-Wasserstein distance, as formally stated in Theorem 4.2. proved Lu & Lu [48]. To enhance the empirical performance, our case adopts the 2-Wasserstein distance and an FNN with d-dim output instead of the gradient of an FNN with 1-dim outout. Here, the reparameterization trick [42] needs to be used:
FNN- transformation of Gaussian
Vi (= (E). 5 No Ei),
(16)
14 = (T)
(Fag(exp(FNN,
Theorem 4.2. For any e > 0, the support of the distribution p(() is confined to bounded space of
RR, there exists a FNN ul-): Rd + R (and thus its gradient Vu(() : Rd + +Rd) with sufficiently large
width and depth (depending on E) such that (?!) Vu(G)) < where Vu(g) is the distribution generated through the mapping Vu((). E ~ d-dim non-degenerate Gaussian distribution.
Another challenge is that the Wasserstein distance between tú (x)-1:t) closed form. Thus, we utilize the empirical Wasserstein distance that can provably approximate the population one as in Peyré et al. [57]. For each forward pass, our CONPREDIFF will get q sampled target pixel/feature points ( J S 9} from PAr: Next, get samples from N (H4, Eg). denoted by E1, E2. Eq. and thus {अ(..]) = (E) S j S q} are q samples from the prediction v/n ( (); Adopt the following empirical surrogated loss of ((())((()))))) in Equation (14):
1) and PAç- does not have
(z)-1: Px;)
min
| far (i.))
pred 13 ((,+)))
s.t. is bijective mapping: [4] [g].
(17)
The loss function is based on solving a matching problem and needs the Hungarian algorithm with Oa²²) complexity [33]. A more efficient surooient loss may be needed, such as Chamfer loss based on greedy approximation [18] or Sinkhorn loss based on continuous relaxation [11], whose complexities are O(g²). In our study, as is set to a small constant, we use Equation (17) based on a Hungarian matching and do not introduce much computational overhead. The computational efficiency of design empirically demonstrated in Sec. 5.3.
4.3 Discrete and Continuous CONPREDIFF
In training process, given previously Ft. our CONPREDIFF simultancously predict both I;-1 and the neighborhood distribution PNς around each pixel/feature. Because æ can be pixel, feature or discrete token of input image, we can generalize the CONPREDIFF to existing discrete and continuous backbones form discrete and continuous CONPREDIFF. More concretely, we can substitute the point denoising part in Equation (14) alternatively with the discrete diffusion term Cdis (Equation (3)) or the continuous (Equation (6)) diffusion term frs for generalization:
xXy WE ()
dis CONPREDITT = -Cti
A
f=1
(18)
2Xy
cown CONPREDIEE
C A
Σ
())
where X, E [0. 1] is a time-dependent weight parameter. Note that our CONPREDIFF only performs context prediction in training for optimizing the point denoising network dh,, and thus does not

---

LDD
Imagen
ConPreDiff (Ours)
"The sunlight shines on the leaves, and
"Some ancient stone piliars stand upon the earth, arranged spaced apart from each other."
"There are black suits and white pants hanging
rainy day street, the air near the ground is filled with raindrops."
"A bunch flowers bloomed the grass, with dewdrops
LDM
Imagen
ConPreDiff (Ours)
"The blankated snow, ground bay edorned with stones."
box contains apples, each displaying touch green hue,"
"A reinbow rise after the group shoap graze in the fields. Cows are black and sheep
"An elderly fisherman sits fishing aside, gazing towards the distant horizon. The last the ripples on the water's surface."
"On town's nighttime streets, strips
pulled up busy
People dressed in various attire and holding umbrellas strolled along the streets."
Figure 2: Synthesis examples demonstrating text-to-image capabilities of for various text prompts with LDM, Imagen, and ConPreDiff (Ours). Our model can better express local contexts and semantics of the texts marked blue.
introduce extra parameters the inference stage, which is computationally efficient. Equipped with our proposed context prediction term, existing diffusion models consistently gain performance promotion. Next, use extensive experimental results to prove the effectiveness.
5
Experiments
5.1 Experimental Setup
Datasets and Metrics for evaluation: CelebA-HQ [34], FFHQ [35], L.SUN-Church-outdo [102], and L.UNN-beroomss [102]. We evaluate the sample quality and their coverage the data manifold using FID [26] and Precision ancaRe [43]. For text-to-image generation, train the model with LAION [73, 74]
Regarding unconditional image generation, we choose four popular datasets
7

---

Table I: Quantitative evaluation of FID MS-COCO for 256 256 image resolution.
Zero-shot
Approach
Model Type
FID-30K
FID-30K
AttnGAN [96]
GAN
35.49
DM-GAN [113]
GAN
32.64
DF-GAN [86]
GAN
21.42
DM-GAN CL [100]
GAN
20.79
XMC-GAN [107]
GAN
9.33
LAFITE [112]
GAN
8.12
Make-A-Scene [22]
Autoregressive
7.55
DALL-E [61]
Autoregressive
17.89
LAFITE [112]
GAN
26.94
LDM [65]
Continuous Diffusion
12.63
GLIDE [54]
Continuous Diffusion
12.24
DALL-E [62]]
Continuous Diffusion
10.39
Improved VQ-Diffusion [85]
Discrete Diffusion
8.44
Simple Diffusion [31]
Continuous Diffusion
8.32
Imagen [69]
Continuous Diffusion
7.27
Parti [104]
Autoregressive
7.23
Muse [7]
Non-Autoreeressive
7.88
eDiff-I [3]
Continuous Diffusion
6.95
CONPREDIFF
Discrete Diffusion Continuous Diffusion
6.67
CONPREDIFF
6.21
and some internal datasets, and conduct evaluations on MS-COCO dataset with zero-shot FID and CLIP score [25, 59], which aim assess the generation quality and resulting image-text alignment. For image inpainting, we choose CelebA-HQ [34] and ImageNet [12] for evaluations, and evaluate all 100 images of the test datasets for the following masks: Wide, Narrow, Every Second Line, Half Image, Expand, and Super-Resolve. We report the commonly reported perceptual metric LPIPS [109], which is learned distance metric based on the deep feature space.
Baselines To demonstrate the effectiveness of CONPREDIFF, we compare with the latest dif- fusion and non-diffusion models. Specifically, for unconditional image generation, we choose ImageBART( 16,, U-Net GAN (+aug) [72]. UDM [39], StyleGAN [36], ProjectedGAN [71], DDPM [28] and ADM [14] for comparisons. As for text-to-image generation, we choose DM-GAN [113],
DF-GAN [86], DM-GAN CL [100], XMC-GAN [107] LAFITE [112], Make-A-Scene [22], DALL- [61], LDM [65], GLIDE [54], DALL-E [62], Improved VQ-Diffusion [85], Imagen-3.4B [69], Parti [104], Muse [7], and eDiff-I [3] for comparisons. For image inpainting, choose autoregres-
sive methods( DSI [56] and ICT [91]). the GAN methods (DeepFilly2 [103], AOT [105], and LaMa
[84]) and diffusion based model (RePaint [49]). All the reported results are collected from their
published papers or reproduced by open source codes.
Implementation Details For text-to-image generation, similar to Imagen [68], our continuous diffusion model CONPREDIFF consists of base text-to-image diffusion model (64x64) [53], two super-resolution diffusion models [29] upsample the image, first 64 256 256, and then 256x256 - 1024x 1024. The model is conditioned on both T5 [60] and CLIP [59] text embeddings. The T5 encoder is pre-trained on a C4 text-only corpus and the CLIP text encoder trained on an image-text corpus with an image-text contrastive objective. We use the standard Adam optimizer with learning rate of 0.0001, weight decay of 0.01, and a batch size of 1024 to optimize the base model and two super-resolution models on NVIDIA 1100 GPUs, respectively. equipped with multi-scale training technique image scales). We generalize our context prediction discrete diffusion models [24, 85] to form our CONPREDIFF For image inpainting, we adopt a same pipeline as RePaint [49], and retrain diffusion backbone with our context prediction loss. We use T 250 time steps, j= 10. For unconditional generation tasks, we use the same denoising architecture like LDM [65] for fair comparison. The max channels are 224, and use T=2000 time steps, linear noise schedule and an initial learning rate of 0.000096.
and applied
10 times resampling with jumpy size

---

Table 2: Quantitative evaluation of image inpainting on CelebA-HQ and ImageNet.
Ce
Wide LPIPS 4
Narrow LPIPS
Super-Resolve LPIPS
Altern. Lines
Half LPIPS 4
Expand
Method
LPIPS
LPIPS
AOT [105]
0.104
0.047
0.714
0.667
0.287
0.604
DSI [56]
0.067
0.038
0.128
0.049
0.211
0.487
ICT [91]
0.063
0.036
0,483
0.353
0.166
0,432
DeepFillv2 [103]
0.066
0.049
0.119
0.049
0.209
0.467
LaMa [84]
0.045
0.028
0.177
0.083
0.138
0.342
RePaint [49]
0.059
0.028
0.029
0.009
0.165
0.435
CONPREDIFF
0.042
0.022
0.023
0.022
0.139
0.297
ImageNet
Wide
Narrow LPIPS
Super-Resolve 2x LPIPS
Altern. Lines LPIPS
Half
Expand
Method
LPIPS +
LPIPS &
LPIPS
DSI [56]
0.117
0.072
0.153
0.069
0.283
0.583
ICT [91]
0.107
0.073
0.708
0.620
0.255
0.544
LaMa [84]
0,105
0.061
0.272
0.121
0.254
0,534
RePaint [49]
0.134
0.064
0.183
0.089
0.304
0,629
CONPREDIFF 0.098
0.057
0.129
0.107
0.285
0,506
Our context prediction head contains two non-lineat blocks (e.g.. Conv-BN-ReLL resnet block or transformer block), and its choice can be flexible according to specific task. The prediction head does not incur significant training costs, and can be removed in inference stage without introducing extra testing costs. We set the neighborhood stride to for all experiments, and carefully choose the specific layer for adding context prediction head near the end of denoising networks.
5.2 Main Results
Text-to-Image Synthesis We conduct text-to-image generation on MS-COCO dataset, and quan- titative comparison results are listed in Tab. 1. We observe that both discrete and continuous CONPREDIFF substantially surpasses previous diffusion and non-diffusion models terms of FID score, demonstrating the new state-of-the-arr performance. Notably, our discrete and continuous CONPREDIFF achieves an FID score of 6.67 and 6.21 which are better than the score of 8.44 and 7.27 achieved by previous SOTA discrete and continuous diffusion models. We visualize text-to-image generation results Figure 2, and find that our CONPREDIFF can synthesize images that are semantically better consistent with text prompts. It demonstrates our CONPREDIFF can make promising cross-modal semantic understanding through preserving visual context information generating process, and effectively associating with contextual text information. Moreover, we observe that CONPREDIFF can synthesize complex objects and scenes consistent with text prompts as demonstrated by Figure in Appendix A.4, proving the effectiveness of our designed neighborhood context prediction. Human evaluations are provided in Appendix A.S.
Image Inpainting Our CONPREDIFF naturally fits image inpainting task because we directly predict the neighborhood context of each pixel/feature in diffusion generation. We compare our CONPREDIFF against ttate-of-the-an benchmarking. As in Tab. 2, our CONPREDIFF outperforms previous SOTA method for most kinds of masks. We also put some qualitative results in Figure 3, and observe that CONPREDIFF produces a semantically meaningful filling, demonstrating the effectiveness of our context prediction.
standard mask distributions, commonly employed for
Unconditional Image Synthesis We list the quantitative results about unconditional image genera- tion Tab. of Appendix A.3. We observe that our CONPREDIFF significantly improves upon the oate-of-tth-art in FID and Precision and---an---eee mnc-R scores on FFHQ and LSUN-Bedroome Bedrooms datasets. The CONPREDIFF obtains high perceptual quality superior to prior GANs and diffusion models, while maintaining higher coverage of the data distribution measured by recall.
5.3 The Impact and Efficiency of Context Prediction
In Sec. 4.2, we tackle the complexity problem by transforming the decoding target from entire neighborhood features to neighborhood distribution. Here we investigate both impact and efficiency of the proposed neighborhood context prediction. For fast experiment, we conduct ablation study with the diffusion backbone of LDM [65]. As illustrated in Figure 4, the FID score of CONPREDIEE

---

Original Image
Masked Image
CONPREDDIY
Figure 3: Inpainting examples generated by our CONPREDIFF.
better with the neighbors of more strides and 1-stride neighbors contribute the most performance gain, revealing that preserving local context benefits the generation quality. Besides, we observe that increasing neighbor strides significantly increases the training cost when using feature decoding. while it has little impact distribution decoding with comparable FID score. To demonstrate the generalization ability, we equip previous diffusion models with our context prediction head. From results in Figure 5, we find that our context prediction can consistently and significantly improve the FID scores of these diffusion models, sufficiently demonstrating the effectiveness and extensibility of method. We further conduct ablation study the trade-off between CLIP and FID scores across a range of guidance weights in Appendix A.2, the results exhibit our superior generation ability.
Ablation Study
Effectiveness Context Prediction
Feature Distrbution
Feature
Original Model
13.0
Distribution
-8
Context Prediction
Required Secondsfii
FID Score
12.0
10
0 11.5
10.5
Baseline S-Srid 2-Stride 3-Stride 4-5tride 5-Stride
LDM
DALL-E 2 Improved VOD Imagen
Figure 4: Bar denotes FID and line denotes time cost.
Figure 5: Equip diffusion models
with our context prediction.
Conclusion
In this paper, for the first time propose CONPREDIFF improve diffusion-based image synthesis with context prediction. We explicitly force each point to predict its neighborhood context with an efficient context decoder near the end of diffusion denoising blocks, and remove the decoder for inference. CONPREDIFF can generalize to arbitrary discrete and continuous diffusion backbones and consistently improve them without extra parameters. We achieve new SOTA results on unconditional image generation, text-to-image generation and image inpainting tasks.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (No.61 61830013 and U22B2037).
10

---

References
[1] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Struc- tured denoising diffuusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021. 1,2
[2] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In JEEE Conference on Computer Vision and Pattern Recognition, pp.
19208-18211 2022.
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to- -moge diffusion models with an ensemble of expert denoisers. arXiv preprint 1xXiv:221..01322 2022. 8
[4] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference Learning Representations, 2021. 1
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018. 1
[6] Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li.
A survey on generative diffusion model. arXiv preprint urXiv:2209.026444 2022. 2
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hssan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint (exxix2331.007000 2023. 8
[8] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin. Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.030026, 2022. 2,
[9] Florinel-Alin Croitoru. Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. arXiv preprint arXiv:2209.04744 044422779.4777, 2022. 2
[10] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204.0855 2022.
[11] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 2013,
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009. 8
[13] Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, 2019. 2
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 19
2021. 1, 2,
[15] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015. 2,
[16] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirec- tional context with multinomial diffusion for autoregressive image synthesis. Advances in Neural Information Processing Systems, 34:3518-3523 2021. 8, 19

---

[17] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 12873-12883, 2021. 2, 19
[18] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR, 2017. 6
[19] Wan- Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. arXiv preprint arXiv:2208. 2022. 2
[20] Mary Anne Franks and Ari Ezra Waldman. Sex, lies, and videotape: Deep fakes and free speech delusions. Md. L. Rev., 78:892, 2018. 18
[21] Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya-Polo, and Tomaso A Poggio. Learning with a wasserstein loss. In NeurlPS, 2015. 2,5
[22] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene based text-to-image generation with human priors. arXiv preprint (rXiv:2203.13131 2022. 1, 8
[23] Clark R Givens and Rae Michael Shortt. A class of wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31(2):231-240, 1984. 5
[24] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In /EEE Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022. 1, 3, 3,
[25] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021. 8
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 7, 18
[27] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.125540 2022. 1,2
[28] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020. 1, 2, 4,
8, 19
[29] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23:47-1, 2022. 1, 8
[30] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, volume 34, pp. 11454-12465, 2021. 1,2
[31] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans, simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:2301. 2023. 8
[32] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, pp. 3464-3473, 2019. 2,
[33] Roy Jonker and Anton Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38(4):325 -300, 1987.
[34] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Represen- tations, 2018. 7,
12

---

[35] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019. 7
[36] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019. 8, 19
[37] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 11100-11 8118,119,,,111,, 2020. 19
[38] Bahjat Kawar, Shiran Zada, Oran lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint 17Xi::2210009276 2022.
[39] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Score matching model for unbounded data score. arXiv preprint urXiv:2/06.05527 2021. 8, 19
[40] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2426 2426-2435, 2022. 1,2
[41] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 2
[42] Diederik P Kingma and Max Welling. Auto-encoding variational bayess. ICLR, 2014.
[43] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019. 7, 19
[44] Zhi Lei, Guixian Zhang, Lijuan Wu, Kui Zhang. and Rongjiao Liang. A multi-level mesh mutual attention model for visual question answering. Data Science and Engineering, 7(4): 339-353, 2022. 3
[45] Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In ICCV, pp. 4170-4179, 2019. 2,
[46] Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, pp. 14429-14460, 2022. 1
[47] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206. 2.X0022206.00927, 2022.
[48] Yulong Lu and Jianfeng Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. NeurlPS, 2020.
[49] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11661
1 11771, June 2022. 8,
[50] Jiayi Ma. Ji Zhao, Junjun Jiang, Huabing Zhou, and Xiaojie Guo. Locality preserving matching. International Journal of Computer Vision, 127(5):512-55 531, 2019. 2,
[51] Chenlin Meng. Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[52] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013. 2
13

---

[53] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, 2021. 1.
[54] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804, 2022. 1, 2,
[55] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive gen- erative autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 823-832, 2021. 19
[56] Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical vae. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10775-10784, 2021. 8,
[57] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-60 300777 2019. 6
[58] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom.. Diffusion autoencoders: Toward meaningful and decodable representation. In IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, 2021. 2,8 8
[60] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified texi-to-text transformer. J. Mach. Learn. Res., 21 (110))11163) 2020. 2, 8
[61] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18 24 Jul 2021. 8
[62] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06/25, 2022. 2, 8, 21
[63] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in Neural Information Processing Systems, 32, 2019. 1
[64] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pp. 1278-1286, 2014. 2
[65] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition. pp. 10684-10695, 2022. 1, 2, 8, 9, 19
[66] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
[67] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-imas iooee diffusion models. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, pp.
1-10, 2022. 1,2
[68] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 11487, 2022. 1, 1, 8, 21
14

---

[69] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.1148)) 2022. 8
[70] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[71] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34:1 34:17480-17492, 2021. 8, 19
[72] Edgar Schönfeld, Bernt Schiele, and Anna Khoreva. u-net based discriminator for generative adversarial networks. In CVPR, pp. 8204-8213, 2020. 8, 19
[73] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021. 7
[74] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 -57294, 2022. 7
[75] Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. Kan-diffusion: Image generation via large-scale retrieval. arXiv preprint 3xXiv:224400284/ 2022. 2
[76] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 1256-2266 2015. 1,
[77] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
[78] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32, 2019. 1,
[79] Yang Song and Stefano Ermon. Improved techniques for training score-basce based generative models. In Advances Neural Information Processing Systems, volume 33, pp. 12438-12448, 2020.
[80] Yang Song. Jascha Dohk-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020. 1,2
[81] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of bosee diffusion models. In Advances in Neural Information Processing Systems, volume 34, pp. 1415-1428, 2021.
[82] Ramya Srinivasan and Kanji Uchino. Biases in generative art: A causal look from the lens of art history. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 41-51, 2021. 18
[83] Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM conference on fairness, account- ability, and transparency, pp. 701-713, 2021. 18
[84] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempit- sky. Resolution-robust large mask inpainting with fourier convolutions. Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 2149-2159, 2022, 8,9
15

---

[85] Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv preprint arXiv:2205. 16007, 2022. 2,
[86] Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing- Bing-Kun Bao, and Changsheng Xu. Df-gan:
A simple and effective baseline for text-to-image synthesis. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16515-16525, 2022. 8
[87] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems, volume 34, pp- 12.11122 11287-11302, 2021.
1, 2, 19
[88]
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[89] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 2
[90] Cédric Villani. Optimal transport: old and new, volume 338. 2008. 5
[91] Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High fideliiiy pluralistic image completion with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4692-4701, 2021. 8, 9
[92] Haixin Wang, Jianlong Chang, Xiao Luo, Jinan Sun, Zhouchen Lin, and Qi Tian. Lion: Implicit vision prompt tuning. arXiv preprint arXiv:2303.0991 2023. 18
[93] Haixin Wang, Xinlong Yang, Jianlong Chang, Dian Jin, Jinan Sun, Shikun Zhang, Xiao Luo, and Qi Tian. Mode approximation makes good vision-language prompts. arXiv preprint arXiv:2305.083383 2023. 18
[94] Jing Wang, Yehao Li, Yingwei Pan, Ting Yao, Jinhui Tang, and Tao Mei. Contextual and selective attention networks for image captioning. Science China Information Sciences, 65 (12):222103, 2022.
[95] Meng Wang, Yinghui Shi, Han Yang. Ziheng Zhang, Zhenxi Lin, and Yefeng Zheng. Probing the impacts of visual context in multimodal entity alignment. Data Science and Engineering, 8(2):124-134 2023.
[96] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp.
2018.
[97] Ling Yang, Liangliang Li, Zilun Zhang, Xinyu Zhou, Erjin Zhou, and Yu Liu. Dpgn: Distri- bution propagation graph network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition. pp. 13390-13399 2020. 18
[98] Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming- -ssan Yang. Diffusion-based scene graph to image generation with masked contrastive gre-training. arXiv preprint arXiv:2211.11138 2022. 2
[99] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 2022. 1,2
[100] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to-image synthesis using contrastive learning. arXiv preprint 1.Xi:2207700222. 2021.
[101] Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al. Visual tuning. arXiv preprint arXiv:2305.0600 2023. 18
16

---

[102] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 150.00333 2015.
[103] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018. 8.9
[104] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprini 220. 2022.
21
[105] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Aggregated contextual transformations for high-resolution image inpainting. IEEE Transactions on Visualization and Computer Graphics, 2022. 8, 9.9
[106] Dong Zhang, Liyan Zhang, and Jinhui Tang. Augmented fen: rethinking context modeling for semantic segmentation. Science China Information Sciences, 142105, 2023. 3
[107] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modis contrastive learning for text-to-image generation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 233-842, 2021.
[108] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902 2022.
[109] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp- 586-595, 2018.
[110] Xinyu Zhang, Jiahui Chen, Junkun Yuan, Qiang Chen, Jian Wang, Xiaodi Wang, Shumin Han, Xiaokang Chen, Jimin Pi, Kun Yao, et al. Cae v2: Context autoencoder with clip target. arXiv preprint arXiv:2211.00 01.222211099999 1.09799, 2022. 2,
[111] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighbor- hood features for point cloud processing. In CVPR, pp. 5565-5573, 2019. 2, 3
[112] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free training for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17907-17917, 2022. 8
[113] Minfeng Zhu, Pingho Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5802-5810, 2019. 8
[114] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross-modal and conditional generation. arXiv preprint arXiv:2006.0777,, 2022.
2
17

---

A Appendix
A.1 Limitations and Broader Impact
While our ConPreDiff boosts performance of both discrete and continuous diffusion models without introducing additional parameters in model inference, our models still have more trainable parameters than other types of generative models, GANs. Furthermore, we note the long sampling times of both and compared to single step generative approaches like GANs or VAEs. However, this drawback is inherited from the underlying model class and is not property of our context prediction approach. Neighborhood context decoding is fast and incurs negligible computational overhead in training stage. For future work, we will try to find more intrinsic information to preserve for improving existing point-wise denoising diffusion models, and extend to more challenging tasks like text-to-3D and text-to-video generation.
Limitations
Broader Impact Recent generative image models enable creative applications and autonomous media creation, but can also be viewed as dual-use technology with negative implications. In this paper, we use human face datasets only for evaluating the image inpainting performance of our method, and our method is not intended to create content that is used mislead deceive. However, like other related image generation methods, it could still potentially be misused in the realm of human impersonation. A notorious example so-called "deep fakes" that have been used, for example, to create pornographic "undressing" applications. We strongly disapprove of any actions aimed at producing deceptive methods have the capacity to be harnessed for other malicious intentions, including harassment and misinformation spread [20], and give rise to significant concerns pertaining to societal and cultural exclusion as well as biases [83, 82). These considerations guide our decision not to release the source code public demo at this point in time.
harmful content featuring real individuals. Besides, generative
Furthermore, the immediate availability of mass mass- produced high-quality images can be used to spread misinformation and spam, which in turn can be used for targeted manipulation in social media. Datasets are crucial for deep learning as they are the main input information [101, 92, 93, 97]. Large-scale data requirements of text-to-image models have led researchers rely heavily large, mostly uncurated, web-scraped datasets. While this approach has enabled rapid algorithmic advances recently, datasets of this nature have been critiqued and contested along various ethical dimensions. One should consider the ability to curate the database to exclude (or explicitly contain) potential harmful source images. Creating public API could offer cheaper way offer safe model than retraining a model on a filtered subset of the training data or doing difficult prompt engineering. Conversely, including only harmful content is easy way build a toxic model.
A.2 Guidance Scale vs. FID
To further demonstrate the effectiveness of our proposed context prediction, we quantitatively conduct evaluations about the trade-off between MS MS-COCO zero-shot FID [26] and CLIP scores. The results in Figure 6 indicate that the guidance hurts the diversity of GLIDE much more than DALL-E 2 and CONPREDIFF. The phenomenon reveals that the proposed CONPREDIFF can overall improve the generation quality of diffusion models.
18
DALLE-2
16
ConPreDiff
FID Score
14
12
10
2.0
2.5 Scale
Guidance
Figure 6: Trade-off between guidance scale and FID.
18

---

A.3 More Quantitative Results
We list the unconditional generation results on FFHQ. CelebA-HQ, LSUN-Churches, and LSUN- Bedrooms in Tab. 3. We find CONPREDIFF consistently outperforms previous methods, demonstrat- ing the effectiveness of the CONPREDIFF.
Table 3: Evaluation results for unconditional image synthesis.
FFHQ 256
256
Method
FID
Prec.
Recall f
ImageBART(16]
9.57
U-Net GAN (+aug) [72]
7.6
UDM [39]
5.54
StyleGAN [36]
4.16
0.71
0.46
ProjectedGAN [71]
3.08
0.65
0.46
LDM [65]
4.98
0.73
0.50
CONPREDIFF
2.24
0.81
0.61
LSUN Bedrooms 256
256
Method
FID
Prec.
Recall f
ImageBART [16]
5.51
DDPM [28]
4.9
UDM [39]
4.57
StyleGAN [36]
2.35
0.59
0.48
ADM [14]
1.90
0.66
0.51
ProjectedGANN [71]
1.52
0.61
0.34
LDM-4 [65]
2.95
0.66
0.48
CONPREDIFF
1.12
0.73
0.59
CelebA-HQ 256 x 256
Method
FID
Prec.
Recall 1
DC- DC-VAE [55]
15.8
VQGAN+T. [17] (k=400)
10.2
PGGAN [43]
8.0
LSGM [87]
7.22
UDM [39]
7.16
LDM [65]
5.11
0.72
0.49
CONPREDIFF
3.22
0.83
0.57
LSUN- LSUN-Churches 256 x 256
Method
FID
Prec.
Recall 1
DDPM [28]
7.89
ImageBART [16]
7.32
PGGAN [43]
6.42
StyleGAN [36]
4.21
StyleGAN2 [37]
3.86
ProjectedGAN [71]
1.59
0.61
0.44
LDM [65]
4.02
0.64
0.52
CONPREDIFF
1.78
0.74
0.61
A.4 More Synthesis Results
We visualize more text-to-image synthesis results on MS-COCO dataset in Figure 7. We observe that compared with previous powerful LDM and DALL-E 2, our CONPREDIFF generates more natural and smooth images that preserve local continuity.
19

---

"A leddy bear sitting chair.
"A holding bunch of bananas on a table."
"A photo Goth
"A group of elephants walking muddy water."
"Grean frog green grass"
"The plane wing above the clouds.
big round hole brick wall"
"Reflection of lake"
orange ball is put the ground
"A red bydrant sitting in the
" "Trees Alrican grassland" "
" "Cat fiell asleup the owner's bed
"A photo of white garlic ice cream"
Pancakes with ketchup
"A photo of an adult lion."
Figure 7: Synthesis examples demonstrating text-to-image capabilities various text prompts.
20

---

A.5 Human Evaluations
As demonstrated in qualitative results, our CONPREDIFF is able to synthesize realistic diverse, context-cohenee images. However, using FID to estimate the sample quality is not always consistent with human judgment. Therefore, we follow the protocol of previous works [104, 68, 62], and conduct systematic human evaluations to better assess the generation capacities of our COF from the aspects of image photorealism and image-text alignment. We conduct side-by-side human evaluations, in which well-trained users are presented with two generated images for the same prompt and need choose which image is of higher quality and more realistic (image photorealism) and which image better matches the input prompt (image-text alignment). For evaluating the coherence of local context, we propose new evaluation protocol, which users are presented with 1000 pairs of images and must choose which image better preserves local pixel/semantic continuity. The evaluation results are in Tab, 4, CONPREDIFF performs better in pairwise comparisons against both Improved VQ- Og-Diffusion and Imagen. We find that CofFREDifF is preferred in terms of all three evaluations, and CONFREDIFF is strongly preferred regarding context coherence, demonstrating that preserving local neighborhood context advances sample quality and semantic alignment.
Table 4: Human evaluation comparing CONPREDIFF to Improved VQ- VQ-Diffusion and Imagen.
Improved VQ-Diffusion
Imagen
Image Photorealism
72%
65%
Image-Text Alignment
63% 68%
Context Coherence
84%
78%
21


Reviews:
[{"review_id": "UiMN0vAhZQ", "reviewer": "Reviewer_m5Qc", "summary": "This paper is proposing context-aware Diffusion Models. They make the models learn the context information by setting up auxiliary networks to estimate the neighbor distributions from the estimated denoised sample from Diffusion Models. The benefit of this approach is that additional cost from the auxiliary networks are not applied during the sampling. Both quantitative and qualitative experiments are reported.", "questions": "\n* The meaning of \"FNN\" is not defined.\n* What is the meaning of “(0)” of the delta function in line 138?\n* Why not let $FNN_{\\mu}$ and $FNN_{\\sigma}$ take target index as additional input and skip Eq. 7? Once the target index is specified, the matching algorithm may not be needed anymore.\n* What is the relationship between $q$ in Eq. 7 and stride $s$ and $K$? \n* It would be better if performance of directly estimating the neighbors is reported as well since it would be more accurate and simple setup to implement the motivation of this paper (even though it has the memory inefficiency, as mentioned in L132).\n* Can it be described more specifically why KL or JSD cannot be applied (L155)? It is not straightforward to me why KL cannot be applied to this task. \n* Are both terms in Eq. 3 used together for finetuning?\n* What happens if Diffusion Models are trained from scratch with the proposed objectives?\n", "limitations": "- The most important part of the paper would be “context prediction” term in Eq . 3. It’s motivation is understandable, but it is not interpreted in terms of optimizing variational bound of negative log likelihood. Is it just “additional” term? or can it be interpreted as a term playing a certain role in maximizing ELBO or likelihood?\n- This paper is proposing to 1. estimate the neighborhood distribution (instead of directly estimating the neighborhood) and 2. minimize Wasstertein distance as a core objective. Though the authors made an attempt to justify the design choice, I believe it could be compared as a sort of ablation study, which might strengthen the proposed method. --- I found Fig.6 and the first concern is resolved.\n- \bI believe training time needs to be compared together in Fig. 7.\n- Although performance improvement is shown by quantitaive experiments, it is not straightforward how the \"context prediction\" makes model performance better (qualitatively).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Motivation is agreeable.\n- Good writing.\n- Reasonable method for motivation.\n- Experiments are done well.", "weaknesses": "Specifics of the weaknesses of this paper are written below as questions and limitations, but I believe most of them can be resolved during the rebuttal. I will increase my rating if my concerns can be resolved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer m5Qc for the positive review and valuable feedback. We are glad that the reviewer found that the motivation is agreeable, the writing is good, the method for motivation is reasonable, and experiments are done well. Please see below for our responses to your comments.*\n\n**Q1: Interpret “context prediction” term from the perspective of maximizing ELBO or likelihood.**\n\nA1: Connection between our context prediction loss in Eq.3 and ELBO:\n\nAt time t: \n\nLoss of DDPM ( after reparamterization and scaling): $||x_0-\\hat{x}_0(x_t,t)||_2^2$\n\nLoss of ConPreDiff:  $\\sum\\_{i = 1}^{x*y} [\\mathcal{M}\\_p(x\\_0^{i} ,\\hat{x}\\_0^{i})+ \\mathcal{M}\\_n(\\mathcal{H}\\_{\\mathcal{N}\\_i},\\hat{\\mathcal{H}}\\_{\\mathcal{N}\\_i})]$\n\nWe let $\\mathcal{M}_p,\\mathcal{M}_n$ be square loss,\n  $\\mathcal{M}_n(\\mathcal{H}\\_{\\mathcal{N}\\_i},\\hat{\\mathcal{H}}\\_{\\mathcal{N}\\_i})=\\sum\\_{j\\in \\mathcal{N}\\_i}(x\\_0^{i,j}-\\hat{x}\\_0^{i,j})^2$,\n\n  where $x_0^{i,j}$ is the j-th neighbor in the context of $x_0^i$ and $\\hat{x}_0^{i,j}$ is the prediction of $x_0^{i,j}$ from a denoising neural network. Following the notation in the main paper, we have\n\n  $\\hat{x}_0^{i,j} = \\psi_n(\\psi_p(x_t,t)(i))(j)$,\n\n  where $\\psi_p(x_t,t)(i)$ is the prediction of $x_0^i$ and $\\psi_n$ is the neighborhood decoder. Compactly, we can write the denoising network (equation (1)) as:\n\n  $$\\Psi(x_t,t)(i,j) =\\left\\\\{ \\begin{array}{l}\\psi_n(\\psi_p(x_t,t)(i))(j), & j \\in \\mathcal{N}\\_i, \\\\\\\\ \\psi_p(x_t,t)(i), & j=i \\end{array} \\right.$$\n\n  We can show that the DDPM loss is upper bounded by ConPreDiff loss, by reparameterizing $\\hat{x}_0(x_t,t)$.\n  Specifically, for each unit i in the feature map, we predict the unit i and its neighbors using equation (1), and then we use the mean of  predicted value in the neighborhood as the final prediction:\n\n  $$\\hat{x}\\_0(x_t,t)(i) = 1/(|\\mathcal{N}\\_i|+1)*\\sum\\_{j \\in \\mathcal{N}\\_{i}\\ \\cup\\ \\\\{i\\\\}}\\Psi(x\\_t,t)(i,j)$$\n\n  Now we can show the connection between DDPM loss with the mean prediction and ConPreDiff loss:\n\n$$\\begin{array}{rl}||x_0-\\hat{x}\\_0(x_t,t)||_2^2 & = \\sum_i (x_0^i-\\hat{x}\\_0(x_t,t)(i))^2,\\\\\\\\ &=\\sum_i (x_0^i-\\sum\\_{j \\in \\mathcal{N}_i\\cup\\{i\\}}\\Psi(x_t,t)(i,j)/(|\\mathcal{N}_i|+1))^2,\\\\\\\\ &=\\sum_i(\\sum\\_{j \\in \\mathcal{N}_i\\cup\\{i\\}}(\\Psi(x_t,t)(i,j)-x_0^i))^2/(|\\mathcal{N}_i|+1)^2, \\\\\\\\ (Cauchy\\ inequality) & \\leq \\sum_i \\sum\\_{j \\in \\mathcal{N}\\_i\\cup\\{i\\}} (\\Psi(x_t,t)(i,j)-x_0^i)^2/(|\\mathcal{N}_i|+1), \\\\\\\\ &=1/(|\\mathcal{N}\\_i|+1)\\sum_i [(x_0^i- \\psi_p(x_t,t)(i))^2+\\sum\\_{j \\in \\mathcal{N}\\_i} (x_0^{i,j}-\\hat{x}\\_0^{i,j})^2] \\end{array}$$\n  \n  \n  in the last equality, we assume that the feature is padded so that each unit i has the same number of neighbors $|\\mathcal{N}|$.\n  As a result, the ConPreDiff loss is an upper bound of the negative log likelihood.\n\n\n**Q2: Training time comparison.**\n\nA2: We report the approximate time cost (seconds) of these models to train in a single step on the MS-COCO dataset in the table below. The additional context prediction head added to the models accounts for only a small portion of the parameters (0.13M), and the neighborhood distribution decoding of our method also does not incur significant additional training costs.\n\n|Training cost (secs/step) |LDM | DALL·E 2| Improved VQ-Diffusion| Imagen|\n| :-----| :----: | :----: |:----: |:----: |\n|Original Model  | 11.9 | 74.7 | 83.9 |198.3 |\n| + Context Prediction | 13.1 | 77.5 |87.6 |206.3 |\n\n**Q3: Qualitative comparison.**\n\nA3: More visualization results can be found in **global response pdf**, ConPreDiff better expresses local contexts and consistent semantics in generated images compared to other methods.\n\n\n**Q4: The meanings of \"FNN\" and “(0)” in line 138.**\n\nA4: We first mention FNN in L145 with feedforward neural networks, which contains two linear-BN-ReLU modules in our experiments. \"(0)\" is the superscript of $h_u$, which denotes the center point of neighborhoods $N^s_{i}$ (i.e., $h_i$).\n\n**Q5: Why not let FNN take target index as additional input and skip Eq. 7?**\n\nA5: Our distribution decoding loses original spatial orders of neighborhoods, and thus we use a **permutation invariant loss (Wasserstein distance)** for optimization. However, the Wasserstein distance between decoded neighborhood distribution and ground truth does not have a closed form. Thus we design **Eq.7 as an empirical surrogated loss of Wasserstein distance**. Taking target index as additional input may not well approximate Wasserstein distance.\n\n**Q6: What is the relationship between q in Eq.7 and stride s and K?**\n\nA6: \n1. $K={(2s+1)}^2-1$\n2. $q<K$\n\n**Q7: It is better to report the performance of directly estimating the neighbors as well.**\n\nA7: As you found in the Fig.6 of our paper, direct estimation of neighbors slightly improves the generation results but significantly incurs more training costs with larger strides. And our distribution decoding can achieve a better trade-off between FID score and training cost.\n\n\n**Q8: Why KL or JSD cannot be applied (L155)?**\n\nA8: Wasserstein distance can effectively measure structural similarity and impose structural constraint between the decoded distribution and the ground truth, via an optimal-transport loss. And Wasserstein distance can measure the similarity **between continuous and discrete distributions**, but KL or JSD can not well characterize the structural similarity between such distributions. \n\n**Q9: Are both terms in Eq. 3 used together for finetuning? What happens if Diffusion Models are trained from scratch with the proposed objectives?**\n\nA9: Both terms are used together. The convergence of our ConPreDiff would be quicker than the diffusion models that only use point-based reconstruction objective, because both terms play a cooperative role in maximizing ELBO as demonstrated in above responses."}, {"review_id": "ajzGhaexcx", "reviewer": "Reviewer_qzoS", "summary": "This paper proposes an idea of context prediction to boost difussion-based image generation.\nThe core idea is that in each step of diffusion, after the denoised point is generated, neighborhood context prediction is performed.\nIn particular, to maintain the spatial orders of the neighborhood, a permutation invariant loss is used for optimization by replacing the context prediction with neighborhood distribution prediction. Performance improvement against standard diffusion models were presented. in experiments.\n", "questions": "1. By introducing context, is any sign of \"blurriness\" introduced?\n2. What are the typical cases that do worse compared to standard diffusion? ", "limitations": "Similar with text2image papers.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The idea is very interesting and sound.\nFrom an image denoising point of view, neighborhood info is commonly used, so it's a natural extension of diffusion-denoising models.\n2. Performance improvement showed in experiments are promising. ", "weaknesses": "The proposed approach probably takes longer to train. Can you discuss from that perspective?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer qzoS for the positive review and valuable feedback. We are glad that the reviewer found that the idea is very interesting and sound, and performance improvement showed in experiments is promising. Please see below for our responses to your comments.*\n\n**Q1: The proposed approach probably takes longer to train. Can you discuss from that perspective?**\n\nA1: Thanks for your interesting question, we here discuss this problem from two training scenarios: \n\n1.Training ConPreDiff from scratch\n\nWe introduce the \"context prediction\" term as in Eq.1. Actually the proposed \"context prediction\" term plays a additional role in maximizing ELBO and will accelerate the convergence (i.e. less training steps) of diffusion models. We here provide the formula derivation：\n\nAt time t: \n\nLoss of DDPM ( after reparamterization and scaling): $||x_0-\\hat{x}_0(x_t,t)||_2^2$\n\nLoss of ConPreDiff:  $\\sum\\_{i = 1}^{x*y} [\\mathcal{M}\\_p(x\\_0^{i} ,\\hat{x}\\_0^{i})+ \\mathcal{M}\\_n(\\mathcal{H}\\_{\\mathcal{N}\\_i},\\hat{\\mathcal{H}}\\_{\\mathcal{N}\\_i})]$\n\nWe let $\\mathcal{M}_p,\\mathcal{M}_n$ be square loss,\n  $\\mathcal{M}_n(\\mathcal{H}\\_{\\mathcal{N}\\_i},\\hat{\\mathcal{H}}\\_{\\mathcal{N}\\_i})=\\sum\\_{j\\in \\mathcal{N}\\_i}(x\\_0^{i,j}-\\hat{x}\\_0^{i,j})^2$,\n\n  where $x_0^{i,j}$ is the j-th neighbor in the context of $x_0^i$ and $\\hat{x}_0^{i,j}$ is the prediction of $x_0^{i,j}$ from a denoising neural network. Following the notation in the main paper, we have\n\n  $\\hat{x}_0^{i,j} = \\psi_n(\\psi_p(x_t,t)(i))(j)$,\n\n  where $\\psi_p(x_t,t)(i)$ is the prediction of $x_0^i$ and $\\psi_n$ is the neighborhood decoder. Compactly, we can write the denoising network as:\n\n  $$\\Psi(x_t,t)(i,j) =\\left\\\\{ \\begin{array}{l}\\psi_n(\\psi_p(x_t,t)(i))(j), & j \\in \\mathcal{N}\\_i, \\\\\\\\ \\psi_p(x_t,t)(i), & j=i \\end{array} \\right.$$\n\n  We can show that the DDPM loss is upper bounded by ConPreDiff loss, by reparameterizing $\\hat{x}_0(x_t,t)$.\n  Specifically, for each unit i in the feature map, we predict the unit i and its neighbors using equation (1), and then we use the mean of  predicted value in the neighborhood as the final prediction:\n\n  $$\\hat{x}\\_0(x_t,t)(i) = 1/(|\\mathcal{N}\\_i|+1)*\\sum\\_{j \\in \\mathcal{N}\\_{i}\\ \\cup\\ \\\\{i\\\\}}\\Psi(x\\_t,t)(i,j)$$\n\n  Now we can show the connection between DDPM loss with the mean prediction and ConPreDiff loss:\n\n$$\\begin{array}{rl}||x_0-\\hat{x}\\_0(x_t,t)||_2^2 & = \\sum_i (x_0^i-\\hat{x}\\_0(x_t,t)(i))^2,\\\\\\\\ &=\\sum_i (x_0^i-\\sum\\_{j \\in \\mathcal{N}_i\\cup\\{i\\}}\\Psi(x_t,t)(i,j)/(|\\mathcal{N}_i|+1))^2,\\\\\\\\ &=\\sum_i(\\sum\\_{j \\in \\mathcal{N}_i\\cup\\{i\\}}(\\Psi(x_t,t)(i,j)-x_0^i))^2/(|\\mathcal{N}_i|+1)^2, \\\\\\\\ (Cauchy\\ inequality) & \\leq \\sum_i \\sum\\_{j \\in \\mathcal{N}\\_i\\cup\\{i\\}} (\\Psi(x_t,t)(i,j)-x_0^i)^2/(|\\mathcal{N}_i|+1), \\\\\\\\ &=1/(|\\mathcal{N}\\_i|+1)\\sum_i [(x_0^i- \\psi_p(x_t,t)(i))^2+\\sum\\_{j \\in \\mathcal{N}\\_i} (x_0^{i,j}-\\hat{x}\\_0^{i,j})^2] \\end{array}$$\nAs a result, the ConPreDiff loss is an upper bound of the negative log likelihood. Thus both terms are cooperative for convergence, and adding \"context prediction\" term would accelerate the convergence with less training steps.\n  \n2.Training ConPreDiff from pre-trained diffusion models\n\n Initializing ConPreDiff with any pre-trained diffusion models will need additional fine-tuning time for adjusting base denoising network and training our context prediction head. \n\n**Q2: By introducing context, is any sign of \"blurriness\" introduced?**\n\nA2: There is no sign of \"blurriness\" by introducing context, which can be partially explained by above derivations. We also put more qualitative comparison results  **in global response pdf**. From the results, we can easily find that our ConPreDiff can better generate the images semantically consistent with text prompts compared to other methods, and preserve local context well without any sign of \"blurriness\".\n\n**Q3: What are the typical cases that do worse compared to standard diffusion?**\n\nA3: Currently, we do not find obvious cases or typical cases that do worse than standard diffusion. However, we will continue to improve our model, explore more potential applications, and address possible limitations for future work.\n"}, {"review_id": "JD96OLsG7O", "reviewer": "Reviewer_DpwQ", "summary": "This paper presents ConPreDiff, a method introduced to improve the performance of diffusion models by preserving the neighborhood context of predicted pixels/features. They achieve this by predicting the neighborhood context during the diffusion generation process. To simplify the modeling complexity, they propose predicting distributions instead of directly reconstructing the neighborhood. The method's effectiveness is demonstrated through extensive experiments on unconditional image generation, text-to-image generation, and image inpainting.", "questions": "* L138: Why do you choose $\\mathcal P_{\\mathcal N_i^s}$ in this form? Is there any insight that motivates you doing this?\n* L115: What is $\\mathbf h_{t-1}$? I suppose it should be $\\mathbf x_{t-1}$.\n* L119: What is $\\mathbf h_{t-1}$? I think it should be $\\mathbf h_i$.\n* L138: What is $h_u^{(0)}$?\n* Figure 4: What is NDM?", "limitations": "The authors did not discuss their limitations and societal impact in the paper. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea is intuitive and easy to understand. \n* The proposed method is general and can be easily applied to recent diffusion models.\n* The performance of the proposed method is very impressive.", "weaknesses": "* Recent diffusion models use UNet backbone, which stacks many convolutional and self-attention layers. Thus, it has a large receptive field. Additionally, LDM also has a decoder, which also has a decent receptive field. Therefore, I am confused about the paper's main claim that the point-wise reconstruction neglects to fully preserve the local context.\n* There are no visual comparisons of the proposed method and baselines. I am not sure if ConPreDiff can really be more local-context consistent compared to other methods.\n* The authors need to discuss the additional training cost. Besides, they also need to provide the additional parameters they use for the context prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer DpwQ for your valuable feedback. We are glad that the reviewer found that our idea is intuitive and easy to understand, the proposed method is general and can be easily applied to recent diffusion models, and the performance of the proposed method is very impressive. Please see below for our responses to your comments.*\n\n**Q1: Being confused about the paper's main claim that the point-wise reconstruction neglects to fully preserve the local context.**\n\nA1: Although existing UNet-based diffusion models has a decent receptive filed, the process of acquiring receptive field is progressive, where the each point's local context information of previous layers may partially reduce or be filtered by multiple non-linear and pooling functions. Thus we explicitly add \"context prediction\" term at the end of the denoising network to explicitly reconstruct local context for each point，and maximally preserve useful context information from the perspective of reconstructing neighborhood distribution.  \n\n**Q2: More visual comparison.**\n\nA2: More visualization comparison results can be found in **global response pdf**, from which we can easily find that our ConPreDiff can better express the local contexts and consistent semantics of text prompts in the generated images compared to other methods.\n\n**Q3: The authors need to discuss the additional training cost. Besides, they also need to provide the additional parameters they use for the context prediction.**\n\nA3: We report the approximate time cost (seconds) of these models to train in a single step on the MS-COCO dataset in the table below. The additional context prediction head (two linear-BN-ReLU modules) added to the models accounts for **only a small portion of the parameters (around 0.13M)**, and the neighborhood distribution decoding of our method also does not incur significant additional training costs. Notably, our \"context prediction\" head can be removed in inference stage without introducing extra testing costs.\n|Training cost (secs/step) |LDM | DALL·E 2| Improved VQ-Diffusion| Imagen|\n| :-----| :----: | :----: |:----: |:----: |\n|Original Model  | 11.9 | 74.7 | 83.9 |198.3 |\n| + Context Prediction | 13.1 | 77.5 |87.6 |206.3 |\n\n\n**Q4: (1)** Why do you choose P_NS in this form? Is there any insight that motivates you doing this? **(2)** L115: What is $h_{t-1}$? I suppose it should be $x_{t-1}$. **(3)** L119: What is $h_{t-1}$? I think it should be $h_i$. **(4)** L138: What is $h_{u}^{(0)}$? Figure 4: What is NDM?\n\nA4: **(1)** In L138, $P_{N^s_i}$ is defined by uniform distribution, which means we sample neighbors from local context using equal probabilities. The main insight is after the distribution decoding, neighbors are sampled without spatial orders and ground truth neighbors in a local area are all important to local semantics. Thus we equally treat sampled neighbors for reconstruction.\n\n**(2)** $h_{t-1}$ denotes one point in $x_{t-1}$, which is equal to $h_i$. And we represent each point of $x_{t-1}$ in both time form ($h_{t-1}$) and spatial form ($h_{i}$) for better explanations.\n\n**(3)** \"(0)\" is the superscript of $h_u$, which denotes the center point of neighborhoods $N^s_{i}$ (i.e., $h_i$).\n\n**(4)** NDM is a typo, we previously name our model Neighborhood Diffusion Model (NDM).\n\n\n**Q5: The authors did not discuss their limitations and societal impact in the paper.**\n\nA5:  We will add these in final version\n\nLimitations:\n\nWhile our ConPreDiff boosts performance of both discrete and continuous diffusion models without introducing additional parameters in model inference, \nour models still have more trainable parameters than other types of generative models, e.g GANs. Futhermore, we note the long sampling times of both $\\text{ConPreDiff}\\_{dis}$ and $\\text{ConPreDiff}\\_{con}$ compared to single step generative approaches like GANs or VAEs. However, this drawback is inherited from the underlying model class and is not a property of our context prediction approach. Neighborhood context decoding is fast and incurs negligible computational overhead in training stage. For future work, we will try to find high-order and more intrinsic information to preserve for improving existing point-wise denoising diffusion models.\n\nBroader Impact:\n\nRecent generative image models enable creative applications and autonomous media creation, but can also be viewed as a dual-use technology with negative implications. In this paper, we use human face datasets only for evaluating the image inpainting performance of our method, and our method is not intended to create content that is used to mislead or deceive. However, like other related image generation techniques, it could still potentially be misused for impersonating humans.  A notorious example are so-called “deep fakes” that have been used, for example, to create pornographic “undressing” applications. We condemn any behavior to create misleading or harmful content of a real person. Furthermore, the immediate availability of mass-produced high-quality images can be used to spread misinformation and spam, which in turn can be used for targeted manipulation in social media. Datasets are crucial for deep learning as they are the main input of information.\nLarge scale data requirements of text-to-image models have led researchers to rely heavily on large, mostly uncurated, web-scraped datasets. While this approach has enabled rapid algorithmic advances in recent years, datasets of this nature have been critiqued and contested along various ethical dimensions. Furthermore, one should consider the ability to curate the database to exclude (or explicitly contain) potential harmful source images. When creating a public API that approach could offer a cheaper way to offer a safe model than retraining a model on a filtered subset of the training data or doing difficult prompt engineering. Conversely, including only harmful content is an easy way to build a toxic model. "}, {"review_id": "WRouW6BQds", "reviewer": "Reviewer_ydZz", "summary": "This paper proposes to improve diffusion based image generative training objectives by adding context prediction loss. The motivation of predicting context comes from other non-diffusion based models like semantic segmentation and representation learning. To mitigate the complexity of predicting large per pixel neighbourhood context, the author further models the context as a probability distribution using Wasserstein distance. Experiments show the proposed model achieves new SoTA generation on MSCOCO for both discrete and continuous diffusion models.\n", "questions": "In Table 1, it is not clear which diffusion model is used with the proposed method for both discrete and continuous diffusion models.\n\nFor qualitative comparison in Figure 2 and 8, there is no head to head comparison with other baselines, and it is hard to appreciate the improvement.\n", "limitations": "There is no limitation/future work discussion in the paper\n\nAnd there is also no training / implementation details in the paper, which cause concerns on reproducibility. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduction is well-written and the motivation of predicting context in the diffusion based model is easy to follow.\n\nThe presentation of the method including the loss derivation and the training pipeline is easy to understand.\n\nThe authors conduct intensive experiments showing the proposed context prediction loss can be used on various DM models, achieving SoTA performance on MSCOCO FID and inpainting tasks.\n", "weaknesses": "The paper lacks training and implementation details. For example, the text-to-image experiment uses T5 encoder as the text encoder, but did not mention architecture details and training details.\n\nOne big motivation to model context as a probability distribution is to improve the training efficiency. As shown in Figure 6, feature matching has lower throughput compared to distribution matching, but it has better FID. I think an important baseline is missing – sampling based feature matching, i.e. use the same number of random samples as proposed in the distribution matching, 9 instead of the full neighbourhoods features for context prediction.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer ydZz for the positive review and valuable feedback. We are glad that the reviewer found that the introduction is well-written, the motivation is easy to follow, the presentation of the method including the loss derivation and the training pipeline is easy to understand, and the conducted experiments are extensive. Please see below for our responses to your comments.*\n\n**Q1: The paper lacks training and implementation details. For example, the text-to-image experiment uses T5 encoder as the text encoder, but did not mention architecture details and training details.**\n\nA1: We provide some important implementation details in section 4.1. Because our ConPreDiff can generalize to both continuous and discrete diffusion models, our training and implementation details mainly follow the diffusion models that we generalize to for fair comparisons. Here we provide more details for clarity:\n\nFor text-to-image tasks, like Imagen, an Efficient U-Net architecture with 22 resnet blocks at total is utilized. A frozen T5-xl encoder is utilized as our text encoder. The text encoder is a 12 layers Transformer with 32-head multi-head attention and is pre-trained on a C4 text-only corpus with our context prediction based denoising objective. We use the standard Adam optimizer with 1e-4 learning rate.  We use the same cosine noise schedule as the Imporved DDPM [1]. For inpainting tasks, we adopt a same pipeline as RePaint, and a smaller U-Net architecture with 12 resnet blocks at total is utilized. The image size is 256x256. We use T = 250 time steps, and applied r = 10 times resampling with jumpy size j = 10.  For unconditional generation tasks, like LDM, we use a 8 layers U-Net. The max channels are 224. We use T=2000 time steps and linear noise schedule. The initial learning rate is 9.6e-5.\n\n**We are committed to open sourcing all the codes and trained models for all datasets and all tasks upon the acceptance.**\n\n[1] Nichol A Q, Dhariwal P. Improved denoising diffusion probabilistic models[C]//International Conference on Machine Learning. PMLR, 2021: 8162-8171.\n\n**Q2: Adding another baseline – sampling based feature matching, i.e. use the same number of random samples as proposed in the distribution matching, instead of the full neighborhood features for context prediction.**\n\nA2: Following your suggestions, we conduct fast experiments with sampling based feature matching, on the CelebA-HQ dataset. Compared with our approach, it predicts neighbors in a faster way, but it performs much worse than our \"context prediction\" which is based on distribution decoding, because our ConPreDiff adopts a surrogated loss of the entire neighborhoods matching.  And our approach can achieve a better trade-off between the FID score and training cost.\n|CelebA-HQ |LDM| LDM + Context Prediction| LDM + Sampling-based Feature Matching|\n| :-----| :----: | :----: |:----: |\n|Training time （sec/step） | 4.02 | 4.79| 4.22 |\n|FID score| 5.11 | **3.22**| 4.35 |\n\n\n**Q3: In Table 1, it is not clear which diffusion model is used with the proposed method for both discrete and continuous diffusion models.**\n\nA3: In the implementation details of section 4.1, we introduce that we generalize our ConPreDiff to discrete diffusion models (Improved VQ-Diffusion) to form our discrete $\\text{ConPreDiff}\\_{dis}$ and to continuous diffusion models (DALL-E 2, Imagen) to form our continuous $\\text{ConPreDiff}\\_{con}$.\n\n**Q4: More qualitative comparisons to appreciate the improvement.**\n\nA4: More qualitative comparison results can be found in **global response pdf**. From the results, we can easily find that our ConPreDiff can better express the local contexts and consistent semantics in the generated images compared to other methods.\n\n\n**Q5: Limitation/future work.**\n\nA5: We will add these in final version.\n\nLimitation/future work:\n\nWhile our ConPreDiff boosts performance of both discrete and continuous diffusion models without introducing additional parameters in model inference, our models still have more trainable parameters than other types of generative models, e.g GANs. Futhermore, we note the long sampling times of both $\\text{ConPreDiff}\\_{dis}$ and $\\text{ConPreDiff}\\_{con}$ compared to single step generative approaches like GANs or VAEs. However, this drawback is inherited from the underlying model class and is not a property of our context prediction approach. Neighborhood context decoding is fast and incurs negligible computational overhead in training stage. For future work, we will try to find high-order and more intrinsic information to preserve for improving existing point-wise denoising diffusion models."}, {"review_id": "tPTiXR9isS", "reviewer": "Reviewer_2mXr", "summary": "This paper proposes to improve diffusion-based image synthesis by explicitly reinforcing each point to predict its neighborhood context during training, without extra cost at inference. To reduce computation/time complexity of context decoding the authors propose efficient large context decoding adopting Wasserstein distance to characterize the distribution reconstruction loss. The method is applicable to both discrete and continuous diffusion backbones and achieves new SOTA text-to-image generation on MS-COCO with FID 6.21", "questions": "1. L260-262: regarding to Fig. 5 I'm not convinced by the observation and conclusion, could the authors make it more clear ? \n2. I'm curious about the comparision with Dalle-2 and Imagen in Fig. 7 as the models are not open-sourced.", "limitations": "No.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. \n2. The method of explicitly reinforcing each point to predict its neighborhood context for diffusion models is well-motivated with effective designs to reduce substantial computation complexity for large-context neighborhood reconstruction.\n3. The method is proven effective in boosting FID scores on MS-COCO text-to-image synthesis for both continuous (eDiff-I) and discrete diffusion (VQ-Diffusion) backbones.", "weaknesses": "1. The main results are on text-to-image synthesis and image inpainting. It would be good to add unconditional generation results. \n2. The method emphasizes on diffusion with better neighbouring context, leading to generations \"semantically better consistent with the text prompts\"(L233), \"prommising cross-modal semantic understanding\" (L234), \"can synthesize more complex objects and scenes\" (L236-237). I don't think the claims are well-justified: e.g. Fig. 2 and Fig. 3 only presents results of the proposed method without any comparison to warrant the aforementioned conclusions. More analysis and evidence of \"better semantics\" are required other than the overall FID score.\n3. Some notations are misleading, e.g. L 114-116, for h_i (h_{t-1}), the subscript is used to indicate both spatial and time; x_t is not defined in the main text. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer 2mXr for the positive review and valuable feedback. We are glad that the reviewer found that the paper is well-written and easy to follow, the method is well-motivated with effective designs, and the method is proven effective in boosting FID scores with both continuous and discrete diffusion backbones. Please see below for our responses to your comments.*\n\n**Q1: The main results are on text-to-image synthesis and image inpainting. It would be good to add unconditional generation results. More evidence of \"better semantics\" are required other than the overall FID score.**\n\nA1: We have listed unconditional generation results on four datasets in the appendix, and consistently outperform previous methods. More qualitative comparison results for demonstrating  \"better semantics\" of our method can be found in **global response pdf**. From the visual comparison, we conclude that our ConPreDiff can sufficiently capture the semantics in the text prompt, and can better express them in the generated images compared to powerful LDM and Imagen.\n\n**Q2: Some notations are misleading, e.g. L 114-116, for $h_i (h_{t-1})$, the subscript is used to indicate both spatial and time; x_t is not defined in the main text.**\n\nA2: $h_{t-1}$ and $x_t$ are used to denote the predicted point and previously-predicted image in the diffusion process, respectively. We will make the notations more clear, thanks for your constructive suggestions.\n\n**Q3: L260-262: regarding to Fig. 5 I'm not convinced by the observation and conclusion, could the authors make it more clear?**\n\nA3: As demonstrated in Fig.5, the neighborhood decoding error, the point-wise reconstruction error, and FID score consistently decline in the training process. The phenomenon means that our \"context prediction\" term and standard \"point-wise reconstruction\" term are cooperative, they simultaneously make better FID score. We theoretically prove this cooperation (from the perspective of maximizing ELBO/likelihood) as the following:\n\nAt time t: \n\nLoss of DDPM ( after reparamterization and scaling): $||x_0-\\hat{x}_0(x_t,t)||_2^2$\n\nLoss of ConPreDiff:  $\\sum\\_{i = 1}^{x*y} [\\mathcal{M}\\_p(x\\_0^{i} ,\\hat{x}\\_0^{i})+ \\mathcal{M}\\_n(\\mathcal{H}\\_{\\mathcal{N}\\_i},\\hat{\\mathcal{H}}\\_{\\mathcal{N}\\_i})]$\n\nWe let $\\mathcal{M}_p,\\mathcal{M}_n$ be square loss,\n  $\\mathcal{M}_n(\\mathcal{H}\\_{\\mathcal{N}\\_i},\\hat{\\mathcal{H}}\\_{\\mathcal{N}\\_i})=\\sum\\_{j\\in \\mathcal{N}\\_i}(x\\_0^{i,j}-\\hat{x}\\_0^{i,j})^2$,\n\n  where $x_0^{i,j}$ is the j-th neighbor in the context of $x_0^i$ and $\\hat{x}_0^{i,j}$ is the prediction of $x_0^{i,j}$ from a denoising neural network. Following the notation in the main paper, we have\n\n  $\\hat{x}_0^{i,j} = \\psi_n(\\psi_p(x_t,t)(i))(j)$,\n\n  where $\\psi_p(x_t,t)(i)$ is the prediction of $x_0^i$ and $\\psi_n$ is the neighborhood decoder. Compactly, we can write the denoising network as:\n\n  $$\\Psi(x_t,t)(i,j) =\\left\\\\{ \\begin{array}{l}\\psi_n(\\psi_p(x_t,t)(i))(j), & j \\in \\mathcal{N}\\_i, \\\\\\\\ \\psi_p(x_t,t)(i), & j=i \\end{array} \\right.$$\n\n  We can show that the DDPM loss is upper bounded by ConPreDiff loss, by reparameterizing $\\hat{x}_0(x_t,t)$.\n  Specifically, for each unit i in the feature map, we predict the unit i and its neighbors using equation (1), and then we use the mean of  predicted value in the neighborhood as the final prediction:\n\n  $$\\hat{x}\\_0(x_t,t)(i) = 1/(|\\mathcal{N}\\_i|+1)*\\sum\\_{j \\in \\mathcal{N}\\_{i}\\ \\cup\\ \\\\{i\\\\}}\\Psi(x\\_t,t)(i,j)$$\n\n  Now we can show the connection between DDPM loss with the mean prediction and ConPreDiff loss:\n\n$$\\begin{array}{rl}||x_0-\\hat{x}\\_0(x_t,t)||_2^2 & = \\sum_i (x_0^i-\\hat{x}\\_0(x_t,t)(i))^2,\\\\\\\\ &=\\sum_i (x_0^i-\\sum\\_{j \\in \\mathcal{N}_i\\cup\\{i\\}}\\Psi(x_t,t)(i,j)/(|\\mathcal{N}_i|+1))^2,\\\\\\\\ &=\\sum_i(\\sum\\_{j \\in \\mathcal{N}_i\\cup\\{i\\}}(\\Psi(x_t,t)(i,j)-x_0^i))^2/(|\\mathcal{N}_i|+1)^2, \\\\\\\\ (Cauchy\\ inequality) & \\leq \\sum_i \\sum\\_{j \\in \\mathcal{N}\\_i\\cup\\{i\\}} (\\Psi(x_t,t)(i,j)-x_0^i)^2/(|\\mathcal{N}_i|+1), \\\\\\\\ &=1/(|\\mathcal{N}\\_i|+1)\\sum_i [(x_0^i- \\psi_p(x_t,t)(i))^2+\\sum\\_{j \\in \\mathcal{N}\\_i} (x_0^{i,j}-\\hat{x}\\_0^{i,j})^2] \\end{array}$$\n  \n  \n  in the last equality, we assume that the feature is padded so that each unit i has the same number of neighbors $|\\mathcal{N}|$.\n  As a result, the ConPreDiff loss is an upper bound of the negative log likelihood. Thus both terms are cooperative for optimizing the generation quality of the model.\n\n**Q4: The comparison with Dalle-2 and Imagen.**\n\nA4: For the results of Dalle-2 and Imagen, we directly adopt published results from the papers for fair comparison. We adopt the codes reproduced by open source community, and further modify them to implement our ConPreDiff. And we are committed to open sourcing all the codes and trained models for all datasets and all tasks upon the acceptance.\n"}]

Author rebuttal:
## Global Response
We sincerely thank all the reviewers for the thorough reviews and valuable feedback. We are glad to hear that the idea is interesting and well-motivated (all reviewers), the paper is well-written and easy to follow (Reviewer m5Qc, ydZz, and 2mXr), the proposed method is general and can be easily applied to various diffusion models (Reviewer DpwQ, ydZz and 2mXr), and performance improvement showed in experiments are promising (all reviewers). 

We here summarize and highlight our responses to the reviewers:
* We make more visual comparisons to previous methods in the attachment pdf for more directly demonstrating the performance improvement (Reviewer m5Qc, DpwQ and ydZz) and better semantics expression (Reviewer 2mXr) of our proposed ConPreDiff.
* We provide additional theoretical derivation from the perspective of maximizing ELBO/likelihood (Reviewer m5Qc) to explain the cooperative relationships between our "context prediction" term and existing point-based denoising diffusion term (DDPMs), which is beneficial for model optimization and convergence (Reviewer qzoS and 2mXr).
* We also add some experiments to demonstrate the efficiency and soundness of the proposed "context prediction" (Reviewer m5Qc, DpwQ, and ydZz), and add more discussions about model limitations and societal impact (Reviewer DpwQ and ydZz, ethics reviews). 

We reply to each reviewer's concerns in detail below their reviews. Please kindly check out them. Thank you and please feel free to ask any further question.

Meta-review: This paper introduces a new method to improve diffusion-based image generation by predicting neighborhood context during generation without increasing inference cost. It outperforms standard diffusion models in various tasks and achieves state-of-the-art text-to-image generation on MSCOCO. Reviewers appreciate for the well-motivated idea, well-written paper and generalization for various diffusion models. The rebuttal also provides additional experiments and valuable insights to address reviewers' concerns very well. As a result, all reviewers have expressed scores above the acceptance threshold, and AC concurs with their recommendation to accept this paper. AC strongly encourages the authors to incorporate these valuable insights and constructive comments, possibly including discussion of potential negative applications and limitations, from the reviewers into the final version of the manuscript.
Decision: Accept (poster)