SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions
Authors: Yuseung Lee
Kunho Kim
Hyunjin Kim
Minhyuk Sung
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion model
Text-to-image generation
Panorama generation
TL;DR: We propose SyncDiffusion, a module that synchronizes multiple diffusions to create coherent image montages.

Abstract: The remarkable capabilities of pretrained image diffusion models have been utilized not only for generating fixed-size images but also for creating panoramas. However, naive stitching of multiple images often results in visible seams. Recent techniques have attempted to address this issue by performing joint diffusions in multiple windows and averaging latent features in overlapping regions. However, these approaches, which focus on seamless montage generation, often yield incoherent outputs by blending different scenes within a single image. To overcome this limitation, we propose SyncDiffusion, a plug-and-play module that synchronizes multiple diffusions through gradient descent from a perceptual similarity loss. Specifically, we compute the gradient of the perceptual loss using the predicted denoised images at each denoising step, providing meaningful guidance for achieving coherent montages. Our experimental results demonstrate that our method produces significantly more coherent outputs compared to previous methods (66.35% vs. 33.65% in our user study) while still maintaining fidelity (as assessed by GIQA) and compatibility with the input prompt (as measured by CLIP score). We further demonstrate the versatility of our method across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation. Our project page is at https://syncdiffusion.github.io.

SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions
Yuseung Lee
Kunho Kim
Hyunjin Kim
Minhyuk Sung
KAIST keitss98 rlaguswls98 phang (kaassttn kr
Abstract
The remarkable capabilities of pretrained image diffusion models have been uti- lized not only for generating fixed-size images but also for creating panoramas. However, naive stitching of multiple images often results in visible seams. Recent techniques have attempted address this issue by performing joint diffusions in multiple windows and averaging latent features in overlapping regions. However, these approaches, which focus seamless montage generation, often yield inco- herent outputs by blending different scenes within single image. To overcome this limitation, we propose SYNCDIFFSSON,, a plug-and-play module that syn- chronizes multiple diffusions through gradient descent from a perceptual similarity loss. Specifically, compute the gradient of the perceptual loss using the pre- dicted denoised images at each denoising step, providing meaningful guidance for achieving coherent montages. Our experimental results demonstrate that our method produces significantly more coherent outputs for text eext-guiedd panorama generation compared previous methods (66.35% vs. 33.65% our user study) while still maintaining fidelity (as assessed by GIQA) and compatibility with the input prompt measured by CLIP score). We further demonstrate the versatility of our method across three alug-and-play applications: mayout-guided image gen- eration, conditional image generation and 360-degree panorama generation. Our project page is at
github.io
Figure 1: Comparison panoramas generated with prompt "A photo of a rock concert" by Blended Latent Diffusson [ (top), MultiDiffusion (3) (middle), and our SYNCDIFFUSION (bottom). Blended Latent Diffusion, when applied on image extrapolation, often generates visible seams and repetitive patterns. MultiDiffusion creates seamless panoramas but fails to achieve global coherence across the image. In contrast, our SYNCDIFFUSION synchronizes windows across the panorama by increasing the perceptual similarity of the denoised output predictions. This results in significantly more coherent panorama outputs.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Diffusion models have recently emerged as the forefront of generative models, Recent breakthroughs in text-to-image generation such as DALL-E 2 [66] and Stable Diffusion [39 are diffusion models trained with billions of images. Compared with GANs [16], diffusion models offer advantages not only in producing high-quality realistic images but also in being utilized for conditional generation
setups such as inpainting [30] 34 41 editing [3 18] 7 2 34, and deblurring [48] 9), with
adaptation. The iterative reverse diffusion process can easily incorporate diverse conditions and regularizations at each step while guiding the entire process to produce realistic images. Hence, the diffusion model, once trained on a large-scale dataset, becomes a versatile and multi-purpose pretrained model that can be applied to various tasks and applications.
few-shot [25 12 40) or even zero-shot [37
36
28
46
49
Recent work has extended the versatility of pretrained diffusion models to generate arbitrary-sized images panoramas using either sequential [ or joint 33 diffusion processes. Since typical image diffusion models are trained to generate fixed-sized images, creating panoramic images requires stitching multiple fixed-size images together, which can be impossible do seamlessly without considering integration in the generation process. Two approaches have been proposed to tackle this issue. The first approach (2] involves generating the final output as a sequence of inpainting starting from an initial image, where each consecutive view image is produced while fixing the overlapped region (which referred to as image extrapolation in their work (2). However, this approach often struggles to seamlessly extend the given image and also tends to repeat similar patterns, resulting in unrealistic panoramas as shown the first row of Fig. | The other approach is joint diffusion B], which operates the reverse generative process simultaneously across multiple views while averaging the intermediate noisy images (or the noisy latent features) in the overlapped regions at each reverse process step. The blending of noisy latent features among the views at each denoising step can effectively generate a seamless montage of images. However, is important to note that the content and styles of the images may vary across the views, resulting in a mixture of colorful and fack-and-white images in single panorama, as shown in the second row of Fig. [ The lack of consistency occurs because the latent features of the overlapped regions are simply averaged without considering the coherence between them.
To address the limitation of previous work that produces unrealistic or incoherent montages, we present novel synchronization module for joint diffusion, dubbed SYNCDITTUIONN This module guides the reverse diffusion processes to achieve global coherence across different areas of the panorama image. Similar to previous guided diffusion methods (36 30 B4 41 our SYNcDIfFUSION guides the reverse diffusion process while adjusting the intermediate noisy images at each step. Our guidance is specifically provided as a gradient descent from a perceptual similarity loss calculated across multiple windows. Various off-the-sheif perceptual similarity losses such as LPIPS (53 or Style Loss 14 can be utilized in our framework. However, perceptual similarity losses computed with noisy images cannot effectively guide the denoising process. Thus, we draw inspiration from the non- non-Markovian formulation of DDIM [45) leveraging the prediction of the denoised output from the current noisy image at each denoising step. We compute the perceptual similarity loss using the foreseen denoised images at each step and then backpropagate the gradient through the noisy images. By leveraging the synergy with prior seamless stitching technique 33 based averaging latent features at each denoising step, our joint diffusion framework demonstrates the capability to generate montages that exhibit both local seamlessness and global coherence, shown in the last row of Fig. T This is achieved in a zero-shot manner, without the need for retraining or fine-tuning of existing diffusion models.
In our experiments on text- text-guided panorama generation using Stable Diffusion 2.0 [39 model, the results demonstrate that our method achieves significantly higher coherence compared to previous methods. Quantitatively, as we increase the weight of the gradient descent, we observe improved coherence, measured by LPIPS [53) and Style Loss [14, while maintaining fidelity (measured by GIQA I7) and compatibility with the input prompt (assessed by the CLIP score (19).. Diversity, measured by FID and KID, exhibits trade-off with coherence, but our method still achieves much better scores compared to the baseline method. Our user studies confirm a significant preference for our method (66.35%) over the previous method (33.65%) in terms of coherence, while also suggesting superior image quality and higher prompt compatibility. Moreover, we further demonstrate the versabtility of S SYNCDIFFUSION across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation.

---

2
Related Work
2.1 Diffusion Models
Diffusion probabilistic models [4 sampling from initial Gaussian distribution and iteratively applying a denoising process (referred to as the reverse process). These methods have achieved remarkable strides in image generation
21 33
are group of generative models that generate data by
42 36], surpassing previous image generation models including GANs [24] 6 DDPM [21] was among the pioneering models that showcased the impressive capability of image generation through Markovian forward and reverse processes, albeit with longer computation times in generation. This limitation was addressed by DDIM 451, which significantly reduced the sampling time in the reverse process using a non- non-Markovian transition formulation conditioned on the predicted denoised data. Furthermore, LDM [39 proposed incorporating the diffusion process into the latent space, achieving state-of-the-ar realism generated images and gaining attention the text-to-image generation. Diffusion models have also demonstrated their applicability to diverse data modalities
39
such as audio 50 23
29
15
videos [22 5 and 3D objects 35 28 46 49).
2.2 Few-Shot or Zero-Shot Adaptation of Diffusion Models
Building upon the remarkable generation capabilities of pretrained public text-to-image image diffusion models such as Stable Diffusion [39], recent research has introduced various methodologies for leveraging the pretrained models in diverse tasks including conditional generation, image editing, and manipulation, without the need to retrain the models from scratch. ControlNet [51] is an example of a method that enables the incorporation of additional conditions into existing text-to-image diffusion models through few-shot finetuning, wherein the image encoder is duplicated to handle the additional conditional image, and only a carefully selected subset of parameters is modified during the finetuning process. Custom Diffusion (25] also introduces similar idea of enabling few-shot tuning while keeping the majority of parameters in the neural network frozen, but with applications of finctuning the model for particular class concept of images. Other previous work has also demonstrated that diffusion models can even be applied to novel tasks in a zero-shot manner. SDEdit (31 was the
first show zero-shot conditional image generation using pretrained diffusion model by dispersing
noise over the conditional image and denoising it back to a real image. RePaint [30) introduced
an image inpainting idea by combining a generated foreground image and noised background image at each time step. Similar guided diffusion ideas have also been explored for various tasks,
such image super-resolution (27 13 9 43 colorization [1 10], deblurring 48 91, and style
We propose a novel guided diffusion framework for image montage generation via joint diffusion.
transfer [26] 25 12 40
2.3 Montage Generation via Diffusion Models
Panorama generation is one of the zero-shot applications of diffusion models. Since diffusion models are trained to generate images of specific size and on a 2D plane, stitching is required to generate panoramas or textures. Most previous methods 38 g 2 have employed inpainting- based approaches for seamless stitching. These methods extrapolate the accumulated image and fill only the missing regions to generate the panorama or texture. In contrast, MultiDiffusion B and DiffCollage [52] conduct diffusion in multiple views jointly while combining noisy latent features or scores at each reverse diffusion step. While both approaches have successfully produced continuous images, they have limitations in enforcing global coherence across the panorama texture. MVDiffusion [47, a concurrent work, extends milti-view diffusion to produce non-square panorama images such as 360 panorama images by leveraging pixel-wise correspondence and attention modules However, it focuses on achieving smooth stitching, without addressing global coherence. In this work, we propose simple yet effective synchronization module that can be integrated into any joint diffusion process to achieve global semantic coherence.
3 Backgrounds
3.1 Diffusion Models
In this section, we provide brief overview of the Denoising Diffusion Probabilistic Models (DDPM) 21 and Denoising Diffusion Implicit Models (DDIM) (45). which are the foundations of recent pretrained image diffusion models. The aim of DDPM is to approximate the data distribution q(xo) with tractable model distribution pe (xo). which takes the form of Markov chain with
3

---

learned Gaussian transitions pa (xx-1(xx) from p(xr) -x(x;; 0, I):
po(xu) /
po )dxxx, where po(xu:)) = p(xr)
II po (x(xx-1(x))
(1)
The parameters of the joint distribution (known the reverse process) 0 are learned by minimizing the negative evidence lower bound (ELBO):
min Eg (xx)) |- log pe [xo)] S min By(xo.x)..... xx) |- log Poggo((X07) + log (xx):(((x))))
(2)
where qo X1:7 Xn) is the forward process adding a sequence of Gaussian noise the data while increasing the noise scale. Among the variations of the forward processes, DDPM uses the variance- preserving diffusion that parameterizes the Gaussian transitions as follows with decreasing sequence 01:7 (0, 1)
((x1:T (x)) = II (x((xxx)) where ((x,/xx-1) N N
(Vorr x1-1;
Ot a1-1
(1 (1- Ott
a4-1 )
(3)
The definition of the Gaussian transitions in the forward process derives the following property:
q(x, (xx; va, II)
and thus matches the choice of the starting distribution in the reverse process unit Gaussian) since ((xr(x)) converges to unit Gaussian when or is set close to O. It also allows expressing x, with Xy and unit Gaussian noise variable
X
x0 + /1 Org6, where (0,
In DDPM (21), the Gaussian transition pe(x, (x:) for each in the reverse process is modeled as follows
px xx) = N (v
04--
Xt
1 VI at
(1
ai a--1
ea (x)x))) .
of.).
(6)
at
where of 1-a0-1 T-or
and (x1, 1) a learned function that optimizes the objective in Eq. 2 when it maps each x, at time & to unit Gaussian noise, thus resulting in the following simplitied loss:
(1- -ل_
I(co)= = Pxxro((x)..e [lloo (llo(van (Varxo + v1 a.4,,) 1) 62
DDIM 45 provides different perspective of seeing the same forward process as a non-Markovian process while taking the input data X into consideration reversed transitions:
T
((x1r1 (xo) = g(xr) xn) II
Xx, Xa), (xx 2=
where ((xr|xx) = (NVaTaxxpp xo, (1 --rr)
(8)
(i:==NN(va (vaxxo + 11 a, - of X1
VayX0
and vstt, g(x,|x,,x)) xp) = N
V1
aq
Then, each transition in the reverse process is also redefined as first predicting the denoised observa- tion Xo given each and then sampling X,-1 via the conditional distribution g(x,-1 Xt, Xo):
(x = (9(x1-1lxx) (x(,))))
ift 2 2
(9)
(x) 11
if = 1,
Note that 3, in the DDPM 21 equivalent to
in our paper and DDIM 45
0e-1

---

where
Do xx 1)
(x1-xxii -I- (xe ())
(10)
Var
the predicted denoised observation. The key observations of DDIM are twofold. First, the same simplified objective (Eq. () can be used to find the best models co (x t) in Eq. 10 that minimize the negative ELBO (Eq. 2 This means that the DDIM reverse process can be used with pretrained DDPM without retraining. Second, a subset of the time sequence [1.... . can be used in the reverse process of DDIM since X, for any & < can be sampled from x, via the Xo prediction, enabling a significant boost in the reverse process computation.
In the rest of the paper, the operation sampling the next denoised data in the reverse process with the learned distribution pa (x defined in either Eq. (DDPM) or Eq. (DDIM) is denoted as:
S(x,,t, c),,
(11)
which takes a noisy data x, at timestep / and unit Gaussian noise (O, as input.
3.2 Joint Diffusion
In an image diffusion model, each sample from the data distribution is either a 2D grid of per-pixel colors, or a 2D grid of latent features (as in Latent Diffusion (39]) that can be encoded from or decoded to a real image through a pretrained encoder & and decoder D. In the rest of the paper, the term image will thus be used refer to either color image or latent feature image, unless explicitly stated otherwise. Image diffusion models pretrained on fixed-size images cannot be used directly to produce arbitrary-sicc size images. MultiDiffusion [3] has addressed this limitation by using a multi-window joint diffusion approach. The framework integrates images generated from multiple windows seamlessly by averaging colors or features across the windows at every reverse diffusion step. For instance, consider the case of generating panorama image z E IR H. W.
The image at each window x(i) E x x is subarea of the panorama image whose union
across all the windows covers the entire panorama image. Let m (6) E [o, 1]Ha*H. denote a binary mask for the subregion in the panorama image corresponding the i-th window. The function Ta-+i TR. H, xW, x > H2 x W, X maps (crops) the panorama image to the i-th window image, while Ti-z H2 X JR xW: x is its inverse function that fills the region outside of the mask m, with zeros. During the joint diffusion process running the reverse process simultaneously
for each window, the noisy images from the windows are first averaged in the panorama space:
(6)
E. Ti Ti-xx Σ. m()
(12)
and then, the resulting combined noisy image is cropped again for each window xx" = Ta Tz))) modifying the noisy image at each window with a inter-window regularization.
Figure Panoramas generated by MultiDiffusion 3 (left) and our SYNCDIFFUSION (right), with prompt "A photo of a mountain range at twilight". MultiDiffusion often combines various scenes, such as mountains with trees and snow, and even awkwardly blends them. In contrast, SYNCDIFFUSION generates panoramas that are significantly more coherent.
SyncDiffusion
While MultiDiffusion 3 can generate seamless panorama images from joint diffusion, it often fails to produce coherent and realistic montages. The left image in Fig. demonstrates that the resulting image often oddly combines various scenes, such as mountains with trees and snow. Also, the blending occasionally fails to merge them in a realistic manner, as shown in the figure where distant objects are connected closer objects. This incoherence issue in MultiDiffusion arises due to two main reasons. Firstly, the averaging operation only aligns the colors or features in the overlapped regions but does not match the content or style of the images. Secondly, only enforces adjacent views to influence each other, and thus global coherence between distant windows cannot be achieved.
5

---

(x,,
D(Ss (x,,
(xx))
0.6485
0.6433
0.5416 0,3501
0,5911
0.3704
Figure 3: LPIPS [53] scores computed across the noisy images ((x;) at the intermediate step (t 45 out of 50) of the reverse process (left), the predicted denoised images D ( t)) at the same timestep t (middle), and the final generated images (xxo) at timestep t = (right). The indistinguishable noisy images yield similar LPIPS scores among them, whereas the predicted denoised images, which closely resemble the final outputs even at the beginning of the denoising process, exhibit LPIPS scores that align with those of the final generated images. This indicates that the predicted denoised images can provide meaningful guidance for producing coherent panoramas the diffusion process.
Algorithm 1: Pseudocode of one-time denoising in SYNCDIFFUSION.
Parameters I00
// Gradient deacent weight. // Noisy images at timestep t. // Noisy images at timestep 1.
Inputs: {x) Ji=o..N-1;
Outputs: {x [x-_] 10.......
Function SyncDiffusion(( x(")
xi
(4)
// The anchor window index is changed.
for ..... do X
w x
(D(co(x()) ,0), ()),, ((x :t)):// // Gradient descent (Eq. 14]
return (x Function MultiDiffusion (1x(()))
Σ, )x Σ, m(+)
// Averaging in the global space (Eq. 120.
for O....,N do
(ze): return {x; (1) ):
10
11 Function Denoossng@neetepp }):
[x(*) <- SyncDiffusion (xx; ())
12
13
for 0...., do
14
x S(x e),
// Sampling the next denoised data (Eq. 11
15
(x(2)) + Mult:iDiffusioo(((())
16
return
[x 1
To address this problem, we introduce a module called SYNCDIFFUSION which enables the generation of coherent montages, shown on the right in Fig. This module be easily integrated into an existing joint diffusion framework. Similar to MultiDiffusion, our SYNCDIFFUSION module updates the noisy image at every step of the reverse diffusion process. In contrast to averaging the colors or latent features in the overlapped regions. however, SYNCDIFFUSION employs the backpropagation of gradients from perceptual similarity loss computed across the windows to perform the update. The perceptual similarity loss, denoted as L, can utilize any off-the-shelf loss function for perceptual similarity, such as LPIPS 5 and Style Loss [1]. To facilitate efficient computation, we designate
an anchor window with an index of 0. For each view's noisy color image D(x!")) and the anchor window's noisy color image D(x(")) (where the decoder D can be treated as an identity function the given diffusion model operates in image space rather than latent space), one can measure the coherence using the images and conduct gradient descent through (
w L D(xx)))
(13)
where is the weight of the gradient descent. However, the coherence measured with the noisy images cannot provide meaningful guidance. Fig. 3 shows examples where the left three images are the intermediate noisy images D(x;) at timestep 45 out of total of 50 timesteps in the DDIM

---

"A photo of a city skyline at night"
"A photo of mountain range at twilight"
BLD
MD 3
Syne Diftusion
"A panorama of spring beautiful nature"
"A natural landscape in anime style illustration"
BLD
X
Sync
Diflusion
Figure 4: Qualitative comparisons. Blended Latent Diffusion ① (BLD, the first row of each case) tends exhibit visible seams and repetitive patterns. MultiDiffusion K (MD, the second row of each case) generates seamless results but lacks coherence, such as blending sunset sky with blue sky (top right), and displaying a combination of purple, pink, and blue backgrounds (bottom). In contrast, our SIFFUIIN (the last in each case) produces seamless panoramas with significantly improved coherence. Best view in zoom and color.
reverse process. Note that the LPIPS scores among the noisy images are indistinguishable. Hence, similar the DDIM reverse process, we utilize the foreseen denoised observation of each noisy data 60 [x] t) in Eq. 10 We measure the coherence not with the current noisy color images (D(x)))) (2)
but with the predicted denoised color images (D(oe(x)) (6) t))} with the timestep and perform the backpropagation of the gradient through x resulting in the following updated formulation:
x (2) wV (D(oo(x))) ()) ((ou(x)
(0) .0)).
(14)
In Fig. 3 the middle three images depict the predicted denoised images D( (x,, ()) at timestep 45, which closely resemble the final generated images D(xn) timestep = 0, even during the initial stages of the reverse diffusion process. Therefore, the LPIPS scores among the predicted denoised images also match those of the generated images, providing meaningful guidance for maintaining coherence. During each denoising step in the joint reverse process, we apply this update the noisy images for windows (x("), and sample the one-step denoised images. MultiDiffusion is also applied to average the sampled images at the end. Refer to Alg. T for detailed pseudocode.
Results
5.1 Text-Guided Panorama Generation
In our experiments, generate panorama images using our SYNCDIFFUSION method and the pretrained Stable Diffusion 2.0 (39 model. Stable Diffusion model operates latent space of [2o1x61X4 and generates images of R512x512x3 We generate panorama images of resolution 512 3072 (64 384 in the latent space), where the width six times the width the output of Stable Diffusion. Each window x(i) has an image resolution of 512 512, with stride of 128 pixels along the width in the image space which is equivalent to stride 16 in the latent space, resulting in total 21 windows to operate diffusion processes jointly. use six text prompts from MultiDiffusiin 3 (see S.8 of the supplementary) and generate 500 panoramas per prompt. For gradient descent weight w (Eq. T4., we experiment with various initial values while applying weight decay with rate of 0.95. We also set the center window as the anchor window with an index of O.
We compare our SYNCDIFFUSION with previous methods that generate panoramas using pretrained diffusion model. Blended Latent Diffusion is inpainting-bai based method that
Baselines

---

extrapolates a single window image. MultiDiffusion [1, served as the base of our framework, is special case of our method when the weight of gradient descent w is O. The same Stable Diffusion 2.0 model is used for all the methods for a fair comparison.
Evaluation Metrics We utilize range of metrics to assess the coherence, fidelity, diversity, and compatibility of the output panoramas with the input prompt.
(Coherence) Intra-LPIPS [53] and Intra-Style-L [14] To assess the coherence of the generated
panoramas, we introduce two metrics. Intra-LPIPS and Intra-Style-L, which are computed as the averages of LPIPS [53] and Style Loss [4. respectively, between a pair of non-overlapping window images from the same panorama. Specifically, we divide the panorama into windows, each with dimensions of 512 x 512 and then compute the average of LPIPS and Style Loss across the 15 combinations of these cropped views. To provide a reference for the scale of these values, we generate 500 singlewwidow: the LPIPS and Style Loss for randomly selected 1,000 pairs of these reference images.
images using the same Stable Diffusion model and compute
(Fidelityy Mean-GIQA [17] GIQA quantifies the fidelity of individual images by calculating the inverse of the distance between a query image and a reference set in a feature space. Mean-GIQA is computed by taking a single random crop of each panorama in 512 512 size and computing the average GIQA score from each cropped image to the reference set of images mentioned above. (Fidelity & Diversity) FID 20 and KID FID [0] and KID 4 are used to measure both fidelity and diversity. Both of them are measured with the aforementioned randomly cropped images and the set of reference images.
(Compatibility with the Input Prompt) Mean-CLIP-S 19 The compatibility with the input prompt is assessed using the mean of CLIP scores [9] denoted as Mean -L.P-SL This met- ric is calculated using the same set of cropped images and the input prompt.
Qualitative Comparisons Fig. 4 showcases qualitative comparisons between our method and the baseline methods. Here, we show the results of our method generated with a weight parameter of w = 20. Blended Latent Diffusion [ often exhibits visible seams due the sequential inpainting scheme and produces repetitive patterns in the extrapolation, as illustrated by the mountains in the second case of the top row and the flowers and trees in both cases of the bottom row. MultiDiffusion achieves seamless outputs, although it often produces incoherent outputs, such as mixing sunset sky with a blue sky, as shown in the second cases of the first and second rows, and pink and purple backgrounds with blue backgrounds in the two cases of the bottom row. Our SYNCDIFFUSION generates visually and semantically more coherent panoramas with all the prompts. More qualitative comparisons are provided in the supplementary (S.1, S.8).
Quantitative Results Fig. 5.1 presents a quantitative comparison among the methods. Our method's results are displayed using two different gradient descent weights, w = 10 and 20. MultiDiffusion 3 is also the case when w 0 in our framework. For results obtained with different weights, please refer to the supplementary (S.2). The color bars in the plots indicate the average scores across the six prompts, while the black lines depict the standard deviation. Note that as the gradient weight increases, both Intra-LPIPS and Intra-Style-L decrease. When w = 20, the Intra-LPIPS and Intra-Style-L of our method are approximately 3/4 and 1/6 of those computed with the reference set images (referred to as SD, Stable Diffusion). respectively, indicating significantly higher coherence. Moreover, the Mean- CLIP-S and Mean-GIQA scores are comparable to those computed with the reference set, meaning that the compatibility with the input prompt and fidelity are not compromised by our diffusion synchronization. The results of FID and KID demonstrate the trade-off between coherence and diversity. As the gradient descent weight w increases, FID and KID also increase slightly, although they are still much lower compared to Blended Latent Diffusion D. This implies that for certain images, it is more difficult to find coherent images. In the supplementary (S.3), we substantiate this claim with the results of shorter generated panoramas. Blended Latent Diffusion results in low due to its tendency to repeat similar patterns, but it leads to low Mcan-GIQA and very high FID and KID, indicating a significant degradation in fidelity.
User Study We conducted three user studies to further evaluate the coherence, image quality and prompt compatibility of the generated panoramas, respectively. Following Ritchie [32], participants were presented with panorama images generated by both MultiDiffusion [3] and our SYNCDIF- FUSION methods (with w = 20). They were then asked to choose one of them by answering the question: Which one appears more coherent panorama image to you? (Coherence), Which one is of higher quality? (Image Quality), or Which one best matches the shared caption? (Prompt Compatibility). We collected 25 responses each, including 5 vigilance
8

---

0883
Intra LPIPS
Intru-Style-_ (x10-7)
Moan CLIPS
0.8
34
30-
30
0.50
088
60
10.0
BLD
MD
Diex
50
31D
380
Dar
Ok
[W] (k 20)
Moan-GiOA (x10
(x10
1:00
80-
80
20
40
5
20
(9 10) Of- 200
(e 300 201
(r- 10) (A-
Figure 5: Quantitative results. MultiDiffusion [3] (MD) can be considered as special case of our method when the gradient descent weight w is Intra-Style-L) improves while maintaining the compatibility with the input prompt AMean-CLIP-- and fidelity (Mean-GIQM There is trade-off between coherency and diversity, as indicated by the FID and KID results. Note that the FID and KID of our method are still significantly lower than those Blended Latent Diffusion [T (BLD). (Stable Diffusion) the score with reference set images. Refer to the text for the details.
As increases, coherence (Intra-LPIPS and
tasks, from 100 participants for each user study. The results in Tab. Taffirm that human evaluators per- ceive SIFFUSSINO as producing more coherent results compared to MultiDiffusion, while also demonstrating superior image quality and higher prompt compatibility. Refer the supplementary (S.7) for detailed setups for the user study.
Coherence (%)
Image Quality (%)
Prompt Compatibility (%)
MultiDiffusion 3 SYNCDIFFUSION 66.35
33.65
42.81
40.50
57.19
59.50
Table I: User study results.
5.2 Additional Applications of SYNCDIFFUSION
We further demonstrate the versatility of SYNCDIFFUSION through three additional plug-and sogg-and-pla and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation.
Layout-Guided Image Generation Plugging SYNCDIFFUSION into the layout-to-image pipeline in MultiDiffusion 33 leads to notable enhancement in the global coherence displayed in Fig. (A). While MultDiffusiinn (middle row) generates an unnatural image with incoherent background around the house and the bear, our method produces natural image with globally coherent background.
Conditional Image Generation When integrated with ControlNet (51). SYNCDIFFUSION extends the conditional image generation to arbitrary resolutions. Let E H. xW x3 denote an input condition and vi-.i R M, M.xxW.33 + R XW x3 be mapping from to the i-th cropped condition
corresponding to the window x; We define conditional S SYNCDUFUSION by substituting de (X t) Alg. with (x, L, cl)), where (()) == (c). Fig. (B) illustrates that combination of NontrolNet and SYNCDIFFUSION generates coherent panoramas while reflecting the given condition
Canny edge map (top row).
360-degree Panorama Generation further plug SYNCDIFFUSION into MVDiffusion 47 concurrent work that generates 360-degree panoramas from text prompts via multi-view diffusion. As shown Fig. (C)(),, our SYNCDIFFUSION distinctly improves the global coherence the generated panorama. The increase in coherence becomes more apparent when comparing perspective views

---

(A) Layout-Guided Guided Imuge Generation
(B) Conditional Image Generation
Caryy odge mX)
gussllan ander beaurifwi
wooden Aons
bear
boanthi' cal paintiys
Stuchiffaion
SI Aigiall raisity is faranay planss
(0) 360-degree Panorama Generation
View
View 2
"Digial illomaation of moderns loiing room
Perspestive Plens
Synchimund
360
SynclMlfashin
Panaroma
Figure Plug-and -lay applications of SYNCDIFFUSIN..
from different angles. While View I and View 2 from the vanilla MVDiffusion (top row) seem to be from two different rooms, with our method the generated images better depict two views from the same room (bottom row).
While our SYNCDIFFUSION module can significantly enhance the coherence of generated panoramas, it relies appropriate input prompts to achieve realistic results, illustrated in Fig. 7 Also, the SYNCDIFFUSION module that includes a forward pass through the neural network and gradient descent computation introduces additional computational overhead.
Limitations
Supplementary Due to space constraints, present the following additional results in the Supple- mentary: more qualitative comparisons with various prompts (S.1, S.8), details about the quantitative evalution (S.2). evaluation on different resolutions (S.3), results with Style Loss [14 as the perceptual loss (S.4), an ablation study using Eq. [3 instead of Eq. (S.5), an analysis of computation time (S.6), and details on the user study (S.7).
"A red sports car"
fancy hotel room"
Figure 7: Our failure cases. A suitable input prompt is required to generate realistic panoramas.
6 Conclusion
We presented SYNCDIFFUSION, diffusion synchronization module designed generate coherent montages through joint diffusions. Using a pretrained diffusion model, we propose guiding the reverse process by updating the noisy images each intermediate step using gradient descent. This update is based on a perceptual similarity loss calculated with the predictions the denoised images. Moreover, the idea of SYNCDIFFUSION plan to investigate such possibilities in future work.
be applied to generating textures for 3D models. We
Potential Negative Societal Impacts Image generative models can potentially generate deepfakes, images resembling copyrighted material, biased or discriminatory images, and harmful outputs. Future research is needed to advance the detection of manipulated content and establish societal barriers to protect intellectual property.
10

---

Acknowledgments and Disclosure of Funding
We thank Juil Koo for valuable discussions on diffusion models and Eunji Hong for help in conducting user studies. This work was partially supported by the NRF grant (RS2023-00209723) and IITP grants (2019-0-00075, 2022-0-00594, RS-2023-00222592) funded by the Korean government (MSIT), the Technology Innovation Program (20016615) funded by the Korean government (MOTIE), grants from ETRI, KT, NCSOFT, and Samsung Electronics, and computing resource support from KISTI.
References
[1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. In SIGGRAPH, 2023.
[2] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
[3] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusiin: Fusing diffusion paths for controlled image generation. In /CML,, 2023.
[4] Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR, 2018.
[5] Andreas Blattmann. Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[7] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
[8] Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2Tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. 1/396, 2023.
Hsin-Ying
[9] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In ICLR, 2023.
[10] Hyungjin Chung, Bycongsu Sim, and Jong Chul Yc. Improving diffusion models for inverse problems using manifold constraints. In NewrlPS, 2022.
[11] Prafulla Dhariwal and Alex Nichol. Diffusion models heat GANs on image synthesis. In NeurlPS, 2021.
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
[13] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li. Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusson models continuous super-resolution. In CVPR, 2023.
[14] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. CVPR, 2016.
[15] Deepanway Ghosal, Navonil Majumder, Ambuj Mchrish, and Soujanya Poria. Text-to Text-to-aue audiioo- generation using instruction tuned IIm and latent diffusion model. arXiv preprint arXiv:2304.13331 2023.
[16] Ian Goodfellow, Jcan Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[17] Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. GIQA: Generated image quality assessment. In ECCV, 2020.
[18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. In ICLR, 2023.
[19] Jack Hessel, Ari Holtzman, Maxwell Forbes. Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning, In EMNLP, 2021.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two scals update rule converge to local nash equilibrium. In NIPS, 2018.

---

[21]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[22]
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In ICIR,, 2022.
[23] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Yc, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-An-Audio: Text-to-audit generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301. 12661, 2023.
[24] Tero Karras, Samuli Laine, Miik Aittla,, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.
[25] Nupur Kumari, Bingliang Zhang, Richard Zhang. Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-imap diffusion. CVPR, 2023.
[26] Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. ICLR, 2023.
[27] Haoying Li, Yifan Yang. Meng Chang, Huajun Feng. Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing. 2022.
[28] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3D: High-resolution text-to-3D content creation. In CVPR, 2023.
[29] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Dud Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 12500, 2023.
[30] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
[31] Chenlin Meng, Yutong He. Yang Song. Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
[32] Rudimentary framework for running two-alternative forced choice (2afc) perceptual studies on mechanical turk. /tttps///github.
www...ooooooooceeeee wttp////wow///ttkkkoeeoeeoereeer 2
[33] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In JCML, 2021.
[34] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In ICMI,, 2022.
[35] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DrcamFusion: Text-to-3D using 2D diffusion. In ICLR, 2023.
[36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25 2022.
[37] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. shot io-age- generation. JCML, 2021.
[38] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. TEXTure: Text-guided texturing of 3D shapes. arXiv preprint XXX::2302.0/721,, 2023.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[40] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dream- Booth: Fine tuning text-to-image diffusion models for subject-drived generation. In CVPR, 2023.
[41] Chitwan Saharia, William Chan, Huiwen Chang. Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH, 2022.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-iman to-image diffusion models with deep language understanding. In NeurIPS, 2022.
12

---

[43] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[44] Jascha Sohl-Dicke Dickkstiin, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In ICML, 2015.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[46] Junshu Tang. Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-It-3D: High-fidelity 3D creation from single image with difflusio prior. arXiv preprint arXiv:2303 14184, 2023.
[47] Shitao Tang. Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. MVDiffusion: Enahling holistic multi-view image generation with correspondence-wware diffusion. arXiv preprint arXiv:2307.00097, 2023.
[48] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. CVPR, 2022.
[49] Jiale Xu, Xintao Wang. Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. 3ream3D: Zero-shot text-to-3D synthesis using 3D shape prior and text-to-image diffiusion models. CVPR, 2023.
[50] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model text-to-sound generation. IEEE Transactions on Audio, Speech and Language Processing, 2022.
[51] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffiusion models. arXiv preprint arXiv:2302.05544 2023.
[52] Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, Ming- Liu. DiffCollage: Parallel generation of large content with diffusion models. In CVPR, 2023.
[53] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
13


Reviews:
[{"review_id": "Y5z2SmEgve", "reviewer": "Reviewer_X4Nm", "summary": "The paper addresses the problem of lack of global coherence in panoramas generated with diffusion models (in a tiled manner with overlapping regions). This work builds upon MultiDiffusion [1], which proposed the technique of averaging multiple predictions for an overlapping region at each denoising step to avoid edge artifacts. However, MultiDiffusion lacked long range coherence.\nThis paper proposes interleaving an additional guidance step using perceptual similarity loss to ensure that each region is coherent with an anchor region.\n\n[1] MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, Bar-Tal et al, 2023", "questions": "None", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The solution of guiding the denoised image with a perceptual similarity loss for global coherence is simple yet effective\n- Qualitative and quantitative results are thorough and back the effectiveness of the approach in achieve global coherence\n- Paper is very well written, adequately discusses background, related work and limitations.", "weaknesses": "- This slows down the inference process as the authors discuss in the supplementary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your review, acknowledging that our method is \"simple yet effective\", the experimental results support “the effectiveness of the approach,\" and the \"paper is well-written.\"\n\nIndeed, the computational overhead is a drawback of our method. However, we would like to reiterate the discussion in Section S.6 of our supplementary, where we highlight that computing the gradient descent not at every timestep, but rather at only a few steps with regular intervals, can still enhance coherence (for instance, reducing Intra-LPIPS by 0.07 by performing the gradient descent every 5 timesteps), while alleviating the computational overhead and also minimizing its impact on fidelity and diversity.\n\nMoreover, in Section (1) of our global response above (Additional Experimental Results), we present the results when involving the computation of the SyncDiffusion function only at the initial 3 or 5 steps of the reverse process. The results demonstrate that the early-stage synchronization is even more effective in improving coherence. Specifically, the initial 3-step synchronization yields better results than a 10-step synchronization with uniform intervals. This finding further contributes to the reduction of computational overhead.\n\nWe plan to further explore methods for accelerating the synchronization computation through recent distillation training approaches, as presented in \"Consistency Model\" [1] or \"On Distillation of Guided Diffusion Models\" [2].\n\n**References**\n\n[1] Song, Yang, et al. \"Consistency models.\" ICML (2023).\n\n[2] Meng, Chenlin, et al. \"On distillation of guided diffusion models.\" CVPR (2023).\n"}, {"review_id": "HsOrfdOKJQ", "reviewer": "Reviewer_AhXa", "summary": "The paper tackles the problem of image diffusion models for generation montages/panoramas. Their proposed method, dubbed SyncDiggusion, improves on the state-of-the-art (both qualitatively, through user studies, and in quantitative metrics) by synchronizing multiple diffusions through gradient descent using a perceptual similarity loss. This provides a guidance in the reverse diffusion process by adjusting intermediate noisy images at each step. Notably, off-the-shelf perceptual similarity losses can be interchangeably used with this method.", "questions": "1. Can this work be extended to other applications such as videos?\n1. It was mentioned that choice of prompt is critical in this method, and can easily break it. How were the prompts for the user study chosen? Choice of prompts would greatly influence the outcome of the study, potentially in favour of the proposed algorithm.\n", "limitations": "Yes, limitations are discussed in brief.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel and interesting solution to the problem of consistent panoramic image generation using diffusion models\n1. The method is original, to my knowledge\n1. The paper is well written and clear\n1. The paper presents adequate theoretical and algorithmic explanations\n1. The paper presents a comprehensive set of both qualitative and quantitative results that demonstrate the benefits of their method over previous state-of-the-art, highlighting their contributions well", "weaknesses": "1. Coherence is measured not with the current noisy color images $\\{D(x_t^{(i)})\\}$ but with the predicted denoised color images $\\{D(\\phi(x_t^{(i)}, t))\\}$ (L195). What is the computational overhead of this computation? Similarly, What is the added computational overhead of this method as compared to the reported baselines? A discussion of this is omitted, although mentioned in brief in the limitations sections.\n1. In turn, a discussion of the tradeoffs between consistency vs. diversity vs. computational overhead would then be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review, recognizing the novelty and originality of our method, as well as the comprehensive qualitative and quantitative results. Here, we address the concerns and questions that have been raised.\n\n**(1) Tradeoffs between consistency vs. diversity vs. computational overhead**\n\nWe draw your attention to Section S.6 in our supplementary material, where we analyze the trade-offs among coherence, diversity, and computation overhead by adjusting the frequency of gradient descent computation in the reverse process. The results reveal that reducing the frequency of gradient descent computation can lower computation overhead and also impact fidelity and diversity (Mean-GIQA, FID, and KID), while still enhancing coherence (Intra-LPIPS, Intra-Style-L). Notably, performing gradient descent not at every timestep, but only ten times out of 50 timesteps, can decrease Intra-LPIPS from 0.69 to 0.62, while utilizing fewer than 1/3 of the computation time compared to gradient descent at every timestep. \n\nMoreover, in Section (1) of our global response above (Additional Experimental Results), we present the results when involving the computation of the SyncDiffusion function only at the initial 3 or 5 steps of the reverse process. The results demonstrate that the early-stage synchronization is even more effective in improving coherence. Specifically, the initial 3-step synchronization yields better results than a 10-step synchronization with uniform intervals. This finding further contributes to the reduction of computational overhead.\n\n**(2) Application to video generation**\n\nWhile we haven't yet attempted to apply this concept to video generation, we believe that it holds potential for enhancing the video generation process as well. We appreciate your comment and will delve deeper into this direction for further exploration.\n\n**(3) Choice of prompts**\n\nIn Section 5, for the purpose of comparisons, we employed the same prompts that were utilized in MultiDiffusion [1], aiming to eliminate any potential biases in the comparisons. Note that we utilized only six out of the eight prompts that were originally used in MultiDiffusion, as the authors did not provide the remaining two prompts.\n\n**References**\n\n[1] Bar-Tal, Omer, et al. \"Multidiffusion: Fusing diffusion paths for controlled image generation.\" ICML (2023).\n\n\n\n"}, {"review_id": "kXFl4JhCJl", "reviewer": "Reviewer_SgT1", "summary": "This paper proposes a diffusion synchronization module, a plug-and-play module for text-to-panorama generation. The key idea is based on the observation that predicted denoised images closely resemble the final outputs even at the beginning of the denoising process. Authors proposed to synchronize multiple diffusions through gradient descent from a perceptual similarity loss. Experiments demonstrate the effectiveness of the proposed method in generating coherent montages with good fidelity and compatibility with prompts.", "questions": "See weakness.", "limitations": "Yes, authors mentioned the limitation of the proposed in its high sensitivity to prompt and additional computation overhead.\nThey also mentioned its potential negative sociatal impact on deepfake. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)\tThis work is well motivated based on the observation that Predicted denoised images closely resemble the final outputs even at the beginning of the denoising process. In addition, it also helps local seamlessness and global coherence.  \n(2)\tEvaluation metrics used in quantitative comparison experiments are elaborate.  \n(3)\tThe paper has high writing quality which makes it easy to understand.  \n", "weaknesses": "(1)\tBased on MultiDiffusion, the author introduces guidance information provided solely by gradient descent from a perceptual similarity loss. I wonder if this guidance is too weak or have they explored other guidance?  \n(2)\tPretrained stable diffusion v2 is used to generate 512x512-sized images. How can stable diffusion be utilized to generate wider images? Please provide a detailed explanation in the paper. The setting of text-to-panorama generation task should be further clarified.   \n(3)\tThrough the quantitative comparisons in Fig. 5, it can be observed that the setting of gradient descent weights has significant impact on the generation results. However, it is unclear how this affects the qualitative or visual effects. I wonder if the authors have experimented with other weights and is it possible that a weight other than w=20 could yield better results.  \nAdditional Comments  \n(1)\tL171, L179, writing errors, Fig. 4 -> Fig. 2\n(2)\tL191, L198, writing errors, Fig. 4 -> Fig. 3\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review, recognizing that \"evaluation metrics are comprehensive\" and \"the paper is of high writing quality.\" We address specific queries below.\n\n**(1) Is the guidance not strong enough?**\n\nWe have quantitatively and qualitatively showcased the impact of our synchronization guidance in Figures 1, 4, and 5 in the main paper, as well as in the supplementary results. We would like to draw your attention to our extensive qualitative comparison in Section S.1 of our supplementary, where we illustrate the significant distinction between MultiDiffusion [1] and our SyncDiffusion across a variety of examples.\n\n**(2) Other guidance**\n\nIn Section S.4 of our supplementary, we presented results obtained by employing our SyncDiffusion method using Style Loss [2] instead of LPIPS. The qualitative and quantitative outcomes are depicted in Figure S3 and Table S2, respectively. These results illustrate that this alternative guidance can also enhance global coherence in the output panorama images, albeit at the expense of greater trade-offs in terms of fidelity and diversity (FID and KID) compared to when LPIPS is used in SyncDiffusion. We plan to further investigate the potential of alternative guidance losses.\n\n**(3) Details about wide image generation**\n\nHere we briefly clarify our algorithm outlined in Sections 3 and 4. For a given target resolution of a panorama image—512x3072 in our experiments—we define a set of windows spanning the image, each being of Stable Diffusion's resolution, i.e., 512x512, and mutually overlapping. In our experiments, this set of windows is established with a stride of 16 in the latent space. We execute the reverse process of Stable Diffusion within each window. However, preceding each step of this reverse process, as detailed in Algorithm 1, we apply our SyncDiffusion module, computing gradient descent with the loss defined across these windows. Furthermore, after each step of the reverse process, we apply the idea of MultiDiffusion [1] and average the latent features of images in the regions of overlap.\n\n**(4) Impacts of the weight ‘w’**\n\nThe impact of the weight 'w' is analyzed in Section 5 of the paper and also in Section S.2 of our supplementary. Qualitative and quantitative experimental results obtained by varying the weight 'w' are presented in Figure S1, Table S2, and Figure S2. As expected, the plots in Figure 5 and Figure S2 (which include more variations of ‘w’) in the supplementary illustrate a discernible trend: as the weight increases, global coherence improves (Intra-LPIPS and Intra-Style-L decrease), while fidelity and diversity experience more compromise (FID and KID increase). We anticipate observing a similar trend as the weight 'w' is further increased.\n\n**References**\n\n[1] Bar-Tal, Omer, et al. \"Multidiffusion: Fusing diffusion paths for controlled image generation.\" ICML (2023).\n\n[2] Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. \"Image style transfer using convolutional neural networks.\" CVPR (2016)."}, {"review_id": "QKfe5YFmgt", "reviewer": "Reviewer_DATV", "summary": "In this paper, the authors propose a new module called SyncDiffusion that enables to synchronize multiple outputs from diffusion models to create coherent montage. This module guides the reverse diffusion process in each window by shifting x_t with gradients of a perceptual loss between the target window and an anchor window. The experimental results show that SyncDiffusion contributes to produce more coherent montage compared to previous methods.", "questions": "- Please see Weaknesses.\n- FID of MultiDiffusion in Fig. 5 seems much different from that shown in the original paper [3] (around 30 vs ~10). Why is it? \n- It seems straightforward to apply SyncDiffusion to layout-to-image task by just adding an appropriate cropping process before computing the perceptual loss. This should make quantitative comparison much easier as done in MultiDiffusion or [R1]. Why did the authors focus on generating panorama images in the experiments?\n  - [R1] \"Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC,\" ICLR 2023.\n\n---\n\nGiven the authors' response, I updated the rating accrodingly.", "limitations": "- Several limitation are described in the manuscript.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This study adresses an important problem, namely how to synchronize multiple estimates from diffusion models to produce a single output. In particular, how to guarantee a global conherence in the output image is highly non-trivial, but should be critical for practical applications such as panorama image generation.\n- The main idea, which is to use gradients of a perceptual loss between target and anchor windows to ensure the global coherence, is simple and can be jointly used with various methods for combining multiple outputs from diffusion models.", "weaknesses": "- The current design of the proposed algorithm relies heavily on the homogeneity of the target scene, which may limit its applicability. This is due to LPIPS used for guidance, which forces every window share a similar composition of the image. Personally, I wonder this causes the failure cases shown in Fig. 6.\n- The evaluation by Intra-LPIPS and Intra-Style-L is interesting to see a global coherence in a resultant image, but the superiority of the values is not clear, because it is unknown how much the values will be in real panorama images.\n- Somewhat related to above, the effectiveness of the proposed method is not clear. In the quantitative comparison, Mean-CLIP-S and Mean-GIQA are on par, and FID is even worse than that of MD (I understood why it happans, though). On the other hand, the user study is conducted only for evaluating the coherence, not for image quality nor prompt fidelity.\n- Although the authors argue that SyncDiffusion is a plug-and-play module, it is evaluated only with MultiDiffusion in the experiments. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your review, acknowledging that we \"address an important problem\" which is also \"highly non-trivial\", and also recognizing that our solution is \"simple and can be jointly used with various methods\". We address the concerns and questions raised in the review.\n\n\n**(1) Relying on homogeneity limiting its applicability?**\n\nFirst of all, using LPIPS as the guidance for the synchronization does not result in the failure cases shown in Figure 6. Please refer to Figure 2 in the PDF file attached to the above global response (Additional Experimental Results). You can find the link at the bottom of the text. The figure displays the results of performing MultiDiffusion [1] with the same prompts. These results exhibit the same failure cases due to the description of an \"object\" instead of a \"scene.\"\n\nFurthermore, the LPIPS-based guidance enforces perceptual similarity across the windows while still allowing for varying content. For instance, the last row in Figure 4 illustrates examples of compositing images with distinct contents such as trees, mountains, flowers, ponds, and the sky in various locations. \n\nWe would also like to highlight that our approach is versatile, as it allows for the incorporation of any guidance loss function. As demonstrated in Section S.4 of our supplementary, we have also experimented with Style Loss [2]. Furthermore, the weight assigned to the guidance can be adjusted, as explored in Section S.2 of the supplementary. We intend to continue investigating various types of guidance losses within our SyncDiffusion framework.\n\n\n**(2) Comparison with real panorama images**\n\nPlease refer to Section (4) of our global response above (Additional Experimental Results), where we report the quantitative comparison of Intra-LPIPS and Intra-Style-L for our output images and real panorama images. As demonstrated in the results, the comparison with real panorama images shows that SyncDiffusion is capable of generating realistic panorama images.\n\n\n**(3) Mean-CLIP-S and Mean-GIQA are on par, and FID is even worse than that of MultiDiffusion.**\n\nWe would like to reiterate our analysis of the trade-off between coherence and diversity, as explained in Figure 5, Table S2, and Figure S2 in our supplementary materials. As the weight for perceptual similarity guidance increases, our quantitative evaluations demonstrate an improvement in coherence (lower Intra-LPIPS and Intra-Style-L scores), while fidelity remains unaffected (Mean-CLIP-S remains unchanged), and diversity decreases (higher FID and KID scores). This outcome is expected, given the greater difficulty in generating coherent images compared to producing incoherent ones. Users can adjust the guidance weight to prioritize either coherence or diversity. Furthermore, our additional user study results also highlight the effectiveness of our method (please refer to the following answer and Section (2) of our global response).\n\n\n**(4) Further User Studies Evaluating Image Quality and Prompt Fidelity**\n\nPlease refer to Section (2) of our global response above (Additional Experimental Results), where we report the results of supplementary user studies designed to assess the image quality and prompt fidelity. The results indicate that individuals perceive the images generated by SyncDiffusion as having superior image quality and prompt fidelity compared to the baseline images.\n\n\n**(5) Layout-to-image generation and more applications**\n\nIn Section (3) of our previous global response (Additional Experimental Results) and Figure 3 of the PDF file attached to the same response (you can locate the link at the bottom of the text), we present an example of layout-to-image generation using SyncDiffusion. We aim to include additional experimental outcomes along with quantitative assessments in the revised version. We appreciate your feedback.\n\n\n**(6) Difference in FID from MultiDiffusion [1]**\n\nThe difference in FID originates from different experimental setups, specifically the different numbers of images (500/prompt in ours vs. 2000/prompt in MultiDiffusion), different resolutions of panorama images (512x3072 in ours vs. 512x4608 in MultiDiffusion), and different strides for the windows in the latent space (16 in ours vs. 8 in MultiDiffusion). We adjusted the experiment setup due to limited computation resources. Moreover, as MultiDiffusion employed eight prompts in their experiments, we utilized six of them since the authors publicly disclosed only these six prompts.\n\nBelow, we present the results using a single prompt (\"a photo of a mountain range at twilight\") when we align our experiment setup with that of MultiDiffusion. The quantitative outcome employing the MultiDiffusion experiment setup still highlights that our SyncDiffusion can significantly reduce Intra-LPIPS while causing only marginal changes to FID. Based on the observations in Section (1) of our global response, we computed the gradient descent for only the initial 5 steps for SyncDiffusion.\n\n| |FID ($\\downarrow$)|Intra-LPIPS ($\\downarrow$)|\n|:--|:--:|:---:|\n|Stable Diffusion| 5.09 | 0.728 |\n|MultiDiffusion| 11.04 | 0.640 |\n|SyncDiffusion (Ours)| 13.04 | 0.547 |\n\n**References**\n\n[1] Bar-Tal, Omer, et al. \"Multidiffusion: Fusing diffusion paths for controlled image generation.\" ICML (2023).\n\n[2] Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. \"Image style transfer using convolutional neural networks.\" CVPR (2016).\n"}]

Author rebuttal:
We extend our gratitude to all the reviewers for their invaluable feedback. Here, we summarize our additional experimental results.

**[Additional Experiment Results]**

**Please refer to the attached document for the figures.**

**(1) Speeding up with early-stage synchronization**

In addition to the computation time analysis in Section S.6 of the supplementary, where we explored the variation in the frequency of gradient descent computation (SyncDiffusion Function in Algorithm 1) during the reverse process, we conducted further experiments involving the computation of the SyncDiffusion function only at the early stage of the reverse process.

The table below presents the results when the gradient descent computation is performed at each time step for the initial three or five steps, out of the total 50 steps. Comparing row 4 and row 7 shows that by computing the SyncDiffusion function for just the initial three steps is analogous to computing it for ten times at regular intervals in terms of coherence (Intra-LPIPS, Intra-Style-L) and superior in terms of fidelity and diversity (FID and KID), while taking less than 70% of the latter’s computation time. The qualitative comparisons of the early-stage synchronization are shown in Figure 1 of the attached PDF (you can locate the link at the bottom of this global response).

| |Intra-LPIPS ($\downarrow$)|Intra-Style-L ($\downarrow$) ($\times 10^{-3}$)|Mean-GIQA ($\uparrow$) ($\times 10^{-3}$) | FID ($\downarrow$) | KID ($\downarrow$) ($\times 10^{-3}$) | Mean-CLIP-S ($\uparrow$) | Time (s)|
|:--|:--:|:---:|:---:|:---:|:---:|:---:|:---:|
|Stable Diffusion|0.74|8.40|26.70|28.31|<0.01|31.63|-|
|MultiDiffusion|0.69|2.98|**28.54**|**33.52**|**9.04**|31.77|46.10|
|Ours (f=50)|**0.56**|**1.39**|27.17|44.60|21.00|31.84|339.53|
|Ours (f=10)|0.62|2.14|28.43|36.22|12.84|**31.85**|104.83|
|Ours (f=5)|0.64|2.33|28.44|35.18|11.43|31.81|81.17|
|**Ours (Init. 5 Steps)**|0.61|1.96|28.21|36.31|12.09|31.77|79.55|
|**Ours (Init. 3 Steps)**|0.62|2.07|28.43|35.40|11.15|31.79|71.20|


**(2) Further User Studies Evaluating Image Quality and Prompt Fidelity**

The table presented below showcases the results of supplementary user studies designed to assess image quality and prompt fidelity. To evaluate image quality and prompt fidelity, we employed the same user study setup utilized in our previous experiment, while changing the questions to the ones used in Imagen [2] for image quality and DALL-E [3] for prompt fidelity:

* Image Quality: "Which panorama image is of higher quality? Choose one of the following images."
* Prompt Fidelity: "Which panorama image best matches the shared caption? Choose one of the following images." 

The results affirm that human evaluators perceive SyncDiffusion results as demonstrating superior image quality and higher prompt fidelity compared to the MultiDiffusion outcomes.

| |Coherence (%)|Image Quality (%)|Prompt Fidelity (%)|
|:--|:--:|:---:|:---:|
|MultiDiffusion| 33.65 | 42.81 | 40.50|
|SyncDiffusion (Ours)| **66.35** | **57.19** | **59.50**|

**(3) Layout-to-image and more applications**

In Figure 3 of the attached PDF below (you can locate the link at the bottom of this global response), we showcase a result of layout-to-image generation and offer a qualitative comparison with MultiDiffusion. Note that MultiDiffusion generates an unnatural image with an incoherent background, whereas our SyncDiffusion produces a superior outcome with a coherent background. In the revised version, we aim to incorporate more experimental results for layout-to-image generation along with quantitative evaluations.

**(4) Comparison with real panorama images**

The table below demonstrates a comparison of Intra-LPIPS and Intra-Style-L for our output images and the real panorama images. Since there is no existing benchmark for real panorama images (especially non-360 flat panorama images), we collected 100 images from Unsplash with a width at least twice larger than the height. Then, we scaled the images to match the height of our generated panorama images. For a fair comparison, we also randomly selected 100 generated panorama images and then cropped them along the width to align the resolution with that of the corresponding real images.

As demonstrated in the results below, our SyncDiffusion achieves similar or even slightly lower Intra-LPIPS and Intra-Style-L scores compared to the real images. In contrast, MultiDiffusion yields much higher Intra-LPIPS and Intra-Style-L scores. This illustrates that our SyncDiffusion is capable of generating more realistic panorama images.

| |Intra-LPIPS ($\downarrow$)|Intra-Style-L ($\downarrow$) ($\times 10^{-3}$)|
|:--|:--:|:---:|
|Stable Diffusion| 0.74 | 8.4 |
|MultiDiffusion| 0.67 | 2.6 |
|SyncDiffusion (Ours)| 0.55 | 1.1 |
|Real Panorama| 0.51 | 1.6 |

**References**

[1] Bar-Tal, Omer, et al. "Multidiffusion: Fusing diffusion paths for controlled image generation." ICML (2023).

[2] Saharia, Chitwan, et al. "Photorealistic text-to-image diffusion models with deep language understanding." NeurIPS (2022).

[3] Ramesh, Aditya, et al. "Zero-shot text-to-image generation." ICML (2021).

Meta-review: The AC has carefully read the paper, reviews, author responses, and discussions. This paper tackles the problem of image diffusion models for generation montages/panoramas. SyncDiggusion improves on the state-of-the-art (both qualitatively, through user studies, and in quantitative metrics) by synchronizing multiple diffusions through gradient descent using a perceptual similarity loss. All reviewers are positive towards acceptance. The AC agrees with the reviewers that this is a solid submission to NeurIPS and thus recommends acceptance with a certain confidence.
Decision: Accept (poster)