Title: SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion model

Text-to-image generation

Panorama generation
Abstract: The remarkable capabilities of pretrained image diffusion models have been utilized not only for generating fixed-size images but also for creating panoramas. However, naive stitching of multiple images often results in visible seams. Recent techniques have attempted to address this issue by performing joint diffusions in multiple windows and averaging latent features in overlapping regions. However, these approaches, which focus on seamless montage generation, often yield incoherent outputs by blending different scenes within a single image. To overcome this limitation, we propose SyncDiffusion, a plug-and-play module that synchronizes multiple diffusions through gradient descent from a perceptual similarity loss. Specifically, we compute the gradient of the perceptual loss using the predicted denoised images at each denoising step, providing meaningful guidance for achieving coherent montages. Our experimental results demonstrate that our method produces significantly more coherent outputs compared to previous methods (66.35% vs. 33.65% in our user study) while still maintaining fidelity (as assessed by GIQA) and compatibility with the input prompt (as measured by CLIP score). We further demonstrate the versatility of our method across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation. Our project page is at https://syncdiffusion.github.io.

SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions

Abstract

The remarkable capabilities of pretrained image diffusion models have been utilized not only for generating fixed-size images but also for creating panoramas. However, naive stitching of multiple images often results in visible seams. Recent techniques have attempted to address this issue by performing joint diffusions in multiple windows and averaging latent features in overlapping regions. However, these approaches, which focus on seamless montage generation, often yield incoherent outputs by blending different scenes within a single image. To overcome this limitation, we propose SYNCDIFFUSION, a plug-and-play module that synchronizes multiple diffusions through gradient descent from a perceptual similarity loss. Specifically, we compute the gradient of the perceptual loss using the predicted denoised images at each denoising step, providing meaningful guidance for achieving coherent montages. Our experimental results demonstrate that our method produces significantly more coherent outputs for text-guided panorama generation compared to previous methods (66.35% vs. 33.65% in our user study) while still maintaining fidelity (as assessed by GIQA) and compatibility with the input prompt (as measured by CLIP score). We further demonstrate the versatility of our method across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation. Our project page is at https://syncdiffusion.github.io .

1 Introduction

Diffusion models have recently emerged as the forefront of generative models. Recent breakthroughs in text-to-image generation such as DALL · E 2 [36] and Stable Diffusion [39] are diffusion models trained with billions of images. Compared with GANs [16], diffusion models offer advantages not only in producing high-quality, realistic images but also in being utilized for conditional generation setups such as inpainting [30, 34, 41], editing [31, 18, 7, 1, 2, 34], and deblurring [48, 9], with few-shot [25, 12, 40] or even zero-shot [37, 36, 28, 46, 49] adaptation. The iterative reverse diffusion process can easily incorporate diverse conditions and regularizations at each step while guiding the entire process to produce realistic images. Hence, the diffusion model, once trained on a large-scale dataset, becomes a versatile and multi-purpose pretrained model that can be applied to various tasks and applications.

Recent work has extended the versatility of pretrained diffusion models to generate arbitrary-sized images or panoramas using either sequential [2] or joint [3] diffusion processes. Since typical image diffusion models are trained to generate fixed-sized images, creating panoramic images requires stitching multiple fixed-size images together, which can be impossible to do seamlessly without considering integration in the generation process. Two approaches have been proposed to tackle this issue. The first approach [2] involves generating the final output as a sequence of inpainting starting from an initial image, where each consecutive view image is produced while fixing the overlapped region (which is referred to as image extrapolation in their work [2]). However, this approach often struggles to seamlessly extend the given image and also tends to repeat similar patterns, resulting in unrealistic panoramas as shown in the first row of Fig. 1. The other approach is joint diffusion [3], which operates the reverse generative process simultaneously across multiple views while averaging the intermediate noisy images (or the noisy latent features) in the overlapped regions at each reverse process step. The blending of noisy latent features among the views at each denoising step can effectively generate a seamless montage of images. However, it is important to note that the content and styles of the images may vary across the views, resulting in a mixture of colorful and black-and-white images in a single panorama, as shown in the second row of Fig. 1. The lack of consistency occurs because the latent features of the overlapped regions are simply averaged without considering the coherence between them.

To address the limitation of previous work that produces unrealistic or incoherent montages, we present a novel synchronization module for joint diffusion, dubbed SYNCDIFFUSION. This module guides the reverse diffusion processes to achieve global coherence across different areas of the panorama image. Similar to previous guided diffusion methods [36, 30, 34, 41], our SYNCDIFFUSION guides the reverse diffusion process while adjusting the intermediate noisy images at each step. Our guidance is specifically provided as a gradient descent from a perceptual similarity loss calculated across multiple windows. Various off-the-shelf perceptual similarity losses such as LPIPS [53] or Style Loss [14] can be utilized in our framework. However, perceptual similarity losses computed with noisy images cannot effectively guide the denoising process. Thus, we draw inspiration from the non-Markovian formulation of DDIM [45] leveraging the prediction of the denoised output from the current noisy image at each denoising step. We compute the perceptual similarity loss using the foreseen denoised images at each step and then backpropagate the gradient through the noisy images. By leveraging the synergy with a prior seamless stitching technique [3] based on averaging latent features at each denoising step, our joint diffusion framework demonstrates the capability to generate montages that exhibit both local seamlessness and global coherence, as shown in the last row of Fig. 1. This is achieved in a zero-shot manner, without the need for retraining or fine-tuning of existing diffusion models.

In our experiments on text-guided panorama generation using Stable Diffusion 2.0 [39] model, the results demonstrate that our method achieves significantly higher coherence compared to previous methods. Quantitatively, as we increase the weight of the gradient descent, we observe improved coherence, measured by LPIPS [53] and Style Loss [14], while maintaining fidelity (measured by GIQA [17]) and compatibility with the input prompt (assessed by the CLIP score [19]). Diversity, measured by FID and KID, exhibits a trade-off with coherence, but our method still achieves much better scores compared to the baseline method. Our user studies confirm a significant preference for our method (66.35%) over the previous method (33.65%) in terms of coherence, while also suggesting superior image quality and higher prompt compatibility. Moreover, we further demonstrate the versabtility of SYNCDIFFUSION across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation.

2 Related Work

2.1 Diffusion Models

Diffusion probabilistic models [44, 11, 21, 33] are a group of generative models that generate data by sampling from an initial Gaussian distribution and iteratively applying a denoising process (referred to as the reverse process). These methods have achieved remarkable strides in image generation [39, 11, 42, 36], surpassing previous image generation models including GANs [24, 6]. DDPM [21] was among the pioneering models that showcased the impressive capability of image generation through Markovian forward and reverse processes, albeit with longer computation times in generation. This limitation was addressed by DDIM [45], which significantly reduced the sampling time in the reverse process using a non-Markovian transition formulation conditioned on the predicted denoised data. Furthermore, LDM [39] proposed incorporating the diffusion process into the latent space, achieving state-of-the-art realism in generated images and gaining attention in the text-to-image generation. Diffusion models have also demonstrated their applicability to diverse data modalities such as audio [50, 23, 29, 15], videos [22, 5], and 3D objects [35, 28, 46, 49].

2.2 Few-Shot or Zero-Shot Adaptation of Diffusion Models

Building upon the remarkable generation capabilities of pretrained public text-to-image diffusion models such as Stable Diffusion [39], recent research has introduced various methodologies for leveraging the pretrained models in diverse tasks including conditional generation, image editing, and manipulation, without the need to retrain the models from scratch. ControlNet [51] is an example of a method that enables the incorporation of additional conditions into existing text-to-image diffusion models through few-shot finetuning, wherein the image encoder is duplicated to handle the additional conditional image, and only a carefully selected subset of parameters is modified during the finetuning process. Custom Diffusion [25] also introduces a similar idea of enabling few-shot tuning while keeping the majority of parameters in the neural network frozen, but with applications of finetuning the model for a particular class or concept of images. Other previous work has also demonstrated that diffusion models can even be applied to novel tasks in a zero-shot manner. SDEdit [31] was the first to show zero-shot conditional image generation using a pretrained diffusion model by dispersing noise over the conditional image and denoising it back to a real image. RePaint [30] introduced an image inpainting idea by combining a generated foreground image and a noised background image at each time step. Similar guided diffusion ideas have also been explored for various tasks, such as image super-resolution [27, 13, 9, 43], colorization [41, 10], deblurring [48, 9], and style transfer [26, 25, 12, 40]. We propose a novel guided diffusion framework for image montage generation via joint diffusion.

2.3 Montage Generation via Diffusion Models

Panorama generation is one of the zero-shot applications of diffusion models. Since diffusion models are trained to generate images of a specific size and on a 2D plane, stitching is required to generate panoramas or textures. Most previous methods [38, 8, 2, 1] have employed inpaintingbased approaches for seamless stitching. These methods extrapolate the accumulated image and fill only the missing regions to generate the panorama or texture. In contrast, MultiDiffusion [3] and DiffCollage [52] conduct diffusion in multiple views jointly while combining noisy latent features or scores at each reverse diffusion step. While both approaches have successfully produced continuous images, they have limitations in enforcing global coherence across the panorama or texture. MVDiffusion [47], a concurrent work, extends multi-view diffusion to produce non-square panorama images such as 360 panorama images by leveraging pixel-wise correspondence and attention modules However, it focuses on achieving smooth stitching, without addressing global coherence. In this work, we propose a simple yet effective synchronization module that can be integrated into any joint diffusion process to achieve global semantic coherence.

3 Backgrounds

3.1 Diffusion Models

In this section, we provide a brief overview of the Denoising Diffusion Probabilistic Models (DDPM) [21] and Denoising Diffusion Implicit Models (DDIM) [45], which are the foundations of recent pretrained image diffusion models. The aim of DDPM is to approximate the data distribution q ( x 0 ) with a tractable model distribution p θ ( x 0 ) , which takes the form of a Markov chain with

learned Gaussian transitions p θ ( x t -1 | x t ) from p ( x T ) = N ( x T ; 0 , I ) :

The parameters of the joint distribution (known as the reverse process ) θ are learned by minimizing the negative evidence lower bound (ELBO):

where q θ ( x 1: T | x 0 ) is the forward process adding a sequence of Gaussian noise to the data while increasing the noise scale. Among the variations of the forward processes, DDPM uses the variancepreserving diffusion that parameterizes the Gaussian transitions as follows with a decreasing sequence α 1: T ∈ (0 , 1] T :

The definition of the Gaussian transitions in the forward process derives the following property:

and thus matches the choice of the starting distribution in the reverse process (a unit Gaussian) since q ( x T | x 0 ) converges to a unit Gaussian when α T is set close to 0. It also allows expressing x t with x 0 and a unit Gaussian noise variable ϵ :

In DDPM [21], the Gaussian transition p θ ( x t -1 | x t ) for each x t in the reverse process is modeled as follows 1 :

where σ 2 t = 1 -α t -1 1 -α t · ( 1 -α t α t -1 ) , and ϵ θ ( x t , t ) is a learned function that optimizes the objective in Eq. 2 when it maps each x t at time t to a unit Gaussian noise, thus resulting in the following simplified loss:

DDIM [45] provides a different perspective of seeing the same forward process as a non-Markovian process while taking the input data x 0 into consideration in reversed transitions:

Then, each transition in the reverse process is also redefined as first predicting the denoised observation x 0 given each x t and then sampling x t -1 via the conditional distribution q ( x t -1 | x t , x 0 ) :

where

is the predicted denoised observation. The key observations of DDIM are twofold. First, the same simplified objective (Eq. 7) can be used to find the best models ϵ θ ( x t , t ) in Eq. 10 that minimize the negative ELBO (Eq. 2). This means that the DDIM reverse process can be used with a pretrained DDPM without retraining. Second, a subset of the time sequence [1 , · · · , T ] can be used in the reverse process of DDIM since x s for any s < t can be sampled from x t via the x 0 prediction, enabling a significant boost in the reverse process computation.

In the rest of the paper, the operation sampling the next denoised data in the reverse process with the learned distribution p θ ( x t -1 | x t ) defined in either Eq. 6 (DDPM) or Eq. 9 (DDIM) is denoted as:

which takes a noisy data x t at timestep t and a unit Gaussian noise ϵ ∼ N ( 0 , I ) as input.

3.2 Joint Diffusion

In an image diffusion model, each sample from the data distribution is either a 2D grid of per-pixel colors, or a 2D grid of latent features (as in Latent Diffusion [39]) that can be encoded from or decoded to a real image through a pretrained encoder E and decoder D . In the rest of the paper, the term image will thus be used to refer to either a color image or a latent feature image, unless explicitly stated otherwise. Image diffusion models pretrained on fixed-size images cannot be used directly to produce arbitrary-size images. MultiDiffusion [3] has addressed this limitation by using a multi-window joint diffusion approach. The framework integrates images generated from multiple windows seamlessly by averaging colors or features across the windows at every reverse diffusion step. For instance, consider the case of generating a panorama image z ∈ R H z × W z × D . The image at each window x ( i ) ∈ R H x × W x × D is a subarea of the panorama image whose union across all the windows covers the entire panorama image. Let m ( i ) ∈ [0 , 1] H x × H x denote a binary mask for the subregion in the panorama image corresponding to the i -th window. The function T z → i : R H z × W z × D → R H x × W x × D maps (crops) the panorama image z to the i -th window image, while T i → z : R H x × W x × D → R H z × W z × D is its inverse function that fills the region outside of the mask m i with zeros. During the joint diffusion process running the reverse process simultaneously for each window, the noisy images from the windows x ( i ) t are first averaged in the panorama space:

and then, the resulting combined noisy image z t is cropped again for each window x ( i ) t = T z → i ( z t ) , modifying the noisy image at each window with a inter-window regularization.

4 SyncDiffusion

While MultiDiffusion [3] can generate seamless panorama images from joint diffusion, it often fails to produce coherent and realistic montages. The left image in Fig. 2 demonstrates that the resulting image often oddly combines various scenes, such as mountains with trees and snow. Also, the blending occasionally fails to merge them in a realistic manner, as shown in the figure where distant objects are connected to closer objects. This incoherence issue in MultiDiffusion arises due to two main reasons. Firstly, the averaging operation only aligns the colors or features in the overlapped regions but does not match the content or style of the images. Secondly, it only enforces adjacent views to influence each other, and thus global coherence between distant windows cannot be achieved.

𝒟(𝐱

!

)

𝒟(𝜙

"

𝐱

!

, 𝑡

)

𝒟(𝐱

#

)

0.6485

0.6433

0.5416

0.3501

0.5911

0.3704

Algorithm 1: Pseudocode of one-time denoising in SYNCDIFFUSION.

To address this problem, we introduce a module called SYNCDIFFUSION which enables the generation of coherent montages, as shown on the right in Fig. 2. This module can be easily integrated into an existing joint diffusion framework. Similar to MultiDiffusion, our SYNCDIFFUSION module updates the noisy image at every step of the reverse diffusion process. In contrast to averaging the colors or latent features in the overlapped regions, however, SYNCDIFFUSION employs the backpropagation of gradients from a perceptual similarity loss computed across the windows to perform the update. The perceptual similarity loss, denoted as L , can utilize any off-the-shelf loss function for perceptual similarity, such as LPIPS [53] and Style Loss [14]. To facilitate efficient computation, we designate an anchor window with an index of 0 . For each view's noisy color image D ( x ( i ) t ) and the anchor window's noisy color image D ( x (0) t ) (where the decoder D can be treated as an identity function if the given diffusion model operates in image space rather than latent space), one can measure the coherence using the images and conduct gradient descent through x ( i ) t :

where w is the weight of the gradient descent. However, the coherence measured with the noisy images cannot provide meaningful guidance. Fig. 3 shows examples where the left three images are the intermediate noisy images D ( x t ) at timestep t = 45 out of a total of 50 timesteps in the DDIM

'A photo of a city skyline at night'

'A photo of a mountain range at twilight'

BLD [1]

MD[3]

Sync

Diffusion

'A cartoon panorama of spring summer beautiful nature'

'A natural landscape in anime style illustration'

BLD [1]

MD[3]

Sync

Diffusion

reverse process. Note that the LPIPS scores among the noisy images are indistinguishable. Hence, similar to the DDIM reverse process, we utilize the foreseen denoised observation of each noisy data ϕ θ ( x ( i ) t , t ) in Eq. 10. We measure the coherence not with the current noisy color images {D ( x ( i ) t ) } but with the predicted denoised color images {D ( ϕ θ ( x ( i ) t , t )) } with the timestep t and perform the backpropagation of the gradient through x ( i ) t , resulting in the following updated formulation:

In Fig. 3, the middle three images depict the predicted denoised images D ( ϕ θ ( x t , t )) at timestep t = 45 , which closely resemble the final generated images D ( x 0 ) at timestep t = 0 , even during the initial stages of the reverse diffusion process. Therefore, the LPIPS scores among the predicted denoised images also match those of the generated images, providing meaningful guidance for maintaining coherence. During each denoising step in the joint reverse process, we apply this update to the noisy images for all windows { x ( i ) t } , and sample the one-step denoised images. MultiDiffusion is also applied to average the sampled images at the end. Refer to Alg. 1 for detailed pseudocode.

5 Results

5.1 Text-Guided Panorama Generation

In our experiments, we generate panorama images using our SYNCDIFFUSION method and the pretrained Stable Diffusion 2.0 [39] model. Stable Diffusion model operates in a latent space of R 64 × 64 × 4 and generates images of R 512 × 512 × 3 . We generate panorama images of resolution 512 × 3072 ( 64 × 384 in the latent space), where the width is six times the width of the output of Stable Diffusion. Each window x ( i ) has an image resolution of 512 × 512 , with a stride of 128 pixels along the width in the image space which is equivalent to stride 16 in the latent space, resulting in a total 21 windows to operate diffusion processes jointly. We use six text prompts from MultiDiffusion [3] (see S.8 of the supplementary ) and generate 500 panoramas per prompt. For the gradient descent weight w (Eq. 14), we experiment with various initial values while applying a weight decay with a rate of 0 . 95 . We also set the center window as the anchor window with an index of 0 .

Baselines We compare our SYNCDIFFUSION with previous methods that generate panoramas using a pretrained diffusion model. Blended Latent Diffusion [1] is an inpainting-based method that

extrapolates a single window image. MultiDiffusion [3], served as the base of our framework, is a special case of our method when the weight of gradient descent w is 0 . The same Stable Diffusion 2.0 model is used for all the methods for a fair comparison.

Evaluation Metrics We utilize a range of metrics to assess the coherence, fidelity, diversity, and compatibility of the output panoramas with the input prompt.

(Coherence) Intra-LPIPS [53] and Intra-Style-L [14]: To assess the coherence of the generated panoramas, we introduce two metrics. Intra-LPIPS and Intra-Style-L, which are computed as the averages of LPIPS [53] and Style Loss [14], respectively, between a pair of non-overlapping window images from the same panorama. Specifically, we divide the panorama into 6 windows, each with dimensions of 512 × 512 and then compute the average of LPIPS and Style Loss across the 15 combinations of these cropped views. To provide a reference for the scale of these values, we generate 500 single-window-size images using the same Stable Diffusion model and compute the LPIPS and Style Loss for randomly selected 1,000 pairs of these reference images.

(Fidelity) Mean-GIQA [17]: GIQA quantifies the fidelity of individual images by calculating the inverse of the distance between a query image and a reference set in a feature space. Mean-GIQA is computed by taking a single random crop of each panorama in 512 × 512 size and computing the average GIQA score from each cropped image to the reference set of images mentioned above.

(Fidelity & Diversity) FID [20] and KID [4]: FID [20] and KID [4] are used to measure both fidelity and diversity. Both of them are measured with the aforementioned randomly cropped images and the set of reference images.

(Compatibility with the Input Prompt) Mean-CLIP-S [19]: The compatibility with the input prompt is assessed using the mean of CLIP scores [19], denoted as Mean-CLIP-S. This metric is calculated using the same set of cropped images and the input prompt.

Qualitative Comparisons Fig. 4 showcases qualitative comparisons between our method and the baseline methods. Here, we show the results of our method generated with a weight parameter of w = 20 . Blended Latent Diffusion [1] often exhibits visible seams due to the sequential inpainting scheme and produces repetitive patterns in the extrapolation, as illustrated by the mountains in the second case of the top row and the flowers and trees in both cases of the bottom row. MultiDiffusion [3] achieves seamless outputs, although it often produces incoherent outputs, such as mixing a sunset sky with a blue sky, as shown in the second cases of the first and second rows, and pink and purple backgrounds with blue backgrounds in the two cases of the bottom row. Our SYNCDIFFUSION generates visually and semantically more coherent panoramas with all the prompts. More qualitative comparisons are provided in the supplementary (S.1, S.8) .

Quantitative Results Fig. 5.1 presents a quantitative comparison among the methods. Our method's results are displayed using two different gradient descent weights, w = 10 and 20 . MultiDiffusion [3] is also the case when w = 0 in our framework. For results obtained with different weights, please refer to the supplementary (S.2) . The color bars in the plots indicate the average scores across the six prompts, while the black lines depict the standard deviation. Note that as the gradient weight increases, both Intra-LPIPS and Intra-Style-L decrease. When w = 20 , the Intra-LPIPS and Intra-Style-L of our method are approximately 3/4 and 1/6 of those computed with the reference set images (referred to as SD, Stable Diffusion), respectively, indicating significantly higher coherence. Moreover, the MeanCLIP-S and Mean-GIQA scores are comparable to those computed with the reference set, meaning that the compatibility with the input prompt and fidelity are not compromised by our diffusion synchronization. The results of FID and KID demonstrate the trade-off between coherence and diversity. As the gradient descent weight w increases, FID and KID also increase slightly, although they are still much lower compared to Blended Latent Diffusion [1]. This implies that for certain images, it is more difficult to find coherent images. In the supplementary (S.3) , we substantiate this claim with the results of shorter generated panoramas. Blended Latent Diffusion results in low Intra-LPIPS due to its tendency to repeat similar patterns, but it leads to low Mean-GIQA and very high FID and KID, indicating a significant degradation in fidelity.

User Study We conducted three user studies to further evaluate the coherence, image quality and prompt compatibility of the generated panoramas, respectively. Following Ritchie [32], participants were presented with panorama images generated by both MultiDiffusion [3] and our SYNCDIFFUSION methods (with w = 20) . They were then asked to choose one of them by answering the question: Which one appears a more coherent panorama image to you? (Coherence) , Which one is of higher quality? (Image Quality) , or Which one best matches the shared caption? (Prompt Compatibility) . We collected 25 responses each, including 5 vigilance

0.85

Intra-LPIPS↓

Intra-Style-L↓(x10-3)

Mean-CLIP-S↑

0.80

14

34

0.75

12

0.70

10

32

0.65

8

30

0.60

6

0.55

28

0.50

0.45

2

26

0.40

SD

BLD

MD

Ours

Ours

SD

BLD

MD

Ours

Ours

SD

BLD

MD

Ours

Ours

(w=10)(w=20)

(w =10) (w=20)

(w =10) (w =20)

FID←

KID↓ (×10-3)

35

120

100

30

100

80

25

80

60

20

60

15

40

40

10

20

5

20

2.4 × 10-3

±0.13

01

SD

BLD

MD

Ours

sIno

0

SD

BLD

MD

Ours

Ours

0

SD

BLD

MD

Ours

Ours

(w =10) (w = 20)

(w =10) (w = 20)

(w =10) (w =20)

tasks, from 100 participants for each user study. The results in Tab. 1 affirm that human evaluators perceive SYNCDIFFUSION as producing more coherent results compared to MultiDiffusion, while also demonstrating superior image quality and higher prompt compatibility. Refer to the supplementary (S.7) for detailed setups for the user study.

5.2 Additional Applications of SYNCDIFFUSION

We further demonstrate the versatility of SYNCDIFFUSION through three additional plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation.

Layout-Guided Image Generation Plugging SYNCDIFFUSION into the layout-to-image pipeline in MultiDiffusion [3] leads to a notable enhancement in the global coherence as displayed in Fig. 6-(A). While MultiDiffusion (middle row) generates an unnatural image with incoherent background around the house and the bear, our method produces a natural image with a globally coherent background.

Conditional Image Generation When integrated with ControlNet [51], SYNCDIFFUSION extends the conditional image generation to arbitrary resolutions. Let c ∈ R H z × W z × 3 denote an input condition and ψ c → i : R H z × W z × 3 → R H x × W x × 3 be a mapping from c to the i -th cropped condition corresponding to the window x ( i ) t . We define conditional SYNCDIFFUSION by substituting ϕ θ ( x t , t ) in Alg. 1 with ϕ θ ( x t , t, c ( i ) ) , where c ( i ) := ψ c → i ( c ) . Fig. 6-(B) illustrates that the combination of ControlNet and SYNCDIFFUSION generates coherent panoramas while reflecting the given condition Canny edge map (top row).

360-degree Panorama Generation We further plug SYNCDIFFUSION into MVDiffusion [47], a concurrent work that generates 360-degree panoramas from text prompts via multi-view diffusion. As shown Fig. 6-(C), our SYNCDIFFUSION distinctly improves the global coherence of the generated panorama. The increase in coherence becomes more apparent when comparing perspective views

MultiDiffusion

+  SyncDiffusion

'A beautiful city on a sunny day in oil painting'

'A beautiful city under the sunset'

'A digital painting of a city in a faraway planet'

MVDiffusion

Layout

+ SyncDiffusion

'Digital illustration of a modern living room'

(A) Layout-Guided Image Generation

(B) Conditional Image Generation

View 1

View 2

MVDiffusion

+ SyncDiffusion

360

Panorama

Perspective

View

Canny edge map

agrassland underabeautiful sky

awooden

house

abear

from different angles. While View 1 and View 2 from the vanilla MVDiffusion (top row) seem to be from two different rooms, with our method the generated images better depict two views from the same room (bottom row).

Limitations While our SYNCDIFFUSION module can significantly enhance the coherence of generated panoramas, it relies on appropriate input prompts to achieve realistic results, as illustrated in Fig. 7. Also, the SYNCDIFFUSION module that includes a forward pass through the neural network and gradient descent computation introduces additional computational overhead.

Supplementary Due to space constraints, we present the following additional results in the Supplementary: more qualitative comparisons with various prompts (S.1, S.8), details about the quantitative evalution (S.2), evaluation on different resolutions (S.3), results with Style Loss [14] as the perceptual loss (S.4), an ablation study using Eq. 13 instead of Eq. 14 (S.5), an analysis of computation time (S.6), and details on the user study (S.7).

'A red sports car'

'A fancy hotel room'

6 Conclusion

We presented SYNCDIFFUSION, a diffusion synchronization module designed to generate coherent montages through joint diffusions. Using a pretrained diffusion model, we propose guiding the reverse process by updating the noisy images at each intermediate step using gradient descent. This update is based on a perceptual similarity loss calculated with the predictions of the denoised images. Moreover, the idea of SYNCDIFFUSION can be applied to generating textures for 3D models. We plan to investigate such possibilities in future work.

Potential Negative Societal Impacts Image generative models can potentially generate deepfakes, images resembling copyrighted material, biased or discriminatory images, and harmful outputs. Future research is needed to advance the detection of manipulated content and establish societal barriers to protect intellectual property.

Acknowledgments and Disclosure of Funding

We thank Juil Koo for valuable discussions on diffusion models and Eunji Hong for help in conducting user studies. This work was partially supported by the NRF grant (RS2023-00209723) and IITP grants (2019-0-00075, 2022-0-00594, RS-2023-00227592) funded by the Korean government (MSIT), the Technology Innovation Program (20016615) funded by the Korean government (MOTIE), grants from ETRI, KT, NCSOFT, and Samsung Electronics, and computing resource support from KISTI.

References

Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. In SIGGRAPH , 2023.

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR , 2022.

Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In ICML , 2023.

Mikołaj Bi´ nkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR , 2018.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High-resolution video synthesis with latent diffusion models. In CVPR , 2023.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR , 2019.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR , 2023.

Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2Tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11396 , 2023.

Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In ICLR , 2023.

Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In NeurIPS , 2022.

Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS , 2021.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR , 2023.

Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In CVPR , 2023.

Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR , 2016.

Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction-tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731 , 2023.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.

Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. GIQA: Generated image quality assessment. In ECCV , 2020.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-toprompt image editing with cross attention control. In ICLR , 2023.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP , 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS , 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In ICLR , 2022.

Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-An-Audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661 , 2023.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR , 2020.

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR , 2023.

Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. In ICLR , 2023.

Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing , 2022.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3D: High-resolution text-to-3D content creation. In CVPR , 2023.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 , 2023.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR , 2022.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2022.

Rudimentary framework for running two-alternative forced choice (2afc) perceptual studies on mechanical turk. https://github.com/brownvc/mturk-2afc .

Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , 2021.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML , 2022.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In ICLR , 2023.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML , 2021.

Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. TEXTure: Text-guided texturing of 3D shapes. arXiv preprint arXiv:2302.01721 , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR , 2023.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-It-3D: High-fidelity 3D creation from a single image with diffusion prior. arXiv preprint arXiv:2303.14184 , 2023.

Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. MVDiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint arXiv:2307.01097 , 2023.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR , 2022.

Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3D: Zero-shot text-to-3D synthesis using 3D shape prior and text-to-image diffusion models. In CVPR , 2023.

Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE Transactions on Audio, Speech and Language Processing , 2022.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming-Yu Liu. DiffCollage: Parallel generation of large content with diffusion models. In CVPR , 2023.

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new module called SyncDiffusion that enables to synchronize multiple outputs from diffusion models to create coherent montage. This module guides the reverse diffusion process in each window by shifting x_t with gradients of a perceptual loss between the target window and an anchor window. The experimental results show that SyncDiffusion contributes to produce more coherent montage compared to previous methods.\n\nSTRENGTHS:\n- This study adresses an important problem, namely how to synchronize multiple estimates from diffusion models to produce a single output. In particular, how to guarantee a global conherence in the output image is highly non-trivial, but should be critical for practical applications such as panorama image generation.\n- The main idea, which is to use gradients of a perceptual loss between target and anchor windows to ensure the global coherence, is simple and can be jointly used with various methods for combining multiple outputs from diffusion models.\n\nWEAKNESSES:\n- The current design of the proposed algorithm relies heavily on the homogeneity of the target scene, which may limit its applicability. This is due to LPIPS used for guidance, which forces every window share a similar composition of the image. Personally, I wonder this causes the failure cases shown in Fig. 6.\n- The evaluation by Intra-LPIPS and Intra-Style-L is interesting to see a global coherence in a resultant image, but the superiority of the values is not clear, because it is unknown how much the values will be in real panorama images.\n- Somewhat related to above, the effectiveness of the proposed method is not clear. In the quantitative comparison, Mean-CLIP-S and Mean-GIQA are on par, and FID is even worse than that of MD (I understood why it happans, though). On the other hand, the user study is conducted only for evaluating the coherence, not for image quality nor prompt fidelity.\n- Although the authors argue that SyncDiffusion is a plug-and-play module, it is evaluated only with MultiDiffusion in the experiments.\n\nQUESTIONS:\n- Please see Weaknesses.\n- FID of MultiDiffusion in Fig. 5 seems much different from that shown in the original paper [3] (around 30 vs ~10). Why is it? \n- It seems straightforward to apply SyncDiffusion to layout-to-image task by just adding an appropriate cropping process before computing the perceptual loss. This should make quantitative comparison much easier as done in MultiDiffusion or [R1]. Why did the authors focus on generating panorama images in the experiments?\n  - [R1] \"Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC,\" ICLR 2023.\n\n---\n\nGiven the authors' response, I updated the rating accrodingly.\n\nLIMITATIONS:\n- Several limitation are described in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a diffusion synchronization module, a plug-and-play module for text-to-panorama generation. The key idea is based on the observation that predicted denoised images closely resemble the final outputs even at the beginning of the denoising process. Authors proposed to synchronize multiple diffusions through gradient descent from a perceptual similarity loss. Experiments demonstrate the effectiveness of the proposed method in generating coherent montages with good fidelity and compatibility with prompts.\n\nSTRENGTHS:\n(1)\tThis work is well motivated based on the observation that Predicted denoised images closely resemble the final outputs even at the beginning of the denoising process. In addition, it also helps local seamlessness and global coherence.  \n(2)\tEvaluation metrics used in quantitative comparison experiments are elaborate.  \n(3)\tThe paper has high writing quality which makes it easy to understand.\n\nWEAKNESSES:\n(1)\tBased on MultiDiffusion, the author introduces guidance information provided solely by gradient descent from a perceptual similarity loss. I wonder if this guidance is too weak or have they explored other guidance?  \n(2)\tPretrained stable diffusion v2 is used to generate 512x512-sized images. How can stable diffusion be utilized to generate wider images? Please provide a detailed explanation in the paper. The setting of text-to-panorama generation task should be further clarified.   \n(3)\tThrough the quantitative comparisons in Fig. 5, it can be observed that the setting of gradient descent weights has significant impact on the generation results. However, it is unclear how this affects the qualitative or visual effects. I wonder if the authors have experimented with other weights and is it possible that a weight other than w=20 could yield better results.  \nAdditional Comments  \n(1)\tL171, L179, writing errors, Fig. 4 -> Fig. 2\n(2)\tL191, L198, writing errors, Fig. 4 -> Fig. 3\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes, authors mentioned the limitation of the proposed in its high sensitivity to prompt and additional computation overhead.\nThey also mentioned its potential negative sociatal impact on deepfake.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of image diffusion models for generation montages/panoramas. Their proposed method, dubbed SyncDiggusion, improves on the state-of-the-art (both qualitatively, through user studies, and in quantitative metrics) by synchronizing multiple diffusions through gradient descent using a perceptual similarity loss. This provides a guidance in the reverse diffusion process by adjusting intermediate noisy images at each step. Notably, off-the-shelf perceptual similarity losses can be interchangeably used with this method.\n\nSTRENGTHS:\n1. The paper proposes a novel and interesting solution to the problem of consistent panoramic image generation using diffusion models\n1. The method is original, to my knowledge\n1. The paper is well written and clear\n1. The paper presents adequate theoretical and algorithmic explanations\n1. The paper presents a comprehensive set of both qualitative and quantitative results that demonstrate the benefits of their method over previous state-of-the-art, highlighting their contributions well\n\nWEAKNESSES:\n1. Coherence is measured not with the current noisy color images $\\{D(x_t^{(i)})\\}$ but with the predicted denoised color images $\\{D(\\phi(x_t^{(i)}, t))\\}$ (L195). What is the computational overhead of this computation? Similarly, What is the added computational overhead of this method as compared to the reported baselines? A discussion of this is omitted, although mentioned in brief in the limitations sections.\n1. In turn, a discussion of the tradeoffs between consistency vs. diversity vs. computational overhead would then be beneficial.\n\nQUESTIONS:\n1. Can this work be extended to other applications such as videos?\n1. It was mentioned that choice of prompt is critical in this method, and can easily break it. How were the prompts for the user study chosen? Choice of prompts would greatly influence the outcome of the study, potentially in favour of the proposed algorithm.\n\nLIMITATIONS:\nYes, limitations are discussed in brief.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper addresses the problem of lack of global coherence in panoramas generated with diffusion models (in a tiled manner with overlapping regions). This work builds upon MultiDiffusion [1], which proposed the technique of averaging multiple predictions for an overlapping region at each denoising step to avoid edge artifacts. However, MultiDiffusion lacked long range coherence.
This paper proposes interleaving an additional guidance step using perceptual similarity loss to ensure that each region is coherent with an anchor region.

[1] MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, Bar-Tal et al, 2023

STRENGTHS:
- The solution of guiding the denoised image with a perceptual similarity loss for global coherence is simple yet effective
- Qualitative and quantitative results are thorough and back the effectiveness of the approach in achieve global coherence
- Paper is very well written, adequately discusses background, related work and limitations.

WEAKNESSES:
- This slows down the inference process as the authors discuss in the supplementary

QUESTIONS:
None

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07