SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuseung Lee Kunho Kim Hyunjin Kim Minhyuk Sung
{phillip0701,kaist984,rlaguswls98,mhsung}@kaist.ac.kr
The remarkable capabilities of pretrained image diffusion models have been utilized not only for generating fixed-size images but also for creating panoramas. However, naive stitching of multiple images often results in visible seams. Recent techniques have attempted to address this issue by performing joint diffusions in multiple windows and averaging latent features in overlapping regions. However, these approaches, which focus on seamless montage generation, often yield incoherent outputs by blending different scenes within a single image. To overcome this limitation, we propose SYNCDIFFUSION, a plug-and-play module that synchronizes multiple diffusions through gradient descent from a perceptual similarity loss. Specifically, we compute the gradient of the perceptual loss using the predicted denoised images at each denoising step, providing meaningful guidance for achieving coherent montages. Our experimental results demonstrate that our method produces significantly more coherent outputs for text-guided panorama generation compared to previous methods (66.35% vs. 33.65% in our user study) while still maintaining fidelity (as assessed by GIQA) and compatibility with the input prompt (as measured by CLIP score). We further demonstrate the versatility of our method across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation. Our project page is at https://syncdiffusion.github.io.
Figure 1: Comparison of panoramas generated with prompt "A photo of a rock concert" by Blended Latent Diffusion [1] (top), MultiDiffusion [3] (middle), and our SYNCDIFFUSION (bottom). Blended Latent Diffusion, when applied on image extrapolation, often generates visible seams and repetitive patterns. MultiDiffusion creates seamless panoramas but fails to achieve global coherence across the image. In contrast, our SYNCDIFFUSION synchronizes windows across the panorama by increasing the perceptual similarity of the denoised output predictions. This results in significantly more coherent panorama outputs.
Diffusion models have recently emerged as the forefront of generative models. Recent breakthroughs in text-to-image generation such as DALL-E 2 [36] and Stable Diffusion [39] are diffusion models trained with billions of images. Compared with GANs [16], diffusion models offer advantages not only in producing high-quality, realistic images but also in being utilized for conditional generation setups such as inpainting [30, 34, 41], editing [31, 18, 7, 1, 2, 34], and deblurring [48, 9], with few-shot [25, 12, 40] or even zero-shot [37, 36, 28, 46, 49] adaptation. The iterative reverse diffusion process can easily incorporate diverse conditions and regularizations at each step while guiding the entire process to produce realistic images. Hence, the diffusion model, once trained on a large-scale dataset, becomes a versatile and multi-purpose pretrained model that can be applied to various tasks and applications.
Recent work has extended the versatility of pretrained diffusion models to generate arbitrary-sized images or panoramas using either sequential [2] or joint [3] diffusion processes. Since typical image diffusion models are trained to generate fixed-sized images, creating panoramic images requires stitching multiple fixed-size images together, which can be impossible to do seamlessly without considering integration in the generation process. Two approaches have been proposed to tackle this issue. The first approach [2] involves generating the final output as a sequence of inpainting starting from an initial image, where each consecutive view image is produced while fixing the overlapped region (which is referred to as image extrapolation in their work [2]). However, this approach often struggles to seamlessly extend the given image and also tends to repeat similar patterns, resulting in unrealistic panoramas as shown in the first row of Fig. 1. The other approach is joint diffusion [3], which operates the reverse generative process simultaneously across multiple views while averaging the intermediate noisy images (or the noisy latent features) in the overlapped regions at each reverse process step. The blending of noisy latent features among the views at each denoising step can effectively generate a seamless montage of images. However, it is important to note that the content and styles of the images may vary across the views, resulting in a mixture of colorful and black-and-white images in a single panorama, as shown in the second row of Fig. 1. The lack of consistency occurs because the latent features of the overlapped regions are simply averaged without considering the coherence between them.
To address the limitation of previous work that produces unrealistic or incoherent montages, we present a novel synchronization module for joint diffusion, dubbed SYNCDIFFUSION. This module guides the reverse diffusion processes to achieve global coherence across different areas of the panorama image. Similar to previous guided diffusion methods [36, 30, 34, 41], our SYNCDIFFUSION guides the reverse diffusion process while adjusting the intermediate noisy images at each step. Our guidance is specifically provided as a gradient descent from a perceptual similarity loss calculated across multiple windows. Various off-the-shelf perceptual similarity losses such as LPIPS [53] or Style Loss [14] can be utilized in our framework. However, perceptual similarity losses computed with noisy images cannot effectively guide the denoising process. Thus, we draw inspiration from the non-Markovian formulation of DDIM [45] leveraging the prediction of the denoised output from the current noisy image at each denoising step. We compute the perceptual similarity loss using the foreseen denoised images at each step and then backpropagate the gradient through the noisy images. By leveraging the synergy with a prior seamless stitching technique [3] based on averaging latent features at each denoising step, our joint diffusion framework demonstrates the capability to generate montages that exhibit both local seamlessness and global coherence, as shown in the last row of Fig. 1. This is achieved in a zero-shot manner, without the need for retraining or fine-tuning of existing diffusion models.
In our experiments on text-guided panorama generation using Stable Diffusion 2.0 [39] model, the results demonstrate that our method achieves significantly higher coherence compared to previous methods. Quantitatively, as we increase the weight of the gradient descent, we observe improved coherence, measured by LPIPS [53] and Style Loss [14], while maintaining fidelity (measured by GIQA [17]) and compatibility with the input prompt (assessed by the CLIP score [19]). Diversity, measured by FID and KID, exhibits a trade-off with coherence, but our method still achieves much better scores compared to the baseline method. Our user studies confirm a significant preference for our method  $(66.35\%)$  over the previous method  $(33.65\%)$  in terms of coherence, while also suggesting superior image quality and higher prompt compatibility. Moreover, we further demonstrate the versatility of SYNCDIFFUSION across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation.
Diffusion probabilistic models [44, 11, 21, 33] are a group of generative models that generate data by sampling from an initial Gaussian distribution and iteratively applying a denoising process (referred to as the reverse process). These methods have achieved remarkable strides in image generation [39, 11, 42, 36], surpassing previous image generation models including GANs [24, 6]. DDPM [21] was among the pioneering models that showcased the impressive capability of image generation through Markovian forward and reverse processes, albeit with longer computation times in generation. This limitation was addressed by DDIM [45], which significantly reduced the sampling time in the reverse process using a non-Markovian transition formulation conditioned on the predicted denoised data. Furthermore, LDM [39] proposed incorporating the diffusion process into the latent space, achieving state-of-the-art realism in generated images and gaining attention in the text-to-image generation. Diffusion models have also demonstrated their applicability to diverse data modalities such as audio [50, 23, 29, 15], videos [22, 5], and 3D objects [35, 28, 46, 49].
2.2 Few-Shot or Zero-Shot Adaptation of Diffusion Models
Building upon the remarkable generation capabilities of pretrained public text-to-image diffusion models such as Stable Diffusion [39], recent research has introduced various methodologies for leveraging the pretrained models in diverse tasks including conditional generation, image editing, and manipulation, without the need to retrain the models from scratch. ControlNet [51] is an example of a method that enables the incorporation of additional conditions into existing text-to-image diffusion models through few-shot finetuning, wherein the image encoder is duplicated to handle the additional conditional image, and only a carefully selected subset of parameters is modified during the finetuning process. Custom Diffusion [25] also introduces a similar idea of enabling few-shot tuning while keeping the majority of parameters in the neural network frozen, but with applications of finetuning the model for a particular class or concept of images. Other previous work has also demonstrated that diffusion models can even be applied to novel tasks in a zero-shot manner. SDEdit [31] was the first to show zero-shot conditional image generation using a pretrained diffusion model by dispersing noise over the conditional image and denoising it back to a real image. RePaint [30] introduced an image inpainting idea by combining a generated foreground image and a noised background image at each time step. Similar guided diffusion ideas have also been explored for various tasks, such as image super-resolution [27, 13, 9, 43], colorization [41, 10], deblurring [48, 9], and style transfer [26, 25, 12, 40]. We propose a novel guided diffusion framework for image montage generation via joint diffusion.
2.3 Montage Generation via Diffusion Models
Panorama generation is one of the zero-shot applications of diffusion models. Since diffusion models are trained to generate images of a specific size and on a 2D plane, stitching is required to generate panoramas or textures. Most previous methods [38, 8, 2, 1] have employed inpainting-based approaches for seamless stitching. These methods extrapolate the accumulated image and fill only the missing regions to generate the panorama or texture. In contrast, MultiDiffusion [3] and DiffCollage [52] conduct diffusion in multiple views jointly while combining noisy latent features or scores at each reverse diffusion step. While both approaches have successfully produced continuous images, they have limitations in enforcing global coherence across the panorama or texture. MVDiffusion [47], a concurrent work, extends multi-view diffusion to produce non-square panorama images such as 360 panorama images by leveraging pixel-wise correspondence and attention modules. However, it focuses on achieving smooth stitching, without addressing global coherence. In this work, we propose a simple yet effective synchronization module that can be integrated into any joint diffusion process to achieve global semantic coherence.
In this section, we provide a brief overview of the Denoising Diffusion Probabilistic Models (DDPM) [21] and Denoising Diffusion Implicit Models (DDIM) [45], which are the foundations of recent pretrained image diffusion models. The aim of DDPM is to approximate the data distribution  $q(\mathbf{x}_0)$  with a tractable model distribution  $p_{\theta}(\mathbf{x}_0)$ , which takes the form of a Markov chain with
learned Gaussian transitions  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  from  $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T;\mathbf{0},\mathbf{I})$  ..
$$
p _ {\theta} \left(\mathbf {x} _ {0}\right) = \int p _ {\theta} \left(\mathbf {x} _ {0: T}\right) d \mathbf {x} _ {1: T}, \quad \text {w h e r e} \quad p _ {\theta} \left(\mathbf {x} _ {0: T}\right) = p \left(\mathbf {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right). \tag {1}
$$
The parameters of the joint distribution (known as the reverse process)  $\theta$  are learned by minimizing the negative evidence lower bound (ELBO):
$$
\min  _ {\theta} \mathbb {E} _ {q \left(\mathbf {x} _ {0}\right)} \left[ - \log p _ {\theta} \left(\mathbf {x} _ {0}\right) \right] \leq \min  _ {\theta} \mathbb {E} _ {q \left(\mathbf {x} _ {0}, \mathbf {x} _ {1}, \dots , \mathbf {x} _ {T}\right)} \left[ - \log p _ {\theta} \left(\mathbf {x} _ {0: T}\right) + \log q \left(\mathbf {x} _ {1: T} \mid \mathbf {x} _ {0}\right) \right], \tag {2}
$$
where  $q_{\theta}(\mathbf{x}_{1:T}|\mathbf{x}_0)$  is the forward process adding a sequence of Gaussian noise to the data while increasing the noise scale. Among the variations of the forward processes, DDPM uses the variance-preserving diffusion that parameterizes the Gaussian transitions as follows with a decreasing sequence  $\alpha_{1:T}\in (0,1]^T$ :
$$
q \left(\mathbf {x} _ {1: T} \mid \mathbf {x} _ {0}\right) := \prod_ {t = 1} ^ {T} q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right), \text {w h e r e} q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right) := \mathcal {N} \left(\sqrt {\frac {\alpha_ {t}}{\alpha_ {t - 1}}} \mathbf {x} _ {t - 1}, \left(1 - \frac {\alpha_ {t}}{\alpha_ {t - 1}}\right) \mathbf {I}\right). \tag {3}
$$
The definition of the Gaussian transitions in the forward process derives the following property:
$$
q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right) := \mathcal {N} \left(\mathbf {x} _ {t}; \sqrt {\alpha_ {t}} \mathbf {x} _ {0}, (1 - \alpha_ {t}) \mathbf {I}\right), \tag {4}
$$
and thus matches the choice of the starting distribution in the reverse process (a unit Gaussian) since  $q(\mathbf{x}_T|\mathbf{x}_0)$  converges to a unit Gaussian when  $\alpha_T$  is set close to 0. It also allows expressing  $\mathbf{x}_t$  with  $\mathbf{x}_0$  and a unit Gaussian noise variable  $\epsilon$ :
$$
\mathbf {x} _ {t} = \sqrt {\alpha_ {t}} \mathbf {x} _ {0} + \sqrt {1 - \alpha_ {t}} \epsilon , \quad \text {w h e r e} \quad \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I}). \tag {5}
$$
In DDPM [21], the Gaussian transition  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  for each  $\mathbf{x}_t$  in the reverse process is modeled as follows:
$$
p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) := \mathcal {N} \left(\sqrt {\frac {\alpha_ {t - 1}}{\alpha_ {t}}} \left(\mathbf {x} _ {t} - \frac {1}{\sqrt {1 - \alpha_ {t}}} \left(1 - \frac {\alpha_ {t}}{\alpha_ {t - 1}}\right) \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t\right)\right), \sigma_ {t} ^ {2} \mathbf {I}\right), \tag {6}
$$
where  $\sigma_t^2 = \frac{1 - \alpha_{t-1}}{1 - \alpha_t} \cdot \left(1 - \frac{\alpha_t}{\alpha_{t-1}}\right)$ , and  $\epsilon_\theta(\mathbf{x}_t, t)$  is a learned function that optimizes the objective in Eq. 2 when it maps each  $\mathbf{x}_t$  at time  $t$  to a unit Gaussian noise, thus resulting in the following simplified loss:
$$
L \left(\epsilon_ {\theta}\right) := \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} _ {0} \sim q \left(\mathbf {x} _ {0}\right), \epsilon_ {t} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} \left[ \| \epsilon_ {\theta} \left(\sqrt {\alpha_ {t}} \mathbf {x} _ {0} + \sqrt {1 - \alpha_ {t}} \epsilon_ {t}, t\right) - \epsilon_ {t} \| _ {2} ^ {2} \right]. \tag {7}
$$
DDIM [45] provides a different perspective of seeing the same forward process as a non-Markovian process while taking the input data  $\mathbf{x}_0$  into consideration in reversed transitions:
$$
q \left(\mathbf {x} _ {1: T} \mid \mathbf {x} _ {0}\right) := q \left(\mathbf {x} _ {T} \mid \mathbf {x} _ {0}\right) \prod_ {t = 2} ^ {T} q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right), \quad \text {w h e r e} \quad q \left(\mathbf {x} _ {T} \mid \mathbf {x} _ {0}\right) = \mathcal {N} \left(\sqrt {\alpha_ {T}} \mathbf {x} _ {0}, (1 - \alpha_ {T}) \mathbf {I}\right) \tag {8}
$$
$$
\mathrm {a n d} \forall s <   t, q (\mathbf {x} _ {s} | \mathbf {x} _ {t}, \mathbf {x} _ {0}) = \mathcal {N} \left(\sqrt {\alpha_ {s}} \mathbf {x} _ {0} + \sqrt {1 - \alpha_ {s} - \sigma_ {t} ^ {2}} \cdot \frac {\mathbf {x} _ {t} - \sqrt {\alpha_ {t}} \mathbf {x} _ {0}}{\sqrt {1 - \alpha_ {t}}}, \sigma_ {t} ^ {2} \mathbf {I}\right).
$$
Then, each transition in the reverse process is also redefined as first predicting the denoised observation  $\mathbf{x}_0$  given each  $\mathbf{x}_t$  and then sampling  $\mathbf{x}_{t - 1}$  via the conditional distribution  $q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)$ :
$$
p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) := \left\{ \begin{array}{l l} q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \phi_ {\theta} \left(\mathbf {x} _ {t}, t\right)\right) & \text {i f} t \geq 2 \\ \mathcal {N} \left(\phi_ {\theta} \left(\mathbf {x} _ {t}, t\right), \sigma_ {t} ^ {2} \mathbf {I}\right) & \text {i f} t = 1, \end{array} \right. \tag {9}
$$
<sup>1</sup> Note that  $\beta_{t}$  in the DDPM [21] is equivalent to  $1 - \frac{\alpha_{t}}{\alpha_{t-1}}$  in our paper and DDIM [45].
$$
\phi_ {\theta} \left(\mathbf {x} _ {t}, t\right) = \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathbf {x} _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t\right)\right) \tag {10}
$$
is the predicted denoised observation. The key observations of DDIM are twofold. First, the same simplified objective (Eq. 7) can be used to find the best models  $\epsilon_{\theta}(\mathbf{x}_t,t)$  in Eq. 10 that minimize the negative ELBO (Eq. 2). This means that the DDIM reverse process can be used with a pretrained DDPM without retraining. Second, a subset of the time sequence  $[1,\dots ,T]$  can be used in the reverse process of DDIM since  $\mathbf{x}_s$  for any  $s < t$  can be sampled from  $\mathbf{x}_t$  via the  $\mathbf{x}_0$  prediction, enabling a significant boost in the reverse process computation.
In the rest of the paper, the operation sampling the next denoised data in the reverse process with the learned distribution  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  defined in either Eq. 6 (DDPM) or Eq. 9 (DDIM) is denoted as:
$$
\mathcal {S} \left(\mathbf {x} _ {t}, t, \epsilon\right), \tag {11}
$$
which takes a noisy data  $\mathbf{x}_t$  at timestep  $t$  and a unit Gaussian noise  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  as input.
3.2 Joint Diffusion
In an image diffusion model, each sample from the data distribution is either a 2D grid of per-pixel colors, or a 2D grid of latent features (as in Latent Diffusion [39]) that can be encoded from or decoded to a real image through a pretrained encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$ . In the rest of the paper, the term image will thus be used to refer to either a color image or a latent feature image, unless explicitly stated otherwise. Image diffusion models pretrained on fixed-size images cannot be used directly to produce arbitrary-size images. MultiDiffusion [3] has addressed this limitation by using a multi-window joint diffusion approach. The framework integrates images generated from multiple windows seamlessly by averaging colors or features across the windows at every reverse diffusion step. For instance, consider the case of generating a panorama image  $\mathbf{z} \in \mathbb{R}^{H_z \times W_z \times D}$ . The image at each window  $\mathbf{x}^{(i)} \in \mathbb{R}^{H_x \times W_x \times D}$  is a subarea of the panorama image whose union across all the windows covers the entire panorama image. Let  $\mathbf{m}^{(i)} \in [0,1]^{H_x \times H_x}$  denote a binary mask for the subregion in the panorama image corresponding to the  $i$ -th window. The function  $\mathcal{T}_{\mathbf{z} \to i}: \mathbb{R}^{H_z \times W_z \times D} \to \mathbb{R}^{H_x \times W_x \times D}$  maps (crops) the panorama image  $z$  to the  $i$ -th window image, while  $\mathcal{T}_{i \to \mathbf{z}}: \mathbb{R}^{H_x \times W_x \times D} \to \mathbb{R}^{H_z \times W_z \times D}$  is its inverse function that fills the region outside of the mask  $\mathbf{m}_i$  with zeros. During the joint diffusion process running the reverse process simultaneously for each window, the noisy images from the windows  $\mathbf{x}_t^{(i)}$  are first averaged in the panorama space:
$$
\mathbf {z} _ {t} = \frac {\sum_ {i} \mathcal {T} _ {i \rightarrow \mathbf {z}} \left(\mathbf {x} _ {t} ^ {(i)}\right)}{\sum_ {i} \mathbf {m} ^ {(i)}}, \tag {12}
$$
and then, the resulting combined noisy image  $\mathbf{z}_t$  is cropped again for each window  $\mathbf{x}_t^{(i)} = \mathcal{T}_{\mathbf{z}\to i}(\mathbf{z}_t)$ , modifying the noisy image at each window with a inter-window regularization.
Figure 2: Panoramas generated by MultiDiffusion [3] (left) and our SYNCDIFFUSION (right), with a prompt "A photo of a mountain range at twilight". MultiDiffusion often combines various scenes, such as mountains with trees and snow, and even awkwardly blends them. In contrast, SYNCDIFFUSION generates panoramas that are significantly more coherent.
4 SyncDiffusion
While MultiDiffusion [3] can generate seamless panorama images from joint diffusion, it often fails to produce coherent and realistic montages. The left image in Fig. 2 demonstrates that the resulting image often oddly combines various scenes, such as mountains with trees and snow. Also, the blending occasionally fails to merge them in a realistic manner, as shown in the figure where distant objects are connected to closer objects. This incoherence issue in MultiDiffusion arises due to two main reasons. Firstly, the averaging operation only aligns the colors or features in the overlapped regions but does not match the content or style of the images. Secondly, it only enforces adjacent views to influence each other, and thus global coherence between distant windows cannot be achieved.
Figure 3: LPIPS [53] scores computed across the noisy images  $\mathcal{D}(\mathbf{x}_t)$  at the intermediate step  $(t = 45$  out of 50) of the reverse process (left), the predicted denoised images  $\mathcal{D}(\phi (\mathbf{x}_t,t))$  at the same timestep  $t$  (middle), and the final generated images  $\mathcal{D}(\mathbf{x}_0)$  at timestep  $t = 0$  (right). The indistinguishable noisy images yield similar LPIPS scores among them, whereas the predicted denoised images, which closely resemble the final outputs even at the beginning of the denoising process, exhibit LPIPS scores that align with those of the final generated images. This indicates that the predicted denoised images can provide meaningful guidance for producing coherent panoramicas in the diffusion process.