SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: Pseudocode of one-time denoising in SYNCDIFFUSION.
Parameters:w; //Gradient descent weight. Inputs:  $\{\mathbf{x}_t^{(i)}\}_{i = 0\dots N - 1}$  //Noisy images at timestep t. Outputs:  $\{\mathbf{x}_{t - 1}^{(i)}\}_{i = 0\dots N - 1}$  // Noisy images at timestep  $t - 1$    
Function SyncDiffusion  $(\{\mathbf{x}_t^{(i)}\})$  ..  $\hat{\mathbf{x}}_t^{(i)}\gets \mathbf{x}_t^{(i)};$  //The anchor window at index O is not changed. for  $i = 1,\dots ,N - 1$  do  $\begin{array}{r}\left\lfloor \hat{\mathbf{x}}_t^{(i)}\leftarrow \mathbf{x}_t^{(i)} - w\nabla_{\mathbf{x}_t^{(i)}}\mathcal{L}\left(\mathcal{D}(\phi_\theta (\mathbf{x}_t^{(i)},t)),\mathcal{D}(\phi_\theta (\mathbf{x}_t^{(0)},t))\right);\right.\\ \left.\right.\end{array}$  Gradient descent (Eq. 14) return  $\{\hat{\mathbf{x}}_t^{(i)}\}$  .   
Function MultiDiffusion  $(\{\tilde{\mathbf{x}}_t^{(i)}\})$  ..  $\mathbf{z}_t\gets \frac{\sum_i\mathcal{T}_{i\to\mathbf{z}}(\tilde{\mathbf{x}}_t^{(i)})}{\sum_i\mathbf{m}^{(i)}};$  //Averaging in the global space (Eq. 12). for  $i = 0,\ldots ,N - 1$  do  $\begin{array}{rl}{\mathrm{\bf~x}_{t}^{(i)}\leftarrow T_{\mathrm{z}\rightarrow i}(\mathrm{\bf z}_{t});} & {} \end{array}$    
return  $\{\mathbf{x}_t^{(i)}\}$  .   
Function DenoisingOneStep  $(\{\mathbf{x}_t^{(i)}\})$  ..   
 $\{\hat{\mathbf{x}}_t^{(i)}\} \leftarrow \mathrm{SyncDiffusion}(\{\mathbf{x}_t^{(i)}\})$  . for  $i = 0,\dots ,N - 1$  do  $\begin{array}{rl}{\mathrm{\bf~x}_{t - 1}^{(i)}\leftarrow S(\hat{\mathbf{x}}_t^{(i)},t,\epsilon);} & {} \end{array}$  // Sampling the next denoised data (Eq. 11).   
 $\{\mathbf{x}_{t - 1}^{(i)}\} \leftarrow \mathrm{MultiDiffusion}(\{\tilde{\mathbf{x}}_{t - 1}^{(i)}\})$  .   
return  $\{\mathbf{x}_{t - 1}^{(i)}\}$
To address this problem, we introduce a module called SYNCDIFFUSION which enables the generation of coherent montages, as shown on the right in Fig. 2. This module can be easily integrated into an existing joint diffusion framework. Similar to MultiDiffusion, our SYNCDIFFUSION module updates the noisy image at every step of the reverse diffusion process. In contrast to averaging the colors or latent features in the overlapped regions, however, SYNCDIFFUSION employs the backpropagation of gradients from a perceptual similarity loss computed across the windows to perform the update. The perceptual similarity loss, denoted as  $\mathcal{L}$ , can utilize any off-the-shelf loss function for perceptual similarity, such as LPIPS [53] and Style Loss [14]. To facilitate efficient computation, we designate an anchor window with an index of 0. For each view's noisy color image  $\mathcal{D}(\mathbf{x}_t^{(i)})$  and the anchor window's noisy color image  $\mathcal{D}(\mathbf{x}_t^{(0)})$  (where the decoder  $\mathcal{D}$  can be treated as an identity function if the given diffusion model operates in image space rather than latent space), one can measure the coherence using the images and conduct gradient descent through  $\mathbf{x}_t^{(i)}$ :
$$
\hat {\mathbf {x}} _ {t} ^ {(i)} = \mathbf {x} _ {t} ^ {(i)} - w \nabla_ {\mathbf {x} _ {t} ^ {(i)}} \mathcal {L} \left(\mathcal {D} \left(\mathbf {x} _ {t} ^ {(i)}\right), \mathcal {D} \left(\mathbf {x} _ {t} ^ {(0)}\right)\right), \tag {13}
$$
where  $w$  is the weight of the gradient descent. However, the coherence measured with the noisy images cannot provide meaningful guidance. Fig. 3 shows examples where the left three images are the intermediate noisy images  $\mathcal{D}(\mathbf{x}_t)$  at timestep  $t = 45$  out of a total of 50 timesteps in the DDIM
Figure 4: Qualitative comparisons. Blended Latent Diffusion [1] (BLD, the first row of each case) tends to exhibit visible seams and repetitive patterns. MultiDiffusion [3] (MD, the second row of each case) generates seamless results but lacks coherence, such as blending a sunset sky with a blue sky (top right), and displaying a combination of purple, pink, and blue backgrounds (bottom). In contrast, our SYNCDIFFUSION (the last in each case) produces seamless panoramicas with significantly improved coherence. Best view in zoom and color.
reverse process. Note that the LPIPS scores among the noisy images are indistinguishable. Hence, similar to the DDIM reverse process, we utilize the foreseen denoised observation of each noisy data  $\phi_{\theta}(\mathbf{x}_t^{(i)},t)$  in Eq. 10. We measure the coherence not with the current noisy color images  $\{\mathcal{D}(\mathbf{x}_t^{(i)})\}$  but with the predicted denoised color images  $\{\mathcal{D}(\phi_{\theta}(\mathbf{x}_t^{(i)},t))\}$  with the timestep  $t$  and perform the backpropagation of the gradient through  $\mathbf{x}_t^{(i)}$ , resulting in the following updated formulation:
$$
\hat {\mathbf {x}} _ {t} ^ {(i)} = \mathbf {x} _ {t} ^ {(i)} - w \nabla_ {\mathbf {x} _ {t} ^ {(i)}} \mathcal {L} \left(\mathcal {D} \left(\phi_ {\theta} \left(\mathbf {x} _ {t} ^ {(i)}, t\right), \mathcal {D} \left(\phi_ {\theta} \left(\mathbf {x} _ {t} ^ {(0)}, t\right)\right)\right). \right. \tag {14}
$$
In Fig. 3, the middle three images depict the predicted denoised images  $\mathcal{D}(\phi_{\theta}(\mathbf{x}_t,t))$  at timestep  $t = 45$ , which closely resemble the final generated images  $\mathcal{D}(\mathbf{x}_0)$  at timestep  $t = 0$ , even during the initial stages of the reverse diffusion process. Therefore, the LPIPS scores among the predicted denoised images also match those of the generated images, providing meaningful guidance for maintaining coherence. During each denoising step in the joint reverse process, we apply this update to the noisy images for all windows  $\{\mathbf{x}_t^{(i)}\}$ , and sample the one-step denoised images. MultiDiffusion is also applied to average the sampled images at the end. Refer to Alg. 1 for detailed pseudocode.
5.1 Text-Guided Panorama Generation
In our experiments, we generate panorama images using our SYNCDIFFUSION method and the pretrained Stable Diffusion 2.0 [39] model. Stable Diffusion model operates in a latent space of  $\mathbb{R}^{64\times 64\times 4}$  and generates images of  $\mathbb{R}^{512\times 512\times 3}$ . We generate panorama images of resolution  $512\times 3072$  ( $64\times 384$  in the latent space), where the width is six times the width of the output of Stable Diffusion. Each window  $\mathbf{x}^{(i)}$  has an image resolution of  $512\times 512$ , with a stride of 128 pixels along the width in the image space which is equivalent to stride 16 in the latent space, resulting in a total 21 windows to operate diffusion processes jointly. We use six text prompts from MultiDiffusion [3] (see S.8 of the supplementary) and generate 500 panoramas per prompt. For the gradient descent weight  $w$  (Eq. 14), we experiment with various initial values while applying a weight decay with a rate of 0.95. We also set the center window as the anchor window with an index of 0.
Baselines We compare our SYNCDIFFUSION with previous methods that generate panoramicas using a pretrained diffusion model. Blended Latent Diffusion [1] is an inpainting-based method that
extrapolates a single window image. MultiDiffusion [3], served as the base of our framework, is a special case of our method when the weight of gradient descent  $w$  is 0. The same Stable Diffusion 2.0 model is used for all the methods for a fair comparison.
Evaluation Metrics We utilize a range of metrics to assess the coherence, fidelity, diversity, and compatibility of the output panoramas with the input prompt.
- (Coherence) Intra-LPIPS [53] and Intra-Style-L [14]: To assess the coherence of the generated panoramas, we introduce two metrics. Intra-LPIPS and Intra-Style-L, which are computed as the averages of LPIPS [53] and Style Loss [14], respectively, between a pair of non-overlapping window images from the same panorama. Specifically, we divide the panorama into 6 windows, each with dimensions of  $512 \times 512$  and then compute the average of LPIPS and Style Loss across the 15 combinations of these cropped views. To provide a reference for the scale of these values, we generate 500 single-window-size images using the same Stable Diffusion model and compute the LPIPS and Style Loss for randomly selected 1,000 pairs of these reference images.
- (Fidelity) Mean-GIQA [17]: GIQA quantifies the fidelity of individual images by calculating the inverse of the distance between a query image and a reference set in a feature space. Mean-GIQA is computed by taking a single random crop of each panorama in  $512 \times 512$  size and computing the average GIQA score from each cropped image to the reference set of images mentioned above.
- (Fidelity & Diversity) FID [20] and KID [4]: FID [20] and KID [4] are used to measure both fidelity and diversity. Both of them are measured with the aforementioned randomly cropped images and the set of reference images.
- (Compatibility with the Input Prompt) Mean-CLIP-S [19]: The compatibility with the input prompt is assessed using the mean of CLIP scores [19], denoted as Mean-CLIP-S. This metric is calculated using the same set of cropped images and the input prompt.
Qualitative Comparisons Fig. 4 showcases qualitative comparisons between our method and the baseline methods. Here, we show the results of our method generated with a weight parameter of  $w = 20$ . Blended Latent Diffusion [1] often exhibits visible seams due to the sequential inpainting scheme and produces repetitive patterns in the extrapolation, as illustrated by the mountains in the second case of the top row and the flowers and trees in both cases of the bottom row. MultiDiffusion [3] achieves seamless outputs, although it often produces incoherent outputs, such as mixing a sunset sky with a blue sky, as shown in the second cases of the first and second rows, and pink and purple backgrounds with blue backgrounds in the two cases of the bottom row. Our SYNCDIFFUSION generates visually and semantically more coherent panoramicas with all the prompts. More qualitative comparisons are provided in the supplementary (S.1, S.8).
Quantitative Results Fig. 5.1 presents a quantitative comparison among the methods. Our method's results are displayed using two different gradient descent weights,  $w = 10$  and 20. MultiDiffusion [3] is also the case when  $w = 0$  in our framework. For results obtained with different weights, please refer to the supplementary (S.2). The color bars in the plots indicate the average scores across the six prompts, while the black lines depict the standard deviation. Note that as the gradient weight increases, both Intra-LPIPS and Intra-Style-L decrease. When  $w = 20$ , the Intra-LPIPS and Intra-Style-L of our method are approximately 3/4 and 1/6 of those computed with the reference set images (referred to as SD, Stable Diffusion), respectively, indicating significantly higher coherence. Moreover, the Mean-CLIP-S and Mean-GIQA scores are comparable to those computed with the reference set, meaning that the compatibility with the input prompt and fidelity are not compromised by our diffusion synchronization. The results of FID and KID demonstrate the trade-off between coherence and diversity. As the gradient descent weight  $w$  increases, FID and KID also increase slightly, although they are still much lower compared to Blended Latent Diffusion [1]. This implies that for certain images, it is more difficult to find coherent images. In the supplementary (S.3), we substantiate this claim with the results of shorter generated panoramas. Blended Latent Diffusion results in low Intra-LPIPS due to its tendency to repeat similar patterns, but it leads to low Mean-GIQA and very high FID and KID, indicating a significant degradation in fidelity.
User Study We conducted three user studies to further evaluate the coherence, image quality and prompt compatibility of the generated panoramas, respectively. Following Ritchie [32], participants were presented with panorama images generated by both MultiDiffusion [3] and our SYNCDIFFUSION methods (with  $w = 20$ ). They were then asked to choose one of them by answering the question: Which one appears a more coherent panorama image to you? (Coherence), Which one is of higher quality? (Image Quality), or Which one best matches the shared caption? (Prompt Compatibility). We collected 25 responses each, including 5 vigilance
Figure 5: Quantitative results. MultiDiffusion [3] (MD) can be considered as a special case of our method when the gradient descent weight  $w$  is set to 0. As  $w$  increases, coherence (Intra-LPIPS and Intra-Style-L) improves while maintaining the compatibility with the input prompt (Mean-CLIP-S) and fidelity (Mean-GIQA). There is a trade-off between coherency and diversity, as indicated by the FID and KID results. Note that the FID and KID of our method are still significantly lower than those of Blended Latent Diffusion [1] (BLD). SD (Stable Diffusion) is the score with the reference set images. Refer to the text for the details.
tasks, from 100 participants for each user study. The results in Tab. 1 affirm that human evaluators perceive SYNCDIFFUSION as producing more coherent results compared to MultiDiffusion, while also demonstrating superior image quality and higher prompt compatibility. Refer to the supplementary (S.7) for detailed setups for the user study.
Coherence (%)	Image Quality (%)	Prompt Compatibility (%)
MultiDiffusion [3]	33.65	42.81	40.50
SYNCDIFFUSION	66.35	57.19	59.50
Table 1: User study results.
5.2 Additional Applications of SYNCDIFFUSION
We further demonstrate the versatility of SYNCDIFFUSION through three additional plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation.
Layout-Guided Image Generation Plugging SYNCDIFFUSION into the layout-to-image pipeline in MultiDiffusion [3] leads to a notable enhancement in the global coherence as displayed in Fig. 6-(A). While MultiDiffusion (middle row) generates an unnatural image with incoherent background around the house and the bear, our method produces a natural image with a globally coherent background.
Conditional Image Generation When integrated with ControlNet [51], SYNCDIFFUSION extends the conditional image generation to arbitrary resolutions. Let  $c \in \mathbb{R}^{H_z \times W_z \times 3}$  denote an input condition and  $\psi_{c \to i}: \mathbb{R}^{H_z \times W_z \times 3} \to \mathbb{R}^{H_x \times W_x \times 3}$  be a mapping from  $c$  to the  $i$ -th cropped condition corresponding to the window  $\mathbf{x}_t^{(i)}$ . We define conditional SYNCDIFFUSION by substituting  $\phi_\theta(\mathbf{x}_t, t)$  in Alg. 1 with  $\phi_\theta(\mathbf{x}_t, t, c^{(i)})$ , where  $c^{(i)} := \psi_{c \to i}(c)$ . Fig. 6-(B) illustrates that the combination of ControlNet and SYNCDIFFUSION generates coherent panoramicas while reflecting the given condition Canny edge map (top row).
360-degree Panorama Generation We further plug SYNCDIFFUSION into MVDiffusion [47], a concurrent work that generates 360-degree panoramas from text prompts via multi-view diffusion. As shown Fig. 6-(C), our SYNCDIFFUSION distinctly improves the global coherence of the generated panorama. The increase in coherence becomes more apparent when comparing perspective views
Figure 6: Plug-and-play applications of SYNCDIFFUSION.
from different angles. While View 1 and View 2 from the vanilla MVDiffusion (top row) seem to be from two different rooms, with our method the generated images better depict two views from the same room (bottom row).
Limitations While our SYNCDIFFUSION module can significantly enhance the coherence of generated coronas, it relies on appropriate input prompts to achieve realistic results, as illustrated in Fig. 7. Also, the SYNCDIFFUSION module that includes a forward pass through the neural network and gradient descent computation introduces additional computational overhead.
Supplementary Due to space constraints, we present the following additional results in the Supplementary: more qualitative comparisons with various prompts (S.1, S.8), details about the quantitative evaluation (S.2), evaluation on different resolutions (S.3), results with Style Loss [14] as the perceptual loss (S.4), an ablation study using Eq. 13 instead of Eq. 14 (S.5), an analysis of computation time (S.6), and details on the user study (S.7).
Figure 7: Our failure cases. A suitable input prompt is required to generate realistic panoramas.
We presented SYNCDIFFUSION, a diffusion synchronization module designed to generate coherent montages through joint diffusions. Using a pretrained diffusion model, we propose guiding the reverse process by updating the noisy images at each intermediate step using gradient descent. This update is based on a perceptual similarity loss calculated with the predictions of the denoised images. Moreover, the idea of SYNCDIFFUSION can be applied to generating textures for 3D models. We plan to investigate such possibilities in future work.
Potential Negative Societal Impacts Image generative models can potentially generate deepfakes, images resembling copyrighted material, biased or discriminatory images, and harmful outputs. Future research is needed to advance the detection of manipulated content and establish societal barriers to protect intellectual property.
We thank Juil Koo for valuable discussions on diffusion models and Eunji Hong for help in conducting user studies. This work was partially supported by the NRF grant (RS2023-00209723) and IITP grants (2019-0-00075, 2022-0-00594, RS-2023-00227592) funded by the Korean government (MSIT), the Technology Innovation Program (20016615) funded by the Korean government (MOTIE), grants from ETRI, KT, NCSOFT, and Samsung Electronics, and computing resource support from KISTI.
[1] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. In SIGGRAPH, 2023.
[2] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
[3] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. MultiDiffusion: Fusing diffusion paths for controlled image generation. In ICML, 2023.
[4] Mikołaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR, 2018.
[5] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your Latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
[7] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
[8] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2Tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11396, 2023.
[9] Hyungjin Chung, Jeongsol Kim, Michael Thompson McCann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In ICLR, 2023.
[10] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. In NeurIPS, 2022.
[11] Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021.
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
[13] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In CVPR, 2023.
[14] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, 2016.
[15] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction-tuned lIm and latent diffusion model. arXiv preprint arXiv:2304.13731, 2023.
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
[17] Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. GIQA: Generated image quality assessment. In ECCV, 2020.
[18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. In ICLR, 2023.
[19] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, 2018.
[43] Chitwan Sahara, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[44] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[46] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-It-3D: High-fidelity 3D creation from a single image with diffusion prior. arXiv preprint arXiv:2303.14184, 2023.
[47] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. MVDiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint arXiv:2307.01097, 2023.
[48] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Sahara, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022.
[49] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3D: Zero-shot text-to-3D synthesis using 3D shape prior and text-to-image diffusion models. In CVPR, 2023.
[50] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE Transactions on Audio, Speech and Language Processing, 2022.
[51] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[52] Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming-Yu Liu. DiffCollage: Parallel generation of large content with diffusion models. In CVPR, 2023.
[53] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.