Patched Denoising Diffusion Models For High-Resolution Image Synthesis
Authors: Zheng Ding
Mengqi Zhang
Jiajun Wu
Zhuowen Tu
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Models
Primary area: generative models
TL;DR: We propose Patch-DM which uses a feature collage mechanism to produce high-resolution images based on diffusion models.

Abstract: We propose an effective denoising diffusion model for generating high-resolution images (e.g., 1024$\times$512), trained on small-size image patches (e.g., 64$\times$64). We name our algorithm Patch-DM, in which a new feature collage strategy is designed to avoid the boundary artifact when synthesizing large-size images. Feature collage systematically crops and combines partial features of the neighboring patches to predict the features of a shifted image patch, allowing the seamless generation of the entire image due to the overlap in the patch feature space. Patch-DM produces high-quality image synthesis results on our newly collected dataset of nature images (1024$\times$512), as well as on standard benchmarks of LHQ(1024$\times$ 1024), FFHQ(1024$\times$ 1024) and on other datasets with smaller sizes (256$\times$256), including LSUN-Bedroom, LSUN-Church, and FFHQ. We compare our method with previous patch-based generation methods and achieve state-of-the-art FID scores on all six datasets. Further, Patch-DM also reduces memory complexity compared to the classic diffusion models. Project page: https://patchdm.github.io.

Published as a conference paper at ICLR 2024
PATCHED DENOISING DIFFUSION MODELS FOR HIGH- RESOLUTION IMAGE SYNTHESIS
Zheng Ding
Mengqi Zhang
Jiajun Wu² Zhuowen Tu
UC San Diego
2 Stanford University
Figure 1: Generated image of size 1024x512 using the model trained on 21k natural images using a 148M-
parameters model.
ABSTRACT
We propose an effective denoising diffusion model for generating high-resolution images (e.g., 1024x512), trained small-size image patches (e.g., 64x 64). We name our algorithm Patch-DM. in which a new feature collage strategy designed to avoid the boundary artifact when synthesizing large-size images. Feature col- lage systematically crops and combines partial features of the neighboring patches to predict the features of shifted image patch, allowing the scamless generation the entire image due to the overlap in the patch feature space. Patch-DM pro- duces high-quality image synthesis results images (0024x512) as well as on standard benchmarks of LHQ(1024x 1024), FFHQ(1024x 1024) and other datasets with smaller sizes (256x256)) includ- ing LSUN-Bedroom, LSUN-Church, and FFHQ We compare our method with previous patch-based generation methods and achieve state-of-the-art FID scores on all six datasets. Further, Patch-DM, also reduces memory complexity compared the classic diffusion models. Project page: httpss:/patchdmmgglleee github.io githuu
newly collected dataset of nature
INTRODUCTION
There have been explosive developments in generative adversarial learning [Tu 2007 Goodfellow al. 2014| Radford al., 20166] (Arjovsyy et al. 2017 Karras et al., 2019 Donahue et al.] (017).
though many GAN models remain hard to train. VAH models (Kingma & Welling 2014) 2014)) are easier
to train, but the resulting image quality is often blurry. Diffusion generative models have lately
gained tremendous popularity with generated images of superb quality (Sohl-Dickstein et al. 2015
Ho et al.) 2020 Song & Ermon 2020 Song et al., 20200 Croitoru et al. 2023 Ramesh Rameshh et al. 2022
Despite the excellent modeling capability of generative diffusion models, the current models still face challenges in both training and synthesis.
"Equal Contribution.

---

Published as a conference paper at ICLR 2024
Due to direct optimization in the pixel space and multi-timestepy training and inference, diffusion models are hard to scale up to high-resolution image generation. Therefore, current state-of-the-an art models either use super-resolution methods to increase the generated images to higher resolutions (Ramesh et al. 2022 Saharia et al. 2022a), or optimize the latent space instead of the pixel space (Rombach et al. 2022). However, both types of approaches still consist of high-resolution image generators that consume a large memory with a big model size.
To ameliorate the limitations in the current diffusion models, we propose a new method, Patch- DM, to generate high-resolution images with a newly-introduced feature collage strategy. The basic operating point for Patch-DM
patch-level model that is relatively compact compared to those modeling the entire image. Though it appears to have introduced compromises for a patch-base representation, Patch-DM can perform seamless full-size high-resolution image synthesis without artifacts of the boundary effects for pixels near the borders of the image patches. The effectiveness of Patch-DM in directly generating high-resolution images enabled by a novel feature collage strategy. This strategy helps feature sharing by implementing a -widdow based shifted im- age patch generation process, ensuring consistency across neighboring image patches; this a key design in our proposed Patch-DM method to alleviate the boundary artifacts without requiring addi- tional parameters. To summarize, the contributions of our work are listed as follows:
We develop new denoising diffusion model based on patches, Patch-DM, to generate images of high-resolutions. Patch-DM can perform direct high-resolution image synthesis without introducing boundary artifacts.
We design new feature collage strategy where each image patch to be synthesized obtains features partially from its shifted input patch. Through systematic window sliding, the en- tire image is being synthesized by forcing feature consistency across neighboring patches. This strategy, named feature collage, gives rise to a compact model of Patch-DM that is patch-based for high-resolution image generation.
Patch-DM points to a promising direction for generative diffusion modeling at flexible patch- representation level, which allows high-resolution image synthesis with lightweight models.
RELATED WORK
Generative diffusion models. Generative diffusion models (Sohl-Dickstein et 2015 Ho al. 2020 Song & Ermon 2020) which learn to denoise noisy images into real images have gained much attention lately due its training stability and high image quality. Lots of progress has been made in diffusion models such as faster sampling(Song et al. 2021], conditional generation(Nicho) & Dhariwal 202T Dhariwal & Nichol, 202I) or high-resolution image synthesis(Rombach et al., 2022). The traits of diffusion models have been amplified particularly by the success of DALLE
(Ramesh et al. (022) and Imagen (Saharia et al. 2022a) which generate high quality images from
the given texts.
Patch-based image synthesis. The practice of employing image patches of relatively small sizes to generate images of larger sizes has been longstanding technique in computer vision and graphics, particularly in the context of exemplar-based based texture synthesis (Efros & Leung 1999). While gen- erative adversarial networks (GANs) have been utilized for expanding non-stationary textures Zhou ett al., 2018), image synthesis is still considered more challenging due to the complex structures present in images. To address this challenge, COCO-GAN(Lin et al. 2019) uses micro coordinates and latent vectors to synthesize large images by generating small patches first. InfinityGAN(Lin et al. 2022) further improves this by introducing Structure Synthesizer and Padding Free Generator to disentangle global structures and local textures and also generate consistent pixel values at the same spatial locations. ALIs(Skorokhodov et al.] (02T) proposes an alignment mechanism on la- tent and image space to generate larger images. Anyres-GAN adopts a two-stage training method by first learning the global information from low-resolution downsampled images and then learning the detailed information from small patches. There are also some works share similar directions 2022) does a reshaping operation on the input image which pushes the dimensions of the height and width to the channels. The model still takes the whole image input just with the shape changed. Wang et al. (2023) does a patch operation during the training stage by concatenating an- other position embedding layer to the input while still requires full-resolution during the inference
et al.) 2002), on the other hand,
patch-based diffusion models. Luhman & Luhman
2

---

Published as a conference paper at ICLR 2024
stage. There are also other works applying patch-based diffusion models to specific applications like image restoration under weather annditions(Ozdenizze & Legenstein 2023) and anomaly detection in brain MRI(Bccrendt et al. 2024). Both of them utilize conditional diffusion mechanism by utilizing the weather-degraded images or images that miss some patches as conditions.
Our work, Patch-DM consists of new design, feature collage, in which partial features of neighbor- ing patches are cropped and combined for predicting a shifted patch. We borrow the term "collage" from the picture collage task (Wang et al. 2006) for the ease of understanding of our method, though our feature collage strategy only has a loose conceptual connection to picture collage (Wang et al.) 2006). Adopting positional embedding in Patch-Do DM also makes it casier to maintain spatial regu- larity. Although Patch-DM employs a shifted window strategy. its motivation and implementation different from those of the widely-known Swin Transformers (Liu et al. 2021).
BACKGROUND
Denoising diffusion models generate real images from randomly sampled noise images by learning a denoising function (Ho et al., (0200). Instead of directly denoising the random noise image to a real image, denoising diffusion models learn to denoise the noise image through T steps. The forward process adds noise to the image Tu gradually while the learned denoising function fo tries to reverse this process from the ØT (0, I). More formally, the forward process time step i(I 1...T) can be defined as
It (z V11 ,,
(1)
where 3g are hyperparameters that control the noise, making the noise level of x; gradually larger through the timesteps. Note that 2, can be directly derived from the original image Zo since Eq. can be rewritten as
Ft N(ro Varano, (1 o;)
where a2 = II- - B.). In order to generate the images from the noise input, the denoising model fe learns to reverse from oe to It-1, which is defined as
G = fo t),
(3)
1
Ft-1 N(
-T- Be (x+
oI)
(4)
-T-
Cz.
where og are hyperparameters that control the variance of the denoising process. The objective of the denoising model is ||+, = et is ground truth noise added image.
Therefore, after the denoising model is trained, the model can generate real images from random
noise using Eq. [ As can be seen, the whole generation process depends fully on the denoising
process. Since the model denoises the image in the pixel space directly, the computation would be very expensive once the resolution gets higher.
PATCHED DENOISING DIFFUSION MODEL
In this section, describe our proposed Patched Denoising Diffusion Model (Patch-DM). Rather than using entire complete images for training, our model only takes patches for training and infer- ence, and
proposed feature collage mechanism systematically combine partial features of neighboring patches. Consequently, Patch-DM is capable of resolving the issue of high compu- tational costs associated with generating high-resolution images,
resolution-agnostic.
Before we dive into our model's training details, we first give an overview of the image generation process of our method. The training image from the dataset Jr RxxH we split zu into (t.f) where i.j is the row and column number of the patch, (3.f) E R.xxx Instead of di- zi
rectly generating zo like most of methods do, our model only generates (i.5) and concatenate them together to form a complete image.
very basic way do this is what show in Figure 2(a) where the denoising model takes the
noised image patch z; (1d) as input and output the corresponding noise & (15)) However, since the patches do not interact with each other, there will be severe borderline artifacts.
3

---

Published as a conference paper at ICLR 2024
(a) Collage on Patches
(b) Patch Collage in Pixel Space
(c) Patch Collage Feature Space (Ours)
Figure 2: Patch Generation For Image Synthesis. (a) shows very basic method of patch-wise image synthesis by simply splitting the images and generating patches independently. This method brings severe border artifacts. (b) alleviates the border artifacts by using shifted windows while generating images and doing patch collage in pixel space. (c) is our proposed method which collages the patches in the feature space. The features for neighboring features will be split and collaged for new patch synthesis. We will show this method is key design for generate high-quality images without border artifacts.
A further way to do this is to shift image patches during each time step depicted in Figure 2(b). At different time steps, the model will take either the original split patch si (3.3) or the shifted split patch (6:5 so that the border artifacts can be alleviated which we call "Patch Collage in Pixel Space". However, in Section we show that the border artifacts still exist.
To further improve this method, we propose novel feature collage mechanism depicted in Figure 2(c). Instead of performing patch collage the pixel space, we perform it in the feature space. The patch collage in the feature space is more in-depth and supports multi-level interaction. This allows the patches to be more cognizant of the adjacent features and prevent border artifacts from appearing while generating the complete images, More formally,
[ (id) (i.)
..... (1.5) fo (x, (f.1) ,t),
(5)
where f/ is the UNet encoder and z 22 .... 2n are the internal feature maps. We then split the feature maps and collage the split feature maps to generate shift patches
(1.5)
(i,i)
(P (6.1), Pa(#k ((++1)
P3(# ((+11))))) Pa(e
(1+1 (a( ((+++++))))
where P1, P2, P3, P4 split functions shown in Figure 2(c). Then send these collaged
shift features 2((15) to the UNet decoder to get the predicted shift patch noise:
(1.5) S2 (1=( =2 ((,j) "(."))....,...)))1)) ()))),
In order make the model generate semantically consistent images, also add position emhedding and semantic embedding to the model so that fo will take another two inputs which are P(i,j) and E(xo).
During inference time, we take a 3x3 example illustrated in Figure in order to generate the border patches, we first pad the images so that the feature collage can be done for each patch without information loss. At each time step f, image T; is decomposed into patches which are fed into the subsequent encoder. Be- fore a feature map goes through the decoder, a split and collage operation is applied to it. Thus, the decoder outputs the predicted noise of the shifted patch. According Eq. 4 able to obtain - and thus generate the final complete images.
pathh
Poilaar suse
Figure 3: Detailed Inference pracess at each time step.
4

---

Published as a conference paper at ICLR 2024
Table Quantitative comparison with previous patch-based image generation methods. models are trained on the natural images dataset (1024x512) standard benchmarks x 1024) and FFHQ(1024x 1024). We use FID to measure the overall quality of generated images and sFID for the quality of high-level structures.
Nutue-21K(1024x351
LHQ(1024x 1024)
FFHQX1024x 1024)
Method
Putch Size
FDD
sFID
FID
sFDD
FID
sFD
Li
2019
64x64
70,980
74,208
35.693
88.988
80.059
209 309.683
InfinityGAN Lin 2024
101x 101
46,550
70.041
50.646
92.577
174.789
270.699
Anyres-AA
64x64
44.173
34,430
130.591
100.041
67.076
170.911
Patch-DM (Ours)
64x64
20.369
34,405
23.777
37.217
19.696
36.512
Figure 4: Generated 2048x 1024 image. We double the number of patches that the model generate images with 2x resolution from 1024x512. The left image a 2048x 1024 image, and the right image is zoom-in of the red bounding box, with resolution 256x256.
EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
Architecture. For the model architecture, we base our denoising U-Net model from (Dhariwal & Nichol, (02T) with changes taking global conditions and positional emheddings. We use two methods to obtain the global conditions. The first is to use pretrained model to obtain the image features and use the image features as the global conditions, while optimizing the features directly during training. In this case, we do not have to increase the model parameters and can scale to high-resolution images. However, when the number of images in the training dataset is too large, optimizing the pre-obtained image features requires more effort. We use this approach for global conditioning when training on datasets of 1024x512 images. The pretrained model we use for obtaining the image embeddings is CLIPP(Rddford al. 2021). We resize the images to 224x 224 and send them to ViT-B/16 6 to obtain the features as global conditions; we then optimize these global conditions directly.
The second is jointly training an image encoder and using its output as the global conditions. Here, the jointly training image encoder may borrow the architecture in the denoising U-Net's encoder. It works particularly well when the training dataset is large. However, it requires another model. which would be bottleneck in training on high-resolution datasets, since the computation would increase significantly as the resolution increases. We use this approach when training on large datasets of x56 256 images. We utilize global conditions with dimension of 512 in both methods.
Classifier-free guidance. We also use the classifier-free guidance (Ho & Salimans 2022) to im- prove the training speed and quality. We use classifier-free guidance on both the global conditions and position embeddings. The dropout rate is 0.1 for the global conditions and 0.5 for position embeddings.
Patch size. We use patch size of 64x64 in all our experiments. The denoising U-Net model's architecture the same across all the datasets, as it is only related to the patch size regardless of the training images' resolution.
Inference. Once the denoising U-Net model has been trained, we train another latent diffusion model for unconditional image synthesis. The latent diffusion model's architecture is based on the

---

Published as a conference paper at ICLR 2024
FFH
LSUN-Churd
Figure 5:
Generated images on FFHQ, LSUN-Bedroom, and L.SU-Chuue datasets using our pro- posed method. All the resolutions are 256x256.
one described in (Preechakul et al. 2022). The data we use for training the latent diffusion model is either from the output of the trained image encoder or the directly optimized image embeddings. To synthesize an image, we will first sample a latent code from the latent diffusion model and then use this latent code to serve as the global conditions for sampling an image. During the sampling stage, we use the inference process proposed by DDIM (Song et al. 2021) and set the sampling step to 50.
Evaluation. We conduct both qualitative and quantitative evaluations on four datasets. For quan- titative evaluation, we use FID, popular metric in generative modeling(Heusel et al., (2017) and sFID al. (202T) for the quality of high-level structures. To compute FID, we follow the set- ting of (Heusel et al. 2017) and generate 50K images compute the metrics over the full dataset. We apply the same setting for sFID.
5.2 RESULTS IK-RESOLUIIO IMAGES
Setup. To show our model's capability on direct high-resolution image synthesis, we show our model's 's performance three datasets: LHQ(1024x 1024), FFHQ(1024X 1024) and self-collected 21443 natural images from Wallpaperscraft 2013-2023 across all datasets. Therefore, each image split into 16x 16 for 1024x 1024-resolution and 16x8 for 1024x51 2-resolution.
We use a patch size 64 x6
We use the CLIP ViT-B/16 pretrained image visual encoder to obtain the image embeddings first. While downsampling to fit the CLIP model may result in the loss of some detailed image informa- tion in the embeddings, these details can still be "recovered" during the embedding optimization in the training process, aided by the supervision of the original high-resolution image. Each image patch can be trained and sampled independently with feature collage assisting surrounding information. Since every image is segmented into smaller patches, the total number of model parameters is much smaller than other large diffusion models.
aware of the
As most existing diffusion models merely can directly generate images of 1k resolution, and the general strategy for high- nigh-resolution synthesis to sample hierarchically (generate relatively low- resolution images first and then perform super-resolution), our Patch-DM simplifies sample proce- dure using much more lightweight models, which one of the main advantages.
Results. We compare our model with previous patch-based image generation methods in Table 1. From the table, we can see our method delivers the best overall quality of the generated images on both FID and sFID scores. Apart from the quantitative evaluation, also present image generated by our model in Figure For more generated images, please refer to our supplementary material.
5.3 RESULTS ON 256x256 IMAGES
Setup. To compare with other existing generative models, also train our Patch-DM three standard public datasets: FFHQ, L.SUN-Bedroom, and LSUN-Church, and evaluate its sampling performance. All the resolution is 256x Thus, the number of patches is 4x4. Notice that the

---

Published as a conference paper at ICLR 2024
model architecture keeps the same; the only change here is the number of patches during training and inference. We use the same training setting across the three datasets.
Results. We report the quantitative results in Table lnd qualitative results in Figure respectively. In Table 2 we can see our model achieves competitive results while still outperforms previous patch-based methods. Figure illustrates FiguutaSillustrates that despite producing small image patches, our denoising model exhibits minimal boundary artifacts and offers good visual quality. This demonstrates the effectiveness of feature collage mechanism.
Table 2: Evaluation Metries unconditional image synthesis on three 256 datasets: FFHQ, LSUN-
Bedroom, and LSUN-Church For fair comparison, results are reproduced in the same sampling steps as ours i.. 50, using provided pretrained checkpoints of other diffusion models. We adopt a patch size of 64 x66 for Patch-DM, DM. Anyres GAN, COCO- CANN and 101 x 101 for InfinityGAN. We bold the numbers denote the best numbers the same category (top: non non-patch-base based methods, bottom:patch based methods).
FFIQQ
LSUN- Bedooom
LSUN-Church
Method
FID
sFID
FID
sFID
FID
sFID
LDM Rombach et
2022
8.76
7.09
3.40
7.53
4.23
11.44
UDM Kim el 2022
5.54
4.57
DiffAE
Preechakul al. 2022
9.71
10.24
PGGAN Karrus et al. 12018
8.34
9.21
6.42
10.48
StyleGAN Karras el al. 2018
2.35
6.62
4.21
COCO-GAN Lin et
2019
34.02
37.44
41.84
62.69
17.91
73.94
InfinityGANNI Jan et 2022
28.87
127.92
10.71
19.28
7.08
33.588
Anyres-GAN
Chai et
24.48
55.77
15.65
56.24
17.09
80.66
Patch-DM (Ours)
10.02
10.58
6.04
9.93
5.49
14.80
Model size comparison. Compared with other widely used diffusion models, our proposed method could achieve competitive performance using a smaller model with above mentioned in- dispensable components. Patch-DM is fully built upon the network on 64 patches regardless of the target image resolution and uses optimized global conditions to avoid the increase of model parameter amounts brought by higher input resolution. Comparison model parameters with other classic diffusion models 256 256 resolution is shown in Table 3 Notice that use the same diffusion model architecture for the 1024x 1024 and the 1024x5 12 resolutions.
Table 3: Number of parameters comparison between different diffusion models 256x256 resolution. SE means semantic encoder to extract global information. The size of our previously trained 1024x 1024 and 1024x512 models is 70M+(size of optimized semantic embeddings] during training and [63M latent DPM] in inference.
Method
Model Size
Base model + Super-resalution
SR3 SR3 Saharia et al. 2022b)
B[64]+625M
Direct generation
ADM Phariwal & Nichol 2021
552M
DifFA 2
Preechakul al 2022
232M
LDM Rombach
274M
Patch DM (Ours, full model)
154M
Patch-DM (Ours, w/ SF, w/o latent DPM)
9IM
Patch-DM (Ours, w/o
DPM)
70M
5.4 APPLICATIONS
We now demonstrate several applications of our Patch-DM. All of them are conducted without post- training.
Beyond patch generation. Since our method samples images using patches, we have the option to incorporate more patches during testing. This enables the model to produce images with higher resolutions compared to the ones in the training set without requiring further training. We adopt two ways to achieve this.
The first one is to add more patches internally. We test this on our Nature-214 dataset. To generate 512x 1024 images which has the same resolution the training dataset, we need 8x 16 patches (random gaussian noise maps) to start with. To generate a 2 resolution images, we can add another 8x 16 patches internally that the total patch number will become 16x32 that can generate images
7

---

Published as a conference paper at ICLR 2024
COCO-GAN
InfinityGAN
(ach-DMM(Ours)
Figure 6:
Synthesized 384x384 images on LSUN-Bedroom (256x256) and LSUN-Church (256x 2he left half of cach group -SUN-Bedroom and the right half of each group is LSUN-Church. Despite only being trained on 256x256 images, our model can generate 384x384 images by adding more patches (outside the red bounding box). Extended images generated by our models are compared with COCO- GAN and InfinityAAN, which also have the ability extend fields without further training.
Figure 7: Image outpainting on LSUN-Church and LSUN-Bedroom. The image inside the red bounding box the input image from the validation dataset. We pad the image patches from 4x4 to 6x6 to enable the image outpainting. The image parts outside the red bounding box are the outpainting results.
with a resolution of 1024x2048. The global conditions we use for these patches is the same as the original, while we interpolate the position embeddings in this setting. We provide a generated 2048 1024 image in Figure 4. As can be seen, our model can still generate consistent patches even though the newly added patches have never been used in the training process.
The second one is to add patches outside the original image. This way is similar to reyond-boundard generation in COCO-GAN(Lin et al. 2019) with a key difference that it needs a post-training pro- cess to improve the continuity among patches. We conduct an experiment on .SUN-Bedroom and
by adding more patches. The original resolution in the training data we use is 256x 256, which is divided by 4x4 patches. We add more patches to the existing 4x4 patches so that the number grows to 6x6. Therefore, the model can generate an image with a resolution of 384x384. For the original 4x4 patches, model and the position embeddings as pre-defined. For the additional patches, we use the same se- mantic condition, while don't use position embeddings for these added ones as this case we'r re adding patches outside which the position embeddings could not be interpolated. Thus the model needs to synthesize the additional patches only according to the global conditions and the neighbor- ing context information. We present our results in Figure 6 and compare it with COCO-GAN( et al. 2019) without post-training and InfinityGAN et al. 2022) under the same setting.
the condition generated from the latent diffusion
Image outpainting. Another practical application would be image outpainting that only draws the outer part of the image while keeping the input image the same. To do this, we experiment using the model trained on LSUN-Church and Church validation dataset and W.SUUN-Bedroooo validation dataset to the image encoder to obtain the global conditions. Then, to keep the original image the same, replace the inner predicted noised image patches with the ground truth noised images patches (adding corresponding noise to the input images) during each timestep while sampling. We present results in Figure 7 It can be from the results that our model can "imagine" the surrounding areas reasonably and generate rather consistent outer parts of the image without obvious border effects.
First, we send images from the LSUN-
Image inpainting. In this task, we infill the corrupted images with random masks, which requires the restored results to be consistent in context. We experiment on the LSUN-Church validation set using already trained models without further tailored training. The original images are masked by different numbers of blocks ranging from 6, and the sampling process only conditioned on local position embeddings w/o global conditions. The results are presented in Figure X From the figure, we can see that our model can infill the blocks consistently using surrounding patches, demonstrating that feature collage facilitates the model with the capability to be aware of adjacent information, enabling it to be naturally applied to inpainting tasks.
ABLATION STUDY
Three indispensable components: semantic code, position embeddings and shift window strategy on feature levels, considerably eliminate border artifacts and improve model performance. Here,
8

---

Published as a conference paper at ICLR 2024
Figure 8: Image Inpainting on LSUN-Church. Six pairs of images are presented here. For cach pair, the left side is the masked image: the right one is the inpainted result by our model without further training on this task. The number of masked patches increasing from left right.
we conduct ablation study to investigate the effects of these modules. We provide both qualitative results and quantitative results in Figure and Table respeettvvy
Global conditions. We study the problem without global conditions; thus, the generation process will fully rely on the positional embedding and neighboring context information. We present our images in Figure 9 (a). It's interesting to see how the model generates when no global conditions are given, which
strong constraint for the model to generate semantic-related patches. From the given image, we can see that the model can still generate locally-consisent images; the image quality is however relatively low.
Position embeddings. The last section shows that global conditions are necessary for our model to generate high-quality images. We then condition the model only those to investigate the role of positional embeddings. The results are shown in Figure (b). Without the position information, the model would generate distorted images with naeelohnengg to where they should be, although the whole image may follow a certain style. Hence, the positional embeddings are vital to our model.
Collage in the pixel space. A straightforward idea is to perform collage in the pixel space as present in Figure 2(b); the images are decomposed by window-shifee patches from their original positions. To maintain patch size consistency, we add zero padding around the image. For the sampling procedure: In an odd-number step. original patches are generated independently, while in an wwen-number step, patches with shifted positions are sampled.
Under this scheme, we experiment two different settings. The first is to take fixed shift step (half patch size) along the height and width direction. The sample result is shown in Figure pc)). There are still apparent artifacts along the border. This proves that even though the shift window on the image level could enable patches to be aware of surroundings during sampling, the awareness level quite limited, and the final generation similar breaking the image into smaller patches.
The second setting is to shift the patch position with randomly sampled step ranging from zero to patch size. The inference result is shown in Figure Id). The sample quality is much improved compared to the previous situation. However, the result is still not as photo-realisth as Patch-DM. The reason is that although the random shift enables finer surrounding awareness, lacks in-depth model does. Therefore, the feature-level window shift and collage can
feature interaction significantly eliminate border artifacts and improve final inference quality.
Table 4: FID evaluation 1,000 images of different ablation settings to investigate the importance of semantic condition, position embedding, and feature-level window shift.
Methud
FID (Ik)
No global semantic condition
79.33
embedking space
spuce rendom shift
Patch-DM (Ouns)
37.99
Figure 9: Ablation study global conditions (a), position embeddings (b), and feature level shift (c, d).
CONCLUSION
We have presented a new algorithm, Patch-DM, a patch-based denoising diffusion model for gen- erating high-resolution images. We introduce a feature collage strategy to combat the boundary effect for patch- based image synthesis. Patch-DM achieves a significant reduction in model size and training complexity compared the standard diffusion models trained the original size images. Competitive quantitative and qualitative results are obtained for Patch- DM when trained on several image datasets.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work supported by NSF Award IIS-2127544 and IIS-2211258.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
Finn Behrendt, Debayan Bhattacharya, Julia Krüger, Roland Opfer, and Alexander Schlaefer. Patched diffuusion models for unsupervised anomaly detection in brain mri. In Medical Imag- ing with Deep Learning, pp. 1019-1032. PMLR, 2024.
Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution train- ing for high-resolution image synthesis. In Computer Vision-ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI, pp. 170-188, Springer, 2022.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.
Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In Proceed- ings of the seventh IEEE international conference on computer vision, pp. 1033-1038, 1999.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv: 12000., 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neurul Information Processing Systems, 68511, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Proc. NeurIPS, 2021.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In International Conference on Machine Learning, pp. 1101--12288 PMLR, 2022.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Chich Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei. and Hwann- -zong Chen. Coco-gan: generation by parts via conditional coordinating. In ICCV, pp. 4512-4521, 2019.
10

---

Published as a conference paper at ICLR 2024
Ming-Hsuun Chich Hubert Lin, Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, and Ming- Hsuan Yang. Infin- ityGAN: Towards infinite- infinite-pixel image synthesis. ICLR, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Troy Luhman and Eric Luhman. Improving diffusion model efficiency through patching. arXiv preprint arXiv:2207.0431 2022.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.0384, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Ozan Ōzdenizci and Robert Legenstein. Restoring vision in adverse weather conditions with patch- based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasor Suwajanakorn Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Alec Radford, Luke Metz, and Soumith Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2xXXi:2204006125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi. Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 10X1 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. In ICLR Worshop Track, 2022b.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14144-14153 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Yang Song and Stefano Ennon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-1244 12448, 2020.

---

Published as a conference paper at ICLR 2024
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020.
Zhuowen Tu. Learning generative models via discriminative approaches. In CVPR, 2007.
Wallpaperscraft. Wallpaperscraft. htttpss/welllppperssrff...cc.
2013-2023.
Jingdong Wang, Long Quan, Jian Sun, Xiaoou Tang, and Heung-Yeung Shum. Picture collage. In CVPR. pp. 347-354, 2006.
Zhendong Wang, Yifan Jiang, Huangjie Zheng. Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, Mingyuan Zhou, et al. Patch diffusion: Faster and more data-efficient training of diffusion models. Advances in Neural Information Processing Systems, 36, 2023.
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski. Daniel Cohen-Or and Hui Huang. Non- stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics (TOG), 17(4) 2018.
12

---

Published as a conference paper at ICLR 2024
A MODEL HYPERPARAMETERS
For model architecture, we base our diffusion model from Dhariwal & Nichol (2021) with changes of taking global semantic condition and positional embedding. The hyperparameters for the main denoising U-Net model are specified in Table Since the model is resolution agnostic, the main architectures for all datasets keep the same. We adopt two methods to obtain global semantic con- ditions: for relatively low-resolution images, an encoder is trained with architecture borrowed from the first half of the U-Net model, and the architecture details of the encoder are shown in Table 6 For high-resolution images such as 1024x512, a pretrained image encoder is used to avoid scaling up the overall model size. We use ViT-B/16 CLIP to obtaín the image embeddings and optimize them during training. For position embeddings, we use sinusoidal positional embeddings. Time embedding and positional embedding are concatenated and modulated into ResBlocks together with the global code.
For realizing unconditional image synthesis, a latent diffusion model is trained on semantic embed- dings. The implementation is based the one proposed in PPrechakul et al. (2022) with MLP skip connections architecture. The parameter details are specified in Table 7
B MORE QUALITATIVE RESULTS
We provide unconditional sampling results with the models trained self-collected
We also provide results on LHQ(1024 1024) and FFHQ(1024x 1024) in Figures [4 and 1 respectively as well three other standard benchmarks
nature images (1024x512) Figure T0 12
with a resolution of x26: BedBoooo (Figure [5), LSUN-Church (Figure I6). and FFHQ
(Figure
13

---

Published as a conference paper at ICLR 2024
Parameter
Patch-DM
Patch input size
34 x64
Channel multiplier
[1, 2, 8]
Net channel
64
ResBlock number
Attention resolution
16
Batch size
16
Diffusion steps
1000
Noise scheduler
Linear
I Learning rate
0.0001
Optimizer
Adam
Table 5: Model Architecture for diffusion model.
Parameter
Semantic Encoder
Input size
252
Channel multiplier
[1, 2, 4, 8,
Net channel
64
ResBlock number
Attention resolution
16
Global condition dimension
512
Batch size
16
Learning rate
0.0001
Optimizer
Adam
Table 6: Model Architecture for image semantic encoder.
Parameter
Latent Diffusion
Input size
512
MLP layers
10
MLP hidden size
2048
Noise scheduler
Constant 0.008
Batch size
256
Learning rate
0.0001
Optimizer
Adam (weight decay 0.01)
Table 7: Model Architecture for latent diffusion model.
14

---

Published as a conference paper at ICLR 2024
Figure 10: Additional qualitative results on self-collected nature dataset (1024x512).
15

---

Published as a conference paper at ICLR 2024
Figure 11: Additional qualitative results on self-collected nature dataset (1024x512).
16

---

Published as a conference paper at ICLR 2024
Figure 12: Additional qualitative results on self-collected nature dataset (1024x512).
17

---

Published as a conference paper at ICLR 2024
Figure 13: Additional qualitative results on high resolution FFHQ dataset (1024x 1024).
18

---

Published as a conference paper at ICLR 2024
Figure 14: Additional qualitative results on high resolution LHQ dataset (1024x 1024).
19

---

Published as a conference paper at ICLR 2024
B&V
Figure 15: Additional qualitative results on --UN-BBrroom (256x 256).
20

---

Published as a conference paper at ICLR 2024
Figure 16: Additional qualitative results on LSUN-Church (256x256).
21

---

Published as a conference paper at ICLR 2024
Figure 17: Additional qualitative results on FFHQ (256x256).
22

---

Published as a conference paper at ICLR 2024
C FAILED CASES COMPARISON ON FFHQ1024
To provide a more thorough understanding of the limitations of our proposed method, we compare our FFHQ1024 results with other non-patch-based methods. We FFHQ1024 it's more structural dataset while other landscape datasets are not that structural, thus FFHQ1024 could better show how well the models leam the structural information. As there lacks results for FFHQ1024 using diffusion models, we compare our results with two state-of-the- GAN-based methods on FFHQ1024: StyleGAN3(Karras et al. 2021) and StyleGAN-XL(SSann can be seen that non-patch GAN-based methods generally perform well on global consistency while ours may not perform well such cases. For example, the first row of our results, the wrinkles around the mouth asymmetrical. And in the second row of our results, the eycs/eyeglasses are asymmetrical. As our method uses patch size of 64x64 and there are a total 16x 16=256 patches, the global consistency sometimes may not perform well. We think this by utilizing larger patch size or introducing better global-conssitencyyenfoooo which we leave for future work.
et al., 2022) in Figure [8]]
further improve mechanisms
I
StyleGAN3-T
StyleGAN-XL
Our Failed Cases
Comparison FFHQ1024 with StyleGAN3-T, StyleGAN-XL, and our failed cases. Ours might fail on global consistency such as the wrinkles around the mouth in the first row and eyes/glasses in the second row.
Figure 18:
23


Reviews:
[{"review_id": "4YYTO2QybW", "reviewer": "Reviewer_7jCc", "summary": "The paper proposes a denoising diffusion model, Patch-DM, for generating high-resolution images (e.g., 1024×512), trained on small-size image patches (e.g., 64×64). The major contribution of the paper is a new feature collage strategy, which is designed to avoid the boundary artifact when synthesizing large-size images. The authors demonstrate the effectiveness of  Patch-DM on mage synthesis results on their newly collected dataset of nature images (1024×512), as well as on standard benchmarks of LHQ(1024× 1024), FFHQ(1024× 1024) and on other datasets with smaller sizes (256×256), including LSUN-Bedroom, LSUN-Church, and FFHQ. The show state-of-the-art FID scores on all six datasets for the proposed model. Further, Patch-DM also reduces memory complexity compared to the classic diffusion models.", "questions": "1) Can you please add comparison to other techniques in the supplementary? It might be useful to reduce the example to great examples vs. poor examples and provide some discussion on failure modes\n2) In Table 2 - \"We bold the numbers to denote the best numbers in the same category.\" --> can you please explain what you mean by \"the same category\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is reasonably well written and easy to follow \n2)  The quantitative results demonstrated in Table 1 and Table 2 shows that the model outperforms state of the art.", "weaknesses": "1) The paper is not sufficiently novel. I'm not working in this domain, but the only novel part that the authors state is creating the collage of the patches in the feature / latent space based on their spatial embeddings. This does not sound like something that has not been done before in the field of image generation. It would  be helpful if the author come up with a more comprehensive literature survey that provides more related works to this particular  design choice and clearly shows the difference. For example, from a short search I found the following relevant paper: [1] https://arxiv.org/pdf/2207.04316.pdf --  Improving Diffusion Model Efficiency Through Patching \n[2] https://arxiv.org/abs/2304.12526 -- Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models (note the paper was first submitted on April 2023).\n\n2) Even if we consider the combination of Patch Diffusion in the latent space sufficiently novel, from the analysis in the supplementary material, I find that faces demonstrate usual artifacts around eyes and mouse (and I think that this is happening despite training on a dedicated dataset). Midjourney models generate much better faces. It would be great to understand why the proposed model fails on those.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aQCy0Oy3ff", "reviewer": "Reviewer_3rts", "summary": "This paper proposes a patch-based denoising diffusion model called Patch-DM for generating high-resolution images. The key contributions are: 1. Introduces a feature collage strategy to avoid boundary artifacts when synthesizing images from patches. It combines partial features from shifted patches to predict features for a new patch. 2. Achieves state-of-the-art FID scores on 1024x512 natural images and 1024x1024 LSUN/FFHQ images using a lightweight model. 3. Demonstrates Patch-DM can directly generate high-fidelity 1K resolution images with minimal patch boundary effects. 4.Reduces memory complexity compared to full-image diffusion models for high-res synthesis. 5.Shows applications like image outpainting, inpainting, super-resolution without any post-training. 6. Validates through ablation studies that feature collage is better than pixel collage for spatial consistency. 7. Provides an effective patch-based generative modeling approach using diffusion models for high-resolution image synthesis with reduced costs.", "questions": "•\tWhy does the Patch Collage in Feature Space strategy avoid artifacts? Can a detailed analysis and explanation be provided? This is crucial for future work.\n\n•\tWhat is the running speed of the proposed method? How much slower does the Patch Collage in Feature Space strategy make the model inference speed?\n\n•\tWhat are the limitations or further areas of exploration for the proposed method?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "•\tProposes Patch-DM, a novel patch-based denoising diffusion model that can generate high-resolution images directly without relying on hierarchical sampling. This simplifies the sampling procedure.\n•\tIntroduces a new feature collage strategy to avoid boundary artifacts when synthesizing images from patches. It forces consistency by combining partial features from shifted patches.\n•\tAchieves state-of-the-art FID scores on generating natural images and LSUN/FFHQ images using a lightweight model, outperforming prior patch-based methods.\n•\tQualitative results show Patch-DM can produce high-fidelity 1K resolution images with minimal patch boundary effects.", "weaknesses": "•\tBased on my experience and recent related publications (e.g., \"Weather Diffusion-PAMI'23\"), patch-based diffusion models often lead to reduced inference efficiency. I hope the authors can provide specific comparisons of inference time and overall efficiency, especially compared to previous GAN methods.\n•\tThe proposed method has limited technical contributions. The authors did not provide detailed explanations or theoretical justifications to explain why the Patch Collage in Feature Space strategy can avoid artifacts. Furthermore, the research and exploration of Semantic Code are not sufficiently in-depth.\n•\tThe authors should provide a quantitative comparison of image inpainting and image outpainting results. Quantitative results would better demonstrate the superiority of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Z4K6x4fOr", "reviewer": "Reviewer_Tdt5", "summary": "This work aims to resolve the limitation of the existing diffusion models towards generating high-resolution images. To this end, the authors present the ideas of feature collage, position embedding, and global conditioning to develop a unified approach, namely Patch-DM, to generate high-resolution images. Moreover, potential applications of outpainting and inpainting are also demonstrated. Comparisons on low-resolution images are conducted to reveal that the proposed method doesn't underperform by a large margin compared to other methods. Comparison of high-resolution images reveals that the proposed method achieves state-of-the-art results.", "questions": "1. Section 5.2: Since every image is segmented into smaller patches, the total number of model parameters is much smaller than other large diffusion models. => The computations might be lesser, why should the number of model parameters be lesser? Authors argue that they use light weight models, but it’s unclear what architectural changes they made for it as compared to existing diffusion models, and how those changes are justifiable. In fact, authors might be able to generate better quality images with heavier architectures and make Table 2 performance comparable to that of previous diffusion models. \n\n2. Beyond patch generation: The first one is to add patches inside the original images so that the generated images can have a 2× resolution compared to the ones in the training dataset. => Why do we need to add patches to original images? During test time the images are supposed to be generated from random noise and hence there is no need for adding patches to original images. This sentence is confusing and should be rewritten for clarity. Also the position embedding adaptations is not well explained, so is the reason for different choices with respect to the two methods mentioned under this category.\n\n3. Image inpainting: No details of the position embedding and the related changes is mentioned in this case. \n\n4. Image inpainting: The details could be incomplete. It is unclear how the original contents in the non-masked region is maintained in the output. Why is the global embedding not used?\n\nMinor Fix: \n- Section 5.2, still be ”recovered” during => still be ``recovered'' during", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The ideas of feature collage, position embedding, and global conditioning and their impact on image generation is interesting.\n2. The proposed method achieves state-of-the-art results on high-resolution image synthesis", "weaknesses": "1. On Low resolution image synthesis as in Table 2, the proposed method is not comparable to that of state-of-the-art. \n2. Some of the implementation details, reasoning behind choices, are missing in the description. Please refer to the comments under Questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mKTMC6nhvk", "reviewer": "Reviewer_oyqb", "summary": "This paper proposes a new feature collage strategy for the generative diffusion model to avoid boundary artifacts when synthesizing large-size images, termed Patch-DM. Feature collage systematically crops and combines partial features of the neighboring patches to predict the features of a shifted image patch, allowing the seamless generation of the entire image due to the overlap in the patch feature space. Experiments reveal the superiority of 1K resolution generation results on several datasets with 64×64 patches.", "questions": "Can the model be performed to generate images of more than 1K or arbitrary resolutions?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a new feature collage strategy for generative diffusion model to avoid boundary artifact when synthesizing large-size images.", "weaknesses": "1.\tThe novelty is relatively small and the impact of this paper may be limited since the proposed method only focuses on the boundary artifacts produced by the patch collage.\n2.\tLacking comparisons with aggregation sampling strategies proposed in StableSR [1], the sampling strategy in StableSR can be performed without more parameters and training. \n3.\tThe experiments only measure models with FID, lacking results measured under other metrics, e.g., CLIPScore.\n4.\tFigures 5 and 6 look confusing. (Which pictures are from which datasets or methods seem to be unclear)\n\n[1] Wang, J., Yue, Z., Zhou, S., Chan, K. C., & Loy, C. C. (2023). Exploiting Diffusion Prior for Real-World Image Super-Resolution. arXiv preprint arXiv:2305.07015.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)