Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach
Authors: Shaofeng Zhang
Jinfa Huang
Qiang Zhou
zhibin wang
Fan Wang
Jiebo Luo
Junchi Yan
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion models
image outpainting
Primary area: generative models
TL;DR: We propose a new method to outpaint images with arbitrary multiples in one step.

Abstract: Image outpainting aims to generate the content of an input sub-image beyond its original boundaries. It is an important task in content generation yet remains an open problem for generative models. This paper pushes the technical frontier of image outpainting in two directions that have not been resolved in literature: 1) outpainting with arbitrary and continuous multiples (without restriction), and 2) outpainting in a single step (even for large expansion multiples). Moreover, we develop a method that does not depend on a pre-trained backbone network, which is in contrast commonly required by the previous SOTA outpainting methods. The arbitrary multiple outpainting is achieved by utilizing randomly cropped views from the same image during training to capture arbitrary relative positional information. Specifically, by feeding one view and positional embeddings as queries, we can reconstruct another view. At inference, we generate images with arbitrary expansion multiples by inputting an anchor image and its corresponding positional embeddings. The one-step outpainting ability here is particularly noteworthy in contrast to previous methods that need to be performed for $N$ times to obtain a final multiple which is $N$ times of its basic and fixed multiple. We evaluate the proposed approach (called PQDiff as we adopt a diffusion-based generator as our embodiment, under our proposed \textbf{P}ositional \textbf{Q}uery scheme) on public benchmarks, demonstrating its superior performance over state-of-the-art approaches. Specifically, PQDiff achieves state-of-the-art FID scores on the Scenery (\textbf{21.512}), Building Facades (\textbf{25.310}), and WikiArts (\textbf{36.212}) datasets. Furthermore, under the 2.25x, 5x and 11.7x outpainting settings, PQDiff only takes \textbf{40.6\%}, \textbf{20.3\%} and \textbf{10.2\%} of the time of the benchmark state-of-the-art (SOTA) method.

Published as a conference paper at ICLR 2024
CONTINUOUS-MULTIPLE IMAGE OUTPAINTING IN ONE- STEP VIA POSITIONAL QUERY AND A DIFFUSION- BASED APPROACH
Shaofeng Zhang', Jinfa Huang", Qiang Zhou', Zhibin Wang", Fan Wang", Jicbo Luo", Junchi Yan *Department of CSE & MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University Univerriity of Rochester, "INF Tech Co., Ltd., *Alibaba Group
Junchi
(sherrylone, yanjunchi ) jes etu.edu.cn (jhuang90@ur, jluo@es) ji00@ess rochester. edu
Code: betps://githnn com/ Sherrylocc/ooiift http://hhh...ccc.cccoopooo/////p
ABSTRACT
Image outpainting aims to generate the content of an input sub-image beyond its original boundaries. It is an important task in content generation yet remains an open problem for generative models. This paper pushes the technical frontier of image outpainting in two directions that have not been resolved in literature: 1) outpainting with arbitrary and continuous multiples (without restriction). and outpainting in a single step (even for large expansion multiples). Moreover, we develop method that does not depend on pre-trained backbone network, which is in contrast commonly required by the previous SOTA outpainting methods. The arbitrary multiple outpainting is achieved by utilizing randomly cropped views from the same image during training capture arbitrary relative positional infor- mation. Specifically, by feeding one view and positional embeddings as queries, reconstruct another view. At inference, we generate images with arbitrary expansion multiples by inputting an anchor image and its corresponding positional embeddings. The one-step outpainting ability here is particularly noteworthy in contrast to previous methods that need to be performed for N times to obtain final multiple which N times of its basic and fixed multiple. We evaluate the proposed approach (called PQDiff as we adopt diffusion- based generator as our embodiment, under our proposed Positional Query scheme) on public bench- marks, demonstrating its superior performance over state-of-the-art approaches. Specifically, PQDiff achieves state-of-the-ar FID scores on the Scenery (21.512), Building Facades (25.310), and WikiArts (36.212) datasets. Furthermore, under the 2.25x, 5x and 11.7x outpainting settings, PQDiff only takes 40.6%. 20.3 % and 10.2% % of the time of the benchmark state-of-the art (SOTA) method.
INTRODUCTION
Image outpainting (Lin et al., 2021a; Cheng et al., 2022; Wang et al., 2021), a.k.a. image extrapola- tion (Wang et al., 2022; Kim et al., 2021; Zhang et al., 2020), is to generate content beyond the original boundaries of a given sub-image. It is technically an essential problem for generative models and remains relatively open (compared with other condition-batt
generation settings e.g. image inpainting (Bertalmio et al., 2000), style transfer (Luan et al., 2017) etc.), which meanwhile can find wide applications in automatic creative image, virtual reality. Usually, ideal outpainter is expected to achieve the following basic functions (Yao et al., 2022): 1) determining where the missing regions should be located relative to the output's spatial locations for both nearby and faraway features; 2) guaranteeing that the extrapolated image has a consistent structural layout with the given sub-image; and 3) the borders between extrapolated regions and original input images should be visually smooth.
Junchi Yan is the correspondence author. The SJTU authors are partly supported by NSFC (62222607) and Shanghai Municipal Science and Technology Major Project (021SH2DDZ00022 and SITU Trans-med Awards Research (STAR) 20210106.

---

Published as a conference paper at ICLR 2024
Model
PQDiff
One Step
Stops
Origin
Nx
Origin
1.0x
3.6x
9.7x
21.8x
99.0x
(a)
(b)
Figure 1: PQDiff can outpaint images with arbitrary and continuous multiples in one step (b). In contrast, previous methods (a) outpaint images with discrete multiples in multiple steps. Note that Nx here means an N-times larger image needs to be generated, while 2.25x, 5x, and 11.7x are adopted in the experiment following the setting of the previous work (Yao et al., 2022) for fair comparisons.
Table Methodology comparisons of the proposed PQDiff with recent advanced image outpainting methods. Pixel loss means pixel-wise 1 or l2 loss on the generated images. Feature loss means l2 loss on the feature map. No pretrain means the backbone randomly initialized.
Method
Generation type
Objectives
Encoder
Continurus
One step
No pretrain
NSIPO (Yang 2019)
GAN-based
Pisel+GAN loss
ResNet-50
x
x
[OII (Van Hoorick, 2019)
GAN based
Pisel+GAN loss
Conv layers
x
x
Uformer (Gao et al., 2023)
GAN-based
PixcH4AA++eaauu Loss
Swin Trans
x
x
QueryOTR (Gao et al., 2023)
GAN+MAE
PixelaGAN Loss
VIT
x
Vanilla Diff (Ho et al., 2020)
Diffusion-based
Pixel Loss
VIT
x
X
PQDiI
DifTusion based
Pixel Loss
VIT
These requirements have been mainly addressed by existing outpainting methods which in general fall into two categories: i) GAN-based methods (Van Hoorick, 2019; Yang al., 2019), whereby random noise and the initial input sub-images (as conditions) are used to generate the fake surrounding image content, and the discriminator is to classify the generated images fake real; ii) MAE- based methods (Yao et al., 2022) use MAE (Masked Autoencoder) (He et al., 2022) as the main architecture. They model the extrapolation as the MIM (Masked Image Modeling) problem (Xie et al., 2022) by replacing the extrapolated regions around the input sub-images with masked tokens and predicting the pixels of the masked patches. Specifically, these MAE-based methods also employ a discriminator to enhance the smoothness of the borders between extrapolated regions and the original sub-imaa sub-images. The above two kinds of methods mainly suffer from two applicability limitations. First, as shown in Fig. (a)), they require running multiple times to outpaints the image (e.g.. the SOTA method in (Yao et al., 2022) outpaints 11.7x images by passing through the model three times (x + forward + 2.25x + forward + 5x forward - 11.7x), which is inefficient especially when the required expansion multiple is large (e.g. 99x); Second, the discriminator based architecture can slow down the convergence speed (Goodfellow et al., 2014; Adler & Lunz, 2018), and require pre-trained encoder. In other words, the computational cost of training their model in fact includes the pretraining cost, which is usually very high (e.g. up 1,000 or longer epochs ImageNet).
In this paper, we emphasize and tackle two less-studied challenges, especially the flexibility, and efficiency for practical tool: the model is expected to i) outpaint images in arbitrary and continuous multiples (i.e. by N E IR times), where Nx multiples mean the contents of the extrapolated images N >> 1 times larger than the input sub-images, especially without resorting retrain model for every different N; and ii) outpaint with any multiple N in one step'.
Recently diffusion models (Dhariwal & Nichol, 2021; Pokle al., 2022; Jin al., 2023b;e; Augustin et al., 2022) have shown success for multi-modal (Jin et al., 2023a; 2022), segmentation (Tan et al., 2022; Wu et al., 2021), backbone designing (Wu al., 2022: Dai et al., 2022), and image generation (Saharia et al., 2022) with a progressive denoising procedure. Yet such an iterative step-by-step (also called timesteps) the sampling (i.e. testing) stage can be too tedious for outpainting, especially considering image outpainting itself also still requires its own iterations
Note that the step here refers to how many iterations are needed to be passed through the model instead of timesteps in the diffusion models.
2

---

Published as a conference paper at ICLR 2024
(e.g. QueryOTR (Yao et al., 2022), IOH (Van Hoorick, 2019)). To achieve one-step tiffusion-basee generation, we propose to use relative positional queries and input sub-images as conditions. Since the relative positional embedding can represent any positional relationship between the input sub-image and the extrapolated image, we can outpaint the sub-image in controllable and continuous multiples in one step (Fig. ((b).. We make methodology comparisons in Table to better position our method. The main contributions include:
Continuous multiples for image outpainting. We propose PQDiff, which learns the positional relationships and pixel information at the same time. Specifically, in the training stage, PQDiff first randomly crops the given images twice to generate two views. Then, PQDiff learns one cropped view from the other cropped view and the pre-calculated relative positional embeddings (RPE) of the two views. Since the RPE can represent continuous relationships between two views, PQDiff can outpaint the images in continuous multiples. To our best knowledge, we are the first to outpaint images in continuous multiples (e-g.. 1x, 2.25x, 3.6x, 21.8x), whereas the SOTA QueryOTR (Yao al., 2022) can only outpaint images in discrete multiples.
ii) One-step image outpainting. We propose a position-aware cross-attention mechanism between relative positional embedding and input sub-image patches, which helps PQDiff to outpaint images in only one step for any multiple settings. As far as we know, PQDiff is the first to achieve this capability, whereas (Yao et al., 2022; Yang al., 2019) can only outpaint images step-by kepp-by-step, which severely limits their sampling, i.e. generation efficiency. Under the 2.25x, 5x and 11.7x outpainting settings, PQDiff only takes 40.6 % 20.3%, and 10.2% % of the time of QueryOTR (Yao al., 2022).
iii) New SOTA performance. Experimental results on outpainting benchmarks (Gao et al., 2023; Yang et al., 2019) show that PQDiff significantly surpasses QueryOTR (Yao et al., 2022) and achieves new SOTA 21.512, 25.310 and 36.212 FID scores with the challenging 11.7x multiple setting on the Scenery, Building Facades, and WikiArts datasets, respectively. Moverover, PQDiff achieves new SOTA results in most settings (2.25x, 5x, and 11.7x).
2 BACKGROUND AND RELATED WORK
Image Outpainting. It aims to generate the surrounding regions from the visual content, which can be considered as an image-conditioned generation task (Odena et al., 2017: Kang et al., 2021: Guo et al., 2020; Arjovsky et al., 2017; Gulrajani et al., 2017). The work (Sabini & Rusak, 2018) brings the image outpainting task to attention with a deep neural network inspired by image inpainting (Bertalmio et al., 2000). It focuses on enhancing the quality of generated images smoothly by using GANs and post-processing to perform horizontal outpainting. The work (Van Hoorick, 2019) designs a CNN-based -nooder-to-decoder framework by using GAN for image outpainting. In (Wang et al., 2019), a Semantic Regeneration Network is proposed to directly learn the semantic features from the conditional sub- sub-image. While 3-stage model is developed in (Lin et al., 2021b) with edge-guided generative network to produce semantically consistent output. Although these methods avoid bias in the general padding and up-sampling pattern, they still suffer from blunt structures and abrupt color issues, which tend to ignore spatial and semantic consistency. To tackle these issues, a Recurrent Content Transfer (RCT) block is devised (Yang et al., 2019) for temporal content prediction with Long Short Term Memory (LSTM) networks (Hochreiter & Schmidhuber, 1997). To enrich the context, (Lu et al., 2021) additionally switches the outer area of images into its inner area.
The SOTA Outpainting Method QueryOTR (Yao et al., 2022). We particularly discuss Query- OTR for its so-far best performance as well as its adoption of ViT module as also will be used in our approach. QueryOTR (Yao et al., 2022) proposes to adopt the ViT-based encoder and MIM- based (Zhang et al., 2023c) architecture for outpainting. Specifically, given an input sub-image x E T #x x3 QueryOTR first partitions x into regular non-overlapping patches with the patch size HxW where P is usually set as 16 and L = PS The goal of QueryOTR is to predict the extra sequence
P X P to obtain the patch tokens x2. xi................ ,
(-++1x(+2 ..... xL+R) representing the extrapolated regions. In line with MAE (He et al., 2022), the sin-cos positional embedding of input patches is pre-defined. Thus, the positional embedding of the extrapolated patches can be obtained. The training of QueryOTR also follows MAE (He et al., 2022), where the visible patches sequence {xx, xz. ... x is fed to the encoder. Then, the fixed positional embedding of the masked tokens sequences (xp 1 L+2
x .... xL+R) is used as input to the decoder predict the extrapolated
3

---

Published as a conference paper at ICLR 2024
patches. Finally, QueryOTR copies the input sub-image to the generated image in the corresponding position, followed by the Patch Smoothing Module (PSM) to smooth the border.
Diffusion Models. These models (c.g. the seminal work (Ho et al., 2020)) gradually inject noise into data and then reverse this process to generate data from noise. The noise-injection process is also called the forward process. Given the original data X (clean), then, the forward process can be formalized as Markov chain: ((x1:T (xo) = I ((xx(x;-1) where q the forward process and (x- = (xxalvaixi- /a: 8,I), and a and / represent the noise schedule and a + 3 = 1. [N10 1) means the standard Gaussion noise. To reverse this process, a Gaussion model p(x(-1(xx)) transition 9x.-1 Specifically, the optimal mean value of x, can be written as (Bao et al., 2022):
( ( is adopted to approximate the ground truth reverse
E((x) where (. = II a,, and is the standard Gaussian noises injected X,. Thus, the learning is equivalent to noise prediction task. Formally, noise prediction network (x,, t) is used to learn [[<]xe] by minimizing the noise prediction objective. For l2 loss, we can formulate the objective of noise prediction task as ming Ex,x,.ellc c co (x,, t)ll2. where is uniform between 1 and T. On the basis of the plain diffusion models, LDM (Rombach et al., 2022) proposes to add noise and denoise in the latent space, which greatly improves the training efficiency, Followed by LDM, ViTDiff (Bao et al., 2023) proposes to replace CNN-base U-net (Ronneberger et al., 2015) with ViTs (Dosovitskiy et al., 2021) to estimate the backward process in diffusion models.
1 X4 (x)) = vas
3. -I-a
THE PROPOSED POSITIONAL-QUERY BASED DIFFUSION MODEL
We provide a concrete embodiment based on the diffusion model and we term approach as PQDiff, whereby the continuous multiples and one-step generation are achieved. In fact, our PQ framework with these two advantages performance comparison given in our ablation studies. We also show the significant improvements of our approach compared with a vanilla diffusion model directly for outpainting.
also incorporate other generative models e.g. GANs, and the empirical
Approach Overview. Our approach mainly consists of key modules: relative positional embedding, diffusion process, cross attention in the position-aware transformer model, and sampling pipeline.
Relative Positional Embedding. Given the image x E H Xx3 from the training set, we first randomly crop the image twice and resize the cropped image to generate two views x, E A X x3 and X E R22xxwxx x3xx3 where X., and X₁, are denoted as the anchor and target view which are not necessarily the same size. We also denote (w, h) as the predefined resolution to be generated. Then, we resize both the anchor view and target view to (w, h). As illustrated in Fig. 2, we can first obtain the prior information of positional relationship (m, n) between the anchor view X. and target view X6. Then, we design the relative positional embedding to represent the position relation:
m (201/d)
m
m
12
Em,.
sin
COS
(222//
sin
sin (2211/d)
COS
2222/d
sin
where e = 10, 000 is the pre-defined parameter, as also commonly used in (He et al., 2022; Zhang et al., 2023a;b). (m, n)) means the position top-left patch position (see Fig. 2). Note that we randomly crop two views, and as result, the positional relationship of the two views might be either containing, overlapping, or non-overlapping. and our model can jointly learn these three relationships.
Forward and Backward of Conditioned Diffuusion Given the anchor view X. E A target view X₁, E xx3 and the relative positional embedding E E L (L and D are the patches' length and the predefined dimension), we first encode the two views by VQVAE (Van Den Oord et al., 2017) (frozen) compress the images to latent space, resulting in 2, E I xw' and z₁, RR'' x xc (usually, 1/ < ht w' < w). The compression aims to improve the training efficiency and convergence speed of the diffusion models (Bao al., 2023; Rombach et al., 2022). After obtaining the two latent views, we patchify the two views and obtain the anchor sequence {zz, z² ... z and target sequence {z], zi..-. ... ) (typically, we set L Ax Then, the forward process of diffusion on the target sequence can be formulated as:
qZz,, [2,, ,) N( .Iv anzs, 19 3.1,) and q(26, (2h))) ) - (20, VTe, (1 I.),,
(2)
where 2hn is the original target sequence {z), 2 ... 2 and a, = Ii'_ a, For the backward process, suppose we have neural network go (will be described later), taking the noisy target 26,-
4

---

Published as a conference paper at ICLR 2024
Training
X.
Arbitrary
VQVAE
Presl
Sampling
Encoder
Position-
Randon
Relative Positional
Farboldnn
F
Transformer Model
Aware
Urgniff
B VQQAR Encoder
Zoz
Mode
Ingut image
Targer
Caussian moise
Center Sampling
I iterations
Mode
VQVAE
Zyp
YT
Aware Transformer
VOVAE Decoder
Encoder
giy
Pros
Outpermed image
Relative Pasitional
Embecliigg
Frozon
Tunod
Target
Anchor
Center Mode
Mode
Figure 2: Framework of PQDiff. RPE in Eq. means relative positional embeddings (we give the pseudo-code to calculate the RPE in Appendix A). For training, we randomly crop the image twice with different random crop ratios to obtain two views. Then, compute the relative positional embeddings of the anchor view (red box) and the target view (blue box). For sampling, i.e. testing or generation, first compute the target view (blue box) based the anchor view (red box) to form mode that means a positional relation. With different types of modes, we can perform arbitrary and controllable image outpainting. Then, feed the RPE, random Gaussian noise, and input sub-image to perform outpainting. In theory, our PQDiff can outpaint (predict) the region at any location, due to the randomness of cropping in the training stage. We illustrate how calculate the relative position in Appendix C. Mode means the positional relations between the anchor view and the target view.
clean anchor sequence Zu and relative positional embeddings E as input. Then, the network aims to predict the added noise on the target sequence Xb,, Then, the objective of PQDiff can be written as: LrQDIfS || <ellp and E, go (Xa: X6, E)
We set = in line with the previous generative methods (Rombach et al., 2022; Bao et al., 2023).
The Position-Aware Transformer Model ge- Here, deseribe the architecture of the neural network used in the diffusion model in detail. Consider we have the noisy target Zi., clean anchor sequence 2a, relative positional embeddings E and the timestep I, we first concatenate the noisy target and the anchor sequence at the channel dimension, followed by linear layer to map to original dimension to reduce the computational cost, and denote the mapped embedding as Zg E L D, where D is the predefined hidden dimension in transformer network. Then, we feed the Zg into the transformer encoder, which is composed of several transformer blocks (Vaswani al., 2017). After the transformer encoder, the position- position-awarr aware cross- aotssaattnttion mechanism is proposed to learn positional relationship, which can formulated as:
QEKI
T ag
Attn (QE, K2,, Vz,)) Softmax
V ,,,
(4)
VD
where Q EW K2 my WR, Vz, W. and Wg. We, W, are learnable parameters. After capturing the information of the target position we directly feed the z into the transformer decoder composed of several transformer blocks. followed by convolutional layer to predict noise.
Sampling Pipeline. After training the network well, we can outpaint the image in any controlled multiples, since the designed relative positional encoding can represent any positional relationship between two images. In the sampling stage, we can simply take the input sub- wub-image as the anchor view, and input any position we want. Then, we calculate the positional encoding of the given position and feed the RPE to the network. Then, the network can predict the noise mentioned in Eq. 3. Finally, through Eq. 2, we can simply compute the fake Zoy and predict Zo.., step-by-step by:
g(Zb,-1 1Zb, Aboo -N(z6, 37o %00 1.1.)).
Vor-1.3 Zbo
vo(1 at-1)
Zb, and Br 1 04-1
(5)
(Zb, (ooo)
3r-
a
1 α2
5

---

Published as a conference paper at ICLR 2024
QueyOTR
igDdft (Ous)
GT
QueyOTR
PQD (Chns)
Inconsistent Noise
Speckle Noise
Beneficial Phenomenon
Figure 3: Comparison on the 2.25x, 5x, and 11.7x settings with the SOTA method QueryOTR. The images generated by QueryOTR come from the pre-trained model in their official repository. We highlight two kinds of noises from QueryOTR. The red box indicates that the boundary of the input sub-image is inconsistent with the generated region, and yellow box contains noise and spots. We also find some interesting phenomena (highlighted in the green ovals of generated images in the right figure), where PQDiff can notice the generated "clouds", and reflect the "clouds" in "water". Moreover, the shape of the clouds in the sky and the reflections in the water are also consistent. In contrast, the previous method only generates "clouds", but ignores the reflection the water.
After several iterations, when 0, can obtain the extrapolated images. The training and sampling algorithms are given in Alg. 1 and Alg. 2 in Appendix A, respectively.
Discussion. As DDPM (Ho et al., 2020) requires step-by-step sampling. DDIM (Song et al., 2021a) is proposed to use ODE (Song et al., 2021b) equation for faster sampling. Our PQDiff can also use the DDIM for faster sampling. Specifically, through the Euler method and probability flow ODE
proposed in (Song et al., 2021b), we can obtain Zb,-ae att by:
Zb. az
1
Ot-Ax 1- Ott
Ot
(6)
Qt-At
az
att
Then, the one-timestep sampling each iteration and replaced with A, timesteps in each iteration.
4 EXPERIMENTS
4.1 EXPERIMENTAL RESULTS
Quantitative Results. Table shows that PQDiff with copy operation outperforms in all metrics on 2.25x, 5x, and 11.7x experiments. In particular, with a larger outpainting multiple, PQDiff can obtain better FID and scores. For 11.7x outpainting, PQDiff surpasses the previous SOTA QueryOTR (Yao et al., 2022) 16.460, 25.310, and 36.216 FID scores on Scenery, Building and WikiArt datasets, respectively. We also find an interesting phenomenon that, WikiArt, PQDiff can surpass PQDiff + Copy on 5x and 11 1.7x experiments. In other words, the generated images without copy could be more realistic than the ones with the centroid copy operation. This perhaps because the boundary region of the input sub-image is slightly inconsistent with the generated image.
Qualitative Results. Examples of visual results are shown in Fig. (the "copy" operation is added on all methods for better comparisons). PQDiff effectively outpaints the images by querying the global semantic-similar image patches. As seen from the 2.25x outpainting results, PQDiff could generate more realistic images with vivid details and enrich the contents of the generated regions. In addition, although QueryOTR (Yao et al., 2022) adds the smoothing module to handle the inconsistency of the boundary region, there still few noisy spots highlighted in red boxes. For 5x and 1 117xx results, we can clearly see the images generated from the QueryOTR are much vaguer than those generated by PQDiff. Moreover, PQDiff can handle details well. For example, the "clouds' reflection"

---

Published as a conference paper at ICLR 2024
Mode 1
Mode
Mode
Mode
Ori
Gen
Figure 4: Example images generated by PQDiff via random relative positional embedding. The original image (Ori) input is mapped the corresponding location in the generated image (Gen). Note that the generated images do not explicitly undergo any "copy" operation, but still reserve the input pixels. Moreover, PQDiff learns the scales input images according to the given mode setting.
Table Results different multiples. 2.25x. 5x, and 11.7x mean input sizes are 128x 128, 86x86, and 56x56, respectively. After feeding to the network go, all input sub-images are first resized to 192x192. Copy" operation means after the network yields the extrapolated images, the model copies the input sub-images to the corresponding center region in the extrapolated images. The best results and second best results are indicated by boldface and underline, respectively.
Multipee Methods
Type
Encoler
FID.
Secnery
18
Building Facades FID |
15
FID
WikiArt
SRN (Wing et 2019)
GAK-hused
VGG (Simonyan & Zivermton, 2014)
47.781
2.981
38 38.544
3.862
76:749
3,629
SSIPO (Yang et 20199)
GAN based
ResNet (Ile et al.. 2016)
25,917
3.059
30.465
4,153
22.242
5.600
108 (Van Hoorick, 2019)
GAN-Jusal
Con loyas
32.167
2.885
40.481
3,924
40.184
4.835
2.25x
Uloraerr (Ciao et al. 2023)
GAN-hasal
Swin Transfommer (Liu at 20113
20.575
3.249
30.542
4.189
15:904
6.507
QurryOTR (Yao al. 2022)
Copy
GAN-MAE
ViT (Desoviesity al. 2021)
20.340 366
3,955
22. 373
4.978
14.955
7,896
PQDiif (Chm)
Diffusion-hraol
VT (Deunviskkt at a 2021)
29:446
3,809
78 855
4.879
10:454
7.374
PQDuff Copy
Diffusion based
VIT (Desovissily
20,100
3,961
5.350
7.968
8.605
SRN (Wang
GAN based
VGG (Simonyan Zissermun, 2014)
83.772
2.349
NSIPO
GAN-lusal
RooNet (H: 2016)
45:980
58.
101l 2019)
bused
layers
20033)
(Lio 2001)
QueryOOR (Yan al., 2022) Copy
GAN-MAB
VII (Desovisally al.. 2021)
39.237
41.273
PqDat (Ours)
Diftuzion based
VIT (Desovissily 2021)
34,492
3,527
15,297
6,971
PQDdIT+ Capy (Duss
Dillision-bssal
ViT (Dosovitskly al. 2021)
28.668
3,712
29.396
4.763
15.772
7,876
SRN (Wang 2019)
GAN-Jusal
VGG (Simceyyn Ziveenton, 2014)
2.087
2.988
NSIP (Ying et 2019)
GAN bused
ResNet (Ile al..
64.457
2.405
$1.301
3.431
75.785
4.225
(Yan Hoozisk, 2019)
GAN-bused
Con layass
$6.629
2.432
95.039
2.90
108.328
3.728
11.7x
Utorzer (Gao et al., 2023)
GAN based
Swin Trunsfoemer (Liu et al. 20213
60.497
2.638
93.89M
3.388
72.923
5.904
QueryOTR (Yao al., 2002) + Copy
GAN-MAR
VIT (Dosuvitskly et 2021)
60.977
1.144
64 63.976
4.6
69.951
688
RgDii (Chuns
Dillision-bisal
VIT (Desovitakl al... 2021)
44.517
3,260
43.971
4.620
33.735
6.957
PQDaff Copy (Oum)
Diffusion based
VIT (Desuvinskly at.... 2021)
39,465
3,574
39.616
4.754
36.326
7.724
Table 3: Comparison of sampling time, FID, and inception scores with 2.25x, 5x, 11.7x settings on Scenery dataset. The sampling time is the wall-clock time generating 64 images on a single GPU.
2.25x
11.7x
Method
Time (Sec.) 4
FID
IS
Time (Sec.)
FID
18 1
Time (Sec.)
FID L
QueryOTR (Yao al., 2022) PQDiIl (20 timesteps)
23.728
20.366
3.955
47.456
39.237
3.431
71.184
60.977
3.114
9.638
20.593
4.026
9.638
28.138
3,707
9.638
42.348
3.737
in the "water" is consistent with the generated "clouds" in the sky, as highlighted in the green ovals. We also provide more qualitative comparisons and generated images in Appendix E.
Sampling (i.e. Generation) Speed. We also compare the sampling speed of PQDiff with different timesteps. Specifically, we first train PQDiff with 80,000 iterations. Then, for the sampling stage, we evaluate the pre-trained PQDiff with different timesteps. Table 3 reports the wall-clock time spent on generating 64 images on 8 V100 GPUs. Since PQDiff can outpaint images with any multiples in one step, the cases for 2.25x, 5x, and 11.7x spend almost the same time. In contrast, previous methods will take much more time under the 5x and 11.7x settings than in 2.25x. It is worth noting that, under the 2.25x setting, the inception score of PQDiff with 200 timesteps (4.111) 1) is even higher than the ground truth (4.091) (refer Appendix E).
4.2 ABLATION STUDIES
Outpainting in an arbitrary position. Previous outpainting methods mainly plot the same multiples around the top, down, left, and right regions, and they simply find where the input sub-ing sub-image should locate in the generated image. Thus, the "copy" operation can be simply finished. However, for outpainting in random positions, it is difficult to find the corresponding locations, since the
7

---

Published as a conference paper at ICLR 2024
Table 4: Comparisons of the center PSNR score over 2.25x, 5x, and 11.7x. Note that we find PQDiff calculates several infinite values, which indicates the center parts in images generated by PQDiff are completely the same with the imput sub-images without any bias. Hence, to make our model comparable, we only include the images whose PSNR scores are below 1,000. For QueryOTR, we do not find the infinite value phenomenon, and we directly report the averaged PSNR scores.
Scenery 5x
Building 5x
WikiArt 5x
Mothod
2.25x
11.7x
2.25x
11.7x
2.25x
11.7x
QueryOTR (Yao et al., 2022)
22.146
18.926
15.375
18.591
15.318
13.119
19.726
15.874
14.016
PQDiff (Ours)
27.676
27.267
24.697
28.831
28.614
27.219
25.946
25.673
23.372
Two images (generated and original) could have different scales. Moreover, the two images may also not intersect. Hence, we directly illustrate the images generated by PQDiff without the "copy" operation. Some examples generated from the controlled position are shown in Fig. 4, where the images generated by PQDiff without the "copy" iteration can be still vivid and realistic. Furthermore, without the "copy" operation, the generated images can also record the pixel information and put it into the corresponding locations. Meanwhile, since the scales of input sub-images and generated images may be different, PQDiff implicitly learns to scale the input sub-images as well.
Diversity of generated images and PSNR in center regions. We also show five generated images with fixed positions and input in Fig. 11 in Appendix E, showing PQDiff can generate diverse content in the generated regions. Furthermore, it also retains the input pixels in the center parts of generated samples. Recall that in Sec. B, we choose not to use the PSNR score as evaluation metries, as we also need to account for the diversity. Here we provide further analysis with PSNR score whose definition is as follows, and a higher score suggests a smaller mean square error:
MAX - - yr,) ) O OSLJSH.
PSNR(x,)) y) = =10 logio =10-1og10 [x,
where x, y are input images, and H, W are the height and width of the input images, and M MAX; is constant. We then report the PSNR score of the generated center region and input images in Table 4. Note that since the objective of QueryOTR is only added to the generated position, the center part is not regularized in the training stage. Hence, the generated center content completely black area. Hence, for better comparison, we modify QueryOTR, adding the objective to the whole generated images with the Lree loss. The PSNR scores in Table and the generated samples in Fig. 11 demonstrate our model can: 1) memorize the input pixels and put them into corresponding locations; 2) learn the semantic information of input images, and outpaint the image with consistent contents around the images. It is worth noting that the central regions in images generated by our PQDiff without the "copy" operation can be completely the same with input images (E,,, [x -)yy.,l² = 0), which indicates, our PQDiff can reconstruct the images without any bias in some cases. Since these cases without bias would lead to infinite values of the PSNR score, which will impact the average values, we directly remove these infinite values. Experimental results show even if we remove these infinite values, our method can still produce much higher center PSNR scores than QueryOTR.
x
Impact of random crop ratio. For better consistency of inputs in the training stage and sampling stage, we crop the view x2 with a larger crop ratio than view X, (since the outpainted images are usually larger than input images). We conduct experiments ratios. Specifically, we fix the crop ratio of target view X as (0.8, 1.0). and switch the crop ratio of anchor views x, from 0-.500.. We train the model with 80,000 iterations on the Scenery dataset and show the results in Fig. 5. The results show the query crop ratio influences 2.25x, 5x, and 1 11.7x experiments in different manners. Specifically, for 2.25x experiments. since the input images are 128x128, and the outpainted images are 192x192, where the extrapolated image is only a little larger than the input sub-image. Hence, PQDiff with larger crop ratios outperforms PQDiff with smaller ones. For 5x and 1 11.7x experiments, the size of extrapolated images is much larger than the given input sub-image. As a result, PQDiff with smaller crop ratios outperforms PQDiff with larger ones. On top of that, we also find when the anchor crop ratio equals 0.50, the inception score drops with a large range, and we guess that is because when the random crop ratio is set (0.50, 0.50), we always feed the images with the same scales to PQDiff, and correspondinlly. PQDiff can not learn to scale images. Then, in the sampling stage. since images in the test set are usually in different scales, is difficult for PQDiff to handle the scaling gaps between the training stage and the testing stage.
analyze the effect of random crop
8

---

Published as a conference paper at ICLR 2024
Table Performance plain diffusion models and PQGAN (integrate PQ scheme in GAN) on the Scenery dataset without the "copy" operation (as we report the center PSNR score).
2.25x
Method
FID L
IS
Center PSNR
FID J.
18
Center PSNR
FID 4
18
Center PSNR
Valiona diffusion (Ho 2020)
57.425
3.816
11.538
59.475
3.286
10.894
73.587
10.347
POGAN (Ouns)
$2.139
3.811
27,097
38.466
3.216
26.893
49.943
2.853
24.139
PQDiit (Ours)
29,446
3,849
27,676
34,492
3.547
27.267
44.517
3.269
24.797
Table 6: Impact of different types of positional embeddings on Scenery dataset. All methods are
without "copy" operation. The sin-cos positional embedding is also used in the main experiments.
2.25x
5x
11.7x
Method
FID
f
Center PSNR t
FID
15
Center PSNR
FID L
Center PSNR
None
61.382
3.811
10.382
75.281
3,298
10.285
90.271
2,974
10.286
Learnable
32.141
3.816
27.263
37,971
3.387
26.885
49.136
3.179
23.461
Sin-Cos (Eq.1)
29.446
3,849
27.676
34.492
3.547
27.267
44.517
3.269
24.797
1.911
3.5
3.900
100epib0 200e
3.00
Incopiion ScCITe
Inceptonnt CCOTE
0.99
5,75
5,80
PQUesT
PQUDe
QueryOTR
Qpery13R PQDxet (defouh)
QueryOTR (ackii)
132
A25
0:00
645
025
1.20
0.5
0.45
6.2
425
009
Query crop ratio
Quary crop mallo
Query crop maio
Figure 5: Inception scores on the Scenery dataset with different random crop ratios anchor view X. over 2.25x (left), 5x (middle), and 11.7x (right). In some cases, PQDiff outperforms PQDiff (Default), as we do not heavily tune the hyper-parameters in the default version. The query crop ratio randomly sampled from 0.5, where is the values in the horizontal axis of the plots.
Integrate PQ scheme into other generating models. Beyond diffusion, we also integrate our PQ learning paradigm into GAN-based model. In addition, analyze the effect of the positional query, we also report the performance when directly using diffusion models (Ho et al., 2020). Table shows the results. Specifically, we find the quality of the images generated by GAN lower than the diffusion model (Ho et al., 2020), as the inception score of PQGAN is lower than diffusion models (Ho al., 2020). However, the diffusion model is not well conditioned by the input sub-image (the FID) and Center PSNR scores are much worse than PQGAN and PQDiff). The high FID and Center PSNR scores indicate the proposed PQ scheme can provide a strong condition, enhancing the generative models (Goodfellow et al., 2014; Ho et al., 2020) to learn where to outpaint.
Impact of the Positional Embedding. To analyze the effect the positional embedding, we conduct group of experiments with different types of positional embeddings. We mainly consider two types of embeddings (sin-cos and learnable). Learnable embedding means we take the relative position MI.P composed of two linear layers with the activation function map the 2-dimensions to D-dimensions. Table shows the results with different positional embedding. Note that None means without relative positional embedding. Thus, the model can not learn the positional relationships between input sub-image and extrapolated images. Correspondingly, the inception score only drops with a little range, but the FID and Center PSNR drop with large range.
(m, n) as input and
5 CONCLUSION
We have proposed PQDiff, which learns the positional relationships and pixel information at the same time. Methodically, PQDiff can outpaint at any multiple in only step, greatly increasing the applicability of image outpainting. We conduct experiments on three standard outpainting datasets, where PQDiff achieves almost all settings. We also conduct comprehensive ablation studies to show the robustness of our approach, including crop ratios, the center PSNR score, and the relative positional embeddings.
SOTA results that surpass previous methods by large margin under
Ethics Statement. PQDiff generates images conditioned by positional embeddings, learning pixel information and positional relationships simultaneously. As the datasets used in PQDiff primarily

---

Published as a conference paper at ICLR 2024
focus scenery, buildings, and arts, there are currently minimal negative potential impacts ethics and crime-related aspects. We are aware that any technology could be abused for purposes.
Reproducibility Statement. We have clarified training and sampling details including hyper- parameters, pseudo code of the relative positional embeddings, and training pipeline in Sec. B in the Appendix. In addition, all the datasets used in this paper open-source and can be accessed online.
REFERENCES
Jonas Adler and Sebastian Lunz. Banach wasserstein gan. NeurIPS, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICMI,, 2017.
Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein. Diffusion visual counterfactual explanations. NeurlPS, 2022.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-pmm an analytic estimate of the optimal reverse variance diffusion probabilistic models. ICLR, 2022.
Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu, All are worth words: a vit backbone for score-based diffusion models. In CVPR, 2023.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In SIGGRAPH, 2000.
Yen-Chi Cheng, Chich Hubert Lin, Hsin- Ying Lee, Jian Ren, Sergey Tulyakov, and Ming-Hsuan Hsuan Yang. Inout: Diverse image outpainting via gan inversion. In CVPR, 2022.
Jifeng Dai, Min Shi, Weiyun Wang, Sitong Wu, Linjie Xing, Wenhai Wang, Xizhou Zhu, Lewei Lu, Jie Zhou, Xiaogang Wang, al. Demystify transformers & convolutions modern image deep networks. arXiv preprint arXbv:2211.05781, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthia Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Penglei Gao, Xi Yang, Rui Zhang, John Y Goulermas, Yujie Geng, Yuyao Yan, and Kaizhu Huang. Generalized image outpainting with u-transformer. Neurul Networks, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. NeurIPS, 2017.
Dongsheng Guo, Hongzhi Liu, Haoru Zhao, Yunhao Cheng, Qingwei Song, Zhaorui Gu, Haiyong Zheng, and Bing Zheng. Spiral generative network for image extrapolation. In ECCV, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
10

---

Published as a conference paper at ICLR 2024
Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. iopectation-maxxiiizee contrastive learning for compact aadeo-and-langueee representations. NeurlPS, 2022.
Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In CVPR, 2023a.
Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen. Diffu- sionret: Generative text-video retrieval with diffusion model. arXiv preprint arXiv:2303.0980 2023b.
Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Yang Wei, and Li Yuan. Act you wish: Fine- grained control of motion diffusion model with hierarchical semantic graphs. arXiv preprint arXiv:2311.01015, 2023c.
Minguk Kang, Woohyeon Shim, Minsu Cho, and Jaesik Park. Rebooting aegan: Auxiliary classifier gans with stable training. NewrIPS, 2021.
Kyunghun Kim, Yeohun Yun, Keon-Woy Woo Kang, Kyeongbo Kong, Siyeong Lee, and Suk-Ju Kang. Painting outside as inside: Edge guided image outpainting via bidirectional rearrangement with progressive step learning. In WACV, 2021.
Chieh Hubert Lin, Hsin-Ying Lee, Yen-Chi Cheng, Sergey Tulyakov, and Ming-Hsuan Yang. Infini- tygan: Towards infinite- fiinite-pixel image synthesis. In ICLR, 2021a.
Han Lin, Maurice Pagnucco, and Yang Song. Edge guided progressively generative image outpainting. In CVPR, 2021b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Chia-Ni Lu, Ya-Chu Chang, and Wei-Chen Chiu. Bridging the visual gap: Wide-range image blending. In CVPR, 2021.
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In CVPR, 2017.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In ICML, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. NeurIPS, 2019.
Ashwini Pokle, Zhengyang Geng, and Zico Kolter. Deep equilibrium approaches to diffusion models. NeurlPS, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Mark Sabini and Gili Rusak. Painting outside the box: Image outpainting with gans. arXiv preprint 1808 08483, 2018
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NewrIPS, 2016.

---

Published as a conference paper at ICLR 2024
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. ccore-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Haoru Tan, Sitong Wu, and Jimin Pi. Semantic diffusion network for semantic segmentation. NeurIPS. 2022.
Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi. Data pruning via moving-one-ssmple-ouu NeurIPS, 2024.
Wei Ren Tan, Chee Seng Chan, Hernán E Aguirre, and Kiyoshi Tanaka. Ceci n n'est pas une pipe: deep convolutional network for fine-art paintings classification. In ICIP, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 2017.
Basile Van Hoorick. Image outpainting and harmonization using generative adversarial networks. arXiv preprint a9Xi: 10960, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Sindiffusion: Learning a diffusion model from a single natural image. arXiv preprint arXiv:2211.124455 2022.
Yaxiong Wang, Yunchao Wei, Xueming Qian, Li Zhu, and Yi Yang. Sketch seiech-guided scenery image outpainting. IEEE Transactions on Image Processing, 2021.
Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Wide-context semantic image extrapolation. In CVPR, 2019.
Sitong Wu, Tianyi Wu, Fangjian Lin, Shengwei Tian, and Guodong Guo. Fully transformer networks for semantic image segmentation. arXiv preprint 206.v 04108, 2021.
Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: general vision transformer backbone with pale-shaped attention. In AAAI, 2022.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. ICCV, 2019.
Kai Yao, Penglei Gao, Xi Yang, Jie Sun, Rui Zhang, and Kaizhu Huang. Outpainting by queries. In ECCV, 2022.
Lingzhi Zhang, Jiancong Wang, and Jianbo Shi. Multimodal image outpainting with regularized normalized diversification. In WACV, 2020.
Shaofeng Zhang, Qiang Zhou, Zhibin Wang, Fan Wang, and Junchi Yan. Patch-level contrastive learning via positional query for visual pre-training. In ICML, 2023a.
Shaofeng Zhang, Feng Zhu, Rui Zhao, and Junchi Yan. Patch-level contrasting without patch correspondence for accurate and dense contrastive representation learning. In ICLR, 2023b.
Shaofeng Zhang, Feng Zhu, Rui Zhao, and Junchi Yan. Contextual image masking modeling via synergized contrasting without view augmentation for faster and better visual pretraining. In ICLR, 2023c.
12

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
Pseudo Code of the Relative Positional Embeddings
14
Training Details
15
Illustration of the RPE During the Training and Sampling
16
D More results in continuous multiples
17
Impact of the Continuous and Diserete Positional Embeddings
17
F Sampling Speed
18
G Impact of Predicting To or Noise.
18
Incorporate with Pretrained Models
18
I More Generated Examples Facade datasets
19
J Absolute Position Embedding v.s. Relative Position Embedding
19
More Cases Generated by PQDiff
More discussions
22
13

---

Published as a conference paper at ICLR 2024
A PSEUDO CODE OF THE RELATIVE POSITIONAL EMBEDDINGS
Pytorch-liked code of the relative positional embeddings.
def get_views_and_ ppilimagee, scales, ratios, size)
:param =il_in images with PIL format sparam scales pil_image: random crop scale :param ratios pil_image: random crop ratios
:param size: size of cropped views
Calculate the position of the anchor view ia,, i,a, a w_a RandomResizedCrop get params pril_image, scales,
ratios)
anchor_view resized_crop (pi immage La, i_a, ha, w_a, (size, sizc))
Calculate the position of the target view
itt, itt, h_t, w_t MandomResizedCran perpaii....r.......apaaa.. (pi pppllimaae Lmmge, scales, ratios)
target_viee virew_viewwt resized_croo (pil image, i,, it,, nt, w_t, (size, size))
Derive the relative positions of two views
grid ceuulaat_sin_cos ([i_a, i_a, h_a, w_al, [i_t, j_t, b_t, m_tl)
embed_dim is relevant to the medel, which is predefined (embed_ (embed_din, grid)
pos_embed
return pos_ pos_embed
def calculate_sin sacee_ssi______ee (anchor_pos, target_pos, anchor_gridd anchor_grid target_gridd_sze-11
:param target_pos: [i_t, i_t, h_t, M_t]
:param anchor_pos: [i_a, j_a, h_a, w_a]
:param target_grid_sizes sequence length of the target view (relative to patch size of models)
:para scales pil_image: sequence length of the anchor view (relative to patch size of nodels)
rreturn grid: 2-d grid of the relative positions
[3]] anchor_grid_size
calculate bias of width
w bias
[1] ancher_pos kg
kl = target_pos [3] / target_griddsize
calculate scales of width
w_scale kl
kg ancher_posi2] anchor_ggid_siz
calculate bias of height
biba - [target_pos ancher_pos[0]) / kg
calculate scales of height
kl target_pos[2] target_grid_sizz
h scale kl kg
grid (h.
(h (habi grid size * s_scele + bias--e--, h_scale, dtype=np.float32))
grid_w ar.arang (w_bass, grid_size w_scale w_bias-Se-3,
R_scale, dtype-np.float
make the width and height grids, and width goes first grid = np. np.meshgrid(gridd.. grid_h) grid
grid np.stack (grid, axis=0)
grid reshape [2,, 1, grid_size, grid_size])
return grid
14

---

Published as a conference paper at ICLR 2024
B TRAINING DETAILS
Datasets. We use three datasets without any data selection strategies Tan et al. (2024): Scenery (Yang et al., 2019), Building Facades (Gao et al., 2023), and WikiArt (Tan et al., 2016), in line with (Yang et al., 2019; Yao et al., 2022; Van Hoorick, 2019). Scenery is natural scenery with diverse natural scenes, consisting of about 5,000 images for training and 1,000 images for testing. Building Facades is a city scenes dataset consisting of about 16,000 and I 1,500 images for training and testing, respectively. WikiArt is fine-art paintings dataset, which can be obtained from wikiart.org. We use the split manner of genres datasets (used in (Yao et al., 2022; Gao et al., 2023)), which contain 45,503 training images and 19,492 testing images.
Training Details. We implement our approach with PyTorch (Paszke et al., 2019) on a platform equipped with 8 V100 GPUs The encoder is composed of 8-10 stacked transformer blocks. Then, a cross-attention block is employed, followed by the decoder made of 8-10 transformer blocks. Finally, 3x3 convolutional layer is adopted smooth the generated image. In line with previous methods (Yao et al., 2022), we copy the ground truth (input) to the corresponding location in the generated image. We find the copy operation will make a great impact on previous methods, while PQDiff is much more robust to this operation, which we will discuss later. The number of parameters upproximately equal QueryOTR (Yao al., 2022). which contains 12 transformer blocks in the encoder and 4 transformer blocks in the decoder. We adopt Adam W optimizer (I (Loshchilov & Hutter, 2019), and we set the learning rate 0.0002, weight decay to 0.03, and betas to 0.99. In the training stage, we set the random crop ratio of the anchor view to (0.15, 0.5) and the ratio of the target view to (0.8, 1.0), aiming to use the small view to predict the larger view. We train PQDiff 80k, 150k, and 300k iterations with 64 images per GPU on the Scenery, Building Facades, and WikiArt datasets, respectively. Following QueryOTR, we set the resolution of each cropped view as 192x192. Our core idea of PQ-Diff is to utilize the randomly cropped two views (anchor and target) to learn the positional relation between them.
Evaluation and Baselines. We Inception Score (IS) (Salimans al., 2016), Frechet Inception Distance (FID) (Heusel et al., 2017) to measure the generative quality. Note that the upper bounds of IS are 4.091, 5.660, and 8.779 for Scenery, Building Facades, and WikiArt, respectively, which are calculated by real images in the test set. Here we do not use PSNR (peak signal-to-noise ratio) as once used in outpainting because PSNR cannot reflect the diversity of generated images, which is important for generative models, and more details are given in Sec. 4.2. Alternatively, we report the PSNR score between the input sub-images and the center region of the generated images. Because we think this score is more meaningful to detect whether the network ignores the input conditions. We make comparisons with five SOTA image outpainting methods, NSIPO (Yang et al., 2019), SRN (Wang et al., 2019), IOH (Van Hoorick, 2019), Uformer (Gao et al., 2023) and QueryOTR (Yao et al., 2022).
Sampling Details. In line with the previous method (Yao et al., 2022; Gao et al., 2023), for the testing stage, all images are resized to 192x192 as the ground truth, and then the input images are obtained by center cropping the sizes 128x128, 86x86, and 56x56 for 2.25x, 5x, and 11.7x outpainting, respectively. The total output sizes are 2.25, 5, and 11.7 times the input in terms of 2 2.25x, 5x, and 11.7x outpainting, respectively.
15

---

10 < Fy(yo). Output: y. Published as a conference paper at ICLR 2024
Algorithm 1: Training pipeline of PQDiff.
Algorithm 2: Sampling pipeline of PQDiff.
Input: Raw complete image the generating network for training: ge.
Input: Input sub-image x, the pretrained network ge. VQVAE encoder and decoder Fg. FD. timestep T, the specified outpainting multiple K.
VQVAE encoder and decoder FE, FD. timestep T.
Result: The pretrained network go.
Result: Outpainted image y.
Initialize go.
Compute the relative position p of y through K (illustrated in Fig. 2).
repeat
Randomly sample t from T.
Acquire the positional embedding E through p.
Xa. Xy + RandoomCropResize(x))
Compute the relative position E.
Randomly sample Gaussian noise yr.
Obtain latent embeddings
Zo.b - (xx,))))
repeat
Acquire latent embeddings z + FE(x).
Randomly sample Gaussian noise Ec.
Predict ěT by go YT, T))
Add noise to obtain Eb, via Eq.2. Predict & by go (za. Zb.
Compute Yo and yr-1 through Eq. 5.
T T+-T2
10
Compute the loss | ce via Eq.3. Backward and update the network go-
until T
11
12 until Converge;
Output: 9o
C ILLUSTRATION OF THE RPE DURING THE TRAINING AND SAMPLING
Anchor
A7
Al
Target
A20
AS
Input Image
Target
05
(7.40
Random Crop
Contimauss Outpainting in Sample Stage
(a) Relative Position
Training Stuge
Figure 6: Here is the detailed working mechanism of Relative Position Embeddings (RPE). Among them are (a) Specific calculation example diagram of RPE. (b) RPE captures different position relationships by randomly sampling the anchor view and target view during the training stage. (c) RPE achieves continuous outpainting through the learned positional relation in the sample stage.
Relative positional embeddings. Instead of simply using learnable positional embeddings (com- monly used in previous transformer-basse learning methods (Gao et al., 2023)), which not represent the relation between the anchor view and target view (as the positional relation random for each sample at each iteration due to the randomness of the RandomCrop augmentation), adopt fixed positional encoding to represent the relative positions between the anchor view and each query tokens (i.e., each relative positional token in the target view), which is illustrated in Fig. 6. Given the positions Р = {pAi, PAjx PAh, PAr (top location, left location, height, and width), Pr (PTh, PPpr, PTh: } } of the two views X, and Xy, the objective is calculating the relative
(K7) in the target view based on the anchor view X,,, where position of each patch [xT XT ..... XT
(2)
(K3) is the number of patches in the target view. Specifically, calculate the patch-level relative position of the target view via the following equation:
Ka (pes pai)
Pth Ká (m (m-1) Ka (Ptj Pas) + Ptw KT (n 1).
Pt
[p" .P.
Pah
K, Pah
Pow
Ki Paw
Column
where K2 means the number patches of the anchor view. pt m.n means the relative position of the patch located m-th row and n-th column in the target view. Then, for each patch, the
16

---

Published as a conference paper at ICLR 2024
following popular form in transformers (Vaswani et al., 2017) to generate the relative positional embedding:
pm..
sin
22+1/d
Pt
cos
e2+2/d
Pt
sin
Pr Pf
(9)
Pt
Pi
sin
c2+1/d
.008
2222/d
sin
e
where Pr E RIxDi is the relative positional embeddings of the patch located at m-th row and n-th column in target view. D is the hidden dimension of the model and D d. is the pre-defined parameter that will be universally set to 10,000 in our experiments, in line with MAE (He et al., 2022). (m, n) means the position of the top left patch position (as described in Eq. and illustrated in Fig. 6).
D MORE RESULTS IN CONTINUOUS MULTIPLES
410
102
QueyOTTR
rypaur
32
223
Suare
SKN NSPP 2081
225
5.0
300
10.0
200
Corpltting Mupiplas
Ougulering
Figure 7: Comparisons between PQDiff and previous methods v.s. different outpainting multiples. where our PQDiff achieves fate-of-the--a results under almost all settings. Note that since PQDiff can additionally outpaint images with continuous multiples, we show the results of PQDiff continuous curve instead of scatter plots.
E IMPACT OF THE CONTINUOUS AND DISCRETE POSITIONAL EMBEDDINGS
To further explore how PQDiff learns the relative position, we conduct a set of ablation studies. Specifically, we remove the randomness of the positions of the anchor views and target views. Specifically, the anchor view the center region of the target view, and the target view is set 2.25x, 5x, and 11.7x larger than the anchor view with the same probability at each training iteration (1/3 for each multiple) to better compare with the main results in our paper. Therefore, the sizes of the anchor view and target view are discrete, as they are fixed as three numbers. We train the Discrete version on 8 V100 GPUs with 80k iterations (in line with the main results). Table 7 shows the results when evaluating with variant outpainting multiples. We find under 2.25x, 5x, and 11.7x settings (inner distribution), discrete training, and achieve similar results with PQDiff. However, when transferred to other outpainting multiples (4x, 9x, 16x), both three scores of discrete training drop, especially for Center PSNR. We guess the reason behind the interesting phenomenon is that the "Discrete training" strategy only memorizes the pixel information of the input sub-image, but fails to scale the sub-image with proper ratio due the outpainting multiples gaps in the training and inference stage.
17

---

Published as a conference paper at ICLR 2024
Inner dist
Outer dist
Metric
Method
2.25x
5x
11.7x
4x
9x
16x
Discrete
3.813
3.553
3.207
3.626
3.391
2.891
IS
PQDiff (Ours)
3.849
3.547
3.269
3,827
3.435
3.012
Discrete
29.481
33.917
47.239
35.183
50.183
53.174
FID
PODifT (Ours)
29.446
34.492
44.517
31.274
39,927
49,271
Discrete
27.814
27.017
23.917
18,271
17.238
14.927
Center PSNR
PQDiff (Ours)
27.676
27.267
24.697
27.477
25.820
23.175
Table 7: Performance when transferring the model to different evaluation distributions. "Inner dist" means evaluating the model under the same settings as the training stage. "Outer dist" means evaluating the model under different settings with the training stage.
F SAMPLING SPEED
For better comparisons, we provide results of PQDiff with more timesteps in Table Specifically, in the training phase, set timesteps 1,000. In the testing phase, change the timesteps from 50~500 (since we observe with larger timesteps, the IS and FID score won't improve anymore). We find when timesteps set 300, PQDiff achieves 4.203 IS scores, while ground truth only achieves 4.091, which further demonstrates the vividness of images generated by PQDiff.
Table 8: Comparison of sampling time, FID, and inception scores with 2.25x, 5x, 11.7x settings Scenery dataset. The sampling time is the wall-clock time spent on generating 64 images on V100 GPUs. The best results and second best results are boldface and underlined, respectively.
2.25x
5x
11.7x
Method
Time (Sec.)
FIID
f
Time (Sec.)
FID
Time (Sec.)
FID |
IS
PQDiff (50 timesteps)
12.175
20.103
4.046
12.175
27.051
3.710
12.175
41.242
3,741
PODiff (100 timesteps)
24.481
20.878
3.968
24,481
28.424
3.719
24.481
39.194
3.496
PODifT (200 timesteps)
49.021
19.838
4.111
49.021
31.039
3.940
49.021
40.265
3.583
PQDiff (300 timesteps)
73,586
21.072
4.203
73.586
28.856
3.957
73,586
39.607
3.479
PQDiff (400 timesteps)
98.101
20.623
4.028
98.101
28.939
3.761
98.101
39.618
3.508
PQDiff (500 timesteps)
122.739
19,828
4.098
122.739
27.561
3.728
122.739
41.066
3.709
Ground Truth
4.091
4.091
4.091
G IMPACT OF PREDICTING To OR NOISE.
PQDiff can also be thought of as predictive task conditioned by the relative information and the anchor views. Hence, conduct the experiments to directly predict Xs. which is similar to QueryOTR (Yao et al., 2022), and we report the inception score in Table 9. We find by predicting Xs- PQDiff is much worse than predicting noise under all settings (especially in the 1 11.7x setting), and the phenomenon is also consistent with previous generative tasks (Bao et al., 2023; Ho et al., 2020). We guess that's because predicting x makes the task more predictive but not generative, and the learned network will overfit in the training set, resulting in bad generalization for the generative task in the test set.
H INCORPORATE WITH PRETRAINED MODELS
As shown in Tab. 10 on the scenery dataset, We consider adding an ablation study to analyze the usage of the pretrained model. First, we try load the stable diffusion pretrained model in PQDiff. As VQVAE and cross attention in our model (our model is mansformer-basee model, while stable diffusion mainly uses block), we can only load weights of the spatial transformer layer.
Then, try to train model on ImageNet with 80,000 iterations, and then, re-train the model on the Scenery dataset. We can find that pretrained with the imagenet has improved consistency, which provides insights for the subsequent scale-up of our framework. We find that pretrained with the Imagenet has improved consistency, which provides insights for the subsequent scale-up of framework.
18

---

Published as a conference paper at ICLR 2024
1 MORE GENERATED EXAMPLES ON FACADE DATASETS
put more generated examples on facade datasets Fig. 9. We have observed two interesting phenomena when our PQDiff extends semantic structures in facade scenes.
Our PQDiff' can expand the reliable semantic structure that aligns with human cognition. In contrast, the previous SOTA QueryOTR model pretends generate physically distorted and abnormally deformed semantic structure. (As shown in the middle column (Fig. 9), the comparison includes the First row(lane), second rwoodnn building), and third row (teaching buildings).
Our PQDiff can expand more texture semantic details in facade scenes, thereby generating high-fidelity expanded images. However, QueryOTR will lose detailed information, and the generated extended image has lot noise. (Representative examples are shown in the first column (First row, second row, and third row).
J ABSOLUTE POSITION EMBEDDING V.S. RELATIVE POSITION EMBEDDING
Fixed Mode
Flexible Modes
Anchor
Anchor
Target
Target
(a) Absolute Position Embedding
(b) Relative Position Embedding
Figure 8: The difference: (a) Absolute Position Embeddings and (b) Relative Position Embeddings.
Table 9: Comparison of Xo prediction and noise prediction Scenery dataset. Since the copy operation would lead the infinite value in the center PSNR score, we only report the center PSNR score without the copy operation. 2.25x, 5x, and 11.7x follow the setting in the main experiments.
2.25x
5x
11.7x
Method
IS
Center
FTD
Center
FID J
Center
OTR (Yan al., 2022)
20.366
3.955
39.237
3.431
60.977
3.114
Xo pred
37.645
3.353
26.478
$2.505
3,230
26,027
69.406
2.984
23,835
Copy
27.356
3.920
43.877
3.676
67.170
3.556
pred
29.446
3.849
27.676
34,492
3,547
27.267
44.517
3,269
24.797
Noise pred Copy
20.100
3.981
28.668
3.712
39.465
3.574
19

---

Published as a conference paper at ICLR 2024
QuRrOTR
PQUBI
QueyOTR
Queyorn
Figure 9: Comparisons of PQDiff with the SOTA method QueryOTR on the facades building dataset.
We highlight the noise and spots generated by QueryOTR with yellow boxes.
PODiI
QuaryOTR
PODIIT
Figure 10: Qualitative comparisons with the previous SOTA method QueryOTR (Yao et al., 2022).
Input
Generated samples
Figure 11: Example of images generated by PQDiff with fixed positions.
20

---

Published as a conference paper at ICLR 2024
Table 10: The experiment that incorporates our PQDiff with pretrained stable diffusion.
2.25x
11.7x
Method
15
FID
IS
pretrain
20.100
3.981
28. 668
3.712
39,465
3.574
SD pretrain
20.164
3,985
28.537
3.692
38.917
3.619
Imagenet pretrain
19.791
3,993
28.016
3,731
36.271
3.629
K MORE CASES GENERATED BY PQDIFF
We show more examples generated by PQDiff in Fig. 12, Fig. 13 and Fig. 14.
PODiff
PQDiff'
PQDiff'
Figure 12: Example of images generated by PQDiff in the testing set.
PQDif
PQDiff
PODiff
Figure 13: Example of images generated by PQDiff in the testing set.
21

---

Published as a conference paper at ICLR 2024
PQDiff
PQDiM'
PQDiff
Figure 14: Example of images generated by PQDiff in the testing set.
MORE DISCUSSIONS
Broader impact and potential applications. As PQDiff generate images conditioned by arbitrary relative positions, we believe our method has great potential in image-inpainting (change the RPE of the target view with the inpainting regions), and super-resolutions (interpolate the relative position of the anchor view) if we properly change the relative positional in the training and sampling stages.
22


Reviews:
[{"review_id": "JL8tR6dZJK", "reviewer": "Reviewer_urTZ", "summary": "Most image outpainting method is limited by the requirement of a pre-defined output ratio before training the models. The authors propose a diffusion-based framework to solve this problem by introducing a positional query. The framework can generate output images with arbitrary and continuous ratios within one step.", "questions": "## Questions\n- 1) Is it precise to say that PQDiff handles inputs with various sizes and ratios while the output size is fixed? Assume the goal is to generate an outpainting image with a multiple of 10x given an input image of size (192, 192). Does PQDiff generate a (192, 192) version of the 10x outpainting image and have to leverage an additional decoder (perhaps super-resolution methods) to resize it to (1920, 1920)?\n\n- 2) Can the author provide results on more multiple besides (2.5x, 5x, 11.7x) used in other sota to justify their claim of continuous multiples? For example, how is the performance of 99x? My suggestion would be a figure of FID over continuous multiples or spectrums of outpainting images with increasing multiples.\n\n- 3) What is the quantitative evaluation of Sec 4.2 **Outpainting in an arbitrary position**, i.e. FID under different positions? Does the model prefer certain positions?\n\n## Suggestion\n- 1) The generated and input images in Supp. Sec. J (Fig. 11, 12, ...) can be put side by side for clearer comparison.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- 1) The authors solve an interesting lasting problem in image outpainting research: the requirement of a pre-defined outpainting multiple.\n- 2) The extensive experiments prove the superiority of the proposed framework in comparison to sota.\n- 3) The paper is mostly well-written with a clear explanation of the methodology and implementation", "weaknesses": "1. Question regarding the wording of the contribution or the presentation in Fig. 1. See Q1.\n2. More results are needed to justify the claim of continuous results. See Q2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohU5CdlpGI", "reviewer": "Reviewer_HK74", "summary": "Image outpainting is practically useful yet very challenging, and current methods often require a fixed ratio of the input and output resolution beforehand. Specifically, the devised PQDiff is a single model trained for all outpainting ratios, by introducing a position query in diffusion training. The proposed method can generate output images with arbitrary and continuous multiple. Such abilities are often missing in literature.", "questions": "Are there other methods using POSITIONAL QUERY? As current existing work part does not mention them (is there were) too much.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper provided a simple yet elegant and effective approach for the challenging and important task of image outpainting. Specifically, it can naturally ensure the outpaint images with continuous multiple, and also generates the output in a single forward pass. \n2) This paper shows that the position query techniques can also be applied to GAN-based models which further expand the potential impact of the paper.\n3) The experimental results propsoed in Sec. 4 are convincing, in terms of both quantative results and qualitative results. Meanwhile the method is very efficient due to its single-pass inference nature against arbitary input ratio and continuous multiple.\n4) Supplemental material gives detailed empirical studies to verify the effectiveness of the approach.", "weaknesses": "I think the authors should further discuss the potential of their method to other potential applications beyond outpainting to further expand the potential impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KHCgCTvYQx", "reviewer": "Reviewer_xgEg", "summary": "This paper proposes an diffusion-based image outpainting model that can outpaint images with arbitrary and continuous multiples. This is achieved with the help of the relative positional embedding that can encode any relative position relations. Experimental results validate the effectiveness of the proposed model.", "questions": "The questions and confused points are given in Weakness Section. I am looking forward to the responses from the authors. I may improve my rating if the questions have satisfactory answers.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The target of outpainting images with arbitrary and continuous multiples is very attractive and highly practical.\n2. Using relative positional embedding to deal with the problem of arbitrary and continuous multiples is simple but effective.", "weaknesses": "1. The main weakness of this paper is some unclear explanations and descriptions.\n\n(1) The authors categorize existing methods into two classes, GAN-based and MAE-based methods, and emphasize that one of their  limitations is  the multiple running times to achieve the large expansion multiple. However, it is unclear why these two types of methods need to run multiple times. Using GAN or MAE is not the crux.\n\n(2) As the authors fail to clearly explain why previous methods cannot outpaint images with arbitrary and continuous multiples. The strengths and the contributions of the proposed method become unclear and unconvincing.\n\n(3) The authors say \"to achieve one-step diffusion-based generation, we propose to use relative positional queries and input sub-images as conditions\". This is really strange. Why relative positional queries can reduce the number of diffusion steps?\n\n(4) In section 3, the authors say that they set h1 = h2 = h and w1 = w2 = w for notation simplicity. It looks improper.\n\n(5) Some parts of Figure 2 are confusing. For example, in the training part, the relative positional embedding takes the two image views as inputs. But Eq.1 shows that only the relative position (m,n) is needed to generate the embedding. The images are not the inputs of it. In the sampling part, it is unclear what the inputs are. I cannot know which of the original image, the cropped image, and the masked image are the inputs\n\n(6) In Eq.2, it seems Xb0 should be Zb0\n\n2. There is a strange point in the proposed model. As shown in Figure 2, the relative position only reflects the position relations of the top-left corners between the two views. In other words, even the anchor view and the relative position are fixed, there can still be many different target views. How does this one-to-many mapping influence the model? Furthermore, in the sampling stage, if the input noise, image and relative position are all fixed, how to ensure the outpainting ratios on four edges are the same as what we expected?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)