InfinityGAN: Towards Infinite-Pixel Image Synthesis
Authors: Chieh Hubert Lin
Hsin-Ying Lee
Yen-Chi Cheng
Sergey Tulyakov
Ming-Hsuan Yang
Venue: ICLR 2022 Poster (2022)
Keywords: generative modeling
image synthesis
generative adversarial networks
infinite-pixel synthesis
GANs

Abstract: We present InfinityGAN, a method to generate arbitrary-sized images. The problem is associated with several key challenges. First, scaling existing models to an arbitrarily large image size is resource-constrained, both in terms of computation and availability of large-field-of-view training data. InfinityGAN trains and infers patch-by-patch seamlessly with low computational resources. Second, large images should be locally and globally consistent, avoid repetitive patterns, and look realistic. To address these, InfinityGAN takes global appearance, local structure and texture into account. With this formulation, we can generate images with spatial size and level of detail not attainable before. Experimental evaluation supports that InfinityGAN generates images with superior global structure compared to baselines and features parallelizable inference. Finally, we show several applications unlocked by our approach, such as fusing styles spatially, multi-modal outpainting and image inbetweening at arbitrary input and output sizes.

Published as a conference paper at ICLR 2022
INFINITYGAN: TOWARDS INFINITE-PIEEL IMAGE S SYNTHESIS
Chieh Hubert Lin' Hsin- Ying Lee" Yen-Chi Cheng Sergey Tulyakov?
Ming-Hsuan Yang4.45 44.55
UC Merced SSap Inc. *Carnegie Mellon University
Yonsei University
Google Research
ABSTRACT
We present novel framework, InfinityGAN, for arbitrary-sizedt image generation. The task is associated with several key challenges. First, scaling existing models to an arbitrarily large image size is resource-constraine. in terms of both compu- tation and availability of large-field-of training data. InfinityGAN trains and infers in seamless patch-by-patch manner with low computational resources. Second, large images should be locally and globally consistent, avoid repetitive patterns, and look realistic. To address these, InfinityGAN disentangles global ap- pearances, local structures, and textures. With this formulation, we can generate images with spatial size and level of details not attainable before. Experimen- tal evaluation validates that InfinityGAN generates images with superior realism compared baselines and features parallelizable inference. Finally, we show several applications unlocked by our approach, such as spatial style fusion, multi- modal outpainting. and image inbetweening. All applications can be operated with arbitrary input and output sizes.
Style
Style B
Toinir Paich
(Ultttn
StyleA
Style
Figure Synthesizing infinite-pixel images from finite-sized training data. x024x2048 image composed of 242 patches, independently synthesized by InfinityGAN with spatial fusion of two styles. The generator trained on 101 x 101 patches (e.g., marked top-left) sampled from x97x 197 real images. Note that training and inference (of any size) are performed single GTX TITAN X GPU. Zoom-in for better experience.
INTRODUCTION
"To infinity and beyond!"
Buzz Lightyear
Generative models witness substantial improvements in resolution and level of details. Most im- provements come at a price of increased training time (Gulrajani et al., 2017; Mescheder et al., 2018), larger model size (Balaji al., 2021), and stricter data requirements (Karras et al., 2018). The most recent works synthesize images at 1024x 1024 resolution featuring high level of details
trained models are publicly Project page: http::/fuhen00527 innfiiiinnnnnnnnGGGAAA

---

Published as a conference paper at ICLR 2022
and fidelity. However, models generating high resolution images usually still synthesize images of limited beld-of-view bounded by the training data. It is not straightforward to scale these models to generate images of arbitrarily large field-of-view. Synthesizing aefinite-pixdl images is constrained by the finite nature of resources. Finite computational resources (e.g., memory and training time) set bounds for input receptive field and output size. A further limitation is that there exists infinite- pixel image dataset. Thus, to generate infinite-pixel images, a model should learn the implicit global structure without direct supervision and under limited computational resources.
Repetitive texture synthesis methods (Efros & Leung, 1999; Xian et al., 2018) generalize to large spatial sizes. Yet, such methods are not able to synthesize real-world images. Recent works, such as SinGAN (Shaham et al., 2019) and InGAN (Shocher et al., 2019), learn an internal patch distri- bution for image synthesis. Although these models can generate images with arbitrary shapes, in Section 4.1, we show that they do not infer structural relationships well, and fail to construct plau- sible holistic views with spatially extended latent space. A different approach, COCO-GAN (Lin et al., 2019), learns coordinate-conditione patch distribution for image synthesis. As shown in Figure 4, despite the ability to slightly extend images beyond the learned boundary, fails to main- tain the global coherence of the generated images when scaling to 2 larger generation size.
How to generate infinite-pixel images? Humans are able to guess the whole scene given a partial observation of it. In a similar fashion, we aim to build generator that trains with image patches, and inference images of unbounded arbitrary-large size. An example of a synthesized scene containing globally- structure and heterogeneous textures is shown in Figure 1.
We propose InfinityGAN, a method that trains on a finite-pixel dataset, while generating infinite- pixel images at inference time. InfinityGAN consists of neural implicit function, termed structure synthesizer, and a padding-free StyleGAN2 generator, dubbed texture synthesizer. Given a global appearance of an infinite-pixel image, the structure synthesizer samples a sub-region using coor- dinates and synthesizes an intermediate local structural representations. The texture synthesizer then seamlessly synthesizes the final image by parts after filling the fine local textures to the local structural representations. InfinityGAN can infer a compelling global composition of a scene with realistic local details. Trained on small patches, InfinityGAN achieves high-quality, seamless and arbitrarily-sized outputs with low computational resources- single TITAN X to train and test.
We conduct extensive experiments to validate the proposed method. Qualitatively, we present the everlastingly long landscape images. Quantitatively, we evaluate InfinityGAN and related methods using user study and a proposed ScaleInv FID metric. Furthermore, we demonstrate the efficiency and efficacy of the proposed methods with several applications. First, we demonstrate the flexibility and controllability of the proposed method by spatially fusing structures and textures from different distributions within image. Second, we show that our model is an effective deep image prior for the image outpainting task with the image inversion technique and achieves multi-modal out- painting of arbitrary length from arbirrarily-hhaed inputs. Third, with the proposed model we can divide-anddcomquer the full image generation into independent patch generation and achieve 7.2x of inference up with parallel computing, which is critical for high-resolution image synthesis.
2 RELATED WORK
Latent generative models. Existing generative models are mostly designed to synthesize images of fixed sizes. A few methods (Karras et al., 2018: 8; 2020) have been recently developed to train latent generative models high-resolution images, up to 1024x 1024 pixels. However, latent generative models generate images from dense latent vectors that require synthesizing all structural contents at once. Bounded by computational resources and limited by the learning framework and architecture, these approaches synthesize images of certain sizes and are non-trivial to generalize to different output size. In contrast, patch- based GANs trained on image patches (Lin et al., 2019; Shaham al., 2019: Shocher et al., 2019) are less constrained by the resource bottleneck with the synthesis-by by-part approach. However, (Shaham et al., 2019; Shocher et al., 2019) can only model and repeat internal statistics of a single image, and (Lin et al., 2019) can barely extrapolate few patches beyond the training size. ALIS (Skorokhodov et al., 2021) is a concurrent work that also explores synthesizing infinite-pixel images. It recursively inbetweens latent variable pairs in the horizontal direction. We further discuss the method in Appendix A.I. Finally, autoregressive models (Oord et al., 2016; Razavi et al., 2019; Esser et al., 2021) can theoretically synthesize at arbitrary image sizes. Despite (Razavi et al., 2019) and (Esser et al., 2021) showing unconditional images synthesis at 1024x 1024 resolution, their application in infinite- pixell image synthesis has not yet been well-explored.
2

---

Published as a conference paper at ICLR 2022
Section 3:2
Section 3.3
Implicit Image (Infinitely Large)
Section 3.5
Figure
Derierii No
Latentt)
Coctibats
(raglii Famctoc)
Syntasiaer (PAly Conrotticoal
Figure 2: Overview. The generator of InfinityGAN consists of two modules, structure synthesizer based on neural implicit function, and a fully-convolutional texture synthesizer with all positional information removed (see Figure 3). The two networks take four sets of inputs, a global latent variable that defines the holistic appearance of the image, a local latent variable that represents the local and structural variation, a continuous coordinate for learning the neural implicit structure synthesizer, and set of randomized noises model fine-grained texture. InfinityGAN synthesizes images of arbitrary size by learning spatially extensible representations.
Conditional generative models. Numerous tasks such as image super-resolution, semantic image synthesis, and image extrapolation often showcase results over x024 1024 pixels. These tasks are less related to our setting, as most structural information is already provided in the conditional inputs. We illustrate and compare the characteristies of these tasks against ours in Appendix B.
Image outpainting. Image outpainting (Abdal et al., 2020; Liu et al., 2021; Sabini & Rusak, 2018; Yang et al., 2019) is related to image inpainting (Liu et al., 2018a; Yu et al., 2019) and shares similar issues that the generator tends copy-and- and-paaaphaasee the conditional input or create mottled textural samples, leading to repetitive results especially when the outpainted region is large. InOut (Cheng al., 2021) proposes to outpaint image with GANs inversion and yield results with higher diversity. We show that with InfinityGAN as the deep image prior along with InOut (Cheng et al., 2021), we obtain the state-of-the-ar outpainting results and avoids the need of sequential outpainting. Then, we demonstrate applications in arbitrary-distant image inbetweening, which is at the intersection of image inpainting (Liu al., 201 8a; Nazeri al., 2019; Yu et al., 2019) and outpainting research. Neural implicit representation. Neural implicit functions (Park et al., 2019; Mescheder et al., 2019; Mildenhall et al., 2020) have been applied to model the structural information of 3D and con- tinuous representations. Adopting neural implicit modeling, our query-by- by-ry--b-cooriiatte synthesizer is able to model structural information effectively. Some recent works (De et al., 2021; Niemeyer & Geiger, 2021; Chan et al., 2021) also attempt to integrate neural implicit function into generative models, but aiming at 3D-structure modeling instead of extending the synthesis field-of-view.
3 PROPOSED METHOD
3.1 OVERVIEW
An arbitrarily large image can be described globally and locally. Globally, images should be coher- ent and hence global characteristics should be expressible by a compact holistic appearance (e.g., medieval landscape, ocean view panorama). Therefore, we adopt a fixed holistic appearance for each infinite-pixel image to represent the high-level composition and content of the scene. Locally, close-up view of an image is defined by its local structure and texture. The structure represents objects, shapes and their arrangement within local region. Once the structure is defined, there exist multiple feasible appearances ture and texture should conform to the global holistic appearance to maintain the visual consistency among the neighboring patches. Given these assumptions, we can generate infinite-pixe image by first sampling a global holistic appearance, then spatially extending local structures and textures following the holistic appearance.
textures to render realistic scenes. At the same time, struc-
Accordingly, the InfinityGAN generator G' consists of structure synthesizer Gs and a texture syn- thesizer Gr. Gs an implicit function that samples a sub-region with coordinates and creates local structural features. G is a fully convolutional StyleGAN2 (Karras et al., 2020) modeling textural properties for local patches and rendering final image. Both modules follow a consistent holistic appearance throughout the process. Figure 2 presents the overview of our framework.
3

---

Published as a conference paper at ICLR 2022
3.2 STRUCTURE SYNTHESIZER (Gs)
Structure synthesizer is neural implicit function driven by three sets of latent variables: A global latent vector 2g representing the holistic appearance of the infinite-pixel image (also called implicir image since the whole image is never explicitly sampled), local latent tensor 21 expressing the local structural variation of the image content, and a coordinate grid c specifying the location of the patches to sample from the implicit image. The synthesis process is formulated as:
= (z(z. Zi, c),
(1)
where zy denotes the structural latent variable that is later used as an input to the texture synthesizer.
We sample Zg RDxx from unit Gaussian distribution once and inject z into every layer and pixel in Gg via feature modulation (Huang & Belongie, 2017; Karras et al., 2020). As local variations are independent across the spatial dimension, we independently sample them from unit Gaussian prior for each spatial position of 2 E R#* XD,, Dai, where H and W can be arbitrarily extended.
We then use coordinate grid c to specify the location of the target patches to be sampled. To be able to condition Gş with coordinates infinitely far from the origin, we introduce a prior by exploiting the nature of landscape images: (a) self-similarity for the horizontal direction, and (b) rapid saturation (e.g., land, sky or ocean) for the vertical direction. To implement this, we use the positional encoding for the horizontal axis similar to (Vaswani al., 2017: Tancik et al., 2020; Sitzmann et al., 2020). We use both sine and cosine functions to encode each coordinate for numerical stability. For the vertical axis, to represent saturation, we apply the tanh function. Formally, given horizontal and vertical indexes (In,, by)) of tensor, we encode them as c (tanh(,y)) cos(iz/ ( sin(i,,/T), where T is the period of the sine function and c controls the location of the patch to generate.
To prevent the model from ignoring the variation of 2 and generating repetitive content by following the periodically repeating coordinates, we adopt a mode-seeking diversity loss (Mao et al., 2019; Lee et al., 2020) between pair of local latent variables Z), and zı, while sharing the same Zg and
Caiv
|Z| 21 / ||Gs(zy Z,, c) Gs (Zg. Z c)ll1
(2)
Conventional neural implicit functions produce outputs for each input query independently, which is pixel in zj for InfinityGAN. Such a design causes training instabilities and slows convergence, as we show in Figure 37. We therefore adopt the feature unfolding technique (Chen et al., 2021) to enable Gs to account for the information in broader neighboring region of z and c, introducing a larger receptive field. For each layer in Gs. before feeding forward to the next layer, we apply k x & feature unfolding transformation at each location (i.j) of the origin input f to obtain the unfolded input f':
(3) where Concat(-) concatenates the unfolded vectors in the channel dimension. In practice, as the grid- shaped z and e are sampled with equal spacing between consecutive pixels, the feature unfolding can be efficiently implemented with CoordConv (Liu et al., 2018b).
fan = Concat (( n, + m.)).mm(-+/22A22
3.3 TEXTURE SYNTHESIZER (GT)
Texture synthesizer aims to model various realizations of local texture given the local structure zs generated by the structure synthesizer. In addition to the holistic appearance Zg and the local structural latent zş. texture synthesizer uses noise vectors Zu to model the finest-grained textural variations that are difficult to capture by other variables. The generation process can be written as:
Ps GT(zs, Zgz 2n),
(4)
where Pa is a generated patch at location c (i.e., the used in Eq for generating Zz).
We implement upon StyleGAN2 (Karras et al., 2020). First, we replace the fixed constant input with the generated structure zş. Similar to StyleGAN2, randomized noises 2n are added to all layers of GT, representing the local variations of fine- sine-granedd textures. Then, a mapping layer projects zg to a style vector, and the style is injected into all pixels in each layer via feature modulation. Finally, remove all rero-naddings from the generator, as shown in Figure 3(b).
Both sero-padding and Gg can provide positional information to the generator, and we later show that positional information is important for generator learning in Section 4.2. However, it is nec- essary to remove all zero eero-paddings from Gr for three major reasons. First, zero-padding has a consistent pattern during training, due to the fixed training image size. Such behavior misleads
4

---

Published as a conference paper at ICLR 2022
(a) Conventional G enerator
(b) Tadding-Free G Generator
Pash.
Feluee
Local Labent
Coordinate
Podiiaalltee
Esample Cawa
Cena Transpoo
Dikziin
2sample 33 Comaaa
XY2
Transooe
AR
X2X
Podizaal Pallnay
Kiokedd Ipand
Image
Conaideal
Figure 3: Padding-free generator. (Left) Conventional generators synthesize inconsistent pix- els due to the zero-paddings. Note that the inconsistency region grows exponentially as the net- work deepened. (Right) In contrast, our padding-free generator can synthesize consistent pixel value regardless of the position in the model receptive field. Such a property facilitates spatially- independently generating patches and forming into seamless image with consistent feature values.
the generator to memorize the padding pattern, and becomes vuinerable to unseen padding patterns while attempting to synthesize at different image size. The third column of Figure 4 shows when we extend the input latent variable of the StyleGAN2 generator multiple times, the center part of the features does not receive expected coordinate information from the paddings, resulting in extensively repetitive textures in the center area of the output image. Second, zero-paddings can only provide positional information within a limited distance from the image border. However, while generating infinite-pixe images, the image border is considered infinitely far from the generated patch. Finally, as shown in Figure 3, the existence of paddings hampers GT from generating separate patches that can be composed together. Therefore, remove all paddings from Gr. facilitating the synthesis- by-parts of arbitrary-sized images. We refer to the proposed GT as a padding-fre generator (PFG).
3.4 SPATIALLY INDEPENDENT GENERATION
InfinityGAN enables spatially independent generation thanks to two characteristics of the proposed modules. First, Gg. neural implicit function, naturally supports independent inference each spatial location. Second, GT, as a fully convolutional generator with all paddings removed, can synthesize consistent pixel values the same spatial location in the implicit image, regardless of different querying coordinates, as shown in Figure 3(b). With these properties, we can independently query and synthesize a patch from the implicit image, seamlessly combine multiple patches into an arbitrarily large image, and maintain constant memory usage while synthesizing images of any size.
In practice, having a single center pixel in zs slice that aligns to the center pixel of the corre- sponding output image patch can facilitate 2 and c indexing. We achieve the goal by shrinking the StyleGAN2 blur kernel size from to 3, causing the model to generate odd-sized features in all layers, due to the convolutional transpose layers.
3.5 MODEL. TRAINING
The discriminator D of InfinityGAN is similar to the one in the StyleGAN2 method. The detailed architectures of and are presented in Appendix D. The two networks are trained with the non- saturating logistic loss Ladv (Goodfellow et al., 2014), R₁ regularization LR. (Mescheder et al., 2018) and path length regularization Cpath (Karras al., 2020). Furthermore, to encourage the generator to follow the conditional distribution in the vertical direction, train G and D with auxiliary task (Odena et al., 2017) predicting the vertical position of the patch:
Lar = ||cy C
(5)
where čy is the vertical coordinate predicted by D. and c. is either (for generated images) cy = tanh(iy) (for real images) the vertical position of the patch in the full image. We formulate Lar as a regression task. The overall loss function for the InfinityGAN is:
min Ladv + Cac + AB, CR, 17
(6)
min - Ladv + Aarhar + Aaiv Laiv + ApathCpath 6
where X are the weights.
5

---

Published as a conference paper at ICLR 2022
Table 1: Quantitative evaluation on Flickr-Landscape. Despite we use disadvantageous setting for our InfinityGAN (discussed in Section 4.1), it still outperforms all baselines after extending the size to 4x larger. Furthermore, the user study shows an over 90% preference favors our Infinity- GAN results. The preference is marked as x% when x% of selections prefer the results from the corresponding method over InfinityGAN, 'he images are first resized to 128 before resizing to 197.
Method
Image Size
FID
Scaletnv FID
Preference v.s. Ous
Inference
Fall
Train
Test 8x
Train
1x
2x
4x
8x
4x
8x
Memory
SinGAN
128
128
1024
4.21
4.21
57. 10
145.12 12
210.22
0.80%
1.60%
Osize
COCO-GAN
1024
17.52
41.32
258.51 51
376.69
387.15
0%
0%
O(1)
SYycGANN+4CC
128
128
1024
4.19
4.19
18.31
79.83
189.65
9.20%
7.20%
Osize
StyleGAN2+NCI (Patchod)
128
1024
5.35
21.06
58.84
165.65 65
234.19
Osize
SyyleGAN+NNCI-PG
1971
1576
86.76
90.79
126.88
211.22
272.80
0.40%
1.20%
O(1)
InfinityGAN (Ours)
(SyUUGANN+NNCIIHGGGGG] 1971
101
1576
11.03
21.84
28.83
61.41
12L.8
(
COCO-GAN
SinGAN
StyleGAN+NCI
SCC-
InfinityGAN (ours)
Generated Full Image
128x 128
128 128
128 128
197x197
197x 197
Test- set-time extension
to x024x 1024
Figure 4: Qualitative comparison. We show that InfinityGAN can produce more favorable holistic appearances against related methods while testing with an extended size 1024x 1024. (NCI: Non- Constant Input, PFG: Padding- FFree Generator). More results are shown in Appendix E.
EXPERIMENTAL RESULTS
Datasets. We evaluate the ability of synthesizing at extended image sizes on the Flickr-Landscape dataset consists of 450,000 high-quality landscape images, which are crawled from the Landscape group on Flickr. For the image outpainting experiments, we evaluate with other baseline meth- ods scenery-
subsets from the Place365 (Zhou et al., 2017) (62,500 images) and Flickr- Scenery (Cheng et al., 2021) (54,710 images) datasets. Note that the Flickr-Scenery here is different from our Flickr-Landscape. For image outpainting task, split the data into 80%, 10%, 10% for training, validation, and test. All quantitative and qualitative evaluations are conducted on test set.
Hyperparameters. We Aar = 1, Adiv 1, AR, 10, and Apath for all datasets. All models are trained with x01 x 101 patches cropped from 197x197 real images. Since our InfinityGAN synthesizes odd-sized images, we choose 101 that maintains sufficient resolution that humans can still recognize its content. On the other hand, 197 is the next output resolution if stacking another upsampling layer to InfinityGAN, which also provides 101 101 patches sufficient feld-of-view. We adopt the Adam (Kingma & Ba, 2015) optimizer with 31 0, ,33 0.99 and batch size 16 for 800,000 iterations. More details are presented in Appendix C.
Metrics. We first evaluate Fréchet Inception Distance (FID) (Heusel et al., 2017) at G training res- olution. Then, without access to real images at larger sizes, we assume that the real landscape with a larger FoV will share a certain level of self-similarity with its smaller FoV parts. We accordingly propose a ScaleIny FID, which resizes larger images to the training data size with bilinear interpo- lation, then computes FID. We denote Nx ScaleInv FID when the metric is evaluated with images Nx larger than the training samples.
Evaluated Method. We perform the evaluation Flickr-Landsean with the following algorithms:
SinGAN. We train an individual SinGAN model for each image. The images at larger sizes generated by setting spatially enlarged input latent variables. Note that we do not compare with the super-resolution setting from SinGAN since we focus on extending the learned structure rather than super-resolve the high-frequency details.

---

Published as a conference paper at ICLR 2022
COCO-GAN. Follow the "Beyond- -oonnddr Generation" protocol of COCO-GAN, we transfer a trained COCO-GAN model to extended coordinates with a post-training procedure.
StyleGAN2 (+NCI). We replace the constant imput of the original StyleGAN2 with of the same shape, we call such a replacement as "non-constant input (NCI)". This modification enables StyleGAN2 to generate images at different output sizes with different z sizes.
4.1 GENERATION AT EXTENDED SIZE.
Additional (unfair) protocols for fairness. We adopt two additional pre- and post-processing to ensure that InfinityGAN does not take advantage of its different training resolution. To ensure Infin- ityGAN is trained with the same amount of information as other methods, images are first bilinear interpolated into 128 x 128 before resized into x97 197. Next, for all testing sizes in Table 4, In- finityGAN generates at 1 1.54x (=197/128) larger size to ensure final output images share the same FoV with others. In fact, these corrections make the setting disadvantageous for InfinityGAN, as it is trained with patches of 50% FoV, generates 54% larger images for all settings, and requires to composite multiple patches for its 1 ScaleInv FID.
Quantitative analysis. For all the FID metrics in Table 1, unfortunately, the numbers are not directly comparable. since InfinityGAN is trained with patches with smaller FoV and at a different resolution. Nevertheless, the trend in ScaleInv FID is informative. It reflects the fact that the global structures generated from the baselines drift far away from the real landscape as the testing FoV enlarges. Meanwhile, InfinityGAN maintains more steady slope, and surpasses the strongest baseline after 4x ScaleInv FID. Showing that InfinityGAN indeed performs favorably better than all baselines as the testing size increases.
Qualitative results. Figure 4, we show that all base- lines fall short of creating reasonable global structures with spatially expanded input latent variables. COCO-GAN is unable to transfer to coordinates when the extrapolated coordinates are too far away from the training distribution. Both SinGAN and StyleGAN2 implicitly establish image features based on position encoded by zero padding, as- suming the training and testing position encoding should be the same. However, when synthesizing at extended image sizes, the inevitable change in the spatial size of the input and the features leads to drastically different position en- coding in all model layers. Despite the models can still syn- thesize reasonable contents near the image border, where the position encoding is still partially correct, they fail to synthesize structurally sound content in the image center. Such a result causes ScaleInv FID to rapidly surge as the ex- tended generation size increases to 1024x 1024. Note that at the 16x setting, StyleGAN2 runs out of memory with a batch size of 1 and does not generate any result. In com- parison, InfinityGAN achieves reasonable global structures with fine details. Note that the 1024 x 1024 image from InfinityGAN is created by compositing 121 independently synthesized patches. With the ability of generating con- sistent pixel values (Section 3.4). the composition is guar- anteed to be seamless. We provide more comparisons in Appedix E, larger set of generated samples in Appendix F, results from models trained at a higher resolution in Ap- pendix G, and a very-long synthesis result in Appendix J.
Figure 5: LSUN bridge and tower. InfinityGAN synthesize at 512x512 pixels. We provide more details and samples in Appendix H.
Ditterent local varkablis
Figure 6: Diversity. InfinityGAN synthesizes diverse samples at the same coordinate with different local latent and styles. More samples are shown in Appendix
In Figure 5, we further conduct experiments on LSUN bridge and tower datasets, demonstrating InfinityGAN is applicable other datasets. However, since the two datasets are object centric with a low view-angle variation in the vertical direction, InfinityGAN frequently fills the top and bottom area with blank padding textures.
In Figure 6, we switch different z and GT styles (i.e., Zg projected with the mapping layer) while sharing the same More samples be found in Appendix I. The results show that the structure
7

---

Published as a conference paper at ICLR 2022
Figure 7: Spatial style fusion. We present a mechanism in fusing multiple styles together to increase the interestingness and interactiveness of the generation results. The 512x 4096 image fuses four styles across 258 independently generated patches.
Table 2: Outpainting performance. The combi- nation of IntOut (Cheng et al., 2021) and Infin- ityGAN achieves state-of-thh--an art IS (higher bet- ter) and FID (lower better) performance on image outpainting task.
Table 3: Inference speed up with parallel batching. Benefit from the spatial independent generation nature, InfinityGAN achieves up x.20x inference speed up by with parallel batch-
ing at 8192x pixels. The complete table can be found in Appendix P.
Place365
Fliekr-Scenery
Method
FID J
IS
FID
18 1
Method
Parallel
GPUs
Inference Time (second / image)
Speed Up
Batch Size
Boundless
35 02
6.15
61.98
6,98
NS-outpaint
58.68
4.70
61.16
4.76
StyleGAN2
N/A
OOM
In&Out
23.57
7.18
30.34
7.10
137.44 19.09
x1.00 x7.20
In&OOut- InfinityGAN
Ours
128
and texture are disentangled and modeled separately by Gs and Gr. The figure also shows that Gs can generate a diverse set of structures realized by different zl.
User study. We use two-alternative forced choice (2AFC) between InfinityGAN and other baselines on the Lancss-pa dataset. A total of 50 participants with basic knowledge in computer vision engage the study, and we conduct 30 queries for each participant. For each query. show two separate grids of 16 random samples from each of the comparing methods and ask the participant to select "the one you think is more realistic and overall structurally plausible." As presented in Table 1, the user study shows an over 90% of preference favorable to InfinityGAN against all baselines.
4.2 ABLATION STUDY: THE POSITIONAL INFORMATION IN GENERATOR
As discussed in Section 3.3, we hypothesize that StyleGAN2 highly relies on the positional in- formation from the zero- prr--paddings. In Table 1 and Figure 4, we perform an ablation by removing all paddings from StyleGAN2+NCI, yielding StyleGAN2+NCI+PPF that has no positional infor- mation in the generator. The results show that StyleGAN2+NCI+PFG fails to generate reasonable image structures, and significantly degrades in all FID settings. Then, with the proposed Gs. the positional information is properly provided from 2g. and resumes the generator performance back to a reasonable state.
4.3 APPLICATIONS
Spatial style fusion. Given single global latent variable Zg. the corresponding infinite pixel image is tied to a single modal of global structures and styles. To achieve greater image diversity and allow the user to interactively generate images, propose a spatial fusion mechanism that can spatially combine two global latent variables with smooth transition between them. First, we manually define multiple style centers the pixel space and then construct an initial fusion map by assigning pixels to the nearest style center. The fusion map consists of one-hot vectors for each pixel, forming a style assignment map. According to the style aesienmsss assignment map, we then propagate the styles in all intermediate layers. Please refer to Appendix L for implementation details. Finally, with the fusion maps annotated for every layer, we can apply the appropriate zg from each style center to each pixel using feature modulation. Note that the whole procedure has a similar inference speed as the normal synthesis. Figure 7 shows synthesized fusion samples.
Outpainting via GAN Inversion. We leverage the pipeline proposed in In&Out (Cheng et al., 2021) to perform image outpainting with latent variable inversion. All loss functions follow the ones proposed in Out. We first obtain inverted latent variables that generates an image similar to the given real image via GAN inversion techniques, then outpaint the image by expanding z and zn with their unit Gaussian prior. See Appendix K for implementation details.
In Table 2, our model performs favorably against all baselines in image outpainting (Bound- less (Teterwak et al., 2019), NS-outpaint (Yang et al., 2019), and In&Out (Cheng al., 2021)).
8

---

Published as a conference paper at ICLR 2022
Bounlless
NS-adpaint
InkOut
In&Out + InfinityGAN
Multi-modal outpainting, Infinity- GAN natively achieve multi-modal outpaint- ing by sampling different local latents in the out- painted region. The real image annotated with red box is 256 128 pixels. We present more out- painting samples in Appendix M.
Figure 9:
Figure 8: Outpainting long-range area. Infin- ityGAN synthesizes continuous and more plau- sible outpainting results for arbitrarily large painting areas. The real image annotated with red box is 256x 128 pixels.
A Arbitwey- Lnngh Inbetwenning
Arbitrary Cyclk
Figure 10: Image inbetweening with inverted latents. The InfinityGAN can synthesize arbitrary- length cyclic panorama and inbetweened images by inverting real image different position. The top-row image size is 256 2080 pixels. We present more samples in Appendix N and Appendix O.
As shown in Figure 8, while dealing with a large outpainting area (e.g., panorama), all previous outpainting methods adopt a sequential process that generates a fixed region at each step. This intro- duces obvious concatenation seams, and tends to produce repetitive contents and black regions after the multiple steps. In contrast, with InfinityGAN the image prior in the pipeline of (Cheng et al., 2021), we can directly outpaint arbitrary size target region from inputs of arbitrary shape. More- over, in Figure 9, show that our outpainting pipeline natively supports multi-modal outpainting by sampling different local latent codes in the outpainting area.
Image inbeiweening with inverted latent variables. We show another adaptation of outpainting with model inversion by setting two sets of inverted latent variables at two different spatial locations, then perform spatial style fusion between the variables. Please refer to Appendix K for implemen- tation details. As shown in Figure 10, we can naturally inbetween (Lu et al., 2021) the area between two images with arbitrary distance. A cyclic panorama of arbitrary width can also be naturally generated by setting the same image on two sides.
Parallel batching. The nature of single image. As shown in Table 3, by stacking batch of patches together, InfinityGAN can sig- nificantly speed up inference at testing up to 7.20 times. Note that this speed-up is critical for high-resolution image synthesis with large number of FLOPs.
generation enables parallel inference on
CONCLUSIONS
In this work, we propose and tackle the problem of synthesizing infinite pixel images, and demon- strate several applications of InfinityGAN, including image outpainting and inbetweening.
Our future work will focus improving Infinity AN in several aspects. First, our Flickr-Landscape dataset consists of images taken at different FoVs and distances the scenes. When InfinityGAN composes landscapes of different scales together, synthesized images may contain artifacts. Second, similar to the FoV problem, some images intentionally include tree leaves on top of the image as a part of the photography composition. These greenish textures cause InfinityGAN sometimes synthesizing trees or related elements the sky region. Third, there still slight decrease in FID score in comparison to StyleGAN2. This may be related to the convergence problem in video synthesis (Tian et al., 2021). in which the generator achieves inferior performance if a preceding network (e.g., the motion module in video synthesis) is jointly trained with the image module.

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENTS
This work is supported in part by the NSF CAREER Grant #1149783 and a gift from Snap Inc.
ETHICS STATEMENT
Our work follows the General Ethical Principles listed at ICLR Code of Ethics (https://icl) cc cc/publ i c Codeofft SedeeffEthie cs).
The research in generative modeling is frequently accompanied by concerns about the misuse of ma- nipulating or hallucinating information for improper use. Despite none of the proposed techniques aiming at improving manipulation of fine-graiie fine-grained image detail hallucinating human activities, we cannot rule out the potential of misusing the framework to recreate fake scenery images for any inap- propriate application. However, we do drastically alter the plausibility of synthesis results in the eigh-frequeccy domain, our research is still covered by continuing research in ethical generative modeling and image forensics.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegantt: How to edit the embedded images? In IEEE Conference on Computer Vision and Pattern Recognition, 2020. 33
Yogesh Balaji, Mohammadmabdi Sajedi, Neha Mukund Kalibhat, Mucong Ding, Dominik Stöger, Mahdi Soltanolkotabi, and Soheil Feizi. Understanding over-paraanttriiaationn in generative ad- versarial networks. In International Conference on Learning Representations, 2021.
Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic spatial gan. 2017. 41
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In IEEE Conference Computer Vision and Pattern Recognition, 2021.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Yen-Chi Cheng, Chich Hubert Lin, Hsin- Lee, Sergey Tulyakov, Jian Ren, and Ming-Hsuan Yang. In&Out: Diverse image outpainting via gan inversion. arXiv preprint XiX::I004.00675 2021. 3, 6, 92
Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. In IEEE International Conference on Computer Vision, 2021.
Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In IEEE
International Conference on Computer Vision, 1999.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Puttern Recognition, 2021. 2
Anna Frühstück,, Ibraheem Alhashim, and Peter Wonka. Tilegan: synthesis of large-scale non- homogeneous textures. ACM Transactions on Graphics, 2019. 41
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farler Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Pro- cessing Systems, 2014. 5
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville, Im- proved training of wasserstein gans. In Neural Information Processing Systems, 2017.
10

---

Published as a conference paper at ICLR 2022
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Neural Information Processing Systems, 2017. 6
Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. Improved techniques for training single-image gans. In IEEE Winter Conference Applications of Computer Vision, 2021. 41
Xun Huang and Serge Belongie. Arbitrary style transfer real-time with adaptive instance normal- ization. In IEEE International Conference Computer Vision, 2017.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to tomageg translation with con- ditional adversarial networks. In IEEE Conference on Computer Vision and Putlern Recognition, 2017. 18
Nikolay Jetchev, Urs Bergmann, and Gokhan Yildirim. Copy the old paint anew? an adversar- ial framework for (non-) parametric image stylization. Neural Information Processing Systems Workshops, 2018. 41
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. 1,2
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In IEEE Conference on Computer Vision and Pattern Recognition, 2020. 3, 4. 5, 33
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. 33
Hsin-Ying Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Kumar Singh, and Ming- Yang. Drit++: Diverse image-to-image translation viadisentangled representations. International Journal of Computer Vision, pp. 1-16, 2020. 4
Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong Chen. COCO-GAN: Generation by parts via conditional coordinating. In IEEE International Conference on Computer Vision, 2019.
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In IEEE International Conference on Computer Vision, 2021. 3
Guilin Liu, Fitsum Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In European Conference on Com- puter Vision, 2018a.
Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. In Neural Information Processing Systems, 2018b. 4
Chia-Ni Lu, Ya-Chu Chang, and Wei-Chen Chiu. Bridging the visual gap: Wide-range image blend- ing. In IEEE Conference Computer Vision and Pattern Recognition, 2021. 9
Qi Mao, Hsin-Ying Lee, Hung-Ye Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking gener- ative adversarial networks for diverse image synthesis. In JEEE Conference Computer Vision and Pattern Recognition, 2019. 4
Lars Mescheder, Andreas Geiger. and Sebastian Nowozin. Which training methods for gans do actually converge? In International Conference on Machine Learning, 2018. 1,
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3

---

Published as a conference paper at ICLR 2022
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, 2020.
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Struc- ture guided image inpainting using edge prediction. In IEEE International Conference on Com- puter Vision Workshops, 2019. 3
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In IEEE Conference on Computer Vision and Pattern Recognition, 2021. 3
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier GANs. In International Conference on Machine Learning, 2017.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, 2016. 2
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In JEEE Con- ference on Computer Vision and Pattern Recognition, 2019.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAB-2. In Neural Information Processing Systems, 2019. 2
Mark Sabini and Gili Rusak. Painting outside the box: Image outpainting with gans, arXiv preprint arXiv:1808.000 1808,0883, 2018. 3
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from a
single natural image. In IEEE International Conference on Computer Vision, 2019. 2
Assaf Shocher, Shai Bagon. Phillip Isola, and Michal Irani. Ingan: Capturing and retargeting the"
dna" of a natural image. In IEEE International Conference on Computer Vision, 2019. 2
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Neural Information Processing Systems, 2020. 4
Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. In IEEE International Conference on Computer Vision, 2021. 2,
15
Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Kei., Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let net- works learn high frequency functions low dimensional domains. In Neural Information Pro- cessing Systems, 2020. 4
Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. In IEEE International Conference Computer Vision, 2019.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng. Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator what you need for high-resolution video synthesis. In International Conference Learning Representations, 2021. 9
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
Jonas Wulff and Antonio Torralba. Improving inversion and generation diversity in stylegan using a gaussianized latent space. arXiv preprint arXiv:2009.06:: 006699999999......... 2020. 33
Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj, Jingwan Lu, Chen Fang. Fisher Yu, and James Hays. Texturegan: Controlling deep image synthesis with texture patches. In IEEE Conference Computer Vision and Pattern Recognition, 2018. 2
12

---

Published as a conference paper at ICLR 2022
Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, and Chen Change Loy. Positional encoding spatial inductive bias in gans, In IEEE Conference on Computer Vision and Pattem Recognition, 2021. 16
Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. In IEEE International Conference on Computer Vision, 2019. 3,
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. IEEE International Conference on Computer Vision, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 33
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.6
13

---

Appendix Table of Contents
A Comparisons with Concurrent Work
15
Conceptual Comparisons Among Different Tasks...
17
Implementation Details of Coordinates..
17
D InfinityGAN Architecture
20
E More Comparison with baselines...
21
More InfinityGAN Qualitative Results
22
InfinityGAN Results At Higher Resolution.
25
H More Results on Other Datasets..
29
I More Diversity Visualization
31
J Our Best Attempt in Including An Everlasting Image
32
K Implementation Details of Image Outpainting and Inhetweening via Inversion
32
L Implementation Details of Spatial Style Fusion
34
M More Qualitative Results of Outpainting via Inversion
36
More Qualitative Results of Inbetweening via Inversion
37
More Qualitative Results of Cyelic Panoramic Inhetweening via Inversion.
38
P Experimental Details of The Speed Benchmark with Parallel Batching
39
Ablation: Feature Unfolding
39
R Limitations.......
40
Implementation Illustration of ScaleInv FID
40
T More Comparisons with Texture Synthesis Method
U More Comparisons with SinGAN-based Models.
41
Ablation: Mode-Secking Diversity Loss (Caiv)
43
W Ablation: Auxiliary Loss (Car)
14

---

Published as a conference paper at ICLR 2022
A COMPARISONS WITH CONCURRENT WORK
A.1 ALIS (SKOROKIOODOY ET AL., 2021)
ALIS concurrent work that achieves a similar application in infinite-pixel generation by itera- tively inbetween pairs of anchor patches. Here, we discuss some of the critical differences.
ALIS has limited ability in extending vertically. InfinityGAN achieves both vertical and hori- zontal extension while maintaining a plausible holistic appearance. ALIS only presents horizontal extension in the paper. Vertically connecting anchors on any dataset used their paper will pro- duce invalid structures (c.g., layered landscapes periodically stacking in the sky). Therefore, vertical anchor connection is limited to certain datasets, such as pattern-like textures or satellite images.
Differences in the problem formulation. InfinityGAN directly models each infinite-pixel image with shared global latent variable using an implicit funetion. In contrast, ALIS learns to inbetween two independent global latent variables. Furthermore, InfinityGAN can still achieve the inbetween- ing setup similar to ALIS with spatial style fusion. However, applying ALIS to synthesize images with shared global context will lead to periodically repeating patches, shown in Figure 13.
InfinityGAN allows free-form anchor placements without training. ALIS has to designate a constant relative position between the anchors before training starts. It is also non-trivial for ALIS to inbetween multiple anchors. Meanwhile, as shown in Figure 12, our training-free spatial style fusion allows placing any number of anchors any place. However, the flexibility also comes with a trade-off, our spatial style fusion not trained with adversarial learning, causing the synthesis performance not compatible the regularly synthesized images. Following the ALIS evaluation protocol, we train our InfinityGAN on the ALIS LHQ dataset at 256 256 resolution, yielding an FID 11.82 with regular synthesis and an 3o-FID 19.22 with spatial style fusion. In contrast, ALIS shows negligible performance gap between FID (10.48) and oo-FID (10.64). Such a result suggest a flexibility accurracy trade-off between InfinityGAN and ALIS.
ALIS generates blocky and high-frequency lattice artifacts. ALIS adopts the generation-be -arioon----y--- by-parts design from COCO-GAN. As shown in Figure 11, a critical consequence is creating the blocky and lattice artifacts between patches, since the inter-patch continuity is unstably maintained with adversarial learning. In contrast, InfinityGAN is an improved version COCO-GAN that the inter- patch continuity is guaranteed with implicit-functinn and padding-free generator design.
ALIS still suffers from content repetition. The content repetition problem is discussed in the ALIS paper as a limitation. InfinityGAN addresses the issue with a local latent space and enforces the contribution of the local variables with a diversity loss (2) between Z and Zg. We demonstrate that InfinityGAN does not have the content repetition problem in Figure 31.
Blocky Artifacts
Discontinuity
Lattice Artifacts
Figure 11: ALIS can suffer from blocky artifacts, inter-patch discontinuity and lattice artifacts. We train ALIS with the official implementation 1024x 1024 resolution. We focus on the failure cases caused by COCO-GAN A--0-GAN-baaced generation-by by-prtsss framework, which artificially enforces the learned inter-patch continuity with adversarial learning.
15

---

Published as a conference paper at ICLR 2022
InfinityGAN with free-form anchor placements. The spatial style fusion of Infinity- GAN place any number of style centers (called anchors in ALIS) at any location. We show (top) a five-anchor example and (bottom) the corresponding regions covered by each style center.
Figure 12:
Figure 13: ALIS cannot synthesis with a single holistic appearance. ALIS synthesizes repetitive content if all anchors share the same latent vector.
A.2 MS-PIE (XU ET AL., 2021).
MS-PIE found the positional information is crucial for GANs training. The paper explores different positional encoding schema, including sinusoidal coordinate encoding and padding removal. We further discuss the relation and distinction between MS-PIE and InfinityGAN in these modules.
Extensible coordinate encoding. The "expand" configuration of MS-PIE allows coordinate value extrapolation in the spatial dimension. However, its training sticks to fixed coordinate matrix for each synthesis scale, and the framework aligns the real images to the fixed coordinate matrix with bilinear interpolation during training. Such a design causes the synthesized content attach the coordinate matrix, thus inevitably creates repetitive content when the coordinate value periodically repeats as the matrix expanding. In Figure 14, we train no-padding StyleGAN2 model with MS- PIE-expand setting (i.e., config (k) in Table 5 of MS-PIE paper) on our Flickr-Landscape dataset. The model is trained at scales 256, 384 and 512, then test at scale 1408.
https: / /ttgtttttbbb/////// com/ open- mmlch /mgonaarttoo///lll.. 96746488481 72c01 34c4591e96 3de36 imgee is/ gans//eppteeeee e_sty leganZ. py41.1 5.
The corresponding
official
implementattin::
16

---

Published as a conference paper at ICLR 2022
Figure 14: MS-PIE creates repetitive content in the expand setting. We train the no-padding StyleGAN2 with MS-PIE at 256, 384 and 512 scales, then synthesize at 1408 scale.
Distinctions between no-padding and padding-frre freee generator. The no-padding generator (NPG) in MS-PIE is conceptually related to our padding- free generator (PFG), but fundamentally different while considering how the information of border pixels in the feature space is processed. NPG grad- ually loses the information from border pixels after each convolution, since the border pixels are less visited than the other pixels (consider 3x3 convolutional kemel, most pixels will be scanned for 9 times, while edge pixels will be only scanned for 4 times, and corner pixels only time). Further note that the information loss worsens exponentially as the network stacking more convolutional layers. In contrast, our PFG pads feature values from the neighbor context before the no-padding convolution applied. The information loss caused by the no- po-padding convolution is natural way to discard the information that is spatially too far away from and less related to the current context.
B CONCEPTUAL COMPARISONS AMONG DIFFERENT TASKS
(b) Texture Synthesis
Task
(a) Super Resolation
(includes
(c) Image Extrapolation
(d) COCO-GAN
(e) InfinityGAN
Input
Condition
None
None
None
Training-Dati
Distribution
Further Extended Resolution
(Orly frw srpp))
Figure 15: (a) Super Resolution: The final outputs inherit the coarse structure from and share the same field-of-view with the original input condition. (b) Texture Synthesis: Due to coordinate en- coding, objects are generated near image the border, and the center the image is filled with repet- itive textures. (c) Image Extrapolation: Current extrapolation models tend to hooyyand-paraphasss the conditional imput or create mottled textural samples, leading to repetitive results especially when the outpainted region is large. (d) COCO-GAN: COCO-GAN can only synthesize samples slightly larger than its training distribution. (e) InfinityGAN: Ours InfinityGAN can synthesize a more favorable global structure arbitrary resolutions without input condition.
C IMPLEMENTATION DETAILS OF COORDINATES
We first derive the receptive field size R of an L-layer Gg after adding the 7x7 feature unfolding to all layers. Assume that the size of zş (i.e., output Gg. and input of G.) is M. The value R can be derived as M + (2x3) TT, where is the half-size of the feature unfolding area. In practice, with
17

---

Published as a conference paper at ICLR 2022
an architecture shown in Figure 18 with training patch size 101 x 101, we have M 11, L 4, and R = 35.
Horizontal direction. In order to avoid the generator discovering and exploiting any property of the periodic coordinate, we use period T much larger than R. Mcanwhile, the period should be short enough to avoid the differences between consecutive coordinates vanish to almost zero. In practice, we use T 4x R for both of the cosine and sine coordinates.
Vertical direction. We utilize the property of the tanh function that its slope rapidly saturates to nearly zero and its value range is bounded by [-1, 1]. Such property is aligned to the property of real-world scenery- the structural interactions among landscape objects are mostly concentrated around the horizon and rapidly saturates to single modal (i.c., sky, ground, or water) in the vertical direction.
There are two hyperparameters while constructing the tanh coordinates: (i) a pair of cutoff points, and (ii) the sampling period of z in the spatial dimension.
Cutoff points (Cent) are a pair of values that, during training, we only sample the coor- dinates between the value pair. Such a hyperparameter required since we are unable to sample all coordinates from infinitely large coordinate system within the finite training steps. In practice, we set the cutoff points at +00995 of tanh-projected coordinate. Note that the underlying effect of using different cutoff point values is not well-investigated, we do not observe obvious changes with slightly different values. Sampling period (d) defines the distance between two spatially consecutive values of z are sampled within a training sample. Such a value relates to the grain the representation that the generator overall models. It is equivalent to the occupation ratio of z between the cutoff points. Thus we can alternatively define a hyperparameter V that defines the occupation ratio of 2i by R+V. Then, the sampling period can be derived with di
2 x Ceut X R+V. In practice, we use = 10, resulting in dj = 0.13.
1
C.1 MORE DISCUSSIONS ON THE CHOICE OF COORDINATE SYSTEM
In this paper, we mainly tackle the scenery image datasets, such as landscape, LSUN bridge and LSUN tower. The coordinate prior we introduced in Section 3.2 specifically designed for such types of data, which has self-similarity in the horizontal direction and rapid mode-saturatio in the vertical direction. However, we want to emphasize that the choice of coordinate system is hyper- parameter depending on the dataset. For instance, in Figure 16, we show that InfinityGAN can also work on satellite image dataset (Isola et al., 2017), which is more frequently used in texture synthe- sis task. A more natural choice of coordinate system in such setting using periodic coordinates in both horizontal and vertical directions. Nevertheless, in Figure 17 we observe that InfinityGAN can still produce visually plausible and globally sound appearances with using saturating (i.e., tanh) coordinates in the vertical direction and periodic coordinates in the horizontal direction.
In Table 4, identify three categories of coordinate systems, but these options may not have covered all possibilities. One may need to be aware of the characteristics of the data and the intended outcome before selecting the appropriate coordinate system.
Dataset Type
Object-centric
Texture
Scenery
Example
ImageNet, CelebA
Texture, satellite images
Landscape
Spatial Distribution of Content
(Vertical) Spatially varying
N/A
Spatially agmostic
(Horizontal) Spatially agnostic
Horizontal Coordinate
Constant
Periodic (e.g.. sin/cos)
Periodic (e.g.. sin/cos)
Vertical Coordinate
Constant
Periodic (e.g., sin/cos)
Saturate (e.g., tanh)
Table 4: Possible coordinate designs for different types of dataset. The choice of coordinate
system for InfinityGAN is dataset-dependent hyperparameter.
18

---

Published as a conference paper at ICLR 2022
Real Images
Generated (1x)
Generated (2x)
Generated (4xt
Figure 16: Qualitative results on satellite image dataset. InfinityGAN trained on satellite image
dataset with cyclic coordinates on both vertical and horizontal directions.
Real Images
Genersted (1x)
Generated (2x)
Generated (4x)
Figure 17: Qualitative results satellite image dataset. InfinityGAN trained on satellite im- age dataset with saturating coordinates (i.e., tanh) on vertical direction and cyclic coordinates on horizontal direction.
19

---

Published as a conference paper at ICLR 2022
D INFINITYGAN ARCHITECTURE
Generator
Discriminator
Stradtre SystestriiC
Trstrn SyriesooriGy
(aape Bel12)
5,ttape 33
codape: Belo35.00)
Scléaçe
2,0ip Bed12i
Magpeng Nanwaks
Cawal
Torme: 3e3. paldlag
Builbonde
Soekdan 23
Spheitmaaaltled
Di
Boolennd Mooktmsfooontteen pastigg
2 Butthonds
Sphhitasbaitlad publeg Falas)
DReddheffiin ock
parling
DiboskaadEackk
Janai stide paddeg
pestin
Garrd: Sod. palling
Dilesdaddf oct
[hamel 707 ordee 1 paddig Paiee"
lkatet 243 pubdic: File))
(bemel 360l, paldiig: Trie andl at ch Triel
Garri: fnd. palling Trac)
Lioaz Leao BALL
2- Adbenaral Frddecie
Outpot
P. Vaaae Bo2el0bet0))
Figure 18: A high-level overview of InfinityGAN model architecture.
Modak Name
MedibateeCeenorrra (kanc) T.2, pabding: Fdlo)
Stpedheddealtick pallig
DResdtaBBlck poddagg
(Soope Contgg
Daput
Saape
(dage: BeluBaM thage BxPoBlWW
3, or Saaaety
Istpe Bs5121
3 ot fattaty tiage Bubilili
200 (Sage: B47.lB))
oc fostu Iaage BxPs55W1
Lice
Weighs
Caerdd
Weigat Modicates
Donsddactiie
Coeoseain D
Coerid tadyytet2
Conrid Weights
Canvial a Lado Hall
Creedd
(Madiiaadd
(siske 2t
(drske 31
Outpet
1dape (oG)
tdoge Tx0y23--3EMW-3
sus
sca
Medule Name Cuxcgi
Stylid ooy
StyldtSipp [bere) 36, pabking Trlee)
(hemed 3ol, podding Polo)
Ikzzd 3cl, pubdap Faks))
Inpat
Idupe hv4171
fodam, telpee Tabslton)
odupe hus12)
Serrr, Johpe BuFoliok)
jahae huts7)
nr twitim (xhqe ftuillllli
sdps
Lbarr
Ocar Tanspoodd
Weigla
1hser
Cocr2d
Wepte
Liner
Ccar2d Wagta
Weight
&Dmnoditiinn
herecubetn
Modabtten
Weights (Mialeed)
Obtte 20
Cocr21 Vede INtdilant
Cocd25
Cuar2l
(Hedazand
Cervid
lpperplgg
(betzd 3:)
Laaiy/aLU
LedohaLU
LoakytoU
Dotpst
Baage B.0 SWE
Gtape: B:0xc 20
30 (Sage: BvOx 21:W
Figure 19: The low-level design of each module within InfinityGAN
20

---

Published as a conference paper at ICLR 2022
E
MORE COMPARISON WITH BASELINES
SinGAN
StyleGAN2 NCI
StyleGAN2 + NCI + FCG
InfinityGAN (ours)
Figure 20: More qualitative comparisons. We show more samples Flick--Landseag
x024x 1024 pixels.
21

---

Published as a conference paper at ICLR 2022
F MORE INFINITYGAN QUALITATIVE RESULTS
More qualitative results. We provide more images synthesized at 1024x 1024 pixels with our InfinityGAN trained Flickr-Landscape. All images synthesized with the same model presented in the paper, which is trained with 101 x 101 patches cropped from 197x 197 resolution real images. All images share the same coordinate and present high structural diversity. Note that the images are down-sampled 2 to reduce file size.
Figure 21:
22

---

Published as a conference paper at ICLR 2022
Figure 22: More qualitative results. We provide more images synthesized at 1024 1024 resolution with our InfinityGAN trained on Plickr-Landscape.. All images are synthesized with the same model presented in the paper, which is trained with 101 x 101 resolution patches cropped from 197x 197 resolution real images. All images share the same coordinate and present high structural diversity. Note that the images are down-sampled 2x to reduce file size.
23

---

Published as a conference paper at ICLR 2022
Figure 23: More qualitative results. We provide more images synthesized 1024 1024 resolution
with our InfinityGAN trained on Plickr-Landdsape.. All images are synthesized with the same model presented in the paper, which is trained with 101 x 101 resolution patches cropped from 197x x 197 resolution real images. All images share the same coordinate and present high structural diversity. Note that the images are down-sampled 2x to reduce file size.
24

---

Published as a conference paper at ICLR 2022
G INFINITYGAN RESULTS AT A HIGHER RESOLUTION
Figure 24: InfinityGAN samples training at a higher resolution. We synthesize 4096 pixel images using InfinityGAN trained Flickr-Laddseape at 397x397 pixels patches cropped from 773 773 full images. The top and bottom rows are zoom-in view of the image. Note that the figure is 2x down-sampled reduce file size.
25

---

Published as a conference paper at ICLR 2022
Figure 25: InfinityGAN samples training a higher resolution. We synthesize 4096x pixel images using InfinityGAN trained on Flickr-Laddscape at 397x397 pixels patches cropped from 773x773 full images. The top and bottom rows are zoom-in view of the image. Note that the figure is 4x down-sampled reduce file size.
26

---

Published as a conference paper at ICLR 2022
Figure 26: InfinityGAN samples training a higher resolution. We synthesize 4096xx096 pixel images using InfinityGAN trained on Flickr-Ladsscape at 397x397 pixels patches cropped from 773x773 full images. The top and bottom rows are zoom-in view of the image. Note that the figure is 4x down-sampled reduce file size.
27

---

Published as a conference paper at ICLR 2022
Figure 27: InfinityGAN samples training a higher resolution. We synthesize 4096 4096 pixel images using InfinityGAN trained on Flickr-Ladssaape at 397x397 pixels patches cropped from 773x773 full images. The top and bottom rows are zoom-in view of the image. Note that the figure is 4x down-sampled to reduce file size.
28

---

Published as a conference paper at ICLR 2022
H MORE RESULTS ON OTHER DATASETS
Training Size (101x101)
Test Size (512x512)
Figure 28: LSUN bridge category. InfinityGAN synthesis results at 512x512 pixels on LSUN bridge category. The model is trained with 101 x x001 pixels patches cropped from 197 197 resolu- tion real images.
29

---

Published as a conference paper at ICLR 2022
Training Size (101x101)
Test Size (512x512)
Figure 29: LSUN tower category. InfinityGAN synthesis results 512x512 pixels on LSUN tower category. The model is trained with 101 x 101 pixels patches cropped from 197 197 resolution real images.
30

---

Published as a conference paper at ICLR 2022
MORE DIVERSITY VISUALIZATION 1
Ditterent styles
Different local latent variables
Figure 30: Generation diversity. We show that structure synthesizer and texture synthesizer sepa- rately models structure and texture by changing either the local latent or style while all other vari- ables are fixed. The results also show that InfinityGAN can synthesize a diverse set of landscape structures at the same coordinate. All samples are synthesized at 389x389 pixels with InfinityGAN trained at 101 x 101.
31

---

Published as a conference paper at ICLR 2022
J OUR BEST ATTEMPT IN INCLUDING AN EVERLASTING IMAGE
Figure 31: We provide a 256x9984 pixels sample synthesized with InfinityGAN, The sample shows that (a) our InfinityGAN can generalize arbitrarily- sizes, and (b) the synthesized contents do not self-repeat while using the sample global latent variable Zg.
K IMPLEMENTATION DETAILS OF IMAGE OUTPAINTING AND INBETWEENING VIA INVERSION
Our pipeline similar to that of InOut (Cheng et al., 2021). With a given image æ, the objective of GAN-model inversion is to recover set of generator-parantee paraaaeee--eeeeeeeee input latent variables z* that can synthesize a resulting image x² that is similar to r. There exist multiple different imple- mentations to recover 2", adopt the cradient-descent-ae basee method, which optimizes 2 set of learnable parameters with carefully designed objective functions. In the context of InfinityGAN, the W+ space
optimize four groups of variables: Zg. 21, style (ZT) and 2n- In particular,
32

---

Published as a conference paper at ICLR 2022
formulation (Abdal et al., 2020; Wulff & Torralba, 2020) for zr, named z where each layer in Gr has its own set of 2T² and each of the zr" is optimized separately.
Objective fuunctions We first introduce two image-distance losses to the inversion objectives:
pix 2 - x 22
ppeccept LPIPS (z 2
which LPIPS is the learned perceptual distance proposed by Zhang al. (Zhang et al., 2018).
We utilize the Gaussianized latent space technique proposed by Wulff and Torralha (Wulff & Torralba, 2020), which uses LeakyReLU of slope to discount the last activation function (a LeakyReL.U with a slope 0.2) in the StyleGAN2 mapping layer, and recovers Gaussian-like marginal distribution of z7. With the Gaussian distribution prior, we can use the empirical mean vi and covariance matrix E/ (computed by sampling 10,000 zit via Zy) recover an estimated zip distri- bution. With the empirical statistics, (Wulff & Torralba, 2020) proposes to compute Mahalanobis distance:
(as H4 Σ) (= 11)² E (Zi p.)
(8)
Then, we construct a prior loss Sprior (Wulff Torralba, 2020) that regularizes the Zg. Z and z with Mahalanobis distance:
ppior = Aad 0,1) + (2, 0.I) + N, pT E
which Aas Ag and X, are weight factors. For the cases of zy and z with zero means and unit variances, the prior loss degenerates to an l2 loss.
Following StyleGAN2, we adopt a noise regularization loss fareg and noise renormalization. The full objective function of the inversion is:
Livv = Apix + Aperoept Sperceps + Sprior + Anregfarez
(10)
where As are the weighting factor of each loss terms.
Hyperparameters. For all tasks and datasets, we set Apix = 10, Apercept = 10, Anreg = 1,000, Xa 10, Ag 10, and X, 0.01. We use Adam (Kingma & Ba, 2015) optimizer with learning annealing (Karras et al., 2020) from 0.1 to for 1000 iterations. We use a batch size of to avoid batch samples interfere with each other. Despite we observe batched inversion can sometimes yield superior results, it is not a conventional setting for real-world applications that batched inputs are mostly unavailable, it also significantly increases the stochasticity while reproducing the results.
Outpainting and inbetweening with inverted latent variables. With the inverted latent variables, we perform image outpainting by spatially extend the z with its unit Gaussian prior, while using 2 and x everywhere. For image inbetweening, 2 also extended with its unit Gaussian prior, while z and xt" are fused with spatial style fusion. Notice that we do not optimize in our pipeline, since inverting is non-trivial and requires additional regularization losses. This introduces a limitation that the spatial position of the images is fixed after the inversion, the users have to perform the inversion optimization again if they want to assign z a different location.
Another limitation is that, despite the use of the prior loss, some dimensions of the inverted latents tend to drift far away from the normal distribution. In combination with the use of the W+ space of Zr, the inverted latents are of high-instability, sometimes introduce checkerboard-like artifacts, and frequently mix multiple irrelevant contexts together. We develop an interactive tool (released along with the code release) that allows users interactively and regionally resampling the undesired local latent variables z in the outpainting area. Such limitation is highly related to the generalization of the inverted latents, put it as important future working direction.
33

---

Published as a conference paper at ICLR 2022
L IMPLEMENTATION DETAILS OF SPATIAL STYLE FUSION
(a) Image Inference (Single Style)
(b) Fusion Map Calibration
(c) Spatial Style Fusion Generation
Yodtamn Pack
Latest
Courdinate
Dalles
Example 3x3 Clonidd
Trasspose
Exammpl 3xl Com/2d
SaleT)
Image Space
Figure 32: Illustration of fusion map creation procedure and spatial fusion generation. With a toy architecture example shown in (a) and a style fusion map in the pixel space (bottom of (b)), can reversely create spatially aligned fusion maps in all intermediate layers by padding or terpolating the fusion map in the previous layer. Spatial style fusion in (c) uses the fusion maps to synthesize images with a natural style transition in the pixel space.
To achieve spatial style fusion in the existing InfinityGAN pipeline, we introduce two additional procedures: "style fusion map creation" and "fused mnduulaioondemmodulationn The former creates per-layer style fusion maps that specify the geometry of the style fusion area. The latter one is that processes the volumetric styles created
modified version of feature from the style fusion map.
Style fusion map creation. Given N style centers designated by the user, in the pixel space, the target of style fusion map creation is to construct a set of style fusion maps for each layer of both Gs and GT. The fusion map is a spatially-shaped (i.c., batch N X x W) tensor with N channels that specifies the weight of the style for each spatial location, which the weights sums up to one across the N dimension for each spatial position.
We first construct an initial fusion map in the pixel space by finding the spatially nearest style center, then assign a one-hot label for each spatial position in the initial fusion map. Since the spatial style fusion happens in all layers in the generator, from the output of the generator to its input, we call such a procedure fusion map calibration. We show illustration of fusion map calibration in Figure 32(b). The fusion map calibration starts from the image space and sequentially For each pair (output-side and input-side) of the fusion maps in network layer, we match the spatial dimension of the fusion map pair by padding or interpolating the output fusion map into spatially aligned input fusion map. For different types of intermediate layers, the underlying implementation of the fusion map calibration can be slightly different, but shared principle is to maintain a consistent geometrical position of the style fusion center throughout the generator.
therefore reversely propagate the fusion map
the fusion maps for all generator layers.
In practice, such binary map creates a sharp style transition that produces visible straight lines dividing the style regions. Accordingly, we apply a mean filter that smooths the style transition border. While different kernel sizes for the mean filter only alter the range of style transition and the visual smoothness, we use kernel size of 127 our experiments as it empirically produces good visual results.
Fused modulation-demooullatue After constructing the per-layer style fusion map, we can use the fusion maps to create volumetric styles (i.c., batch Dx H W' by weighted-sum the styles by the importance weights (D-channel dimension) in each spatial position. The volumetric styles are applied to each layer of the generator. As the feature StyleGAN2 and InfinityGAN is pixel-wise operator, we can easily adapt it to volumetric styles. We demonstrate possible implementation of the fused modulation-ddmmodllttooo in Figure 33.
strategy used in both
2The forward function is hased on the implementation from https:://g http:///ttttta/ee thub. com/ cooomrrrsing lity/ pytorch.
34

---

Published as a conference paper at ICLR 2022
egort torch.an, an, Fundt ex
Import tordh.rn funct Fonal es
de forxard (self, feature, style)
cet rused. forwerd (selt, teature, stylet:
Deabure: Fealure with shape (B, CI, H, W]
fealuze: Peature with shaps 18, Cl, 3, M) style Fesion style with shage (8, C2, H, exn
G
style Single style wich shape (3, 62)
7
8
batel, inge inghy
Eealure,
batch, inLe, st o style shapel
featase shape
II
11
12
Hyperparenen
Hyperporameters
Penv asre
out s-lf. o rmppad k // 2
Eapeot ed outpus channel beco-peoding renoval
out.: a=1f. ou Expect ed a dutipl charnel toyad & //
15
+ zero-padoing removal
15
16
acaling (aryleaae y1eGN22)
scaling 181y1-GA12)
18
Shope:
Snape:
19
Il, ) loat iy on by ky k] lest_e,
O I (ost or in k, kl
19
20
Neight selt scale selt weigat
veight seale weight
23
+ Neight nomistiin (GtyLeGAN2)
Keight modelation Icasted; tol innfiiafeteee,
23
24
style mode lation (style)
24
style style. tberch, Shape:
sewel git etty exeeood)
corviin tattrr arwelgati daned
26
27
Bl out o, in by ky k) lostchy 1y in Cy 1,
Style eyyle gees the torner ore for speed,
27
lestch, Neigat weight ensquetze style
inue,
J, resaspe st_c) style selt nocu (ntynl)tyle)
style permate
31
style stylc sisvibetch, ia_h, 1a.s, in H, Keight demodulation (eproximated) atatistios.
31
32
style sty.e. pernute (sty
32
35
Meiigtt demodr on (SCA12 cenod
35
36
Feature cenedol laciiin approxination here similar to near of statistics trom all styles.
36
weight. (2) 22, weigat demod, (batch, out_o,, 1, 1,
39
denod tozch. zcros (bazca, out_e, in_h, iagxt
39
J in sarge(in_
style_y stylelt, i i, 31
43
viewibatch,, 3, in_e,
43
44
style_y weight unsqueeze 101 style_y
44
I seyle_y shapes IE, eat_sh, 1a &,
46
denocl Fo t, 11 tarch zoqet
47
47
stylc_v (2) 31,
50
Convelution
Convolution
50
51
festure frature, xl2v(1, batch-in_s, ingh, ingal upsarple:
(A11 fecture uses same veicht, 3o aoed to group)
51
it selt
52
weight vaiche. view (batch, out_e, 1ago, 2. weight veight, transpose [1, reshapel k, k) soooo_ttrnnpoaa2dt festurs, weight, Padding-C, stside-,,
waighc weight viawiout__, ings, k.
woizat wei (S, 1) coariguuus (1
56
batch
ca
55
foasure, weicht.
pasding Stride 2, groups
3W
batel)
58
59
59
Clipping paoding (Ceavt special case) outli, is empad: iupad, empad: smpad]
clipping padding IcorvT special but out Iut f, impad: Enpad, supad: rmaad]
62
4 hate demiiaaal lon (natich ou shape) danod P.eetespolace
63
denod,
66
zize- (auL. shape( 21, sat dlape 11), mode mode-"bille "Lilineas" " clien_carrars True)
66
67
67
70
out self. blut (out) StyleOMM Gaussian bluz
ouc sel2 blus (out) BEy1edAR2 Gausalan bluz
70
71
clss:
eise:
71
weight veagh.. Bautcch inge, k, conv2d1
F.uaav21
Zeattre, weight, paddlnc- aroupa denoi dened :, empads rmged, repads rmped. denad
74
feel are, wright, parkling-O, groupe-batch)
74
75
Recovos batch-charnel sh, out_x mut shape
out out. pont taguux
78
oat eat. viww (batch, ous_e, b, eat_x)
78
79
79
Figure 33: Implementation of spatial style fusion. We present (left) the original StyleGAN2 forward function, and (right) corresponding implementation for the spatial style fusion. We align the related code blocks on the left and right.
35

---

Published as a conference paper at ICLR 2022
M MORE QUALITATIVE RESULTS OF OUTPAINTING VIA INVERSION
Figure 34: More outpainting via model inversion. We present more outpainting results from InfinityGAN on Slickr-Scenery. We invert the latent variables from 256x 128 pixels real images (marked with red box), then outpaint 256x640 area (5x real image size).
36

---

Published as a conference paper at ICLR 2022
N
MORE QUALITATIVE RESULTS OF INBETWEENING VIA INVERSION
Figure 35: More image inbetweening with model inversion. By inverting the latent variables
that reconstruct the two real images two sides (marked with red box), InfinityGAN can natu- rally inbetween the two images arbitrarily distant away. We synthesize the 256x 1280 images using InfinityGAN trained on Flickr-Scenery 101 x 101 pixels.
37

---

Published as a conference paper at ICLR 2022
0 MORE QUALITATIVE RESULTS OF CYCLIC PANORAMIC INBETWEENING VIA INVERSION
Figure 36: More cyclic panorama synthesized with model inversion. By setting the same real image on two sides (marked with red box) and inverting the latent variables that reconstruct the real image, InfinityGAN can naturally synthesize horizontally cyclic panoramic images with image inbetweening. We synthesize the 256x 220 images using InfinityGAN trained on Flickr-Scenery at 101 101 pixels.
38

---

Published as a conference paper at ICLR 2022
P EXPERIMENTAL DETAILS OF THE SPEED BENCHMARK WITH PARALLEL BATCHING
We perform all the experiments workstation with Intel Xeon CPU (E5-2650 2.20GHz) and GTX 2080Ti GPUs. We implement our framework with Pytorch 1.6, and execute an environment with Nvidia driver version 440.44, cuDNN version 4.6.5, and Cuda version 10.2.89.
We report the summation of pure GPU execution time and data scatter-collection time introduced by data parallelism. The model synthesizes single image for each trial. We first warm up the GPUs with 10 proceeding trials without recording their statistics, then compute the mean and variance over 100 trials. The numbers are reported in Table
Table 5: Inference speed up with parallel batching, Benefit from the spatial independent gener- ation nature, InfinityGAN achieves up 7.20x inference speed up by with parallel batching. We conduct all experiments at batch size of 1, and OOM indicates out-of-memory. Note that the GPU time here accounts for pure GPU execution time and (if applicable) data-pane data-parallel ceatter-aggregation time.
Generation Paradigm
Paraliel
Time Inference Size (scc/image)
Speed Up
MFLOPs
Method
GPUs
Batch Size
1024 1024
2048x2044
4096 4096
8192x8192
8192
1024x 1024
StyleGAN2
One-Shot
1
0.60 1: 0.01
OOM
OOM
OOM
6,642
One-Shot
0.67 + 0.01
0OM
0OM
00M
6,815
1
1.24 1: 0.15
7.96
0.17
34.35
1.69
137.44 1.85
x1.00
2
1.58 1: 0.09
5.31
1: 0.13
24.13
0.42
0.42
95.77 = 1.63
x1.44
InfinityGAN (Ours)
1.35
5.20
0.02
20.93
82.52
0.08
x..67
Spatially Independent
8
1.28 1: 0.01
5.14
11.41:00 0.02
19.63
0.002
78.41 1: 0.17
x1.75
17,901
Generation
1.23 +
0.01
19.11
0.02
x1.80
32
2
0.96 + 0.01
3.90
0.06
59.33 = 0.15
x2.32
64
0.56 4 0.01 2.25
8.64
0.11
35.20
0.39
X3.90
128
0.32 + 0.05
1.30
0.05
4.82 0.06
19.09 = 0.16
X7.20
Q ABLATION: FEATURE UNFOLDING
Figure 37: We plot the FID curve for a training episode of our complete InfinityGAN (red curve) and InfinityGAN without feature unfolding (blue curve). We observe the FID saturates at early stage without feature unfolding.
39

---

Published as a conference paper at ICLR 2022
R
LIMITATIONS
Here we discuss some empirical limitations of InfinityGAN.
Patch-training leads to performance degradation. In order construct better conditional dis- tribution in the vertical direction with an auxiliary loss Car, InfinityGAN trains the generator with patches instead of full images. However, training with patches reduces the training field-of-view, thus leading to an inferior performance at the same field-of-view compared to model trained with full images. Designing unsupervised mechanisms in learning Car without patch-cropping can im- prove InfinityGAN performance.
Long-range coherence. InfinityGAN assumes local coherence following shared holistic appear- ance can achieve visually plausible synthesis. However, certain physical relationships still require long-range dependency, such as we can observe two suns in the top-left image Figure 23, and twilight should only happen near the horizon in Figure 25. Despite InfinityGAN can independently sample pixels arbitrarily distant away, it remains unclear how to construct unsupervised losses for such conditions, only have finite-ixel images in training data.
Strip-shaped artifacts. We observe InfinityGAN creates a unique type of artifact that forms a strip-shaped structure sweeping through the sky ground for long distance, such the clouds in the bottom-two image Figure 23. We hypothesize the root cause of such an artifact is that the model attends too much to the strong structural characteristics of the horizon line and accidentally shares the representation with other less related contexts. We anticipate improving the modulation coordinate encoding mechanisms may help suppress such behavior.
IMPLEMENTATION ILLUSTRATION OF SCALEINV FID
import torch. on. functions as
def eval_scaleinv_ (real imgs, fake_ scale)
seal imgs: tensor of real images, shape (B, C, H, X0
$ Take_imgst tensor of fake images, shape (B, C, H, W)
$ scale: the scale of the current Scale F FID. Lake_imagen
P. interpolated
fake images,
scale scale_factor-e wac.to-toctcacttao
10
mode= "iil-""biirea"" align_corners-TTHe True)
The regular FID evaluation
return eval_ frea images, take_images]
40

---

Published as a conference paper at ICLR 2022
T MORE COMPARISONS WITH TEXTURE SYNTHESIS METHOD
As we discussed in Section that texture synthesis models are not directly applicable to real-world image synthesis. In Figure 38, demonstrate such a problem by running TileGAN (Frühstück et al., 2019) on our Flickr-Landscape dataset. The results show that the random texture synthesis cannot produce plausible global structures. is important note that the images are shown in Tile- GAN and other texture synthesis papers (Bergmann et al., 2017; Jetchev et al., 2018 118) with plausible global structure, such as the satellite map of Jurassic Park and the paintings in TileGAN, are all conditioned on an image that gives the blueprint of the global structure.
Implementation details TileGAN experiment. We use the officially released TileGAN pipeline to synthesize the results with randomly sampled latent variables. We follow the instructions and train a PGGAN model at 256 resolution, then test the model to synthesize at 512x512, 1024x 1024, and 2048 x 2048 pixels. Note that we discovered that TileGAN alters the PGGAN architecture from residual-base ToRGB branch to single ToRGB projection Such a modification is not described in the TileGAN paper, but can be found by diagnosing the model checkpoints released by the authors. The modification leads to significant visual-quality degradation. However, even without the visual- quality degradation, the lack of structural clues makes TileGAN impossible to infer coherent global structure while synthesizing at larger image sizes.
Generotal (1x) Gererales
Generoted (4x)
Generateel (Bk)
Figure 38: Qualitative results of TileGAN on Flickr-Landscapp Landscape dataset. The results show that random texture synthesis models are not directly applicable to worldwoldd image synthesis.
U MORE COMPARISONS WITH SINGAN-BASED MODELS
We conduct additional experiments ConSinGAN (Hinz al., 2021), concurrent work that proposes several improvements upon SinGAN. We use the officially released codes and hyperpa- rameters train the ConSinGAN models. ConSinGAN has two types of training modes, generation and retargeting. The author mentions the "retargeting" mode is more suitable for extending synthe- sis size in their GitHub release. As shown Figure 39. similar to SinGAN, ConSinGAN does not have specialized mechanisms to deal with different positional information while tested at a different synthesis size. Therefore, neither of the training modes can produce a plausible global view while tested at extended synthesis sizes.
Codes: htceeh/whhhhhcaciiceeeeeeeeeeeen L294.
Codess: https//t L.355.
41

---

Published as a conference paper at ICLR 2022
Generated (1x)
Generated (2x)
Generated (4x)
Generated (8x)
Generated (1x)
Generated (2x)
Generated (4x)
Generated (8x)
Figure 39: Qualitative results of ConSinGAN on Flickr-Landscape dataset. We run ConSinGAN under two different configurations released by the authors, (top) generation and (bottom) retarget. The results show that ConSinGAN inherits similar behaviors from SinGAN and fails to produce images with plausible global structure while synthesizing at larger image sizes.
42

---

Published as a conference paper at ICLR 2022
V
ABLATION: MODE-SEEKING DIVERSITY Loss (Ldiv)
@ InfinityGAN
(b) Infinity Diversity Loss
Figure 40: Ablation on the mode-seeking diversity loss. We show that the mode-seeking diversity loss discourages the model from synthesizing similar appearance at the same coordinate. Both Infin- ityGAN models are trained patch size, then synthesize at 101 101 pixcls at testing. In this figure, all images share the same coordinate grid, and each row shares the same global latent variable. Therefore, only the local latent variables are varying in each row. In Figure (a), regular InfinityGAN shows high diversity in each row, and obvious mruuturee-cordiigg high-level layout is shared among each row, while the differences between samples are mostly local variations. In particular, the third, sixth, and seventh rows share similar layout, which is sign that the model learns a correspondence between the image structure and the coordinates. However, it is difficult to quantify such problem since the repetition is not an exact repetition of content but structural/semantical level similarity.
data with 197x 197 full-image size and 101 x 101
relation is presented. In contrast, in Figure (b), consistent
W ABLATION: AUXILIARY LOSS (Lor)
Figure 41: Ablation on the auxiliary loss. We show that InfinityGAN with auxiliary loss (or- ange curve) can provide slight improvement compared the variant without the auxiliary loss (pink curve). However, such a performance difference is not very significant. We believe the additional supervision the vertical position should provide important clues in helping the model learn the spatial-varying distribution in the vertical direction. Our approach in modeling such information with two MLP-layers (see Figure 18) may be too naive. Future studies on improving the model- ing performance with better loss functions or architecture may improve the overall performance of InfinityGAN further.
43


Reviews:
[{"review_id": "cx0KhIylgH_", "reviewer": "Reviewer_1Z86", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel framework, InfinityGAN, for infinitely-wide landscape image generation, combining recent advances on style-based generative adversarial networks and implicit neural representations. Leveraging parallelizable inference, patch-based generative modelling and hand-crafted landscape image priors, the proposed framework can synthesize landscape images of seemingly infinite width and a large height. The paper further proposes a new evaluation metric, ScaleInv FID, and a set of applications for their framework, including image outpainting and inbetweening, and spatial style fusion. The results of InfinityGAN are validated using user studies and some image-space metrics. \n\nThe paper proposes very interesting ideas on image generation, including a padding-free StyleGAN architecture, feature unfolding and neural implicit representations for adding variation to the generated image contents, as well as hand-crafted priors on landscape image generation, including positional encodings and disentanglements between global and local styles. The paper is well-written and presented and is extended with a very valuable supplementary material. It includes sufficient details on implementation for allowing reproducibility. The proposed applications could be impactful in the image generation literature. \n\nOverall, this paper shows potential for somewhat significant algorithmic contributions for image generative models, but it falls short on important aspects that need to be adressed before it can be recommended for acceptance. First, several of the claims of the paper should be toned down. The paper argues it proposes a generic framework for infinite image synthesis, but the hand-crafted priors embedded in the method and the datasets used for validation are very specific for landscape photography, making it more limited in its scope than the paper claims. Second, many decisions taken throughout the design of the framework are not well justified or validated empirically, making it hard to understand what the actual contributions of the paper really are.  Finally, this paper misses important references and comparisons to prior work, the comparisons that are present in the paper are somewhat arbitrary and the proposed evaluation metric lacks implementation details and justification. \n\n*Detailed review* \n\n- This paper should tone down its claims of generality. The paper claims to present \"a novel framework for arbitrary-sized image generation\". However, the design of the method and the datasets used for evaluating it are limited to landscape photography, which shows particularities that InfinityGAN exploits in clever ways but limits its generality. In particular, the hand-crafted prior of self-similarity on the horizontal axis but rapid saturation on the vertical axis only holds for this particular image domain. Therefore, it is unlikely to work in other domains where arbitrary-sized image generation is important, including texture synthesis or satellite photography. This limitation is implicitly acknowledged on the conclusions section, but is not properly discussed throughout the paper.\n\n- Several important references are missing on the related work. In the context of neural implicit representations, relevant recent advances should be included and discussed (eg Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, Tancik et al, NeurIPS 2020; Implicit Neural Representations with Periodic Activation Functions, Sitzmann et al, NeurIPS 2020). \n\n- Important comparisons with previous work on image generation are missing: TileGAN: Synthesis of Large-Scale Non-Homogeneous \nTextures, Frühstück et al, SIGGRAPH 2019; Learning Texture Manifolds with the Periodic Spatial GAN,  Bergmann et al, ICML 2017;  Improved Techniques for Training Single-Image GANs, Hinz et al, WACV 2021. Concurrent work that could be mentioned includes Taming Transformers for High-Resolution Image Synthesis, Esser et al, CVPR 2021. Finally, the paper proposes an inbetweening application, which should be better contextualized on the image inpainting literature. \n\n- Some parts of the design of the framework need to be better empirically demonstrated. \n1) What will happen if the self-similarity prior is introduced on both axes? \n2) What is the impact of the period T of the positional encoding? \n3) What is the impact of the mode-seeking diversity loss introduced to the generator training? \n4) Why is the auxiliary task of the generator needed? What happens if it is not done? \n5) What will happen when a non-landscape dataset is used? \n\n- With regards to Section 4:\n1) The proposed ScaleInv FID seems rather arbitrary and its implementation is not clear. Could the authors elaborate further on this metric? From the current level of detail about this metric, it is quite hard to draw conclusions from it.\n2) The comparison with SinGAN is rather unfair to their method. The paper could include more implementation details on how they implement this comparison. It also remains unclear if the improved results of InfinityGAN are due to its inductive biases (hand crafted priors, neural implicit representations, etc), or simply because InfinityGAN is trained on tens of thousands of images, whilst SinGAN is trained on a single image.\n3) The user study needs further explanation. What are the demographics of this user set? Which images were they presented with (eg what dataset). What were the questions they were asked?\n4) Why is the inception score (IS) used on Table 2 but not on Table 1? Could the authors extend these analyses with other metrics, including SSIM or LPIPS?\n\nIn terms of results, on the Supplementary Material, it is clear that InfinityGAN struggles to add variation to the uppermost parts of the images (eg Figures 20 and 21). This is likely due to the tanh() operation done to the vertical coordinates, which limits the variation on this axis compared to the horizontal, positonal-encoded, axis. Could the authors elaborate on this and maybe show more examples on vertical expansion?\n\nI have other minor comments:\n- Was any type of data augmentation considered, rather than random cropping? This could alleviate the need for such massive datasets. (Training Generative Adversarial Networks with Limited Data, Karras et al 2020)\n- It would be really interesting to see spatial fusion and outpainting applications in the LSUN dataset and other architectural datasets. Did the authors consider this?\n- Could this method be extended for 360º photography for VR applications? For example, by making the positional encoding periodic for both axes? It would be great if authors could ellaborate on this.\n\n\n** Update after rebuttal period** \n\n\nThe authors have successfully adressed many of my concerns during the rebuttal period, including new extensive comparisons with other methods, ablation studies, discussion of limitations and clarifications on the implementation of the Scale-Invariant FID metric. I am overall more positive about this submission and thereby recommend it for acceptance, as this paper proposes novel and well-evaluated ideas, despite some of the claims being somewhat oversold. The responses to concerns by other reviewers were also convincing. \n\n", "summary_of_the_paper": "This paper proposes a novel framework, InfinityGAN, for infinitely-wide landscape image generation, combining recent advances on style-based generative adversarial networks and implicit neural representations. Leveraging parallelizable inference, patch-based generative modelling and hand-crafted landscape image priors, the proposed framework can synthesize landscape images of seemingly infinite width and a large height. The paper further proposes a new evaluation metric, ScaleInv FID, and a set of applications for their framework, including image outpainting and inbetweening, and spatial style fusion. The results of InfinityGAN are validated using user studies and some image-space metrics. ", "summary_of_the_review": "This paper proposes very interesting ideas on image generation, including a padding-free StyleGAN architecture, feature unfolding and neural implicit representations for adding variation to the generated image contents, as well as hand-crafted priors on landscape image generation, including positional encodings and disentanglements between global and local styles. It includes sufficient details on implementation for allowing reproducibility. The proposed applications could be impactful in the image generation literature. \n\nOverall, even though this paper shows potential for somewhat significant algorithmic contributions for image generative models, it falls short on important aspects that need to be adressed before it can be recommended for acceptance. First, several of the claims of the paper should be toned down. The paper argues it proposes a generic framework for infinite image synthesis, but the hand-crafted priors embedded in the method and the datasets used for validation are very specific for landscape photography, making it much more limited in its scope than the paper claims to be. Second, many decisions taken throughout the design of the framework are not well justified or validated empirically, making it hard to understand what the actual contributions of the paper really are.  Finally, this paper misses important references and comparisons to prior work, the comparisons that are present in the paper are somewhat arbitrary and the proposed evaluation metric lacks implementation details and justification. \n\n** Update after rebuttal period** \nThe authors have adressed many of my concerns during the rebuttal period, including new extensive comparisons with other methods, ablation studies and discussion of limitations. I am overall more positive about this submission and thereby recommend it for acceptance.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZGzJ5LWx9IL", "reviewer": "Reviewer_gucW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is a very solid paper. The writing is good and the proposed model seems novel.  The task is very interesting and the authors addressed several challenges in very smart ways. For example, I think generating two patches that can be combined seamlessly is hard. The authors proposed a modification to StyleGAN to eliminate zero padding. Position information is further added to generate plausible results.\nThe experiments are sufficient and comprehensive. A lot of details are provided in the supplementary material. Applications including spatial style fusion, multimodal outpainting, and image inbetweening are also impressive.\n \n\nThe major concern I have is about the problem formulation. The whole image generation pipeline involves 3 random variables as input: $z_g$, $z_l$, and $z_n$. $z_n$ is used as noise input in StyleGAN. The structural latent variable $z_S$ is generated from $G_S(z_g, z_l, c)$. However, $z_S$ is only used in $G_T$, while there is another input $z_g$ in G_T. I was wondering if the $z_g$ input in $G_S$ is necessary? As shown by Fig 6, $z_l$ determines the layout while $z_g$ in $G_T$ for style, I did not see the role of $z_g$ in $G_S$. In other words, what if we use different $z_g$ in $G_S$ and $G_T$? Which $z_g$ will determine the style?\n \nHere are some minor issues that will not affect my rating on this paper. But I would appreciate it if the authors could share more insight:\n\n1 In the previous work section, I wondered if the authors also want to include autoregressive models like PixelCNN. These models can generate images of arbitrary sizes or shapes by nature. The authors did cite the VQVAE2 paper and mentioned it could only generate images of fixed sizes.\n \n2 I found the visual result on the Flickr-Landscape data is better than that on the LSUN bridge or tower datasets. For example, Fig 27 top left shows repetitive tower patterns. Is this limited by the proposed pipeline, or the datasets, or StyleGAN?", "summary_of_the_paper": "This paper proposed a new model called InfinityGAN, for generating images of arbitrary sizes. These images are generated holistically: the authors proposed using a structure synthesizer first to generate a latent structural representation. Then a modified StyleGAN based texture synthesizer generates the image patches at the corresponding locations. The positional encoding and PFG enable generating patches in an arbitrary large image and aggregating these patches seamlessly. The experiments on the Flickr-Landscape dataset show promising large image synthesis performance. On the Place365 and Flickr-Scenery datasets, the experiment results show superior image outpainting performance. The authors also demonstrated other applications including spatial style fusion and image inbetweening.", "summary_of_the_review": "I did not find big issues with this submission. It is a very good paper, and I think the proposed techniques can benefit other large image generation research. I am only concerned about the $z_g$ in both $G_S$ and $G_T$, which I think might be redundant. Therefore I recommend accepting this submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RDBlIcPNNaO", "reviewer": "Reviewer_TkQk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper advances the state of art in image synthesis and there are multiple contributions that could be re-used in other contexts. For example, I really liked the intuition behind the padding-free generator and how the authors showed its effectiveness in both quantitative and qualitative results.\n+ The paper is well written and justified. All the technical details are well elaborated and additional supplementary material provides all the details needed to ensure reproducibility. The contributions seem to be well placed with respect to the state of art\n+ The experimental section is impressive, plenty of examples showing the effectiveness of the approach. The supplementary materials show multiple comparisons and visuals as well as implementation details for the spatial style fusion.\n\nThere are a couple, mostly minor weak points, that I hope could help the authors to make the paper even stronger:\n- Evaluation on temporal consistency: it would be great to assess how the method performs on temporal sequences and assess how stable/consistent it is. I understand that the method was not designed for this purpose, but could still provide a baseline for future research.\n- Ablation study: there are lots of interesting contributions here, from architecture specific modules to loss functions. I would suggest to elaborate more the ablation study section, which mostly focuses on the padding-free generator at the moment.", "summary_of_the_paper": "The paper proposes a framework to generate high resolution images of outdoor scenes with any desired style. The approach extends image synthesis techniques and proposes a system that uses both global and local information. This ensures that the model can be trained using low resolution patches (e.g. 101 x 101) and, at inference time, it can be used to generate high resolution renderings: images up to 4096 x 4096 are showed.\n\nThe combination of the global and local appearance, coupled with interesting modules such as the padding-free generator, ensure high quality results that are consistently outperforming other approaches.", "summary_of_the_review": "This paper is a solid submission, that really pushes the state of art in image synthesis for outdoor scenes. The problem is well justified and placed with respect to related work and the contributions are well described. The results prove that the method is outperforming baselines with comparisons on both quantitative and qualitative results. Given the complexity of the framework, I would have appreciated a more in-depth analysis (i.e. ablation study) of the various components to help other researchers to reproduce the results and focus on the right modules, however I acknowledge that authors did their best with experiments and evaluations: the supplementary material is actually impressive. \n\nI would suggest authors to also show some results on temporal sequences, to assess temporal consistency across frames and/or long sequences. This would also help to establish a baseline for future work in this area.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8zUaHa-qzXx", "reviewer": "Reviewer_bTWK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1) The idea to generate infinity-pixel images in a seamless patch-by-patch manner is quite novel.\n2) This approach considers both global and local consistency in generating large images. Though straightforward, the designed framework based on it is effective and elegant.\n3) The experiments and the appendix are comprehensive to reproduce this work.\n\nWeakness:\nNo obvious weakness but there are some issues.\n1) It is better to add the symbols in Sec 3.2 & 3.3 to Figure 2, e.g., $G_s$, $Z_s$, $G_T$, $p_c$.\n2) Although the concurrent work ALIS (SKOROKHODOV ET AL., 2021) has been discussed in appendix A, it is suggested to make a fair comparison in Table 1 with the same metrics in the future version, since ALIS address nearly the same task to generate the infinity-pixel images.\n", "summary_of_the_paper": "This paper proposes a novel framework, InfinityGAN, which could generate arbitrary-sized images with infinite pixels.\n\nBy disentangling global appearance, local structures and textures, InfinityGAN generates local and global consistent large images with a seamless patch-by-patch paradigm. \n\nExperiments validate its superiority compared to some other baselines, and several applications are raised based on their approaches.", "summary_of_the_review": "Based on its strength and weakness, I think it is a good and quite novel paper considering both its framework and applications. I recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: 3 reviewers recommend accept, 1 rates the paper marginally above acceptance. The authors provided satisfactory answers to criticism -- all in all this is a paper worth accepting at ICLR. Please make sure that criticism in the reviews is adequately addressed in the final version, e.g. include various experimental results in the rebuttal, add the symbols in sec 3.2 & 3.3 to fig. 2, add a related discussion on ablations when the model is fully trained, etc.
Decision: Accept (Poster)