InfinityGAN: Towards Infinite-Pixel Image Synthesis

1 INTRODUCTION . Generative models witness substantial improvements in resolution and level of details . Most improvements come at a price of increased training time ( Gulrajani et al. , 2017 ; Mescheder et al. , 2018 ) , larger model size ( Balaji et al. , 2021 ) , and stricter data requirements ( Karras et al. , 2018 ) . The most recent works synthesize images at 1024×1024 resolution featuring a high level of details and fidelity . However , models generating high resolution images usually still synthesize images of All codes , datasets , and trained models will be made publicly available . limited field-of-view bounded by the training data . It is not straightforward to scale these models to generate images of arbitrarily large field-of-view . Synthesizing infinite-pixel images is constrained by the finite nature of resources . Finite computational resources ( e.g. , memory and training time ) set bounds for input receptive field and output size . A further limitation is that there exists no infinitepixel image dataset . Thus , to generate infinite-pixel images , a model should learn the implicit global structure without direct supervision and under limited computational resources . Repetitive texture synthesis methods ( Efros & Leung , 1999 ; Xian et al. , 2018 ) generalize to large spatial sizes . Yet , such methods are not able to synthesize real-world images . Recent works , such as SinGAN ( Shaham et al. , 2019 ) and InGAN ( Shocher et al. , 2019 ) , learn an internal patch distribution for image synthesis . Although these models can generate images with arbitrary shapes , in Section 4.1 , we show that they do not infer structural relationships well , and fail to construct plausible holistic views with spatially extended latent space . A different approach , COCO-GAN ( Lin et al. , 2019 ) , learns a coordinate-conditioned patch distribution for image synthesis . As shown in Figure 4 , despite the ability to slightly extend images beyond the learned boundary , it fails to maintain the global coherence of the generated images when scaling to a 2× larger generation size . How to generate infinite-pixel images ? Humans are able to guess the whole scene given a partial observation of it . In a similar fashion , we aim to build a generator that trains with image patches , and inference images of unbounded arbitrary-large size . An example of a synthesized scene containing globally-plausible structure and heterogeneous textures is shown in Figure 1 . We propose InfinityGAN , a method that trains on a finite-pixel dataset , while generating infinitepixel images at inference time . InfinityGAN consists of a neural implicit function , termed structure synthesizer , and a padding-free StyleGAN2 generator , dubbed texture synthesizer . Given a global appearance of an infinite-pixel image , the structure synthesizer samples a sub-region using coordinates and synthesizes an intermediate local structural representations . The texture synthesizer then seamlessly synthesizes the final image by parts after filling the fine local textures to the local structural representations . InfinityGAN can infer a compelling global composition of a scene with realistic local details . Trained on small patches , InfinityGAN achieves high-quality , seamless and arbitrarily-sized outputs with low computational resources—a single TITAN X to train and test . We conduct extensive experiments to validate the proposed method . Qualitatively , we present the everlastingly long landscape images . Quantitatively , we evaluate InfinityGAN and related methods using user study and a proposed ScaleInv FID metric . Furthermore , we demonstrate the efficiency and efficacy of the proposed methods with several applications . First , we demonstrate the flexibility and controllability of the proposed method by spatially fusing structures and textures from different distributions within an image . Second , we show that our model is an effective deep image prior for the image outpainting task with the image inversion technique and achieves multi-modal outpainting of arbitrary length from arbitrarily-shaped inputs . Third , with the proposed model we can divide-and-conquer the full image generation into independent patch generation and achieve 7.2× of inference speed-up with parallel computing , which is critical for high-resolution image synthesis . 2 RELATED WORK . Latent generative models . Existing generative models are mostly designed to synthesize images of fixed sizes . A few methods ( Karras et al. , 2018 ; 2020 ) have been recently developed to train latent generative models on high-resolution images , up to 1024×1024 pixels . However , latent generative models generate images from dense latent vectors that require synthesizing all structural contents at once . Bounded by computational resources and limited by the learning framework and architecture , these approaches synthesize images of certain sizes and are non-trivial to generalize to different output size . In contrast , patch-based GANs trained on image patches ( Lin et al. , 2019 ; Shaham et al. , 2019 ; Shocher et al. , 2019 ) are less constrained by the resource bottleneck with the synthesis-by-part approach . However , ( Shaham et al. , 2019 ; Shocher et al. , 2019 ) can only model and repeat internal statistics of a single image , and ( Lin et al. , 2019 ) can barely extrapolate few patches beyond the training size . ALIS ( Skorokhodov et al. , 2021 ) is a concurrent work that also explores synthesizing infinite-pixel images . It recursively inbetweens latent variable pairs in the horizontal direction . We further discuss the method in Appendix A.1 . Finally , autoregressive models ( Oord et al. , 2016 ; Razavi et al. , 2019 ; Esser et al. , 2021 ) can theoretically synthesize at arbitrary image sizes . Despite ( Razavi et al. , 2019 ) and ( Esser et al. , 2021 ) showing unconditional images synthesis at 1024×1024 resolution , their application in infinite-pixel image synthesis has not yet been well-explored . Conditional generative models . Numerous tasks such as image super-resolution , semantic image synthesis , and image extrapolation often showcase results over 1024×1024 pixels . These tasks are less related to our setting , as most structural information is already provided in the conditional inputs . We illustrate and compare the characteristics of these tasks against ours in Appendix B . Image outpainting . Image outpainting ( Abdal et al. , 2020 ; Liu et al. , 2021 ; Sabini & Rusak , 2018 ; Yang et al. , 2019 ) is related to image inpainting ( Liu et al. , 2018a ; Yu et al. , 2019 ) and shares similar issues that the generator tends to copy-and-paraphrase the conditional input or create mottled textural samples , leading to repetitive results especially when the outpainted region is large . InOut ( Cheng et al. , 2021 ) proposes to outpaint image with GANs inversion and yield results with higher diversity . We show that with InfinityGAN as the deep image prior along with InOut ( Cheng et al. , 2021 ) , we obtain the state-of-the-art outpainting results and avoids the need of sequential outpainting . Then , we demonstrate applications in arbitrary-distant image inbetweening , which is at the intersection of image inpainting ( Liu et al. , 2018a ; Nazeri et al. , 2019 ; Yu et al. , 2019 ) and outpainting research . Neural implicit representation . Neural implicit functions ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Mildenhall et al. , 2020 ) have been applied to model the structural information of 3D and continuous representations . Adopting neural implicit modeling , our query-by-coordinate synthesizer is able to model structural information effectively . Some recent works ( DeVries et al. , 2021 ; Niemeyer & Geiger , 2021 ; Chan et al. , 2021 ) also attempt to integrate neural implicit function into generative models , but aiming at 3D-structure modeling instead of extending the synthesis field-of-view . 3 PROPOSED METHOD . 3.1 OVERVIEW . An arbitrarily large image can be described globally and locally . Globally , images should be coherent and hence global characteristics should be expressible by a compact holistic appearance ( e.g. , a medieval landscape , ocean view panorama ) . Therefore , we adopt a fixed holistic appearance for each infinite-pixel image to represent the high-level composition and content of the scene . Locally , a close-up view of an image is defined by its local structure and texture . The structure represents objects , shapes and their arrangement within a local region . Once the structure is defined , there exist multiple feasible appearances or textures to render realistic scenes . At the same time , structure and texture should conform to the global holistic appearance to maintain the visual consistency among the neighboring patches . Given these assumptions , we can generate an infinite-pixel image by first sampling a global holistic appearance , then spatially extending local structures and textures following the holistic appearance . Accordingly , the InfinityGAN generator G consists of a structure synthesizer GS and a texture synthesizer GT . GS is an implicit function that samples a sub-region with coordinates and creates local structural features . GT is a fully convolutional StyleGAN2 ( Karras et al. , 2020 ) modeling textural properties for local patches and rendering final image . Both modules follow a consistent holistic appearance throughout the process . Figure 2 presents the overview of our framework . 3.2 STRUCTURE SYNTHESIZER ( GS ) Structure synthesizer is a neural implicit function driven by three sets of latent variables : A global latent vector zg representing the holistic appearance of the infinite-pixel image ( also called implicit image since the whole image is never explicitly sampled ) , a local latent tensor zl expressing the local structural variation of the image content , and a coordinate grid c specifying the location of the patches to sample from the implicit image . The synthesis process is formulated as : zS = GS ( zg , zl , c ) , ( 1 ) where zS denotes the structural latent variable that is later used as an input to the texture synthesizer . We sample zg ∈ RDzg from a unit Gaussian distribution once and inject zg into every layer and pixel in GS via feature modulation ( Huang & Belongie , 2017 ; Karras et al. , 2020 ) . As local variations are independent across the spatial dimension , we independently sample them from a unit Gaussian prior for each spatial position of zl ∈ RH×W×Dzl , where H and W can be arbitrarily extended . We then use coordinate grid c to specify the location of the target patches to be sampled . To be able to conditionGS with coordinates infinitely far from the origin , we introduce a prior by exploiting the nature of landscape images : ( a ) self-similarity for the horizontal direction , and ( b ) rapid saturation ( e.g. , land , sky or ocean ) for the vertical direction . To implement this , we use the positional encoding for the horizontal axis similar to ( Vaswani et al. , 2017 ; Tancik et al. , 2020 ; Sitzmann et al. , 2020 ) . We use both sine and cosine functions to encode each coordinate for numerical stability . For the vertical axis , to represent saturation , we apply the tanh function . Formally , given horizontal and vertical indexes ( ix , iy ) of zl tensor , we encode them as c = ( tanh ( iy ) , cos ( ix/T ) , sin ( ix/T ) ) , where T is the period of the sine function and c controls the location of the patch to generate . To prevent the model from ignoring the variation of zl and generating repetitive content by following the periodically repeating coordinates , we adopt a mode-seeking diversity loss ( Mao et al. , 2019 ) between a pair of local latent variables zl1 and zl2 while sharing the same zg and c : Ldiv = ‖zl1 − zl2‖1 / ‖GS ( zg , zl1 , c ) −GS ( zg , zl2 , c ) ‖1 . ( 2 ) Conventional neural implicit functions produce outputs for each input query independently , which is a pixel in zl for InfinityGAN . Such a design causes training instabilities and slows convergence , as we show in Figure 37 . We therefore adopt the feature unfolding technique ( Chen et al. , 2021 ) to enable GS to account for the information in a broader neighboring region of zl and c , introducing a larger receptive field . For each layer in GS , before feeding forward to the next layer , we apply a k × k feature unfolding transformation at each location ( i , j ) of the origin input f to obtain the unfolded input f ′ : f ′ ( i , j ) = Concat ( { f ( i+ n , j +m ) } n , m∈ { −k/2 , k/2 } ) , ( 3 ) where Concat ( · ) concatenates the unfolded vectors in the channel dimension . In practice , as the gridshaped zl and c are sampled with equal spacing between consecutive pixels , the feature unfolding can be efficiently implemented with CoordConv ( Liu et al. , 2018b ) . 3.3 TEXTURE SYNTHESIZER ( GT ) Texture synthesizer aims to model various realizations of local texture given the local structure zS generated by the structure synthesizer . In addition to the holistic appearance zg and the local structural latent zS , texture synthesizer uses noise vectors zn to model the finest-grained textural variations that are difficult to capture by other variables . The generation process can be written as : pc = GT ( zS , zg , zn ) , ( 4 ) where pc is a generated patch at location c ( i.e. , the c used in Eq 1 for generating zS ) . We implement upon StyleGAN2 ( Karras et al. , 2020 ) . First , we replace the fixed constant input with the generated structure zS . Similar to StyleGAN2 , randomized noises zn are added to all layers of GT , representing the local variations of fine-grained textures . Then , a mapping layer projects zg to a style vector , and the style is injected into all pixels in each layer via feature modulation . Finally , we remove all zero-paddings from the generator , as shown in Figure 3 ( b ) . Both zero-padding and GS can provide positional information to the generator , and we later show that positional information is important for generator learning in Section 4.2 . However , it is necessary to remove all zero-paddings from GT for three major reasons . First , zero-padding has a consistent pattern during training , due to the fixed training image size . Such a behavior misleads the generator to memorize the padding pattern , and becomes vulnerable to unseen padding patterns while attempting to synthesize at a different image size . The third column of Figure 4 shows when we extend the input latent variable of the StyleGAN2 generator multiple times , the center part of the features does not receive expected coordinate information from the paddings , resulting in extensively repetitive textures in the center area of the output image . Second , zero-paddings can only provide positional information within a limited distance from the image border . However , while generating infinite-pixel images , the image border is considered infinitely far from the generated patch . Finally , as shown in Figure 3 , the existence of paddings hampers GT from generating separate patches that can be composed together . Therefore , we remove all paddings from GT , facilitating the synthesisby-parts of arbitrary-sized images . We refer to the proposed GT as a padding-free generator ( PFG ) .

Summary: The paper proposes a framework to generate high resolution images of outdoor scenes with any desired style. The approach extends image synthesis techniques and proposes a system that uses both global and local information. This ensures that the model can be trained using low resolution patches (e.g. 101 x 101) and, at inference time, it can be used to generate high resolution renderings: images up to 4096 x 4096 are showed.  The combination of the global and local appearance, coupled with interesting modules such as the padding-free generator, ensure high quality results that are consistently outperforming other approaches.