InfinityGAN: Towards Infinite-Pixel Image Synthesis

1 INTRODUCTION . Generative models witness substantial improvements in resolution and level of details . Most improvements come at a price of increased training time ( Gulrajani et al. , 2017 ; Mescheder et al. , 2018 ) , larger model size ( Balaji et al. , 2021 ) , and stricter data requirements ( Karras et al. , 2018 ) . The most recent works synthesize images at 1024×1024 resolution featuring a high level of details and fidelity . However , models generating high resolution images usually still synthesize images of All codes , datasets , and trained models will be made publicly available . limited field-of-view bounded by the training data . It is not straightforward to scale these models to generate images of arbitrarily large field-of-view . Synthesizing infinite-pixel images is constrained by the finite nature of resources . Finite computational resources ( e.g. , memory and training time ) set bounds for input receptive field and output size . A further limitation is that there exists no infinitepixel image dataset . Thus , to generate infinite-pixel images , a model should learn the implicit global structure without direct supervision and under limited computational resources . Repetitive texture synthesis methods ( Efros & Leung , 1999 ; Xian et al. , 2018 ) generalize to large spatial sizes . Yet , such methods are not able to synthesize real-world images . Recent works , such as SinGAN ( Shaham et al. , 2019 ) and InGAN ( Shocher et al. , 2019 ) , learn an internal patch distribution for image synthesis . Although these models can generate images with arbitrary shapes , in Section 4.1 , we show that they do not infer structural relationships well , and fail to construct plausible holistic views with spatially extended latent space . A different approach , COCO-GAN ( Lin et al. , 2019 ) , learns a coordinate-conditioned patch distribution for image synthesis . As shown in Figure 4 , despite the ability to slightly extend images beyond the learned boundary , it fails to maintain the global coherence of the generated images when scaling to a 2× larger generation size . How to generate infinite-pixel images ? Humans are able to guess the whole scene given a partial observation of it . In a similar fashion , we aim to build a generator that trains with image patches , and inference images of unbounded arbitrary-large size . An example of a synthesized scene containing globally-plausible structure and heterogeneous textures is shown in Figure 1 . We propose InfinityGAN , a method that trains on a finite-pixel dataset , while generating infinitepixel images at inference time . InfinityGAN consists of a neural implicit function , termed structure synthesizer , and a padding-free StyleGAN2 generator , dubbed texture synthesizer . Given a global appearance of an infinite-pixel image , the structure synthesizer samples a sub-region using coordinates and synthesizes an intermediate local structural representations . The texture synthesizer then seamlessly synthesizes the final image by parts after filling the fine local textures to the local structural representations . InfinityGAN can infer a compelling global composition of a scene with realistic local details . Trained on small patches , InfinityGAN achieves high-quality , seamless and arbitrarily-sized outputs with low computational resources—a single TITAN X to train and test . We conduct extensive experiments to validate the proposed method . Qualitatively , we present the everlastingly long landscape images . Quantitatively , we evaluate InfinityGAN and related methods using user study and a proposed ScaleInv FID metric . Furthermore , we demonstrate the efficiency and efficacy of the proposed methods with several applications . First , we demonstrate the flexibility and controllability of the proposed method by spatially fusing structures and textures from different distributions within an image . Second , we show that our model is an effective deep image prior for the image outpainting task with the image inversion technique and achieves multi-modal outpainting of arbitrary length from arbitrarily-shaped inputs . Third , with the proposed model we can divide-and-conquer the full image generation into independent patch generation and achieve 7.2× of inference speed-up with parallel computing , which is critical for high-resolution image synthesis . 2 RELATED WORK . Latent generative models . Existing generative models are mostly designed to synthesize images of fixed sizes . A few methods ( Karras et al. , 2018 ; 2020 ) have been recently developed to train latent generative models on high-resolution images , up to 1024×1024 pixels . However , latent generative models generate images from dense latent vectors that require synthesizing all structural contents at once . Bounded by computational resources and limited by the learning framework and architecture , these approaches synthesize images of certain sizes and are non-trivial to generalize to different output size . In contrast , patch-based GANs trained on image patches ( Lin et al. , 2019 ; Shaham et al. , 2019 ; Shocher et al. , 2019 ) are less constrained by the resource bottleneck with the synthesis-by-part approach . However , ( Shaham et al. , 2019 ; Shocher et al. , 2019 ) can only model and repeat internal statistics of a single image , and ( Lin et al. , 2019 ) can barely extrapolate few patches beyond the training size . ALIS ( Skorokhodov et al. , 2021 ) is a concurrent work that also explores synthesizing infinite-pixel images . It recursively inbetweens latent variable pairs in the horizontal direction . We further discuss the method in Appendix A.1 . Finally , autoregressive models ( Oord et al. , 2016 ; Razavi et al. , 2019 ; Esser et al. , 2021 ) can theoretically synthesize at arbitrary image sizes . Despite ( Razavi et al. , 2019 ) and ( Esser et al. , 2021 ) showing unconditional images synthesis at 1024×1024 resolution , their application in infinite-pixel image synthesis has not yet been well-explored . Conditional generative models . Numerous tasks such as image super-resolution , semantic image synthesis , and image extrapolation often showcase results over 1024×1024 pixels . These tasks are less related to our setting , as most structural information is already provided in the conditional inputs . We illustrate and compare the characteristics of these tasks against ours in Appendix B . Image outpainting . Image outpainting ( Abdal et al. , 2020 ; Liu et al. , 2021 ; Sabini & Rusak , 2018 ; Yang et al. , 2019 ) is related to image inpainting ( Liu et al. , 2018a ; Yu et al. , 2019 ) and shares similar issues that the generator tends to copy-and-paraphrase the conditional input or create mottled textural samples , leading to repetitive results especially when the outpainted region is large . InOut ( Cheng et al. , 2021 ) proposes to outpaint image with GANs inversion and yield results with higher diversity . We show that with InfinityGAN as the deep image prior along with InOut ( Cheng et al. , 2021 ) , we obtain the state-of-the-art outpainting results and avoids the need of sequential outpainting . Then , we demonstrate applications in arbitrary-distant image inbetweening , which is at the intersection of image inpainting ( Liu et al. , 2018a ; Nazeri et al. , 2019 ; Yu et al. , 2019 ) and outpainting research . Neural implicit representation . Neural implicit functions ( Park et al. , 2019 ; Mescheder et al. , 2019 ; Mildenhall et al. , 2020 ) have been applied to model the structural information of 3D and continuous representations . Adopting neural implicit modeling , our query-by-coordinate synthesizer is able to model structural information effectively . Some recent works ( DeVries et al. , 2021 ; Niemeyer & Geiger , 2021 ; Chan et al. , 2021 ) also attempt to integrate neural implicit function into generative models , but aiming at 3D-structure modeling instead of extending the synthesis field-of-view . 3 PROPOSED METHOD . 3.1 OVERVIEW . An arbitrarily large image can be described globally and locally . Globally , images should be coherent and hence global characteristics should be expressible by a compact holistic appearance ( e.g. , a medieval landscape , ocean view panorama ) . Therefore , we adopt a fixed holistic appearance for each infinite-pixel image to represent the high-level composition and content of the scene . Locally , a close-up view of an image is defined by its local structure and texture . The structure represents objects , shapes and their arrangement within a local region . Once the structure is defined , there exist multiple feasible appearances or textures to render realistic scenes . At the same time , structure and texture should conform to the global holistic appearance to maintain the visual consistency among the neighboring patches . Given these assumptions , we can generate an infinite-pixel image by first sampling a global holistic appearance , then spatially extending local structures and textures following the holistic appearance . Accordingly , the InfinityGAN generator G consists of a structure synthesizer GS and a texture synthesizer GT . GS is an implicit function that samples a sub-region with coordinates and creates local structural features . GT is a fully convolutional StyleGAN2 ( Karras et al. , 2020 ) modeling textural properties for local patches and rendering final image . Both modules follow a consistent holistic appearance throughout the process . Figure 2 presents the overview of our framework . 3.2 STRUCTURE SYNTHESIZER ( GS ) Structure synthesizer is a neural implicit function driven by three sets of latent variables : A global latent vector zg representing the holistic appearance of the infinite-pixel image ( also called implicit image since the whole image is never explicitly sampled ) , a local latent tensor zl expressing the local structural variation of the image content , and a coordinate grid c specifying the location of the patches to sample from the implicit image . The synthesis process is formulated as : zS = GS ( zg , zl , c ) , ( 1 ) where zS denotes the structural latent variable that is later used as an input to the texture synthesizer . We sample zg ∈ RDzg from a unit Gaussian distribution once and inject zg into every layer and pixel in GS via feature modulation ( Huang & Belongie , 2017 ; Karras et al. , 2020 ) . As local variations are independent across the spatial dimension , we independently sample them from a unit Gaussian prior for each spatial position of zl ∈ RH×W×Dzl , where H and W can be arbitrarily extended . We then use coordinate grid c to specify the location of the target patches to be sampled . To be able to conditionGS with coordinates infinitely far from the origin , we introduce a prior by exploiting the nature of landscape images : ( a ) self-similarity for the horizontal direction , and ( b ) rapid saturation ( e.g. , land , sky or ocean ) for the vertical direction . To implement this , we use the positional encoding for the horizontal axis similar to ( Vaswani et al. , 2017 ; Tancik et al. , 2020 ; Sitzmann et al. , 2020 ) . We use both sine and cosine functions to encode each coordinate for numerical stability . For the vertical axis , to represent saturation , we apply the tanh function . Formally , given horizontal and vertical indexes ( ix , iy ) of zl tensor , we encode them as c = ( tanh ( iy ) , cos ( ix/T ) , sin ( ix/T ) ) , where T is the period of the sine function and c controls the location of the patch to generate . To prevent the model from ignoring the variation of zl and generating repetitive content by following the periodically repeating coordinates , we adopt a mode-seeking diversity loss ( Mao et al. , 2019 ) between a pair of local latent variables zl1 and zl2 while sharing the same zg and c : Ldiv = ‖zl1 − zl2‖1 / ‖GS ( zg , zl1 , c ) −GS ( zg , zl2 , c ) ‖1 . ( 2 ) Conventional neural implicit functions produce outputs for each input query independently , which is a pixel in zl for InfinityGAN . Such a design causes training instabilities and slows convergence , as we show in Figure 37 . We therefore adopt the feature unfolding technique ( Chen et al. , 2021 ) to enable GS to account for the information in a broader neighboring region of zl and c , introducing a larger receptive field . For each layer in GS , before feeding forward to the next layer , we apply a k × k feature unfolding transformation at each location ( i , j ) of the origin input f to obtain the unfolded input f ′ : f ′ ( i , j ) = Concat ( { f ( i+ n , j +m ) } n , m∈ { −k/2 , k/2 } ) , ( 3 ) where Concat ( · ) concatenates the unfolded vectors in the channel dimension . In practice , as the gridshaped zl and c are sampled with equal spacing between consecutive pixels , the feature unfolding can be efficiently implemented with CoordConv ( Liu et al. , 2018b ) . 3.3 TEXTURE SYNTHESIZER ( GT ) Texture synthesizer aims to model various realizations of local texture given the local structure zS generated by the structure synthesizer . In addition to the holistic appearance zg and the local structural latent zS , texture synthesizer uses noise vectors zn to model the finest-grained textural variations that are difficult to capture by other variables . The generation process can be written as : pc = GT ( zS , zg , zn ) , ( 4 ) where pc is a generated patch at location c ( i.e. , the c used in Eq 1 for generating zS ) . We implement upon StyleGAN2 ( Karras et al. , 2020 ) . First , we replace the fixed constant input with the generated structure zS . Similar to StyleGAN2 , randomized noises zn are added to all layers of GT , representing the local variations of fine-grained textures . Then , a mapping layer projects zg to a style vector , and the style is injected into all pixels in each layer via feature modulation . Finally , we remove all zero-paddings from the generator , as shown in Figure 3 ( b ) . Both zero-padding and GS can provide positional information to the generator , and we later show that positional information is important for generator learning in Section 4.2 . However , it is necessary to remove all zero-paddings from GT for three major reasons . First , zero-padding has a consistent pattern during training , due to the fixed training image size . Such a behavior misleads the generator to memorize the padding pattern , and becomes vulnerable to unseen padding patterns while attempting to synthesize at a different image size . The third column of Figure 4 shows when we extend the input latent variable of the StyleGAN2 generator multiple times , the center part of the features does not receive expected coordinate information from the paddings , resulting in extensively repetitive textures in the center area of the output image . Second , zero-paddings can only provide positional information within a limited distance from the image border . However , while generating infinite-pixel images , the image border is considered infinitely far from the generated patch . Finally , as shown in Figure 3 , the existence of paddings hampers GT from generating separate patches that can be composed together . Therefore , we remove all paddings from GT , facilitating the synthesisby-parts of arbitrary-sized images . We refer to the proposed GT as a padding-free generator ( PFG ) .

Summary: This paper proposed a new model called InfinityGAN, for generating images of arbitrary sizes. These images are generated holistically: the authors proposed using a structure synthesizer first to generate a latent structural representation. Then a modified StyleGAN based texture synthesizer generates the image patches at the corresponding locations. The positional encoding and PFG enable generating patches in an arbitrary large image and aggregating these patches seamlessly. The experiments on the Flickr-Landscape dataset show promising large image synthesis performance. On the Place365 and Flickr-Scenery datasets, the experiment results show superior image outpainting performance. The authors also demonstrated other applications including spatial style fusion and image inbetweening.