GENERATING HIGH FIDELITY IMAGES WITH SUBSCALE PIXEL NETWORKS AND MULTIDIMENSIONAL UPSCALING

1 INTRODUCTION . A successful generative model has two core aspects : it produces targets that have high fidelity and it generalizes well on held-out data . Autoregressive ( AR ) models trained by conventional maximum likelihood estimation ( MLE ) have produced superior scores on held-out data across a wide range of domains such as text ( Vaswani et al. , 2017 ; Wu et al. , 2016 ) , audio ( van den Oord et al. , 2016a ) , images ( Parmar et al. , 2018 ) and videos ( Kalchbrenner et al. , 2016 ) . These scores are a measure of the models ’ ability to generalize in that setting . From the perspective of sample fidelity , the outputs generated by AR models have also achieved state-of-the-art fidelity in many of the aforementioned domains with one notable exception . In the domain of unconditional large-scale image generation , AR samples have yet to manifest long-range structure and semantic coherence . One source of difficulties impeding high-fidelity image generation is the multi-faceted relationship between the MLE scores achieved by a model and the model ’ s sample fidelity . On the one hand , MLE is a well-defined measure as improvements in held-out scores generally produce improvements in the visual fidelity of the samples . On the other hand , as opposed to for example adversarial methods ( Arora & Zhang , 2017 ) , MLE forces the model to support the entire empirical distribution . This guarantees the model ’ s ability to generalize at the cost of allotting capacity to parts of the distribution that are irrelevant to fidelity . A second source of difficulties arises from the high dimensionality of large images . A 256× 256× 3 image has a total of 196,608 positions that need to be architecturally connected in order to learn dependencies among them ; the representations at each position require sufficient capacity to express their respective surrounding contexts . These requirements translate to large amounts of memory and computation . ∗Equal contributions . 256 x 256 x 3 x 3 256 x 256 x 3 x 8 32 x 32 x 3 x 3 128 x 128 x 3 x 3 128 x 128 x 3 x 8 . These difficulties notwithstanding , we aim to learn the full distribution over 8-bit RGB images of size up to 256× 256 well enough so that the samples have high fidelity . We aim to guide the model to focus first on visually more salient bits of the distribution and later on the visually less salient bits . We identify two visually salient subsets of the distribution : first , the subset determined by sub-images ( “ slices ” ) of smaller size ( e.g . 32 × 32 ) sub-sampled at all positions from the original image ; and secondly , the subset determined by the few ( e.g . 3 ) most significant bits of each RGB channel in the image . We use Multidimensional Upscaling to map from one subset of the distribution to the other one by upscaling images in size or in depth . For example , the generation of a 128× 128 8-bit RGB image proceeds by first upscaling it in size from a 32 × 32 3-bit RGB image to a 128 × 128 3-bit RGB image ; we then upscale the resulting image in depth to the original resolution of the 128× 128 8-bit RGB image . We thus train three networks : ( a ) a decoder on the small size , low depth image slices subsampled at every n pixels from the original image with the desired target resolution ; ( b ) a size-upscaling decoder that generates the large size , low depth image conditioned on the small size , low depth image ; and ( c ) a depth-upscaling decoder that generates the large size , high depth image conditioned on the large size , low depth image . Figure 1 illustrates this process . To address the latter difficulties that ensue in the training of decoders ( b ) and ( c ) , we develop the Subscale Pixel Network ( SPN ) architecture . The SPN divides an image of size N×N into sub-images of size NS × N S sliced out at interleaving positions ( see Figure 2 ) , which implicitly also captures a form of size upscaling . The N ×N image is generated one slice at a time conditioned on previously generated slices in a way that encodes a rich spatial structure . SPN consists of two networks , a conditioning network that embeds previous slices and a decoder proper that predicts a single target slice given the context embedding . The decoding part of the SPN acts over image slices with the same spatial structure and it can share weights for all of them . The SPN is an independent image decoder with an implicit size upscaling mechanism , but it can also be used as an explicit size upscaling network by initializing the first slice of the SPN input at sampling time with one generated separately during step ( a ) . We extensively evaluate the performance of SPN and the size and depth upscaling methods both quantitatively and from a fidelity perspective on two unconditional image generation benchmarks , CelebAHQ-256 and ImageNet of various sizes up to 256 . From a MLE scores perspective , we compare with previous work to obtain state-of-the-art results on CelebAHQ-256 , both at full 8-bit resolution and at the reduced 5-bit resolution ( Kingma & Dhariwal , 2018 ) , and on ImageNet-64 . We also establish MLE baselines for ImageNet-128 and ImageNet-256 . From a sample fidelity . x x ( a ) ( b ) Figure 2 : The receptive field in a Subscale Pixel Networks ( a ) and the four image slices subsampled from the original image ( b ) perspective , we show the strong benefits of multidimensional upscaling as well as the benefits of the SPN . We produce CelebAHQ-256 samples ( at full 8-bit resolution ) that are of similar visual fidelity to those produced with methods such as GANs that lack however an intrinsic measure of generalization ( Mescheder , 2018 ; Karras et al. , 2017 ) . We also produce some of the first successful samples on unconditional ImageNet-128 ( also at 8-bit ) showing again the striking impact of the SPN and of multidimensional upscaling on sample quality and setting a fidelity baseline for future methods . 2 MODEL . 2.1 CONVENTIONAL GENERATION ORDERING . A standard AR image model such as the PixelCNN ( van den Oord et al. , 2016b ) generates an HxW colour image starting at the top-left position and ending at the bottom-right position , fully generating the three 8-bit channels of each pixel in a given position : P ( x ) = H∏ h=1 W∏ w=1 { R , G , B } ∏ c P ( xh , w , c|x < ) ( 1 ) where x < corresponds to all previously generated intensity values in the ordering and h , w , and c are row , column , and colour channel indices . The raster scan ordering ( Figure 3 ( a ) ) is conventionally used in AR models . Each conditional distribution P ( xh , w , c|x < ) is parametrized by a deep neural network ( van den Oord et al. , 2016b ) . 2.2 SUBSCALE ORDERING IN IMAGES . We define an alternative ordering that divides a large image into a sequence of equally sized slices and has various core properties . First , it makes it easy to compactly encode long-range dependencies across the many pixels in the large images . It also induces a spatial structure over the original image by aligning the subsampled slices ; this also has an implicit size upscaling side effect . From the perspective of the neural architecture , it makes it possible for the same decoder within the SPN to be consistently applied to all slices , since they are structurally similar ; the smaller slices also allow for self-attention ( Vaswani et al. , 2017 ) in the SPN to be used without local contexts ( Parmar et al. , 2018 ) . We think of the ordering as the two-dimensional analogue of the one-dimensional subscale ordering introduced in Kalchbrenner et al . ( 2018 ) . The subscale ordering is defined as follows : P ( x ) = S∏ i=1 S∏ j=1 H/S∏ h=1 W/S∏ w=1 { R , G , B } ∏ c P ( xi+S∗h , j+S∗w , c|x < ) ( 2 ) where x < corresponds to all previously generated intensity values according to this ordering . Figure 3 ( d ) illustrates the subscale ordering . A scaling factor S is selected and each slice of size H/S×W/S is obtained by selecting a pixel every S pixels in both height and width ; there are thus S2 interleaved slices in the original image , with each specified by its row and column offset ( i , j ) . We sometimes refer to this offset as the “ meta-position ” of a slice . 2.3 SIZE UPSCALING IN SUBSCALE ORDERING . The subscale ordering itself already captures size upscaling implicitly . Analogous to the multi-scale ordering ( van den Oord et al. , 2016b ) , and depicted in 3 ( b ) , we can perform size upscaling explicitly , by training a single slice decoder on subimages and generate the first slice of a subscale ordering from the single slice decoder itself . The rest of the image is then generated according to the subscale ordering by the main network ( see 3 ( e ) ) . The single-slice model can be trained on just the first slices of images , or on slices at all positions in all images given the shared spatial structure among the slices . For this reason , the same SPN that captures the subscale ordering can act simultaneously as a full-blown image model as well as a size upscaling model if initialized with the outputs of a single-slice decoder . A separate formulation of size upscaling is the Parallel Multi-Scale ( Reed et al. , 2017 ) ordering where the pixels in an image are doubled at every stage by distinct neural networks and are generated in parallel without sequentiality ( 3 ( c ) ) . 2.4 DEPTH UPSCALING . Multidimensional upscaling applies upscaling not just in the height and width of the image , but also in the remaining dimension that is channel depth . This is performed in stages such that a network first generates the d1 most significant bits of an image using a conventional or subscale ordering ; then a second network generates the next d2 most significant bits of the image conditioned on all the d1 bits of the image ; and so on to further stages . Using the conventional ordering as basis , the first stage of depth upscaling looks as follows : P ( x : d1 ) = H∏ h=1 W∏ w=1 { R , G , B } ∏ c P ( x : d1h , w , c|x : d1 < ) ( 3 ) Next , a second stage of depth upscaling has the following form , conditioned on the first d1 bits of each channel : P ( xd1 : d2 ) = H∏ h=1 W∏ w=1 { R , G , B } ∏ c P ( xd1 : d2h , w , c |x d1 : d2 < , x : d1 ) ( 4 ) We do not share weights among the networks at different stages of depth upscaling . We note that in depth upscaling bits of lower significance are only generated when the more significants bits at all positions have been generated in a previous stage . Just like for size upscaling from the previous Previous slices in relative position Slice Embedder Masked Decoder Slice n , m Slice n , m Slice 0,0 Slice 0 , k Slice n , m Slice k,0 Slice k , k Slice n , m Slice n , m 1D Transformer over Slice Gated PixelCNN Reshape to 1D Reshape back to 2D ( a ) ( b ) Figure 4 : ( a ) The architecture of a Subscale Pixel Network , with a conditional and a decoding part . ( b ) Scheme of the parts in the decoder itself . section , the goal of multidimensional upscaling is to let the model focus on visually salient bits of an image unaffected by less salient and less predictable bits of the image . Depth upscaling is related to the method underlying the Grayscale PixelCNN that models 4-bit greyscale images subsampled from colored images Kolesnikov & Lampert ( 2016a ) .

Summary: The paper tackles a problem of learning long-range dependencies in images in order to obtain high fidelity images. The authors propose to use a specific architecture that utilizes three main components: (i) a decoder for sliced small images, (ii) a size-upscaling decoder for large image generation, (iii) a depth-upscaling decoder for generating high-res image. The main idea of the approach is slicing a high-res original image and a new factorization of the joint distribution over pixels. In this model various well-known blocks are used like 1D Transformer and Gated PixelCNN. The obtained results are impressive, the generated images are large and contain realistic details.