Semantically-Adaptive Upsampling for Layout-to-Image Translation

1 INTRODUCTION . The layout-to-image translation task aims to translate input layouts to realistic images ( see Fig . 1 ( a ) ) , which have many real-world applications and draw much attention from the community ( Park et al. , 2019 ; Liu et al. , 2019 ; Jiang et al. , 2020 ; Tang et al. , 2020 ) . For example , Park et al . ( 2019 ) propose GauGAN with a novel spatially-adaptive normalization to generate realistic images from semantic layouts . Liu et al . ( 2019 ) propose CC-FPSE , which predicts convolutional kernels conditioned on the semantic layout and then generate the images . Tang et al . ( 2020 ) propose LGGAN with several local generators for generating realistic small objects . Despite the interesting exploration of these methods , we can still observe artifacts and blurriness in their generated images because they always adopt the nearest-neighbor interpolation to upsample feature maps and then to generate final results . Feature upsampling is a key operation in the layout-to-image translation task . Traditional upsampling methods such as nearest-neighbor , bilinear , and bicubic only consider sub-pixel neighborhood ( indicated by white circles in Fig . 1 ( b ) ) , failing to capture the complete semantic information , e.g. , the head and body of the dog , and the front part of the car . Learnable upsampling methods such as Deconvolution ( Noh et al. , 2015 ) and Pixel Shuffle Shi et al . ( 2016 ) are able to obtain the global information with larger kernel size , but learns the same kernel ( indicated by the white arrows in Fig . 1 ( c ) ) across the image , regardless of the semantic information . Other feature enhancement methods such as Spatial Attention ( Fu et al. , 2019 ) can learn different kernels ( indicated by different color arrows in Fig . 1 ( d ) ) , but it still inevitable captures a lot of redundant information , i.e. , ‘ grasses ’ and ‘ soil ’ . Also , it is prohibitively expensive since it needs to consider all the pixels . To fix these limitations , we propose a novel Semantically-Adaptive UpSampling ( SA-UpSample ) for this challenging task in Fig . 1 ( e ) . Our SA-UpSample dynamically upsamples a small subset of relevant pixels based on the semantic information , i.e. , the green and the tangerine circles represent the pixels within the dog and the car , respectively . In this way , SA-UpSample is more efficient than both Deconvolution , Pixel Shuffle , and Spatial Attention , and can capture more complete semantic information than traditional upsampling methods such as the nearest-neighbor interpolation . We perform extensive experiments on six popular datasets with diverse scenarios and different image resolutions , i.e. , Cityscapes ( Cordts et al. , 2016 ) , ADE20K ( Zhou et al. , 2017 ) , COCO-Stuff ( Caesar et al. , 2018 ) , DeepFashion ( Liu et al. , 2016 ) , CelebAMask-HQ ( Lee et al. , 2020 ) , and Facades ( Tyleček & Šára , 2013 ) . We show that with the help of SA-UpSample , our framework can synthesize better results compared to several state-of-the-art methods . Moreover , an extensive ablation study shows the effectiveness of SA-UpSample against other feature upsampling and enhancement methods for the layout-to-image translation task . 2 RELATED WORK . Feature Upsampling . Traditional upsampling methods such as nearest-neighbor and bilinear interpolations use spatial distance and hand-crafted kernels to capture the correlations between pixels . Recently , several deep learning methods such as Deconvolution ( Noh et al. , 2015 ) and Pixel Shuffle Shi et al . ( 2016 ) are proposed to upsample feature maps using learnable kernels . However , these methods either exploit semantic information in a small neighborhood or use a fixed kernel . Some other works of super-resolution , inpainting , denoising ( Mildenhall et al. , 2018 ; Wang et al. , 2019 ; Jo et al. , 2018 ; Hu et al. , 2019 ) also explore using learnable kernels . However , the settings of these tasks are significantly different from ours , making their methods can not be used directly . Layout-to-Image Translation tries to convert semantic layouts into realistic images ( Park et al. , 2019 ; Liu et al. , 2019 ; Jiang et al. , 2020 ; Tang et al. , 2020 ; Zhu et al. , 2020a ; Ntavelis et al. , 2020 ; Zhu et al. , 2020b ) . Although existing methods have generated good images , we still see unsatisfactory aspects mainly in the generated content details and intra-object completions , which we believe is mainly due to they always adopt the nearest-neighbor interpolation to upsample feature maps and then generate final results . To fix this limitation , we propose a novel Semantically-Adaptive UpSampling ( SA-UpSample ) for this task . To the best of our knowledge , we are the first to investigate the influence of feature upsampling on this challenging task . 3 SEMANTICALLY-ADAPTIVE UPSAMPLING ( SA-UPSAMPLE ) . An illustration of the proposed Semantically-Adaptive UpSampling ( SA-UpSample ) is shown in Fig . 2 , which mainly consists of two branches , i.e. , the Semantically-Adaptive Kernel Generation ( SAKG ) branch predicting upsample kernels according to the semantic information , and the Semantically-Adaptive Feature Upsampling ( SAFU ) branch selectively performs the feature upsampling based on the kernels learned in SAKG . All components are trained in an end-to-end fashion so that the two branches can benefit from each other . Specifically , given a feature map f∈RC×H×W and an upsample scale s , SA-UpSample aims to produce a new feature map f ′∈RC×Hs×Ws . For any target location l′= ( i′ , j′ ) in the output f ′ , there is a corresponding source location l= ( i , j ) at the input f , where i=bi′/sc , j=bj′/sc . We denote N ( l , k ) as the k×k sub-region of f centered at the location l in , i.e. , the neighbor of the location l. See Fig . 1 and 2 for illustration . 3.1 SEMANTICALLY-ADAPTIVE KERNEL GENERATION ( SAKG ) BRANCH . This branch aims to generate a semantically-adaptive kernel at each location according to the semantic information , which consists of four modules , i.e. , Feature Channel Compression , Semantic Kernel Generation , Feature Shuffle , and Channel-wise Normalization . Feature Channel Compression . This module is used to reduce the network parameters and computational cost . Specifically , the input feature f is fed into a convolution layer with 1×1 kernel to compress the input channel from C to C ′ , making SA-UpSample with fewer network parameters . Semantic Kernel Generation . This module receives the feature fc∈RC ′×H×W as input ( where H and W denotes the height and width of the feature map ) and ties to generate different semanticallyadaptive kernels , which can be represented as fk∈Rk 2s2×H×W , where k is the semantically-adaptive upsampling kernel size and s is the expected upsample scale . Feature Shuffle . We then feed the feature fk through a feature shuffle layer for rearranging elements in fk , leading to a new feature map fs∈Rk 2×Hs×Ws , where k2=k×k represents the learned semantic kernel . Note that the learned semantic kernels are quit different at different locations l′ , as shown in Fig . 1 and 3 . Channel-wise Normalization . After that , we perform a channel-wise softmax operation on each semantic kernel fs to obtain the normalized kernel fn , i.e. , the sum of the weight values in k2 is equal to 1 . In this way , we can guarantee that information from the combination would not explode . Also , the semantically-adaptive kernels can encode where to emphasize or suppress according to the semantic information . 3.2 SEMANTICALLY-ADAPTIVE FEATURE UPSAMPLING ( SAFU ) BRANCH . This branch aims to upsample the input feature f based on the kernel fn learned in the SAKG branch in a semantically-adaptive way , which contains four modules , i.e. , Feature Spatial Expansion , Sliding Local Block Extraction , Feature Reshape , and Upsampling Feature Selection . Feature Spatial Expansion . The input feature f is fed into this module to expand the size of spatial from H×W to Hs×Ws . Specifically , we adopt the nearest interpolation to achieve this process . Sliding Local Block Extraction . Then the expanded feature fe∈RC×Hs×Ws is fed into this module to extract sliding local block of each location in fe , leading to the new feature fl∈RCk 2×Hs×Ws . Feature Reshape . Thus , we reshape fl by adding a dimension , resulting in a new feature fr∈RC×k 2×Hs×Ws . In this way , we can do multiplication between the reshaped local block fr and the learned kernel fn . Upsamling Feature Selection . Finally , the feature map fr and the kernel fn learned in the SAKG branch are fed into the Upsampling Feature Selection module to generate the final feature map f ′∈RC×Hs×Ws by an weighted sum manner . The computation process at the location l= ( i , j ) can be expressed as follow , f ′ = bk/2c∑ p=−bk/2c bk/2c∑ q=−bk/2c fr ( i+ p , j + q ) × fn ( p , q ) . ( 1 ) In this way , the pixels in the learned kernel fn contributes to the upsampled pixel l′ differently , based on the semantic information of features instead of the spatial distance of locations . The semantics of the upsampled feature map can be stronger than the original one , since the information from relevant points in a local region can be more attended , and the pixels with the same semantic label can achieve mutual gains , improving intra-object semantic consistency . 3.3 WHY DOES THE SA-UPSAMPLE WORK BETTER ? . A short answer is that it can better preserve semantic information against common upsampling methods . Specifically , while other upsampling methods such as nearest-neighbor interpolation and Deconvolution are essential parts in almost all the state-of-the-art image generation ( Radford et al. , 2016 ) and translation ( Park et al. , 2019 ) models , they tend to ‘ pollute ’ semantic information when performing feature upsampling since it would inevitably incorporate contaminating information from irrelevant regions ( see Fig . 1 ) . In contrast , the proposed SA-UpSample performs feature upsampling by using itself , i.e. , it uses the pixels belonging to the same semantic label to upsample the feature maps . Hence , the generator can better preserve semantic information . It enjoys the benefit of feature upsampling without losing the input semantic information . In Fig . 3 , we show some examples of the learned semantically-adaptive kernels . We can easily observe that the proposed SA-UpSample upsamples features by leveraging complementary features in the regions of the same semantic information than local regions of fixed shape to generate consistent objects/scenarios , further confirming our motivations . 3.4 OPTIMIZATION OBJECTIVE AND TRAINING DETAILS . We follow GauGAN ( Park et al. , 2019 ) and use three losses as our training objective , i.e. , L=λganLgan+λfLf+λpLp , where Lgan , Lf and Lp represent adversarial , discriminator feature matching , and perceptual loss , respectively . We set λgan=1 , λf=10 , and λp=10 in our experiments . We use the multi-scale discriminator ( Park et al. , 2019 ) as our discriminator . Moreover , we set C ′=64 , k=5 and s=2 in the proposed SA-UpSample . We replace the upsampling layers in GauGAN with our SA-UpSample layers . The proposed method is implemented by using PyTorch ( Paszke et al. , 2019 ) . We conduct the experiments on NVIDIA DGX1 with 8 32GB V100 GPUs .

Summary: This paper proposes a semantically-adaptive upsampling approach for layout-to-image translation. It uses the semantic label map to predict spatially-adaptive upsampling kernels for feature map upsampling.  Compared with traditional upsampling methods, it has a larger receptive field to focus on not only nearby pixels, but also semantically-related pixels at a longer distance. Experiments are conducted on Cityscapes, ADE20K, COCO-Stuff, DeepFashion, CelebAMask-HQ, and Facades datasets, and the proposed approach achieves better results compared with the baseline.