Semantically-Adaptive Upsampling for Layout-to-Image Translation
Authors: Hao Tang
Nicu Sebe
Venue:  (2021)
Keywords: Feature upsampling
semantically-adaptive
layout-to-image translation

Abstract: We propose the Semantically-Adaptive UpSampling (SA-UpSample), a general and highly effective upsampling method for the layout-to-image translation task. SA-UpSample has three advantages: 1) Global view. Unlike traditional upsampling methods (e.g., Nearest-neighbor) that only exploit local neighborhoods, SA-UpSample can aggregate semantic information in a global view. 
2) Semantically adaptive.  Instead of using a fixed kernel for all locations (e.g., Deconvolution), SA-UpSample enables semantic class-specific upsampling via generating adaptive kernels for different locations. 3) Efficient. Unlike Spatial Attention which uses a fully-connected strategy to connect all the pixels, SA-UpSample only considers the most relevant pixels, introducing little computational overhead. We observe that SA-UpSample achieves consistent and substantial gains on six popular datasets. The source code will be made publicly available.

Under review as a conference paper at ICLR 2021
SEMANTICALLY-ADAPIII LAYOUT-TO- IMAGE TRANSLATION
UPSAMPLING
FOR
Anonymous authors Paper under double-blind review
ABSTRACT
We propose the cnnaanticallyAAaaptii UpSampling (SA-UpSample), general and highly effective upsampling method for the layout-to-image translation task. SA-UpSample has three advantages: 1) Global view. Unlike traditional upsam- pling methods (e.g.. hearest-neighhee UpSample can aggregate semantic information in global view. 2) Semantically adaptive. Instead of using a fixed kernel for all locations (e.g.. Deconvolution), SA- LA-UpSample enables semantic class-specific upsampling via generating adap- tive kernels for different locations. 3) Efficient. Unlike Spatial Attention which uses a fully-connected strategy to connect all the pixels, SA-UpSample only con- siders the most relevant pixels, introducing little computational overhead. We ob- serve that SA-UpSample achieves consistent and substantial gains on six popular datasets. The source code will be made publicly available.
that only exploit local neighborhoods, SA-
INTRODUCTION
The layout-to mnyout-to-image to-image translation task aims to translate input layouts to realistic images (see Fig. I(a),, which have many real-world applications and draw much attention from the community (Park et al., 2019; Liu et al., 2019; Jiang et al., 2020; Tang et al., 2020). For example, Park et al. (2019) propose GauGAN with a novel spatially-adaptive normalization to generate realistic images from semantic layouts. Liu et al. (2019) propose CC-FPSE, which predicts convolutional kernels conditioned on the semantic layout and then generate the images. Tang et al. (2020) propose LGGAN with several local generators for generating realistic small objects. Despite the interesting exploration of these methods, we can still observe artifacts and blurriness in their generated images because they always adopt the nearest-neighhor interpolation to upsample feature maps and then to generate final results.
Feature upsampling is a key operation in the layout-to-image translation task. Traditional upsam- pling methods such as nearest-neighhor, bilinear, and bicubic only consider sub-pixel neighborhood (indicated by white circles in Fig. 1(b)), failing capture the complete semantic information, e.g., the head and body of the dog, and the front part of the car. Learnable upsampling methods such as Deconvolution (Noh et al., 2015) and Pixel Shufile Shi et al. (2016) are able to obtain the global information with larger kernel size, but learns the same kernel (indicated by the white arrows in Fig. 1(c)) across the image, regardless of the semantic information. Other feature enhancement methods such as Spatial Attention (Fu et al., 2019) can learn different kernels (indicated by different of redundant information, i.e., 'grasses'
color arrows in Fig. (d)), but it still inevitable captures and 'soil'. Also, it is prohibitively expensive since it needs to consider all the pixels.
To fix these limitations, we propose a novel Semantically Adaptiv UpSampling (SA-UpSample) for this challenging task in Fig. I(e). Our SA-UpSample dynamically upsamples small subset of relevant pixels based on the semantic information, i.e., the green and the tangerine circles represent the pixels within the dog and the car, respectively. In this way, SA- SA-UpSample is more efficient than both Deconvolution, Pixel Shuffle, and Spatial Attention, and can capture more complete semantic information than traditional upsampling methods such the nearest-neighhor interpolation.
We perform extensive experiments on six popular datasets with diverse scenarios and different image resolutions, i.e., Cityscapes (Cordts al., 2016), ADE20K (Zhou et al., 2017), COCO-Stuff (Cae- sar et al., 2018), DeepPashion (Liu et al., 2016), CelebAMask-HHQ (Lee et al., 2020), and Facades (Tylečck & Šára, 2013). We show that with the help of SA-UpSample, our framework can syn- thesize better results compared several state-of-the-art methods. Moreover, extensive ablation

---

Under review as a conference paper at ICLR 2021
(a) Input Layout
(b) Nearest, etc.
(c) Deconvolution
(d) Spatial Attention
(c) SA-UpSample
Figure I: Comparison with different feature upsampling and enhancement methods on the layout-to- image translation task. Given two locations (indicated by red and megenta squares) in the output feature map f'. our goal is to generate these locations by selectively upsampling several points N (1, k) (indicated by circles) in the input feature map f.
study shows the effectiveness of SA-UpSample against other feature upsampling and enhancement methods for the syout-to-image translation task.
RELATED WORK
Feature Upsampling. Traditional upsampling methods such as nearest-neighbor and bilinear inter- polations use spatial distance and hand-crafted kernels to capture the correlations between pixels. Recently, several deep learning methods such as Deconvolution (Noh et al., 2015) and Pixel Shuffle Shi et al. (2016) are proposed to upsample feature maps using learnable kernels. However, these fixed kernel. Some
methods either exploit semantic information in small neighborhood other works of super-resolution, inpainting, denoising (Mildenhall et al., 2018; Wang et al., 2019; Jo al., 2018; Hu al., 2019) also explore using learnable kernels. However, the settings of these tasks are significantly different from ours, making their methods cannot be used directly.
Layout-to-Iman Image Translation tries convert semantic layouts into realistic images (Park al., 2019; Liu et al., 2019; Jiang et al., 2020; Tang et al., 2020; Zhu et al., 2020a; Ntavelis et al., 2020; Zhu et al., 2020b). Although existing methods have generated good images, we still see unsatisfac- tory aspects mainly in the generated content details and intra-object completions, which we believe mainly due to they always adopt the saieest-neighor interpolation to upsample feature maps and then generate final results. To fix this limitation, we propose a novel comantically-Adaptii Up- Sampling (SA-UpSample) for this task. To the best our knowledge, we are the first to investigate the influence of feature upsampling this challenging task.
3 SEMANTICALL-ADAPIII UPSAMPLING (SA-UPSAMPLE)
An illustration of the proposed Semantically-Aapplie UpSampling (SA-UpSample is shown in Fig. 2, which mainly consists of two branches, i.e., the CemanticallyAAdapiiv Kernel Genera- tion (SAKG) branch predicting upsample kernels according to the semantic information, and the Semantically-AAaatiie Feature Upsampling (SAFU) branch selectively performs the feature upsam- pling based on the kernels learned in SAKG. All components are trained in an end-to-end fashion so that the two branches can benefit from each other.
Specifically, given feature map XxW and an upsample scale s, SA- aims to
produce new feature map f'eRCxHaxWs, For any target location I'=(1', j') in the output f', there is a corresponding source location i=(i.j) i) at the input f. where i= //]] j=(j'/s]. We denote (1, k) as the kxk sub-region of centered at the location I in, i.e., the neighbor of the location I. See Fig. and for illustration.
3.1 SEMANTICALLY-ADAPIII KERNEL GENERATION (SAKG) BRANCH
This branch aims generate maanticallly-daptiio kernel at each location according to the mantic information, which consists four modules, i.c., Feature Channel Compression, Semantic Kernel Generation, Feature Shuffle, and Channel-wise Normalization.
2

---

Under review as a conference paper at ICLR 2021
ceaenntically-Appptiii Feature Upsampling (SAFU) Branch
Feature Spatial
Sliding Local Block
Upsampling Feature Selection
Feature Reshape
Expansion
CxHaxWa
Extraction
Hax
Cx Hax
CxHaxWa
Channel
Feature Sc
Semantic
Kernel
Feature
Shuffle
-S 1 #sxxWs
Channel-wise Normalization
C
Compression
C'xHXW
Generation
A Hax Wa
Semantically-Adepee Kernel Generation (SAKG) Branch
Figure 2: Overview of the proposed SA-UpSample, which consists of two branches, i.e., SAKG and SAFU. SAKG branch aims to generate semantically-adaptii layout. SAFU branch aims to selectively upsample the feature fECx xW to the target one f'ECxHsWWs based on the kernels learned in SAKG, where is the expected upsample scale.
kernels according the input
Feature Channel Compression. This module is used to reduce the network parameters and com- putational cost. Specifically, the input feature f is fed into convolution layer with 1x1 1 kernel to compress the input channel from C' to C², making SA-U SA-UpSample with fewer network parameters.
Semantic Kernel Generation. This module receives the feature feeRC' xxW as input (where H and W denotes the height and width of the feature map) and ties to generate different semantically- adaptive kernels, which can be represented as R x xW where is the semantically-adaptiv upsampling kernel size and 8 the expected upsample scale.
Feature Shuffle. We then feed the feature f& through a feature shuffle layer for rearranging ele- ments in fs. leading to new feature map fI Hs*WW, where =²= represents the learned semantic kernel. Note that the learned semantic kernels are quit different at different locations 1', as shown in Fig. 1 and 3.
Channel-wise Normalization. After that, we perform a channel-wise softmax operation on each semantic kernel fe to obtain the normalized kernel fn, i.e., the sum of the weight values in 122 is equal to 1. In this way, we can guarantee that information from the combination would not explode. Also, the semantically-adaatiee kernels can encode where emphasize or suppress according the semantic information.
3.2 SEMANTICALLY-ADAPII FEATURE UPSAMPLING (SAFU) BRANCH
This branch aims to upsample the input feature f based on the kernel fn learned in the SAKG branch in semantically-adaptiv way, which contains four modules, i.e., Feature Spatial Expansion, Sliding Local Block Extraction, Feature Reshape, and Upsampling Feature Selection.
Feature Spatial Expansion. The input feature f is fed into this module to expand the size of spatial from H x X to HsxW Ws. Specifically, we adopt the nearest interpolation to achieve this process.
Sliding Local Block Extraction. Then the expanded feature fe Cx HsxWss is fed into this module extract sliding local block of each location in fe, leading the new feature fIERCK* HaxWa
Feature Reshape. Thus, we reshape ft by adding a dimension, resulting in a new feature f.EROx x x# In this way, we can do multiplication between the reshaped local block f. and the learned kernel fn-
Upsamling Feature Selection. Finally, the feature map fr and the kernel f., learned in the SAKG branch are fed into the Upsampling Feature Selection module to generate the final feature map f' f'EROx Hxx by an weighted sum manner. The computation process at the location (=( can be expressed as follow,
1/21
[k/2]
Σ
Σ
A(i+p.j (x(
1K/2
q=- 1R-1
In this way, the pixels in the learned kernel fn contributes the upsampled pixel I' differently, based on the semantic information of features instead of the spatial distance of locations. The semantics of the upsampled feature map be stronger than the original one, since the information from
3

---

Under review as a conference paper at ICLR 2021
Input
Generated Image
Learned SemanticalyyAdaapiie Kernels
Figure 3: Visualization of learned semantically semanticall--adptive kernels COCO-Stur In each row, show three representative locations with different color squares in the generated image. The other three images are learned semantically-adaativ kernels for those three locations, with correspond- ing color arrows summarizing the most-attended regions for upsampling the target location. We observe that the network learns to allocate attention according to regions within the same semantic information, rather than just spatial adjacency.
relevant points in local region can be more attended, and the pixels with the same semantic label can achieve mutual gains, improving intra-object semantic consistency.
3.3 WHY DOES THE SA-UPSAMPLE WORK BETTER?
A short answer is that it can better preserve semantic information against common upsampling methods. Specifically, while other upsampling methods such as nearest-neighbor interpolation and Deconvolution are essential parts in almost all the state-of-the-art image generation (Radford et al., 2016) and translation (Park et al., 2019) models, they tend to 'pollute' semantic information when performing feature upsampling since it would inevitably incorporate contaminating information from irrelevant regions (see Fig. 1).
In contrast, the proposed SA-UpSample performs feature upsampling by using itself, i.c., uses the pixels belonging to the same semantic label to upsample the feature maps. Hence, the generator can better preserve semantic information. It enjoys the benefit of feature upsampling without losing the input semantic information. In Fig, 3, we show some examples of the learned semantically-adaative kernels. We can easily observe that the proposed SA-UpSample upsamples features by leveraging complementary features in the regions of the same semantic information than local regions of fixed shape to generate consistent objects/scenarios, further confirming our motivations.
3.4 OPTIMIZATION OBJECTIVE AND TRAINING DETAILS
We follow GauGAN (Park et al., 2019) and use three losses our training objective, i.e.. C=Agon A Cy Cp. where Lyan. Ly and Cp represent adversarial, discriminator feature matching, and perceptual loss, respectively. We set =1 Ay=10, and >p=10 in our experi- ments. We use the multi-scale discriminator (Park et al., 2019) as our discriminator. Moreover, set C'=64, k=5 and s=2 in the proposed SA-UpSample. We replace the upsampling layers in GauGAN with our SA-UpSample layers. The proposed method is implemented by using Py PyTocc (Paszke al., 2019). We conduct the experiments on NVIDIA DGXI with 32GB V 1000 GPUs.
EXPERIMENTS
Datasets. We first follow GauGAN (Park al., 2019) and conduct experiments Cityscapes (Cordts et al., 2016), ADE20K (Zhou et al., 2017), and COCO-Stuff (Caesar et al., 2018). Then conduct experiments three more datasets with diverse scenarios. Facades (Tyleček & Šára, 2013) contains different city images with diverse architectural styles. The training and test set
4

---

Under review as a conference paper at ICLR 2021
hout
OC .PSE
StudWW
-SA UpSample
Ground Truhh
(a) Cityscapes.
(b) Facades.
Figure 4: Qualitative comparison on Cityscapes and Facades. From left to right: Input, CC- FPSE (Liu et al., 2019), GauGAN (Park et al., 2019), GauGAN+SA-UpSampp (Ours), and GT.
Input
GauGAN
Sam-le
Ground Truth
Input
GauGAN
.SA-UpSample
Ground
(a) DeepFashion.
(b) CelebAMask-HH Mask-HQ.
Figure 5: Qualitative comparison on DeepFashion and CelebAMask-HO, From left right: Input, GauGAN (Park et al., 2019), BRuGAN+SA-UpSampe (Ours), and GT.
Table I: User study. The numbers indicate the percentage of users who favor the results of our
method (i.e., GauGAN+SA-UpSampee over GauGAN.
AMT
Cityscapes
ADE20K
COCO-Stuff
DeepHashion
Facades
CelebAMask-HO
Ours vs. GauGAN
63.8
65.7
62.4
60.1
58.3
70.5
sizes 378 and 228, respectively. We resize the images to 512x512 for high high-resolution layout- to-image translation tasks. 2) CelebAMask-HO (Lee et al., 2020) contains face images with 19 semantic facial attributes. The training and test set sizes resize the images to 512x 512. 3) DeepFashion (Liu et al., 2016) contains human body images. The training and test sizes are 30,000 and 2,247, respectively. We resize the images to x56x256, and use well-trained model (Li et al., 2019) to extract input semantic layouts for this dataset.
14,183 and 2,842, respectively. We also
Evaluation Metrics. We follow GauGAN (Park al., 2019) and use mean Intersection- -veeronno-vve--eee veionUnion (mloU), pixel accuracy (Acc), and Fréchet Inception Distance (FID) (Heusel et al., 2017) as the evaluation metrics on Citysacapes, ADE20K, and COCO-Stuff. For DeepFashion, CelehAMask- HQ, and Facades, we use FID and LPIPS (Zhang et al., 2018) as the evaluation metrics.
4.1 COMPARISONS WITH STATE-OF-THE-AAT
Qualitative Comparisons. We first compare SA-UpSample with GauGAN (Park et al., 2019) on Desponn, CelebAMask-HOQ and Facades. Note that used the source code provided by the
5

---

Under review as a conference paper at ICLR 2021
Input
CC-FPSE
GauGAN
SSA-UpSample Ground Truth
Input
CC-FPSE
GauGAN
SS-UUSample Ground Truff
(a) ADE20K.
(b) COCO-Stuff.
Figure 6: Qualitative comparison ADE20K and cOcO-Stuff. From left to right: Input, CC- FPSE (Liu et al., 2019), GauGAN (Park et al., 2019), GuUGAN+SA-Upample (Ours), and GT.
GauGAN
SA-UpSammpl
GauGAN
USA.ppSample
Ground Trush
Input
GauGSAN
-SA-ppSample
Cityscapes.
(b) CelebAMask HQ.
Figure 7: (a) Visualization of generated semantic maps compared with those from GauGAN (Park ct al., 2019) on Cityscapes. Most improved regions are highlighted the ground truths with white dash boxes. (b) Comparison in zoomed-in manner CelebAMask-HO.
Table 2: Quantitative comparison on DeepFashion, Facades, and CelebA
DecpFashion
Facades
CelebAMask-
Method
FID J
LPIPS
FID 1
LPIPS |
FID
LPIPS
GauGAN (Park et al., 2019)
22.8
0.2476
116.8
0.5437
42.2
0.4870
+SA- +SA-UpSampe UpSample (Ours)
20.8
0.2446
112.4
0.5387
33.6
0.4788
authors to generate the results of GauGAN on these three datasets for fair comparisons. Visualization results are shown in Fig. 4(b) and 5. We can that the model with SA-UpSample generates more photo-realistic results than the original GauGAN, Moreover, we compare GauGAN and the proposed method in zoomed-in manner on CelebAMask--HO in Fig. 7(b). We can see that the model with our SA-UpSample can generate more vivid content than the original GauGAN model, further validating effectiveness
AA-pplepee. Lastly, we compare the proposed method with GauGAN and CC-FPSE (Liu et al., 2019) on Cityscapes, ADE20K, and COCO-Stuff. Comparison results are shown in Fig. 4(a) and 6. We can see that our method produces more clear and visually plausible results than both leading methods, further demonstrating design motivation.
User Study, We follow the same evaluation protocol GauGAN and also perform user study. The results compared with the original GauGAN shown in Table 1. We that users strongly favor the results generated by our proposed method on all datasets, further validating that the generated images by upsampling method more cholo-realistic.
Quantitative Comparisons. Although the user study is more suitable for evaluating the quality of the generated images, we also follow GauGAN and use mloU, Acc, FID, and LPIPS for quantitative

---

Under review as a conference paper at ICLR 2021
NearosiNeghht Upsomplin
Blineer Upsomelin
Bkupio Upcampinn
Deconsutton
Plol Shuffs
SA-UpSammle (Oun)
Figure 8: Qualitative comparison of different upsampling methods on Cityscapes. Key differences are highlighted by red boxes.
Table 3: Quantitative comparison Cityscapes, ADE20K, and COCO-Stuff: The results of other methods are reported from their papers.
Cityscapes
ADE20K
COCO-Stuff
Method
mloU
Acc f
FID
mloU
Acc
FID
mloU
Acc
FID
CRN (Chen Koltun, 2017)
52.4
77.1
104.7
22.4
68.8
73.3
23.7
40.4
70.4
SIMS (Qi at. 2018)
47.2
75.5
49.7
Pix2pixHID (Wang et al., 2018)
58.3
81.4
95.0
20.3
69.2
81.8
14.6
45.7
111.5
CC-FPSE (Liu et al., 2019)
65.5
82.3
54.3
43.7
82.9
31.7
41.6
70.7
19.2
PIS (Dundar et al., 2020)
64.8
82.4
96.4
38.6
69.0
28.8
TSIT (Jiang et al,, 2020)
65,9
82.7
39.2
38.6
80.8
31.6
LGGAN (Tang et ol., 2020)
68.4
83.0
57.7
41.6
81.8
31.6
GauGAN (Park et al., 2019)
62.3
81.9
71.8
38.5
79.9
33.9
37.4
67.9
22.6
+Smple (Ours)
65.5
82.5
48.3
39.8
80.7
32.0
39.0
69.1
20.1
evaluation. The results compared with several leading methods are shown in Tables 2 and 3. Firstly, observe that the model with SA-UpSample achieves better results compared with GauGAN on DeepFashion, CelebAMask-HOQ. and Facades in Table 2. Moreover, we can see that our method (i.e.. GauGAN+SA- --pSppll)) achieves competitive results compared with other leading methods on Cityscapes, ADE20K, and COCO- Souff-tuffiii in Table 3. Notably, LGGAN (Tang et al., 2020) achieves better results than our method, however, trains each local generator for each semantic class, leading to more parameters and the training time will be significantly increased when increasing the number of semantic classes.
Visualization of Generated Semantic Maps. We follow GauGAN and adopt the pretrained DRN- D-105 (Yu et al., 2017) on the generated Cityscapes images to produce semantic maps. The results compared with those produced by GauGAN are shown in Fig. 7(a). We see that the method with our proposed SA-UpSemple generates more semantically
results than the original GauGAN.
4.2 ABLATION STUDIES
Baselines. We conduct an extensive ablation study on Cityscapes to evaluate the effectiveness of the proposed SA- SA-UpSample. As shown in Table 4, B1, B2, and B3 are three traditional upsampling methods. B4 and B5 are two learnable upsampling methods. B6 the Spatial Attention module proposed in (Fu et al., 2019). B7 is proposed SA-UpSample
Ablation Analysis. We first compare the proposed SA-U SA-UpSample with different upsampling strate- gies (i.e., B1-B5). The results of the ablation study are shown in Table 4 and Fig. 8. We can see that the proposed SA-UpSample achieves significantly better FID than other upsampling methods in Ta- ble 4, indicating that the design of effective upsampling methods is critical for this challenging task. We also observe that the proposed SA-UpSample generates more photo-realistic and semantically- consistent results with fewer artifacts than other upsampling methods in Fig. 8. Moreover, add

---

Under review as a conference paper at ICLR 2021
Inage
Irpu
Image
image
Irput
teage
Change Building
Change Buiding to Sky
Adi
Add zebras
(a) Cityscapes.
(b) ADE20K.
(c)
COCO-Stuff.
Generated
Geeosated
Ferove Earrings
Remove Eyegiassee
Muti-Mocal Synthesis
(d) CelebAMMask-IIQ.
(e) DeepFashion.
Figure Exemplar applications of the proposed method on different datasets.
the Spatial Attention module (Fu et al., 2019) GauGAN, obtaining 56.2 on FID. We can see that our method still significantly outperforms Spatial Attention.
Comparisons of Model Parameters. We also compare the number of gen- erator parameters with different base- lines. The results are shown in Ta- ble 4. Traditional upsampling meth-
Table 4: Quantitative comparison of different feature up-
sampling and enhancement methods on Cityscapes.
No.
Method
FID
Params
B1
Nearest-Neighhor Upsampling
58.7
93.0M
ods B1-B3) have the same number of parameters. Also, we can see that the proposed SA-UpSample achieves perior model capacity than the learn- able upsampling methods (i.e., B4 and B5) and Spatial Attention.
B2
Bilinear Upsampling
52.9
93.0M
Bicubic Upsampling
54.4
93.0M
B4
Deconvolution (Noh et at., 2015)
54.0
98.6M
B5
Pixel Shuffle (Shi et al., 2016)
59.1
143.2M
B6
Spatial Attention (Fu et al., 2019)
56.2
97.4M
SA-UpSample (Ours)
48.3
93.4M
Generalization of SA-UpSample. Our SA- LA-UpSemple general and be seamlessly integrated into existing GANs. Thus, to validate the generalization ability of SA-UpSample, we further conduct more experiments on Facades. Specifically, we replace the upsampling layers CC-FPSE (Liu et al., 2019) with our SA-UpSample. We observe that the CC-FPSE model with SA-UpSample further decreases FID from 93.98 90.49, validating the generalization ability of SA-UpSample.
4.3 APPLICATIONS
Semantic Manipulation. Our model also supports semantic manipulation. For instance, we can replace the building with trees in Fig. 9(a), insert a bed in the room in Fig. 9(b), add a few zebras to the grass in Fig. 9c), or remove earrings and eyeglasses from faces in Fig. 9()). These applications provide users more controllability during the translation process.
Multi-Modes Synthesis. By using random vector the input of the generator, our model can perform multi-modal synthesis (Zhu et al., 2017). The results are shown in Fig. 9e)). We can see that our model generates different outputs from the same input layout.
CONCLUSION
We present novel Semantically-Adape UpSampling (SA- SSA-UpSamlle), highly effective upsam- pling method for the eeyout-to-mmage translation task. We observe that SA-UpSample consistently boosts the generation performances on six datasets with diverse scenarios. More importantly, SA- UpSample introduces little computational overhead and can be readily plugged into existing GAN architectures to solve other tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In ICCV, 2017.7 7
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 1, 4
Aysegul Dundar, Karan Sapra, Guilin Liu, Andrew Tao, and Bryan Catanzaro. Panoptic-based image synthesis. In CVPR, 2020. 7
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 1, 7,
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPS, 2017. 5
Xuecai Hu, Haoyuan Mu, Xiangyu Zhang. Zilei Wang, Tieniu Tan, and Jian Sun. Meta-sr: A magnification-arbitaa network for super-resolution. In CVPR, 2019. 2
Liming Jiang, Changxu Zhang. Mingyang Huang, Chunxiao Liu, Jianping Shi, and Chen Change Loy. Tsit: A simple and versatile framework for image-to-image translation. In ECCV, 2020. 1, 2,7
Younghyun Jo, Scoung Wug Oh, Jacyeon Kang, and Scon Joo Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. CVPR, 2018.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020. 1,5
Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing, arXiv preprint arX1v:1910.09777, 2019. 5
Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, et al. Learning to predict layout-to-image
conditional convolutions for semantic image synthesis. In NeurIPS, 2019. 1, 2, 5, 6, 8,8
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, 2016. 1,5
Ben Mildenhall, Jonathan T Barron, Jiawen Chen. Dillon Sharlet, Ren Ng, and Robert Carroll. Burst denoising with kernel prediction networks. In CVPR, 2018. 2
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for seman-
tic segmentation. In ICCV, 2015. 1, 2, 8
Evangelos Ntavelis, Andrés Romero, Iason Kastanis, Luc Van Gool, and Radu Timofte. Sesame:
Semantic editing of scenes by adding, manipulating erasing objects. In ECCV, 2020. 2
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019. 1, 2, 5,
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi Semi-parametio image synthesis. In CVPR, 2018.

---

Under review as a conference paper at ICLR 2021
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016. 4
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In CVPR, 2016. 1, 2, 8
Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe. Local class-specific and global image- level generative adversarial networks for semantic-guizn guided scene generation. In CVPR, 2020. 1, 7.7
Radim Tyleček and Radim Šára. Spatial pattern templates for recognition of objects with regular structure. In GCPR, 2013. 1,4
Jiaqi Wang, Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, and Dahua Lin. Carafe: Content-awarg reassembly of features. In CVPR, 2019. 2
Ting-Chur Wang, Ming-Ye Yu Liu, Jun Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In CVPR, 2017. 7
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features perceptual metric. In CVPR, 2018. 5
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 1,4
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-imagee translation. In NeurIPS, 2017. 8
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region-adaptive normalization. In CVPR, 2020a. 2
Zhen Zhu, Zhiliang Xu, Ansheng You, and Xiang Bai. Semantically multi-modal image synthesis. In CVPR, 2020b.
10


Reviews:
[{"review_id": "muVc4QLdZ3A", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a module named semantically-adaptive upsampling (SA-UpSample) to achieve layout-to-image translation. The proposed method is able to aggregate semantic information in the layout input and adaptively conducts class-specific upsampling in the translation process. Experiments on six datasets demonstrate the effectiveness of the proposed mehtod. However, there are still several issues to be addressed.\n\n1. Novelty. The proposed semantically-adaptive kernel generation branch and semantically-adpative feature upsampling branch share very similar idea as the filter generation network and dynamic filtering layer in [a], and adaptive interpolation kernel generation and adaptive image resampling in [b], except that the input of generation branch is layout map.\n\n[a] B. De Brabandere et al., Dynamic Filter Networks, NIPS 2016.\n\n[b] X. Jia et al., Super-Resolution with Deep Adaptive Image Resampling, CoRR abs/1712.06463, 2017\n\n2. What if applying the proposed module to also the place after conv layers that corresponds to 1x upsampling? Will performance be further improved.\n\n3. Given the experiments in Fig. 5 (comparing 3rd column and the 4th column), I wonder whether there is any overfitting with the results.\n\n4. It only shows that it is effective in image translation but does not mention how efficient the proposed module is.\n\n5. In Table 3, it seems that the semantic consistency with layout input is worse than TSIT and LGGAN. It is not clear why the row of the proposed method is marked bold although it is not the best in terms of some metrics.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EuNEqGztJmZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the Semantic-Adaptive UpSampling method to do feature upsample in layout-to-image translation task. The SA-UpSample module exploit the semantic information to learn adaptive upsample kernels for different input features. The proposed method has the advantage of global view, semantically-adaptive and efficient. Extensive experimental results are shown to prove the effectiveness of the proposed method. The paper is well organized and easy to follow.\n\nStrength:\n\n(+) The experiments are sufficient and extensive. The proposed method is evaluated on many different datasets and generate satisfactory results in all of these datasets. Both qualitative and Quantitative results are shown, along with user study and visualization. Comparison results with different upsampling methods and state-of-the-art methods guarantee the effectiveness of the proposed methods.\n\n(+) This paper is well organized and the description of the proposed method is easy to understand.\n\nWeakness:\n\n(-) How to determine some hyper-parameters of the proposed method is not clear. E.g. C’, k and s are very important hyper-parameters of SA-UpSample. The authors set C’=64, k=5 and s=2. Are they determined by manual or from experimental results? \n\n(-) The proposed method is efficient because it has a feature channel compression step. If we ignore the computational complexity and enlarge the channel number of this step (i.e. enlarge C’), will the proposed method generate better results (images with more realistic)? Whether the authors choose a trade off between effectiveness and efficiency by setting a relatively small C’?", "extra_scores": {}, "extra_text": {}}, {"review_id": "ASEhtbaYljc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper presents a content-aware upsampling approach for layout-to-image translation problems. The core idea is to learn a spatially varying upsampling kernel. The paper applied the proposed upsampling method to GauGAN and showed improved performance over the nearest neighbor upsampling operators on six layout-to-image datasets.\n\n**Strength**\n+ The results clearly show that the proposed upsampling operator can improve the visual quality over the baseline nearest neighbor upsampling. The quantitative results also support the effectiveness of the proposed upsampling operation. \n+ The motivation and the differences over alternative methods is clear (Figure 1).\n+ The overview figure helps readers understand the high-level ideas. \n\n**Weakness**\n\n== Exposition ==\n\nThe exposition of the proposed method can be improved. \nFor examples, \n-\tit’s unclear how the “Semantic Kernel Generation” is implemented. I can probably guess this step is essentially a 1 x 1 convolution, but it would be better to fill in the details. \n-\tIn the introduction, the paper mentioned that the translated images often contain artifacts due to the nearest-neighbor upsampling. Yet, in the Feature Spatial Expansion step it also used nearest-neighbor interpolation. This needs some more justification.\n-\tFor Figure 3, it’s unclear what the learned semantically adaptive kernel visualization means. At each upsampling layer, the kernel is only K x K. \n-\tAt the end of Section 2, I do not understand what it means by “the semantics of the upsampled feature map can be stronger than the original one”. \n-\tThe proposed upsampling layer is called “semantically aware”. However, I do not see anything that’s related to the semantics other than the fact that the input is a semantic map. I would suggest that this should be called “content aware” instead. \n\n== Technical novelty ==\n-\tMy main concern about the paper lies in its technical novelty. There have been multiple papers that proposed content aware filter. As far as I know, the first one is [Xu et al. 2016].\nJia, Xu, et al. \"Dynamic filter networks.\" Advances in neural information processing systems. 2016.\n\n- The work most relevant to this paper is the CARAFE [Wang et al. ICCV 2019], which can be viewed as a special case of the dynamic filter network for feature upsampling. By comparing Figure 2 in this paper and Figure 2 in the CARAFE paper [Wang et al. ICCV 2019], it seems to me that the two methods are * the same*. The only difference is the application of the layout-to-image task. The paper in the related work section suggests, “the settings of these tasks are significantly different from ours, making their methods cannot be used directly.” I respectfully disagree with this statement because both methods take in a feature tensor and produce an upsampled feature tensor. I believe that the application would be straightforward without any modification. Given the similarity to prior work, it would be great for the authors to (1) describe in detail the differences (if any) and (2) compare with prior work that also uses spatially varying upsampling kernel.\n\nMinor:\n-\tCARAFE: Content-Aware ReAssembly of Features. The paper is in ICCV 2019, not CVPR 2019\n\nIn sum, I think the idea of the spatially adaptive upsampling kernel is technically sound. I also like the extensive evaluation in this paper. However, I have concerns about the high degree of similarity with the prior method and the lack of comparison with CARAFE. \n\n** After discussions **\n\nI have read other reviewers' comments. Many of the reviewers share similar concerns regarding the technical novelty of this work. I don't find sufficient ground to recommend acceptance of this paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0aRleGLPEHb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes a semantically-adaptive upsampling approach for layout-to-image translation. It uses the semantic label map to predict spatially-adaptive upsampling kernels for feature map upsampling.  Compared with traditional upsampling methods, it has a larger receptive field to focus on not only nearby pixels, but also semantically-related pixels at a longer distance. Experiments are conducted on Cityscapes, ADE20K, COCO-Stuff, DeepFashion, CelebAMask-HQ, and Facades datasets, and the proposed approach achieves better results compared with the baseline.\n\n##########################################################################\n\nPros: \n\n1. The semantically-adaptive upsampling approach considers the semantic information for upsampling. It has a larger receptive field and can take far-away pixels which are semantically related for upsampling, so that it can better preserve the semantic consistency within the instance or stuff with the same semantic label.\n\n2. The proposed semantically-adaptive upsampling is efficient compared with spatial attention or prediction convolution kernels.\n\n3. The writting and explanations are clear.\n \n##########################################################################\n\nCons: \n\n1. In Figure.2, both the SAFU branch and the SAKG branch take the feature f as input. Based on my understanding of this paper, the input of the SAKG branch should be the semantic layout map, rather than the feature map f. I think this figure is a little confusing and misleading.\n\n2. The novelty of the semantically-adaptive upsampling is limited. On the one hand, semantic-adaptive operations have been proposed in previous work SPADE (semantically-adaptive normalization) and CC-FPSE (semantically-adaptive convolution). The difference from the conditional convolution in CC-FPSE is that the semantically-adaptive upsampling does not learn the feature transformation across channels, so the parameter size to be predicted is smaller. On the other hand, content-adaptive upsampling operations have also been explored in previous work CARAFE [A]. Especially the architecture design of this paper is almost the same as CARAFE (even with the same submodule names such as \"channel compression\" and \"channel-wise normalization\"). The only difference from CARAFE is that the proposed method uses the semantic label maps rather than the feature maps to predict the upsampling kernels.\n\n[A] CARAFE: Content-Aware ReAssembly of FEatures, ICCV 2019\n\n3. The authors claim that the proposed upsampling approach can aggregate information in a global view, but the receptive field of the upsampling layer is still limited to the kernel size of the upsampling kernels. The kernel size used in this paper is k=5. Have the authors experiment using different kernel sizes to see the effect of kernel size on the image generation performance?\n\n4. The visual results do not seem to be significantly better than previous methods.\n\n##########################################################################\n\nReasons for score: \n\nMy main reason for rating this paper as below the acceptance threshold is that the noveltly is limited (see the 2nd point in Cons for explanation), and there is not a large improvement on the quality of the synthesized images.\n\n##########################################################################\n\nQuestions during the rebuttal period: \n\nPlease address my concerns in the Cons part.\n\n##########################################################################\n\nPost-rebuttal:\n\nI have read other reviewers' comments. Since the authors did not provide feedback to our reviews, I would change my score from 5 to 4.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an upsampling layer design for converting layouts to images. Three reviewers rate the paper below the bar, while one reviewer rates the paper marginally above the bar. The main concern that several reviewers raise is the novelty. Particularly, R1 and R3 point out that the proposed method shares great similarity to CARAFE [Wang et al. ICCV 2019]. The AC agrees with the reviewers.

Decision: Reject