Learning Spatially-Adaptive Squeeze-Excitation Networks for Image Synthesis and Image Recognition
Authors: Jianghao Shen
Tianfu Wu
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Squeeze Excitation
Spatially-Adaptive Squeeze Excitation
Low-shot Image Synthesis
Image Classification

Abstract: Learning light-weight yet expressive deep networks in both image synthesis and image recognition remains a challenging problem. Inspired by a more recent observation that it is the data-specificity that makes the multi-head self-attention (MHSA) in the Transformer model so powerful, this paper proposes to extend the widely adopted light-weight Squeeze-Excitation (SE) module to be spatially-adaptive to reinforce its data specificity, as a convolutional alternative of the MHSA,  while retaining the efficiency of SE and the inductive basis of convolution. It presents two designs of spatially-adaptive squeeze-excitation (SASE) modules for image synthesis and image recognition respectively. For image synthesis tasks, the proposed SASE is tested in both low-shot and one-shot learning tasks. It shows better performance than prior arts. For image recognition tasks,  the proposed SASE is used as a drop-in replacement for convolution layers in ResNets and achieves much better accuracy than the vanilla ResNets, and slightly better than the MHSA counterparts such as the Swin-Transformer and Pyramid-Transformer in the ImageNet-1000 dataset, with significantly smaller models.

Learning Spatiall--dapttive Squeeze-Excitation Networks for Image Synthesis and Image Recognition
Anonymous Author(s)
Affiliation
Address
email
Abstract
Learning light-weight expressive deep networks in both image synthesis and image recognition remains a challenging problem. Inspired by a more recent observation that it is the data-specificity that makes the multi-head self-attention (MHSA) in the Transformer model so powerful, this paper proposes to extend the widely adopted light-weight Squeeze-Excitatiin adaptive to reinforce its data specificity. as a convolutional alternative of the MHSA, while retaining the efficiency of SE and the inductive basis of convolution. It presents two designs of spatially-adaptiin squeeze-exeitation (SASE) modules for image synthesis and image recognition respectively. For image synthesis tasks, the proposed SASE tested in both low-shot and one-shot learning tasks. It shows better performance than prior arts. For image recognition tasks, the proposed SASE is used as a drop-in replacement for convolution layers in ResNets and achieves much better accuracy than the vanilla ResNets, and slightly better than the MHSA counterparts such as the Wwin-Transformet and Sraamdd-Tranffomee in the ImageNet- 1000 dataset, with significantly smaller models.
2
(SE) module to be spatially-
6
10
12
13
14
15
Introduction
17 Both image synthesis and image recognition remain challenging problems computer vision and machine learning. Despite remarkable progress has been made since the recent resurgence of deep 19 neural networks (DNNs). both synthesizing high-fidelity and high- nigh-resolution images and classifying images accurately scale typically entails computationally expensive training and inference, which have shown to lead to potential environmental issues due to the carbon footprint [ Along with the progress, learning light-weight yet highly-expressive deep models also remains important and 23 interesting research direction, especially with less data. This paper focuses on learning low-shot (e.g.. 100 1000 images training) and one-shot image synthesis models and learning smaller yet expressive models for image recognition at scale.
21
24
25
Consider generative adversarial networks (GANs), state-of-the-art methods such BigGANs 2 and StyleGANs 3 4 utilize ResNets as their backbones. Although powerful, as the resolution of synthesized images goes higher, the width and the depth of generator network goes wider and
27
29 deeper accordingly. leading to much increased memory footprint and longer training time. The more recent Transformer based models often further increase the complexities (5). To address
31 these issues, Liu et (6) present a FastGAN approach which introduces a Skip-Layer channel-wist
32 Excitation (SLE) module to reduce the computation and memory complexities of both generator
33 and discriminator networks (Fig. D. together with exploiting the differenniablee data augmentation
34
methods 77]. FastGANs have shown exciting results which outperform the well-known and powerful
35
StyleGANv2 4 under the low-shot training settings.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38
What make the light-weight SE/SLE an effective drop-in module? One possible explanation lies in its data specificity that
Resicual Building Block (ResBiock)
HasElock Squeeze- Excitation (SE)
37
x(X)
38
39
enables on-the-fly feature modulation be- tween feature responses in both training and inference. More recently, the data specificity of the multi-head self-attention (MHSA) module in the Transformer model has been shown to be the key to its rep- resentational power (rather than its long- range contextual modeling capability) [TO]. However, SE/SLE a channel-wise real- ization of the data specificity, without counting for the spatial feature modula- tion/attention (the spatial dimensions are entirely squeezed). So, a question nat- urally arises: Can we extend SE/SLE be spatially-adapioe
40
F(X)
P(X)
SE
41
Skip-LLyyer Excitation (SLE)
43
x
P1(
AO
SUE
45
40
100
40
47
Channel-wise Attention, Modulation: SE (X = SLE (X+
49
Gr x1x1
Koyw
By
51
Top: Illustration of the ubiquitous residual building block [8], and its reinforced variant with the popular squeeze- excitation (SE) module [ that learns channel-wise wise feature attention. F(X) represents the transformation applied to input feature map X. ((X) represents the skip connec- tion. Middle: Illustration of the Skip-Laye Excitation (SLE) module in FastGANs [6]. With the SLE, the original layer- wise skip-connections are removed to reduce computational and memory complexities (e.g., T1() to Ta() shown in dot- ted blocks). Bottom: Both SE: and SLLE realize channel-wise feature attennion/moouulaton to re-calibrate the feature maps.
Figure 1:
52
58
such that we can build wiight convolutional alterna- tive to the MHSA to retain the efficiency of SE/SLE and the inductive basis of con- volution for both fast and low-shot image synthesis and large scale image recognition applications.
54
55
56
57
58
59
To address the question, this paper pro- poses to learn spatially-adaptive squeeze-excitation (SASE) networks with two realizations for 61
80
62 image synthesis and image recognition respectively (Fig. We give brief overview of the proposed
63 modules below.
for Image Synthesis (ayer)
Acy xllyxxiy x
SASE for Image Recagnition
Solmaa along
soltt (mult-heatt
Ki=tk stly
Ao Krutt Jly,
2
Xo,
Xexisw (mub-head)
Qiyx3aa
Co 5XW
Woy
YoyxByxTH
X <ByxWy
Conv ConyBlock Block Core BatchNorm ReLU
Cx
Ox
Global posling
x 1
Convelock 1x1
x Mx Wx
Global pooking
ComBBook
(equesze)
Cx
ConvBlock Bxs, diactoned
[squ8620)
Conv
11 W
9'
BatdhNorm
ConvBlock 1x1 (squesze)
BatchNorm
Ja-w
Noise Injection:
Hx Wx
ReLu
ReL.U
Sigmold
Canv 3x3
FC (exoiation)
(oschation)
Liseempl (Neaceol)
Qeyaasa
V
Figure 2: Illustration of the proposed SASE module. It resembles the multi-head computation in the Transformer
model [T]. It exploits different strategies computing the attention. 9 represents the number of hcads/groups used to split the input along the channel dimension (c.g., g = 4), and r represents the squeezing ratio (c.g..
r
4). See text for details.
64
SASE for Image Synthesis. The left of Fig. 2 iillstrates the proposed design of SASE to facilitate efficient low-shot and one-shot image synthesis. Unlike the channel-wiss wise 1-D attention weights for target feature map X in both the SE and SLE modules (i.e., Cy 1 x 1, see the bottom of Fig. [T]. the proposed SASE aims to learn full 3D attention weights (i.e., Ac, x wy.) in multi- way. Each 3D attention matrix computed by broadcasting and multiplying the learned Query vector Qcyxixx (accounting for the latent style information for image synthesis by squeezing the spatial dimensions) and the learned Key map Kix My xWy (accounting for the spatial mask by squeezing the channel dimensions). The multi-head 3D attention matrices are summed together and normalized
65
66
67
88
70

---

72 by the sum of the Key maps. The resulting final 3D attention matrix used for modulating the feature map Ycy x Hy xWy integrates both spatial and thaanel-wise attention. This 3D attention matrix enables richer information flow from a source feature map X to a target one Y. which leads to better generation quality for low-shot and one-shot image synthesis. Fig. 3 shows the deployment of the
73
74
75
76
SASE module in FastGANs [6] and SinGANs [2]
77
SASE for Image Recognition. As illustrated in the right of Fig. 2 the learned full 3D attention matrix resembles the role of the self-attention weights in the Transformer model M. In the Transformer model, the attention is explicitly calculated between all pairs of "tokens" (e.g., patches after embedding) after the query and key transformation respectively. The resulting full attention matrix thus quadratic in terms of the number of "tokens". In the proposed SASE, the channel-wise attention squeezes all spatial locations, and the resulting ID Query (style) vector conveys/squeezes information from all locations. The spatial Key map squeezes the channels, and the resulting 21) spatial masks (heatmaps) forms the soft grouping of pixels in each mask. 'he resulting 31D attention matrix thus implicitly measure the attention weights used in the Transformer model a much coarser level, but will be more efficient to compute. Softmax is applied along the channel dimension of the 3D attention matrix. It is then used to re-calibrate the output of the Value (convolutional response)
78
79
80
81
82
83
84
85
86
87
map in element-wise / apatially-adaptive way. As shown in Fig. 4 it is used to replace all the 3x3 convolution layers in a feature backbone (e.g.. ResNet-50). It achieves much better accuracy with significantly smaller model complexity in ImageNet- 1000 [3].
89
Our Contributions. In summary, this paper makes three main contributions as follows: (i) presents a Spatially- of generative models from low-shot one-shot images, and for large-scale discriminative learning like the Transformer model, but in more efficient way, respectively. (ii) It shows significantly better performance for high-resolution image synthesis at the resolution of 1024x 1024 when deployed 96 in the FastGANs (6), while retaining the efficiency. It also shows better performance in image classification and object detection with much smaller models. (iii) It enables a simplified workflow for SinGANs 2 and shows stronger capability of preserving image structures than prior arts
91
Squeeze-Fxcitation (SASE) module with two realizations for better learning
93
94
95
97
2
Approach
99
100
2.1
The SASE for Image Synthesis
101
Uncondiational Image Synthesis. The goal is to learn a generator network which maps a latent code
102
to an image.
(I) where: 2 represents a 1-D latent code (e.g., in FastGANs 6 or a 2-D latent code (e.g.. in Sin- GANs (12)), which is typically drawn from standard Normal distribution (i.e., white noise). Og
I = Oc):
103
104
collects all the parameters of the generator network G. Given a latent code, it is straightforward to
105
106
generate image.
107
FastGANs. As shown in the left of Fig. 3 the generator network used in FastGANs (6) adopts a minim yet elegantly chosen design methodology. Given an input latent code, the initial block applies the transpose convolution to map the latent code to a 4 x 4 feature map. Then, a composite block (UpCompBlock) and a plain block (UpBlock) are interleaved to map the 4 x 4 feature map to the one at a given target resolution (e.g.. 1024x x 1024). Batch normalization [14] and gated linear unit (GLU) [15] are used in the building blocks. In composite upsample block, noise injection is used right after the convolution operation. Please refer the original paper (6] for details.
109
111
112
113
114
SinGANs. The right-top of Fig. 3 shows a stage in the generator of SinGANs 112. A SinGAN
115
is progressively trained from a chosen low resolution to the resolution of the single input image. Following the notation usage in SinGANs [12). the start resolution is indexed by N and the end resolution by 0. At the very beginning, a 2-D latent code, Zn is sampled and the initial generator > 0), the generator G, has been progressively grown from Gv and we have X', = GZ,, (X f;0,) where (X² 1 represents the up- ap-sampeed synthesized image from the previous stage n + 1 with respect to the predefined ratio used in the construction of the image pyramid. More specifically,
117
X (ZN; 0x) is trained under the vanilla GAN settings. Then, at the stage (N
119
120
121
122
X (X++1) 1 + (zn + (X,+1) t: O.).. More details are referred to the original paper [12..
123
With the proposed SASE module, we substantially change the workflow of the generator as shown in the right-bottom of Fig. 3 Each stage of the vanilla SinGAN is to learn the residual image on
3

---

BLE/ BASE PaNIDAN
The Van Sindii Slage
(Ru41)
-W(9.1)
Chontiack: Camntoi BatchNarn Leakytlaayya.
SpCoepihace
Upploce
The Propossd SAGD-SHGAA Stage
Condiall
Can/Toneel
Nale Feceiie
x2
Rasinieee
Conto
CLU
Figure 3: Left: The generator network of FastGANs and drop-in replacement of SLE module by our SASE module. The network specification is reproduced based on the officially released code of FastGANs (link). Right: Illustration of deploying the proposed SASE module in SinGANs [12]. See text for details.
125
top of the output from the previous stage. As we shall elaborate, the proposed SASE module is modulating a target feature map using a source feature map, so we remove the residual learning setting. We keep the discriminator of SinGANs unchanged in our experiments.
126
spatially-adaptie
127
128
The SASE Module. Fig. shows the proposed SASE module. We first compare the formulation
129
between the SE module [9], the SLE module (6) and the proposed SASE module. Focusing on how an input target feature map Y is transformed to the output feature map Y', denote by Y = "Yi""",,C where represents single channel slice of the tensor Y for 1 < Cy, we have,
130
SE: Y' = [a1 Y1," .CC yCv),
(2)
SLE: Y = (31 Y11."...... y1. ... yc,), where the channel importance coefficient a, FSE(() in the SE module, and B. = FF.=FLLdd FSLE(X)) in the SLE module. So, the SE module realizes self-attention between channels (a.k.a. "neurons"), while the SLE module realizes crossaatentionn And, the former is special case of the latter when X = Y. Both Or and Se are scalar and shared by all spatial locations in the same channel slice. For discriminative learning tasks such as image classification, this channel-wis feature attention works very well since spatial locations will be discarded by the classification head sub-network (typically via global average pooling followed by fully-connected layer). For image synthesis tasks whose outputs are becation-sensitvve, it may not be sufficient to deliver the entailed modulation effects.
(3)
133
135
137
138
139
The proposed SASE module aims to facilitate apatially-adaptivy attention by extending the SLE module. It learns a 3D weight matrix WCy x from the source feature map X in modulating the target feature map Ycy xWy (that is to "pay full attention" )), and we have,
141
SASE: Y WoY,
143
where o represent the Hadamard product.
144
Learning the spatially-adaptive attention matrix W from X. We want to distill two types of information: One represents ID latent style codes (as the Query vector) that are informed by the source feature map X, and then are used to induce the modulated target feature map Y' to focus on. The other reflects 2D latent spatial masks (as the Key maps) that are used distribute the latent Query/style codes. Decoupling these two is beneficial to enable them learning faster and more accurate.
145
146
147
148
149
150
Decoupling the Style and Layout. We decouple the channels (" ("ncurons") ") in an input source feature map by splitting them into number of groups (e.g., 4), that is to exploit mixture modeling or clustering of the "neurons" " in a building block, as suggested by the theoretical study of how to construct optimal neural architecture in layer-wise manner with
152
of constraints satisfied [16] and as typically done in the MHSA of the Transformer model. For each group, we apply the decoupled channel-wise and spatial transformation for learning the latent style codes and the latent spatial masks
154
155
156
concurrently.
To up, from the channel-win wise attention branches, compute a group of g latent Query/style vectors, Qcy xcx1'sx's 's. From the spatial attention branches, we compute a group of g latent spatial Key masks. KixHy xwy. Then, the 3-D weight matrix W in Eqn.
158
4 is computed by.
9 Σi_ K
W
(5)
D911 K'
where Q and I are broadcasted from Q and K to match the dimensions respectively.

---

161
2.2
The SASE for Image Recognition
Xeupor
Xounve
Xoepor
162
The right Fig. 2 illlstrates the proposed SASE for
0,1x1
image recognition. Its implementation is straightfor- ward following the discussions above. It is used for substituting the 3x3 Convolutions in network (e.g..
164
J
MISA
NABUL
186
the ResNets [8 ), as shown in the right of Fig. 4
G aaxt
167
More specifically, we can compare the computation workflows between our SASE and the MHSA mod- ule. Let c be the input dimension (i.e., c c= (), g the number of heads, and d = the head dimension. For simplicity, we omit the additive positional encoding used in integrating the MHSA in ResNets. Please refer to [17] for more details. Following the terminol- ogy used in the Transformer model [T. denote by
188
Aoutru
169
170
Figure 4: Comparisons between variants of the vanilla ResNet bottleneck block [8] (left) with the proposed SASE bottleneck: the SE bot- tlneckk [9] is widely adopted design, and the Transformer bottleneck recently proposed
171
173
method [1. r the bottleneck ratio (c.g., r 4).
175
N = HH x W the number of "tokens". In MHSA and
176
SASE, the query, key and value are then defined respectively by:
MHSA:
ZgxNxd Reshape (W exe x XexHxw), Z E (Q,
(6)
ANxNN Softmax Kaxx)/Va),
i= 1=
XxXHXxW Reshape(Concat( (o'onaatA''xx X vi
(8)
SASE:
DxxIXI = -SE(XxxHx)): 1- 9.
(9)
KaxHxx =
(10)
VåxHxw = Conv3x
(11)
AxxHxW Softmax Softmax(QaxxxI KdxHxw),
(12)
X Xx H Concat(A'áx X V VdxH>w)
(13)
177
where the MHSA often suffers from the quadratic complexities of computint time and memory footprint in terms of the input number of "tokens" (N). Our SASE can retain linear complexities.
178
179
In terms of maintaining the on-the-fly data-specificity as pointed out in [00, our SASE offers an alternative and efficient computation workflow. In our SASE, the query attempts to summarize information from all spatial locations, and the key attempts maintain the locality. The resulting attention via broadcasting and multiplying the query and key facilitate integrating the global and local information, which is then used to modulate the value.
180
182
183
Experiments
194
185
In this section, test the proposed SASE module on four tasks: low-shot image synthesis using FastGANs (6, one-shot image synthesis using SinGANs [12,, ImageNet- classification using ResNets (8) and MS-COCO object detection and instance segmentation using Mask R-CNN [18] with ResNets backbone. Our Py Torch source code is provided in the supplementary materials.
186
188
189
3.1
Low-Shot Image Synthesis Results
190
Data and Settings. We adopt the datasets used in the vanilla FastGANs 6 for fair comparisons with the SLE. There are 5 categories tested at the resolution of 256 x 256 each of which uses around 100 training images. There are 7 categories tested at the resolution of 1024 x 1024, four of which use around 1000 training images and the remaining of which use around 100 training images. The categories are listed in Table I We follow settings used in the official code of FastGANs. One thing worth clarifying is the output size of the discriminator. There are two different settings used for different categories in the original experiments by FastGANs [6: 1 x or 5, which show different performance on different categories. For simplicity. we use 5 x 5 consistently throughout the experiments as the output size, so some of the results of the proposed SASE module could be further improved. A single GPU is used in training.
192
193
194
196
198
200
Metries. To evaluate the quality of synthesized images, we adopt the widely used Fréchet Inception Distance (FID) 19 and Kernel Inception Distance (KID) 20 metrics. KID has better sample- efficiency and lower estimation bias than FID, more suitable for low-shot image synthesis. We further use the density and coverage [21 metric for evaluating the reliable fidelity and diversity, where
202
5

---

204
5. To assess the potential memorization in low-shot image synthesis methods, we use the Kolmogorov- -iiinvv (KS) p-value proposed in the latent
use the default k-nearest neighbours with k =.
205
recovery method [22]. We use I and + alongside each of the metric in the tables to indicate whether
206
207
the larger/smaller its value is, the better a model is.
Metric
Method (DillAug)
Ubana
256x226.
<< -100
Coumpy
PeT chllggrry
FUEQ
1024x
1024.
~1100
Fosemon
1024x 1024,
Skalls
~> imsgges
Shalls
SLE
41.05
50.66
35.11
26
10.03
44.3
45.08
31.7
57.1
59.38
130.05
155,47
FID,
SPAP
40.05
51.98
58.45
54.31
36.14
30.15
14.41
10:14
78.37
58.23
61
60.15
114.98
76.76
92.53
125.61
118.09
160.12
156.76
SASE (oues)
36.4
33.55
26.01
9.48
39.59
43.46
51.2
54.22
101.16
140.45
0.112
0.005
0.001
0.012
0.011
0.018
0.054
0.068
KID
CBAM
0.455
0.012
0.026
0.016
0.014
0.007
0.007
0.009
0.004
0.15
0.21
0.54
0.45
0.122
0.058
0.01
0.15
0.13
0.045
0.051
0.071
0.11
SASE (OUPS)
0.005
0.012
0.004
0.004
0
0
0.009
0.006
0.911
0.014
0.030
0.044
SLE
1.3
0.79
0.95
1.25
1.78
1.18
1.38
0.85
1.14
1.17
0.90
0.29
Density
CBAM
SPAP
0.11
1.35
0.53
0.79
0.89
0.91
1.01
1.25
1.32
1.75
0.88
0.81
0.74
0.81
0.73
0.66
0.67
0.54
0.79
0.61
0.92
0.90
0.27
0.26
SASE (Ours)
1.38
1184
1,07
1.38
1,89
1.20
1.41
0.92
1.21
1.21
1,18
0.52
SLE
1.0
0.96
1,0
1.0
0.95
0.95
0.95
0.89
0.85
Coverge-
SPAP
0.56
1.9
0.92
1.0
0.95
1.0
0.91
0.78
0.90
0.92
0.83
SASE (ours)
1.0
0.98
1.0
1.0
0.96
0.96
0.95
0.96
1.0
1.0
0.91
Table 1: Fidelity (FID, KID. Density) and diversity (Coverage) comparisons between our SASE and three baseline modules in low-shot image synthesis, including the vanilla SLE (6). the SPAP modulc (23) and the CBAM module 24 using the FastGAN pipeline R that utilizes the differentiable data augmentation (DiffAug) method 7 in training. Our SASE consistently better than the three bascline modules.
208
Model and Data Augmentation Baselines: To evaluate the effectiveness of the proposed SASE, in addition to the vanilla SLE (6. we also compare with: the CBAM module [24] which leverages spatial and channel attention sequentially for better representation learning, and the SPAP module (3 which leverages multi-scale spatial attention in GANs.
209
210
211
212
For low-shot image synthesis, data augmentation plays important role. The vanilla FastGAN 6 utilizes the differentiable data augmentation (DiffAug) method 77. More recently, the adaptive data augmentation (ADA) method (25 is proposed with even better support for low-shot image synthesis.
213
215
To evaluate whether the proposed SASE retains its effectiveness, we compare the SLE and our SASE
216
in modified FastGAN pipeline with the ADA in training. We follow the original ADA settings to
217
set the target value to 0.6, and set the increasing rate of augmentation probability such that it can
218
increase from 0 within 10k iterations (1/5 of the total training time).
Metric
Method (ADA)
256x226,
.~100 -1000
images per calegorr
1024x
1024.
<-. ,.10000 images
1024x 1024,
~ 100
imsgees Shell
Usama
Dog
Cat
Grampy CR
Panda
FFRQ
Art
Plower
Pokemon
AmimeFace
SKDIES
FID;
SASE (ours)
SLE
34,5
38.9
52.04
49.83
31.2
34.5
26,03
26.83
9.50
9.87
39.12
44.43
43.53
45.1
31.
29.63
55.67
48.56
53,31
59.11
120.62
96.56
153.47
1401.7
KID,
SLE
SASE (ours)
0.004
0.01
0.015
0.012
0.004
0.002
0:007
0,004
0.003
0.002
0.012
0,011
0.009
0,011
0.006
0.006
0.013
0.009
0,013
0.018
0,349
0,025
0.044
0.021
Density;
SLE
1.35
0.78
0.96
1,28
1.82
1.17
1.39
0.85
1.13
1.18
0.91
0.28
SANE (omes)
1.40
1.24
Covengo
SASE (ours)
1.0
1.0
130
1.0
1.0
1.0
0.96
0.97
095
0.96
0ST
0.95
039
1.0
1.0
0.92
Table 2:
Fidelity (FID, KID, Density) and diversity (Coverage) comparisons between our SASE and SLE
using a modified FastGAN pipeline in which the differentiable data augmentation method is replaced by more recent adaptive data augmentation method (ADA) 25 that facilitates low-shot image synthesis. Compared with Table the ADA method shows better overall performance than the differentiable data augmentation method 77. Our SASE remains consistently better than the SLE w.r.t. the new data augmentation method, justifying the architectural contributions by our SASE.
Merric
DataAng
Method
Oluani
256x2266,
Dag
~-100
Cal
intges catogory Grungy Car
Punds
HFRQ
1024x
Ait
1024,
Flawer
~1000
inages Pakeaaa
AlimiiFacc
1024 1024,
Skulls
>-100 images
Slells
SIF
0.57
0.77
0.32
0.19
0.81
0.30
0.025
0.06
0.78
0.75
0.17
0.89
KS p-valun,
DiffAng
CBAM
SPAP
0.86
0:45
0.59
0.65
0.35
0.13
0.11
0.53
0.79
0.55
0.22
0.24
011
0.3
0.03
0.02
0.42
0.37
0.42
0.31
0.58
0:21
0.66
0.49
(ttresboldd d0D1)
SASE (rurs)
0.65
0.45
0.32
0.94
0.76
0.73
0.3
044
081
0.86
0.59
0.96
ADA
SASC (ours)
SLE
0.33
0.71
0.59
0.74
0.35
0.42
0.95
0.19
0.81
0331
0.74
0.27
01227
0.55
0.42
000
0.8
0.76
0.7
033
0.38
0.13
0.30
0.98
Table 3: Memorizattonn/overitin assessment for our SASE and SLE, SPAP adn CBAM in the FastGAN
pipeline with the differentiable data augmentation method. Our SASE shows signs of overfitting across all scenarios, while the SLE shows tendency towards overfitting on some categories such as "Art" and "Flower".
219 Results- Image Synthesis Quality: Table I shows that the proposed SASE is consistently better than all baselines (SLE, SPAP and CBAM) in terms of both traditional metrics, FID and KID and more recently proposed more reliable metrics, Density and Coverage. For high-resolution (1024xA A 1024) image synthesis which uses more SASE components (Fig. 3. the improvements are significantly
221

---

223
better, which shows the effectiveness of our SASE. It is also noted that the SLE is overall better than SPAP and CBAM.
224
225
Table shows that our proposed SASE is still consistently better than the SLE when we replace the
226
DiffAug by the ADA in training.
227
Results- Memorization Assessment: Table 3 shows that our SASE is capable of synthesizing
new images by learning from low-shot images, while other methods have certain tendency towards overfitting on different categories. The results of our SASE are aligned with its diversity evaluation
228
results in Table and Table 2
Madel
Na
Natredf
Natre-tOk
PR
27.54
2015
203
1921
Morkl
Panan
FLOOS
Parmma
Rendheiio
FLOP.
28.53
256-226
9,794GG
29.55
1024 1024
16,1966G
Table 4: FID comparisons (smaller is better) on the categorics with models trained with more data (2k, 5k and 10k) at the resolution of 1024 x 1024.
Table 5: Model complexity comparisons between SASE-FasGGA and ALE-FastGAN.
231
Results- Learning with More Data: To further evaluate the effectiveness of the proposed SASE
when trained with more data, we compare our SASE and the SLE under different settings. With more training data,
233
we use FID in evaluation for simplicity. As shown in
235
Table 4 our SASE retains its stronger effectiveness than the SLE. Both models are trained from scratch with the same data sampled from the original FFHQ and Nature Photograph datasets. Training time are budgeted with around 20 hours for all the experiments.
237
239
240
Qualitative Results and Explanability Visualization.
241
To check what are learned by the spatial attention (the Key maps in Fig. B). Fig. sioows some synthesized face images and the learned latent spatial masks. We can see that the learned masks cover different areas of the face, e.g. starting from left, the fourth column of masks cover the hair area, and the third column covers nose area. Please check the Appendix AA.3 for more qualitative results.
243
245
247
Figure 5: Top: Synthesized face images at the resolution of 1024 x 1024 in the FFHQ dataset B. The model is trained using 2k training FFHQ images for around 15 hours on a single GPU. Bottom: Visualization of the learned Key maps (spatial masks) from the stage 32" to the stage 512?.
248
Model Complexity. Table 5 shows the comparisons of model sizes. Our MASE--aatGAAe have slightly less pa- rameters than the vanilla Fas-FastG,NNs, and has negligi- ble computating cost increase interms of FLOPs. since we split and squeeze the channel dimension (g and r in right of Fig. 2 in learning the channel-wise and spatial attention in our SASE. Although having less number of parameters, the proposed SASE module increases the training time in training models for righuttsolution image synthesis (roughly 1/8 relative increase), which may due the more sophisticated computational graphs to maintain for forward and backward computation after the SASE is used. The training time increase is negligible for training image synthesis models at lower resolutions such 256 x 256.
249
250
251
253
255
257
258
259
3.2
One-Shot Image Synthesis Results
Data. Since our goal is to test if the pro- posed SASE can lead to structure-aware one- shot image synthesis, we select 23 images, among which 14 images are used in the vanilla SinGANs: Brandenberg, bridge, Golden gate, tower, angkorwat, balloons, birds, colusseum, mountains, starry night, tree, cows, volcano; The remaining 9 images are searched from the website. The images cover different structures which often fail the vanilla SinGANs.
251
263
265
267
Figure 6: Left: a real image. Right: from top to bottom, synthesized images by our SASE-SinGAN, the vanilla SinGAN, the ConSinGAN, and the SLE- SinGAN. SASE-SinGAN is better in terms of preserving structure, while producing meaningful semantic varia- tions (e.g., the change of number of Sphinx statues)
269
Settings and Baselines. We follow the settings
271
provided by the vanilla SinGANs [12] Two basclines are used: (i) ConSinGANs (26] for
7

---

273
which we follow the suggestions in the paper to try different combinations between the learning rate and the number of stages jointly trained and select the best results. (ii) SLE-SinGANs in which the
274
275
SLE module is used, instead of SASE, in the right-bottom of Fig. 3
276
Metrics. We evaluate methods with single image FID (SIFID) and Diversity Score proposed in
277
the vanilla SinGANs [[2].
Merrcc SIFDD
SinGAN
01885
ConSinGAN
145
SLE SINGAN
0.8
SASE SinGAN (ours)
0.58I
Model
Panans
Resolution
FLOP=
Panans
Resetution
FLOP%
Diverity
0.345
0.87
0.50
0,25
SiNGIA
102M
102M
330-220
38.20G
SLE-SuGGAN
1633
165=250
19.33G
163M
330-2250
38.21G
Table 6: SIFID and diversity score comparisons using the 23 selected images. Note that SIFID may not reflect the actual quality of synthesized images, as pointed out in ConSinGANs [26].
SASE SRGAA
125M
44.00G
ConSiCIAN
165x250
Table 7: Model complexity comparison between the
proposed SASE-SinGAN and other SinGAN variants.
278
Results. Table 6 the comparison results. In terms of diversity score, Our SASE obtains lower diversity in the trend similar ConSinGAN. The testing images are structure-rich images for which our goal is to study how to preserve the structure. The diversity score should be interpreted jointly with the SIFID. Fig. 6 shows synthesized images for the Egyptian pyramid image. We can see the the
279
280
281
282
proposed SASE is stronger in terms of preserving structures in synthesized images. We observe that ConSinGANs may fail to learn some images c.g., the Golden Gate (Fig. 23 in the appendix), which
283
284
causes the high SIFID. More qualitative results are in the Appendix A.4
Model Complexity. Table 7 shows the complexity comparison between JASE-SinGAN and other SinGAN variants. Both SLE and our SASE increases the FLOPs significantly compare to vanilla Singan, since they are used for connecting
286
resolutinn feature maps to relatively high ones and there are four in total, see Fig. 3 so the overhead is light-weight. For SinGANs, we have SLE and
288
289
SASE between feature maps with the same resolution and have one at every resolution stage. The
spatial attention branch of our SASE increases the FLOPPs even more significantly.
290
291
3.3
Image Classification Results in ImageNet- 1000
292
We test the proposed SASE using ResNet-
293
50 [ (Fig. 4.. First we compare it with SE
Epochs
Method
294
and other variants of attention models used in
RR.NN.G..2.44..5
ResNec50
25.03M
4.27G
77.00
17.14
93.66
295
ResNets, including the Self-
convo-
SC-ReeNert00
296
lutions (SC-ResNets) (27), the Gather-Excite
ResNe::0
X0
24.37M
3,86G
73.48
297
networks (GE-ResNets)
the NetR
AA RecNet50
25.80M
8.30G
73,70
298
local networks meet the SE networks) [29].
SASU Resivec00
Swin Timy
18.6M
25.0084
3,36
4.50G
78.06
81,20
94.14
95.50
299
the Efficient Channel Attention networks (ECA-
Siikhoann (A2)
Siaal
3
24.508
25.60
4.100
3.30G
70:00
79,80
300
ResNets) (30] and the attention augmented net-
NaaE-R (eurs)
15.65M
H1.24
301
works (At-ResNets) 31 For fair comparisons,
BoT
17
20.80M
79.10
94,40
302
we use the most vanilla training settings to verify the effectiveness of the architectural design of
Table 8: Comparisons of ImageNet- 1000
1000 classification results. All models are trained and tested using the resolution of 224 x 224. Top-1 and Top-S accuracy (%) are used. Results are from the MMClassification model zoo
303
304
SASE itself: 100 epochs and the basic data aug- mentation scheme (random crop and horizontal flip). Table 8 (top) shows the results. Compared with the SE, our SASE obtains 0.32% top-1 accuracy increase, while significantly reducing the model parameters (by 9M) and FLOPs. Compared with CesSNeXt32244.
305
307
our SASE obtains 0.16% top-1 accuracy increase with much less parameters too, and our SASE also outperforms other attention variants. These results clearly show the effectiveness of the proposed SASE.
309
311
Further, to compare the recent state of the art image classification models, we follow the improved training procedure, the A2 recipe, proposed in [34] that enables ResNets to strike back in performance compared with variants of Vision Transformer, the results are shown in Table g (middle). The proposed SASE shows very promising performance, bridging the performance gap between the ResNets and the state-of-the-art Swin-Transformer [32] and Pyramid Vision Transformer (PVT) 33
312
313
315
which supports our design hypothesis stated in Section 2
317
3.4
Object Detection and Instance Segmentation in MS-COCO
To check how well the ImageNet- 100 pretrained SASE-ResNetSh will transfer to downstream 319
tasks, we test it the MS-COCO 2017 object detection and instance segmentation dataset [35]
using the Mask R- R-CNN framework [18). Table 9 shows the comparisons. On the one hand, our
8

---

321
SASE significantly outperforms the vanilla ResNet, and is slightly better than the Bottleneck Trans- former [17 with a smaller model complexity, which shows the effectiveness of our SASE. On the other hand, our SASE obtains comparable
322
323
324
performance to the PVT-Small [33. but is sig-
Bockbone
MAMI
AP"
Mosk RANN
AFg
AP
1x
LT opeeins
AP
AP
AP
325
nificantly worse than the Swin-T (32).
PVT-Small 13
44.1
14.2
43.3
41.3
61.7
65.3
445
46.9
37.1
39.9
58.4
62.5
401
42.8
BoF50
43.5
65.3
47.6
36.9
62.5
41.3
326
4 Related Work
Swin-T
SASE ResNec50
13
37.5
48.0
43.7
46.0
68.1
65.2
50.3
47.7
39.5
41.6
65.1
62.0
44.9
423
Table 9: Performance comparisons in MS-COCO.
327
I. Light-weight GANs with Low-Shot Learn-
328
ing. Compared to the extensive research on
light- neural architectures in discrimina-
330
tive learning for mobile platforms, much less work has been done in generative learning [6).. The residual network [8] is the most popular choice, on top of which powerful generative models such as BigGANs 22 and StyleGANs [37 4 have been built with remarkable progress achieved. For high- resolution image synthesis, these models will be very computational expensive in training and in- ference. In the meanwhile, training these models typically require a big dataset, which further increase the training time. low-shot learning is appealing, but very challenging in training GANs, since data augmentation methods that are developed for discriminative learning tasks are not directly applicable.
332
334
336
337
To address this challenge, differentiable data augmentation methods and variants 7 25 [3 39 have been proposed in training GANs with very exciting results obtained. Very recently, a FastGAN
338
339
approach [6] is proposed to realize light-weight yet sufficiently powerful GANs with several novel
340
designs including the SLE module. The proposed SASE is built on the SLE in FastGANs to reinforce
341
its data-specificity.
342
Learning Unconditional GANs from a Single Image. There are several work on learning GANs from single texture image [40] 41 [2] Recently, a SinGAN approach [12] has shown surprsingly good results on learning unconditional GANs from a single non-texture image. It is further improved in ConSinGANs (26) which jointly train several stages in progressively growing the generator network. However, it remains a challenging problem of preserving image structure in synthesis. The proposed SASE is applied to the vanilla SinGANs 122, leading to simplified workflow that can be trained in a stage-wise manner and thus more efficient than ConSinGANs, and facilitating a stronger capability of preserving image structures.
344
345
346
348
349
350
Attention Mechanism in Deep Networks. Attention reallocates the available computational re-
sources to the most relevant components of a signal to the task 443 44
45
46
47
11]. Attention
352
mechanisms have been widely used in computer vision tasks [48]
49
50
51]
The SE module [9]
applies lightweight self gating module to facilitate channel-wise feature attention. Our proposed
354
SASE module incorporates spatially-adaptive feature modulation, while maintaining the light weight design, improving the representation power for efficient discriminative learning.
355
Limitations and Potential Negative Impacts of the Proposed Work
356
The proposed SASE shows worse performance in object detection and instance segmentation than state-of-the-art Transformer based models. One direction to address this is to run more comprehensive experiments the design choices (Eqn. to Eqn. 12). The proposed SASE module does not show
358
359
360
any potential negative impacts with its current form.
361
6
Conclusion
382
This paper proposes to learn ppatially-adaptivn squeeze-cxcitation (SASE) networks for better data- specificity by jointly learning both channel-wist attention as latent style representation and spatial attention as latent layout representation. The resulting SASE module computes a 3D attention matrix for modulating input feature map. In experiments, the proposed SASE module is tested in low- shot image synthesis using FastGANS, one-shot image synthesis using SinGANs, ImageNet- 1000 1000 classification using ResNets, MS-COCO object detection using Mask R-CNN. The AaSE-FastGANs
364
366
368
are consistently better than three strong baselines, and obtain significantly better performance at high-resolution image synthesis. The SASE-SinGANs show stronger capabilities in preserving image
389
370
structures than prior arts. The SASE-ResNets show better performance than the SE variant and
other variants with significantly smaller models, and competitive performance state-of ofte-oofthheart Transformer based models.
372

---

References
373
374
[1]
Emma Strubell, Ananya Ganesh, and Andrew McCallum, Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:/906.000 2019.
375
376
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
378
[3] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
379
380
381
[4] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1110-111, 2020.
382
383
384
[5] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
385
386
387
[6] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. arXiv e-prints, pages arXiv-2101, 2021.
388
389
[7] Shengyu Zhao, Zhijian Liu, Ji Lin, Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. arXiv preprint wrXiv:2006.107366 2020.
391
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
382
383
394
[9] Jie Hu, Li Shen, and Gang Sun. Squeeze-ands -nde--acttttieioon networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
395
396
[10]
Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations, 2021.
387
396
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008 2017.
400
[12] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4570-4580, 2019.
402
404
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
405
406
407
[14] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, pages 448-456, 2015.
408
409
[15] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933-941. PMLR, 2017.
411
412
413
[16] Sanjeev Arora, Aditya Bhaskara, Rong Ge, and Tengyu Ma. Provable bounds for learning some deep representations. In International conference machine learning, pages 584-592. PMLR, 2014.
415
[17] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16519-16529, 2021.
417
10

---

419
[18]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
420
421
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
422
423
[20] Mikolaj Binkkwski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans, arXiv preprint arXiv:1801.01440 2018.
425
426
[21] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models, In International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.
427
428
[22]
Ryan Webster, Julien Rabin, Loie Simon, and Frédéric Jurie. Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11272- 2019.
430
432
[23] Wei Sun and Tianfu Wu. Learning spatial pyramid attentive pooling in image synthesis and image-to-image translation. arXiv preprint 1rXiv:1901.000
2019.
434
[24] Sanghyun Woo, Jongchan Park, Joon- Yoon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
435
438
437
[25] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.0666 066666 2020.
439
440
[26] Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. Improved techniques for training single-image gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages I 1300-1309, 2021.
441
442
[27] liang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Changhu Wang, and Jiashi Feng. Improving convolutional networks with self-calibrated convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10096-10105, 2020.
444
445
446
[28] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. Advances in neural information processing systems, 31, 2018
448
449
[29] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet qqueeze-excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
450
451
[30] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu, Eea- net: Efficient channel attention for deep convolutional neural networks. In CVPR, pages 11531-11539, 2020.
453
454
455
[31] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3286-3295, 2019.
457
458
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu. Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
459
480
[33] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEFE/CVF International Conference on Computer
462
Vision, pages 568-576 578, 2021.
464
[34] Ross Wightman, Hugo Touvron, and Hervé Jégou, Resnet strikes back; An improved training procedure in timm. arXiv preprint 1rX0v:2110.00476, 2021.
11

---

486
[35] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
467
469
[36] Karol Kurach, Mario Lucic, and Xiaohua Zhai Marcin Michalski Sylvain Gelly. The gan landscape: Losses, architectures, regularization, and normalization. arXiv preprini arXiv:1807.047200 2018.
470
472
[37] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4401-4410, 2019
474
475
[38] Ngoc-Trume Trung Tran. Viet-Hung Tran, Ngoe-Bao -roe-Bao Nguyen, Trung- Kien Nguyen, and Ngai-Man Cheung. Towards good practices for data augmentation in gan training. arXiv preprint arXiv:2006.05338, 2020.
478
477
[39] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. arXiv preprini arXiv:2006.02595. 2020.
479
[40] Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic spatial gan. arXiv preprint arXiv: 2017.
481
482
[41] Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. Texture synthesis with spatial generative adversarial networks. arXiv preprint arXiv:1611, 188207, 2016.
483
484
[42]
Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian genera- tive adversarial networks. In European conference on computer vision, pages 702-716. Springer, 2016.
486
487
[43] Bruno A Olshausen, Charles H Anderson, and David C. Van Essen. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. Journal of Neuroscience, 13(1 1):4700-447 4719, 1993.
488
489
[44] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-base -ased visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence, 20(1 10: 1):11)1-212259 1998.
481
482
493
[45] Laurent Itti and Christof Koch. Computational modelling of visual attention. Nature reviews neuroscience, 2(3):194-203 2001.
495
[46] Hugo Larochelle and Geoffrey E Hinton. Learning combine foveal glimpses with third-order boltzmann machine. Advances in neural information processing systems, 23: 12:1 2010.
496
[47] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pages 2204-2212, 2014.
498
[48] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28:2017-2025, 2015.
500
501
[49] Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang. Yongzhen Huang, Liang Wang. Chang Huang, Wei Xu, al. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In Proceedingy of the IEEE international conference on computer vision, pages 2956-2964, 2015.
502
503
504
505
[50]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
507
508
509
[5]] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 5659-5661 2017.
511
12

---

518 [52] Taesung Park, Ming- Yu Liu, Ting- Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with ppatiall--duaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2337-2346, 2019.
Ting-Chun
514
[53] Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In Proceedings of the IEEE International Conference on Computer Vision, pages 10531-10540, 2019.
517
13

---

Checklist
518
519
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or
520
521
[N/A]- - You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
522
523
Did you include the license to the code and datasets? [Yes] See the license file in the code folder in the supplementary material.
625
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
527
528
1. For all authors...
529
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
530
(b) Did you describe the limitations of your work? [Yes] See Section
S
532
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 5
538
534
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
535
538
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
538
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
540
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See the code in the supplementary material.
541
542
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
544
545
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
546
547
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See settings in experiments.
549
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
550
(a) If your work uses existing assets, did you cite the creators? [Yes]
551
(b) Did you mention the license of the assets? [Yes]
552
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code is provided.
553
554
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
555
556
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
557
558
5. If you used crowdsourcing or conducted research with human subjects...
$59
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
562
563
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
584
14

---

585
A
Appendix
566
A.1
Comparing the SASE with Alternative Designs in Image Synthesis
367
The weight modulation method in StyleGANv2 is an elegantly designed operation to achieve detailed style tuning effects. The style code is used to directly modulate the filter kemnels (as model parameters) in an instance specific, and then modulated filter kernels are used in computing the convolution. Although being highly expressive, this weight modulate not ppatially-adaptivn And, it increases the computational burden and the memory footprint in execution. The proposed SASE directly modulates the feature map in lightt manner.
Comparing with the weight modulation in StyleGANy2
568
569
570
571
572
573
574
Comparing with the SPADE in GauGANs (52) and the ISLA-Norm in LosIGANs 153y. Both the SPADE and the ISLA-Norm exploit aadipily-adaptive modulation, but apply it inside the BatchNorm. They replace the vanilla channel-wise affine transformation in the BatchNorm with spatiall--adaptic affine transformation. The spatially-adaplee
575
578
577
affine transformation coefficients are learned either from the input semantic masks in GauGANs or the generated latent masks from the input layouts in
578
579
LostGANs. The proposed SASE is similar in spirit to the ISLA-Norm, but is formulated under the Inception architecture together with the skip-layer idea proposed in FastGANs [6.
580
581
A.2
Training details of EASE-FastGANs
We use the Adam optimizer for training, with 31=0.5, $2=0.999.. We use learning rate of 0.0002 for all datasets except for FFHQ and the panda datasets, in which we use 0.0001. For the architecture of Discriminator, we adopts the output size of 5 x 5; and for SASE-FastGAN model on the 1024 x 1024 datasets, we apply Gaussian noise injection to the spatial masks of SASE (the right-bottom of Fig. ??), with zero mean and unit variance; For the convolution of spatial branch of SASE, we set the dilation rates as 2, 2, 4 at stage 8 x 8, 16 x 16, 32 x 32, respectively.
583
585
587
A.3
More results of SASE-FastGARe
589
A.3.1
Clarification on results on FFHQ-1k
590
We notice there is gap of the performance on FFHQ-IK between our retrained version based on the official FastGAN code and the reported one in the paper. Thanks to the author's feedback via emails, the best configuration for reproducing the FFHQ-1k 1k result of FID=24.45 will NOT be released since is deployed on a commercial platform. So the FID of the SLE- Fas-GaN we trained based on the Aa code is worse than the one reported in the paper (Table 10
591
582
593
FID
SLE-FasIGAN [d] (reported in the paper)
24.45
Retrained from the official FasIGAN code
44.31
Retrained from the official FastGAN code (datiset-ppccficc version)
42.82
Our SASE-FaRGAN buili on the official FastCANN code
39.59
Table 10:
FFHQ-IK Ik performance comparison between the reported result in FasIGAN paper, our retrained version based on their code, and the proposed SASE-FasIGAN
585
A.3.2
Synthesis results of SASE-FastGAN on 1024 x 1024 datasets
596
Fig. 7
shows the example synthesized 1024 x 1024 images of our proposed SASE-FasIGAN.
A.3.3
Backtracking results
596
Settings: (Thanks the clarification by the authors of FastGANs via emails) 1) We first split the dataset into train/test ratio of 9:1. 2) Train the model on the splitted training set. 3) Pick the trained generator checkpoint at iteration (20k, 40k, 80k) respectively, and do latent backtracking for Ik
599
600
iterations on test set. 4) Compute the mean LPIPS between the test images and the reconstructed images from backtracking of the corresponding checkpoints. Where LPIPS is the average perceptual distance between two set of images; in this test, lower LPIPS value indicates less overfitting, since
602
15

---

Shell
Art Painting
Flower
Pokemon
Anime Face
Skull
Real Data
StyleGANv2
Vanilla FastGAN
S (ours)
Figure Examples of synthesized images at the resolution of 1024 1024. Best viewed in magnification.
16

---

Obama 256
Cal 2562
Dog 256
Panda 2562
FFHQ 1024
Art Painting 10242
Real Image
Interpolation between backtracked images
Real Image
Figure 8: Examples of backtracking results,
17

---

604 it means that the model trained on the training set can backtrack images on an unseen testset with
605
small reconstruction error.
606
Results: Fig- g shows the example backtracking results on several of the low-shot datasets. The
smooth transition of the interpolated images between the backtracked test images show that model hasn't 't overfit to the training set.
608
A.3.4
Style mixing results
610
Settings: To demonstrate that the proposed SASE able disentangle the high level semantic attributes of featres at different scales, we conduct the style mixing experiment as done in the FastGAN paper [6], in which for pair of style and content images, we extract channel weights from style images, and use them to modulate the features of content images, while retaining the spatial masks of the content images. The resulting effects as shown in Fig. Iis that the appearance and color scheme of the style image is propagated to the content image, and the spatial structure of the content
612
613
614
615
616
image is unchanged.
A.4
More results of SASE-SinGANS
618
In order to demonstrate the strength of the proposed SASE module in one-shot generative learning. We present qualitative comparison of synthesis results of SASE-SinGAN with other related methods on 9 images, which are buildings that have different global structures. We also show the results of image harmonization and editing under one-shot setting.
620
621
622
A.4.1
Synthesis with example images
623
Fig. 10 to Fig. 23 are the example synthesis results. We can see that compare to ConSinGAN, SinGAN and SLE-SinGAN, SASE-SinGAN captures the global layout of the image better, while
624
625
producing meaningful local semantic variations, also notice that ConSinGAN fails to learn some of
626
the image, as shown Fig. 23
627
A.4.2
Harmonization
628
Fig. 24
shows the comparison one-shot image harmonization task done in 12 It shows that our proposed SASE-SinGAN can realistically blend an object into the background image.
629
630
A.4.3
Editing
Fig. 25 shows the comparison on one-shot image editing task as done in 12 It shows that our
632
proposed MASE-SinGAN/ able to produce seamless composite which image regions have been
copied and pasted in other locations. Note that SASE-SinGAY shows more realistic composite within the edited regions.
634
18

---

Style
Style
Content
Content
Art Painting
Shells
Style
Style
Content
Content
Pokemon
AnimalFace-Cat
Style
Style
Content
Content
AnimalFace-Dog
Obama
Figure 9: Examples of style mixing results, Best viewed in magnification.
19

---

SASE-SHIGA
ContinDAN
Real
SLE-dIGAN
Figure 10: One-Shot synthesis comparison on the bridge image. Note how the synthesized images of SASE-SinGAN capture the global layout of the real image, and at the same time produces semantically meaningful variations (size, number of towers at top).
BASE-SHON SIEES-GGAN
Condroow
Real
SLE-SHGA
Figure 11: One-Shot synthesis comparison on the Great Wall image.
20

---

SABE SEEESMMGALL
SAGAA
ConSiNGAM
Real
SLE SIE-A
Figure 12: One-Shot synthesis comparison on the capitol hill image.
SA-E SHYGAN
SKnCAN
CordSrrCA
Real
Figure 13: One-Shot synthesis comparison the ancient Chinese tower image.
21

---

BABE: SINGAA
SHGANN
ConSeeSAN
St 91E-0004
Figure 14: One-Shot synthesis comparison the Lincoln memorial image.
BABE BIE--BAAANNN
SInGON
ConS
Figure 15: One-Shot synthesis comparison the Temple of Heaven image.
22

---

B SiNDAN
S
Can Bondi-cA CAN
Real
ELEE S-N----ANN
Figure 16: One-Shot synthesis comparison the Temple of ancient Chinese tower image.
SASE SHIGAN
EWWDAN
COMSHOAN
SLE-SI3ION
Figure 17: One-Shot synthesis comparison on the Temple of Heaven image.
23

---

BASE-SinGAN
SinGAN
ConSmGAN
SLE-SiNGAN
Figure 18: One-Shot synthesis comparison on the Brandenberg image.
24

---

SASE-SinGAN
SinGAN
ConSinGAN
Real
SLE-SinGAN
Figure 19: One-Shot synthesis comparison the Biffel tower image.
SASE-SINGAN
SANCAN
DonSaGAN
Real
SIF-SHOAM
Figure 20: One-Shot synthesis comparison the bridge image.
25

---

SASE-SIOGAL
SinDAN
Real
SIF-SINAAN
Figure 21: One-Shot synthesis comparison on angkorwat image.
SASE-SINGAN
SMnGAN
ConSinGANN
Real
SLE-SMGAN
Figure 22: One-Shot synthesis comparison on the zebra image.
26

---

SASE SABE-BINON BIEE-IIIAAN
SINDAN
CoMSirCAN
Real
BLF-SHNOAN
Figure 23: One-Shot synthesis comparison on the Golden Gate image. Notice that training of the ConSinGAN has collapsed.
27

---

Original Image
Naively pasted
SinGAN
ConSInGAN
SLE SSGGS
SASE-SInGAN
Figure 24: One-Shot harmonization comparison example images.
28

---

Original Image
Edit
SinGAN
ConSiNGAN
SLE-SnGAN
SASE-SinGAN
Figure 25: One-Shot Editing comparison example images.
29


Reviews:
[{"review_id": "y8cRCEuDKvE", "reviewer": "Reviewer_bFtN", "summary": "This paper proposes a Spatially-Adaptive Squeeze-Excitation (SASE) module which aims to learn a full attention mechanism as a 3D attention matrix. Two variants are developed to address the image classification and image synthesis tasks. Experimental results on public datasets show its performance.", "questions": "There are some issues which are listed as follows.\n\n1, The idea of SASE is modified based on SE to include the spatial attention mechanism, thus it can learn the so-called full attention as the form of 3D matrix. While this idea is similar to the dual attention mechanism [a], which learns attention matrixes to consider spatial attention and channel attention at the same time. \n\n[a] Dual Attention Network for Scene Segmentation.\n\n2. The experimental results are not sufficiently strong. For example, in table 8, the improvement over the existing methods is only tiny. At the same time, in table 6, the proposed method achieves the worst diversity performance, is there any reason for that?", "limitations": "Please see the above sections.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is easy to follow. Its representation is clear.\n+ The idea of learning a full attention matrix is interesting.\n+ Extensive experimental studies are conducted.\n\n- The experimental results are not strong.\n- The idea is essentially similar to the dual attention mechanism.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bafJNTmgWd9", "reviewer": "Reviewer_NUVi", "summary": "This work presents to extend light-weight SE module to be spatially-adaptive to improve model's efficiency and effectiveness. The proposed module is applied to both image synthesis tasks and image recognition tasks with a little different design. Its effectiveness is validated by  experiments on low-shot image synthesis and image classification to some extent.", "questions": "Please check that in weaknesses.", "limitations": "Authors mention the limitations and potential engative societal impact, however, they do not give detailed anaylsis and constructive suggestions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \nA lightweight SE module is applied to image synthesis tasks and image recognition tasks, whose effectiveness is validated by the corresponding experiments to some extent.\n\nWeaknesses:  \n1. Novelty is quite limited, modules similar to the proposed spatially adaptive SE has been proposed before, such the ones in CBAM, RCAN and SFTGAN, SPADE. Please explain in detail what the main difference from these modules is.\n2. The organization of this work look more like a technical report rather than an academic paper.\n\n==========================post rebuttal==========================  \nAlthough authors addressed my comments to some extent, I still have a little concern about the novelty of this work. Therefore, I only raise the score from reject to borderline accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ViNksnnBOYy", "reviewer": "Reviewer_EeP1", "summary": "This paper propose a self-attention module(SASE) for image synthesis and image recognition.\nThe self-attention is based on the **S**queeze-**E**xcitation(SE) mechanism, which is also \nfree from matrix multiplication. \nBy inplacing the matrix multiplication by Hadamard product and using the SE mechanism, \nthe computational efficiency improves while preserving decent performance.\nSufficient comparisons to traditional attention (not self-attention) mechanism demonstrate \nthe effectiveness of SASE in image synthesis. \nWhile in image recognition, the SASE has slightly disadvantage compared with \nSwin-tiny([32] in the main paper), but with only 67% parameters and 75% FLOPs.", "questions": "1. Why your module is effective in the few-shot and one-shot image systhesis? Is there any special design for this task?\n   If so, what is it? (Q for weakness 1)\n2. Why the SE mechanism is adopted in the $Q, K, V$ calculating procedure? Please explain your intention. (Q for weakness 3)\n3. Why the authors choose $K$ (a.k.a spatial attention) for normalizing $A$ in figure 2, \n   instead of choosing $Q$ (a.k.a channel attention) for normalizing $A$? (Q for weakness 3)\n4. The traditional cross attention in transformer requires that the $K, V$ comes form the same feature[2][3] for better refining.\n   However, the authors chooses $Q, K$ from the same feature. Please explain your intention. (Q for weakness 3)\n\n[2] Chen, Chun-Fu Richard, Quanfu Fan, and Rameswar Panda. \"Crossvit: Cross-attention multi-scale vision transformer for image classification.\" ICCV 2021.\n\n[3] Vaswani, Ashish, et al. \"Attention is all you need.\" NeruIPS 2017.", "limitations": "Yes, the authors adequately the limitations of their work.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The proposed SASE module is parameter-less and computationally inexpensive, \n   while with respectable performance \n2. The proposed SASE module is robust and easy to use. With simply replacing modules \n   in existing network structures, performance can be improved.\n3. The the operation of weighting $Q$ with $K$ in the SASE for image systhesis is novel, \n   which is an effective way for aggregating the spatial attention and channel attention.\n\n**Weakness**\n1. This authors need to explain that why their module is effective in the few-shot and one-shot image systhesis.\n   For example, in FastGAN([6] in the main paper), they proposed a self-supervised Discriminator in Auto-Encoder manner\n   to fully utilize the information for the encoder in the discriminator to extract a more comprehensive representation.\n   That's an effective strategy they designed individually for few-shot image systhesis.\n2. There is no ablation studies in this paper, so that the design of the module is not guaranteed to be optimal.\n   Such as where to adopt the SE mechanism is effective? In the $Q,K,V$ procudure or after the self-attention?\n3. This paper, which does not explain some of the original intentions of the module design, can not convince me.\n   Such as why the authors choose $X$ to compute the $Q, K$ and $Y$ for the $V$ instead of $X$ for the $Q$ and $Y$ for $K, V$?\n   (in the SLE SASE module)\n4. Using the self-attention mechanism, the authors need to compare their modules with currently *state-of-the-arts* \n   self-attention modules in Table 3. Such as Swin-Transformer block([32] in the main paper) and Focal transformer Block[1] (at least one of them).\n\n[1] Yang, Jianwei, et al. \"Focal attention for long-range interactions in vision transformers.\" NeruIPS 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MvbZFwHaCDH", "reviewer": "Reviewer_f98y", "summary": "This work proposes to extend the widely adopted light-weight Squeeze-Excitation (SE) module to be spatially-adaptive to reinforce its data specificity, while retaining the efficiency of SE and the inductive basis of convolution. It presents two designs of spatially-adaptive squeeze-excitation (SASE) modules for image synthesis and image recognition respectively. For image synthesis tasks, the proposed SASE is tested in both low-shot and one-shot learning tasks and shows better performance than prior arts. For image recognition tasks, the proposed SASE is used as a drop-in replacement for convolution layers in ResNets and achieves better accuracy than the vanilla ResNets.", "questions": "See above.", "limitations": "Limitations and Potential Negative Impacts of the Proposed Work have been discussed in the paper.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nIt presents a Spatially-Adaptive Squeeze-Excitation module with two realizations for better learning of generative models from low-shot / one-shot images.\nIt shows better performance for high-resolution image synthesis at the resolution of 1024×1024 when deployed in the FastGANs and better performance in image classification and object detection with smaller models. \nIt enables a simplified workflow for SinGANs, and shows a stronger capability of preserving image structures than prior arts.\nThe experiments are comprehensive.\n\n\n\nWeaknesses: \n1. The formulation introduction part is problematic. The current introduction includes too much detailed designs network structures. The introduction should more focus on the motivation and deliver the contribution in a high-level, instead of giving the detailed network design which should be appear in the method section.\n2. Why put the related work section after the method section?\n3. line 251, interms --> in terms", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Y126jSgcLu", "reviewer": "Reviewer_Xsyp", "summary": "Authors propose a Spatially-Adaptive Squeeze-Excitation drop-in module for GAN-base image synthesis and recognition, with improved accuracy and reduced computational cost.", "questions": "Generalization ability is unconvincing.\n\nVisual effect of the results should be compared too.", "limitations": "Not really. The module could potentially help synthesize or fake very realistic facial images and videos, hence may have negative social impact.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the proposed Spatially-Adaptive Squeeze-Excitation seems rational and meaningful\n\nWeaknesses: limited novelty, weak presentation (typos, verbose statements, too small diagrams, ...)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation", "Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Reject