GUIDEGAN:  ATTENTION  BASED  SPATIAL  GUIDANCE FOR  IMAGE-TO-IMAGE TRANSLATION
Authors: Yu Lin
Yigong Wang
Yifan Li
Zhuoyi Wang
Yang Gao
Latifur Khan
Venue:  (2020)
Keywords: Image-to-Image translation
Attention Learning
GAN

Abstract: Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner. However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice. Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients. To tackle this problem, we propose a GuideGAN based on attention mechanism. More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction. This attention map then assists the generator to produce more plausible and realistic images. We extensively evaluate the proposed GuideGAN framework on a  number of image transfer tasks. Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.

Under review as a conference paper at ICLR 2020
GUIDEGAN: ATTENTION BASED SPATIAL GUIDANCE FOR
TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner. However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator practice. Namely, the relative model capaci- ties of the generator and discriminator do not match, leading mode collapse and/or diminished gradients. To tackle this problem, we propose a GuideGAN based attention mechanism. More specifically, an attention mechanism so not only it estimates the probability that its input is real, but also does it create attention map that highlights the critical features for such prediction. This attention map then assists the generator to produce more plausible and realistic images. We extensively evaluate the proposed GuideGAN framework on a number of image transfer tasks. Both qualitative results and quan- titative comparison demonstrate the superiority of our proposed approach.
arm the discriminator with
INTRODUCTION
Generative Adversarial Networks (GANs) have drawn much attention during the past few years, due to their proven ability to generate realistic and sharp looking images. Various computer vision problems are solved using this framework, such as super-resolution (Ledig et al. 2017 colorization (Cao et al. 2017), denoising (Yang et al. 2018) and style transfer (Zhang et al. (017). All these problems can be considered as an smage-to-imen to-immge translation problem. mapping an image from source domain to target domain, for instance, the super-resolution problem of trying to transfer a low-resolution image (source domain) to a corresponding high-resolution image (target domain). Existing literatures have shows that variants of GAN achieved impressive results in both supervised
and unsupervised setting. (Zhu et al., 2017 Liu et al. 2017 Wang et al. 2018 Isola et al. 2017
Choi et al. 2018 Huang al. 2018)
Even with such great success, most GAN-based approaches are suffering from the imbalance be- tween the generator and discriminator (Arjovsky Bottou| (017). In practice, the discriminator is usually too powerful for its task. Thus, the generator obtains very small gradients from discrimi- nator and is hard converge. Most state-of-the-art solutions are trying to find a new objective or add some new regularization terms to the cost function, which mainly affect the generator (Arjovsky al.) 2017| Arjovsky & Bottou| 2017 Mao al.) 2017 Nowozin al., 2016 Zhang et al., 2018 Hu et al. (0118). To address this problem from different direction, we want to borrow some power from the discriminator by incorporating the attention mechanism to help the generator. this pa- per, we propose that the critical locating areas more significant in the translation. The generator should pay more attention to some particular areas rather than the whole image.
Imagine student is learning how to draw a horse. The standard discriminator, painting master, merely grades the student's s painting and hopes that can help the student improve his work. On the other hand, another master will provide additional information. For instance. each incorrect region. That is exactly our idea: we suggest that the student (generator) gains benefit from the second master (attention embedded discriminator). Our main contribution is threefold:
canvas circling

---

Under review as a conference paper at ICLR 2020
A flexible attention-augmenter discriminator: such discriminator provides not only the proba- bility of realness, but an attention map from its perspective. Both trainable attention module and post hoc attention are implemented.
A unified GAN framework using attention information: to improve the translation of the gen- erator, we combine the attention map with raw input via concatenate methods: 1) convert the input to RGBA image by adding an alpha channel; 2) compute the residual Hadamard production of the attention map and corresponding original input, based on RAM; (Wang et al. 2017)
Extensive experiment validation on different benchmarks: we provide extensive experimental validation of GuideGAN on different benchmarks; both the qualitative results and quantitative comparisons against wate-offtheear methods demonstrate the effectiveness of our approach.
To the best of our knowledge, we are the first to report image-to-image translation results using the attention information from discriminator. Different with previous approaches, our framework strengthens the communication and guidance between the generator and discriminator. level, the significance of our work is also on discovering that the attention information from auxiliary network affects the result of image-to-image translation, which we think would be influential to other related research in the future.
a high
2 RELATIVE WORKS
Generative Adversarial Network GANs have achieved impressive results in image translation
tasks (Denton et al. 2013 Radford et al. 2015 Isola et al. 2017 Kim et al. 2017 Ledig et al.) 2017). Typically, GAN consists of two components: a generator and a discriminator. The generator trained to fool the discriminator which in turn tries to distinguish between real and synthesised samples. Various improvements to GANs have been proposed, like improved objective function (Mao al., 2017 Arjovsky al., (017) and advanced training strategies (Gulrajani et al. 2017 Nowozin et al. 2016). A recent framework, FAL (Huh et al. 2019), iteratively improves the synthe- sized image using a spatial discriminator. However, they either didn't collect enough information from the discriminator or need more forward pass to stabilize the result.
Image Translation Image to image translation can be considered as a generative process condi- tioned on an input image. pix2pix (Isola et al. (017)) was the first unified framework for supervised image-to-ing to-t- translation based on conditional GAN (cGAN) (Mirza Osindero) 2014). Tex- lureGAN (Xian et al., 2018) solves the sketch-to-image problem using user defined texture patch. Gonzalez-Garch et al. (20T8) adopted disentanglement representation to improve the rendering pro-
cess and Tang et al. (2019) utilized the extra semantic information guide the generation.
Despite the promising results they achieved, the above methods are generally not applicable in prac- tice due to the lack of paired data. Several interesting frameworks have be proposed to solve this unsupervised image-to image-to-image translation problem. Cycle consistency loss was first proposed in Cy- cleGAN (Zhu et al.] al.]/20) (2017) and is widely used by other unsupervised image translation frameworks. UNIT (Liu et al.) (217)) improves the translation with shared latent space assumption, which is the fundamental of MUNIT (Huang et al. (018) that handles multi-modal translation. In contrast, our flexible framework can be applied on both supervised and unsupervised settings.
Attention Learning Generally, attention can be viewed as guidance to bias the allocation of avail- able processing resources towards the most informative components of an input. Contemporary ap- proaches are divided into two categories: post hoc network analysis and trainable attention module. The former scheme has been predominantly employed to access network reasoning for the visual object recognition task (Simonyan et al.) 2013 Zhou et al. 2016 Selvaraju et al., 2017: Chattopad- hay et al. 2018). Trainable attention models fall into two main sub-categories, hard (stochastic) that requires reinforcement training and soft (deterministic) that can be trained end-to-end (Wang et al.,
2017 Hu al. 2018 Woo et al. 2018).
Attention also widely used in image translation. Ma et al. (2018) proposed the DA-GAN frame- work, which learns deep attention encoder to discover the instance level correspondences, Meijati jtt al. (2018) separates the object and background using a trainable attention network. InstaGAN (Mo et al. 2018) incorporates the instance information, like segmentation masks, to improve the multi-instance transfiguraiton. Generally, these methods are trying to boost the attention embed- ded component, while we using the attention mechanism to transfer more information from the
2

---

Under review as a conference paper at ICLR 2020
discriminator to the generator. We directly compare against several sfate-of-the-ee approaches in Section
3 METHOD
X' (Source)
Disoriminato
Gx
Xi
Attention
Gx
Dx
M
Gy
Dy
1 y:
(TTaggtt)
Figure 1: Overview of our framework. Left block is standard GAN with attention embedded discriminator. M2 is the attention map provided by the discriminator. The LI loss between generated s/ and corresponding ground truth y4 computed. Right side is the framework for unsupervised translation using cycle consistency. Ground truth Yi is not available and the L1 loss between x and x' is calculated instead.
Consider images from two different domains, source domain X' and target domain V. Data instances in source domain x E X' follow the distribution Pg, whereas instances in target domain y E y follow the distribution Pg- Notice that we do not have labels in both X' and y: Our goal. in the problem setting of image-to-imes different image domains, Gx x - y and/or Gy y - x, such that the differences between Pz and Gy Pg and the difference between Py and Gx o Pa are minimized. From the perspective of statistics, learning those two mapping functions can also be formulated as estimating the conditional distribution P(rly) and P()|z).
translation, aims to learn mapping functions Gss across these two
The main and unique idea of our approach is to incorporate the attention map generated by the discriminator, i.e., augment space of attention information A to the original input space X, to improve the smage-to-image translation. The attention map can be further transformed to an extra alpha channel paper, different attention mechanisms and concatenation methods have been studied and achieve promising results based a different task setting. Formally, our approach a joint-mapping learning from attention-augmennter space X e Ax to Y, and Y @ Ay to X cycle consistency applied, where @ the concatenate operation. Our method explicitly forces the generator to put more processing resources to attended areas so it can conduct sharp and clear translation. Generally, this approach can applied to any conditional GAN-based translation, hence, we call it GuideGAN. We will present the detail of our approach in the following subsections.
mask channel with weight) be interpreted
pixel weight map. In this
be described as
3.1 ARCHITECTURE
Our framework, as illustrated in Figure I is built upon GAN and attention mechanism. For the supervised learning setting, it consists of three components, a generator Gx, discriminator Dy and an attention transfer block T. It be extended to unsupervised setting using CycleGAN framework easily, which now has five components: including two generators G. and Gy, two domain adversarial discriminators Dy and Dx and shared attention transfer component T.
The training is based on each generator- discriminatrrtor pair. Considering a standard GAN, the gen- erator Gx translates image æ; in N' to image in domain y and the discriminator Dy tries to

---

Under review as a conference paper at ICLR 2020
distinguish whether its input is real or fake image in domain V. Here, denote y/ Gx (xx) as the output of generator, given 2;- Our attention embedded discriminator not only returns the prob- abillity of realness. (y() E [0, 1], but also an attention map Az, that highlights the focusing area of Dy. This attention map As, will be passed to the attention transfer block T to create an alpha the concatenation method, which will be described in Section 3.3 For simplicity, the constructed term is denoted as M,, given Agg despite its actual interpolation. Noteworthy is the input of our generator Gx is actually the concatenation of x; and Ma., which is represented as _r = ( Mz. At the start of the training, the attention map of each image is not available so we initialize it as an all-ones matrix As, Rm=x, where m x n is the shape of the input image. Other initialization methods, like random noise, have also be examined but have limited impact on the final result. The translation process of generator Gx x can be formulated as:
channel or a pixel weight map, depending
(1++1] = Gx(x; @ T(D (1) )):)), k 0,
(1)
where k and k + 1 denote the index of iteration and 0 is the parameter of Gx. We emphasize that the attention map from Dy is crucial because it allows Gx focus on informative areas. For example, if we only give the generator the raw input, Gx may waste its processing resources on some inessential locations and Dy will beat it easily. As a consequence. the loss of the discrim- inator quickly converges to zero and the generator can no longer efficiently update its parameter. Alternatively, by concatenating the raw input with Ma. the generator knows exactly where the dis- criminator is looking and allocates its processing resources properly on those areas. As illustrated in Figure ! we can easily extend this framework to perform the unsupervised translation by adding another GAN component and enforcing cycle consistency.
3.2 ATTENTION MAP
Remember that our discriminator provides an extra attention map Az, for each image generated from T₁. Therefore, we consider both post hoc attention (PHA) that does not change the capacity of the discriminator, and trainable attention module (TAM), which enhances the discriminator's distinguishing power.
Given input z, the post hoc attention map is constructed from the backward gradients, forward activation, or the mix of them. We use PatchGAN (Isola et al. (017)) as the bone of our discriminator. where 4 denotes i-th convolution layer in
The network can be formulated D (1o, I1 - (m] the network, and Actp = {01, 02,....... am is the set of activation map of corresponding layer. This kind of attention map is sensitive to layer selection; different layer selection leads to different attention map (Mei et al. 2019). More specifically, if t is the chosen layer, the attention map can be described as:
M = g( - (ar.i))
(2)
where e is the number of channels in t-th layer and g(-) applies the min-max normalization. This attention map only requires minor computation and works surprisingly well in most cases, but it may not achieve promising results when handling complex images. On the contrary, a TAM is suitable for such complex input since it simultaneously increases the capacity of generator and discriminator.
Our TAM follows the same 2-branch architecture of the attention block in RAM (Wang et al., 2017). See Appendix A for implementation details. However, each branch in their implementation contains several ResBlock He et al. (2016), which makes impractical our framework due to two reasons: 1) The discriminator conducts simple binary classification 2) The capacity gap between generator and discriminator is already significant. We replaced the Resblock by a simple convolution layer to simplify the network structure. First few layers of the discriminator extract the low-level information of the input, and passes it to following branches. Given the trunk branch output T(x) with the input x, the mask branch learns an attention map M (x) that softly weights the output of trunk branch. Put these two outputs together:
Fr,e (Mi,( + 11 (2,(x)
(3)
4

---

Under review as a conference paper at ICLR 2020
where i ranges over all spatial positions and {1. C} is the index of channels. Finally, a few consecutive convolutional layers will do the final prediction based on and attention map δ Ee Ma(x) constructed from mask branch output will be returned.
3.3 CONCATENATION
In this section, we propose two methods to blend the attention map M, with its corresponding input J. The first one is based on the aforementioned attention module in RAM (Wang et al. 2017). We compute the Residual Hadamard Production (RHP) of the attention map and original input. The reason of this operation are 1) Dot production with the attention range from zero to one will degrade the pixel value and cause fractional pixel problem (Mejjati et al.) 2018), 2) Attention mask can potentially break good property the raw input. For example, some pixels are not crucial in distinguish real and fake image, but they are still important for the image translation process. This RHP can be formulated as:
x' (g :6) + 1) x
(4)
where g(: Φ) is a transfer function that up-sample the attention map to the shape of original input. Another more intuitive concatenation is converting RGB image its RGBA version. RGBA, as color space, stands for red- with a 4-th alpha channel that indicates how opaque each pixel is. This concatenation somehow makes nonessential areas more transparent thus highlighting the crucial locations. Formally, it is described as:
is the three-channel RGB color model supplemented
x' = (Tr, Fg. Ba ol (Mz ())
(5)
where g(: is transfer function that maps attention map alpha channel. Follow the standard image pre-processing step, this concatenation can also be applied on gray scale image. Gray scale image can be transformed into RGB image by repeating its intensity for each RGB channel.
3.4 TRAINING LOSS
Let's start with supervised translation. The adversarial loss of the generator G and its discriminator D can be expressed as:
LGAN(G. D) Eyspearaly) Dog D(v)] + B (w) (log(1 D(G(# @ Ma)))
(6)
which is the adversarial loss of vanilla GAN. aims to minimize this objective while an adversary D tries maximize it, i.e., ming maxp LGAN D). However, this cost function is well known for its training difficulty. We adopt the modified least-squares loss, proposed in LSGAN (Mao et al., 2017). to further stabilize the training process and improve the quality of generated images. The adversarial loss now becomes:
(G, (w -1)3 + E Ex (z) (G( 4 Mz))
Adversarial loss alone does not guarantee a sound translation. It is beneficial to mix traditional loss like LI distance L2 distance between synthesized image and ground truth. Based the suggestion from pix2pix (Isola et al. 2017) that L1 loss encourages less blurry, we chose L1 loss as part of our training objective:
LG) = Ex,y G(r'Ik1
(8)
The final objective function in this setting is:
G = arg min max La (G)
G
We can easily extend this framework to conduct unsupervised translation by adding another pair of generator and discriminator and enforeing cycle consistency. Assume the generator Gx simulates
5

---

Under review as a conference paper at ICLR 2020
(O)aale
Sjunmer--WWinte
(Orage
(juumerr-
Method
A-:
O-+A
S-W
W-5
A-1
0-+A
S-W
W-+S
PHA+Alpha
3.69
186 0.24
101
0.37
4.11
104
1.23
PHA+RHP
6,31 + 0.60
2.99
0.38
198
0.33
1.03
0.26
369
0.27
4.30
118 0.36
1.55+
TAM-AIpha
10.80 0.71
7.26
0.47
2.37
0.35
1.76 0.37
593
0.31
6.70
1: 0.36
145 0.18
1.71 0.15
TAM+RHP
10.06 0.64
6.81 0.45
1.34
0.29
1.73
0.30
5.54
0.31
6,47
0.37
0.82 0.14
1,72 0.15
Table I: KIDx + std.x100 (Lower better) computed for different combination on ap- ple2orange and sunmer2winter. Left 4 columns shown the target only KID and the rest 4 columns shown the mean KID (Lower the better). Best results are bolded.
the map function x Y and discriminator Dy are trying to distinguish between G(x) and y. the objective of this GAN component EGAN(Gx Dy). The generator Gy and discriminator Dx is doing the same task in an opposite direction, their loss function is LGAN (Gy, Dx). Cycle consistency employed in such unsupervised setting because alleviate the shortness of paired data. It assumes that if image from domain X' has be translated to a fake image in domain y we should get the same image x by applying Gy Y + X. This behavior is formally presented as:
LGye(Gxx Gy) Ex-pasee (I - x1h]+ E Ey-puen (y =vl,] (10)
The final objective function for the unsupervised translation is:
G Gy = arg min max LGAN(Gx,Do) Dv) + LOAN(Gy,Gr + (Gx, Gr) (11) Gx,Gy Dr
EXPERIMENTS
crucial point of our framework is how we perform the inference in test phase. The attention map of same input from previous iterations be used at training phase. However, this information is not available testing, and some placeholders are required. To alleviate the problem that lead to this phenomenon, the generator should not rely much the attention map. Our proposed concatenation methods naturally handle this problem, since the attention map can only amplify the information but never hurt the original input. An all-one attention map is used as the placeholder based on the assumption that whole image is important.
4.1 ATTENTION AND CONCATENATION
Recall that we implemented two attention mechanisms and two concatenations. Then the problem is how to combine them properly. First, qualitative results are presented in Figure As discussed in Section 3 TAM is not good at handling simple datasets, e.g. apple2orangewhile the results are more attractive for more complex summer2winter dataset. By comparing alpha concatenation with RHP concatenation under post hoc attention, we find that the contrast ratio of the synthesized image normally too high and makes the image look unrealistic.
We also present a quantitative evaluation for each method combination in Table T Kernel Inception Distance (KID) (Biřkkowski al., 2018) is used as evaluation metric, which computes the squared MMD (Maximum Mean Discrepancy) between feature representations real and generated images. Different from the Frchet Inception Distance (Heusel et al.) (017). KID more reliable because of the unbiased estimator. While KID is unbounded, the lower its value, the more shared visual similarities there are between real and generated images. Numeric results the table justified our previous observations. Based on the overall performance across different tasks, most experiments are using PHA and RHP by default.
4.2 ОBJECT AND SCENERY TRANSLATION
We first evaluate method on four benchmark datasets. orange2apple. horse2zebra (Deng al. 2009) are for object transfer and day2night, summer2winter (Zhu et al. 2017) are two challenging scenery transfer tasks. day2night is cropped from BDD1 10k YYu et al. 2018), which contains 7870

---

Under review as a conference paper at ICLR 2020
ELORIDA
TEORIM
ELOROM
PHA+ Alpha
PHA+
TAM Alpha
TAM-
Figure 2: Different combination of attention and concatenation for apple2orange and sum- mer2winter. First column the real input. From second column the right: PHA and alpha channel, PHA and RHP, TAM and alpha channel, TAM and RHP
(Apple=+(Orrange
(H)orse-
(Dlay+(Night
(Sjummer-= Wilnter
Method
A-0
O-
H-+Z
7-H
D-
N-ID
S-W
W-1$
CycleGAN
SurGAN
13.32 1 0.52
11.19
0.51
12.42
0.74
12:21
1: 0.89
5.37 1: 0.43
8.49 0.34
8.05 1: 0.37
8.72 1: 0.47
AGGAN
10.61 0.79
4.57 + 0.30
4.12
0.80
+ 0.40
8.09
0.37
7.85 0.29
3.45
4: 0.43
2.75
0.20
UNIT
17.41 1.13
7.26
0.57
12.25
0.74
12:37
1 0.84
283
1 0.30
11.00
0.53
6.20
0.25
5.99
1: 0.28
Ours (PHA-RHP)
6.3 0.60
139
103
0.35
3.221
2.76
1 0.32
6.96
0.38
198
1 0.33
1,03
I 0,26
Table 2: KIDx 100
std. 100 (Lower is better) computed using only target domain for different methods and on different datasets. Best results are bolded.
images of daytime street traffic signs and 8592 night street traffic signs. All data were split into train and test randomly (80%/20% split).
Then for all tasks, we present target KID in Table and mean KID in Table The target only KID meaningful when the background of image not important. For example, in apple2orange and horse2zebra tasks, we only care about objects in those images. Under such scenarios, our proposed framework outperforms all baselines all tasks except one task day2night. Still, our result is very close to the best. This observation is consistent with our qualitative evaluation in Figure 3 where our fake horse (zebra) is much more realistie than the counterparts produced by baselines. However, we can clearly see the background changed using our method, even though we still have leading performance on apple2orange and summer2winter regarding the mean KID between source and target domain. It's surprising to that simplest CycleGAN model got the first place in day2night, which is a very hard task compared to two object transfer datasets. Another interesting observation is our framework got 6.96 for the task night->day while CycleGAN got 7.68, given CycleGAN outperforms our method in the easier day->night direction. That's 's in the opposite direction as presented in Figre

---

Under review as a conference paper at ICLR 2020
StarGAN
Figure 3: Image-to-lmage translation results generated by different approaches apple2orange and horse2zebra. Every two rows from top: apple- orange rappee, zebra- shors, horse-szebra.
4.3 CITYSCAPE TRANSLATION
We then evaluate our method on Cityscape (Cordts et al. (016) using FCN scores. Appendix B shown for detailed evaluation protocol. We train Cityscape, and compare the output label images with the ground truth. We used only RHP concate- nation for this task.
and label-xphoto tasks the
find that our method significantly outperforms the baselines in these experiment, especially when PHA and RHP work together, shown in Table 4 The image translation result is also presented in Figure 6 The significant improvement in the pixel-level accuracy comes from the guidance of the attention map, which aligns with expectations. However, the improvement for metrics,
8

---

Under review as a conference paper at ICLR 2020
CycleGAN
Figure Image-to-Image translation results generated by different approaches on day2night and summer2winter. Every two rows from top: night day--night winter
sum-
(O)rane
(Horse-
(7ebrra
(Night
(Summer--- (Winter
Method
-0
0 +A
I1 iz
2-1
D -N
N D
S-W
W
CycleGAN
11.02 + 0.00
9x2=053
10.25 0.25
11.44 + 0.38
1.95 = 0.13
36146.
0012
1.400.12
StrGAN
9.15
8.4
7.14 0.48
0.20
5.18::23
0.17
4.14
AGGAN
6.44
5.32 044
0.27
671 4 0.27
0.14
0.118
3,15
0..19
2.45 0.13
Ous (PIA-RHPP]
3.69 0.27
4.30
0.31
8.42 0.47
846 0.44
0.15
4.38.02
11860.16
1,55 0.13
Table 3: KIDx 100 + std. x 100 (Lower is better) computed using both target and source domain for different methods and on different datasets. Best results are bolded.
such class accuracy and Intersection over Union (loU), are limited. It is probably because the attention map only focuses on a few domain specific classes, so the generator works too hard on those classes and ignores others. Small class number per image might be another potential reason

---

Under review as a conference paper at ICLR 2020
Label Photo
Photo Photo->Label -Label
Method
Per-pixel acc.
Per-class acc.
ToU
Per-pixel acc.
Per-class acc.
ToU
CycleGAN
0.42
0.15
0.10
0.56
0.21
0.17
UNIT
0.48
0.17
0.11
0.58
0.18
0.14
AGGAN
0.37
0.11
0.09
0.49
0.14
0.10
StarGAN
0.47
0.16
0.11
0.61
0.21
0.17
Ours (PHA)
0.52
0.20
0.12
0.60
0.24
0.19
Ours (TAM)
0.49
0.19
0.10
0.59
0.23
0.19
Table 4: FCN-scores (Higher is better) for different methods, evaluated Cityscape label+>photos in unsupervised setting.
Label Photo
Photo-Labee
Method GAN
Per-pixel acc.
Per-class acc.
loU
Per-pixel acc.
Per-class acc.
loU
0.22
0.05
0.01
0.32
0.08
0.02
cGAN
0.57
0.20
0.14
0.71
0.26
0.21
pix2pix
0.61
0.22
0.16
0.80
0.43
0.32
(PHA))HA) 0.63
0.23
0.16
0.81
0.42
0.32
Ours(TAM) 0.63
0.22
0.16
0.75
0.40
0.30
Table 5: FCN-scores (Higher is better) for different methods, evaluated on Cityscape label+>photos in supervised setting. See Appendix for qualitative result.
for this phenomenon, since cannot increase the accuracy for nonexistent class objects. Empirical justifications are available in Appendix D
Mcanwhile, the improvement of the supervised translation is not sharp the unsupervised trans- lation according to Table 5 Yct still shows that we can further improve the translation results with little extra computation, especially when PHA has been chosen. We believe that the major reason actually due to strong regularizations, which are from the L1 distance between paired images. The generator receives
feedbacks when paired image is available. image and 2) The prediction from the discriminator. Recall that the idea behind our framework is letting the discriminator provide more useful information, but maybe the information from LI loss is already sufficient. Appendix CerlixxCooffe offfers qualitative result.
The LI loss between paired
CONCLUSION
we have proposed a novel method incorporating attention map from discriminator for image-to- image translation. The experiments on different datasets have shown successful translation both supervised and unsupervised setting. We remark that our idea can apply on any GAN-based model with little modification, such those baselines in the paper. Nonetheless, the results are sensitive to the selection of attention module and concatenation. Investigating the impact of different attention mechanism and new tasks could be an interesting research direction in the future.
REFERENCES
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:
7/ //openreview net wwt.:eeoutidddiiieeeee. _0004___55xe__ qw5xe
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875 2017.
Wasserstein gan.
arXiv preprint
Mikolaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint 1rXiv:1801.01400 2018.
Yun Cao, Zhiming Zhou, Weinan Zhang, and Yong Yu. Unsupervised diverse colorization via gener- ative adversarial networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 151-166. Springer, 2017.
10

---

Under review as a conference paper at ICLR 2020
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient- bbseed visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 839-849--7 849.. IEEE, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789- 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference Computer Vision and Patlern Recognition (CVPR), 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Emily Denton, Soumith Chintala, Rob Fergus, al. Deep generative image models using laplacian pyramid of adversarial networks. In Advances in neural information processing systems, pp. 1486-1494, 2015.
Abel Gonzalez-Garcia Joost van de Weijer, and Yoshua Bengio. Image-to-image translation for cross-domain disentanglement, In Advances in Neural Information Processing Systems, pp. 1287- 1298, 2018.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp. 7767-5777,201 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 0026-6637, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatie networks. In CVPR, pp. 7132-7141, 2018.
Xun Huang, Ming-Yt Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 172-189, 2018.
Minyoung Huh. Shao-Hua Sun, and Ning Zhang. Feedback adversarial learning: Spatial feedback for improving generative adversarial networks. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 1476-1485, 2019.
Phillip Isola, Jun-Yen Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning- Haaminng-oolume Volume 70, pp. 1857-1865. JMLR. org, 2017
Christian Ledig, Lucas Theis, Ferenc Huszár. Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to- tonge-to-image translation networks. In Advances neural information processing systems, pp. 700-708, 2017.
11

---

Under review as a conference paper at ICLR 2020
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 31.3-334000 2015.
Shuang Ma, Jianlong Fu, Chang Wen Chen, and Tao Mei. Da-gan: Instance-level image translation by deep attention generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5657-5666, 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 2794-2802, 2017.
Xiaoguang Mei, Erting Pan, Yong Ma, Xiaobing Dai, Jun Huang, Fan Fan, Qinglei Du, Hong Zheng, and Jiayi Ma. Spectral-spatial attention networks for hyperspectral image classification. Remote Sensing. 11(8):963, 2019.
Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim. Unsupervised eecention-guieed mmage-to-image translation. In Advances in Neural Information Processing Systems, pp. 3693-3703, 2018.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. 1rXiv:1411.1788 2014.
arXiv preprint
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Instagan: Instance-aware image-to-image translation. arXiv preprint 1rXiv:1812.1000 2018.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Training generative neural samplers
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- Springer, 2015.
assisted intervention, pp.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, and Yan Yan. Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2417-2426, 2019.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In CVPR, pp. 3156-3164,
2017.
Ting-Chur Wang, Ming- Yu Liu, Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7798-8807, 2018.
Sanghyun Woo, Jongchan Park, Joon- attention module. In ECCV, pp. 3-19, 2018.
Joon-Young Young Lee, and In So Kweon. Cbam: Convolutional block
Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Texturegan: Controlling deep image synthesis with texture patches. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8456-8465, 2018.
12

---

Under review as a conference paper at ICLR 2020
Qingsong Yang, Pingkun Yan, Yanbo Zhang. Hengyong Yu, Yongyi Shi, Xuanqin Mou, Man- nudeep Kalra, Yi Zhang, Ling Sun, and Ge Wang. Low-dose et image denoising using a generative adversarial network with wasserstein distance and perceptual loss. IEEE transactions 2018.
on medical imaging, 37(6):1348-
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv: 180.018805.0487,, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint 1805.0880 2018.
Lvmin Zhang. Ji, Xin Lin, and Chunping Liu. Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier gan. In 2017 4th IAPR Asian Conference Pattern Recog- nition (ACPR), pp. 506-511. IEEE, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 20-222222222 2017.
13

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
For the unsupervised Cityscape translation, we adopted the network architectures of CycleGAN (Zhu et al. (017) as the basic of our proposed model. In specific, we adopted the ResNet 6-blocks (He et al., 2016) generator and the PatchGAN (Isola et al., 2017) discriminator. This generator contains 2 down-sampling blocks, residual blocks and 2 up-sampling blocks. For the supervised translation, adopted the UNet-122 (Ronneberger et al. 2015) generator and same PatchGAN discriminator. The PatchGAN discriminator is composed of convolution layers, including normal- ization and ReLU layers.
Before diving into the detail of our modified discriminator, let us first describe the details of RAM's 2-branch architecture (Wang et al. (017)) They built very deep network with numbers of attention blocks. Each attention block contains two branches: mask branch and trunk branch. Mask branch cascades the input features through bottom-up top-down architecture that mimics human attention. Trunk branch is applied as feature processing. To build a TAM discriminator with this 2-branch architecture, we replaced the ResBlock by a simple convolution layer, as presented in the left part of Figure 5 In this TAM discriminator, we use the first convolution layer as feature extractor, three consecutive convolution layers for trunk branch and the one convolution layer for classifier. The mask branch is composed of two downsampling layers, two convolution layers and one upsampling layer.
As presented in the right side of Figure 5 we selected the 4-th convolution layer compute the post hoc attention map, based on the formula Section 3.2 All attention maps will be detached from the computation graph and be resized the shape of original input. Either by bilinear upsampling layer or by a small 3-layer neural network.
M()
Down
Down
Camu
Up
Ax
Conv
comv
Com-3
cone
Comes 5
Conu.1
Comu2
Donv.3
Domu.S
TO)
Ax
Figure 5: Left: PatchGAN discriminator using TAM, the attention map is denoted As; Right: Patch discriminator using post hoc attention, the attention map Ar is computed from 4-th conv layer.
horse2zebra, apple2orunge and day2night tasks are performed under the unsupervised setting. For this three tasks, we adopted ResNet 9-blocks generator aforementioned PatchGAN discrim- inator. Similar to prior works, applied Instance Normalization (IN) for both generators and discriminators. the preprocessing step, we resized the input image to 143 x 143 then randomly cropping back to 128 x 128 for all Cityscape related tasks. We resized the input image to 286 286 then randomly cropping back to 256 x 256 for the rest tasks.
For all the experiments, simply set the weight factor of the GAN loss 10 and the weight factor of LI loss to 10 for our objective. For example, our implementation uses following objective for supervised training.
14

---

Under review as a conference paper at ICLR 2020
G = arg min max 10LGAN(G, D) + 10Lra(G)
G
We used Adam optimizer with batch size 1, training on Quadro 8000 GPU, All networks were trained from scratch, with learning rate of 0.0002 for both the generator and discriminator, and 81 0.5, 82 = 0.999 for the optimizer. Similar to CycleGAN, we kept learning rate for first 100 epochs and linearly decayed to for next 100 epochs for apple2orange and Cityscape related tasks, and kept learning rate for first 50 epochs and linearly decayed to 0 for next 50 epochs for horse2zebra and day2night datasets.
B EVALUATION METRIC IN DETAIL.
Evaluating the quality of synthesized images is an open and difficult problem. In this paper, we trained network perform label->pholo and photo->label translation in both supervised and unsupervised manner. Classical metrics such LI, or L2, distance between the real image and syn- thesized image
suitable since they do assess joint statísties of the result. Researchers in image segmentation are widely using pretrained semantic classifier to measure the discriminability of the generated image a surrogate metric. The assumption behind such measurement that the generated images are indeed realistic, classifiers pretrained on real images should classify the synthesized image correctly as well, For the Cityscapes dataset, we used the FCN-8s (Long et al.) 2015) network released by Zhu al. (2017), which is pretrained on the Cityscape dataset.
The metrics we used our experiment are per-pixel accuracy, per-class aceuracy. and Intersection over Union (IoU). Per-pixel accuracy, namely, is the ratio between the number of correctly predicted pixels and total number of pixels. can be presented as:
P(x) M x N
per-pix acc.
(12)
where P(x) denotes the number of correctly predicted pixels and M x N is the sharp of input image. Per-class accuracy, also known as macro-average. is self explanatory. computes the the accuracy for each class and then compute the average. It's formulated as:
P(r,k)
per-class acc.
(13)
K
G(k)
KEK
where K is the set of classes, P(r, k)) denotes the number of correctly predicted pixels for class k and G(k) the total number of pixels that belongs to class k in ground truth.
Intersection over Union (loU) is another often used metric for image segmentation. It computes the ratio between the number of pixels seat in the intersection between predicted segmentation mask and ground truth, and the union of them. Let P(x) be the prediction and GT be the ground truth.
IoU = P(x) GT(())
(14)
P(x) (T(()
For all three aforementioned metric, the highest score is one, and the closer to one, the better.
C QUALITATIVE RESULT FOR Cityscape
Cityscape translation result is presented in Figure
D HYPOTHESIS JUSTIFICATION ON Cityscape
In order to empirically justify our hypothesis for the limited improvement over per-class accuracy and loU in Section We conduct two additional experiments and show its result here. We first
15

---

Under review as a conference paper at ICLR 2020
Originl Domain
GydeGAN
StarGAN
UNIT
AGGAN
Curs
Ground Truth
Figure 6: Different unsupervised translation methods for mapping labels<>photos trained on Cityscape images.
10
322
Figure 7: The statistic frequency for all 18 classes presented in the Cityscape dataset.
compute the statistic information of each class. Figure 7 shows the statistic frequency of each class. Namely, it tell us how many images contains the specific class in the dataset. Another useful statistic information, which tell us the average frequency each class, is provided in Figure
This statistic justified our second hypothesis that some class objects merely presented in the image thus it's hard to improve the per-class accuracy and IoU. We then extracted the attention map of whole training set and computed the average per-class attention map intensity. More specific, first perform binary normalization over all attention map using a threshold a (In this experiment 0.5). So assume pixel is crucial if the attention value it is larger than a. For a specific class in one image, it's regarded as attended least half of its pixel crucial. In the Figure show the average per-class attention map intensity in different epochs.
16

---

Under review as a conference paper at ICLR 2020
230
175
125
75-
30
Cinusnana
Fporh Epoch-4e kpoch
0.6
0.4
0.0
indurtioot
Figure 9: The average per-class attention map intensity in epochs 20, 40, 60.
Based on aforementioned figures, it's hard to find out that the attention map are focusing on small classes. For example, rider and terrain. If the generator tried too hard to fix those small wholes but ignore the major classes, like car, the per-class accuracy and loU will also be affected. Since the contribution from generating good riders and terrain is significantly less than the contribution from generating good cars. This experiment also justified our first hypothesis.
E ATTENTION MAP DURING TRAINING
We present some intermediate training results with its attention map in Figure 10 Figure I| and
Figure 12 The white area in the attention map indicates that region is important. Please note that the attention map indicates the behavior of the discriminator thus some of them may not make sense from human's 's perspective.
17

---

Under review as a conference paper at ICLR 2020
Post
RAM
Booiphoe
Input
Output
Attention
Input
Output
Attention
Figure 10: Inputs, outputs and corresponding attention maps at training epoch 10. Left: attention map generated by the post hoc attention; Right: attention map generated by RAM attention mecha- nism.
Post hoc
RAM
Input
Attention
Map
Output
Attention
Map
Figure 11: Inputs, outputs and corresponding attention maps at training epoch 50. Left: attention map generated by the post hoc attention: Right: attention map generated by RAM attention mecha- nism.
F MORE TRANSLATION RESULTS
More translation results are provided in this section.
18

---

Under review as a conference paper at ICLR 2020
Post hoc
RAM
Input
Ouiput
Map
Input
Output
Attention
Map
Figure 12: Inputs, outputs and corresponding attention maps at training epoch 100. Left: atten- tion map generated by the post hoc attention; Right: attention map generated by RAM attention mechanism.
SPEES
15
15
Figure 13: Additional translation results on day2night datasct with default setting (PHA RHP). From left to right: real daytime images, fake night images, real night images, fake daytime images.
19

---

Under review as a conference paper at ICLR 2020
Figure 14: Additional translation results on apple2orange dataset with default setting (PHA RHP).
From left to right: real apple images, fake orange images, real orange images, fake apple images.
Figure 15:
Additional translation results on horse2zebra dataset using RAM + RHP. From left to right: real horse images, fake zebra images, real zebra images, fake horse images.
20


Reviews:
[{"review_id": "rJxCLIITcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an extension of the conditional GAN objective, where the generator conditions on an attention map produced by the discriminator in addition to the input image. The motivation is that the discriminator is usually too powerful, and so the gradient the generator receives is often too small in magnitude. By conditioning on the attention map, the generator could leverage information about the regions in the image that the discriminator attends to and use it to generate a new image that better fools the discriminator. \n\nMy main concern is about whether the proposed extension achieves the desired goal. The intuitive motivation provided in the paper aims to add a cooperative component to the two-player game, but the min-max objective corresponds to a zero-sum adversarial game. As a result, when training the discriminator, the discriminator is encouraged to reveal as little information as possible via the attention map, so that the loss maximized. This appears to be the opposite of the desired behavior, so the objective needs to be reformulated. \n\nAlso, it is unclear how inference is performed: at test time, the attention map is unknown and so some placeholder must be used in its place. The paper should clarify what is done at test time, and clearly state the shortcomings as a result of this, i.e. different procedures are used for training and testing, which is not principled. I imagine the generator could rely too much on the attention map as a result - how this is alleviated/prevented should be explained. \n\nFigure 2: Only the qualitative results for unsupervised image-to-image translation are available; qualitative results for supervised image-to-image translation should also be provided. \n\nWhile the quantitative improvement over existing methods is somewhat insignificant, I appreciate the authors discussing their hypotheses why this might be the case. It would be more useful to empirically validate these hypotheses as well. For example, for the claim that \"maybe the attention map only focuses on a few domain specific classes so the generator works too hard on those classes and ignores others\", it might be good to compute the average per-class attention map intensity to show that some classes appear rarely in the attention map. \n\nThe evaluation protocol should be explained in greater detail (perhaps in the appendix); the segmentation model (which I assume is FCN) should be described and each of the evaluation metrics (per-pixel acc., per-class acc. and IoU) should be described for the benefit of researchers outside the area. \n\npg. 4: \"in their implementation contains several Resblock (He et al., 2016), which makes it infeasible in our framework\". Why is it infeasible?\npg. 6: What are the architectures used by the baselines? Are they comparable to the architecture the proposed method used?\n\nMinor Issues:\n\npg. 3: \"differences between P_x and G_Y \\cdot P_y, P_y and G_X \\cdot P_x are minimized\" - confusing; should rephrase as \"the difference between P_x and G_Y \\cdot P_y and the difference between P_y and G_X \\cdot P_x are minimized\". Also should replace \\cdot with \\circ. \npg. 4: \"like random noisy\" -> \"like random noise\"\npg. 4, last paragraph: \"Our trainable attention module follows the same structure of the attention block in RAM (Wang et al., 2017). They built a very deep network with several such blocks, each containing two branches: mask branch and trunk branch. Mask branch cascades the input features through a bottom-up top- down architecture that mimics human attention. Trunk branch is applied as feature processing.\" - this is very confusing; it would be easier to refer readers to the appendix. \npg. 5 - \"Attention mask can potentially break good property of the raw input.\" - what does this mean?\npg. 6 - \"as showed in Table 4.1\" -> \"as shown in Table 4.1\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkeopBxRFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis paper proposes a GAN with an attention-based discriminator for I2I translation, GuideGAN. The proposed discriminator provides the probability of real /fake and an attention map which reflects the salience for image generation. They apply their method to CycleGAN. GuideGAN is evaluated on the Cityscape dataset, horse2zibra, apple2orange, and day2night, compared to UNIT, CycleGAN, StarGAN, cGAN, and pix2pix.\n\n[Pros]\n- Quantitative results\n\n[Cons]\n- Novelty is restricted. Even if there is no work using attention in the discriminator, it is hard to tell that the use of attention is not novel. \n- The used datasets are not challenging. Why the quantitative results are evaluated on the Cityscape? How about FID on other datasets such as CelebA and Summer2Winter, or higher-resolution data?\n- The quantitative scores are not impressive. The gaps look insignificant.\n- How large are additional parameters? and How long does it spend training compared to CycleGAN?\n- The authors describe two concatenation methods. How about the results of simple RGBA?\n- Why post hoc and RAM are evaluated on different datasets from each other in qualitative results?\n- User study will be helpful.\n- Basically, this architecture can be applied to various GANs. Do the authors have any result on other GANs?\n \n\n[Minor]\nIn Figure 1, M is used without the definition.\nIn row 2 of page 3, follows --> follow\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkltujxotB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a feedback mechanism in the GAN framework which improves the quality of generated images in the context of image-to-image translation. The key contribution is that the discriminator not only predicts the probability of an image being real or fake, but also outputs a map which indicates where the generator should focus in the next iteration in order to make its results more convincing. The paper explores ways of obtaining such a map 1) by summing feature activations of the discriminator on a specific or group of layers  2) by predicting it via augmenting the capacity of the discriminator. After such a map is obtained, it is concatenated with the input image and fed iteratively to the generator. \n\nThe proposed setting have been tested on the setting of supervised and unsupervised image translation on 4 datasets. Quantitative experiments show that the proposed approach improves over other baselines. I think this paper introduces an interesting and important new GAN framework. However, I feel that the paper requires a major revision strengthening the experiments, before it can be reconsidered for ICLR: \n\nMore qualitative results showing comparisons with other algorithms should be shown for Day-Night, Apple-Orange, Horse-Zebra. The only comparison available in the paper is on the well constrained problem of segmentation maps to images.\nMore quantitative experiments should be provided for other datasets (perhaps using FID and KID). It is not entirely clear if the produced results are better than cycleGAN’s (my subjective analysis is that the results of cycleGAM look better on Fig. 2).\nThe paper is closely related to [Huh et al: Feedback Adversarial Learning: Spatial Feedback for Improving Generative Adversarial Networks] in that they share the idea of a feedback mechanism from the discriminator. It hence seems reasonable to compare with this approach.  \nI was struggling to understand precisely how the StarGAN results were obtained on CityScapes: As a multi-modal image-to-image translation model, StarGan takes as input, an image and a binary vector pointing to which modality to transform the image into. In the case of CityScape, there is no such multi-modality (at least none that is provided as ground truths, via for example, a binary vector). More details on this process would make the experimental section clearer. \nTable 4.1 is often used however such a Table does not exist (probably Table 1,2 was meant here).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to augment the conditional GAN discriminator with an attention mechanism, with the aim to  help the generator, in the context of image to image translation. The reviewers raise several issues in their reviews. One theoretical concern has to do with how the training of the attention mechanism (which seems to be collaborative) would interact with the minimax, zero-sum nature of a GAN objective; another with the discrepancy in how the attention map is used during training and testing. The experimental results were not significant enough, and the reviewers also recommend additional experiment results to clearly demonstrate the benefit of the method. 
Decision: Reject