Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an extension of the conditional GAN objective, where the generator conditions on an attention map produced by the discriminator in addition to the input image. The motivation is that the discriminator is usually too powerful, and so the gradient the generator receives is often too small in magnitude. By conditioning on the attention map, the generator could leverage information about the regions in the image that the discriminator attends to and use it to generate a new image that better fools the discriminator. 

My main concern is about whether the proposed extension achieves the desired goal. The intuitive motivation provided in the paper aims to add a cooperative component to the two-player game, but the min-max objective corresponds to a zero-sum adversarial game. As a result, when training the discriminator, the discriminator is encouraged to reveal as little information as possible via the attention map, so that the loss maximized. This appears to be the opposite of the desired behavior, so the objective needs to be reformulated. 

Also, it is unclear how inference is performed: at test time, the attention map is unknown and so some placeholder must be used in its place. The paper should clarify what is done at test time, and clearly state the shortcomings as a result of this, i.e. different procedures are used for training and testing, which is not principled. I imagine the generator could rely too much on the attention map as a result - how this is alleviated/prevented should be explained. 

Figure 2: Only the qualitative results for unsupervised image-to-image translation are available; qualitative results for supervised image-to-image translation should also be provided. 

While the quantitative improvement over existing methods is somewhat insignificant, I appreciate the authors discussing their hypotheses why this might be the case. It would be more useful to empirically validate these hypotheses as well. For example, for the claim that "maybe the attention map only focuses on a few domain specific classes so the generator works too hard on those classes and ignores others", it might be good to compute the average per-class attention map intensity to show that some classes appear rarely in the attention map. 

The evaluation protocol should be explained in greater detail (perhaps in the appendix); the segmentation model (which I assume is FCN) should be described and each of the evaluation metrics (per-pixel acc., per-class acc. and IoU) should be described for the benefit of researchers outside the area. 

pg. 4: "in their implementation contains several Resblock (He et al., 2016), which makes it infeasible in our framework". Why is it infeasible?
pg. 6: What are the architectures used by the baselines? Are they comparable to the architecture the proposed method used?

Minor Issues:

pg. 3: "differences between P_x and G_Y \cdot P_y, P_y and G_X \cdot P_x are minimized" - confusing; should rephrase as "the difference between P_x and G_Y \cdot P_y and the difference between P_y and G_X \cdot P_x are minimized". Also should replace \cdot with \circ. 
pg. 4: "like random noisy" -> "like random noise"
pg. 4, last paragraph: "Our trainable attention module follows the same structure of the attention block in RAM (Wang et al., 2017). They built a very deep network with several such blocks, each containing two branches: mask branch and trunk branch. Mask branch cascades the input features through a bottom-up top- down architecture that mimics human attention. Trunk branch is applied as feature processing." - this is very confusing; it would be easier to refer readers to the appendix. 
pg. 5 - "Attention mask can potentially break good property of the raw input." - what does this mean?
pg. 6 - "as showed in Table 4.1" -> "as shown in Table 4.1"

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: [Summary]
This paper proposes a GAN with an attention-based discriminator for I2I translation, GuideGAN. The proposed discriminator provides the probability of real /fake and an attention map which reflects the salience for image generation. They apply their method to CycleGAN. GuideGAN is evaluated on the Cityscape dataset, horse2zibra, apple2orange, and day2night, compared to UNIT, CycleGAN, StarGAN, cGAN, and pix2pix.

[Pros]
- Quantitative results

[Cons]
- Novelty is restricted. Even if there is no work using attention in the discriminator, it is hard to tell that the use of attention is not novel. 
- The used datasets are not challenging. Why the quantitative results are evaluated on the Cityscape? How about FID on other datasets such as CelebA and Summer2Winter, or higher-resolution data?
- The quantitative scores are not impressive. The gaps look insignificant.
- How large are additional parameters? and How long does it spend training compared to CycleGAN?
- The authors describe two concatenation methods. How about the results of simple RGBA?
- Why post hoc and RAM are evaluated on different datasets from each other in qualitative results?
- User study will be helpful.
- Basically, this architecture can be applied to various GANs. Do the authors have any result on other GANs?
 

[Minor]
In Figure 1, M is used without the definition.
In row 2 of page 3, follows --> follow

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper introduces a feedback mechanism in the GAN framework which improves the quality of generated images in the context of image-to-image translation. The key contribution is that the discriminator not only predicts the probability of an image being real or fake, but also outputs a map which indicates where the generator should focus in the next iteration in order to make its results more convincing. The paper explores ways of obtaining such a map 1) by summing feature activations of the discriminator on a specific or group of layers  2) by predicting it via augmenting the capacity of the discriminator. After such a map is obtained, it is concatenated with the input image and fed iteratively to the generator. 

The proposed setting have been tested on the setting of supervised and unsupervised image translation on 4 datasets. Quantitative experiments show that the proposed approach improves over other baselines. I think this paper introduces an interesting and important new GAN framework. However, I feel that the paper requires a major revision strengthening the experiments, before it can be reconsidered for ICLR: 

More qualitative results showing comparisons with other algorithms should be shown for Day-Night, Apple-Orange, Horse-Zebra. The only comparison available in the paper is on the well constrained problem of segmentation maps to images.
More quantitative experiments should be provided for other datasets (perhaps using FID and KID). It is not entirely clear if the produced results are better than cycleGAN’s (my subjective analysis is that the results of cycleGAM look better on Fig. 2).
The paper is closely related to [Huh et al: Feedback Adversarial Learning: Spatial Feedback for Improving Generative Adversarial Networks] in that they share the idea of a feedback mechanism from the discriminator. It hence seems reasonable to compare with this approach.  
I was struggling to understand precisely how the StarGAN results were obtained on CityScapes: As a multi-modal image-to-image translation model, StarGan takes as input, an image and a binary vector pointing to which modality to transform the image into. In the case of CityScape, there is no such multi-modality (at least none that is provided as ground truths, via for example, a binary vector). More details on this process would make the experimental section clearer. 
Table 4.1 is often used however such a Table does not exist (probably Table 1,2 was meant here).

review_title: Official Blind Review #2

Meta-review: The paper proposes to augment the conditional GAN discriminator with an attention mechanism, with the aim to  help the generator, in the context of image to image translation. The reviewers raise several issues in their reviews. One theoretical concern has to do with how the training of the attention mechanism (which seems to be collaborative) would interact with the minimax, zero-sum nature of a GAN objective; another with the discrepancy in how the attention map is used during training and testing. The experimental results were not significant enough, and the reviewers also recommend additional experiment results to clearly demonstrate the benefit of the method. 
Decision: Reject