Structured Prediction using cGANs with Fusion Discriminator

Abstract: We propose a novel method for incorporating conditional information into a generative adversarial network (GAN) for structured prediction tasks. This method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data. This method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree. The proposed method is conceptually simpler than the joint convolutional neural network - conditional Markov random field (CNN-CRF) models and enforces higher-order consistency without being limited to a very specific class of high-order potentials. Experimental results demonstrate that this method leads to improvement on a variety of different structured prediction tasks including image synthesis, semantic segmentation, and depth estimation.

Reviews and author responses:

This paper presents a new method for incorporating conditional information into a GAN for structured prediction tasks (image conditioned GAN problems). The proposed method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data. The proposed method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree. The proposed method is conceptually simpler than joint CNN-CRF models and enforces higher-order consistency without being limited to a very specific class of high-order potentials. Thorough experimental results on Cityscapes and NYU v2 verify the efficacy of the proposed method.  I believe this paper shows a promising approach to solve the structured prediction problems that I have not seen elsewhere so far. The paper is written clearly, the math is well laid out and the English is fine.

We thank the reviewer for their comments and appreciate the support for our work.

This paper proposes a new method to input data to a conditional discriminator network. The standard setup would simply concatenate the "condition" image with the "output" image (i.e., the real image, or the generator's output corresponding to the condition). The setup here is to feed these two images to two separate encoders, and gradually fuse the features produced by the two. The experiments show that this delivers superior performance to concatenation, on three image-to-image translation tasks.

I think this is a good idea, but it's a very small contribution. The entire technical approach can be summarized in 2-3 sentences, and it is not particularly novel. Two-stream models and skip connections have been discussed and explored in hundreds of papers. Applying these insights to a discriminator is not a significant leap. 

The theoretical "motivation" equations in Sec. 3.1 are obvious and could be skipped entirely. 

In summary, the paper makes sense, but it does not present substantively new ideas. I do not recommend the paper for acceptance. 

This paper presents an a particular architecture for conditional discriminators in the cGAN framework. Different to the conventional approach of concatenating the conditioning information to the input, the authors propose to process them separately with two distinct convolutional networks fusing (by element-wise addition) intermediate features of the conditioning branch into the input branch at each layer.

Pros:
+ The writing is mostly clear and easy to follow.
+ I feel that exploring better conditioning strategies is an important direction. Quite often the discriminator discards additional inputs if no special measures against this behaviour are taken.
+ The proposed method seem to outperform the baselines

Cons:
- I’m generally not excited about the architecture as it seems a slight variation of the existing methods. See, for example, the PixelCNN paper [van den Oord et al., 2016] and FiLM [Perez et al., 2017].
- Theoretical justification of the approach is quite weak. The paper shows that the proposed fusion method may result in higher activation values (in case of the ReLU non-linearity, other cases are not considered at all) but this is not linked properly to the performance of the entire system. Paragraph 3 of section 3.1 (sentence 3 and onward) seems to contain a theoretical claim which is never proved.
- It seems that the authors never compare their results with the state-of-the-art. The narrative would be much more convincing if the proposed way of conditioning yielded superior performance compared to the existing systems. From the paper it’s not clear how bad/good the baselines are.

Notes/questions:
* Section 3.1, paragraph 1: Needs to be rephrased. It’s not totally clear what the authors mean here.
* Section 3.1, paragraph 4: “We observed that the fusion …” -  Could you elaborate on this? I think you should give a more detailed explanation with examples because it’s hard to guess what those “important features” are by looking at the figure.
* Figure 4: I would really want to see the result of the projection discriminator as it seems to be quite strong according to the tables. The second row of last column (which is the result of the proposed system) suspiciously resembles the ground-truth - is it a mistake?
* Figure 5: It seems that all the experiments have not been run until convergence. I’m wondering if the difference in performance is going to be as significant when the model are trained fully.

In my opinion, the proposed method is neither sufficiently novel nor justified properly. On top of that, the experimental section is not particularly convincing. Therefore, I would not recommend the paper in its present form for acceptance.

Meta-review: All three reviewers argue for rejection on the basis that this paper does not make a sufficiently novel and substantial contribution to warrant publication. The AC follows their recommendation.

Decision: rejected-papers