Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a method for improving the diversity of Generative Adversarial Network (GAN) by promoting the Gnet's weights to be as informative as possible. This is achieved by penalizing the correlation between responses of hidden nodes and promoting low entropy intra node. Numerical experiments that demonstrate the diversity increment on the generated samples are shown.

Concerns.

The paper is hard do tear and it is deficit to identify the precise contribution of the authors. Such contribution can, in my opinion, be summarized  in a potential of the form

with

$$
R_BRE = a R_ME+ b R_AC = a \sum_k  \sum_i s_{ki}^2   +  b \sum_{<k,l>} \sum_i \{ s_{ki} s_{li} \}   
$$
(Note that my version of R_ME is different to the one proposed by the authors, but it could have the same effect)

Where a and b are parameters that weight the relative contribution of each term  (maybe computed as suggested in the paper).

In this formulation:

Then R_ME has a high response if the node has saturated responses -1’s or 1``s, as one desire such saturated responses, a should be negative.

The R_AC, penalizes correlation between responses of different nodes.

The point is, 

a) The second term will introduce  low correlation in saturated vectors, then the will be informative. 
 
b) why the authors use the softsign instead the tanh:  $tahnh \in C^2 $! Meanwhile the derivative id softsign is discontinuous.

c)  It is not clear is the softsign is used besides the activation function: In page 5 is said “R_BRE can be applied on ant rectified layer before the nolinearity” . This seems tt the authors propose to add a second activation function (the softsign), why not use the one is in teh layer?

d) The authors found hard to regularize the gradient $\nabla_x D(x)$, even they tray tanh and cosine based activations. It seems that effectively, the  introduce their additional softsign in the process.

e) En the definition of R_AC, I denoted by <k,l> the pair of nodes (k \ne l). However, I think that it should be for pair in the same layer. It is not clear in the paper.

f) It is supposed that the L_1 regularization motes the weights to be informative, this work is doing something similar. How is it compared  the L_1 regularization vs. the proposal?

Recommendation
I tried to read the paper several times and I accept that it was very hard to me. The most difficult part is the lack of precision on the maths, it is hard to figure out what the authors contribution indeed are. I think there is some merit in the work. However, it is not very well organized and many points are not defined. In my opinion, the paper is in a preliminary stage and should be refined. I recommend a “SOFT” REJECT

review_title: Improving GANs by promoting more informative weights in hidden layers

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a regularizer that encourages a GAN discriminator to focus its capacity in the region around the manifolds of real and generated data points, even when it would be easy to discriminate between these manifolds using only a fraction of its capacity, so that the discriminator provides a more informative signal to the generator. The regularizer rewards high entropy in the signs of discriminator activations. Experiments show that this helps to prevent mode collapse on synthetic Gaussian mixture data and improves Inception scores on CIFAR10. 

The high-level idea of guiding model capacity by rewarding high-entropy activations  is interesting and novel to my knowledge (though I am not an expert in this space). Figure `1 is a fantastic illustration that presents the core idea very clearly. That said I found the intuitive story a little bit difficult to follow -- it's true that in Figure 1b the discriminator won't communicate the detailed structure of the data manifold to the generator, but it's not clear why this would be a problem -- the gradients should still pull the generator *towards* the manifold of real data, and as this happens and the manifolds begin to overlap, the discriminator will naturally be forced to allocate its capacity towards finer-grained details. Is the implicit assumption that for real, high-dimensional data the generator and data manifolds will *never* overlap? But in that case much of the theoretical story goes out the window. I'd also appreciate further discussion of the relationship of this approach to Wasserstein GANs, which also attempt to provide a clearer training gradient when the data and generator manifolds do not overlap.

More generally I'd like to better understand what effect we'd expect this regularizer to have. It appears to be motivated by improving training dynamics, which is understandably a significant concern. Does it also change the location of the Nash equilibria? (or equivalently, the optimal generator under the density-ratio-estimator interpretation of discriminators proposed by https://arxiv.org/abs/1610.03483). I'd expect that it would but the effects of this changed objective are not discussed in the paper. 

 The experimental results seem promising, although not earthshattering. I would have appreciated a comparison to other methods for guiding discriminator representation capacity, e.g. autoencoding (I'd also imagine that learning an inference network (e.g. BiGAN) might serve as a useful auxiliary task?). 

Overall this feels like an cute hack, supported by plausible intuition but without deep theory or compelling results on real tasks (yet). As such I'd rate it as borderline; though perhaps interesting enough to be worth presenting and discussing.

A final note: this paper was difficult to read due to many grammatical errors and unclear or misleading constructions, as well as missing citations (e.g. sec 2.1). From the second paragraph alone:
"impede their wider applications in new data domain" -> domains
"extreme collapse and heavily oscillation" -> heavy oscillation
"modes of real data distribution" -> modes of the real data distribution
"while D fails to exploit the failure to provide better training signal to G" -> should be "this failure" to refer to the previously-described generator mode collapse, or rewrite entirely
"even when they are their Jensen-Shannon divergence" -> even when their Jensen-Shannon divergence
 I'm sympathetic to the authors who are presumably non-native English speakers; many good papers contain mistakes, but in my opinion the level in this paper goes beyond what is appropriate for published work. I encourage the authors to have the work proofread by a native speaker; clearer writing will ultimately increase the reach and impact of the paper.  

review_title: a cute hack with unclear impact

final_score: rating: 7: Good paper, accept
review_content: The paper proposed a novel regularizer that is to be applied to the (rectifier) discriminators in GAN in order to encourage a better allocation of the "model capacity" of the discriminators over the (potentially multi-modal) generated / real data points, which might in turn helps with learning a more faithful generator.

The paper is in general very well written, with intuitions and technical details well explained and empirical studies carefully designed and executed.

Some detailed comments / questions:

1. It seems the concept of "binarized activation patterns", which the proposed regularizer is designed upon, is closely coupled with rectifier nets. I would therefore suggest the authors to highlight this assumption / constraint more clearly e.g. in the abstract.

2. In order for the paper to be more self-contained, maybe list at least once the formula for "rectifier net" (sth. like "a^T max(0, wx + b) + c") ? This might also help the readers better understand where the polytopes in Figure 1 come from.

3. In section 3.1, when presenting random variables (U_1, ..., U_d), I find the word "Bernourlli" a bit misleading because typically people would expect U_i to take values from {0, 1} whereas here you assume {-1, +1}. This can be made clear with just one sentence yet would greatly help with clearing away confusions for subsequent derivations.
Also, "K" is already used to denote the mini-batch size, so it's a slight abuse to reuse "k" to denote the "kth marginal".

4. In section 3.2, it may be clearer to explicitly point out the use of the "3-sigma" rule for Gaussian distributions here. But I don't find it justified anywhere why "leave 99.7% of i, j pairs unpenalized" is sth. to be sought for here?

5. In section 3.3, when presenting Corollary 3.3 of Gavinsky & Pudlak (2015), "n" abruptly appears without proper introduction / context.

6. For the empirical study with 2D MoG, would an imbalanced mixture make it harder for the BRE-regularized GAN to escape from modal collapse?

7. Figure 3 is missing the sub-labels (a), (b), (c), (d).
review_title: An interesting novel regularizer for rectifier discriminators in GAN

Meta-review:   + Original regularizer that encourages discriminator representation entropy is shown to improve GAN training.
  + good supporting empirical validation
  - While intuitively reasonable, no compelling theory is given to justify the approach
  - The regularizer used in practice is a heap of heuristic approximations (continuous relaxation of a rough approximate measure of the joint entropy of a binarized activation vector)
  - The writing and the mathematical exposition could be clearer and more precise
Decision: Accept (Poster)