Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

The paper presents an information theoretic regularizer for deep learning
algorithms. The regularizer aims to enforce compression of the learned
representation while conditioning upon the class label so preventing the
learned code from being constant across classes. The presentation of the Z
latent variable used to simplify the calculation of the entropy H(Y|C) is 
confusing and needs revision, but otherwise the paper is interesting.

Major Comments:

- The statement that I(X;Y) = I(C;Y) + H(Y|C) relies upon several properties
  of Y which are not apparent in the text (namely that Y is a function of X,
so I(X;Y) should be maximal, and Y is a smaller code space than X so it should 
be H(Y)). If Y is a larger code space than X then it should still be true, but
the logic is more complicated.

- The latent code for Z is unclear. Given the use of ReLUs it seems like Y
  will be O or +ve, and Z will be 0 when Y is 0 and 1 otherwise, so I'm
unclear as to when the value H(Y|Z) will be non-zero. The data is then
partitioned within a batch based on this Z value, and monte carlo sampling is
used to estimate the variance of Y conditioned on Z, but it's really unclear
as to how this behaves as a regularizer, how the z is sampled for each monte
carlo run, and how this influences the gradient. The discussion in Appendix C
doesn't mention how the Z values are generated.

- The discussion on how this method differs from the information bottleneck is
  odd, as the bottleneck is usually minimising the encoding mutual information
I(X;Y) minus the decoding mutual information I(Y;C). So directly minimising
H(Y|C) is similar to the IB, and also minimising H(Y|C) will affect I(C;Y) as
I(C;Y) = H(Y) - H(Y|C).

- The fine tuning experiments (Section 4.2) contain no details on the
  parameters of that tuning (e.g. gradient optimiser, number of epochs,
batch size, learning rates etc).

- Section 4.4 is obvious, and I'd consider it a bug if regularising with label
  information performed worse than regularising without label information.
Essentially it's still adding supervision after you've removed the
classification loss, so it's natural that it would perform better. This
experiment could be moved to the appendix without hurting the paper.

- In appendix A an upper bound is given for the reconstruction error in terms
  of the conditional entropy. This bound should be related to one of the many
upper bounds (e.g. Hellman & Raviv) for the Bayes rate of a predictor, as
there is a fairly wide literature in this area.

Minor Comments:

- The authors do not state what kind of input variations they are trying to
  make the model invariant to, and as it applies to CNNs there are multiple
different kinds, many of which are not amenable to a regularization based
system for inducing invariance.

- The authors should remind the reader once that I(X;Y) = H(Y) - H(Y|X) = H(X) -
  H(X|Y), as this fact is used multiple times throughout the paper, and it may
not necessarily be known by readers in the deep learning community.

- Computing H(Y|C) does not necessarily require computing c separate
  entropies, there are multiple different approaches for computing this
entropy.

- The exposition in section 3 could be improved by saying that H(X|Y) measures
  how much the representation compresses the input, with high values meaning
large amounts of compression, as much of X is thrown away when generating Y.

- The figures are difficult to read when printed in grayscale, the graphs
  should be made more readable when printed this way (e.g. different symbols,
dashed lines etc).

- There are several typos (e.g. pg 5 "staking" -> "stacking").

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a particular variance regularizer on activations and connect it to the conditional entropy of the activation given the class label. They also present some competitive results on CIFAR-10 and ImageNet.

Despite some promising results, I found some issues with the paper. The main one is that the connection between conditional entropy and the proposed variance regularizer seems tenuous. The chain of reasoning is as follows:

- Estimation of H(Y|C) is difficult for two reasons: 1) when the number of classes is large, the number of samples needed calculate the entropy are high, and 2) naive estimators -- even when the number of classes are small -- have high variance. To solve these issues, the authors propose:

a) Introduce a latent code Z such that H(Y|C) = H(Y|Z). This solves problem 1).

b) Use a variance upper bound on H(Y|Z). This solves problem 2).

My issue is with the reasoning behind a). H(Y|C) = H(Y|Z) relies on the assumption that I(Y;C) = I(Y;Z). The authors present a plausibility argument, but the argument was not sufficiently convincing to me to overcome my prior that I(Y;C) =/= I(Y;Z).

Apart from this, I found some other issues. 

* In the second paragraph of 2.2, the acronym LME in "LME estimator" was not defined, so I checked the reference provided. That paper did not mention a LME estimator, but did present a "maximum likelihood estimator" with the same convergence properties as those mentioned in the SHADE paper. Since the acronym LME was used twice, I'm assuming this was not a typo. Perhaps this is a bug in the reference?

* In section 4.4, it's hard to know if the curves actually show that "SHADE produces less class-information filtering". The curves are close throughout and are nearly identical at epoch 70. It is entirely possible that the difference in curves is due to optimization or some other lurking factor.

* The final form of the regularization makes it look like a more principled alternative to batchnorm. It would have been nice if the authors more directly compared SHADE to BN.

* There are two upper bounds here: that H(Y_l | C) <= \sum_i H(Y_{l, i} | C), and the variance upper bound. The first one does not seem particularly tight, especially at the early layers where the representation is overcomplete. I understand that the authors argue that the upper bound is tight in footnote 3, but it is only plausible for later laters.

My Occam's razor explanation of this paper is that it forces pre-nonlinearity activations (and hence post-nonlinearity activations) to be binary, without having to resort to sigmoid or tanh nonlinearities. This is a nice property, but whether the regularizer connects to H(Y|C) still remains unsolved.
review_title: Some nice ideas but also some tenuous connections

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes another entropic regularization term for deep neural nets. The key idea can be stated as follows: Let X denote the observed input, C the hidden class label taking values in a finite set, and Y the representation computed by a neural net. Then C -> X -> Y is a Markov chain. Moreover, assuming that the mapping X -> Y is deterministic (as is the case with neural nets or any other deterministic representations), we can write down the mutual information between X and Y as

I(X;Y) = H(Y) - H(Y|X) = H(Y).

A simple manipulation shows that H(Y) = I(C;Y) + H(Y|C). The authors interpret the first term, I(C;Y), as a data fit term that quantifies the statistical correlations between the class label C and the representation Y, whereas the second term, H(Y|C), is the amount by which the representation Y can be compressed knowing the class label C. The authors then propose to 'explicitly decouple' the data-fit term I(C;Y) from the regularization penalty and focus on minimizing H(Y|C). In fact, they replace this term by the sum of conditional entropies of the form H(Y_{i,k}|C), where Y_{i,k} is the activation of the ith neuron in the kth layer of the neural net. The final step is to recognize that the conditional entropy may not admit a scalable and differentiable estimator, so they use the relation between a quantity called entropy power and second moments to replace the entropic penalty with the conditional variance penalty Var[Y_{i,k}|C]. Since the class-conditional distributions are unknown, a surrogate model Q_{Y|C} is used. The authors present some experimental results as well.

However, this approach has a number of serious flaws. First of all, if the distribution of X is nonatomic and the mapping X -> Y is continuous (in the case of neural nets, it is even Lipschitz), then the mutual information I(X;Y) is infinite. In that case, the representation of I(X;Y) in terms of entropies is not valid -- indeed, one can write the mutual information between two jointly distributed random variables X and Y in terms of differential entropies as I(X;Y) = h(Y) - h(Y|X), but this is possible only if both terms on the right-hand side exist. This is not the case here, so, in particular, one cannot relate I(X;Y) to I(C;Y). Ironically, I(C;Y) is finite, because C takes values in a finite set, so I(C;Y) is at most the log cardinality of the set of labels. One can start, then, simply with I(C;Y) and express it as H(C) - H(C|Y). Both terms are well-defined Shannon entropies, where the first one does not depend on the representation, whereas the second one involves the representation. But then, if the goal is to _minimize_ the mutual information between I(C;Y), it makes sense to _maximize_ the conditional entropy H(C|Y). In short, the line of reasoning that leads to minimizing H(Y|C) is not convincing. Moreover, why is it a good idea to _minimize_ I(C;Y) in the first place? Shouldn't one aim to maximize it subject to structural constraints on the representation, along the lines of InfoMax?

The next issue is the chain of reasoning that leads to replacing H(Y|C) with Var[Y|C]. One could start with that instead without changing the essence of the approach, but then the magic words "Shannon decay" would have to disappear altogether, and the proposed method would lose all of its appeal.
review_title: an obvious idea supported by flawed reasoning

final_score: rating: 7: Good paper, accept
review_content: the paper adapts the information bottleneck method where a problem has invariance in its structure. specifically, the constraint on the mutual information is changes to one on the conditional  entropy. the paper involves a technical discription how to develop proper estimators for this conditional entropy etc.

this is a nice and intuitive idea. how it interacts with classical regularizers or if it completely dominates classical regularizers would be interesting for the readers.
review_title: nice and intuitive idea

Meta-review: The proposed conditional variance regularizer looks interesting and the results show some promise. However, as the reviewers pointed out, the connection between the information-theoretic argument provided and the final form of the regularizer is too tenuous in its current form. Since this argument is central to the paper, the authors are urged to either provide a more rigorous derivation or motivate the regularizer more directly and place more emphasis on its empirical evaluation.
Decision: Reject