Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper summary
This paper develops a generalization of dropout using information theoretic
principles. The basic idea is that when learning a representation z of input x
with the aim of predicting y, we must choose a z such that it carries the least
amount of information about x, as long as it can predict y. This idea can be
formalized using the Information Bottleneck Lagrangian. This leads to an
optimization problem which is similar to the one derived for variational
dropout, the difference being that Information dropout allows for a scaling
factor associated with the KL divergence term that encourages noise. The amount
of noise being added is made a parameterized function of the data and this
function is optimized along with the rest of the model. Experimental results on
CIFAR-10 and MNIST show (small) improvements over binary dropout.

Strengths
- The paper highlights an important conceptual link between probabilistic
  variational methods and information theoretic methods, showing that dropout
can be generalized using both formalisms to arrive at very similar models.
- The presentation of the model is excellent.
- The experimental results on cluttered MNIST are impressive.

Weaknesses
- The results on CIFAR-10 in Figure 3(b) seem to be on a validation set (unless
  the axis label is a typo). It is not clear why the test set was not used. This
makes it hard to compare to results reported in Springenberg et al, as well as
other results in literature.

Quality
The theoretical exposition is high quality. Figure 2 gives a nice qualitative
assessment of what the model is doing. However, the experimental results
section can be made better, for example, by matching the results on CIFAR-10 as
reported in Springenberg et al. and trying to improve on those using information
dropout.

Clarity
The paper is well written and easy to follow.

Originality
The derivation of the information dropout optimization problem using IB
Lagrangian is novel. However, the final model is quite close to variational
dropout.

Significance
This paper will be of general interest to researchers in representation learning
because it highlights an alternative way to think about latent variables (as
information bottlenecks). However, unless the model can be shown to achieve
significant improvements over simple dropout, its wider impact is likely to be
limited.

Overall
The paper presents an insightful theoretical derivation and good preliminary
results. The experimental section can be improved.

Minor comments and suggestions -
- expecially -> especially
- trough -> through
- There is probably a minus sign missing in the expression for H(y|z) above Eq (2).
- Figure 3(a) has error bars, but 3(b) doesn't. It might be a good idea to have those
for Figure 3(b) as well.
- Please consider comparing Figure 2 with the activity map of a standard CNN
  trained with binary dropout, so we can see if similar filtering out is
happening there already.
review_title: Insightful theoretical derivation, experiments can be improved.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose "information dropout", a variation of dropout with an information theoretic interpretation. A dropout layer limits the amount of information that can be passed through it, and the authors quantify this using a variational bound. 

It remains unclear why such an information bottleneck is a good idea from a theoretical standpoint. Bayesian interpretations lend a theoretical basis to parameter noise, but activation noise has no such motivation. The information bottleneck indeed limits the information that can be passed through, but there is no rigorous argument for why this should improve generalization.

The experiments are not convincing. The CIFAR-10 results are worse than those in the paper that originally proposed the network architecture they use (Springenberg et al). The VAE results on MNIST are also horrible.
review_title: interesting idea, but not very convincing

final_score: rating: 6: Marginally above acceptance threshold
review_content: An interesting connection is made between dropout, Tishby et al's "information bottleneck" and VAEs. Specifically, classification of 'y' from 'x' is split in two faces: an inference model z ~ q(z|x), a prior p(z), and a classifier y ~ p(y|z). By optimizing the objective E_{(x,y)~data} [ E_{z~q(z|x)}[log p(x|y)] + lambda * KL(q(z|x)||p(z))], with lambda <= 1, an information bottleneck 'z' is formed, where lambda controls an upper bound on the number of bits traveling through 'z'.

The objective is equivalent to a VAE objective with downweighted KL(posterior|prior), an encoder that takes as input 'x', and a decoder that only predicts 'x'.

- Related work (section 2) is discussed sufficiently. 
- In section 3, would be better to remind us the definition of mutual information.
- Connection to VAEs in section 5 is interesting.
- Unfortunately, the MNIST/CIFAR-10 results are not great. Since the method is potentially more flexible than other forms of dropout, this is slightly disappointing.
- It's unclear why the CIFAR-10 results seem to be substantially worse than the results originally reported for that architecture.
- It's unclear which version of 'beta' was used in figure 3a.

Overall I think the theory presented in the paper is promising. However, the paper lacks sufficiently convincing experimental results, and I encourage the authors to do further experiments that prove significant improvements, at least on CIFAR-10, perhaps on larger problems.
review_title: Interesting theory, but experimental results not (yet) very convincing, unfortunately.

final_score: rating: null
review_content: It is very interesting how Information Dropout works almost like an attention mechanism! Paper is very well written, so no major clarification questions.

In the discussion on nuisances having the structure of a group action, could you please give some more intuition and examples about which nuisances can be seen as groups ?

review_title: Nuisances as group actions.

Meta-review: The authors all agree that the theory presented in the paper is of high quality and is promising but the experiments are not compelling. The reviewers are concerned that the presented idea and connections to existing methods, while neat, may not be impactful as the promise of the theory does not bear out in practice. One reviewer is concerned that the presented theory is still not useful, stating that the "information bottleneck thus only becomes meaningful when the capacity of the encoding network is controlled in some measurable way, which is not discussed in the paper". In general, they seem to agree that the experimental evaluation is still preliminary and unfinished. As such, it would seem that the authors could make the paper far more compelling by demonstrating more compelling improvements on benchmark experiments and submitting to a future conference.
Decision: Reject