Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: How was beta in Table 1 chosen? If it was chosen based on the test set, the performance may be too optimistic. Since the training set cannot be used to choose beta (one would end up with beta=0), cross-validation has to be used. This step seems important since the optimal beta depends on the size of the bottleneck and the number of samples, so that using a fixed beta would be sub-optimal.
review_title: Optimal beta

final_score: rating: 6: Marginally above acceptance threshold
review_content: Thank you for an interesting read. I personally like the information bottleneck principle and am very happy to see its application to deep neural networks. To my knowledge, this is the first paper that applies IB to train deep networks (the original papers only presented the concept), but see below for the note of independent work claim. 

The derivation of the variational lowerbound is very clear, even for those who are not very familiar with variational inference. Also the explanation of the IB principle is clear. Experimental results seem to be very promising.

I found the presentation for the model a bit confusing. In variational inference/information maximisation, p usually denotes the model and q represents the "inference engine". This means the choice of inference method is independent to the modelling procedure. However the presented VIB assumed p(x, y) as the **underlying data distribution** (and approximated by the empirical distribution), thus here the model is actually q(y|z)p(z|x). Then the authors presented p(y|x) as the **predictive distribution** in page 8, paragraph 2 of section 4.2.3. Predictive in what sense? I guess you meant p(y|x) = \int q(y|z) p(z|x) dz in this case, but this makes the two definitions contradict to each other!

The authors have made an interesting connection to variational auto-encoder and the warm-up training (by tuning beta). However, even when the loss function formula is the same to the variational lowerbound used in VAE (in this case beta = 1), the underlying model is different! For example, r(z) in VIB is the variational approximation to p(z) (which means r(z) is not a component in the model), while in VAE it is the prior distribution which is actually defined in the modelling procedure. Similaly p(z|x) in VIB is included in the model, while in VAE that is the approximate posterior and can be independently chosen (e.g. you can use p(x|z) as a deep NN but p(z|x) as a deep NN or a Gaussian process).

In summary, I think the presentation for the modelling procedure is unclear. I hope these point would be made clearer in revision since the current presentation makes me uncomfortable as a Bayesian person. In the VAE part, it's better to clearly mention the difference between VIB and VAE, and provide some intuitions if the VIB interpretation is preferred.


Typos:
Eq. 9-11: did you mean q(y|z) instead of q(z|y)?
Fig 2 "as beta becomes smaller": did you mean "larger"?

**claim for independent work**
The authors claimed that the manuscript presented an independent work to Chalk et al. 2016 which is online since May 2016. It seems to me that nowadays deep learning research is very competitve that many people publish the same idea at the same time. So I would trust this claim and commend the authors' honesty, but in case this is not true, I would not recommend the manuscript for acceptance.
review_title: A paper with interesting methods, but the presentation is a bit confusing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper “Deep Variational Information Bottleneck” explores the optimization of neural networks for variational approximations of the information bottleneck (IB; Tishby et al., 1999). On the example of MNIST, the authors show that this may be used for regularization or to improve robustness against adversarial attacks.

Review:
The IB is potentially very useful for important applications (regularization, adversarial robustness, and privacy are mentioned in the paper). Combining the IB with recent advances in deep learning to make it more widely applicable is an excellent idea. But given that the theoretical contribution is a fairly straight-forward application of well-known ideas, I would have liked to see a stronger experimental section.

Since the proposed approach allows us to scale IB, a better demonstration of this would have been on a larger problem than MNIST. It is also not clear whether the proposed approach will still work well to regularize more interesting networks with many layers.

Why is dropout not included in the quantitative comparison of robustness to adversarial examples (Figure 4)?

How was the number of samples (12) chosen?

What are the error bars in Figure 1 (a)?

On page 7 the authors claim “the posterior covariance becomes larger” as beta “decreases” (increases?). Is this really the case? It’s hard to judge based on Figure 1, since the figures are differently scaled.

It might be worth comparing to variational fair autoencoders (Louizos et al., 2016), which also try to learn representations minimizing the information shared with an aspect of the input.

The paper is well written and easy to follow.
review_title: Great idea, lacking empirical section

final_score: rating: 7: Good paper, accept
review_content: Update: raised the score, because I think the arguments about adversarial examples are compelling.  I think that the paper convincingly proves that this method acts as a decent regularizer, but I'm not convinced that it's a competitive regularizer.  For example, I don't believe that there is sufficient evidence that it gives a better regularizer than dropout/normalization/etc.  I also think that it will be much harder to tune than these other methods (discussed in my rebuttal reply).  

----

Summary: If I understand correctly, this paper proposes to take the "bottleneck" term from variational autoencoders which pulls the latent variable towards a noise prior (like N(0,1)) and apply it in a supervised learning context where the reconstruction term log(p(x|z)) is replaced with the usual supervised cross-entropy objective.  

The argument is that this is an effective regularizer and increases robustness to adversarial attacks.  

Pros: 

-The presentation is quite good and the paper is easy to follow.  

-The idea is reasonable and the relationship to previous work is well described.  

-The robustness to adversarial examples experiment seems convincing, though I'm not an expert in this area.  Is there any way to compare to an external quantitative baseline on robustness to adversarial examples?  This would help a lot, since I'm not sure how the method here compares with other regularizers in terms of combatting adversarial examples.  For example, if one uses a very high dropout rate, does this confer a comparable robustness to adversarial examples (perhaps at the expense of accuracy)?  

Cons: 

-MNIST accuracy results don't seem very strong, unless I'm missing something.  The Maxout paper from ICML 2013 listed many permutation invariant MNIST results with error rates below 1%.  So the 1.13% error rate listed here doesn't necessarily prove that the method is a competitive regularizer.  I also suspect that tuning this method to make it work well is harder than other regularizers like dropout.  

-There are many distinct architectural choices with this method, particularly in how many hidden layers come before and after z.  For example, the output could directly follow z, or there could be several layers between z and the output.  As far as I can tell the paper says that p(y | z) is a simple logistic regression (i.e. one weight matrix followed by softmax), but it's not obvious why this choice was made.  Did it work best empirically?  

Other: 

-I wonder what would happen if you "trained against" the discovered adversarial examples while also using the method from this paper.  Would it learn to have a higher variance p(z | x) when presented with an adversarial example?  
review_title: Review

final_score: rating: null
review_content: Thank you for an interesting read.

Could you clarify how exactly did you generate the adversarial examples for each method you tested, especially for Deep VIB?
review_title: Adversarial example

Meta-review: This paper discussses applying an information bottleneck to deep networks using a variational lower bound and reparameterization trick. The paper is well written and the examples are compelling. The paper can be improved with more convincing results on MNIST.
Decision: Accept (Poster)