Title: Deep Variational Information Bottleneck
Authors: Alexander A. Alemi
Ian Fischer
Joshua V. Dillon
Kevin Murphy
Venue: ICLR 2017
Keywords: Theory
Computer vision
Deep learning
Supervised Learning

Abstract: We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method “Deep Variational Information Bottleneck”, or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

TL;DR: Applying the information bottleneck to deep networks using the variational lower bound and reparameterization trick.

Review discussion:
Thank you for an interesting read.
Could you clarify how exactly did you generate the adversarial examples for each method you tested, especially for Deep VIB?
Nicholas Carlini shared his implementation of the attacks in his recent paper https://arxiv.org/abs/1608.04644 with us.  We used the code for his L2 attack directly, including using the same parameters as he used for the the results in his paper (which we confirmed by replicating his results on the same network he used in his paper -- he also shared the trained checkpoint for that network).
His L2 attack uses the Adam optimizer to find perturbations to an input image that cause the classifier to change its result on that input image to a different class.  This attack works in both untargeted and targeted variants.  I.e., you can have the optimizer generate perturbations that change the classification to another arbitrary class, or you can specify that the perturbation must change the classification to a particular class.  The loss that is optimized combines an L2 measure between the original image and the perturbed image, and a second loss computed from the original classification (or target classification, if it’s a targeted attack) and the output classification.  In order to constrain the adversarial example to be a valid image (where all of the pixels are in a constrained range), the attack also passes the perturbed image through a tanh function before doing the classification.
Since the Deep VIB model is a classifier, this  adversarial attack (and most others) can be applied directly.  We did not need to change anything to run the attack on the different networks, so all of the networks mentioned in the paper were attacked in exactly the same way, including the same attack parameters.
How was beta in Table 1 chosen? If it was chosen based on the test set, the performance may be too optimistic. Since the training set cannot be used to choose beta (one would end up with beta=0), cross-validation has to be used. This step seems important since the optimal beta depends on the size of the bottleneck and the number of samples, so that using a fixed beta would be sub-optimal.
The beta in Table 1 was chosen as the optimal from the test set.  The regularization experiments were not meant to establish a new state of the art, just to demonstrate that VIB can be competitive.  You are correct: in general, the best beta would have to be chosen by doing cross validation. For these experiments, it seems there is a wide range of beta values that do better than no beta whatsoever, so initial indications are that it is not a very hard hyperparameter to tune decently.
Thank you for an interesting read. I personally like the information bottleneck principle and am very happy to see its application to deep neural networks. To my knowledge, this is the first paper that applies IB to train deep networks (the original papers only presented the concept), but see below for the note of independent work claim.
The derivation of the variational lowerbound is very clear, even for those who are not very familiar with variational inference.
Also the explanation of the IB principle is clear. Experimental results seem to be very promising.
I found the presentation for the model a bit confusing. In variational inference/information maximisation, p usually denotes the model and q represents the "inference engine". This means the choice of inference method is independent to the modelling procedure.
However the presented VIB assumed p(x, y) as the **underlying data distribution** (and approximated by the empirical distribution), thus here the model is actually q(y|z)p(z|x).
Then the authors presented p(y|x) as the **predictive distribution** in page 8, paragraph 2 of section 4.2.3. Predictive in what sense? I guess you meant p(y|x) = \int q(y|z) p(z|x) dz in this case, but this makes the two definitions contradict to each other!
The authors have made an interesting connection to variational auto-encoder and the warm-up training (by tuning beta).
However, even when the loss function formula is the same to the variational lowerbound used in VAE (in this case beta = 1), the underlying model is different!
For example, r(z) in VIB is the variational approximation to p(z) (which means r(z) is not a component in the model), while in VAE it is the prior distribution which is actually defined in the modelling procedure. Similaly p(z|x) in VIB is included in the model, while in VAE that is the approximate posterior and can be independently chosen (e.g. you can use p(x|z) as a deep NN but p(z|x) as a deep NN or a Gaussian process).
In summary, I think the presentation for the modelling procedure is unclear. I hope these point would be made clearer in revision since the current presentation makes me uncomfortable as a Bayesian person. In the VAE part, it's better to clearly mention the difference between VIB and VAE, and provide some intuitions if the VIB interpretation is preferred.
Typos:
Eq. 9-11: did you mean q(y|z) instead of q(z|y)?
Fig 2 "as beta becomes smaller": did you mean "larger"?
**claim for independent work**
The authors claimed that the manuscript presented an independent work to Chalk et al. 2016 which is online since May 2016. It seems to me that nowadays deep learning research is very competitve that many people publish the same idea at the same time.
So I would trust this claim and commend the authors' honesty, but in case this is not true, I would not recommend the manuscript for acceptance.
Thank you for your feedback and apologies for the notational headaches. We added some clarifying remarks regarding p(y|x) (which is, as you pointed out "p(y|x) = \int q(y|z) p(z|x) dz").  We're approximating this distribution in two ways: by averaging over samples, z~p(.|x) and of course using q as a surrogate for p(y|z)=\int dx p(y,x|z).
Unlike standard applications of variational inference our surrogate q(y|z) replaces a marginal (not a posterior).
However, the motivation behind both approaches is identical: the posterior is computationally challenging due to its normalization term, i.e., the computation of some marginal. Hopefully this clears up how our inference engine differs from typical regimes.
Regarding your comment about VAE and r(z): we were making the case that the VIB formalism subsumes the VAE loss function. You are correct that the underlying (assumed) model is different: the VIB makes assumptions about the latent "bottleneck" space which are not made in a VAE. The connection is made only under the variational loss, which has a degree of freedom permitting recovery of the VAE loss when choosing r(z) appropriately.
Thank you for catching the typos as well -- we've fixed those in this draft.
We have added some preliminary experiments based on a SOTA Imagenet classifier, namely Inception ResNet V2.
In particular, we took a pre-trained network, removed the final softmax layer, and used the resulting 1536 dimensional feature vector as the input to our experiments.
Using these features, it is possible to classify with 80.4% accuracy using only logistic regression, which is the same performance as the original network.
We then performed the following experiment. The input was passed through two fully connected layers with 1024 units each, followed by a stochastic layer encoding a diagonal 1024 dimensional Gaussian representation with learned means and standard deviations, which was then passed to our variational classifier, which again for simplicity we took to be simple logistic regression.
Although we were not able to improve classification accuracy, we were able to achieve essentially the same performance (80.12%) using β of 0.01; this corresponds to a very "narrow" (in the information-theoretic sense) bottleneck: we estimate  the mutual information as only I(X,Z) ∼ 45 bits. (Contrast this to the estimate we observe in the β = 0 case, where the capacity is about 10,000 bits, but with an accuracy of only 78.87%.)
More interestingly, we show that this narrow bottleneck is substantially more robust to adversarial attacks than the original deterministic model.
In particular, the VIB network requires much larger perturbations in order to fool the classifier.
See Section 4.2.5 for further details on our new Imagenet experiments.
Summary:
The paper “Deep Variational Information Bottleneck” explores the optimization of neural networks for variational approximations of the information bottleneck (IB; Tishby et al., 1999). On the example of MNIST, the authors show that this may be used for regularization or to improve robustness against adversarial attacks.
Review:
The IB is potentially very useful for important applications (regularization, adversarial robustness, and privacy are mentioned in the paper). Combining the IB with recent advances in deep learning to make it more widely applicable is an excellent idea.
But given that the theoretical contribution is a fairly straight-forward application of well-known ideas, I would have liked to see a stronger experimental section.
Since the proposed approach allows us to scale IB, a better demonstration of this would have been on a larger problem than MNIST. It is also not clear whether the proposed approach will still work well to regularize more interesting networks with many layers.
Why is dropout not included in the quantitative comparison of robustness to adversarial examples (Figure 4)?
How was the number of samples (12) chosen?
What are the error bars in Figure 1 (a)?
On page 7 the authors claim “the posterior covariance becomes larger” as beta “decreases” (increases?). Is this really the case? It’s hard to judge based on Figure 1, since the figures are differently scaled.
It might be worth comparing to variational fair autoencoders (Louizos et al., 2016), which also try to learn representations minimizing the information shared with an aspect of the input.
The paper is well written and easy to follow.
Thank you for your feedback. We've added some additional experiments using fast gradient sign adversarial examples and included dropout in Figures 4 and 5. We've also added a brief comment on how we chose 12 samples (answer: it was a somewhat arbitrary choice which we did not tune yet apparently worked well). The error bars in Figure 1 (a) are one +- std. dev.
Regarding “the posterior covariance becomes larger,” we corrected this typo and clarified the caption.
The variational fair autoencoder is an interesting connection, and we have added a brief comparison in Section 2. Thanks for bringing it to our attention!
We have added some preliminary experiments based on a SOTA Imagenet classifier, namely Inception ResNet V2.
In particular, we took a pre-trained network, removed the final softmax layer, and used the resulting 1536 dimensional feature vector as the input to our experiments.
Using these features, it is possible to classify with 80.4% accuracy using only logistic regression, which is the same performance as the original network.
We then performed the following experiment. The input was passed through two fully connected layers with 1024 units each, followed by a stochastic layer encoding a diagonal 1024 dimensional Gaussian representation with learned means and standard deviations, which was then passed to our variational classifier, which again for simplicity we took to be simple logistic regression.
Although we were not able to improve classification accuracy, we were able to achieve essentially the same performance (80.12%) using β of 0.01; this corresponds to a very "narrow" (in the information-theoretic sense) bottleneck: we estimate  the mutual information as only I(X,Z) ∼ 45 bits. (Contrast this to the estimate we observe in the β = 0 case, where the capacity is about 10,000 bits, but with an accuracy of only 78.87%.)
More interestingly, we show that this narrow bottleneck is substantially more robust to adversarial attacks than the original deterministic model.
In particular, the VIB network requires much larger perturbations in order to fool the classifier.
See Section 4.2.5 for further details on our new Imagenet experiments.
Update: raised the score, because I think the arguments about adversarial examples are compelling.  I think that the paper convincingly proves that this method acts as a decent regularizer, but I'm not convinced that it's a competitive regularizer.
For example, I don't believe that there is sufficient evidence that it gives a better regularizer than dropout/normalization/etc.  I also think that it will be much harder to tune than these other methods (discussed in my rebuttal reply).
Summary: If I understand correctly, this paper proposes to take the "bottleneck" term from variational autoencoders which pulls the latent variable towards a noise prior (like N(0,1)) and apply it in a supervised learning context where the reconstruction term log(p(x|z)) is replaced with the usual supervised cross-entropy objective.
The argument is that this is an effective regularizer and increases robustness to adversarial attacks.
Pros:
-The presentation is quite good and the paper is easy to follow.
-The idea is reasonable and the relationship to previous work is well described.
-The robustness to adversarial examples experiment seems convincing, though I'm not an expert in this area.  Is there any way to compare to an external quantitative baseline on robustness to adversarial examples?  This would help a lot, since I'm not sure how the method here compares with other regularizers in terms of combatting adversarial examples.
For example, if one uses a very high dropout rate, does this confer a comparable robustness to adversarial examples (perhaps at the expense of accuracy)?
Cons:
-MNIST accuracy results don't seem very strong, unless I'm missing something.  The Maxout paper from ICML 2013 listed many permutation invariant MNIST results with error rates below 1%.
So the 1.13% error rate listed here doesn't necessarily prove that the method is a competitive regularizer.  I also suspect that tuning this method to make it work well is harder than other regularizers like dropout.
-There are many distinct architectural choices with this method, particularly in how many hidden layers come before and after z.
For example, the output could directly follow z, or there could be several layers between z and the output.  As far as I can tell the paper says that p(y | z) is a simple logistic regression (i.e. one weight matrix followed by softmax), but it's not obvious why this choice was made.  Did it work best empirically?
Other:
-I wonder what would happen if you "trained against" the discovered adversarial examples while also using the method from this paper.  Would it learn to have a higher variance p(z | x) when presented with an adversarial example?
Thank you for your feedback, particularly with respect to MNIST experiments and architectural choices.  We've revised the text to better reflect what we believe is our primary value-add.  To us the most impressive aspect of the VIB performance on MNIST was not that we exceeded state-of-the-art, but rather how close we got with a relatively simple model.
Furthermore, the VIB framework permits spanning a continuum of model complexities within a given architecture, and in a principled manner.
We did not explore additional model architectures, viz., adding/removing layers before/after z. In a future submission we hope to explore different structures. Along these lines--and perhaps even more exciting--is the prospect of studying applications of the VIB principle to every layer of the net (not just the penultimate).
That said, the goal of this paper was to show that despite p(y|z) being a simple logistic regression--and making the simplest architecture choices we could--we were able to get good results.
Additionally we added several changes to the adversarial section and accompanying figures. Hopefully this improves clarity while showing some additional experiments with the fast gradient sign method of generating adversarial examples.
Your idea of training against the discovered adversarial examples is a good idea -- Goodfellow et al. 2014 demonstrated that doing such training can also improve robustness to adversarial examples, and it’s likely to improve things in the VIB model as well.  We don’t include adversarial training in this work, since we are trying to focus on the question of whether the information bottleneck on its own improves robustness relative to reasonable baselines, but it is certainly something we would want to do when trying to train a maximally robust model.
"To us the most impressive aspect of the VIB performance on MNIST was not that we exceeded state-of-the-art, but rather how close we got with a relatively simple model"
First, why not try to show an improvement on a model that's near the state-of-the-art?
For example, you could fairly easily apply this to the end of the PI-MNIST NN from the maxout paper and see if it improves upon that result.  I'm not sure that there's any point in trying to get a SOTA on PI-MNIST, but it would be nice to see a result on top of a model that's closer to the SOTA.
In terms of the model being relatively simple: I'm somewhat open to the idea of accepting an empirical comparison on a restricted set of models (for example disallowing convolutions or data augmentation) but it needs to be very clear how the models in the comparison are all constrained.  What was the criteria for deciding what models were to be included in Table 1?  The caption implies that it's having a "bottleneck of k=256" but I'm not sure exactly what this means.  All of the results reference Pereya 2016.  Are there no other papers with MNIST results that ought to be included here?
"Furthermore, the VIB framework permits spanning a continuum of model complexities within a given architecture, and in a principled manner."
Can you clarify what you mean here?  It seems like tuning the beta parameter (figure 1a,1b) isn't too different from tuning a dropout rate.  Is your point that one can compute I(x,z) and quantify how much information from x is kept in z?
"We did not explore additional model architectures, viz., adding/removing layers before/after z. In a future submission we hope to explore different structures. Along these lines--and perhaps even more exciting--is the prospect of studying applications of the VIB principle to every layer of the net (not just the penultimate).
That said, the goal of this paper was to show that despite p(y|z) being a simple logistic regression--and making the simplest architecture choices we could--we were able to get good results."
My guess is that if you use this in early layers in a deep network, the model will immediately set mu=0 and sigma=1 for those layers because the gradient from the information bottleneck is less noisy than the gradient from the following layers in the just-initialized deep network.  It may be very hard to find a beta parameter that avoids this.
Thanks for your comments, which we respond to below.
"First, why not try to show an improvement on a model that's near the state-of-the-art?"
We have added some preliminary experiments based on a SOTA Imagenet classifier, namely Inception ResNet V2.
In particular, we took a pre-trained network, removed the final softmax layer, and used the resulting 1536 dimensional feature vector as the input to our experiments.
Using these features, it is possible to classify with 80.4% accuracy using only logistic regression, which is the same performance as the original network.
We then performed the following experiment. The input was passed through two fully connected layers with 1024 units each, followed by a stochastic layer encoding a diagonal 1024 dimensional Gaussian representation with learned means and standard deviations, which was then passed to our variational classifier, which again for simplicity we took to be simple logistic regression.
Although we were not able to improve classification accuracy, we were able to achieve essentially the same performance (80.12%) using β of 0.01; this corresponds to a very "narrow" (in the information-theoretic sense) bottleneck: we estimate  the mutual information as only I(X,Z) ∼ 45 bits. (Contrast this to the estimate we observe in the β = 0 case, where the capacity is about 10,000 bits, but with an accuracy of only 78.87%.)
More interestingly, we show that this narrow bottleneck is substantially more robust to adversarial attacks than the original deterministic model.
In particular, the VIB network requires much larger perturbations in order to fool the classifier.
See Section 4.2.5 for further details on our new Imagenet experiments.
"I'm not sure that there's any point in trying to get a SOTA on PI-MNIST, but it would be nice to see a result on top of a model that's closer to the SOTA".
We deliberately chose to pick a simple baseline that is fairly close to SOTA on MNIST, but which still left some "headroom" in order to show the benefits of our method.
"What was the criteria for deciding what models were to be included in Table 1?"
We chose the architecture from the  Pereya 2016 paper (also in submission to ICLR'17) simply because they are our colleagues, and that way, we could be more confident we were doing a fair comparison.
"You could fairly easily apply this to the end of the PI-MNIST NN from the maxout paper and see if it improves upon that result.  "
Comparing against the permutation invariant maxout network would be difficult, because it introduces another source of stochasticity beyond our own (namely in the dropout that is required with maxout activations).  This would make it hard to discern the source of benefit: is improvement a consequence of our stochasticity or theirs?
"The caption implies that it's having a "bottleneck of k=256" but I'm not sure exactly what this means".
You are right, this is not well explained. We have clarified it in the text.
The Pereya  architecture is 784 - 1024 - 1024 - 10.
In our case all (MNIST) networks were 784 - 1024 - 1024 - K - 10, where K is the size of the latent bottleneck.
"It seems like tuning the beta parameter (figure 1a,1b) isn't too different from tuning a dropout rate.  Is your point that one can compute I(x,z) and quantify how much information from x is kept in z?"
That's exactly right.  By adjusting beta one can target different amounts of information allowed to pass through the network (with respect to input data, X).  By tuning beta, one dials up or down the "fidelity" between the compressed representation Z and the ground-truth evidence X.  Through this mechanism we explore how different architectures respond to varying levels of "fidelity", i.e. strength of the bottleneck.
"My guess is that if you use this in early layers in a deep network, the model will immediately set mu=0 and sigma=1 for those layers because the gradient from the information bottleneck is less noisy than the gradient from the following layers in the just-initialized deep network.  It may be very hard to find a beta parameter that avoids this. "
This is a valid concern. Unfortunately we have not had time to try this experiment (we have instead focused our energies on scaling up the adversarial experiments to Imagenet, as mentioned above). We leave exploration of this issue to future work.

Meta-review:
role: PC
data: comment: This paper discussses applying an information bottleneck to deep networks using a variational lower bound and reparameterization trick. The paper is well written and the examples are compelling. The paper can be improved with more convincing results on MNIST.

Decision: Accept (Poster)