Unpacking Information Bottlenecks: Surrogate Objec- tives for Deep Learning
Year: 2020

Abstract:
The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models. However, multiple competing objectives are proposed in the literature, and the information-theoretic quantities used in these objectives are difficult to compute for large deep neural networks, which in turn limits their use as a training objective. In this work, we review these quantities, and compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly to optimization without relying on cumbersome tools such as density estimation. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on MNIST, CIFAR-10 and Imagenette with modern DNN architectures (ResNets).

1 Introduction
The Information Bottleneck (IB) principle, introduced by Tishby et al. (2000), proposes that training and generalization in deep neural networks (DNNs) can be explained by information-theoretic principles (Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017; Achille and Soatto, 2018a). This is attractive as the success of DNNs remains largely unexplained by tools from computational learning theory (Zhang et al., 2016; Bengio et al., 2009). The IB principle suggests that learning consists of two competing objectives: maximizing the mutual information between the latent representation and the label to promote accuracy, while at the same time minimizing the mutual information between the latent representation and the input to promote generalization. Following this principle, many variations of IB objectives have been proposed (Alemi et al., 2016; Strouse and Schwab, 2017; Fischer and Alemi, 2020; Fischer, 2020; Fisher, 2019; Gondek and Hofmann, 2003; Achille and Soatto, 2018a), which, in supervised learning, have been demonstrated to benefit robustness to adversarial attacks (Alemi et al., 2016; Fisher, 2019) and generalization and regularization against overfitting to random labels (Fisher, 2019).
Whether the benefits of training with IB objectives are due to the IB principle, or some other unrelated mechanism, remains unclear (Saxe et al., 2019; Amjad and Geiger, 2019; Tschannen et al., 2019), suggesting that although recent work has also tied the principle to successful results in both unsupervised and self-supervised learning (Oord et al., 2018; Belghazi et al., 2018; Zhang et al., 2018; Burgess et al., 2018, among others), our understanding of how IB objectives affect representation learning remains unclear.
Critical to studying this question is the computation of the information-theoretic quantities1 used. While progress has been made in developing mutual information estimators for DNNs (Poole et al., 2019; Belghazi et al., 2018; Noshad et al., 2019; McAllester and Stratos, 2018; Kraskov et al., 2004), current methods still face many limitations when concerned with high-dimensional random variables (McAllester and Stratos, 2018) and rely on complex estimators or generative models. This presents a challenge to training with IB objectives.
In this paper, we analyze information quantities and relate them to surrogate objectives for the IB principle which are more friendly to optimization, showing that complex or intractable IB objectives can be replaced with simple, easy-to-compute surrogates that produce similar performance and similar
1We shorten these to information quantities from now on.
behaviour of information quantities over training. Sections 2 & 3 review commonly-used information quantities for which we provide mathematically grounded intuition via information diagrams and unify different IB objectives by identifying two key information quantities, Decoder Uncertainty H[Y | Z] and Reverse Decoder Uncertainty H[Z | Y] which act as the main loss and regularization terms in our unified IB objective. In particular, Section 3.2 demonstrates that using the Decoder Uncertainty as a training objective can minimize the training error, and shows how to estimate an upper bound on it efficiently for well-known DNN architectures. We expand on the findings of Alemi et al. (2016) in their variational IB approximation and demonstrate that this upper bound is equal to the commonly-used cross-entropy loss2 under dropout regularization. Section 3.3 examines pathologies of differential entropies that hinder optimization and proposes adding Gaussian noise to force differential entropies to become non-negative, which leads to new surrogate terms to optimize the Reverse Decoder Uncertainty. Altogether this leads to simple and tractable surrogate IB objectives such as the following, which uses dropout, adds Gaussian noise over the feature vectors f (x; η), and uses an L2 penalty over the noisy feature vectors:
min θ E x,y∼p̂(x,y), ∼N η∼dropout mask
[ − log p(Ŷ = y | z = fθ(x; η) + ) + γ ‖ fθ(x; η) + ‖22 ] . (1)
Section 4 describes experiments that validate our insights qualitatively and quantitatively on MNIST, CIFAR-10 and Imagenette, and shows that with objectives like the one in equation (1) we obtain information plane plots (as in figure 1) similar to those predicted by Tishby and Zaslavsky (2015). Our simple surrogate objectives thus induce the desired behavior of IB objectives while scaling to large, high-dimensional datasets. We present evaluations on CIFAR-10 and Imagenette images3.
Compared to existing work, we show that we can optimize IB objectives for well-known DNN architectures using standard optimizers, losses and simple regularizers, without needing complex estimators, generative models, or variational approximations. This will allow future research to make better use of IB objectives and study the IB principle more thoroughly.
2 Background
Information quantities & information diagrams. We denote entropy H[·], joint entropy H[·, ·], conditional entropy H[· | ·], mutual information I[·; ·] and Shannon’s information content h (·) (Cover and Thomas, 2012; MacKay, 2003; Shannon, 1948). We will further require the Kullback-Leibler divergence DKL(· || ·) and cross-entropy H(· || ·). The definitions can be found in section A.1. We will use differential entropies interchangeably with entropies: equalities between them are preserved in the differential setting, and inequalities will be covered in section 3.3.
2This connection was assumed without proof by Achille and Soatto (2018a;b). 3Recently, Fischer and Alemi (2020) report results on CIFAR-10 and ImageNet, see section F.4.
Information diagrams (I-diagrams), like the one depicted in figure 2, clarify the relationship between information quantities: similar to Venn diagrams, a quantity equals the sum of its parts in the diagram. Importantly, they offer a grounded intuition as Yeung (1991) show that we can define a signed measure µ* such that information quantities map to abstract sets and are consistent with set operations. We provide details on how to use I-diagrams and what to watch out for in section A.2.
Probabilistic model. We will focus on a supervised classification task that makes prediction Ŷ given data X using a latent encoding Z, while the provided target is Y. We assume categorical Y and Ŷ, and continuous X. Our probabilistic model based on these assumptions is as follows:
p(x, y, z, ŷ) = p̂(x, y) pθ(z | x) pθ(ŷ | z). (2) Thus, Z and Y are independent given X, and Ŷ is independent of X and Y given Z. The data distribution p̂(x, y) is only available to us as an empirical sample distribution. θ are the parameters we would like to learn. pθ(z | x) is the encoder from data X to latent Z, and pθ(ŷ | z) the decoder from latent Z to prediction Ŷ. Together, pθ(z | x) and pθ(ŷ | z) form the discriminative model pθ(ŷ | x): pθ(ŷ | x) = Epθ(z|x) pθ(ŷ | z). (3) We can derive the cross-entropy loss H(p̂(y | x) || pθ(Ŷ = y | x)) (Solla et al., 1988; Hinton, 1990) by minimizing the Kullback-Leibler divergence between the empirical sample distribution p̂(x, y) and the parameterized distribution pθ(x) pθ(ŷ | x), where we set pθ(x) = p̂(x). See section D.1. Mickey Mouse I-diagram. The corresponding I-diagram for X, Y, and Z is depicted in figure 2. As some of the quantities have been labelled before, we try to follow conventions and come up with consistent names otherwise. Section B.1 provides intuitions for these quantities, and section B.2 lists all definitions and equivalences explicitly. For categorical Z, all the quantities in the diagram are positive, which allows us to read off inequalities from the diagram: only I[X; Y; Z] could be negative, but as Y and Z are independent given X, we have I[Y; Z|X] = 0, and I[X; Y; Z] = I[Y; Z]−I[Y; Z|X] = I[Y; Z] ≥ 0. Section 3.3 investigates how to preserve inequalities for continuous Z.
3 Surrogate IB & DIB objectives
3.1 IB Objectives
Tishby et al. (2000) introduce the IB objective as a relaxation of a constrained optimization problem: minimize the mutual information between the input X and its latent representation Z while still accurately predicting Y from Z. An analogous objective which yields deterministic Z, the Deterministic Information Bottleneck (DIB) was proposed by Strouse and Schwab (2017). Letting β be a Lagrange multiplier, we arrive at the IB and DIB objectives:
min I[X; Z] − βI[Y; Z] for IB, and min H[Z] − βI[Y; Z] for DIB. (4) This principle can be recast as a generalization of finding minimal sufficient statistics for the labels given the data (Shamir et al., 2010; Tishby and Zaslavsky, 2015; Fisher, 2019): it strives for minimality and sufficiency of the latent Z. Minimality is achieved by minimizing the Preserved Information I[X; Z]; while sufficiency is achieved by maximizing the Preserved Relevant Information I[Y; Z]. We defer an in-depth discussion of the IB principle to the appendix Section C.1. We discuss the several variants of IB objectives, and justify our focus on IB and DIB, in Section C.2.
The information quantities that appear in the IB objective are not tractable to compute for the representations learned by many function classes of interest, including neural networks; for example, Strouse and Schwab (2017) only obtain an analytical solution to their Deterministic Information Bottleneck (DIB) method for the tabular setting. Alemi et al. (2016) address this challenge by constructing a variational approximation of the IB objective, but their approach has not been applied to more complex datasets than MNIST variants. Belghazi et al. (2018) use a separate statistics network to approximate the mutual information, a computationally expensive strategy that does not easily lend itself to optimization.
In this section, we introduce and justify tractable surrogate losses that are easier to apply in common deep learning pipelines, and which can be scaled to large and high-dimensional datasets. We begin by proposing the following reformulation of IB and DIB objectives.
Proposition 1. For IB, we obtain arg min I[X; Z] − βI[Y; Z] = arg min H[Y | Z] + β′ I[X; Z | Y]︸ ︷︷ ︸
=H[Z|Y]−H[Z|X]
, (5)
and, for DIB, arg min H[Z] − βI[Y; Z] = arg min H[Y | Z] + β′H[Z | Y] = arg min H[Y | Z] + β′′H[Z] (6)
with β′ := 1 β−1 ∈ [0,∞) and β′′ := 1 β ∈ [0, 1). The derivation can be found in section C.3.
In the next sections, we show that Decoder Uncertainty H[Y | Z] provides a loss term, which minimizes the training error, and DIB’s Reverse Decoder Uncertainty H[Z | Y] and IB’s Redundant Information I[X; Z | Y], respectively, provide a regularization term, which helps generalization. Another perspective can be found by relating the objectives to the Entropy Distance Metric introduced by MacKay (2003), which we detail in section C.4.
3.2 Decoder Uncertainty H[Y | Z]
The Decoder Uncertainty H[Y | Z] is the first term in our reformulated IB and DIB objectives, and captures the data fit component of the IB principle. This quantity is not easy to compute directly for arbitrary representations Z, so we turn our attention to two related entities instead, where we use θ as subscript to mark dependence on the model: the Prediction Cross-Entropy, denoted Hθ[Y | X] (more commonly known as the model’s cross-entropy loss; see section D.1), and the Decoder Cross-Entropy, denoted Hθ[Y | Z]. Noting that h (x) = − ln x, we define these terms as follows:
Hθ[Y | X] := H(p̂(y | x) || pθ(Ŷ = y | x)) = Ep̂(x,y) h ( Epθ(z|x) pθ(Ŷ = y | z) ) (7)
Hθ[Y | Z] := H(p(y | z) || pθ(Ŷ = y | z)) = Ep̂(x,y)Epθ(z|x) h ( pθ(Ŷ = y | z) ) . (8)
Jensen’s inequality yields Hθ[Y | X] ≤ Hθ[Y | Z], with equality iff Z is a deterministic function of X. The notational similarity4 between Hθ[Y | Z] and H[Y | Z] is deliberately suggestive: this cross-entropy bounds the conditional entropy H[Y | Z], as characterized in the following proposition. Proposition 2. The Decoder Cross-Entropy provides an upper bound on the Decoder Uncertainty:
H[Y | Z] ≤ H[Y | Z] + DKL(p(y | z) || pθ(ŷ | z)) = Hθ[Y | Z], (9) and further bounds the training error:
p(“Ŷ is wrong”) ≤ 1 − e−Hθ[Y|Z] = 1 − e−(H[Y|Z]+DKL(p(y|z)||pθ(ŷ|z))). (10) Likewise, for Hθ[Y | X] and H[Y | X]. See section D.2 for a derivation.
Hence, by bounding DKL(p(y | z) || pθ(ŷ | z)), we can obtain a bound for the training error in terms of H[Y | Z]. We examine one way of doing so by using optimal decoders pθ(ŷ | z) := p(Y = ŷ | z) for the case of categorical Z in section E.
Alemi et al. (2016) use the Decoder Cross-Entropy bound in equation (9) to variationally approximate p(y | z). We make this explicit by applying the reparameterization trick to rewrite the latent z as a parametric function of its input x and some independent auxiliary random variable η, i.e. fθ(x, η) D = z ∼ pθ(z | x), yielding
H[Y | Z] ≤ Hθ[Y | Z] = Ep̂(x,y)Ep(η) h ( pθ(Ŷ = y | z = fθ(x; η)) ) . (11)
Equation (11) can be applied to many forms of stochastic regularization that turn deterministic models into stochastic ones, in particular dropout. This allows us to use modern DNN architectures as stochastic encoders.
Dropout regularization When we interpret η as a sampled dropout mask for a DNN, DNNs that use dropout regularization (Srivastava et al., 2014), or variants like DropConnect (Wan et al., 2013a), fit the equation above as stochastic encoders. Monte-Carlo dropout (Gal and Ghahramani, 2016), for example, even specifically estimates the predictive mean pθ(ŷ | x) from equation (3). The following result extends the observation by Burda et al. (2015) that sampling yields an unbiased estimator for the Decoder Cross-Entropy Hθ[Y | Z], while it only yields a biased estimator for the Prediction Cross-Entropy Hθ[Y | X] (which it upper-bounds).
4This notation is compatible withV-Entropy introduced by Xu et al. (2020).
Corollary 1. Let x, y, z and fθ be defined as previously, with η a sampled stochastic dropout mask. Then h ( pθ(Ŷ = y | z = fθ(x; η)) ) evaluated for a single sample η is an unbiased estimator of the Decoder Cross-Entropy Hθ[Y | Z], and an estimator of an upper bound on the Prediction CrossEntropy Hθ[Y | X].
This distinction between the Decoder Cross-Entropy and the Prediction Cross-Entropy has been observed in passing in the literature, but not made explicit. Multi-sample approaches like Multi-Sample Dropout (Inoue, 2019), for example, optimize Hθ[Y | Z], while Importance Weighted Stochastic Gradient Descent (Noh et al., 2017) optimizes Hθ[Y | X]. Dusenberry et al. (2020) observe empirically in the different context of rank-1 Bayesian Neural Networks that optimizing Hθ[Y | Z] instead of Hθ[Y | X] is both easier and also yields better generalization performance (NLL, accuracy, and ECE), while they also put forward an argument for why the stochastic gradients for Hθ[Y | Z] might benefit from lower variance. We empirically compare training with either cross-entropy in section G.3.3 and show results in figure G.11 in the appendix. We conclude this section by highlighting that H[Y | Z] is therefore already minimized in modern DNN architectures that use dropout together with a cross-entropy loss. This means that, at least for one half of our reformulation of the IB objective, we can apply off-the-shelf, scalable objectives and optimizers for its minimization.
3.3 Surrogates for the regularization terms
In the previous section, we have examined how to tractably estimate the error minimization term H[Y | Z]. In this section, we will examine tractable optimization of the regularization terms H[Z | Y] and I[X; Z | Y], respectively. We discuss how to minimize entropies meaningfully and show how this unifies DIB and IB via the inequality I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] before providing tractable upper-bounds for H[Z | Y] and H[Z]. Differential entropies In most cases, the latent Z is a continuous random variable in many dimensions. Unlike entropies on discrete probability spaces, differential entropies defined on continuous spaces are not bounded from below. This means that the DIB objective is not guaranteed to have an optimal solution and allows for pathological optimization trajectories in which the variance of the latent Z can be scaled to be arbitrarily small, achieving arbitrarily high-magnitude negative entropy. We provide a toy experiment demonstrating this in section G.4.
Intuitively, one can interpret this issue as being allowed to encode information in an arbitrarily-small real number using infinite precision, similar to arithmetic coding (MacKay, 2003; Shwartz-Ziv and Tishby, 2017)5. In practice, due to floating point constraints, optimizing DIB naively will invariably end in garbage predictions and underflow as activations approach zero. It is therefore not desirable
5Conversely, MacKay (2003) notes that without upper-bounding the “power" Ep(z) Z2, all information could be encoded in a single very large integer.
for training. This is why Strouse and Schwab (2017) only consider analytical solutions to DIB by evaluating a limit for the tabular case. MacKay (2003) proposes the introduction of noise to solve this issue in the application of continuous communication channels.
However, here we propose adding specific noise to the latent representation to lower-bound the conditional entropy of Z, which allows us to enforce non-negativity across all IB information quantities as in the discrete case and transport inequalities to the continuous case: for a continuous Ẑ ∈ Rk and independent noise , we set Z := Ẑ+ ; the differential entropy then satisfies H[Z] = H[Ẑ+ ] ≥ H[ ]; and by using zero-entropy noise ∼ N(0, 12πe Ik) specifically, we obtain H[Z] ≥ H[ ] = 0. Proposition 3. After adding zero-entropy noise, the inequality I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] also holds for continuous Z, and we can minimize I[X; Z | Y] in the IB objective by minimizing H[Z | Y] or H[Z], similarly to the DIB objective.
Strictly speaking, zero-entropy noise is not necessary for optimizing the bounds: any Gaussian noise is sufficient, but zero-entropy noise is aesthetically appealing as it preserves inequalities from the discrete setting. In a sense, this propostion bounds the IB objective by the DIB objective. However, adding noise changes the optimal solutions: whereas DIB in Strouse and Schwab (2017) leads to hard clustering in the limit, adding noise leads to soft clustering when optimizing the DIB objective, as is the case with the IB objective. We show in section F.6 that minimizing the DIB objective with noise leads to soft clustering (for the case of an otherwise deterministic encoder). Altogether, in addition to Shwartz-Ziv and Tishby (2017), we argue that noise is essential to obtain meaningful differential entropies and to avoid other pathological cases as described further in section F.7.
It is not generally possible to compute H[Z | Y] exactly for continuous latent representations Z, but we can derive an upper bound. The maximum-entropy distribution for a given covariance matrix Σ is a Gaussian with the same covariance. Proposition 4. The Reverse Decoder Uncertainty can be approximately bounded using the empirical variance V̂ar[Zi | y]:
H[Z | Y] ≤ Ep̂(y) ∑
i
1 2 ln(2πe Var[Zi | y]) ≈ Ep̂(y) ∑ i 1 2 ln(2πe V̂ar[Zi | y]), (12)
where Zi are the individual components of Z. H[Z] can be bounded similarly. More generally, we can create an even looser upper bound by bounding the mean squared norm of the latent: E ‖Z‖2 ≤ C′ ⇒ H[Z | Y] ≤ H[Z] ≤ C, (13) with C′ := ke 2C/k
2πe for Z ∈ Rk. See section F.2 for proof.
Surrogate objectives These surrogate terms provide us with three different upper-bounds that we can use as surrogate regularizers. We refer to them as: conditional log-variance regularizer (log Var[Z | Y]), log-variance regularizer (log Var[Z]) and activation L2 regularizer (E ‖Z‖2). We can now propose the main results of this paper: IB surrogate objectives that reduce to an almost trivial implementation using the cross-entropy loss and one of the regularizers above while adding zero-entropy noise to the latent Z. Theorem 1. Let Z be obtained by adding a single sample of zero-entropy noise to a single sample of the output z of the stochastic encoder. Then each of the following objectives is an estimator of an upper bound on the IB objective. In particular, for the surrogate objective E ‖Z‖2, we obtain:
min H(p(y | z) || pθ(Ŷ = y | z)) + γ‖z‖2; (14) for log Var[Z | Y]:
min H(p(y | z) || pθ(Ŷ = y | z)) + γEp̂(y) ∑
i
1 2 ln(2πe V̂ar[Zi | y]); (15)
and for log Var[Z]: min H(p(y | z) || pθ(Ŷ = y | z)) + γ ∑
i
1 2 ln(2πe V̂ar[Zi]). (16)
For the latter two surrogate regularizers, we can relate their coefficient γ to β′, β′′ and β from section 3. However, as regularizing E ‖Z‖2 does not approximate an entropy directly, its coefficient does not relate to the Lagrange multiplier of any fixed IB objective. We compare the performance of these objectives in section 4.
0.001 0.01 0.1 1
0
0.25
0.50
0.75
1
A cc
u ra
cy
EZ2
γ 10−6 10−4 10−2 1
0.001 0.01 0.1 1
Epsilon
Weight Decay
R o b u stn ess
0.001 0.01 0.1 1
0
0.25
0.50
0.75
1
A cc
u ra
cy EZ2
γ 10−6 10−4 10−2 1
0.001 0.01 0.1 1
Epsilon
Weight Decay
R o b u stn ess
(a) Robustness for different attack strengths . The dashed black line represents a model trained only with cross-entropy and no noise injection. We see that models trained with the surrogate IB objective (colored by γ) see improved robustness over a model trained only to minimize the cross-entropy training objective (shown in black) while the models regularized with weight-decay actually perform worse.
50 100 300
0
0.25
0.50
0.75
1
A cc
u ra
cy
EZ2
γ 10−6 10−4 10−2 1
50 100 300
Preserved Information I[X;Z]
Weight Decay
(b) Average robustness over ∈ [0, 0.1] compared to n rmal accuracy for different amounts of Preserved Information. ◦ markers show robustness. × markers show the normal accuracy. We see that robustness depends on the Preserved Information. If the latent is compressesed too much, robustness (and accuracy) are low. If the latent is not compressed enough, robustness and thus generalization suffer.
Figure 5: Adversarial robustness of ResNet18 models trained on CIFAR-10 with surrogate objectives in comparison to regularization with L2 weight-decay as non-IB method. The robustness is evaluated using FGSM, PGD, DeepFool and BasicIterative attacks of varying values.
4 Experiments
We now provide empirical verification of the claims made in the previous sections. Our goal in this section is to highlight two main findings: first, that our surrogate objectives obtain similar behavior to what we expect of exact IB objectives with respect to their effect on robustness to adversarial examples. In particular, we show that our surrogate IB objectives improve adversarial robustness compared to models trained only on the cross-entropy loss, consistent with the findings of Alemi et al. (2016). Second, we show the effect of our surrogate objectives on information quantities during training by plotting information plane diagrams, demonstrating that models trained with our objectives trade off between I[X; Z] and I[Y; Z] as expected. We show this by recovering information plane plots similar to the ones in Tishby and Zaslavsky (2015) and qualitatively examine the optimization behavior of the networks through their training trajectories. We demonstrate the scalability of our surrogate objectives by applying our surrogate IB objectives to the CIFAR-10 and Imagenette datasets, high-dimensional image datasets.
For details about our experiment setup, DNN architectures, hyperparameters and additional insights, see section G. In particular, empirical quantification of our observations on the relationship between the Decoder Cross-Entropy loss and the Prediction Cross-Entropy are deferred to the appendix due to space limitations as well as the description of the toy experiment that shows that minimizing H[Z | Y] for continuous latent Z without adding noise does not constrain information meaningfully and that adding noise solves the issue as detailed in section 3.3.
Robustness to adversarial attacks Alemi et al. (2016) and Fischer and Alemi (2020) observe that their IB objectives lead to improved adversarial robustness over standard training objectives. We perform a similar evaluation to see whether our surrogate objectives also see improved robustness. We train a fully-connected residual network on CIFAR-10 for a range of regularization coefficients γ using our E ‖Z‖2 surrogate objective; we then compare against a similar regularization method that does not have an information-theoretic interpretation: L2 weight-decay. We inject zero-entropy noise in both cases. After training, we evaluate the models on adversarially perturbed images using the FGSM (Szegedy et al., 2013), PGD (Madry et al., 2018), BasicIterative (Kurakin et al., 2017) and DeepFool (Moosavi-Dezfooli et al., 2016) attacks for varying levels of the perturbation magnitude parameter . We also compare to a simple unregularized cross-entropy baseline (black dashed line). To compute overall robustness, we use each attack in turn and only count a sample as robust if it defeats them all. As depicted in figure 5, we find that our surrogate objectives yield significantly more robust models while obtaining similar test accuracy on the unperturbed data whereas weight-decay regularization reduces robustness against adversarial attacks. Plots for the other two regularizers can be found in the appendix in figure G.13 and figure G.14.
Information plane plots for CIFAR-10 To compare the different surrogate regularizers, we again use a ResNet18 model on CIFAR-10 with zero-entropy noise added to the final layer activations Z, with K = 256 dimensions, as an encoder and add a single K × 10 linear unit as a decoder. We train with the surrogate objectives from section 3.3 for various γ, chosen in logspace from different ranges to compensate for their relationship to β as noted in section 3.3: for log Var[Z], γ ∈ [10−5, 1]; for log Var[Z | Y], γ ∈ [10−5, 10]; and for E ‖Z‖2, by trial and error, γ ∈ [10−6, 10]. We estimate information quantities using the method of Kraskov et al. (2004).
Figure 6 shows an information plane plot for regularizing with E ‖Z‖2 for different γ over different epochs for the training set. Similar to Shwartz-Ziv and Tishby (2017), we observe that there is an initial expansion phase followed by compression. The jumps in performance (reduction of the Residual Information) are due to drops in the learning rate. In figure 4, we can see that the saturation curves for all 3 surrogate objectives qualitatively match the predicted curve from Tishby and Zaslavsky (2015). Figure G.1 shows the difference between the regularizers more clearly, and figure G.3 shows the training trajectories for all three regularizers. More details in section G.3.1.
Information plane plots for Imagenette To show that our surrogate objectives also scale up to larger datasets, we run a similar experiment on Imagenette (Howard, 2019), which is a subset of ImageNet with 10 classes with 224 × 224 × 3 = 1.5 × 105 input dimensions, and on which we obtain 90% test accuracy. See the figure 1, which shows the trajectories on the test set. We obtain similar plots to the ones obtained for CIFAR-10, showing that our surrogate objectives scale well to higher-dimensional datasets despite their simplicity.
5 Conclusion
The contributions of this paper have been threefold: First, we have proposed simple, tractable training objectives which capture many of the desirable properties of IB methods while also scaling to problems of interest in deep learning. For this we have introduced implicit stochastic encoders, e.g. using dropout, and compared multi-sample dropout approaches to identify the one that approximates the Decoder Uncertainty Hθ[Y | Z], relating them to the cross-entropy loss that is commonly used for classification problems. This widens the range of DNN architectures that can be used with IB objectives considerably. We have demonstrated that our objectives perform well for practical DNNs without cumbersome density models. Second, we have motivated our objectives by providing insight into limitations of IB training, demonstrating how to avoid pathological behavior in IB objectives, and by endeavouring to provide a unifying view on IB approaches. Third, we have provided mathematically grounded intuition by using I-diagrams for the information quantities involved in IB, shown common pitfalls when using information quantities and how to avoid them, and examined how the quantities relate to each other. Future work investigating the practical constraints on the expressivity of a given neural network may provide further insight into how to measure compression in neural networks. Moreover, the connection to Bayesian Neural Networks remains to be explored.
References
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):1947–1980, 2018a.
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897–2905, 2018b.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
Rana Ali Amjad and Bernhard Claus Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pages 531–540, 2018.
Yoshua Bengio et al. Learning deep architectures for ai. Foundations and trends R© in Machine Learning, 2(1):1–127, 2009.
J-F Bercher and Christophe Vignat. A renyi entropy convolution inequality with application. In 2002 11th European Signal Processing Conference, pages 1–4. IEEE, 2002.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519, 2015.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in β-vae. arXiv preprint arXiv:1804.03599, 2018.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Michael W Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-an Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. arXiv preprint arXiv:2005.07186, 2020.
Ian Fischer. The conditional entropy bottleneck. arXiv preprint arXiv:2002.05379, 2020.
Ian Fischer and Alexander A. Alemi. Ceb improves model robustness. Entropy, 22(10):1081, Sep 2020. ISSN 1099-4300. doi: 10.3390/e22101081. URL http://dx.doi.org/10.3390/ e22101081.
Ian Fisher. The Conditional Entropy Bottleneck. Submission to ICLR 2019, International Conference on Learning Representations, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059, 2016.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv:1903.12436, 2019.
David Gondek and Thomas Hofmann. Conditional information bottleneck clustering. In 3rd ieee international conference on data mining, workshop on clustering large data sets, pages 36–42. Citeseer, 2003.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. Lecture Notes in Computer Science, page 630–645, 2016b.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
Geoffrey E Hinton. Connectionist learning procedures. In Machine learning, pages 555–610. Elsevier, 1990.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.
Jeremy Howard. Imagewang. 2019. URL https://github.com/fastai/imagenette/.
Hiroshi Inoue. Multi-sample dropout for accelerated training and better generalization. arXiv preprint arXiv:1905.09788, 2019.
Morris A. Jette, Andy B. Yoo, and Mark Grondona. Slurm: Simple linux utility for resource management. In In Lecture Notes in Computer Science: Proceedings of Job Scheduling Strategies for Parallel Processing (JSSPP) 2003, pages 44–60. Springer-Verlag, 2002.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pages 7024–7035, 2019.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066138, 2004.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial machine learning at scale. 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
David J. C. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.04251, 2018.
William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93–111, 1954.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016.
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. In Advances in Neural Information Processing Systems, pages 5109–5118, 2017.
Morteza Noshad, Yu Zeng, and Alfred O Hero. Scalable mutual information estimation using dependence graphs. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2962–2966. IEEE, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024–8035, 2019.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint arXiv:1905.06922, 2019.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411(29-30):2696–2711, 2010.
Claude E Shannon. A mathematical theory of communication. Bell system technical journal, 27(3): 379–423, 1948.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.
Sara A. Solla, Esther Levin, and Michael Fleisher. Accelerated learning in layered neural networks. Complex Systems, 2, 1988.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
DJ Strouse and David J Schwab. The deterministic information bottleneck. Neural computation, 29 (6):1611–1630, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625, 2019.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058–1066, 2013a.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058–1066, 2013b.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689, 2020.
Raymond W Yeung. A new outlook on shannon’s information measures. IEEE transactions on information theory, 37(3):466–474, 1991.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4320– 4328, 2018.
A Information quantities & information diagrams
Here we introduce notation and terminology in greater detail than in the main paper. We review well-known information quantities and provide more details on using information diagrams (Yeung, 1991).
A.1 Information quantities
We denote entropy H[·], joint entropy H[·, ·], conditional entropy H[· | ·], mutual information I[·; ·] and Shannon’s information content h (·) following Cover and Thomas (2012); MacKay (2003); Shannon (1948) :
h (x) = − ln x H[X] = Ep(x) h ( p(x) ) H[X,Y] = Ep(x,y) h ( p(x, y)
) H[X | Y] = H[X,Y] − H[Y]
= Ep(y) H[X | y] = Ep(x,y) h ( p(x | y))
I[X; Y] = H[X] + H[Y] − H[X,Y] = Ep(x,y) h ( p(x) p(y)
p(x,y) ) I[X; Y | Z] = H[X | Z] + H[Y | Z] − H[X,Y | Z],
where X,Y,Z are random variables and x, y, z are outcomes these random variables can take.
We use differential entropies interchangeably with entropies. We can do so because equalities between them hold as can be verified by symbolic expansions. For example,
H[X,Y] = H[X | Y] + H[Y] ⇔Ep(x,y) h ( p(x, y) ) = Ep(x,y) [ h ( p(x | y)) + h (p(y))] = Ep(x,y) [h (p(x | y))] +Ep(y) h (p(y)),
which is valid in both the discrete and continuous case (if the integrals all exist). The question of how to transfer inequalities in the discrete case to the continuous case is dealt with in section 3.3.
We will further require the Kullback-Leibler divergence DKL(· || ·) and cross-entropy H(· || ·): H(p(x) || q(x)) = Ep(x) h ( q(x) ) DKL(p(x) || q(x)) = Ep(x) h ( q(x) p(x)
) H(p(y | x) || q(y | x)) = Ep(x)Ep(y|x) h ( q(y | x))
= Ep(x,y) h ( q(y | x))
DKL(p(y | x) || q(y | x)) = Ep(x,y) h ( q(y|x) p(y|x) )
A.2 Information diagrams
Information diagrams (I-diagrams), like the one depicted in figure 2 (or figure H.1 for a bigger version), visualize the relationship between information quantities: Yeung (1991) shows that we can define a signed measure µ* such that these well-known quantities map to abstract sets and are consistent with set operations.
H[A] = µ*(A)
H[A1, . . . , An] = µ*(∪iAi) H[A1, . . . , An | B1, . . . , Bn] = µ*(∪iAi − ∪iBi)
I[A1; . . . ; An] = µ*(∩iAi) I[A1; . . . ; An | B1, . . . , Bn] = µ*(∩iAi − ∪iBi)
Note that interaction information (McGill, 1954) follows as canonical generalization of the mutual information to multiple variables from that work, whereas total correlation does not.
In other words, equalities can be read off directly from I-diagrams: an information quantity is the sum of its parts in the corresponding I-diagram. This is similar to Venn diagrams. The sets used in I-diagrams are just abstract symbolic objects, however.
An important distinction between I-diagrams and Venn diagrams is that while we can always read off inequalities in Venn diagrams, this is not true for I-diagrams in general because mutual information terms in more than two variables can be negative. In Venn diagrams, a set is always larger or equal any subset.
However, if we show that all information quantities are non-negative, we can read off inequalities again. We do this for figure 2 at the end of section 2 for categorical Z and expand this to continuous Z in section 3.3. Thus, we can treat the Mickey Mouse I-diagram like a Venn diagram to read off equalities and inequalities.
Nevertheless, caution is warranted sometimes. As the signed measure can be negative, µ*(X ∩ Y) = 0 does not imply X ∩ Y = ∅: deducing that a mutual information term is 0 does not imply that one can simply remove the corresponding area in the I-diagram. There could be Z with µ*((X ∩ Y) ∩ Z) < 0, such that µ*(X ∩ Y) = µ*(X ∩ Y ∩ Z) + µ*(X ∩ Y − Z) = 0 but X ∩ Y , ∅. This also means that we cannot drop the term from expressions when performing symbolic manipulations. This is of particular importance because a mutual information of zero means two random variables are independent, which might invite one drawing them as disjoint areas.
The only time where one can safely remove an area from the diagram is for atomic quantities, which are quantities which reference all the available random variables (Yeung, 1991). For example, when we only have three variables X,Y,Z, I[X; Y; Z] and I[X; Y | Z] are atomic quantities. We can safely remove atomic quantities from I-diagrams when they are 0 as there are no random variables left to apply that could lead to the problem explored above.
Continuing the example, 0 = I[X; Y; Z] = µ*(X ∩ Y ∩ Z) would imply X ∩ Y ∩ Z = ∅, and we could remove it from the diagram without loss of generality. Moreover, atomic I[X; Y |Z] = µ*(X∩Y−Z) = 0 then and could be removed from the diagram as well.
We only use I-diagrams for the three variable case, but they supply us with tools to easily come up with equalities and inequalities for information quantities. In the general case with multiple variables, they can be difficult to draw, but for Markov chains they can be of great use.
B MickeyMouse I-diagram
B.1 Intuition for theMickeyMouse information quantities
We base the names of information quantities on existing conventions and come up with sensible extensions. For example, the name Preserved Relevant Information for I[Y; Z] was introduced by Tishby and Zaslavsky (2015). It can be seen as the intersection of I[X; Z] and I[X; Y] in the I-diagram, and hence we denote I[X; Z] Preserved Information and I[X; Y] Relevant Information, which are sensible names as we detail below.
We identify the following six atomic quantities:
Label Uncertainty H[Y | X] quantifies the uncertainty in our labels. If we have multiple labels for the same data sample, it will be > 0. It is 0 otherwise.
Encoding Uncertainty H[Z | X] quantifies the uncertainty in our latent encoding given a sample. When using a Bayesian model with random variable ω for the weights, one can further split this term into H[Z | X] = I[Z;ω | X] + H[Z | X, ω], so uncertainty stemming from weight uncertainty and independent noise (Houlsby et al., 2011; Kirsch et al., 2019).
Preserved Relevant Information I[Y; Z] quantifies information in the latent that is relevant for our task of predicting the labels (Tishby and Zaslavsky, 2015). Intuitively, we want to maximize it for good predictive performance.
Residual Information I[X; Y | Z] quantifies information for the labels that is not captured by the latent (Tishby and Zaslavsky, 2015) but would be useful to be captured.
Redundant Information I[X; Z | Y] quantifies information in the latent that is not needed for predicting the labels6.
We also identify the following composite information quantities:
Relevant Information I[X; Y] = I[X; Y | Z] + I[Y; Z] quantifies the information in the data that is relevant for the labels and which our model needs to capture to be able to predict the labels.
Preserved Information I[X; Z] = I[X; Z | Y] + I[Y; Z] quantifies information from the data that is preserved in the latent.
Decoder Uncertainty H[Y | Z] = I[X; Y | Z] + H[Y | X] quantifies the uncertainty about the labels after learning about the latent Z. If H[Y | Z] reaches 0, it means that no additional information is needed to infer the correct label Y from the latent Z: the optimal decoder can be a deterministic mapping. Intuitively, we want to minimize this quantity for good predictive performance.
Reverse Decoder Uncertainty H[Z | Y] = I[X; Z | Y] + H[Z | X] quantifies the uncertainty about the latent Z given the label Y. We can imagine training a new model to predict Z given Y and minimizing H[Z | Y] to 0 would allow for a deterministic decoder from the latent to given the label.
Nuisance7 H[X | Y] = H[X | Y,Z] + I[X; Z] quantifies the information in the data that is not relevant for the task (Achille and Soatto, 2018a).
B.2 Definitions & equivalences
The following equalities can be read off from figure 2. For completeness and to provide a handy reference, we list them explicitly here. They can also be verified using symbolic manipulations and the properties of information quantities.
Equalities for composite quantities: I[X; Y] = I[X; Y | Z] + I[Y; Z] (17) I[X; Z] = I[X; Z | Y] + I[Y; Z] (18)
H[Y | Z] = I[X; Y | Z] + H[Y | X] (19) H[Z | Y] = I[X; Z | Y] + H[Z | X] (20) H[X | Y] = H[X | Y,Z] + I[X; Z] (21)
We can combine the atomic quantities into the overall Label Entropy and Encoding Entropy: H[Y] = H[Y | X] + I[Y; Z] + I[X; Y | Z] (22) H[Z] = H[Z | X] + I[Y; Z] + I[X; Z | Y]. (23)
We can express the Relevant Information I[X; Y], Residual Information I[X; Y | Z], Redundant Information I[X; Z | Y] and Preserved Information I[X; Z] without X on the left-hand side:
I[X; Y] = H[Y] − H[Y | X], (24) I[X; Z] = H[Z] − H[Z | X], (25)
I[X; Y | Z] = H[Y | Z] − H[Y | X], (26) I[X; Z | Y] = H[Z | Y] − H[Z | X]. (27)
This simplifies estimating these expressions as X is usually much higher-dimensional and irregular than the labels or latent encodings. We also can rewrite the Preserved Relevant Information I[Y; Z] as:
I[Y; Z] = H[Y] − H[Y | Z] (28) I[Y; Z] = H[Z] − H[Z | Y] (29)
6Fisher (2019) uses the term “Residual Information” for this, which conflicts with Tishby and Zaslavsky (2015).
7Not depicted in figure 2.
C Information bottleneck & related works
C.1 Goals & motivation
The IB principle from Tishby et al. (2000) can be recast as a generalization of finding minimal sufficient statistics for the labels given the data (Shamir et al., 2010; Tishby and Zaslavsky, 2015; Fisher, 2019): it strives for minimality and sufficiency of the latent Z. Minimality is about minimizing amount of information necessary of X for the task, so minimizing the Preserved Information I[X; Z]; while sufficiency is about preserving the information to solve the task, so maximizing the Preserved Relevant Information I[Y; Z].
From figure 2, we can read off the definitions of Relevant Information and Preserved Information: I[X; Y] = I[Y; Z] + I[X; Y | Z] (30) I[X; Z] = I[Y; Z] + I[X; Z | Y], (31)
and see that maximizing the Preserved Relevant Information I[Y; Z] is equivalent to minimizing the Residual Information I[X; Y | Z], while minimizing the Preserved Information I[X; Z] at the same time means minimizing the Redundant Information I[X; Z | Y], too, as I[X; Y] is constant for the given dataset8. Moreover, we also see that the Preserved Relevant Information I[Y; Z] is upper-bounded by Relevant Information I[X; Y], so to capture all relevant information in our latent, we want I[X; Y] = I[Y; Z].
Using the diagram, we can also see that minimizing the Residual Information is the same as minimizing the Decoder Uncertainty H[Y | Z]:
I[X; Y | Z] = H[Y | Z] − H[Y | X]. Ideally, we also want to minimize the Encoding Uncertainty H[Z | X] to find the most deterministic latent encoding Z. Minimizing the Encoding Uncertainty and the Redundant Information I[X; Z | Y] together is the same as minimizing the Reverse Decoder Uncertainty H[Z | Y]. All in all, we want to minimize both the Decoder Uncertainty H[Y | Z] and the Reverse Decoder Uncertainty H[Z | Y].
C.2 IB objectives
“The Information BottleneckMethod” (IB)
Tishby et al. (2000) introduce MI(X; X̂) − βMI(X̂; Y) as optimization objective for the Information Bottleneck. We can relate this to our notation by renaming X̂ = Z, such that the objective becomes “min I[X; Z] − βI[Y; Z]”. The IB objective minimizes the Preserved Information I[X; Z] and trades it off with maximizing the Preserved Relevant Information I[Y; Z]. Tishby and Zaslavsky (2015) mention that the IB objective is equivalent to minimizing I[X; Z] + βI[X; Y | Z], see our discussion above. Tishby et al. (2000) provide an optimal algorithm for the tabular case, when X, Y and Z are all categorical. This has spawned additional research to optimize the objective for other cases and specifically for DNNs.
“Deterministic Information Bottleneck” (DIB)
Strouse and Schwab (2017) introduce as objective “min H[Z] − βI[Y; Z]”. Compared to the IB objective, this also minimizes H[Z | X] and encourages determinism. Vice-versa, for deterministic encoders, H[Z | X] = 0, and their objective matches the IB objective. Like Tishby et al. (2000), they provide an algorithm for the tabular case. To do so, they examine an analytical solution for their objective as it is unbounded: H[Z | X]→ −∞ for the optimal solution. As we discuss in section 3.3, it does not easily translate to a continuous latent representation.
“Deep Variational Information Bottleneck”
Alemi et al. (2016) rewrite the terms in the bottleneck as maximization problem “max I[Y; Z] − βI[X; Z]” and swap the β parameter. Their β would be 1/β in IB above, which emphasizes that I[Y; Z] is important for performance and I[X; Z] acts as regularizer.
8That is, it does not depend on θ.
The paper derives the following variational approximation to the IB objective, where z = fθ(x, ) denotes a stochastic latent embedding with distribution pθ(z | x), pθ(ŷ | z) denotes the decoder, and r(z) is some fixed prior distribution on the latent embedding:
minEp̂(x,y)E ∼p( ) [ − log pθ(Ŷ = y | z = fθ(xn, )) + γDKL(p(z|xn)||r(z)) ] . (32)
In principle, the distributions pθ(ŷ | z) and pθ(z | x) could be given by arbitrary parameterizations and function approximators. In practice, the implementation of DVIB presented by Alemi et al. (2016) constructs pθ(z | x) as a multivariate Gaussian with parameterized mean and parameterized diagonal covariance using a neural network, and then uses a simple logistic regression to obtain pθ(ŷ | z), while arbitrarily setting r(z) to be a unit Gaussian around the origin. The requirement for pθ(z | x) to have a closed-form Kullback-Leibler divergence limits the applicability of the DVIB objective.
The DVIB objective can be written more concisely as min Hθ[Y | Z] + γDKL(p(z | x) || r(z))
in the notation introduced in section 3. We discuss the regularizer in more detail in section F.3.
“Conditional Entropy Bottleneck”
In a preprint, Fisher (2019) introduce their Conditional Entropy Bottleneck as “min I[X; Z | Y] − I[Y; Z]”. We can rewrite the objective as I[X; Z | Y] + I[X; Y | Z] − I[X; Y], using equations (30) and (31). The last term is constant for the dataset and can thus be dropped. Likewise, the IB objective can be rewritten as minimizing I[X; Z | Y] + (β − 1)I[X; Y | Z]. The two match for β = 2. Fisher (2019) provides experimental results that favorably compare to Alemi et al. (2016), possibly due to additional flexibility as Fisher (2019) do not constrain p(z) to be a unit Gaussian and employ variational approximations for all terms. We relate CEB to Entropy Distance Metric in section C.4.
“Conditional Entropy Bottleneck” (2020)
In a substantial revision of the preprint, Fischer (2020) change their Conditional Entropy Bottleneck to include a Lagrange multiplier: “min I[X; Z | Y] − γI[Y; Z]”. Their VCEB objective can be written more concisely as
min Hθ[Y | Z] + γ(Hθ[Z | Y] − Hθ[Z | X]), where, without writing down the probabilistic model, we introduce variational approximations for the Reverse Decoder Uncertainty and the Encoding Uncertainty.
They are the first to report results on CIFAR-10. It is not clear how they parameterize the model they use for CIFAR-10. They use one Gaussian per class to model Hθ[Z | Y].
“CEB ImprovesModel Robustness”
Fischer and Alemi (2020) take CEB and switch to a deterministic model which they turn it into a stochastic encoder by adding unit Gaussian noise. They use Gaussians of fixed variance to variationally approximate q(y | z): for each class, q(y | z) is modelled as a separate Gaussian. They are the first to report results on ImageNet and report good rebustness against adversarial attacks without adversarial training.
C.3 Canonical IB & DIB objectives
We expand the IB and DIB objectives into “disjoint” terms and drop constant ones to find a more canonical form. This leads us to focus on the optimization of the Decoder Uncertainty H[Y | Z] along with additional regularization terms. In section 3.2, we discuss the properties of H[Y | Z], and in section 3.3 we examine the regularization terms.
Proposition. For IB, we obtain arg min I[X; Z] − βI[Y; Z] = arg min H[Y | Z] + β′ I[X; Z | Y]︸ ︷︷ ︸
=H[Z|Y]−H[Z|X]
, (33)
and, for DIB, arg min H[Z] − βI[Y; Z] = arg min H[Y | Z] + β′H[Z | Y] = arg min H[Y | Z] + β′′H[Z] (34)
with β′ := 1 β−1 ∈ [0,∞) and β′′ := 1 β ∈ [0, 1).
Proof. For the steps marked with *, we make use of β > 1. For IB, we obtain arg min I[X; Z] − βI[Y; Z] = arg min I[X; Z | Y] + (β − 1)H[Y | Z]
(*) = arg min H[Y | Z] + β′ I[X; Z | Y] arg min H[Y | Z] + β′(H[Z | Y] − H[Z | X]), (IB)
and, for DIB, arg min H[Z] − βI[Y; Z] = arg min H[Z | Y] + (β − 1)H[Y | Z]
(*) = arg min H[Y | Z] + β′H[Z | Y], (DIB)
with β′ := 1 β−1 ∈ [0,∞). Similarly, we show for DIB
arg min H[Z] − βI[Y; Z] = arg min H[Z] + βH[Y | Z] (*) = arg min H[Y | Z] + β′′H[Z],
with β′′ := 1 β ∈ [0, 1), which is relevant in section 3.3.
We limit ourselves to β > 1, because, for β < 1, we would be maximizing the Decoder Uncertainty, which does not make sense: the obvious solution to this is one where Z contains no information on Y, that is p(y | z) is uniform. In the case of DIB, it is to map every input deterministically to a single latent; whereas for IB, we only minimize the Redundant Information, and the solution is free to contain noise. For β = 1, we would not care about Decoder Uncertainty and only minimize Redundant Information and Reverse Decoder Uncertainty, respectively, which allows for arbitrarily bad predictions.
We note that we have β′ = β ′′
1−β′′ using the relations above.
C.4 IB objectives and the Entropy DistanceMetric
Another perspective on the IB objectives is by expressing them using the Entropy Distance Metric. MacKay (2003, p. 140) introduces the entropy distance
EDM (Y,Z) = H[Y | Z] + H[Z | Y]. (35) as a metric when we identify random variables up to permutations of the labels for categorical variables: if the entropy distance is 0, Y and Z are the same distribution up to a consistent permutation of the labels (independent of X). If the entropy distance becomes 0, both H[Y | Z] = 0 = H[Z | Y], and we can find a bijective map from Z to Y.9
We can express the Reverse Decoder Uncertainty H[Z | Y] using the Decoder Uncertainty H[Y | Z] and the entropies:
H[Z | Y] + H[Y] = H[Y | Z] + H[Z], and rewrite equation (35) as
EDM (Y,Z) = 2H[Y | Z] + H[Z] − H[Y]. For optimization purposes, we can drop constant terms and rearrange:
arg min EDM (Y,Z) = arg min H[Y | Z] + 12 H[Z].
C.4.1 Rewriting IB and DIB using the Entropy DistanceMetric
For β ≥ 1, we can rewrite equations (IB) and (DIB) as: arg min EDM (Y,Z) + γ (H[Y | Z] − H[Z | Y]) + (γ − 1) H[Z | X] (36)
for IB, and arg min EDM (Y,Z) + γ (H[Y | Z] − H[Z | Y]) (37)
9The argument for continuous variables is the same. We need to identify distributions up to “isentropic” bijections.
for DIB and replace β with γ = 1 − 2 β ∈ [−1, 1] which allows for a linear mix between H[Y | Z] and H[Z | Y]. DIB will encourage the model to match both distributions for γ = 0 (β = 2), as we obtain a term that matches the Entropy Distance Metric from section C.4, and otherwise trades off Decoder Uncertainty and Reverse Decoder Uncertainty. IB behaves similarly but tends to maximize Encoding Uncertainty as γ − 1 ∈ [−2, 0]. Fisher (2019) argues for picking this configuration similar to the arguments in section C.1. DIB will force both distributions to become exactly the same, which would turn the decoder into a permutation matrix for categorical variables.
D Decoder Uncertainty H[Y | Z]
D.1 Cross-entropy loss
The cross-entropy loss features prominently in section 3.2. We can derive the usual cross-entropy loss for our model by minimizing the Kullback-Leibler divergence between the empirical sample distribution p̂(x, y) and the parameterized distribution pθ(x) pθ(ŷ | x). For discriminative models, we are only interested in pθ(ŷ | x), and can simply set pθ(x) = p̂(x):
arg min θ DKL(p̂(x, y) || pθ(x) pθ(Ŷ = y | x))
= arg min θ DKL(p̂(y | x) || pθ(Ŷ = y | x)) + DKL(p̂(x) || pθ(x))︸ ︷︷ ︸ =0
= arg min θ H(p̂(y | x) || pθ(Ŷ = y | x)) − H[Y | X]︸ ︷︷ ︸ const.
= arg min θ H(p̂(y | x) || pθ(Ŷ = y | x)).
In section 3.2, we introduce the shorthand Hθ[Y | X] for H(p̂(y | x) || pθ(Ŷ = y | x)) and refer to it as Prediction Cross-Entropy.
D.2 Upper bounds & training error minimization
To motivate that H[Y | Z] (or Hθ[Y | Z]) can be used as main loss term, we show that it can bound the (training) error probability since accuracy is often the true objective when machine learning models are deployed on real-world problems10.
Proposition. The Decoder Cross-Entropy provides an upper bound on the Decoder Uncertainty: H[Y | Z] ≤ H[Y | Z] + DKL(p(y | z) || pθ(ŷ | z)) = Hθ[Y | Z],
and further bounds the training error:
p(“Ŷ is wrong”) ≤ 1 − e−Hθ[Y|Z] = 1 − e−(H[Y|Z]+DKL(p(y|z)||pθ(ŷ|z))). Likewise, for the Prediction Cross-Entropy Hθ[Y | X] and the Label Uncertainty H[Y | X].
Proof. The upper bounds for Decoder Uncertainty H[Y | Z] and Label Uncertainty H[Y | X] follow from the non-negativity of the Kullback-Leibler divergence, for example:
0 ≤ DKL(p(y | z) || pθ(ŷ | z)) = Hθ[Y | Z] − H[Y | Z], 0 ≤ DKL(p̂(y | x) || pθ(ŷ | x)) = Hθ[Y | X] − H[Y | X].
The derivation for the training error probability is as follows: p(“Ŷ is correct”) = Ep̂(x,y) p(“Ŷ is correct” | x, y) = Ep̂(x,y)Epθ(z|x) pθ(Ŷ = y | z)
= Ep(y,z) pθ(Ŷ = y | z). We can then apply Jensen’s inequality using convex h (x) = − ln x:
h ( Ep(y,z) pθ(Ŷ = y | z) ) ≤ Ep(y,z) h ( pθ(Ŷ = y | z) ) 10As we only take into account the empirical distribution p̂(x, y) available for training, the following derivation
refers only to the empirical risk, and not to the expected risk of the estimator Ŷ.
⇔ p(“Ŷ is correct”) ≥ e−H(p(y|z)||pθ(Ŷ=y|z))
⇔ p(“Ŷ is wrong”) ≤ 1 − e−Hθ[Y|Z]. For small Hθ[Y | Z], we note that one can use the approximation ex ≈ 1 + x to obtain:
p(“Ŷ is wrong”) / Hθ[Y | Z]. (38) Finally, we split the Decoder Cross-Entropy into the Decoder Uncertainty and a Kullback-Leibler divergence:
Hθ[Y | Z] = H[Y | Z] + DKL(p(y | z) || pθ(Ŷ = y | z)). If we upper-bound DKL(p(y | z) ||pθ(Ŷ = y | z)), minimizing the Decoder Uncertainty H[Y | Z] becomes a sensible minimization objective as it reduces the probability of misclassification.
We can similarly show that the training error is bounded by the Prediction Cross-Entropy Hθ[Y | X].
In the next section, we examine categorical Z for which optimal decoders can be constructed and DKL(p(y | z) || pθ(Ŷ = y | z)) becomes zero.
E Categorical Z
For categorical Z, p(y | z) can be computed exactly for a given encoder pθ(z | x) by using the empirical data distribution, which, in turn, allows us to compute H[Y | Z]11. This is similar to computing a confusion matrix between Y and Z but using information content instead of probabilities.
Moreover, if we set pθ(ŷ | z) := p(Y = ŷ | z) to have an optimal decoder, we obtain equality in equation (9), and obtain Hθ[Y | X] ≤ Hθ[Y | Z] = H[Y | Z]. If the encoder were also deterministic, we would obtain Hθ[Y | X] = Hθ[Y | Z] = H[Y | Z]. We can minimize H[Y | Z] directly using gradient descent. d dθH[Y | Z] only depends on p(y | z) and d dθ pθ(z | x):
d dθ
H[Y | Z] = Ep(x,z) [
d dθ
[ ln pθ(z | x) ] Ep̂(y|x) h ( p(y | z))] .
Proof. d dθ H[Y | Z] = d dθ Ep(y,z) h ( p(y | z)) = d dθ Ep(x,y,z) h ( p(y | z)) = Ep̂(x,y) ddθ Epθ(z|x) h (p(y | z))
= Epθ(z|x)Ep̂(x,y) d dθ
[ h ( p(y | z))] + h (p(y | z)) d dθ [ ln pθ(z | x) ] = Ep(x,y,z)
d dθ
[ h ( p(y | z))] + h (p(y | z)) d dθ [ ln pθ(z | x) ] .
And now we show that Ep(x,y,z) ddθ [ h ( p(y | z))] = 0:
Ep(x,y,z) d dθ [ h ( p(y | z))] = Ep(y,z) ddθ [h (p(y | z))] = Ep(y,z) −1p(y | z) ddθ p(y | z)
= − ∫
p(y, z) p(y | z) d dθ
p(y | z) dy dz = − ∫ p(z) ∫
d dθ p(y | z) dy dz
= − ∫
p(z) d dθ [ ∫ p(y | z) dy︸ ︷︷ ︸
=1
] dz = 0.
Splitting the expectation and reordering ofEp(x,y,z) h ( p(y | z)) ddθ [ln pθ(z | x)], we obtain the result.
The same holds for Reverse Decoder Uncertainty H[Z | Y] and for the other quantities as can be verified easily.
If we minimize H[Y | Z] directly, we can compute p(y | z) after every training epoch and fix pθ(ŷ | z) := p(Y = ŷ | z) to create the discriminative model pθ(ŷ | x). This is a different perspective on the self-consistent equations from Tishby et al. (2000); Gondek and Hofmann (2003).
11p(y | z) depends on θ through pθ(z | x): p(y | z) = ∑ x p̂(x,y) pθ(z|x)∑ x p̂(x) pθ(z|x) .
0 20 40 60
0.5
1 1.5 N at s/ P ro b ab ili ty
Permutation MNIST
Objective minH[Y |Z] minHθ[Y |Z] minHθ[Y |X] Metric H[Y |Z] Hθ[Y |Z] Hθ[Y |X]
0 20 40 60
Epoch
1
2
CIFAR10
Figure E.1: Decoder Uncertainty, Decoder Cross-Entropy and Prediction Cross-Entropy for Permutation-MNIST and CIFAR-10 with a categorical Z. C = 100 categories are used for Z. We optimize with different minimization objectives in turn and plot the metrics. DKL(p(y | z) || pθ(Ŷ = y | z)) is small when training with Hθ[Y | Z] or H[Y | Z]. When training with Hθ[Y | X] on CIFAR-10, DKL(p(y | z) || pθ(Ŷ = y | z)) remains quite large. We run 8 trials each and plot the median with confidence bounds (25% and 75% quartiles). See section E.1 for more details.
E.1 Empirical evaluation of DKL(p(y | z) || pθ(Ŷ = y | z)) during training
We examine the size of the gap between Decoder Uncertainty and Decoder Cross-Entropy and the training behavior of the two cross-entropies with categorical latent Z on Permutation MNIST and CIFAR-10. For Permutation MNIST (Goodfellow et al., 2013), we use the common fully-connected ReLU 784 − 1024 − 1024 −C encoder architecture, with C = 100 categories for Z. For CIFAR-10 (Krizhevsky et al., 2009), we use a standard ResNet18 model with C many output classes as encoder (He et al., 2016a). See section G for more details about the hyperparameters. Even though a C × 10 matrix and a SoftMax would suffice to describe the decoder matrix pθ(ŷ | z)12, we have found that over-parameterization using a separate DNN benefits optimization a lot. Thus, to parameterize the decoder matrix, we use fully-connected ReLUs C − 1024 − 1024 − 10 with a final SoftMax layer. We compute it once per batch during training and back-propagate into it.
Figure E.1 shows the three metrics as we train with each of them in turn. Our results do not achieve SOTA accuracy on the test set—we impose a harder optimization problem as Z is categorical, and we are essentially solving a hard-clustering problem first and then map these clusters to Ŷ. Results are provided for the training set in order to compare with the optimal decoder.
As predicted, the Decoder Cross-Entropy upper-bounds both the Decoder Uncertainty H[Y | Z] and the Prediction Cross-Entropy in all cases. Likewise, the gap between Hθ[Y | Z] and H[Y | Z] is tiny when we minimize Hθ[Y | Z]. On the other hand, minimizing Prediction Cross-Entropy can lead to large gaps between Hθ[Y | Z] and H[Y | Z], as can be seen for CIFAR-10. Very interestingly, on MNIST Decoder Cross-Entropy provides a better training objective whereas on CIFAR-10 Prediction Cross-Entropy trains lower. Decoder Uncertainty does not train very well on CIFAR-10, and Prediction Cross-Entropy does not train well on Permutation MNIST at all. We suspect DNN architectures in the literature have evolved to train well with cross-entropies, but we are surprised by the heterogeneity of the results for the two datasets and models.
F Surrogates for regularization terms
F.1 Differential entropies
Proposition. After adding zero-entropy noise, the inequality I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] also holds in the continuous case, and we can minimize I[X; Z | Y] in the IB objective by minimizing H[Z | Y] or H[Z], similarly to the DIB objective. We present a formal proof in section F.1. Theorem 2. For random variables A, B, we have
H[A + B] ≥ H[B]. 12For categorical Z, pθ(ŷ | z) is a stochastic matrix which sums to 1 along the Ŷ dimension.
Proof. See Bercher and Vignat (2002, section 2.2).
Proposition 1. Let Y, Z and X be random variables satisfying the independence property Z ⊥ Y |X, and F a possibly stochastic function such that Z = F(X) + , with independent noise satisfying ⊥ F(X), ⊥ Y and H( ) = 0. Then the following holds whenever I[Y; Z] is well-defined.
I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z].
Proof. First, we note that H[Z | X] = H[F(X) + | X] ≥ H[ | X] = H[ ] with theorem 2, as is independent of X, and thus H[Z | X] ≥ 0. We have H[Z | X] = H[Z | X,Y] by the conditional independence assumption, and by the non-negativity of mutual information, I[Y; Z] ≥ 0. Then:
I[X; Z | Y] + H[Z | X]︸ ︷︷ ︸ ≥0 = H[Z | Y]
H[Z | Y] + I[Y; Z]︸ ︷︷ ︸ ≥0 = H[Z]
The probabilistic model from section 2 fulfills the conditions exactly, and the two statements motivate our proposition.
It is important to note that while zero-entropy noise is necessary for preserving inequalities like I[X; Z | Y] ≤ H[Z | Y] ≤ H[Z] in the continuous case, any Gaussian noise will suffice for optimization purposes: we optimize via pushing down an upper bound, and constant offsets will not affect this.
Thus, if we had H[ ] , 0, even though I[X; Z | Y] + H[Z | X] 6≤ H[Z | Y], we could instead use I[X; Z | Y] + H[Z | X] − H[ ] ≤ H[Z | Y] − H[ ]
as upper bound to minimize. The gradients remain the same.
This also points to the nature of differential entropies as lacking a proper point of origin by themselves. We choose one by fixing H[ ]. Just like other literature usually only considers mutual information as meaningful, we consider H[Z | X] − H[ ] as more meaningful than H[Z | X]. However, we can side-step this discussion conveniently by picking a canonical noise as point of origin in the form of zero-entropy noise H[ ] = 0.
F.2 Upper bounds
We derive this result as follows: H[Z | Y] = Ep̂(y) H[Z | y]
≤ Ep̂(y) 12 ln det(2πe Cov[Z | y]) ≤ Ep̂(y) ∑ i 1 2 ln(2πe Var[Zi | y])
≈ Ep̂(y) ∑
i
1 2 ln(2πe V̂ar[Zi | y]),
Theorem 3. Given a k-dimensional random variable X = (Xi)ki=1 with Var[Xi] > 0 for all i, H[X] ≤ 12 ln det(2πe Cov[X])
≤ ∑
i
1 2 ln(2πe Var[Xi]).
Proof. First, the multivariate normal distribution with same covariance is the maximum entropy distribution for that covariance, and thus H[X] ≤ ln det(2πe Cov[X]), when we substitute the differential entropy for a multivariate normal distribution with covariance Cov[X]. Let Σ0 := Cov[X] be the covariance matrix and Σ1 := diag(Var[Xi])i the matrix that only contains the diagonal. Because we add independent noise, Var[Xi] > 0 and thus Σ−11 exists. It is clear that tr(Σ −1 1 Σ0) = k. Then, we can use the KL-Divergence between two multivariate normal distributions N0, N1 with same mean 0 and
covariances Σ0 and Σ1 to show that ln det Σ0 ≤ ln det Σ1: 0 ≤ DKL(N0 || N1) = 12 ( tr(Σ−11 Σ0) − k + ln ( det Σ1 det Σ0 )) ⇔ 0 ≤ 12 ln ( det Σ1 det Σ0 ) ⇔ 12 ln det Σ0 ≤ 1 2 ln det Σ1.
We substitute the definitions of Σ0 and Σ1, and obtain the second inequality after adding k ln(2πe) on both sides.
Theorem 4. Given a k-dimensional real-valued random variable X = (Xi)ki=1 ∈ Rk, we can bound the entropy by the mean squared norm of the latent: E ‖X‖2 ≤ C′ ⇒ H[X] ≤ C, (39) with C′ := ke 2C/k
2πe .
Proof. We begin with the previous bound: H[X] ≤ ∑
i
1 2 ln(2πe Var[Xi]) = k 2 ln 2πe + 1 2 ln ∏ i Var[Xi]
≤ k2 ln 2πe + 1 2 ln  1k ∑ i Var[Xi] k = k2 ln 2πek ∑ i Var[Xi]
≤ k2 ln 2πe k E ‖X‖ 2,
where we use the AM-GM inequality:∏ i Var[Xi]  1 k ≤ 1k ∑ i Var[Xi] and the monotony of the logarithm with:∑ i Var[Xi] = ∑ i E [ X2i ] −E [Xi]2 ≤ ∑ i E [ X2i ] = E ‖X‖2 Bounding using E ‖X‖2 ≤ C′, we obtain H[X] ≤ k2 ln 2πe k C
′ = C, and solving for C′ yields the statement.
This theorem provides justification for the use of lnE ‖Z‖2 as a regularizer, but does not justify the use of E ‖Z‖2 directly. Here, we give two motivations. We first observe that ln x ≤ x − 1 due to ln’s strict convexity and ln 1 = 0, and thus:
H[X] ≤ k2 ln 2πe k E ‖X‖ 2 = k2 ( ln 2πk E ‖X‖ 2 − 1 ) ≤ πE ‖X‖2.
We can also take a step back and remind ourselves that IB objectives are actually Lagrangians, and β in min I[X; Z] − βI[Y; Z] is introduced as Lagrangian multiplier for the constrained objective:
min I[X; Z] s.t. I[Y; Z] ≥ C. We can similarly write our canonical DIB objective H[Y | Z] + β′′H[Z] as constrained objective
min H[Y | Z] s.t. H[Z] ≤ C, and use above statement to find the approximate form min H[Y | Z] s.t. E ‖Z‖2 ≤ C′. Reintroducing a Lagrangian multiplier recovers our reguralized E ‖Z‖2 objective:
min H[Y | Z] + γE ‖Z‖2.
F.3 “Deep Variational Information Bottleneck” and E ‖Z‖2
Alemi et al. (2016) model pθ(z | x) explicitly as multivariate Gaussian with parameterized mean and parameterized diagonal covariance in their encoder and regularize it to become close to N(0, Ik) by
minimizing the Kullback-Leibler divergence DKL(pθ(z | x) || N(0, Ik)) alongside the cross-entropy: min Hθ[Y | Z] + γDKL(p(z | x) || r(z)),
as detailed in section C.2.
We can expand the regularization term to DKL(p(z | x) || N(0, Ik))
= Ep̂(x)Ep(z|x) h ( (2π)− k 2 e− 1 2 ‖Z‖2 | ) − H[Z | X]
= Ep(z) [ k 2 ln(2π) + 1 2 ‖Z‖2 ] − H[Z | X].
After dropping constant terms (as they don’t matter for optimization purposes), we obtain
= 1 2 E ‖Z‖2 − H[Z | X].
When we inject zero-entropy noise into the latent Z, we have H[Z | X] ≥ 0 and thus E ‖Z‖2 − H[Z | X] ≤ E ‖Z‖2. Thus, the E ‖Z‖2 regularizer also upper-bounds DVIB’s regularizer in this case. In particular, we have equality when we use a deterministic encoder. When we inject zero-entropy noise and use a deterministic encoder, we are optimizing the DVIB objective function when we use the E ‖Z‖2 regularizer. In other words, in this particular case, we could reinterpret “min Hθ[Y | Z] + γE ‖Z‖2” as optimizing the DVIB objective from Alemi et al. (2016) if they were using a constant covariance instead of parameterizing it in their encoder. This does not hold for stochastic encoders.
We empirically compare DVIB and the surrogate objectives from section 3.3 in section G.5. In the corresponding plot in figure G.15, we can indeed note thatE ‖Z‖2 and DVIB are separated by a factor of 2 in the Lagrange multiplier.
F.4 Detailed Comparison to CEB, VCEB & DVIB
In Fisher (2019), the introduced CEB objective "min I[X; Z | Y] − γI[Y; Z]" is rewritten to "min γH[Y | Z] + H[Z | Y] − H[Z | X]" similar to the IB objective in proposition 1 in section 3.1. However, these atomic quantities are not separately examined in detail.
Both Alemi et al. (2016) and Fischer (2020) focus on the application of variational approximations to these quantities. Using a slight abuse of notation to denote all variational approximations, we can write the VCEB objective13 (Fischer, 2020) and the DVIB objective (Alemi et al., 2016) more concisely as
VCEB ≡ min θ Hθ[Y | Z] + β′(Hθ[Z | Y] − Hθ[Z | X]),
DVIB ≡ min θ Hθ[Y | Z] + β′′(Hθ[Z] − Hθ[Z | X]). DVIB does not specify how to choose stochastic encoders and picks the variational marginal q(z) to be a unit Gaussian. We relate how this choice of marginal relates to the E ‖Z‖2 surrogate objective in section F.3. Alemi et al. (2016) use VAE-like encoders that output mean and standard deviation for latents that are then sampled from a multivariate Gaussian distribution with diagonal covariance in their experiments. They run experiments on MNIST and on features extracted from the penultimate layer of pretrained models on ImageNet.
While VCEB as introduced in Fisher (2019) is agnostic to the choice of stochastic encoder, Fischer (2020) mention that stochastic encoders can be similar to encoders and decoders in VAEs (Kingma and Welling, 2013) or like in DVIB mentioned above. Both VAEs and DVIB explicitly parameterize the distribution of the latent to sample from it before passing samples to the decoder.
Fischer and Alemi (2020) use an existing classifier architecture to output means for a Gaussian distribution with unit diagonal covariance. They further parameterize the variational approximation for the Reverse Decoder Uncertainty q(y | z) with one Gaussian of fixed variance per class and learn this reverse decoder during training as well. Fischer and Alemi (2020) report results on CIFAR-10 and ImageNet that show good robustness against adversarial attacks without adversarial training, similar to the results in this paper.
13We will not examine the original objective without Lagrange multipliers from Fisher (2019) here.
This specific (and not motivated) instantiation of the VCEB objective in Fischer and Alemi (2020) is similar to the log Var[Z | Y] surrogate objective introduced in section 3.3 with a deterministic encoder and zero-entropy noise injection. However, the latter uses minibatch statistics instead of learning a reverse decoder, trading variational tightness for ease of computation and optimization.
Compared to this prior literature, this paper examines the usage of implicit stochastic encoders (for example when using dropout) and presents three different simple surrogate objectives together with a principled motivation for zero-entropy noise injection, which has a dual use in enforcing meaningful compression and in simplifying the estimation of information quantities. Moreover, multi-sample approaches are examined to differentiate between Decoder Cross-Entropy and Prediction CrossEntropy. In particular, implicit stochastic encoders together with zero-entropy noise and simple surrogates make it easier to use IB objectives in practice compared to using explicitly parameterized stochastic encoders and variational approaches.
F.5 An information-theoretic approach to VAEs
While Alemi et al. (2016) draw a general connection to β-VAEs (Higgins et al., 2016), we can use the insights from this paper to derive a simple VAE objective. Taking the view that VAEs learn latent representations that compress input samples, we can approach them as entropy estimators. Using H[X] + H[Z | X] = H[X | Z] + H[Z], we obtain the ELBO
H[X] = H[X | Z] + H[Z] − H[Z | X] (1) ≤ Hθ[X | Z] + H[Z] − H[Z | X] (2) ≤ Hθ[X | Z] + H[Z]. (40)
We can also put eq. (40) into words: we want to find latent representations such that the reconstruction cross-entropy H[X | Z] and the latent entropy H[Z], which tell us about the length encoding an input sample, become minimal and approach the true entropy as average optimal encoding length of the dataset distribution.
The first inequality (1) stems from introducing a cross-entropy approximation Hθ[X | Z] for the conditional entropy H[X | Z]. The second inequality (2) stems from injection of zero-entropy noise with a stochastic encoder. For a deterministic encoder, we would have equality. We also note that (1) is the DVIB objective for a VAE with β = 1, and (2) is the DIB objective for a VAE.
Finally, we can use one of the surrogates introduced in section 3.3 to upper bound H[Z]. For optimization purposes, we can substitute the L2 activation regularizer E ‖Z‖2 from proposition 4 and obtain as objective
min θ
Hθ[X | Z] +E ‖Z‖2.
It turns out that this objective is examined amongst others in the recently published Ghosh et al. (2019) as a CV-VAE, which uses a deterministic encoder and noise injection with constant variance. The paper derives this objective by noticing that the explicit parameterizations that are commonly used for VAEs are cumbersome, and the actual latent distribution does often not necessarily match the induced distribution (commonly a unit Gaussian) which causes sampling to generate out-of-distribution data. It fits a separate density estimator on p(z) after training for sampling. The paper goes on to then examine other methods of regularization, but also provides experimental results on CV-VAE, which are in line with VAEs and WAEs. The derivation and motivation in the paper is different and makes no use of information-theoretic principles. Our short derivation above shows the power of using the insights from section 3.2 and 3.3 for applications outside of supervised learning.
F.6 Soft clustering by entropyMinimization with Gaussian noise
Consider the problem of minimizing H[Z | Y] and H[Y | Z], in the setting where Z = fθ(X) + ∼ N(0, σ2)—i.e. the embedding Z is obtained by adding Gaussian noise to a deterministic function of the input. Let the training set be enumerated x1, . . . , xn, with µi = fθ(xi). Then the distribution of Z is given by a mixture of Gaussians with the following density, where d(x, µi) := ‖x − µi‖/σ2.
p(z) ∝ 1 n n∑ i=1 exp(−d(z, µi))
Assuming that each xi has a deterministic label yi, we then find that the conditional distributions p(y | z) and p(z | y) are given as follows:
p(z | y) ∝ 1 ny ∑ i:yi=y exp(−d(z, µi))
p(y | z) = ∑ i:yi=y p(µi | z) = ∑ i:yi=y p(z | µi) p(µi) p(z)
= ∑ i:yi=y p(z | µi)∑n k=1 p(z | µk) = ∑ i:yi=y exp(−d(z, µi))∑n k=1 exp(−d(z, µk)) ,
where ny is the number of xi with class yi = y. Thus, the conditional Z|Y can be interpreted as a mixture of Gaussians and Y |Z as a Softmax marginal with respect to the distances between Z and the mean embeddings. We observe that H[Z | Y] is lower-bounded by the entropy of the random noise added to the embeddings:
H[Z | Y] ≥ H[ fθ(X) + | Y] ≥ H[ ]
with equality when the distribution of fθ(X)|Y is deterministic – that is fθ is constant for each equivalence class.
Further, the entropy H[Y | Z] is minimized when H[Z] is large compared to H[Z | Y] as we have the decomposition
H[Y | Z] = H[Z | Y] − H[Z] + H[Y].
In particular, when fθ is constant over equivalence classes of the input, then H[Y | Z] is minimized when the entropy H[ fθ(X) + ] is large – i.e. the values of fθ(xi) for each equivalence class are distant from each other and there is minimal overlap between the clusters. Therefore, the optima of the information bottleneck objective under Gaussian noise share similar properties to the optima of geometric clustering of the inputs according to their output class.
To gain a better understanding of local optimization behavior, we decompose the objective terms as follows:
H[Z | Y] = Ep̂(y) H(p(z | y) || p(z | y)) = Ep̂(x,y) H(p(z | x) || p(z | y)) = Ep̂(x,y) DKL(p(z | x) || p(z | y)) + H[Z | x] = Ep̂(x,y) DKL(p(z | x) || p(z | y))
+ H[Z | X]︸ ︷︷ ︸ =const .
To examine how the mean embedding µk of a single datapoint xk affects this entropy term, we look at the derivative of this expression with respect to µk = fθ(xk). We obtain:
d dµk H[Z | Y] = d dµk H[Z | yk]
= d
dµk Ep(x|yk) DKL(p(z | x) || p(z | y))
= ∑
i,i:yi=yk
1 nyk d dµk DKL(p(z | xi) || p(z | yk))
+ 1
nyk d dµk DKL(p(z | xk) || p(z | yk)).
While these derivatives do not have a simple analytic form, we can use known properties of the KL divergence to develop an intuition on how the gradient will behave. We observe that in the left-hand sum µk only affects the distribution of Z|Y (that is we are differentiating a sum of terms that look like a reverse KL), whereas it has greater influence on p(z | xk) in the right-hand term, and so its gradient will more closely resemble that of the forward KL. The left-hand-side term will therefore push µk towards the centroid of the means of inputs mapping to y, whereas the right-hand side term is mode-seeking.
F.7 A note on differential and discrete entropies
The mutual information between two random variables can be defined in terms of the KL divergence between the product of their marginals and their joint distribution. However, the KL divergence is only well-defined when the Radon-Nikodym derivative of the density of the joint with respect to the product exists. Mixing continuous and discrete distributions—and thus differential and continuous entropies—can violate this requirement, and so lead to negative values of the “mutual information”. This is particularly worrying in the setting of training stochastic neural networks, as we often assume that an stochastic embedding is generated as a deterministic transformation of an input from a finite dataset to which a continuous perturbation is added. We provide an examples where naive computation without ensuring that the product and joint distributions of the two random variables have a well-defined Radon-Nikodym derivative yields negative mutual information.
Let X ∼ U([0, 0.1]), Z = X + R with R ∼ U({0, 1}). Then I[X; Z] = H[X] = log 110 ≤ 0. Generally, given X as above and an invertible function f such that Z = f (X), I[X; Z] = H[X] and can thus be negative. In a way, these cases can be reduced to (degenerate) expressions of the form I[X; X] = H[X].
We can avoid these cases by adding independent continuous noise.
These examples show that not adding noise can lead to unexpected results. While they still yield finite quantities that bear a relation to the entropies of the random variables, they violate some of the core assumptions we have such that mutual information is always positive.
G Experiment details
G.1 DNN architectures and hyperparameters
For our experiments, we use PyTorch (Paszke et al., 2019) and the Adam optimizer (Kingma and Ba, 2014). In general, we use an initial learning rate of 0.5 × 10−3 and multiply the learning rate by √ 0.1 whenever the loss plateaus for more than 10 epochs for CIFAR-10. For MNIST and Permutation MNIST, we use an initial learning rate of 10−4 and multiply the learning rate by 0.8 whenever the loss plateaus for more than 3 epochs.
Sadly, we deviate from this in the following experiments: when optimizing the decoder uncertainty for categorical Z for CIFAR-10, we used 5 epochs patience for the decoder uncertainty objective and a initial learning rate of 10−4. We do not expect this difference to affect the qualitative results mentioned in section E when comparing to other objectives. We also only used 5 epochs patience when comparing the two cross-entropies on CIFAR-10 in section 3.2. As this was used for both sets of experiments, it does not matter.
We train the experiments for creating the information plane plots for 150 epochs. The toy experiment (figure 3) is trained for 20 epochs. All other experiments train for 100 epochs.
We use a batchsize of 128 for most experiments. We use a batchsize of 32 for comparing the crossentropies for CIFAR-10 (where we take 8 dropout samples each), and a batchsize of 16 for MNIST (where we take 64 dropout samples each).
For MNIST, we use a standard dropout CNN, following https://github.com/pytorch/ examples/blob/master/mnist/main.py. For Permutation MNIST, we use a fully-connected model (for experiments with categorical Z in section E): 784 × 1024 × 1024 ×C. For CIFAR-10, we use a regular deterministic ResNet18 model (He et al., 2016a) for the experiments in section E. (As the model outputs a categorical distribution it becomes stochastic through that and we don’t need stochasticity in the weights.) For the other experiments as well as the Imagenette experiments, we use a ResNet18v2 (He et al., 2016b). When we need a stochastic model for CIFAR-10 (for continuous Z), we add DropConnect (Wan et al., 2013b) with rate 0.1 to all but the first convolutional layers and dropout with rate 0.1 before the final fully-connected layer. Because of memory issues, we reuse the dropout masks within one batch. The model trains to 94% accuracy on CIFAR-10.
For CIFAR-10, we always remove the maximum pooling layer and change the first convolutional layer to have kernel size 3 with stride 1 and padding 1. We also use dataset augmentation during training, but not during evaluation on the training set and test set for purposes of computing metrics. We crop randomly after the padding the training images by 4 pixels in every direction and randomly flip images horizontally.
We generally sample 30 values of γ for the information plane plots from the specified ranges, using a log scale. For the ablation studies mentioned below, we sample 10 values of γ each. We always sample γ = 0 separately and run a trial with it.
Baselines were tuned by hand (without regularization) using grad-student descent and small grid searches.
G.2 Cluster setup & used resources
We make use of a local SLURM cluster (Jette et al., 2002). We run our experiments on GPUs (Geforce RTX 2080 Ti). We estimate reproducing all results would take 94 GPU days.
G.3 Comparison of the surrogate objectives
0
0.5
1
1.5
2
H θ [Y |Z
] I[ Y
;X |Z
]
log Var [Z] log Var [Z | Y ] EZ2
T rain S et
50 100 300
0
0.5
1
1.5
2
50 100 300 50 100 300
Preserved Information I[X;Z]
T est S et
Figure G.1: Information Plane Plot of the latent Z similar to Tishby and Zaslavsky (2015) but using a ResNet18 model on CIFAR-10 using the different regularizes from section 3.3 (without dropout, but with zero-entropy noise). The dots are colored by γ. See section 4 for more details.
As can be seen in figure G.2, the different surrogate regularizers have very similar effects on H[Z] and H[Z | Y]. Regularizing with E ‖Z‖2 shows a stronger initial regularization effect, but is difficult to compare quantitatively as its hyperparameter does not map to an equivalent β, unlike regularizing using entropy estimates. Overall, we find log Var[Z] to provide stable training trajectories (and expected visualizations) while also having a more meaningful hyperparameter than E ‖Z‖2, though E ‖Z‖2 is trivial to implement and communicate14. log Var[Z | Y] performs worse which we hypothesize is due to the increased variance (given equal batch sizes) from conditioning on Y. It further does not minimize the Preserved Information I[X; Z] as strongly as the other regularizers.
G.3.1 Measurement of information quantities
Measuring information quantities can be challenging. As mentionend in the introduction, there are many complex ways of measuring entropies and mutual information terms. We can side-step the issue by making use of the bounds we have established and the zero-entropy noise we are injecting, and design experiments around that.
First, to estimate the Preserved Information I[X; Z], we note that when we use a deterministic model as encoder and only inject zero-entropy noise, we have H[Z | X] = 0 and I[X; Z] = I[X; Z] + H[Z | X] = H[Z]. We use the entropy estimator from Kraskov et al. (2004, equation (20)) to estimate the Encoding Entropy H[Z] and thus I[X; Z].
14Which is the reason why we showcase it in figure 1 and in equation (1).
100
150
200
250
log Var [Z]
γ 10−6 10−4 10−2 1
log Var [Z | Y ] EZ2 H
[Z ]
0 50 100 150
500
1000
1500
2000
0 50 100 1500 50 100 150
Epoch
E Z
2
Figure G.2: Entropy estimates while training with different γ and with different surrogate regularizers on CIFAR-10 with a ResNet18 model. Entropies are estimated on training data based on Kraskov et al. (2004). Qualitatively all three regularizers push H[Z] and H[Z | Y] down. H[Z | Y] is not shown here because it always stays very close to H[Z]. E ‖Z‖2 tends to regularize entropies more strongly for small γ. See section 4 for more details.
To estimate the Residual Information I[X; Y | Z], we similarly note that I[X; Y | Z] = I[X; Y | Z] + H[Y | X] = H[Y | Z]. Instead of estimating the entropy using Kraskov et al. (2004), we can use the Decoder Cross-Entropy Hθ[Y | Z] which provides a tighter bound as long as we also minimize Hθ[Y | Z] as part of the training objective. When we use stochastic models as encoder, we cannot easily compute I[X; Z] anymore. In the ablation study in the next section, we thus change the X axis accordingly.
Similarly, when we look at the trajectories on the test set instead of the training set, for example in figure G.4, we change the Y axis to signify the Decoder Uncertainty Hθ[Y | Z]. It is still an upper-bound, but we do not minimize it directly anymore.
For the plots in figure 6, we retrained the decoder on the test set to obtain a tighter bound on H[Y | Z] (while keeping the encoder fixed). We then sampled the latent using the test set to estimate the trajectories. We only did this for the CIFAR-10 model without dropout. For our ablations, we did not retrain the decoder and thus only present plots on the test and training set, respectively.
At this point, it is important to recall that the Decoder Uncertainty is also the negative log-likelihood (when training with a single dropout sample), which provides a different perspective on the plots. It makes it clear that we can see how much a model overfits by comparing the best and final epochs of a trajectory in the plot (marked by a circle and a square, respectively).
G.3.2 Ablation study
We perform an ablation study to determine whether injecting noise is necessary. Furthermore, we investigate the more interesting case of using a stochastic model as encoder, and if we can use a stochastic model without injecting zero-entropy noise.
We also investigate whether log Var[Z | Y] performs better when we increase batchsize as we hypothesized that a batchsize of 128 does not suffice as it leaves only ≈ 13 samples per class to approximate H[Z | Y]). Figure G.3 shows a larger version of figure 6 for all three regularizers and also training trajectories on the test set. As described in the previous section, this allows us to validate that the regularizers prevent overfitting on the training set: with increasing γ, the model overfits less.
Figure G.6 and figure G.5 shows that injecting noise is necessary independently of whether we use dropout or not. Regularizing with E ‖Z‖2 still has a very weak effect. We hypothesize that, similar to
the toy experiment depicted in figure 3, floating-point precision issues might provide a natural noise source eventually. This would change the effectiveness of γ and might require much higher values to observe similar regularization effects as when we do inject zero-entropy noise.
Figure G.4 shows trajectories for a stochastic encoder (as described above with DropConnect/dropout rate 0.1). It overfits less than a deterministic one.
Figure G.7 shows the effects of using higher dropout rates (using DropConnect/dropout rates of 0.3/0.5). It overfits less than model with DropConnect/dropout rates of 0.1/0.1.
The plots in figure G.10 show the effects of different γ with different regularizers more clearly. On both training and test set, one can clearly see the effects of regularization.
Overall, log Var[Z | Y] performs worse as a regularizer. In figure G.8, we compare the effect of doubling batchsize. Indeed, log Var[Z | Y] performs better with higher batchsize and looks closer to log Var[Z].
G.3.3 Comparison between Decoder Cross-Entropy and Prediction Cross-Entropy
When training deterministic models or dropout models with a single sample (as one usually does), the estimators for both the Decoder Cross-Entropy Hθ[Y | Z] and the Prediction Cross-Entropy Hθ[Y | X] coincide. In section 3.2, we discuss the differences from a theoretical perspective. Here, we empirically evaluate the difference between optimizing the estimators for each of the two crossentropy losses, for which we will draw multiple dropout samples during training and inference.
We examine models with continuous Z on MNIST and CIFAR-10 (Lecun et al., 1998; Krizhevsky et al., 2009). Specifically, we use a standard dropout CNN as an encoder for MNIST, and a modified ResNet18 to which we add DropConnect in each layer for CIFAR-10. We use K = 100 dimensions for the continuous latent Z in the last fully-connected layer, and use a linear decoder to obtain the final 10-dimensional output of class logits. For MNIST, we compute the cross-entropies using 64 dropout samples; for CIFAR-10, we use 8. For the purpose of this examination of training behavior, it is not necessary to achieve SOTA accuracy: our models obtain 99.2% accuracy on MNIST and 93.6% on CIFAR-10.
Figure G.11 shows the training error probability as well as the value of each cross-entropy loss for models trained either with the Decoder Cross-Entropy or the Prediction Cross-Entropy. The Decoder Cross-Entropy Hθ[Y | Z] outperforms Prediction Cross-Entropy Hθ[Y | X] as a training objective: the training error probability and both cross-entropies are lower when minimizing Hθ[Y | Z] compared to minimizing Hθ[Y | X]. We compare only the training, rather than the test, losses of the models to isolate the effect of each loss term on training performance; we leave the prevention of overfitting to the regularization terms considered later. Recently, Dusenberry et al. (2020) also observed empirically that the Decoder Cross-Entropy Hθ[Y | Z] as an objective is both easier to optimize and provides better generalization performance.
G.4 Differential entropies and noise
We demonstrate the importance of adding noise to continuous latents by constructing a pathological sequence of parameters which attain monotonically improving and unbounded regularized objective values (H[Z]) while all computing the same function. We use MNIST with a standard dropout CNN as encoder, with K = 128 continuous dimensions in Z, and a K × 10 linear layer as decoder. After every training epoch, we decrease the entropy of the latent by normalizing and then scaling the latent to bound the entropy. We multiply the weights of the decoder to not change the overall function. As can be seen in figure 3, without noise, entropy can decrease freely during training without change in error rate until it is affected by floating-point issues; while when adding zero-entropy noise, the error rate starts increasing gradually and meaningfully as the entropy starts to approach zero. We conclude that entropy regularization is meaningful only when noise is added to the latent.
G.5 Comparison between DVIB and surrogate objectives on Permutation-MNIST
Comparing DVIB and our surrogate objectives is not straightforward because DVIB uses a VAE-like model that explicitly parameterize mean and standard deviation of the latent whereas the stochastic models we focus on in section 3.2 and beyond are implicit by using dropout.
For this comparison, we use the same architecture and optimization strategy for DVIB as described in Alemi et al. (2016): the encoder is a ReLU-MLP of the form 796 − 1024 − 1024 − 2K with K=256 latent dimensions that outputs mean and standard deviation explicitly and separately. For the standard deviation, we use a softplus transform with a bias of −5. We use Polyak averaging with a decay constant of 0.999 (Polyak and Juditsky, 1992). We train the model for 200 epochs with Adam with learning rate 10−4, β1 = 0.5, β2 = −.999 (Kingma and Ba, 2014) and decay the learning rate by 0.97 every 2 epochs. The marginal is fixed to a unit Gaussian around the origin. We use a softmax layer as decoder. We use 12 latent samples during training and test time.
For our surrogate objectives, we use a similar ReLU-MLP of the form 796 − 1024 − 1024 − K with K=256 latent dimensions and dropout layers of rate 0.3 after the first and second layer. We use also 12 dropout samples during training and test time. We train for 75 epochs with Adam and learning rate 0.5 × 10−4. We half the learning rate every time the loss does not decrease for 13 epochs. We run 5 trials for each experiments. We were not able to reproduce the baseline of an error of 1.13% for β = 10−3 from Alemi et al. (2016). We show a comparison in figure G.15. Our methods do reach an error of 1.13% overall though, so the simpler surrogate objectives perform as well good or better than DVIB.
From section F.3, we know that DVIB’s β would have to be twice the γ frm our section 3.3. We can see this correspondence in the plot. This also implies that DVIB’s β is not related to the IB objective’s β from section 3. This makes sense as DVIB arbitrarily fixes the marginal to be a unit Gaussian.
1 0 −
4
1 0 −
3
1 0 −
2
1 0 −
11
DecoderCross-EntropyHθ[Y|Z]ResidualInformationI[X;Y|Z]
lo g
V a r
[Z ]
γ 1 0 −
6 1 0 −
4 1 0 −
2 1
E p
o ch
0 5 0
1 0 0
1 5 0
lo g
V a r
[Z |Y
] E Z
2 W
ei g
h t
D ec
ay
Train Set
0 .31 3 .0
Test Set
5 0
1 0
0 3
0 0
0 .3 0 .51
5 0
1 0
0 3
0 0
5 0
1 0
0 3
0 0
5 0
1 0
0 3
0 0
P re
se rv
ed In
fo rm
at io
n I[ X
;Z ]
IP
Fi gu
re G
.3 :W
ith ou
td ro
po ut
bu tw
ith ze
ro -e
nt ro
py no
is e:
In fo
rm at
io n
P la
ne P
lo to
ft ra
in in
g tr
aj ec
to ri
es fo
r R
es N
et 18
m od
el s
on C
IF A
R -1
0 an
d di
ff er
en tr
eg ul
ar iz
er s.
Th e
tra je
ct or
ie s
ar e
co lo
re d
by th
ei rr
es pe
ct iv
e γ
;t he
ir tra
ns pa
re nc
y ch
an ge
s by
ep oc
h. C
om pr
es si
on (P
re se
rv ed
In fo
rm at
io n ↓)
tra de
soff
w ith
pe rf
or m
an ce
(R es
id ua l In fo rm at io n ↓) .S ee se ct io n 4. T he ci rc le m ar ks th e fin al ep oc h of a tr aj ec to ry .T he sq ua re m ar ks th e be st ep oc h (R es id ua lI nf or m at io n ).
1 0 −
2
1 0 −
11
DecoderCross-EntropyHθ[Y|Z]ResidualInformationI[Y;X|Z]
lo g
V a r
[Z ]
γ 1 0 −
6 1 0 −
4 1 0 −
2 1
E p
o ch
0 5 0
1 0 0
1 5 0
lo g
V a r
[Z |Y
] E Z
2
Train Set
5 0
1 0
0 3
0 0
0 .3 0 .51
1 0
0 2
0 0
3 0
0 5
0 1
0 0
3 0
0
E n
co d
in g
E n
tr o
py H
[Z ]
Test Set
Fi gu
re G
.4 :W
ith dr
op ou
ta nd
w ith
ze ro
-e nt
ro py
no is
e: In
fo rm
at io
n P
la ne
P lo
to ft
ra in
in g
tr aj
ec to
ri es
fo r
R es
N et
18 m
od el
s on
C IF
A R
-1 0
an d
di ff
er en
tr eg
ul ar
iz er s. Th e tra je ct or ie s ar e co lo re d by th ei rr es pe ct iv e γ ;t he ir tra ns pa re nc y ch an ge s by ep oc h. C om pr es si on (P re se rv ed In fo rm at io n ↓) tra de soff w ith pe rf or m an ce (R es id ua l In fo rm at io n ↓) .S ee se ct io n 4. T he ci rc le m ar ks th e fin al ep oc h of a tr aj ec to ry .T he sq ua re m ar ks th e be st ep oc h (R es id ua lI nf or m at io n ).
1 0 −
2
1 0 −
11
DecoderCross-EntropyHθ[Y|Z]ResidualInformationI[Y;X|Z]
lo g
V a r
[Z ]
γ 1 0 −
6 1 0 −
4 1 0 −
2 1
E p
o ch
0 5 0
1 0 0
1 5 0
lo g
V a r
[Z |Y
] E Z
2
Train Set
-4 0
0 -2
0 0
0 2
0 0
0 .3 0 .51
-4 0
0 -2
0 0
0 2
0 0
-1 5
0 0
-1 0
0 0
-5 0
0 0
E n
co d
in g
E n
tr o
py H
[Z ]
Test Set
Fi gu
re G
.5 :W
ith dr
op ou
tb ut
w ith
ou tz
er o-
en tro
py no
is e:
In fo
rm at
io n
P la
ne P
lo to
ft ra
in in
g tr
aj ec
to ri
es fo
r R
es N
et 18
m od
el s
on C
IF A
R -1
0 an
d di
ff er
en tr
eg ul
ar iz
er s.
Th e
tra je
ct or
ie s
ar e
co lo
re d
by th
ei rr
es pe
ct iv
e γ
;t he
ir tra
ns pa
re nc
y ch
an ge
s by
ep oc
h. C
om pr
es si
on (P
re se
rv ed
In fo
rm at
io n ↓)
tra de
soff
w ith
pe rf
or m
an ce
(R es
id ua l In fo rm at io n ↓) .S ee se ct io n 4. T he ci rc le m ar ks th e fin al ep oc h of a tr aj ec to ry .T he sq ua re m ar ks th e be st ep oc h (R es id ua lI nf or m at io n ).
1 0 −
4
1 0 −
3
1 0 −
2
1 0 −
11
DecoderCross-EntropyHθ[Y|Z]ResidualInformationI[Y;X|Z]
lo g
V a r
[Z ]
γ 1 0 −
6 1 0 −
4 1 0 −
2 1
E p
o ch
0 5 0
1 0 0
1 5 0
lo g
V a r
[Z |Y
] E Z
2
Train Set
-5 0
0 -2
5 0
0 2
5 0
0 .3 0 .51
-4 0
0 -2
0 0
0 2
0 0
-2 0
0 0
-1 5
0 0
-1 0
0 0
-5 0
0 0
P re
se rv
ed In
fo rm
at io
n I[ X
;Z ]
Test Set
Fi gu
re G
.6 :
W ith
ou td
ro po
ut an
d w
ith ou
tz er
oen
tr op
y no
is e:
In fo
rm at
io n
P la
ne P
lo to
ft ra
in in
g tr
aj ec
to ri
es fo
r R
es N
et 18
m od
el s
on C
IF A
R -1
0 an
d di
ff er
en t
re gu
la ri
ze rs
. T
he tr
aj ec
to ri
es ar
e co
lo re
d by
th ei
r re
sp ec
tiv e γ
; th
ei r
tr an
sp ar
en cy
ch an
ge s
by ep
oc h.
C om
pr es
si on
(P re
se rv
ed In
fo rm
at io
n ↓)
tr ad
es -o
ff w ith pe rf or m an ce (R es id ua lI nf or m at io n ↓) .S ee se ct io n 4. T he ci rc le m ar ks th e fin al ep oc h of a tr aj ec to ry .T he sq ua re m ar ks th e be st ep oc h (R es id ua lI nf or m at io n ).
0 .0
3
1 0 −
1 0 .31
DecoderCross-EntropyHθ[Y|Z]ResidualInformationI[Y;X|Z]
lo g
V a r
[Z ]
γ 1 0 −
6 1 0 −
4 1 0 −
2 1
E p
o ch
0 5 0
1 0 0
1 5 0
lo g
V a r
[Z |Y
] E Z
2
Train Set
5 0
1 0
0 3
0 0
0 .51 2 .0
1 0
0 2
0 0
3 0
0 5
0 1
0 0
3 0
0
E n
co d
in g
E n
tr o
py H
[Z ]
Test Set
Fi gu
re G
.7 :
W ith
m or
e dr
op ou
ta nd
ze ro
-e nt
ro py
no is
e: In
fo rm
at io
n P
la ne
P lo
to ft
ra in
in g
tr aj
ec to
ri es
fo r
R es
N et
18 m
od el
s on
C IF
A R
-1 0
an d
lo g
V ar
[Z |Y ] re gu la ri ze r w ith ba tc hs iz es 12 8 an d 25 6. Th e tra je ct or ie s ar e co lo re d by th ei rr es pe ct iv e γ ;t he ir tra ns pa re nc y ch an ge s by ep oc h. C om pr es si on (P re se rv ed In fo rm at io n ↓) tr ad es -o ff w ith pe rf or m an ce (R es id ua lI nf or m at io n ↓) .S ee se ct io n 4. T he ci rc le m ar ks th e fin al ep oc h of a tr aj ec to ry .T he sq ua re m ar ks th e be st ep oc h (R es id ua l In fo rm at io n ). A D ro pC on ne ct ra te of 0. 3 an d dr op ou tr at e of 0. 4 w er e us ed in st ea d of 0. 1 fo re ac h.
1e-4
1e-3
1e-2
1e-1
1
D ec
o d
er C
ro ss
-E n
tr o
py H θ [Y |Z
] R
es id
u al
In fo
rm at
io n
I[ Y
;X |Z
]
log Var [Z | Y ] (Batchsize 128)
γ 10−5 10−3 10−1 101 Epoch 0 50 100 150
log Var [Z | Y ] (Batchsize 256)
T rain S et
50 100 300
0.3
1
3
100 200 300
Preserved Information I[X;Z]
T est S et
Figure G.8: Without dropout but with zero-entropy noise: Information Plane Plot of training trajectories for ResNet18 models on CIFAR-10 and log Var[Z | Y] regularizer with batchsizes 128 and 256. The trajectories are colored by their respective γ; their transparency changes by epoch. Compression (Preserved Information ↓) trades-off with performance (Residual Information ↓). See section 4. The circle marks the final epoch of a trajectory. The square marks the best epoch (Residual Information ).
0
0.5
1
1.5
2
H θ [Y |Z
] I[ Y
;X |Z
]
log Var [Z] log Var [Z | Y ] EZ2
T rain S et
50 100 300
0
0.5
1
1.5
2
50 100 300 50 100 300
Encoding Entropy H[Z]
T est S et
Figure G.9: Information Plane Plot of the latent Z similar to Tishby and Zaslavsky (2015) but using a ResNet18 model on CIFAR-10 using the different regularizes from section 3.3 (with dropout and zero-entropy noise). The dots are colored by γ. See section 4 for more details.
0.01
0.1
1
10 100 N at s
log Var [Z]
Information Quantities Decoder Cross-Entropy Hθ[Y |Z] Encoding Entropy H[Z]
log Var [Z | Y ] EZ2
T rain S et
10−4 10−2 1
1
10
100
10−4 10−2 1 10−5 10−3 10−1 101
γ
T est S et
(a) With dropout and zero-entropy noise.
0.001
0.1
10
N at
s
log Var [Z]
Information Quantities Decoder Cross-Entropy Hθ[Y |Z] Preserved Information I[X;Z]
log Var [Z | Y ] EZ2
T rain S et
10−4 10−2 1
1
10
100
10−4 10−2 1 10−5 10−3 10−1 101
γ
T est S et
(b) Without dropout but with zero-entropy noise.
-2000
-1500
-1000
-500
0
N at
s
log Var [Z]
Information Quantities Decoder Cross-Entropy Hθ[Y |Z] Preserved Information I[X;Z]
log Var [Z | Y ] EZ2
T rain S et
10−4 10−2 1
-1500
-1000
-500
0
10−4 10−2 1 10−5 10−3 10−1
γ
T est S et
(c) Without dropout and without zero-entropy noise.
Figure G.10: Information quantites for different γ at the end of training for ResNet18 models on CIFAR-10 and log Var[Z | Y] regularizer with batchsizes 128 and 256. Compression (Preserved Information ↓) trades-off with performance (Residual Information ↓). See section 4.
1e-4
1e-3
1e-2
1e-1
N at
s/ E
rr or
Test Set Objective minHθ[Y |Z] minHθ[Y |X]Metric Ep̂(x,y)p(ŷ 6= y) Hθ[Y |Z] Hθ[Y |X] Train Set
M N IS T
1e-6
1e-4
1e-2
P erm u tatio n M N IS T
0 25 50 75 100
0.001
0.01
0.1
1
0 25 50 75 100
Epoch
C IFA R 1 0
Figure G.11: Training error probability, Decoder Cross-Entropy Hθ[Y | Z] and Prediction CrossEntropy Hθ[Y | X] with continuous Z. K = 100 dimensions are used for Z, and we use dropout to obtain stochastic models. Minimizing Hθ[Y | Z] (solid) leads to smaller cross-entropies and lower training error probability than minimizing Hθ[Y | X] (dashed). This suggests a better data fit, which is what we desire for a loss term. We run 8 trials each and plot the median with confidence bounds (25% and 75% quartiles). See section 3.2 and G.3.3 for more details.
0.03
10−1
0.3
1
D ec
o d
er C
ro ss
-E n
tr o
py H θ [Y |Z
] R
es id
u al
In fo
rm at
io n
I[ Y
;X |Z
]
EZ2
T rain S et
γ 10−6 10−4 10−2 1
Epoch 0 50 100 150
70 100 200
Encoding Entropy H[Z]
0.3
0.5
1
T est S et
Imagenette
Figure G.12: Information Plane Plot of the latent Z similar to Tishby and Zaslavsky (2015) but using a ResNet18v2 model on Imagenette using the E ‖Z‖2 surrogate obejctive from section 3.3 (with dropout and zero-entropy noise). The dots are colored by γ.
0
0.25
0.50
0.75
1
A cc
u ra
cy log Var [Z]
γ 10−6 10−4 10−2 1
log Var [Z | Y ] EZ2 Weight Decay B
asicIterative
0
0.25
0.50
0.75
1
D eep F o o l
0
0.25
0.50
0.75
1
F G S M
0
0.25
0.50
0.75
1
P G D
0.001 0.01 0.1 1
0
0.25
0.50
0.75
1
0.001 0.01 0.1 1 0.001 0.01 0.1 1 0.001 0.01 0.1 1
Epsilon
R o b u stn ess
Figure G.13: Adversarial robustness of ResNet18 models trained on CIFAR-10 with surrogate objectives in comparison to regularization with L2 weight-decay as non-IB method for different attack strengths . The robustness is evaluated using FGSM, PGD, DeepFool and BasicIterative attacks of varying values. The dashed black line represents a model trained only with cross-entropy and no noise injection. We see that models trained with the surrogate IB objective (colored by γ) see improved robustness over a model trained only to minimize the cross-entropy training objective (shown in black) while the models regularized with weight-decay actually perform worse.
0
0 .2
5
0 .5
0
0 .7
51
Accuracy
lo g
V a r
[Z ]
γ 1 0 −
6 1 0 −
4 1 0 −
2 1
lo g
V a r
[Z |Y
] E Z
2 W
ei g
h t
D ec
ay BasicIterative
0
0 .2
5
0 .5
0
0 .7
51
DeepFool
0
0 .2
5
0 .5
0
0 .7
51
FGSM
0
0 .2
5
0 .5
0
0 .7
51
PGD
5 0
1 0
0 3
0 0
0
0 .2
5
0 .5
0
0 .7
51
5 0
1 0
0 3
0 0
5 0
1 0
0 3
0 0
5 0
1 0
0 3
0 0
P re
se rv
ed In
fo rm
at io
n I[ X
;Z ]
Robustness
Fi gu
re G
.1 4:
Av er
ag e
ad ve
rs ar
ia lr
ob us
tn es
s ov
er ∈
[0 ,0 .1
]o fR
es N
et 18
m od
el s
tr ai
ne d
on C
IF A
R -1
0 w
ith su
rr og
at e
ob je
ct iv
es in
or L2
w ei
gh t-
de ca
y (a
s no
nIB
m et
ho d)
co m
pa re
d to
no rm
al ac
cu ra
cy fo
r di
ff er
en ta
m ou
nt s
of P
re se
rv ed
In fo
rm at
io n. ◦
m ar
ke rs
sh ow
ro bu
st ne
ss .×
m ar
ke rs
sh ow
th e
no rm
al ac
cu ra
cy .W
e se
e th at ro bu st ne ss de pe nd s on th e Pr es er ve d In fo rm at io n. If th e la te nt is co m pr es se se d to o m uc h, ro bu st ne ss (a nd ac cu ra cy )a re lo w .I ft he la te nt is no tc om pr es se d en ou gh , ro bu st ne ss an d th us ge ne ra liz at io n su ff er .
10−4 10−2 1
γ
10−2
10−1
1
T es
t E
rr or
Regularizer log Var [Z]
log Var [Z | Y ]
EZ2
DVIB
Figure G.15: Comparison of test error for different Lagrange multiplier for DVIB and surrogate objectives from section 3.3 on Permutation-MNIST. The purple strongly dashed line shows the test error reported for DVIB in Alemi et al. (2016). 5 trials with 95% confidence interval shown. Even though we could not reproduce the baseline reported in that paper, the simpler surrogate objective reach at least a similar test error as reported there. We also see that DVIB behaves similar to E ‖Z‖2, but shifted by a factor 2 in γ, as predicted by section F.3.
H L a r g e
V e r si o n o f t h e
M ic k e y
M o u se
ID ia g r a m
H(Y |X) Lab elU
nce rta
inty H(
Z|X )
En co
din g U
nc er
ta int
y
I(Y ;Z ) Pr es er ve d Re le va nt
In fo
rm at
io n
I(X ;Y
|Z )
Re sid
ua l
In fo
rm at
io n
I(X ;Z
|Y )
Re du
nd an t In fo rm at io n
H( Y| Z) De co de r Un ce rta
in ty
H( Z| Y) Re ve rs e De co de r Un ce rta in ty
H( X|
Y, Z)
I(X ;Y ) Re le va
nt
In fo
rm at
io n
I(X ;Z ) Pr es er ve d
In fo
rm at
io n
Fi gu
re H
.1 :M
ic ke
y M
ou se
Idi
ag ra
m .S
ee fig
ur e
2 fo
rd et
ai ls
.

Review summaries:
The authors review the information bottleneck (IB) in the context of deep learning. They discuss the obstacles to applying the IB (and a deterministic variant, the DIB) to modern datasets, review approaches to doing so, and introduce their own scalable approach. Their approach introduces practical surrogate objectives for the information regularizer term, and uses dropout as the source of stochasticity. They take advantage of the scalability of their method to train a ResNet with (D)IB on MNIST, CIFAR-10, and ImageNette and study adversarial robustness and evolution in the information plane.

This paper provides several surrogates for the Information Bottleneck (IB) and Deterministic Information Bottleneck (DIB) loss functions that are more friendly to optimization. For the decoder uncertainty part, the authors show that using Dropout and cross-entropy loss provides an unbiased estimator for the decoder cross-entropy which upperbounds the decoder uncertainty. For the regularization terms in IB/DIB, the authors inject noises to the latent features to lower-bound the conditional entropy of latent representations, and further proposes three types of surrogate objectives for the regularziation terms. Emprical results on CIFAR/ImageNette (a subset of ImageNet of 10 classes) show that the proposed surrogates yield similar behaviours in terms of adversarial robustness and information plane and the scalability of the proposed method.