Unpacking Information Bottlenecks: Surrogate Objectives for Deep Learning
Authors: Andreas Kirsch
Clare Lyle
Yarin Gal
Venue:  (2021)
Keywords: deep learning
information bottleneck
information theory

Abstract: The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as a regularized objective with which to train models. However, multiple competing objectives are proposed in the literature, and the information-theoretic quantities used in these objectives are difficult to compute for large deep neural networks, which in turn limits their use as a training objective. In this work, we review these quantities, compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly to optimization without relying on cumbersome tools such as density estimation. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on MNIST, CIFAR-10 and ImageNette with modern DNN architectures (ResNets).

Under review as a conference paper at ICLR 2021
UNPACKING INFORMATION BOTTLENECKS: SURROGATE OBJEC- TIVES FOR DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The Information Bottleneck principle offers both a mechanism to explain how deep neural networks train and generalize, as well as regularized objective with which to train models. However, multiple competing objectives are proposed the literature, and the information-theorett quantities used in these objectives are difficult to compute for large deep neural networks, which in turn limits their use training objective. In this work, we review these quantities, and compare and unify previously proposed objectives, which allows us to develop surrogate objectives more friendly optimization without relying on cumbersome tools such as density estimation. We find that these surrogate objectives allow us to apply the information bottleneck to modern neural network architectures. We demonstrate our insights on MNIST, CIFAR-10 and Imagenette with modern DNN architectures (ResNets).
INTRODUCTION
The Information Bottleneck (IB) principle, introduced by Tishby et al. (2000), proposes that training and generalization in deep neural networks (DNNs) can be explained by information-theoreti princi- ples (Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017; Achille and Soatto, 2018a). This is attractive as the success of DNNs remains largely unexplained by tools from computational learning theory (Zhang et al., 2016; Bengio et al., 2009). The IB principle suggests that learning consists of two competing objectives: maximizing the mutual information between the latent representation and the label to promote accuracy, while at the same time minimizing the mutual information between the latent representation and the input to promote generalization. Following this principle, many variations of IB objectives have been proposed (Alemi et al., 2016; Strouse and Schwab, 2017: Fischer and Alemi, 2020; Fischer, 2020; Fisher, 2019; Gondek and Hofmann, 2003; Achille and Soatto, 2018a), which, in supervised learning. have been demonstrated benefit robustness to adversarial attacks (Alemi et al., 2016; Fisher, 2019) and generalization and regularization against overfitting to random labels (Fisher, 2019).
Whether the benefits of training with IB objectives are due to the IB principle. or some other unrelated mechanism, remains unclear (Saxe al., 2019; Amjad and Geiger, 2019; Tschannen et al., 2019), suggesting that although recent work has also tied the principle to successful results in both unsupervised and self-supervised learning (Oord et al., 2018; Belghazi et al., 2018; Zhang et al., 2018: Burgess et al., 2018, among others), our understanding of how IB objectives affect representation learning remains unclear.
Critical to studying this question is the computation of the information-theone While progress has been made in developing mutual information estimators for DNNs (Poole et al., 2019; Belghazi et al., 2018; Noshad et al., 2019; McAllester and Stratos, 2018; Kraskov et al., 2004), current methods still face many limitations when concerned with high-dimensional random variables (McAllester and Stratos, 2018) and rely on complex estimators generative models. This presents challenge to training with IB objectives.
quantities' used.
In this paper, analyze information quantities and relate them surrogate objectives for the IB principle which are more friendly to optimization, showing that complex or intractable IB objectives can be replaced with simple, easy-to-compute surrogates that produce similar performance and similar
We shorten these information quantities from on.

---

Under review as a conference paper at ICLR 2021
10-11
10"1
13
Fauth
RY:Z)
Preserved Relevant Information
Y
Z
HEVDO Labet Uncertaing
Infermtion DXX 12
Untertaints
HMYZ) Decoder Uncertainty
(XXY2) Residual Information
10G.ZY) Redundant Information
Reverse
HIZTY
Decoder
Uncertainty
10GZ)
Relevant Information
HIX|Y,ZI ZI
Information
70 220
Excoding Exumps RiA
200
X
Figure I: Information plane plot of the training Figure 2: Mickey Mouse I-diagram for the in- vajectories of ResNet18 models with our surro- formation quantities in the model p(x, y. = gate objective ming HolY [Z] + yI |IZ]I² on Ima-
p(x. pu(z x). X is for the data, Y for the la- bels, Z. for the latent encodings. See section 2 for more details and section B.1 for a description of all the quantities. I[ = I[X; Y: because
genette. Color shows y: transparency the training
epoch. Compression (Encoding Entropy J) trades-
off with test performance (Residual Information 1). See section 4.
I[Y: X] O. Best viewed in color.
behaviour of information quantities over training. Sections 2 & 3 review commonly- information quantities for which provide mathematically grounded intuition via information diagrams and unify different IB objectives by identifying two key information quantities, Decoder Uncertainty H[Y [Z] and Reverse Decoder Uncertainty H[Z| YY which act as the main loss and regularization terms in our unified IB objective. In particular, Section 3.2 demonstrates that using the Decoder Uncertainty as a training objective can minimize the training error, and shows how estimate an upper bound on it efficiently for well-known DNN architectures. We expand on the findings of Alemi al. (2016) in their variational IB approximation and demonstrate that this upper bound is equal to the commonly-us pathologies of differential entropies that hinder optimization and proposes adding Gaussian noise to force differential entropies to become non-negative, which leads to new surrogate terms to optimize the Reverse Decoder Uncertainty. Altogether this leads to simple and tractable surrogate IB objectives such the following, which uses dropout, adds Gaussian noise over the feature vectors f(x; 17), and uses an L2 penalty over the noisy feature vectors:
cross-entropy loss² under dropout regularization. Section 3.3 examines
min Exy-gtxy)ee [- log = =yl== falx; + yllo(xx))) + elE]
(1)
y--rropout mask
Section 4 describes experiments that validate our insights qualitatively and quantitatively on MNIST, CIFAR-10 10 and Imagenette, and shows that with objectives like the one in equation (1) we obtain information plane plots (as in figure 1) similar to those predicted by Tishby and Zaslavsky (2015). Our simple surrogate objectives thus induce the desired behavior of IB objectives while scaling to large, high-dimensional datasets. We present evaluations on CIFAR-10 and Imagenette images)
Compared to existing work, we show that we can optimize IB objectives for well-known DNN architectures using standard optimizers, losses and simple regularizers, without needing complex estimators, generative models, or variational approximations. This will allow future research make better use of IB objectives and study the IB principle more thoroughly.
2 BACKGROUND
Information quantities & information diagrams. We denote entropy H[:]. joint entropy H[---, conditional entropy H[- mutual information I[::-] and Shannon's 's information content h-) (Cover and Thomas, 2012: MacKay, 2003; Shannon, 1948). We will further require the Kullback-Leible divergence ( I) and cross-entropy -l) The definitions can be found in section A. We will use differential entropies interchangeably with entropies: equalities between them are preserved in the differential setting, and inequalities will be covered in section 3.3.
Thiis connection was assumed without proof by Achille and Soatto (2018a;b).
3 Recently, Fischer and Alemi (2020) report results on CIFAR-10 and ImageNet, see section F.4.
2

---

Under review as a conference paper at ICLR 2021
Information diagrams (I-diagrams), like the one depicted in figure 2, clarify the relationship between information quantities: similar to Venn diagrams, a quantity equals the sum of its parts in the diagram. Importantly, they offer a grounded intuition Yeung (1991) show that we can define signed measure J such that information quantities map to abstract sets and are consistent with set operations. We provide details on how to use I-diagrams and what watch out for in section A.2.
Probabilistic model. We will focus on supervised classification task that makes prediction Y given data X using a latent encoding Z, while the provided target is Y. We assume categorical Y and Y, and continuous X. Our probabilistic model based on these assumptions is as follows:
2. p(x., y) pa(z xx (p(f12)
Thus, Z and Y are independent given X, and Y is independent of X and Y given Z. The data distribution f(x, y) is only available to us as an empirical sample distribution. 0 are the parameters we would like learn. Po(z x) is the encoder from data X to latent 7,, and po(2) z) the decoder from latent Z to prediction Y. Together, po(z x) and po(51 2) form the discriminative model pa(O) x):
po x) = Epp po z).
(3)
We can derive the cross-entropy loss H(p(yl x) pa(Y y x)) (Solla et al., 1988; Hinton, 1990) by minimizing the Kullback-Leibler divergence between the empirical sample distribution p(x,y) and the parameterized distribution po(x) po(5 xx), where we set pa(x) = p(x). See section D.I.
Mickey Mouse I-diagram. The corresponding I-diagram for X, Y. and Z is depicted in figure 2. As some of the quantities have been labelled before, we try to follow conventions and come up with consistent names otherwise. Section B. provides inluitions for these quantities, and section B.2 lists all definitions and equivalences explicitly. For categorical Z, all the quantities in the diagram are positive, which allows us to read off inequalities from the diagram: only I[X: Z could be negative. but as Y and Z are independent given X, we have I[Y; Z 0, and I[X; Y; [[: I[;; Z 2 O. Section 3.3 investigates how to preserve inequalities for continuous Z.
3 SURROGATE IB & DIB OBJECTIVES
3.1 IB OBJECTIVES
Tishby et al. (2000) introduce the IB objective relaxation of constrained optimization problem: minimize the mutual information between the input X and its latent representation Z while still accu- rately predicting Y from Z. An analogous objective which yields deterministic Z, the Deterministic Information Bottleneck (DIB) was proposed by Strouse and Schwab (2017). Letting B be a Lagrange multiplier, we arrive the IB and DIB objectives:
min [[X; Z BI(Y;)] Z]
for IB, and min H[Z] - for DIB.
(4)
This principle can be recast as a generalization of finding minimal sufficient statistics for the labels given the data (Shamir et al., 2010; Tishby and Zaslavsky, 2015; Fisher, 2019): it strives for minimality and sufficiency of the latent Z. Minimality is achieved by minimizing the Preserved Information I[x;Z]; while sufficiency is achieved by maximizing the Preserved Relevant Information I[Y; ZJ. We defer an in-depth discussion of the IB principle to the appendix Section C.I. We discuss the several variants of IB objectives, and justify our focus on IB and DIB, in Section C.2.
The information quantities that appear in the IB objective are not tractable to compute for the representations learned by many function classes of interest, including neural networks; for example, Strouse and Schwab (2017) only obtain an analytical solution to their Deterministic Information Bottleneck (DIB) method for the tabular setting. Alemi et al. (2016) address this challenge by constructing a variational approximation of the IB objective, but their approach has not been applied separate statistics
to more complex datasets than MNIST variants. Belghazi et al. (2018) network to approximate the mutual information, a computationally expensive strategy that does not easily lend itself optimization.
In this section, we introduce and justify tractable surrogate losses that are easier to apply in common deep learning pipelines, and which can be scaled to large and high-dimensional datasets. We begin by proposing the following reformulation of IB and DIB objectives.
3

---

Under review as a conference paper at ICLR 2021
Proposition 1. For IB, we obtain
arg min [[X:Z] -][Y: = arg min H[Y| + IX:Z|Y Y)
(5)
=HIZYY)-HIXXI
and, for DIB, arg min -11)B = arg min H g
arg min H[Y [2] + H[Z]
(6)
with 8 = 3-1 E (0) and / := E [0, 1). The derivation can be found in section C.3.
In the next sections, show that Decoder Uncertainty H[Y [Z] provides loss term, which min- imizes the training error, and DIB's Reverse Decoder Uncertainty H[Z] | Y] and IB's Redundant Information I[X; Y], respectively, provide regularization term, which helps generalization. An- other perspective can be found by relating the objectives to the Entropy Distance Metric introduced by MacKay (2003), which we detail in section C.4.
3.2 DECODER UNCERTAINTY H[Y Z]]
The Decoder Uncertainty H[Y Z] is the first term in our reformulated IB and DIB objectives, and captures the data fit component of the IB principle. This quantity not easy to compute directly for arbitrary representations Z, so we turn our attention to two related entities instead, where we use 0 as subscript to mark dependence on the model: the Prediction Cross-Entropy, denoted H&[Y] X] (more commonly known as the model's cross-entropy loss; see section D.1), and the Decoder Cross-Entropy, denoted Hs[Y|Z] [Z]. Noting that (x)
x, we define these terms as follows:
H₂TY [X] == H(pOy x) Pa = x)) = Epexy) (Ereeto Po =y12)
(7)
H [Z] := H(p 2) P y = Epixy Epacao (po = y))))
(8)
Jensen's inequality yields | < H₂[Y ZZ, with equality Z is a deterministic function
of X. The notational similarity* between HalY [Z] and H[Y [2] is deliberately suggestive: this cross-entropy bounds the conditional entropy H[Y ZJ, as characterized in the following proposition.
Proposition 2. The Decoder Cross-Entropy provides an upper bound on the Decoder Uncertainty:
H[Y [Z] S H[Y [Z] (xL(p(y ( 22) po(2) 2)) HolY ZZ].
further bounds the training error:
P is wrong ") S 1 - -ILIZI = 1 - (UIVI/Z--Da...0sttttto00000 Likewise, for HoLY X] and H[Y XX] See section D.2 for derivation.
(10)
obtain bound for the training error in terms of H[Y ZJ. We examine one way of doing so by using optimal decoders paO) z) == p(Y ý 2)) for the
Hence, by bounding DxL (ply 2) pa(O) | z)),
case of categorical Z in section
Alemi et al. (2016) use the Decoder Cross-Entropy bound in equation (9) to variationally approximate p(y | z). We make this explicit by applying the reparameterization trick rewrite the latent z as a parametric function of its input x and some independent auxiliary random variable 17. i.c.
D z Polz x), yielding fe(x, r)
= H[Y Z) S = Epm h ( y
f6(x;)))
(11)
Equation (11) can be applied to many forms of stochastic regularization that turn deterministic models into stochastic ones, in particular dropout. This allows us to modern DNN architectures as stochastic encoders.
Dropout regularization When we interpret 17 sampled dropout mask for DNN, DNNs that use dropout regularization (Srivastava et al., 2014), or variants like DropConnect (Wan et al., 201 3013a), fit the equation above as stochastic encoders. Monte- Carlo dropout (Gal and Ghahramani, 2016), for example, even specifically estimates the predictive mean pa(S) x) from equation (3). The following result extends the observation by Burda al. (2015) that sampling yields an unbiased estimator for the Decoder Cross-Entrope HglY Z], while it only yields a biased estimator for the Prediction Cross-Enin Cross-Entropy H X] (which it upper- boper-bounds).
*hii notation is compatible with V-Entropy introduced by Xu al. (2020).
4

---

Under review as a conference paper at ICLR 2021
Without noise
With noise
Regularizer
= log Var [2]
- log Var [2] YY]
_
E
Residua Information I|Y;X Z]
Training Error
1.5
0.50
0.25
10"
100
10
10P
10
10a
50
100
150
200
250
Preserved Information 1X;Z
Figure 3: Decreasing the entropy of a noise-free
Figure 4: Information plane plot of the latent
latent does not affect the training error. (Though
similar to Tishby and Zaslavsky (2015) but using ResNet18 model on CIFAR-10 using the different
floating point issues start affecting it negatively eventually.) When adding zero-entropy noise, the error rate increases as the entropy approaches zero. See section 3.3 and G.4 for more details.
regularizes from section 3.3. A larger version found in figure G.1. See section 4 for more details. Best viewed in color.
Corollary 1. Let x. and fo be defined as previously, with a sampled stochastic dropout mask. Then h (paCY = = fo(x;1)) rj)) evaluated for a single sample is an unbiased estimator of the Decoder Cross Cross-Entropy H&[Y Z}, and an estimator of an upper bound on the Prediction Cross- Entropy H&[Y [X].
This distinction between the Decoder Cross- Cross-Entropy and the Prediction Cross-Entrope has been ob- served in passing in the literature, but not made explicit. Multi-sample approaches like Multi-Sample Dropout (Inoue, 2019), for example, optimize HolY [Z], while Importance Weighted Stochastic Gra- dient Descent (Noh et al., 2017) optimizes HolY Y X]. Dusenberry et al. (2020) observe empirically in the different context of rank-1 Bayesian Neural Networks that optimizing H [Z] instead of HoIY X] is both easier and also yields better generalization performance (NLI,, accuracy, and ECE), while they also put forward an argument for why the stochastic gradients for HolY Z] might benefit from lower variance. We empirically compare training with either cross-entropy in section G.3.3 and show results in figure G.11 in the appendix. We conclude this section by highlighting that H[Y Z] is therefore already minimized in modern DNN architectures that use dropout together with a cross-entrope loss. This means that, at least for one half of our reformulation of the IB objective, can apply off-the-shelf, scalable objectives and optimizers for its minimization.
3.3 SURROGATES FOR THE REGULARIZATION TERMS
In the previous section, we have examined how to tractably estimate the error minimization term H[Y IZ]. In this section, will examine tractable optimization of the regularization terms H[Z Y] and I[X:2| / Y], respectively. We discuss how to minimize entropies meaningfully and show how this unifies DIB and IB via the inequality I[X; YJ H[Z] Y] & H[Z] before providing tractable upper-bounds for H[Z] YJ and H[Z].
Differential entropies In most cases, the latent Z is continuous random variable in many dimen- sions. Unlike entropies on discrete probability spaces, differential entropies defined on continuous spaces are bounded from below. This means that the DIB objective optimal solution and allows for pathological optimization trajectories in which the variance of the latent can be scaled be arbitrarily small, achieving arbitrarily high-magnitude negative entropy. We provide toy experiment demonstrating this in section G.4.
guaranteed have an
Intuitively, one can interpret this issue as being allowed to encode information in an arbitrarily-small real number using infinite precision, similar to arithmetic coding (MacKay, 2003; Shwartz-Ziv and Tishby, 2017)>. practice, due to floating point constraints, optimizing DIB naively will invariably end in garbage predictions and underflow as activations approach zero. It is therefore not desirable
*Conerselyy, MacKay (2003) notes that without upper-bounding the "power" Feae Z2, all information could he encoded in single very large integer.
5

---

Under review as a conference paper at ICLR 2021
for training. This is why Strouse and Schwab (2017) only consider analytical solutions to DIB by evaluating limit for the tabular case. MacKay (2003) proposes the introduction of noise to solve this issue in the application of continuous communication channels.
However, here we propose adding specific noise to the latent representation to lower-bound the conditional entropy of Z, which allows to enforce non-negativity across all IB information
quantities as in the discrete case and transport inequalities to the continuous case: for a continuous
2 E R and independent noise e, we set := 2 the differential entropy then satisfies H[Z] = H[2+ <] Z H[e]; and by using zero-entropy noise N(0, 11) specifically, obtain H[Z] 2 H[e] = O.
Proposition 3. After adding zero-entropy noise, the inequality I[X:Z & HIZ & H[Z] also holds for continuous Z, and we can minimize I[X:Z] Y) in the IB objective by minimizing H[Z YJ or HIZ), similarly to the DIB objective.
Strictly speaking, zero-entropy noise is not necessary for optimizing the bounds: any Gaussian noise is sufficient, but zero-entropy noise is aesthetically appealing as it preserves inequalities from the discrete setting. In a sense, this propostion bounds the IB objective by the DIB objective. However, adding noise changes the optimal solutions: whereas DIB in Strouse and Schwab (2017) leads to hard clustering in the limit, adding noise leads to soft clustering when optimizing the DIB objective, as is the case with the IB objective. We show in section F.6 that minimizing the DIB objective with noise leads to soft clustering (for the case of an otherwise deterministic encoder). Altogether, in addition to Shwartz Zii and Tishby (2017), we argue that noise is essential to obtain meaningful differential entropies and to avoid other pathological cases as described further in section F.7.
It is not generally possible to compute H[Z| Y] exactly for continuous latent representations Z, but we can derive an upper bound. The maximum-entroopy distribution for a given covariance matrix is Gaussian with the same covariance.
Proposition 4. The Reverse Decoder Uncertainty can be approximately bounded using the empirical variance Var Iy]:
Egt) Σ In(2re Var[Z, 1y)) EgGy Σ (2Te Var[Z, LyD,
H[Z|Y] YJ
(12)
where 74 are the individual components 7. HIZ) can be bounded similarly. More generally, we can create even looser upper bound by bounding the mean squared norm of the latent:
I S C'
H[Z YJ s HIZ] sC
(13)
with C" = 2nee"
ke20a
for 7. E Rt. See section F22 for proof.
Surrogate objectives These surrogate terms provide us with three different upper-bounds that surrogate regularizers. We refer to them conditional log-variance regularizer (log Var[Z YI), log-variance regularizer (log Var[Z]) and activation Le regularizer (I P We can now propose the main results of this paper: IB surrogate objectives that reduce almost trivial implementation using the cross-entropy loss and one of the regularizers above while adding zero-entropy noise the latent Z.
Theorem 1.
Let Z be obtained by adding a single sample of zero-entropy noise to single sample of the output z of the stochastic encoder. Then each of the following objectives is an estimator of an upper bound on the IB objective. In particular, for the surrogate objective E ||Z],, we obtain:
min H(p(y 2) 11 Po = y + ?
(14)
for log VarlZ| Y|:
Hin (2) P = y + Epor Σ In VarlZ, Iyl);
(15)
for log Var[Z]:
Σ 1 In(2xe Var(Z,1)
min minH(ppyll)) 22 Y = y +
(16)
,
For the latter two surrogate regularizers, we can relate their coefficient y to B and from section 3. However, as regularizing IE ||2]|2 does not approximate entropy directly, its coefficient does not relate to the Lagrange multiplier of any fixed IB objective. We compare the performance of these objectives in section 4.

---

Under review as a conference paper at ICLR 2021
10
10-
10-2
Weight Decay
Weight Decay
0.75
Accuracy
0:75
Accuracy
0.50
0.50
0.25
0.25
0.001
0.02
0.3
0.003
0.01
0.3
50
100
Epslion
Preserved Information [[xA7]
(a) Robustness for different attack strengths e. The dashed black line represents a model trained only with cross-entropy and no noise injection. We see that mod- els trained with the surrogate IB objective (colored by y) see improved robustness over a model trained only to minimize the cross-entrop training objective (shown in black) while the models regularized with weight-decay actually perform worse.
(b) Average robustness over e E 00.0.I] compared to
normal accuracy for different amounts of Preserved Information. markers show robustness. x markers show the normal accuracy. We see that robustness depends on the Preserved Information. If the latent compressesed too much, robustness (and accuracy) are low. If the latent is not compressed enough, robustness and thus generalization suffer.
Figure 5: Adversarial robustness of ResNet18 models trained on CIFAR-IO with surrogate objectives in comparison to regularization with L2 weight-decay as non-IB method. The robustness is evaluated using FGSM, PGD, DeepFool and BasicIterative attacks of varying values.
4 EXPERIMENTS
We now provide empirical verification of the claims made in the previous sections. Our goal in this section is to highlight two main findings: first, that our surrogate objectives obtain similar behavior to what we expect of exact IB objectives with respect to their effect on robustness to adversarial examples. In particular, we show that our surrogate IB objectives improve adversarial robustness compared to models trained only the cross-entropy loss, consistent with the findings of Alemi et al. (2016). Second, we show the effect of our surrogate objectives on information quantities during training by plotting information plane diagrams, demonstrating that models trained with our objectives trade off between / [[: 2] and ][Y: Z] as expected. We show this by recovering information plane plots similar the ones in Tishby and Zaslavsky (2015) and qualitatively examine the optimization behavior of the networks through their training trajectories. We demonstrate the scalability of our surrogate objectives by applying our surrogate IB objectives to the CIFAR-10 and Imagenette datasets, high-dimensional image datasets.
For details about our experiment setup, DNN architectures, hyperparameters and additional insights, see section G. In particular, empirical quantification of our observations on the relationship between the Decoder Cross- Eatsrpy loss and the Prediction Cross- Cross-Entropy are deferred to the appendix due to space limitations as well the description of the toy experiment that shows that minimizing H[Z | Y] for continuous latent Z without adding noise does not constrain information meaningfully and that adding noise solves the issue detailed in section 3.3.
Robustness to adversarial attacks Alemi et al. (2016) and Fischer and Alemi (2020) observe that their IB objectives lead to improved adversarial robustness over standard training objectives. We perform a similar evaluation to see whether our surrogate objectives also see improved robustness. We train fully-connected residual network on 1I for range of regularization coefficients y using our IE 2 surrogate objective; we then compare against a similar regularization method that does not have an cemmmation-thhoretin interpretation: L2 weight-decay. We inject zero-entropy noise in both cases. After training, we evaluate the models on adversarially perturbed images using the FGSM (Szegedy et al., 2013), PGD (Madry et al., 2018), Basiclterative (Kurakin et al., 2017) and DeepFool Aalossvvi-Dezoool et al., 2016) attacks for varying levels of the perturbation magnitude parameter e. We also compare to a simple unregularized cross-entropy baseline (black dashed line). To compute overall robustness, we use each attack in turn and only count sample as robust if it defeats them all. As depicted in figure 5, we find that our surrogate objectives yield significantly more robust models while obtaining similar test accuracy the unperturbed data whereas weight-decay regularization reduces robustness against adversarial attacks. Plots for the other two regularizers can be found in the appendix in figure 13 and figure G.14.
7

---

Under review as a conference paper at ICLR 2021
10
10
10-22
50
100
150
Epoch
2
Weight Docay
Residual Information IIX; |2]
0.5
100
Preserved Information [[x; 2]
Figure 6: Information plane plot of the training trajectories of ResNet18 models with the IE |IZ] surrogate objective or 1.2 weight-decas on CIFAR-1O. The color shows y; the transparency the training epoch. Compression (Preserved Information L) trades-off with performance (Residual Information J).
See section 4. While the trajectories are similar, robustness is very different, see figure 5.
Information plane plots for CIEAR-10 To compare the different surrogate regularizers, we again 8 model on CIFAR-10 with zero-entropy noise added to the final layer activations Z, with K 256 dimensions, as an encoder and add a single x 10 linear unit as a decoder. We train with the surrogate objectives from section 3.3 for various y. chosen in logspace from different ranges to compensate for their relationship to B as noted in section 3.3: for log Varl [Z] E [10-S, I]; for log VarlZ YJ. Y [10-5, 10]; and for IE IIZI2. by trial and error, y E [10-6 10]. We estimate information quantities using the method of Kraskov et al. (2004).
Figure shows an information plane plot for regularizing with | for different y over different epochs for the training set. Similar to Shwartz-Ziv and Tishby (2017), we observe that there is an initial expansion phase followed by compression. The jumps in performance (reduction of the Residual Information) are due to drops in the learning rate. In figure 4, we can see that the saturation curves for all 3 surrogate objectives qualitatively match the predicted curve from Tishby and Zaslavsky (2015). Figure G.1 shows the difference between the regularizers more clearly, and figure G.3 shows the training trajectories for all three regularizers. More details in section G.3.1.
Information plane plots for Imagenette To show that our surrogate objectives also scale up to larger datasets, we run similar experiment on Imagenette (Howard, 2019), which is a subset of ImageNet with 10 classes with 224 x 224 x 3 1.5 x 10 input dimensions, and which we obtain 90% test accuracy. See the figure 1, which shows the trajectories on the test set. We obtain similar plots to the ones obtained for CIFAR-10, showing that surrogate objectives scale well higher-dimensional datasets despite their simplicity.
5 CONCLUSION
The contributions of this paper have been threefold: First, we have proposed simple, tractable training objectives which capture many of the desirable properties of IB methods while also scaling to problems of interest in deep learning. For this we have introduced implicit stochastic encoders, e.g. using dropout, and compared multi-sample dropout approaches to identify the one that approximates the Decoder Uncertainty H IZ), relating them to the cross-entropy loss that is commonly used for classification problems. This widens the range of DNN architectures that can be used with IB objectives considerably. We have demonstrated that our objectives perform well for practical DNNs without cumbersome density models. Second, we have motivated our objectives by providing insight into limitations of IB training, demonstrating how to avoid pathological behavior in IB objectives, and by endeavouring to provide a unifying view on IB approaches. Third, we have provided mathematically grounded intuition by using I-diagrams for the information quantities involved in IB, shown common pitfalls when using information quantities and how to avoid them, and examined how the quantities relate to each other. Future work investigating the practical constraints the expressivity of given neural network may provide further insight into how to measure compression in neural networks. Moreover, the connection to Bayesian Neural Networks remains to be explored.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):1947-1980, 2018a.
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 2018b.
Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410. 2016.
Rana Ali Amjad and Bernhard Claus Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pages 531-540, 2018.
Yoshua Bengio et al. Learning deep architectures for ai. Foundations and trends in Machine Learning, 2(1):1-127, 2009.
J-F Bercher and Christophe Vignat. renyi entropy convolution inequality with application. In 2002 11th European Signal Processing Conference, pages 1-4. IEEE, 2002.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509. 555.005599 10500,05100000 2015.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in f-vae. arXiv preprint arXiv: 1804.03599 2018.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Michael W Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-an Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. arXiv preprint arXiv:2005.07186. 2020.
Ian Fischer. The conditional entropy bottleneck. arXiv preprint arXiv:2002.05379 2020.
Ian Fischer and Alexander A. Alemi. Ceb improves model robustness. Entropy, 22(10):1081, Sep 2020. ISSN 1099-4300. doi: 10.3390/e22101081 URL http://dx.. doi www.doio.ooooooooooooooooooooo. wtog//...//// /3903
e22101081.
Ian Fisher. The Conditional Entropy Bottleneck. Submission to ICLR 2019, International Conference Learning Representations, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, 2016.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint 1rX3v:1903.124360 2019.
David Gondek and Thomas Hofmann. Conditional information bottleneck clustering, 3rd ieee international conference data mining, workshop on clustering large data sels, pages 36-42. Citeseer, 2003.
Ian J Goodfellow. Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation catastrophic forgetting in gradient- based neural networks. arXiv preprint 1312 2013.

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 6a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. Lecture Notes in Computer Science, page 630-645, 2016b.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Geoffrey E Hinton. 1990.
Connectionist learning procedures. Machine learning, pages 555 610. Elsevier,
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.57455 2011.
Jeremy Howard. Imagewang. 2019. htl.httss://iiie http://ttgihh.coo/.o...oo /ttp//tttt..cc//ssaaaaaaaa/ttttt
Hiroshi Inoue. Multi-sample dropout for accelerated training and better generalization. arXiv preprint arXivv1905.09788, 2019.
Morris A. Jette, Andy B. Yoo, and Mark Grondona. Slurm: Simple linux utility for resource
management. In In Lecture Notes in Computer Science: Proceedings of Job Scheduling Strategies for Parallel Processing (JSSPP) 2003, pages 60. Springer-Verte 2002.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Andreas Kirsch, Joost Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pages 7024-7035, 2019.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 0966)3, 2004.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, lan J Goodfellow, and Samy Bengio. Adversarial machine learning at scale. 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-23) -3244, 1998.
J. C. MacKay. Information Theory, Inference, and Learning Algorithmss Cambridge University Press, 2003.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. arXiv preprint 181 1.0055,,), 2018.
William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
Seyed-Mohsen Moosavi-Dezfooll: Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574-2582, 2016
10

---

Under review as a conference paper at ICLR 2021
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. In Advances in Neural Information Processing Systems, pages 5109-5118, 2017.
Morteza Noshad, Yu Zeng, and Alfred O Hero. Scalable mutual information estimation using dependence graphs. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2962-2966. IEEE, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.0378, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pages 8024-8035, 2019.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855, 1992.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint 1905 2019.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning, Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 41 (11(29930):296622711 2010.
Claude E Shannon. mathematical theory of communication. Bell system technical journal, 27(3): 379-423, 1948.
Ravid Biwartz-Zivv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 1703.00800 2017.
Sara A. Solla, Esther Levin, and Michael Fleisher. Accelerated learning in layered neural networks. Complex Systems, 2, 1988.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ily Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 1599:1199-11988 2014.
DJ Strouse and David Schwab. The deterministic information bottleneck. Neural computation, 29 (6):1611-1630, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1rXiv:1312.6199, 2013.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
Naftali Tishby, Fernando Percira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004077, 2000.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13622 2019.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058-1066. 2013a.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun. and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058-1066, 2013b.
11

---

Under review as a conference paper at ICLR 2021
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689, 2020.
Raymond W Yeung. A new outlook on shannon's 's information measures. IEEE transactions on information theory, 37634:666477, 1991.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 161 2016.
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4320- 4328, 2018.
12

---

Under review as a conference paper at ICLR 2021
A
INFORMATION QUANTITIES & INFORMATION DIAGRAMS
Here we introduce notation and terminology in greater detail than in the main paper. We review well-known information quantities and provide more details on using information diagrams (Yeung, 1991).
A.
INFORMATION QUANTITIES
We denote entropy H.-]. joint entropy H[:, -), conditional entropy H[-I-], mutual information I[ and Shannon's 's information content (() following Cover and Thomas (2012); MacKay (2003); Shannon (1948)
(x) = In x
H[X] Epcc h (p(x))
H[X, Y] Epoco h (p(x,y))
H[X Y] H[X, Y] -]-
H[Y]
PPo H[X y] = Eetry h ((p(xl))))
[[X: Y] H[X] H[Y] H[X. Y]
pop
p(xy)
I[X; [4] H[X H[Y [2] H[X, 12),
where X, Y, Z are random variables and x. 2. z are outcomes these random variables can take.
We use differential entropies interchangeably with entropies. We can do so because equalities between them hold as can verified by symbolic expansions. For example,
H[X, Y H[X Y] H[Y]
< Eptxy (p(x.))) = Epccy) [h ())(1)))) h (p())) = Epaay [h (p(x yy)) Epg / (p(y)).
which is valid in both the discrete and continuous case (if the integrals all exist). The question of how to transfer inequalities in the discrete case to the continuous case dealt with in section 3.3.
We will further require the Kullback-Leibler divergence D || and cross-entropy H(: I):):
H(p(x) q(x)) Po h (q(x)
(RI(p(x)) q(x)) 1Epco h 4(s))
ptz)
H(p6y x)) q6yl x)) Epco Epotoo (x)1xx)
Eptcy) h (g(y x))
( q6yl x)) Eptcy pots)
(9(xx)
A.2 INFORMATION DIAGRAMS
Information diagrams (E-diagrams). like the one depicted in figure 2 (or figure H.1 for bigger version), visualize the relationship between information quantities: Yeung (1991) shows that we can define signed measure μ such that these well-known quantities map to abstract sets and are consistent with set operations.
H[A] = 1 (A)
H[A .,.,,,A,1111. = μ (((UA))
H[A
Am , = μ "(UIA) - U,))
IAA = μ (((NA))
I A BI...,,,,, "1 (N=A; U,B))
Note that interaction information (McGill, 1954) follows as canonical generalization the mutual information multiple variables from that work, whereas total correlation does not.
13

---

Under review as a conference paper at ICLR 2021
In other words, equalities can be read off directly from I-diagrams: an information quantity is the sum of its parts in the corresponding I-diagram. This is similar to Venn diagrams. The sets used in I-diagrams are just abstract symbolic objects, however.
An important distinction between I-diagrams and Venn diagrams is that while we can always read off inequalities in Venn diagrams, this is not true for I-diagrams in general because mutual information terms in more than two variables can be negative. In Venn diagrams, a set is always larger or equal any subset.
However, if we show that all information quantities are non-negative, we can read off inequalities again. We do this for figure at the end of section 2 for categorical Z. and expand this continuous Z in section 3.3. Thus, we can treat the Mickey Mouse I-diagram like a Venn diagram to read off equalities and inequalities.
Nevertheless, caution is warranted sometimes. As the signed measure can be negative, J (X n Y) = 0 does not imply n Y = 0: deducing that a mutual information term is does not imply that one can simply remove the corresponding area in the I-diagram. There could be Z with μ ((X n Y) Z) O, 0 but X n # This also means that we cannot drop the term from expressions when performing symbolic manipulations. This is of particular importance because a mutual information of zero means two random variables are independent, which might invite one drawing them as disjoint areas.
such that μ (X n Y) = " (X n Y Z) + + (X Z)
The only time where one can safely remove an area from the diagram is for atomic quantities, which are quantities which reference all the available random variables (Yeung, 1991). For example, when we only have three variables X, Y. Z, I[X; Y; Z] and I[X; Y Z] are atomic quantities. We can safely remove atomic quantities from I-diagrams when they 0 as there no random variables left to apply that could lead to the problem explored above.
Continuing the example, 0 I[X; Y; = pX remove it from the diagram without loss of generality. Moreover, atomic I[X; r|2] = =4" mxnnx = then and could be removed from the diagram well.
n Y n Z2) would imply X Knynk Zznz= 0, and could
We only use I-diagrams for the three variable case, but they supply us with tools to easily come up with equalities and inequalities for information quantities. In the general case with multiple variables, they can be difficult to draw, but for Markov chains they can be of great use.
B МICKEY MOUSE I-DIAGRAM
B.1 INTUITION FOR THE MICKEY MOUSE INFORMATION QUANTITIES
We base the names of information quantities on existing conventions and come up with sensible extensions. For example, the name Preserved Relevant Information for I[Y;Z] was introduced by Tishby and Zaslavsky (2015). It can be seen as the intersection of I[X: and I[X; YJ in the I-diagram, and hence we denote [[X: Preserved Information and I[X; Y] Relevant Information, which are sensible names as we detail below.
We identify the following six atomic quantities:
Label Uncertainty H[Y X] quantifies the uncertainty in our labels. If we have multiple labels for the same data sample, it will be O. It is O otherwise.
Encoding Uncertainty H[Z| X] quantifies the uncertainty in our latent encoding given sample.
When using a Bayesian model with random variable w for the weights, one can further split this term into H[Z X] I[Z; w X] + H[Z X, w), so uncertainty stemming from weight uncertainty and independent noise (Houlsby et al., 201 I; Kirsch et al., 2019).
Preserved Relevant Information I[Y: quantifies information in the latent that is relevant for our task of predicting the labels (Tishby and Zaslavsky, 2015). Intuitively, we want to maximize it for good predictive performance.
Residual Information I[X; Y Z] quantifies information for the labels that is not captured by the latent (Tishby and Zaslavsky, 2015) but would be useful be captured.
14

---

Under review as a conference paper at ICLR 2021
Redundant Information I[X:Z] Y] quantifies information in the latent that is not needed for pre- dicting the labels
We also identify the following composite information quantities:
Relevant Information I[:; YJ = I[;; Y Z] + I[Y;Z] quantifies the information in the data that is
relevant for the labels and which model needs capture to be able predict the labels.
Preserved Information I[X: = ZIX: + I[Y:Z] quantifies information from data that preserved in the latent.
Decoder Uncertainty H[YIZ] I[X: Y 2 + X] quantifies the uncertainty about the labels after learning about the latent Z. H[Y Z] reaches 0, it means that no additional information is needed to infer the correct label Y from the latent Z: the optimal decoder can be deterministic mapping. Intuitively, we want to minimize this quantity for good predictive performance.
Reverse Decoder Uncertainty H[Z] YJ I[X:ZI + HIZ X] quantifies the uncertainty about the latent 7. given the label Y. We can imagine training new model predict given and minimizing H[Z. Y] to 0 would allow for deterministic decoder from the latent to given the label.
Nuisance? H[X YJ = H[X Y. I[X; Z] quantifies the information in the data that is not relevant for the task (Achille and Soatto, 2018a).
B.2 DERNITIOOS & EQUIVALENCES
The following equalities can be read off from figure For completeness and to provide a handy reference, we list them explicitly here. They can also be verified using symbolic manipulations and the properties of information quantities.
Equalities for composite quantities:
[[; YJ =
(17)
[[X:Z] = Y) [[Y;
(18)
H[Y 7.] = [[X; [X]
(19)
H[Z| Y] = I]X:Z
(20)
H[X YJ = (XX]
(21)
We can combine the atomic quantities into the overall Label Entropy and Encoding Entropy:
H[Y] H X] + I[Y: Z] I[X;
(22)
H[Z] H|Z| X + I[Y; + +[X;Z|
(23)
We can express the Relevant Information I[X; YJ, Residual Information I[X; Y Y IZ]. Redundant
Information [[X;Z] Y] and Preserved Information [[X; without X on the lent- side:
I[X; Y] = H[Y] H[Y [X],
(24)
I[ = H[Z] - H[Z X],
(25)
I[X; Y Z = H[Y [Z] H[Y X).
(26)
I[X:Z| Y] = H[Z|Y] - X].
(27)
This simplifies estimating these expressions as X is usually much higher-dimensionn and irregular
than the labels or latent encodings. We also can rewrite the Preserved Relevant Information ![ as:
I[ H|Y H[Y 2 [[: H[Z] H]
(28)
(29)
Fisher (2019) uses the term "Residual Information" for this, which conflicts with Tishby and Zaslavsky
(2015). Noo depicted in figure 2.
15

---

Under review as a conference paper at ICLR 2021
C INFORMATION BOTTLENECK & RELATED WORKS
C.1 GOALS & MOTIVATION
The IB principle from Tishby al. (2000) can be recast a generalization of finding minimal sufficient statistics for the labels given the data (Shamir et al., 2010; Tishby and Zaslavsky, 2015; Fisher, 2019): it strives for minimality and sufficiency of the latent Z. Minimality is about minimizing amount of information necessary of X for the task, so minimizing the Preserved Information I[X; Z];
while sufficiency is about preserving the information to solve the task, so maximizing the Preserved Relevant Information Z)
From figure 2, we can read off the definitions of Relevant Information and Preserved Information:
I[;; YJ = I[ I[X; Y Z] I[: = I|::2]+[[X2/
(30)
(31)
and see that maximizing the Preserved Relevant Information I[Y;Z] is equivalent to minimizing the Residual Information I[X; Y 12), while minimizing the Preserved Information I[X: at the
same time means minimizing the Redundant Information I[ YJ, too, as I[X; Y] is constant for the given dataset Moreover, we also see that the Preserved Relevant Information I[Y: 2] is upper-bounded by Relevant Information I[X; Y), to capture all relevant information in our latent, we want [[X; YJ = I[
Using the diagram, we can also see that minimizing the Residual Information is the same as minimiz- ing the Decoder Uncertainty H[Y ZZ]:
[Iz = H[Y [z] H[Y [X].
Ideally, we also want to minimize the Encoding Uncertainty H[Z] X] to find the most deterministic latent encoding Z. Minimizing the Encoding Uncertainty and the Redundant Information I[X; Y] together is the same as minimizing the Reverse Decoder Uncertainty HIZ| YJ.
All in all, we want to minimize both the Decoder Uncertainty H[Y [Z] and the Reverse Decoder Uncertainty H[Z|| Y].
C.2 IB OBJECTIVES
"THE INFORMATION BOTTLENECK METHOD" (IB)
Tishby et al. (2000) introduce MI(X: (0:-8)-MMM0 -MM Y) as optimization objective for the Information Bottleneck. We can relate this to our notation by renaming X = Z, such that the objective becomes "min [[ - BI[Y; ZJ". The IB objective minimizes the Preserved Information I[X: Z] and trades it off with maximizing the Preserved Relevant Information I[Y Tishby and Zaslavsky (2015) mention that the IB objective is equivalent to minimizing I[X; Z] + +I[IX I see our discussion above. Tishby et al. (2000) provide optimal algorithm for the tabular case, when X. Y and Z are all categorical. This has spawned additional research to optimize the objective for other cases and specifically for DNNs.
"DETERMINISTIC INFORMATION BOTTLENECK" (DIB)
Strouse and Schwab (2017) introduce as objective "min H[Z] -B[Y;; ZJ" Compared to the IB objective, this also minimizes H[Z] X] and encourages determinism. Vice-versa, for deterministic encoders, H[Z X]= O, and their objective matches the IB objective. Like Tishby et al. (2000), they provide an algorithm for the tabular case. To do so, they examine an analytical solution for their objective it is unbounded: H[Z| X]
for the optimal solution. As we discuss in section 3.3, it does not casily translate to a continuous latent representation.
"DEEP VARIATIONAL INFORMATION BOTTLENECK"
Alemi et (2016) rewrite the terms in the bottleneck as maximization problem "max I[Y;Z] BI[X; 2]" and swap the B parameter. Their f would be 1/8 in IB above, which emphasizes that I[Y;Z] important for performance and [[X; Z] acts as regularizer.
That is, it does depend
16

---

Under review as a conference paper at ICLR 2021
The paper derives the following variational approximation to the IB objective, where fo(x, e) denotes a stochastic latent embedding with distribution po(z) x), Po( z) denotes the decoder, and r(z) is some fixed prior distribution the latent embedding:
min 18g0 Fe- log Po( =yl fo(x.. + DxL (ptlsssa))))l)r<))
(32)
In principle, the distributions po(S) z) and po(z xx) could be given by arbitrary parameterizations and function approximators. In practice, the implementation of DVIB presented by Alemi et al. (2016) constructs Po(z] xx as a multivariate Gaussian with parameterized mean and parameterized diagonal covariance using a neural network, and then uses simple logistic regression to obtain Po(1 z), while arbitrarily setting r(z) to be unit Gaussian around the origin. The requirement for Po(z) x) to have closed-fome Bellbackk-eibbbee divergence limits the applicability of the DVIB objective.
The DVIB objective can be written more concisely as min Ho[Y Z] + DuL (p(z (())
in the notation introduced in section 3. We discuss the regularizer in more detail section F.3.
"CondItIONaL ENTROPY BOTTLENECK"
In a preprint, Fisher (2019) introduce their Conditional Entropy Bottleneck "min I[X; [| I[Y: We can rewrite the objective I[X: Y] + I[X: Y 2] I[X; Y], using equations (30) and (31). The last term is constant for the dataset and can thus be dropped. Likewise, the IB objective can be rewritten as minimizing I[X:Z] Y] + (B I)[[;; Y Zj. The two match for B = 2. Fisher (2019) provides experimental results that favorably compare Alemi et (2016), possibly due to additional flexibility as Fisher (2019) do not constrain p(z) to be a unit Gaussian and employ variational approximations for all terms. We relate CEB to Entropy Distance Metric in section C.4.
"ConditiOnal ENTROPY BOTTLENECK" (2020)
substantial revision of the preprint, Fischer (2020) change their Conditional Entropy Bottleneck to include Lagrange multiplier: "min I[X; Y] y][" Their VCEB objective can be written more concisely as
min H Z] yHH H XJ),
where, without writing down the probabilistic model, we introduce variational approximations for the Reverse Decoder Uncertainty and the Encoding Uncertainty.
They are the first to report results on CIFAR-IO. not clear how they parameterize the model they use for 10. They use Gaussian per class to model H
"CEB IMPROVES MODEL ROBUSTNESS"
Fischer and Alemi (2020) take CEB and switch deterministic model which they turn it into a stochastic encoder by adding unit Gaussian noise. They use Gaussians of fixed variance to variationally approximate q(y z): for each class, q(y | z) is modelled as separate Gaussian.
They are the first to report results on ImageNet and report good rebustness against adversarial attacks without adversarial training.
C.3 CANONICAL IB & DIB OBJECTIVES
We expand the IB and DIB objectives into "disjoint" terms and drop constant ones to find a more canonical form. This leads us to focus on the optimization of the Decoder Uncertainty H[Y ZZ] along with additional regularization terms. In section 3.2, discuss the properties of H[Y Z], and in section 3.3 we examine the regularization terms.
Proposition. For IB, we obtain
arg min [[X:Z] Z] arg min H[Y []] + B [[X:Z|
(33)
=HIZY]-HIZ2X
and, for DIB, arg min H[Z] -][ arg min H[Y [ + ('H
arg min H[Y [Z] +H"H[[]
(34)
17

---

Under review as a conference paper at ICLR 2021
with 8 == 8ー oo) and 8 := [0,
Proof. For the steps marked with we make use of / 1. For IB, we obtain
arg min [[X; - - Z| = arg min [[;; YJ + (B- 1)H[Y Z
2 arg mig H|Y [2] [2 I[
arg min H[Y [
XD),
(IB)
and, for DIB,
arg min H[Z] - -B]YY arg min H YJ + (8 -8-
arg arg min H + +
(DIB)
with := F-T [0, oo). Similarly, we show for DIB
arg min H[Z] B[[ arg min H[Z] + + ZZ]
arg min H[Y []] + ('H[Z)
with :-
[0, 1), which relevant in section 3.3.
We limit ourselves to B because, for B which does not make sense: the obvious solution to this is one where Z contains no information on Y, that is p(y is uniform. single latent; whereas for IB, we only minimize the Redundant Information, and the solution is free to contain noise. For B I, we would not care about Decoder Uncertainty and only minimize Redundant Information and Reverse Decoder Uncertainty, respectively, which allows for arbitrarily bad predictions.
would maximizing the Decoder Uncertainty,
the case of DIB, to map every input deterministically to
We note that we have B'
8" 1-p
using the relations above.
C.4 IB OBJECTIVES
ENTROPY DISTANCE METRIC
Another perspective on the IB objectives is by expressing them using the Entropy Distance Metric. MacKay (2003, p. 140) introduces the entropy distance
EDM (Y, Z) H[Y Z] + H].
(35)
a metric when we identify random variables up permutations of the labels for categorical
variables: if the entropy distance is O, Y and Z. are the same distribution up to consistent permutation of the labels (independent of X). the entropy distance becomes 0, both H[Y [2] 0 H[Z| Y), and we can find bijective map from Z to Y."
We can express the Reverse Decoder Uncertainty H[Z| Y] using the Decoder Uncertainty H[Y and the entropies:
H[Z YJ H[]] H Z] H[Z],
and rewrite equation (35) as
EDM (Y,Z) 2H[ H[Z] H[Y].
For optimization purposes, we can drop constant terms and rearrange: arg min EDM (Y,Z) arg min H[Y [Z] 4H[Z].
C.4.1 REWRITING AND DIB USING THE ENTROPY DISTANCE METRIC
For B 2 1, we can rewrite equations (IB) and (DIB) as:
arg min EDM (Y,2) (H[Y [Z] H[ZI (y- H) X]
(36)
for IB, and
arg min EDM (Y, 7) (H[Y H[Z||]
(37)
Thhe argument for continuous variables is the same. We need to identify distributions up "isentropic" bijections.
18

---

Under review as a conference paper at ICLR 2021
for DIB and replace f with y 1- [-1, which allows for linear mix between H[Y [Z] and H[Z] Y].
DIB will encourage the model to match both distributions for y 0 (B = 2), as we obtain term that matches the Entropy Distance Metric from section C.4, and otherwise trades off Decoder Uncertainty and Reverse Decoder Uncertainty. IB behaves similarly but tends to maximize Encoding Uncertainty as y [-2, OJ. Fisher (2019) argues for picking this configuration similar the arguments in section C.I. DIB will force both distributions to become exactly the same, which would turn the decoder into a permutation matrix for categorical variables.
D DECODER UNCERTAINTY H[Y [Z]
D.1 CROSS-ENTROPY LOSS
The cross-entropy loss features prominently in section 3.2. We can derive the usual cross-entropy loss for our model by minimizing the Dellback-Leibled divergence between the empirical sample distribution f(x, y) and the parameterized distribution p^(x) pa(5 xx). For discriminative models, we
are only interested in pa(f x), and can simply set po(x) = p(x):
arg min ( y) (o(x) Pa = y))x)
min DKL (p(y pa Y = y x)) (P(()(x) pa(x))
arg min H(p()) Po = y x)) H[Y [X]
coust.
arg min H(p6) x) II Po = y x))-
In section 3.2, we introduce the shorthand H₂lY Y [X] for H(p(y) ( Po y and refer to it as Prediction Cross-Entropy Eatsooy..
D.2 UPPER BOUNDS & TRAINING ERROR MINIMIZATION
To motivate that H[Y Z] (or H ZJ) can be used as main loss term, we show that it can bound the (training) error probability since accuracy is often the true objective when machine learning models are deployed on real-world problems 10.
Proposition. The Decoder Cross-Entropy provides an upper bound on the Decoder Uncertainty:
H[Y IZ] s H[Y + Dxu(p(y 2) po( z)) H&lY [Z],
and further bounds the training error:
y is wrong") - 1-l----lle -IIYZI Dun
Likewise, for the Prediction Cross-Entre Pross-Entropy H&lY X] and the Label Uncertainty [].
Proof. The upper bounds for Decoder Uncertainty H[Y [Z] and Label Uncertainty H[Y | X] follow from the non-negativity of the Kullback-Leibler divergence, for example:
S (KL(p 2) lI p6() 2)) = H ZJ H[Y Z].
0 0 DKL( xx) p x)) = H₂[Y|X] H[Y X].
The derivation for training error probability is as follows:
Y is correct")) Epcs, Y is correct" xx,) EpOs) Epado Po y
Eps.. pa 2).
We can then apply Jensen's 's inequality using convex (x) = In h (Epo. P y 1)) Epo h (poCY y y
As we only take into account the empirical distribution p(x, available for training, the following derivation refers only the empirical risk, and not the expected risk of the estimator Ý.
19

---

Under review as a conference paper at ICLR 2021
p( is correct") Hopps))px<(a)))))
= P("Y is wrong")
For small Ho[Y [Z]. we note that one can use the approximation e' ~ 1 + x to obtain:
P(" is wrong") & HolY Z].
(38)
Finally, we split the Decoder Cross-Entropy into the Decoder Uncertainty and Kullback-Leible divergence:
H H[Y 24] (py( a) yl
If we upper-bound DKL (p(y|2) p Y yl=),, minimizing the Decoder Uncertainty H[Y [Z] becomes
a sensible minimization objective as it reduces the probability of misclassification.
We can similarly show that the training error bounded by the Prediction Cross-Entropy H&LY [X].
In the next section, we examine categorical for which optimal decoders can be constructed and DRL (p6 Po( y2) becomes zero.
CATEGORICAL Z
For categorical Z, p(y z) can be computed exactly for a given encoder Po(z xx) by using the empirical data distribution, which, in turn, allows us to compute H[Y | 1Z]". This similar to computing a confusion matrix between Y and Z but using information content instead of probabilities.
Moreover, if we set Pa(O 12) = p(Y = 2)2) to have an optimal decoder, we obtain equality in equation (9), and obtain H&LY X] H [7] H[Y [Z]. If the encoder were also deterministic, would obtain H₂[Y X] = H Z = H[Y ZZj. We can minimize H[Y [7] directly using gradient descent.
H[Y|| Z] only depends on p =) and d Po(z x):
Y Eptx.
[In po(z x)) Eppco h (p(yl 2))
d0
do
Proof.
d
d
Exp h (p(yl =)) Eaa (p6rl=)) Epaay d de Epcaa (p(yl z))
d
H[Y Z]
d8
d0
do
d [ (ply =))] (p(r|z)) d do
Epcci Epci do
[In pa(z
d
d
Ep do (ply 2))] + h (p6rl 2))
[In (xpp())x))
do
And now we show that Eppcaya de [h (ply) 2))] = 0:
E Eptxya do (p(y 22)]] Eptoa.) do (p +)) Epp -1 d
d
d
p61 2) do p(y 2))
= - S p do
p(y2)) ply 2 = f p(z) S
d
do ply 2) dz
f p(z) da f a dz =
Splitting the expectation and reordering of Epooa h (pry 2)) d [In po(z x)), we obtain the result.
The same holds for Reverse Decoder Uncertainty H[Z] Y] and for the other quantities can be verified easily.
If we minimize H[Y Z] directly, compute p(y after every training epoch and fix z) == p(Y = 3 2) to create the discriminative model paO | x). This is a different perspective on the self-consistent equations from Tishby et al. (2000); Gondek and Hofmann (2003).
'pyy 2z) depends through pa(z) x): p(y Es Paci
2n ((o)peld)
20

---

Under review as a conference paper at ICLR 2021
Objection
min [2] [4] mile [a(ViX]
Metric
M|M21 Ho[r12]
Permutation MNIST
CIFARIO
Nats/ Nats/Proobssiii Prooniiiiiyy
05-
40
40
Epoch
Figure E.1: Decoder Uncertainty, Decoder Cross-Entropy and Prediction Cross-Entron for Permutation- MNIST and CIFAR- with a categorical Z. C = 100 categories are used for We opti- mize with different minimization objectives turn and plot the metrics. ( 2) P& = y =)) is small when training with H,[Y| []] or H[Y Z]. When training with H X on CIFAR-10, (py(p(y z) Pa = z)) remains quite large. We run 8 trials each and plot the median with confidence bounds (25% and 75% quartiles). See section E. for more details.
E.1 EMPIRICAL
Dxu(p(y Pa(Y = y z)) DURING TRAINING
We examine the size of the gap between Decoder Uncertainty and Decoder Cross-Entropp Entropy and the training behavior of the two cross-entropies with categorical latent Z. on Permutation MNIST and CIFAR-10. For Permutation MNIST (Goodfellow et al., 2013), we use the common fully-connected ReLU 784 - 1024 - 1024 - encoder architecture, with 100 categories for Z. For CIFAR- 10 (Krizhevsky et al., 2009), use standard ResNet18 model with C many output classes encoder (He et al., 2016a). See section G for more details about the hyperparameters. Even though a x 10 matrix and SoftMax would suffice to describe the decoder matrix po( 2) we have found that over-parann aver-paramtterization using a separate DNN benefits optimization lot. Thus, to parameterize the decoder matrix, we fully-connected ReLUS C 1024 1024 10 with final SoftMax layer. We compute it once per batch during training and back-propagate into it.
Figure E. shows the three metrics SOTA accuracy on the test set-we impose harder optimization problem as Z is categorical, and we are essentially solving a hard-clustering problem first and then map these clusters to Y, Results are provided for the training in order compare with the optimal decoder.
train with each of them in turn. Our results do not achieve
As predicted, the Decoder Cross- Cooss-Entropy upper-bounds both the Decoder Uncertainty H[Y [Z] and the Prediction Cross- Cross-Entropy in all cases. Likewise, the gap between HofY [Z] and H[Y [Z] tiny when we minimize H&lY Z]. On the other hand, minimizing Prediction Cross-Entropy can lead to large gaps between H Z] and H[Y Z], as can be seen for CIFAR-10.
Very interestingly, on MNIST Decoder Cross-Entropy provides a better training objective whereas on CIFAR-10 Prediction Cross-Entropy trains lower. Decoder Uncertainty does not train very well on CIFAR-10 10, and Prediction Cross-Entrops does not train well on Permutation MNIST at all. We suspect DNN architectures in the literature have evolved train well with cross-entropies, but we are surprised by the heterogeneity of the results for the two datasets and models.
F SURROGATES FOR REGULARIZZATION TERMS
F.1 DIFFERENTIAL ENTROPIES
Proposition. After adding zero-entropy noise, the inequality IIX;ZYY| & H[Z|Y] s H(Z) also
holds in the continuous case, and we can minimize I[X:Z]Y] Y) in the IB objective by minimizing H[Z] Y] or H[Z), similarly to the DIB objective. We present a formal proof in section F.I.
Theorem 2. For random variables A, B, we have
H[A + B] 2 H[B].
For categorical 7,, P&O stochastic matrix which sums along the Y dimension.
21

---

Under review as a conference paper at ICLR 2021
Proof. See Bercher and Vignat (2002, section 2.2).
Proposition 1. Let Y, Z and X be random variables satisfying the independence property Z T Y|X, and a possibly stochastic function such that Z F(X) + e, with independent noise satisfying L F(X), 1 Y and H(c) = 0. Then the following holds whenever I[Y;Z] is well-defined.
I[X:Z|Y] & HIZIY] H[Z].
Proof. First, we note that X H[F(X) X] Hie X] H[e] with theorem 2, independent of X, and thus H[Z X] 2 O. We have H[Z X H(Z X, Y] by the conditional independence assumption, and by the non-negativity of mutual information, I[Y; Z] 2 0. Then:
I[ YJ H
X = H[Z
H YJ
I[Y: Z H[Z]
The probabilistic model from section fulfills the conditions exactly, and the two statements motivate our proposition.
It is important to note that while zero-entropy noise is necessary for preserving inequalities like H[Z] the continuous case, any Gaussian noise will suffice for optimization purposes: we optimize via pushing down upper bound, and constant offsets will not affect this.
I[X:Z] Y] & HIZ/
Thus, we had H[c] # 0, even though I[X: YJ H[Z X] & HIZ| Y), we could instead use I[ YJ + HIZ X] H[e] s H[Z YJ H[e]
as upper bound minimize. The gradients remain the same.
This also points to the nature of differential entropies as lacking proper point of origin by themselves. We choose one by fixing H[e]. Just like other literature usually only considers mutual information as meaningful, we consider H[Z| X] H[c] as more meaningful than H[Z X]. However, we can side-step this discussion conveniently by picking a canonical noise as point of origin in the form of
zero-entropy noise H[e] = 0.
F.2
UPPER BOUNDS
We derive this result as follows:
HIZIY] Eςor H[Z|
Eptn In det(2ize Cov[Z] y))
$ FAO0 Σ In(2 Var(Z, [y])
z Epo) Σ In(2re Var(Z, ly]).
Theorem 3. Given a k-dimensional random variable X = (X)) with Var[X]] for all H[X] Indet(22ee CovlX])
x Σ In(2xe Var(X,]).
Proof. First, the multivariate normal distribution with same covariance is the maximum entropy dis- tribution for that covariance, and thus H[X] < In det(2re Cov[X]), when substitute the differential entropy for a multivariate normal distribution with covariance Cov[X]. Let Σ, == Cov[X] be the covariance matrix and Σ; diag( Var[X,D), the matrix that only contains the diagonal. Because add independent noise, Var[X,] and thus Σ exists. is clear that tr(E, 'In) = k. Then, we can the KI.-Divergence between multivariate normal distributions No. N₁ with same mean O and
22

---

Under review as a conference paper at ICLR 2021
covariances Σg and Σ₁ to show that In det In In det Σj:
2 det En
0 s (No
|| Ni) = |r(z) Zo) k
det
0 s
Indet En s Ind
det Σo
We substitute the definitions of Σn and Σ₁ and obtain the second inequality after adding & In(2re) on both sides.
Theorem 4, Given a k-dimensional real-valued random variable X = (X:) E R we can bound the entropy by the mean squared norm of the latent:
|FI| = C" = H(X]
(39)
with C"
2ne
Proof. We begin with the previous bound:
H[X] < Σ In(2xe Var(X,]) = In 2xe In [ Var[X]]
In 2ne In
8 Σ Var[Xi]
= In 2ne
Σ X
x
S In 2 E [
where we use the AM-GM inequality:
[ Var[X;]
S Σ Var[X,]
and the monotony of the logarithm with:
Σ
Var[X]] Σ
[x}]
EE 2X
18 X
E
|
Bounding using EE ||X]|= S C', obtain
H[X] s C =
and solving for C' yields the statement.
This theorem provides justification for the use In IE |zZl2 as regularizer, but does not justify the use of IE || directly. Here, we give two motivations. We first observe that In x due to In's strict convexity and = 0, and thus:
H[X] S In 24e E |
[In E - x 2
We can also take a step back and remind ourselves that objectives are actually Lagrangians, and B in min [[X:Z] BI[Y: Z] is introduced as Lagrangian multiplier for the constrained objective:
min [[X: Z) s.t. [[Y: 2 C.
We can similarly write our canonical DIB objective H[Y [Z] + +B"H[Z] constrained objective
min H[Y Z] s.t. H[Z] &
and use above statement to find the approximate form min H[Y Z] s.t. IE | < C'. Reintroducing Lagrangian multiplier recovers our reguralized IE [|][2 objective: min H[Y [2] +I ?
F33 "DEEP VARIATIONAL INFORMATION BOTTLENECK" AND IE 2
Alemi et al. (2016) model Po(2 xx) explicitly as multivariate Gaussian with parameterized mean and parameterized diagonal covariance in their encoder and regularize it to become close N(0, 1g) by
23

---

Under review as a conference paper at ICLR 2021
minimizing the Kullback- divergence (polz x)) N(O, 4)) alongside the cross-entropy. min HolY Z] + (pl x) x) (z)),,
as detailed in section C.2.
We can expand the regularization term to (Ra(p(21x)) N(., 1t))
Po Epec (2T)) e-Hilzri) H[Z X]
EEa
2 In(2x) = -H[Z X].
After dropping constant terms (as they don't matter for optimization purposes), we obtain
E IZIP X].
When we inject zero-entropy noise into the latent Z, we have H[Z. X] 2 0 and thus IE |Z H[Z. s ? Thus, the EE ||4|| regularizer also upper-bounds DVIB's regularizer in this case.
In particular, we have equality when use deterministic encoder. When we inject zero-entropy noise and use deterministic encoder, we are optimizing the DVIB objective function when we use the IE |IZI regularizer. In other words, in this particular case, we could reinterpret "min H₂[Y Z] + E || 2d as optimizing the DVIB objective from Alemi et al. (2016) if they were using a constant covariance instead of parameterizing in their encoder. This does not hold for stochastic encoders.
We empirically compare DVIB and the surrogate objectives from section 3.3 in section G.5. In the corresponding plot in figure G.15, of 2 the Lagrange multiplier.
indeed note that EE ||Z]| and DVIB separated by factor
F.4 DETAILED COMPARISON TO CEB, VCEB & DVIB
In Fisher (2019), the introduced CEB objective "min I[ "min yHly [Z] + H[Z YJ H[Z] X]" similar to the IB objective in proposition 1 in section 3.1. However, these atomic quantities are not separately examined in detail.
Z|"
rewritten to
Both Alemi et al. (2016) and Fischer (2020) focus the application of variational approximations to these quantities, Using a slight abuse of notation to denote all variational approximations, we can write the VCEB objective (Fischer, 2020) and the DVIB objective (Alemi et al., 2016) more concisely as
VCEB min H& Z] + ( H XJ).
DVIB min H /2] + (H Hz[Z|
DVIB does not specify how to choose stochastic encoders and picks the variational marginal q(z) to be a unit Gaussian. We relate how this choice of marginal relates to the [|Z]|2 surrogate objective in section F.3. Alemi et al. (2016) use VAE-like encoders that output mean and standard deviation for latents that are then sampled from a multivariate Gaussian distribution with diagonal covariance in their experiments. They run experiments on MNIST and features extracted from the penultimate layer of pretrained models on ImageNet.
While VCEB introduced in Fisher (2019) is agnostic to the choice of stochastic encoder, Fischer (2020) mention that stochastic encoders can be similar to encoders and decoders VAEs (Kingma and Welling, 2013) or like in DVIB mentioned above. Both VAEs and DVIB explicitly parameterize the distribution of the latent to sample from it before passing samples to the decoder.
Fischer and Alemi (2020) distribution with unit diagonal covariance. They further parameterize the variational approximation for the Reverse Decoder Uncertainty q z) with one Gaussian of fixed variance per class and learn this reverse decoder during training as well. Fischer and Alemi (2020) report results on CIFAR-10 and ImageNet that show good robustness against adversarial attacks without adversarial training. similar to the results in this paper.
existing classifier architecture output means for Gaussian
We will examine the original objective without Lagrange multipliers from Fisher (2019) here.
24

---

Under review as a conference paper at ICLR 2021
This specific (and not motivated) instantiation of the VCEB objective in Fischer and Alemi (2020) is similar to the log VarlZ| Y] surrogate objective introduced in section 3.3 with a deterministic encoder and zero-entropy noise injection. However, the latter uses minibatch statistics instead of learning reverse decoder, trading variational tightness for ease of computation and optimization.
Compared to this prior literature, this paper examines the usage of implicit stochastic encoders (for example when using dropout) and presents three different simple surrogate objectives together with a principled motivation for zero-entropy noise injection, which has dual use in enforcing meaningful compression and in simplifying the estimation of information quantities. Moreover, multi-sample approaches are examined differentiate between Decoder Cross-Entrope and Prediction Cross- Entropy. In particular, implicit stochastic encoders together with zero-entropy noise and simple surrogates make it easier to use IB objectives in practice compared to using explicitly parameterized stochastic encoders and variational approaches.
E5 AN INFORMATION-THEORRIIC APPROACH TO VA
While Alemi al. (2016) draw general connection to B-VAES (Higgins al., 2016), we the insights from this paper to derive simple VAE objective. Taking the view that VAEs learn latent representations that compress input samples, we can approach them as entropy estimators. Using
H[X] + H[Z X] H[X ZJ + H[Z], we obtain the ELBO
(I) 3 H&XX ZJ + HIZ] H[ZIX]
2) H&IX ZJ H[Z].
H[X]= H[X| Z] + H[Z] H[Z|X]
(40)
We can also put eq. (40) into words: we want to find latent representations such that the reconstruction cross-entropy H[X] Z] and the latent entropy H[Z], which tell us about the length encoding input sample, become minimal and approach the true entropy as average optimal encoding length of the dataset distribution.
The first incquality (1) stems from introducing a cross-entropy approximation H&[X| Z] for the conditional entropy H[X| Z]. The second incquality (2) stems from injection of zero-entropy noise with stochastic encoder. For a deterministic encoder, we would have equality. We also note that (1) is the DVIB objective for VAE with B = I, and (2) is the DIB objective for a VAE.
Finally, we can use one of the surrogates introduced in section 3.3 to upper bound H[Z]. For optimization purposes, we can substitute the 1.2 activation regularizer IF ||Z][ from proposition 4 and obtain as objective
min | Z E 112]F.
It turns out that this objective is examined amongst others in the recently published Ghosh et al. (2019) as a CV- VAE, which uses a deterministic encoder and noise injection with constant variance. The paper derives this objective by noticing that the explicit parameterizations that are commonly used for VAES are cumbersome, and the actual latent distribution does often not necessarily match the induced distribution (commonly unit Gaussian) which causes sampling to generate cut-of-distributinn data. fits a separate density estimator on p(z) after training for sampling. The paper goes on to then examine other methods of regularization, but also provides experimental results on CV-VAE, which are line with VAFs and WAFs. The derivation and motivation in the paper is different and makes of information-theoreti principles. Our short derivation above shows the power of using the insights from section 3.2 and 3.3 for applications outside of supervised learning.
F.6 SOFT CLUSTERING ENTROPY MINIMIZATION WITH GAUSSIAN NOISE
Consider the problem of minimizing H[Z YJ and H[) in the setting where Z = fa(X) + N(O, r-iee the embedding Z obtained by adding Gaussian noise to a deterministic function of the input. Let the training set be enumerated .p.. with μ. fe(x). Then the distribution of Z is given by a mixture of Gaussians with the following density, where d(x, (.)) := ||x "
p(z) σ
exp(-d(z,H1)
l=1
25

---

Under review as a conference paper at ICLR 2021
Assuming that each has deterministic label y,, we then find that the conditional distributions p(y 2) and p(z y) are given as follows:
p(zly) o
Σ ny
exp(-d(z,pip
2 2)
Σ
p(ui 1))
Σ
(all))(()
p(z)
Leyey p(z (4)
Liyey exp(-
Σ p(z
exp(-dltz
where n, is the number of x; with class y; = =.. Thus, the conditional ZJY can be interpreted mixture Gaussians and r|z as a Softmax marginal with respect to the distances between and the mean embeddings. We observe that H[Z] Y] is lower-bounded by the entropy of the random noise added to the embeddings:
HIZ I YJ 2 H[G() Y H[e]
with equality when the distribution of fa(X) is deterministic that fo is constant for each equivalence class.
Further, the entropy H[Y Z] minimized when H[Z] is large compared to H[Z| Y] as we have the decomposition
H[Y []
HIZ YJ H[Z] H[Y].
In particular, when fa is constant over equivalence classes of the input, then H[Y [Z] is minimized when the entropy H[fx(X) + é] is large i.e. the values of fa(x;) for each equivalence class distant from each other and there is minimal overlap between the clusters. Therefore, the optima of the information bottleneck objective under Gaussian noise share similar properties to the optima of geometric clustering of the inputs according to their output class.
To gain better understanding of local optimization behavior, decompose the objective terms as follows:
H[Z] Y] Epor H(p(z y) p(z 1y))
Eepoo) H(p(z) xx) p(z ()))
REPOOS (pe(
p(z ()) H[Z
Epcry) DRr
(xp22xx))))))))))) x) (pec
p(z
H[Z
mcomer
To examine how the mean embedding μ4 of a single datapoint x affects this entropy term, we look the derivative of this expression with respect to μı fo(x). We obtain:
d
d
H[Z
YJ
HIZ yx] duu
dju
Kppcis (ple p(z Iy))
duu
Σ Mayyyyy
(plz x)) yx))
ny duu
d
DxL (plz) x y())
ny. du4
While these derivatives do not have a simple analytic form, we can use known properties of the KL. divergence to develop an intuition on how the gradient will behave. We observe that in the left-hand sum de only affects the distribution of Z (that is we are differentiating a sum of terms that look like a reverse KL), whereas it has greater influence on p(z X₂) in the right-hand term, and so its gradient will more closely resemble that of the forward KL. The left-hand-side term will therefore push μ towards the centroid the means inputs mapping to y, whereas the right- hand side term is mode-seeking.
26

---

Under review as a conference paper at ICLR 2021
F.7 A NOTE ON DIFFERENTIAL DISCRETE ENTROPIES
The mutual information between two random variables can be defined in terms of the KL divergence between the product of their marginals and their joint distribution. However, the KL divergence is only well-defined when the Radon-Nikk Radon-Nikodym derivative of the density of the joint with respect the product exists. Mixing continuous and discrete distributions and thus differential and continuous entropies- can violate this requirement, and so lead negative values of the "mutual information". This is particularly worrying in the setting of training stochastic neural networks, as we often assume that an stochastic embedding is generated finite dataset to which continuous perturbation is added. We provide an examples where naive computation without ensuring that the product and joint distributions of the two random variables have a well-defined Radon-Nikodie Nikodym derivative yields negative mutual information.
deterministic transformation of input from
Let U([0, 0.1]), Z x + with R U(1O, 1)). Then
[X:Z] = H[X] = log to S O.
Generally, given X above and invertible function f such that % f(X,, I[x H[X] and can thus be negative. In a way, these cases can be reduced to (degenerate) expressions of the form [[X:X]= H[X].
We can avoid these cases by adding independent continuous noise.
These examples show that not adding noise can lead to unexpected results. While they still yield finite quantities that bear relation to the entropies of the random variables, they violate some of the core assumptions have such that mutual information is always positive.
G EXPERIMENT DETAILS
G.I DNN ARCHITECTURES AND HYPERPARAMETERS
For our experiments, we use Py Torch (Paszke al., 2019) and the Adam optimizer (Kingma and Ba, 2014). In general, we use an initial learning rate of 0.5 x 10-3 and multiply the learning rate by VO.I whenever the loss plateaus for more than 10 epochs for CIFAR-10. For MNIST and Permutation MNIST, we use an initial learning rate of 10 and multiply the learning rate by 0.8 whenever the loss plateaus for more than 3 epochs.
Sadly, we deviate from this in the following experiments: when optimizing the decoder uncertainty for categorical Z for CIFAR- 10, we used epochs patience for the decoder uncertainty objective and a initial learning rate of 10-4. We do not expect this difference to affect the qualitative results mentioned in section E when comparing other objectives. We also only used 5 epochs patience when comparing the two cross-entropies on CIFAR-I0 in section 3.2. As this was used for both sets of experiments, it does not matter.
We train the experiments for creating the information plane plots for 150 epochs. The toy experiment (figure 3) is trained for 20 epochs. All other experiments train for 100 epochs.
We use a batchsize of 128 for most experiments. We use a batchsize of 32 for comparing the cross- entropies for CIFAR-10 (where we take dropout samples each), and a batchsize of 16 for MNIST (where we take 64 dropout samples each).
For MNIST, we use standard dropout CNN, following https://gitte examples/ htles//bob/mnee htte///sstrrmmis/////e model (for experiments with categorical Z. in section E): 784 x 1024 x 1024 x C. For CIFAR-10, we use regular deterministic ResNetl 8 model (He et al., 2016al 6a) for the experiments in section E. (As the model outputs a categorical distribution it becomes stochastic through that and we don't need stochasticity in the weights.) For the other experiments as well the Imagenette experiments, we use a ResNet18v2 (He et al., 2016b). When we need a stochastic model for CIFAR-10 (for continuous Z). we add DropConnect (Wan al., 2013b) with rate 0.1 to all but the first convolutional layers and dropout with rate 0.1 before the final fully-connected layer. Because of memory issues, we reuse the dropout masks within one batch. The model trains to 94% accuracy on CIFAR-10.
com/pytorch/ py. For Permutation MNIST, we use a fully-connected
27

---

Under review as a conference paper at ICLR 2021
For CIFAR-10, always remove the maximum pooling layer and change the first convolutional layer to have kernel size with stride and padding 1. We also use dataset augmentation during training. but not during evaluation the training set and test set for purposes of computing metrics. We crop randomly after the padding the training images by pixels in every direction and randomly flip images horizontally.
We generally sample 30 values of y for the information plane plots from the specified ranges, using log scale. For the ablation studies mentioned below, we sample 10 values of y each. We always sample y = 0 separately and run trial with it.
Baselines were tuned by hand (without regularization) using grad-student descent and small grid searches.
G.2 CLUSTER & USED RESOURCES
We make use of local SLURM cluster (Jette et al., 2002). We run our experiments on GPUs (Geforce RTX 2080 Ti). We estimate reproducing all results would take 94 GPU days.
G.3
COMPARISON OF THE SURROGATE OBJECTIVES
log Var [2]
log Var 2 Y]
E 2
2
X 21
1.5
Train
Set
2-
HzYYZZ
1.5
1P5 35R1
0.5
50
100
300
50
100
300
50
100
300
Information [[
Figure G.1: Information Plane Plot of the latent Z similar Tishby and Zaslavsky (2015) but using ResNet18 model on CIFAR-10 using the different regularizes from section 3.3 (without dropout, but with zero-entropy noise). The dots colored by y. See section 4 for more details.
As can be seen in figure G.2, the different surrogate regularizers have very similar effects on H[Z] and Regularizing with EE IIZII2 shows stronger initial regularization effect, but is difficult to compare quantitatively as its hyperparameter does not map to an equivalent B, unlike regularizing us- ing entropy estimates. Overall, we find log Var[Z]] to provide stable training trajectories (and expected visualizations) while also having more meaningful hyperparameter than IF ||Z[|, though IF [| trivial to implement and communicate"4 log Var[Z] Y] performs worse which we hypothesize is due to the increased variance (given equal batch sizes) from conditioning Y. It further does not minimize the Preserved Information I[X; Z) as strongly as the other regularizers.
H[Z Y]. YY.
G.3.1 MEASUREMENT OF INFORMATION QUANTITIES
Measuring information quantities can be challenging. As mentionend in the introduction, there are many complex ways of measuring entropies and mutual information terms. We can side-step the issue by making of the bounds we have established and the zero-entropy noise we are injecting, and design experiments around that.
First, to estimate the Preserved Information I[X; ZJ, note that when
deterministic model as encoder and only inject zero-entropy noise, we have H|Z| X] = and [[X; [X = I[X; Z] + H[Z X] = H[Z]. We use the entropy estimator from Kraskov al. (2004, equation (20)) to estimate the Encoding
Entropy H[Z] and thus I[X; ZJ.
Whiich is the reason why showcase in figure and in equation (1).
28

---

Under review as a conference paper at ICLR 2021
10 10
Joe Var
lox Var
250
150
2000
1000
100 1800 50 100 1500
150
Figure G.2: Entropy estimates while training with different y and with different surrogate regularizers on CIFAR-10 10 with ResNet18 8 model. Entropies estimated training data based on Kraskov et al. (2004). Qualitatively all three regularizers push H[Z] and H[Z| Y] down. H[Z Y] not shown here because it always stays very close to H[Z]. E ! tends to regularize entropies more strongly for small y. See section for more details.
To estimate the Residual Information I[X; Z2] we similarly note that I[X; Y | |X H[Y| H[Y IZ). Instead of estimating the entropy using Kraskov al. (2004), can the Decoder Cross-Entropy Ho[Y ZJ which provides tighter bound as long as we also minimize Ho[Y |2] part of the training objective.
When we use stochastic models as encoder, we cannot easily compute I[X;Z] anymore. In the ablation study the next section, thus change the X axis accordingly.
Similarly, when we look at the trajectories on the test set instead of the training set, for example in figure G.4, we change the axis signify the Decoder Uncertainty HglY Z]. It is still an aoper-bound, but we do not minimize it directly anymore.
For the plots in figure 6, we retrained the decoder on the test set to obtain tighter bound on H[Y [Z] (while keeping the encoder fixed). We then sampled the latent using the test set to estimate the trajectories. We only did this for the CIFAR-10 model without dropout. For our ablations, we did not retrain the decoder and thus only present plots on the test and training set, respectively.
At this point, it important to recall that the Decoder Uncertainty also the negative log- likelihood (when training with single dropout sample), which provides a different perspective on the plots. It makes it clear that we can see how much model overfits by comparing the best and final epochs of trajectory in the plot (marked by circle and square, respectively).
G.3.2 ABLATION STUDY
We perform ablation study to determine whether injecting noise is necessary. Furthermore, investigate the more interesting case of using a stochastic model as encoder, and we can use stochastic model without injecting zero-entropy noise.
We also investigate whether log Var[Z] Y] performs better when we increase batchsize as we hypoth- esized that batchsize of 128 does not suffice as it leaves only 13 samples per class to approximate H[Z|Y])
Figure G.3 shows larger version of figure 6 for all three regularizers and also training trajectories on the test set. As described in the previous section, this allows us to validate that the regularizers prevent overfitting the training set: with increasing y, the model overfits less.
Figure G.6 and figure G.5 shows that injecting noise is necessary independently of whether we use dropout not. Regularizing with IF. | still has very weak effect. We hypothesize that, similar
29

---

Under review as a conference paper at ICLR 2021
the toy experiment depicted in figure 3, floating-pon point precision issues might provide natural noise source eventually. This would change the effectiveness of y and might require much higher values to observe similar regularization effects as when we do inject zero-entropy noise.
Figure G.4 shows trajectories for a stochastic encoder (as described above with DropConnec/drropout rate 0.1). It overfits less than deterministic one.
Figure G.7 shows the effects of using higher dropout rates (using DropConnect/droppout rates of 0.3/0.5). It overfits less than model with DropCommect/rropout rates of 0.1/0.1.
The plots in figure G.10 show the effects different y with different regularizers more clearly. On both training and test set, one clearly see the effects of regularization.
Overall, log Var[Z Y] performs worse as a regularizer. In figure G.8, we compare the effect of doubling batchsize. Indeed, log Var[Z] Y] performs better with higher batchsize and looks closer to log Var[Z].
G.3.3 COMPARISON BETWEEN DECODER
AND PREDICTION -NOSS-ENTROPY
When training deterministic models or dropout models with single sample (as one usually does), the estimators for both the Decoder Cross-Entropy Hg[Y Z] and the Prediction Cross-Entn Enoss-Entropy H,TY X] coincide. In section 3.2, we discuss the differences from theoretical perspective. Here, we empirically evaluate the difference between optimizing the estimators for each of the two cross- entropy losses, for which will draw multiple dropout samples during training and inference.
We examine models with continuous Z on MNIST and CIFAR-10 10 (Lecun et al., 1998; Krizhevsky al., 2009). Specifically, we standard dropout CNN encoder for MNIST, and modified ResNet18 to which we add DropConnect in each layer for CIFAR-10. We use K = 100 dimensions for the continuous latent Z in the last fully-connected layer, and use a linear decoder to obtain the final 10-dimensional output of class logits. For MNIST, we compute the cross-entropies using 64 dropout samples; for CIFAR-IO, we use 8. For the purpose of this examination of training behavior, it is not necessary to achieve SOTA accuracy: our models obtain 99.2% accuracy on MNIST and 93.6% on CIFAR-10.
Figure G.11 shows the training error probability as well as the value of each cross cross-entropy loss for models trained either with the Decoder Eross-Entropy or the Prediction Cross- Poosss-Ettroy.. The Decoder Cross-Entropy HolY /Z) outperforms Prediction Cross-Entrops Ho[Y X] as a training objective: the training error probability and both coos-eensropies are lower when minimizing H&[Y ZZ] compared to minimizing H X]. We compare only the training, rather than the test, losses of the models to isolate the effect of each loss term on training performance; we leave the prevention of overfitting to the regularization terms considered later. Recently, Dusenberry al. (2020) also observed empirically that the Decoder Cross- -nttropy H₂LY| ZZ] as an objective is both easier to optimize and provides better generalization performance.
G.4 DIFFERENTIAL ENTROPIES AND NOISE
We demonstrate the importance of adding noise continuous latents by constructing a pathological sequence of parameters which attain monotonically improving and unbounded regularized objective values (H[Z)) while all computing the same function. We use MNIST with a standard dropout CNN as encoder, with K = 128 continuous dimensions in Z, and a K x 10 linear layer as decoder. After every training epoch, we decrease the entropy of the latent by normalizing and then scaling the latent to bound the entropy. We multiply the weights of the decoder to not change the overall function. As can be seen in figure 3, without noise, entropy can decrease freely during training without change in error rate until it is affected by floating seating-point issues; while when adding vero-entropy noise, the error rate starts increasing gradually and mcaningfully as the entropy starts to approach zero. We conclude that entropy regularization is meaningful only when noise is added to the latent.
30

---

Under review as a conference paper at ICLR 2021
G.5 COMPARISON BETWEEN DVIB AND SURROGATE OBJECTIVES PERMUTATION-MNIIS
Comparing DVIB and our surrogate objectives is not straightforward because DVIB uses a VAE-like model that explicitly parameterize mean and standard deviation of the latent whereas the stochastic models we focus on in section 3.2 and beyond are implicit by using dropout.
For this comparison, we use the same architecture and optimization strategy for DVIB as described in Alemi et al. (2016): the encoder is ReLU-MLP of the form 796 - 1024 - 1024 - 2K with K=256 latent dimensions that outputs mean and standard deviation explicitly and separately. For the standard deviation, we use softplus transform with bias of -5. We use Polyak averaging with a decay constant 0.999 (Polyak and Juditsky, 1992). We train the model for 200 epochs with Adam with learning rate 10 Bi = 0.5,82 -999 (Kingma and Ba, 2014) and decay the learning rate by 0.97 every 2 epochs. The marginal fixed to unit Gaussian around the origin. We use a softmax layer as decoder. We use 12 latent samples during training and test time.
For our surrogate objectives, we use similar ReLU-MLP the form 796 - 1024 - 1024 - K with K=256 latent dimensions and dropout layers of rate 0.3 after the first and second layer. We use also 12 dropout samples during training and test time. We train for 75 epochs with Adam and learning rate 0.5 x 10 We half the learning rate every time the loss does not decrease for 13 epochs.
We run 5 trials for each experiments. We were not able to reproduce the baseline of an error of 1.13% for 8 10Th from Alemi et al. (2016). We show comparison in figure G.15. Our methods do reach an error of 1.13% overall though, so the simpler surrogate objectives perform as well good or better than DVIB.
From section F.3, we know that DVIB's B would have to be twice the y frm our section 3.3. We can this correspondence in the plot. This also implies that DVIB's ss is not related the IB objective's B from section 3. This makes sense as DVIB arbitrarily fixes the marginal to be a unit Gaussian.
31

---

Under review as a conference paper at ICLR 2021
Test
IP
Figure G.3: Without dropout but with zero-entropy noise: Information Plane Plot of training trajectories for ResNer18 models on CIFAR- 10 and different regularizers The trajectories are colored by their respective y: their transparency changes by epoch. Compression PPeserved Information J) trades-off with performance (Residual Information L). See section 4. The circle marks the final epoch of a trajectory. The square marks the best epoch (Residual Information I...
300
Weight Decay
100
300 50
150
E2
100
100
50
Preserved Information 1|x22]
Epoch
300 50
log Var IZ Y
100
00 1000
0000
300 50
loy. Var 12
100
32

---

Under review as a conference paper at ICLR 2021
Set
Figure G.4: With dropout and with zero- eev-eenropyy noise: Information Plane Plot of training trajectories for ResNerl8 18 models on CIFAR- 10 and different regularizers The trajectories are colored by their respective y: their transparency changes by epoch. Compression (Preserved Information J) trades-off with performance (Residual Information L). See section 4. The circle marks the final epoch of a trajectory. The square marks the best epoch (Residual Information W..
3000
200
50
150
100
300
50
0
200
Encoding Entropy H[[]]
Epoch
long Var 22 V
10-3
100
10-1
10-0
200
Jog Var 22
100
50
Secort reaee Srnroee 1o/Y L
Residu Coommaade PP:. X
33

---

Under review as a conference paper at ICLR 2021
Test Set
Figure G.5: With dropout but without zero pero-ee noise: Information Plane Plot of training trajectories for ResNer18 models on CIFAR- 10 and different regularizers The trajectories are colored by their res respective y: their transparency changes by epoch. Compression (Preserved reservee Information J) trades-off with performance (Residual Information L). See section 4. The circle marks the final epoch of a trajectory. The square marks the best epoch (Residual Information W..
.500
E2
-100
1300
150
100
2003
50
0
Encoding Entrogy H[
Epoch
long Var 1Z V
-200
10-2
10-1
4000
10-0
2013
Jog Var 12
2200
400
0.3
becoder Crose Cennroon Ho/Y [2]
Resdun Fooomatio U| 3
34

---

Under review as a conference paper at ICLR 2021
Train
Set
Figure G.6: Without dropout and without zero-entropy noise: Information Plane Plot of training trajectories for ResNer18 models on CIFAR-I R-1O and different regularizers. The trajectories are colored by their respective y; their transparency changes by epoch. Compression (Preserved Information trades-off with performance (Residual Information L). See section 4. The circle marks the final epoch of a trajectory. The square marks the best epoch (Residual Information W.
-500
E2
1100
1500
20000
150
100
2010
50
Bott
0
Preserved Information 1|
Epach
long Var 22 V
-200
10
10-1
400
10-0
250
log Var 12
250
500
Decoder Crose Cennroo Ho/Y [2)
35

---

Under review as a conference paper at ICLR 2021
Train Set
Test
Figure G.7: With more dropout and zero entropy noise: Information Plane Plot of training trajectories for ResNet18 models on CIFAR 10 and log Var(Z Y] regularizer with batchsizes 128 and 256. The trajectories are colored by their respective y; their transparency changes by epoch. Compression (Preserved Information 1) trades-off with performance (Residual Information J). See section 4. The circle marks the final epoch of trajectory. The square marks the best epoch (Residual
EZ
100
150
100
300
Information I.). DropConnect rate of 0.3 and dropout rate of 0.4 were used instead of 0'0.1 for each.
50
200
Encoding Entropy H[2]
Epoch
V
log Var 2
10
100
300
Jog Var 2
100
50
Sepori Cross rrreee Eenropy 71
Rioomaatnn liv X
36

---

Under review as a conference paper at ICLR 2021
10
10
10
10
50
100
150
Epoch
log Var [2 Y] (Batchsize 128)
log Var [2 Y] (Batchsize 256)
Residual Information I|;;
le:1
Train
16.3
1e-4
Decoder Cross- Entropy Hz]
Test Sec
8
0.3
kg
86
100
300
100
200
300
Preserved Information 1[X;
Figure G.8: Without dropout but with zero-entropy noise: Information Plane Plot of training trajectories for ResNet18 models on CIFAR-10 and log Var[Z Y] regularizer with batchsizes 128 and 256. The trajectories Compression (Preserved Information L) trades-off with performance (Residual Information J). See section 4. The circle marks the final epoch of trajectory. The square marks the best epoch (Residual Information Il).
colored by their respective y; their transpareney changes by epoch.
log Var [Z]
log Var
E 2
2
I| 2
1.5
Train
Set
0
14/Y1/1
1.5
1500 SC5
1
0.5
0
100
300
100
300
50
100
300
Encoding Entropy H[Z]
Figure G.9: Information Plane Plot of the latent Z similar to Tishby and Zaslavsky (2015) but using a ResNet18 model on CIFAR-10 10 using the different regularizes from section 3.3 (with dropout and zero-entropy noise). The dots are colored by y. See section 4 for more details.
37

---

Under review as a conference paper at ICLR 2021
Information Quantities
Decoder Cross- Entropy Ha[Y|2)
Encoding Entropy H[Z]
Var [
Var
Z
100
Train Set
Nats
100
125 392
10
10
10
10
10
(a) With dropout and zero- entrrpy noise.
Information Quantities
Decoder Cross-Entrope Ho [Y Z]
Preserved Information [[x]
log Var
log Var 2 Y
10
Train Set
0.1
Nats
100
S25 1991
10
(b) Without dropout but with zero-entropy noise.
Information Quantities
Decoder Cross. Entropy Hy [Z
Preserved Information [X;2
Ing Var [2]
log Var [ Y
500
PP 1121
1000
-1500
Nats
2200
-500
125 159
1000
10-4
10
(c) Without dropout and without zero-entropy noise.
Figure G.10: Information quantites for different y the end of training for ResNet18 models CIFAR-1O and log Var[Z]] Y) regularizer with batchsizes 128 and 256. Compression (Preserved Information 1) trades-off with performance (Residual Information 1). See section 4.
38

---

Under review as a conference paper at ICLR 2021
Objective
min H2
min HY[Y[X]JJa Metteriii
Pptea)p(y #
H.[Y12]
Ha[Y|X]
Test Set
Train Set
1e-1
1e:3
1e.2
Permutation MNIST
Nats /Error
10.4
1e-6
CIFAR10
0.01
0.001
Epoch
Figure G.11: Training error probability, Decoder Cross- Cross-Entropy H&[Y Z] and Prediction Cross- Entropy HoIY X] with continuous Z. K 100 dimensions are used for Z, we use dropout to obtain stochastic models. Minimizing H&[Y Z] (solid) leads to smaller cross-entropicr and lower training error probability than minimizing H, [X] (dashed). This suggests a better data fit, which what desire for a loss term. We run 8 trials each and plot the median with confidence bounds (25% and 75% quartiles). See section 3.2 and G.3.3 for more details.
39

---

Under review as a conference paper at ICLR 2021
Imagenette
E Z
Residual Information I[Y; ;X 21
0.3
Train Set
0.03
Decoder Cross- Entropy Ho [2]
Test Set
0.5
0.3
70
100
200
Encoding Entropy H[Z]
10
10
10
50
100
150
7
Epoch
Figure 12: Information Plane Plot of the latent similar to Tishby and Zaslavsky (2015) but using ResNet18y 8v2 model on Imagenette using the EE !|Z]? surrogate obejctive from section 3.3 (with dropout and zero-entropy noise). The dots are colored by y.
40

---

Under review as a conference paper at ICLR 2021
E22
Weigt Oicer
033
335
075
2.2
335
025
335
2000
100
01
6100
610
03
2.0
Ezslon
Figure G.13; Adversarial robustness of ResNet18 models trained on CIFAR-10 with surrogate objectives in comparison to regularization with L2 weight-decay as non-IB method for different attack strengths The robustness is evaluated using FGSM, PGD, DeepFool and Basiclterative attacks of varying values. The dashed black line represents model trained only with cross-entropy and noise injection. We see that models trained with the surrogate IB objective (colored by y) see improved robustness over a model trained only to minimize the cross-entropy training objective (shown in black) while the models regularized with weight-decay actually perform worse.
41

---

Under review as a conference paper at ICLR 2021
Figure G. 14: Average adversarial robusiness over e [0, 0.10 of ResNet18 18 models trained on CIFAR- 10 with surrogate objectives in or L2 weight-decayy non IB method) compared to normal accuracy for different amounts of Preserved Information, o markers show robustness, X markers show the normal accuracy. We see that robustness depends on the Preserved Information If the latent is compressesal too much, robustness (and accuracy) are low, If the latent not compressed enough, robustness and thus generalization suffer.
Basiclterative
DespFool
FGSM
PGD
Robustness
300
Weight Decay
300
Preserved Information [[X
50
10-2
10-1
10-6
log: Var Z
100
log Var
0.75
0.25
0
0.75
0.
0
0.75
1.50
0.25
0.75
0.50
25
0.75
0.50
0.25
22

---

Under review as a conference paper at ICLR 2021
log Var [Z]
Z
Regularizer
log Var [ Y]]
DVIB
Test Error
10
10
10
10 2
Figure G.15: Comparison of test error for different Lagrange multiplier for DVIB and surrogate objectives from section 3.3 on Pernutation-MNISS The purple strongly dashed line shows the test error reported for DVIB in Alemi et al. (2016). trials with 95% confidence interval shown. Even though we could not reproduce the baseline reported that paper, the simpler surrogate objective reach at least similar test error reported there. We also see that DVIB behaves similar to IE [|]], but shifted by factor 2 in y, as predicted by section F.3.
43

---

Under review as a conference paper at ICLR 2021
H(ZJY)
Reverse
Decoder
Uncertainty
I(X;Z)
Preserved
Information
Z
Encoding Uncertainty
H(Z|X)
Preserved Relevant Information
Figure H.I: Mickey Mouse I-diagram ram. See figure 2 for details.
I(X;Z)Y) Redundant
Information
H(X
I(Y:Z)
X
Information
Residual
I(X;YIZ)
H(Y Label Uncertainty
H LARGE VERSION МICKEY MOUSE I-DIAGRAM
Y
Uncertainty
Information
H(YZZ) Decoder
I(X;)) Relevant
44


Reviews:
[{"review_id": "ynGuKwN76Ll", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of paper:\nThe authors review the information bottleneck (IB) in the context of deep learning. They discuss the obstacles to applying the IB (and a deterministic variant, the DIB) to modern datasets, review approaches to doing so, and introduce their own scalable approach. Their approach introduces practical surrogate objectives for the information regularizer term, and uses dropout as the source of stochasticity. They take advantage of the scalability of their method to train a ResNet with (D)IB on MNIST, CIFAR-10, and ImageNette and study adversarial robustness and evolution in the information plane.\n\nPros:\n1) The surrogate objectives the authors introduce allow the application of IB without restricting the latent distribution to take on a form with analytic entropy (i.e. gaussian), as is the case the deep variational IB (VIB).\n2) The authors are the first, to my knowledge, to scale the DIB from tabular settings (where it was developed) to modern function approximation settings using a clever zero-entropy noise trick (although this did come at the cost of diverging from the deterministic solutions that would be optimal).\n3) The authors are the first, to my knowledge, to use dropout as the source of stochasticity that IB requires. This has the advantage of allowing the authors to use nearly arbitrary DNN architectures with (D)IB, as opposed to inserting an explicitly stochastic (gaussian) layer.\n4) The paper functions as a good review, independent of the authors’ contributions. A common complaint when reading ML papers is that they don’t discuss related work enough, so this paper was refreshing.\n\nCons:\n1) I struggled to disentangle the novel contributions of the authors from the work they were reviewing. The novel contributions to my knowledge are the first 3 pros above. On the other hand, optimizing decoder uncertainty is not, for example (e.g. it is done by VIB). The authors need to do a better job at highlighting their own contributions but also making clear what is not.\n2) I don’t think the authors make a very compelling explicit case for what advantages their approach has over VIB (the main alternative). I do believe there are advantages (see pros 1-3 above), but they are scattered throughout the paper and not always made explicit. I think the authors need a dedicated subsection addressing this. This section should also highlight the disadvantages (looser bound maybe?).\n3) Relatedly, why not include direct comparisons to VIB in the experiments? The authors seem to imply that VIB wouldn’t scale to the datasets they tackle, since the experiments in the VIB paper involve pretrained embeddings and smaller models. But a) the VIB paper was written 4 years ago and hardware/software has improved since then b) the model sizes are the same order of magnitude.\n\nOther comments:\n1) I thought the heavy use of color distracted more than it helped, though I appreciate the effort.\n2) This paper (https://arxiv.org/abs/1712.09657) also attempted to scale DIB to non-tabular problems (although far from the scalability of DNNs). The authors also added noise, but in this case to the data rather than the latents. Different problem being solved, but possibly interesting connection for the authors.\n\nUPDATE\n\nFollowing the author's response and updated draft, I've raised my score from a 6 to a 7.\n\nUPDATE 2\n\nFollowing discussion among the reviewers and especially a summary of experimental results by Reviewer 3, I'm lowering score back to a 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bZw86dHiMs0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: The authors provide a detailed analysis of the information bottleneck principle to explain how neural networks train and generalise. Specifically, since multiple competing IB objectives exist, the authors develop universal surrogate objectives that are easier to optimise and apply these to several neural network architectures for imaging tasks.\n\nQuality and Clarity: The paper is clearly  well written. I particularly like the use of colour to match corresponding terms in each of the objectives which makes it easy to pin-point which pieces correspond.\n\nSignificance: The IB principle is a very useful and relevant concept for specifically optimising models to retain only the relevant information wrt a particular context or prediction task. It has been applied in several contexts eg deep generative models for identifying novel molecule structures (Wieser et al 2020) or deducing sufficient adjustment sets for causal inference (Parbhoo et al 2020) as well as DNNs in general (Tisbhy and Zaslavksy, 2015). Since it relies entirely on information theoretic quantities, it is widely applicable across several domains. Analysing these information theoretic objectives in order to make sense of these models is very important.\n\nPros:  1) The work presents a very rigorous analysis and discussion of multiple competing IB objectives and discusses the implications of each of these.\n\n2) The authors present tractable surrogate objectives that can make optimisation easier. Since these are defined entirely in terms of entropies as well, the work is applicable across various kinds of domains -- a key advantage of the classic IB too.\n\nCons:\n\n1) I think the authors do a good job in theoretically motivating the particular surrogate objectives, but I would have liked to have seen some discussion as to why using a surrogate objective is sensible in the first place, versus say performing comparisons of deep IBs and VAEs. VAEs use maximum marginal likelihood as an objective. How does using the surrogate objective compare to maximum marginal likelihood? What are the implications of this for downstream application of the IB? Should IBs with surrogate objectives only be used for compression or also for prediction tasks like VAEs?\n\n\n2) I would have also like to have seen empirically how the surrogate objectives can generalise across domains that are not images? Are these objectives robust in other applications such as EHR data or say the chemical/molecular domain?\n\n3) I also think the generalisation plots are not the most intuitive to understand from first glance and require more parsing and explanation in the text.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OHiT4xE5n29", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper makes a theoretical contribution of three \"surrogate objectives\" for the information bottleneck principle, followed by empirical results on MNIST, CIFAR-10 and ImageNette (a subset of 10 easily classified classes from ImageNet). The objectives assume we add a single simple of zero-entropy noise to each sample of the output z of the stochastic encoder p(z|x), and then give estimators on an upper bound of the information bottleneck objective.\n\nEvaluation:\nOverall, this is a fine paper - the introduction is especially well-written and I appreciated the inclusion of all the information plane images of training trajectories. However, the contributions are not sufficiently novel for acceptance at ICLR; this paper is mostly a duplicate of prior work in this area.\n\nI am not convinced that their theoretical contribution is novel - it seems to be a variant (or a specific case) of prior work on Conditional Entropy Bottleneck (CEB) given in Fischer 2020 (https://arxiv.org/pdf/2002.05379.pdf). Their initial insight (Proposition 1) that recognizes that the information bottleneck objective I(XZ) - beta * I(YZ) can be rewritten as H(Y|Z) + beta' * I(XZ|Y), is exactly the insight given in CEB. This paper bounds the I(XZ|Y) term by assuming Z has zero-mean Gaussian noise (which can be chosen such that it is also zero-entropy noise). In contrast, the CEB paper gives a variational bound on the rewritten objective, and when optimizing this bound you sample from the encoder and use the samples to parameterize a distribution (where a gaussian is the simplest choice of distribution). It seems like this paper is producing a special case of CEB for gaussian assumptions on that distribution family.\n\nIn addition, the empirical contributions are decidedly not novel. They claim to present \"the first successful evaluation of IB objectives on CIFAR-10 and ImageNette\", but prior work contains these evaluations: Fischer 2020 (linked above) contains CIFAR-10 results and Fischer and Alemi 2020 (https://arxiv.org/pdf/2002.05380.pdf) contains robustness results on CIFAR-10 and ImageNet, on larger ResNets than the experiments in this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JgTHb0uTE5E", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides several surrogates for the Information Bottleneck (IB) and Deterministic Information Bottleneck (DIB) loss functions that are more friendly to optimization. For the decoder uncertainty part, the authors show that using Dropout and cross-entropy loss provides an unbiased estimator for the decoder cross-entropy which upperbounds the decoder uncertainty. For the regularization terms in IB/DIB, the authors inject noises to the latent features to lower-bound the conditional entropy of latent representations, and further proposes three types of surrogate objectives for the regularziation terms. Emprical results on CIFAR/ImageNette (a subset of ImageNet of 10 classes) show that the proposed surrogates yield similar behaviours in terms of adversarial robustness and information plane and the scalability of the proposed method.\n\nStrengths of the paper:\n- As this paper claims, this is the first work that proposes some surrogate of IB loss functions that can be easily optimized and thus be scaled to large models and datasets (CIFAR/ImageNette). Results on both datasets show similar behavior (adversarial robustness, two-phase information plane) to IB loss based optimization.\n- The injection of random noises into the latent representation is interesting and able to enforce lower-bound on the conditional entropy of latent representations, which further induces some surrogates that are optimization-friendly.\n- This paper is well-written, fully-prepared and contains a large amount of results that are of wide interests of researchers working in this topic.\n\nI don't have specific criticisms for this paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We have a very well informed reviewer who strongly feels that this paper is insufficiently novel and significant further discussion on how the paper might be raised to a publishable level with more empirical results.  I will have to side with the more engaged reviewers who feel that the paper should be rejected.
Decision: Reject