Title: THE VARIATIONAL DEFICIENCY BOTTLENECK

Abstract: We introduce a bottleneck method for learning data representations based on channel deficiency, rather than the more traditional information sufficiency. A variational upper bound allows us to implement this method efficiently. The bound itself is bounded above by the variational information bottleneck objective, and the two methods coincide in the regime of single-shot Monte Carlo approximations. The notion of deficiency provides a principled way of approximating complicated channels by relatively simpler ones. The deficiency of one channel w.r.t. another has an operational interpretation in terms of the optimal risk gap of decision problems, capturing classification as a special case. Unsupervised generalizations are possible, such as the deficiency autoencoder, which can also be formulated in a variational form. Experiments demonstrate that the deficiency bottleneck can provide advantages in terms of minimal sufficiency as measured by information bottleneck curves, while retaining a good test performance in classification and reconstruction tasks.

1. INTRODUCTION:

The information bottleneck (IB) is an approach to learning data representations based on a notion of minimal sufficiency. The general idea is to map an input source into a representation that retains as little information as possible about the input (minimality), but retains as much information as possible in relation to a target variable of interest (sufficiency). See Figure 1 . For example, in a classification problem, the target variable could be the class label of the input data. In a reconstruction problem, the target variable could be a denoised reconstruction of the input. Intuitively, a representation which is minimal in relation to a given task, will discard nuisances in the inputs that are irrelevant to the task, and hence distill more meaningful information and allow for a better generalization.

In a typical bottleneck paradigm, an input variable X is first mapped to an intermediate representation variable Z, and then Z is mapped to an output variable of interest Y . We call the mappings, resp., a representation model (encoder) and an inference model (decoder). The channel κ models the true relation between the input X and the output Y . In general, the channel κ is unknown, and only accessible through a set of examples (x (i) , y (i) ) N i=1 . We would like to obtain an approximation of κ using a probabilistic model that comprises of the encoder-decoder pair.

The IB methods (Witsenhausen & Wyner, 1975; Tishby et al., 1999; Harremoës & Tishby, 2007; Hsu et al., 2018) have found numerous applications, e.g., in representation learning, clustering, classification, generative modeling, model selection and analysis in deep neural networks, among others (see, e.g., Shamir et al., 2008; Gondek & Hofmann, 2003; Higgins et al., 2017; Alemi et al., 2018; Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017) .

In the traditional IB, minimality and sufficiency are measured in terms of the mutual information. Computing the mutual information can be challenging in practice. Various recent works have formulated more tractable functions by way of variational bounds on the mutual information (Chalk et al., 2016; Alemi et al., 2016; Kolchinsky et al., 2017) , sandwiching the objective function of interest.

Instead of maximizing the sufficiency term of the IB, we formulate a new bottleneck method that minimizes deficiency. Deficiencies provide a principled way of approximating complex channels

X Y Z κ e d

Figure 1 : The bottleneck paradigm: The general idea of a bottleneck method is to first map an input X to an intermediate representation Z, and then map Z to an output Y . We call the mappings, resp., an encoder (e) and a decoder (d). In general, the true channel κ is unknown, and only accessible through a set of training examples. We would like to obtain an approximation of κ.

by relatively simpler ones. The deficiency of a decoder with respect to the true channel between input and output variables quantifies how well any stochastic encoding at the decoder input can be used to approximate the true channel. Deficiencies have a rich heritage in the theory of comparison of statistical experiments (Blackwell, 1953; Le Cam, 1964; Torgersen, 1991) . From this angle, the formalism of deficiencies has been used to obtain bounds on optimal risk gaps of statistical decision problems. As we show, the deficiency bottleneck minimizes a regularized risk gap. Moreover, the proposed method has an immediate variational formulation that can be easily implemented as a modification of the Variational Information Bottleneck (VIB) (Alemi et al., 2016) . In fact, both methods coincide in the limit of single-shot Monte Carlo approximations. We call our method the Variational Deficiency Bottleneck (VDB).

Perfect maximization of the IB sufficiency corresponds to perfect minimization of the DB deficiency. However, when working over a parametrized model and adding the bottleneck regularizer, both methods have different preferences, with the DB being closer to the optimal risk gap. Experiments on basic data sets show that the VDB is able to obtain more compressed representations than the VIB while performing equally well or better in terms of test accuracy.

We describe the details of our method in Section 2. We elaborate on the theory of deficiencies in Section 3. Experimental results with the VDB are presented in Section 4.

2. THE VARIATIONAL DEFICIENCY BOTTLENECK (VDB):

Let X denote an observation or input variable and Y an output variable of interest. Let p(x, y) = π(x)κ(y|x) be the true joint distribution, where the conditional distribution or channel κ(y|x) describes how the output depends on the input. We consider the situation where the true channel is unknown, but we are given a set of N independent and identically distributed (i.i.d.) samples (x (i) , y (i) ) N i=1 from p. Our goal is to use this data to learn a more structured version of the channel κ, by first "compressing" the input X to an intermediate representation variable Z and subsequently mapping the representation back to the output Y . The presence of an intermediate representation can be regarded as a bottleneck, a model selection problem, or as a regularization strategy.

We define a representation model and an inference model using two parameterized families of channels e(z|x) and d(y|z). We will refer to e(z|x) and d(y|z) as an encoder and a decoder. The encoder-decoder pair induces a model κ(y|x) = d(y|z)e(z|x)dz. Equivalently, we write κ = d•e.

Given a representation, we want the decoder to be as powerful as the original channel κ in terms of ability to recover the output. The deficiency of a decoder d w.r.t. κ quantifies the extent to which any pre-processing of d (by way of randomized encodings) can be used to approximate κ (in the KL-distance sense). Let M(X ; Y) denote the space of all channels from X to Y. We define the deficiency of d w.r.t. κ as follows. Definition 1. Given the channel κ ∈ M(X ; Y) from X to Y , and a decoder d ∈ M(Z; Y) from some Z to Y , the deficiency of d w.r.t. κ is defined as

EQUATION

Here (Csiszár & Körner, 2011) , and π is an input distribution over X . The definition is similar in spirit to Lucien Le Cam's notion of weighted defi-ciencies of one channel w.r.t. another (Le Cam, 1964; Torgersen, 1991, Section 6.2 ) and its recent generalization by Raginsky (2011) .

D KL (• • |•) is the conditional KL divergence

We propose to train the model by minimizing the deficiency of d w.r.t. κ subject to a regularization that penalizes complex representations. The regularization is achieved by limiting the rate I(Z; X), the mutual information between the representation and the raw inputs. We call our method the Deficiency Bottleneck (DB). The DB minimizes the following objective over all tuples (e ∈ M(X ; Z), d ∈ M(Z; Y)):

L DB (e, d) := δ π (d, κ) + βI(Z; X).

(2)

The parameter β ≥ 0 allows us to adjust the level of regularization.

For any distribution r(z), the rate term admits a simple variational upper bound (Csiszár & Körner, 2011, Eq. (8.7 )):

EQUATION

Let pdata be the empirical distribution of the data (input-output pairs). By noting that δ π (d, κ) ≤ D KL (κ d•e|π) for any e ∈ M(X ; Z), and ignoring (unknown) data-dependent constants, we obtain the following optimization objective which we call the Variational Deficiency Bottleneck (VDB) objective:

EQUATION

)

The computation is simplified by defining r(z) to be a standard multivariate Gaussian distribution N (0, I), and using an encoder of the form e(z|x) = N (z|f φ (x)), where f φ is a neural network that outputs the parameters of a Gaussian distribution. Using the reparametrization trick (Kingma & Welling, 2013; Rezende et al., 2014) , we then write e(z|x)dz = p( )d , where z = f (x, ε) is a function of x and the realization of a standard normal distribution. This allows us to do stochastic backpropagation through a single sample z. The KL term admits an analytic expression for a choice of Gaussian r(z) and encoders. We train the model by minimizing the following empirical objective:

EQUATION

For training, we choose a mini-batch size of N = 100. For Monte Carlo estimates of the expectation inside the log, we choose M = 3, 6, 12 samples from the encoding distribution.

We note that the Variational Information Bottleneck (VIB) (Alemi et al., 2016) leads to a similarlooking objective function, with the only difference that the sum over j is outside of the log. By Jensen's inequality, the VIB loss is an upper bound to our loss. If one uses a single sample from the encoding distribution (i.e., M = 1), the VDB and the VIB objective functions coincide.

The average log-loss and the rate term in the VDB objective equation 4 are the two fundamental quantities that govern the probability of error when the model is a classifier. For a discussion of these relations, see Appendix A.

3. BLACKWELL SUFFICIENCY AND CHANNEL DEFICIENCY:

In this section, we discuss an intuitive geometric interpretation of the deficiency in the space of probability distributions over the output variable. We also give an operational interpretation of the deficiency as a deviation from Blackwell sufficiency (in the KL-distance sense). Finally, we discuss its relation to the log-loss.

3.1. DEFICIENCY AND DECISION GEOMETRY:

We first formulate the learning task as a decision problem. We show that δ π (d, κ) quantifies the gap in the optimal risks of decision problems when using the channel d rather than κ.

Let X , Y denote the space of possible inputs and outputs. In the following, we assume that X and Y are finite. Let P Y be the set of all distributions on Y. For every x ∈ X , define κ x ∈ P Y as κ x (y) = κ(y|x), ∀y ∈ Y. Nature draws x ∼ π and y ∼ κ x . The learner observes x and quotes a distribution q x ∈ P Y that expresses her uncertainty about the true value y. The quality of a quote q x in relation to y is measured by an extended real-valued loss function called the score : Y ×P Y → R.

For a background on such special kind of loss functions see, e.g., Grünwald et al., 2004; Gneiting & Raftery, 2007; Parry et al., 2012. Ideally, the quote q x should to be as close as possible to the true conditional distribution κ x . This is achieved by minimizing the expected loss L(κ x , q x ) := E y∼κx (y, q x ), for all x ∈ X . The score is called proper if κ x ∈ arg min qx∈P Y L(κ x , q x ). Define the Bayes act against κ x as the optimal quote

q * x := arg min qx∈P Y L(κ x , q x ).

If multiple Bayes acts exist then select one arbitrarily. Define the Bayes risk for the distribution

p XY (x, y) = π(x)κ(y|x) as R(p XY , ) := E x∼π L(κ x , q * x ).

A score is strictly proper if the Bayes act is unique. An example of a strictly proper score is the log-loss function defined as L (y, q) := -log q(y). For the log-loss, the Bayes act is q *

x = κ x and the Bayes risk is just the conditional entropy

R(p XY , L ) = E x∼π E y∼κx -log q * x (y) = E x∼π E y∼κx -log κ x (y) = H(Y |X). (6)

Given a representation z ∈ Z (output by some encoder), when using the decoder d, the learner is constrained to quote a distribution from a subset of

EQUATION

q *

x Z has an interpretation as the reverse I-projection of κ x to the convex set of probability measures C ⊂ P Y (Csiszár & Matuš, 2003) 1 . We call the associated Bayes risk as the projected Bayes risk R Z (p XY , L ) and the associated conditional entropy as the projected conditional entropy

EQUATION

)

The gap in the optimal risks, ∆R := R Z (p XY , L ) -R(p XY , L ) when making a decision based on an intermediate representation and a decision based on the input data is just the deficiency. This follows from noting that

EQUATION

)

∆R vanishes if and only if the optimal quote against d z , q * x Z matches κ x for all x, y. This gives an intuitive geometric interpretation of a vanishing deficiency in the space of distributions over Y.

Given a decoder channel d, since δ π (d, κ) ≤ D KL (κ d • e|π) for any e ∈ M(X ; Z), the loss term in the VDB objective is a variational upper bound on the projected conditional entropy H Z (Y |X). However, this loss is still a lower bound to the standard cross-entropy loss in the VIB objective (Alemi et al., 2016) , i.e.,

EQUATION

)

This follows simply from the convexity of the negative logarithm function.

3.2. DEFICIENCY AS A KL-DISTANCE FROM INPUT-BLACKWELL SUFFICIENCY:

In a seminal paper David Blackwell (1953) asked the following question: if a learner wishes to make an optimal decision about some target variable of interest and she can choose between two channels with a common input alphabet, which one should she prefer? She can rank the channels by comparing her optimal risks: she will always prefer one channel over another if her optimal risk when using the former is at most that when using the latter for any decision problem. She can also rank the variables purely probabilistically: she will always prefer the former if the latter is an output-degraded version of the former, in the sense that she can simulate a single use of the latter by randomizing at the output of the former. Blackwell showed that these two criteria are equivalent.

Very recently, Nasser (2017) asked the same question, only now the learner has to choose between two channels with a common output alphabet. Given two channels, κ ∈ M(X ; Y) and d ∈ M(Z; Y), we say that κ is input-degraded from d and write d Y κ if κ = d • e for some e ∈ M(X ; Z). Stated in another way, d can be reduced to κ by applying a randomization at its input. Nasser (2017) gave a characterization of input-degradedness that is similar to Blackwell's theorem (Blackwell, 1953) . We say,

d is input-Blackwell sufficient for κ if d Y κ.

Input-Blackwell sufficiency induces a preorder on the set of all channels with the same output alphabet. In practice, most channels are uncomparable, i.e., one cannot be reduced to another by a randomization. When such is the case, the deficiency quantifies how far the true channel κ is from being a randomization (by way of all input encodings) of the decoder d. See Appendix B for a brief summary of Blackwell-Le Cam theory.

3.3. DEFICIENCY AND THE LOG-LOSS:

When Y -X -Z is a Markov chain, the conditional mutual information I(Y ; X|Z) is the Bayes risk gap for the log-loss. This is apparent from noting that

I(Y ; X|Z) = H(Y |Z) -H(Y |XZ) = H(Y |Z) -H(Y |X) = R(p ZY , L ) -R(p XY , L )

. This risk gap is closely related to Blackwell's original notion of sufficiency. Since the log-loss is strictly proper, a vanishing I(Y ; X|Z) implies that the risk gap is zero for all loss functions. This suggests that minimizing the log-loss risk gap under a suitable regularization constraint is a potential recipe for constructing representations Z that are approximately sufficient for X w.r.t. Y , since in the limit when I(Y ; X|Z) = 0 one would achieve I(Y ; Z) = I(Y ; X). This is indeed the basis for the IB algorithm (Tishby et al., 1999) and its generalization, clustering with Bregman divergences (Banerjee et al., 2005; van Rooyen & Williamson, 2015; 2014) .

One can also approximate a sufficient statistic by minimizing deficiencies instead. This is motivated from noting the following proposition.

Proposition 2. When Y -X -Z is a Markov chain, δ π (d, κ) = 0 ⇐⇒ I(Y ; X|Z) = 0.

In general, for the bottleneck paradigms involving the conditional mutual information (IB) and the deficiency (DB), we have the following relationship: 

Our experiments corroborate that for achieving the same level of sufficiency, one needs to store less information about the input X when minimizing the deficiencies than when minimizing the conditional mutual information.

4. EXPERIMENTS:

We present some experiments on the MNIST dataset (LeCun & Cortes, 2010) . Classification on MNIST is a very well studied problem. The main objective of our experiments is to evaluate the information-theoretic properties of the representations learned by the VDB and whether it can match the classification accuracy provided by other bottleneck methods.

For the encoder, we use a fully connected feedforward network with 784 input units-1024 ReLUs-1024 ReLUs-512 linear output units. The deterministic output of this network is interpreted as the vector of means and variances of a 256 dimensional Gaussian distribution. The decoder is a simple logistic regression model with a softmax layer. These are the same settings of the model used by Alemi et al. (2016) . We implement the algorithm in Tensorflow and train for epochs using the Adam optimizer.

As can be seen from the upper left panel in Figure 2 , the test accuracy is stable with increasing M .

Here, M is the number of encoder output samples used in the training objective. We note that M = 1 is just the VIB model (Alemi et al., 2016) . L is the number of encoder output samples used for evaluating the classifier (i.e., we use 1 L L j=1 d(y|z (j) ) where z (j) ∼ e(z|x)). Numerical values of the test accuracies are provided in Table 1 for different values of β and M . We see a slight improvement in the test accuracies for higher values of M . See Figure 5 for train and test accuracies for L = 3 and L = 12 in Appendix D. The traditional IB paradigm traces the mutual information I(Z; Y ) between representation and output (sufficiency) vs. the mutual information I(Z; X) between representation and input (minimality), for different values of the regularization parameter β. This curve is called the information bottleneck curve (Tishby et al., 1999) . In the case of the VDB, we define the corresponding sufficiency term as

J(Z; Y ) := H(Y ) -E (x,y)∼ pdata -log( d(y|z)e(z|x) dz) .

Here H(Y ) = log 2 (10) is the entropy of the output which has 10 classes. In our method, "more informative" means "less deficient". The upper right panel in Figure 2 shows the deficiency bottleneck curve which traces J(Z; Y ) vs. I(Z; X) for different values of β at the end of training. For orientation, lower values of β have higher values of I(Z; X) (towards the right of the plot). For small values of β, when the effect of the regularization is negligible, the bottleneck allows more information from the input through the representation. In this case, J(Z; Y ) increases on the training set, but not necessarily on the test set. This is manifest in the gap between the train and test curves indicative of a degradation in generalization. For intermediate values of β, the gap is smaller for larger values of M (our method). The lower right panel plots the corresponding information bottleneck curve. The lower left panel in Figure 2 plots the minimality term I(Z; X) vs. β. We see that, for β in the range between 10 -8 and 10 -4 , for the same level of sufficiency, setting M = 12 consistently achieves more compression of the input compared to the setting M = 1. The dynamics of the information quantities during training are also interesting. We provide figures on these in Appendix D.

In order to visualize the representations, we also train the VDB on MNIST with a 2 dimensional representation. We use the same settings as before, with the only difference that the dimension of 

5. DISCUSSION:

We have formulated a bottleneck method based on channel deficiencies. The deficiency of a decoder with respect to the true channel between input and output quantifies how well a randomization at the decoder input (by way of stochastic encodings) can be used to simulate the true channel. The VDB has a natural variational formulation which recovers the VIB in the limit of a single sample of the encoder output. Experiments demonstrate that the VDB can learn more compressed representations while retaining the same discriminative capacity. The method has a statistical decision-theoretic appeal. Moreover, the resulting variational objective of the DB can be implemented as an easy modification of the VIB, with little to no computational overhead.

Given two channels that convey information about a target variable of interest, two different notions of deficiencies arise, depending on whether the target resides at the common input or the common output of the given channels. When the target is at the common output of the two channels, as is in a typical bottleneck setting (see Figure 1 ), our Definition 1 has a natural interpretation as a KL-divergence from input-Blackwell sufficiency (Nasser, 2017) . Here sufficiency is achieved by applying a randomization at the input of the decoder with the goal of simulating the true channel. The notion of input-Blackwell sufficiency contrasts with Blackwell's original notion of sufficiency (Blackwell, 1953) in the sense that Blackwell's theory compares two channels with a common input. One can again define a notion of deficiency in this setting (see Appendix B for a discussion on deficiencies in the classical Blackwell setup). The associated channels (one from Y to Z and the other from Y to X ) do not however have a natural interpretation in a typical bottleneck setting. In contrast, the input-Blackwell setup appears to be much more intuitive in this context.

The more detailed view of information emerging from this analysis explains various effects and opens the door to multiple generalizations. In the spirit of the VDB, one can formulate a deficiency autoencoder as well (see sketch in Appendix E). On a related note, we mention that the deficiency is a lower bound to a quantity called the Unique information (Bertschinger et al., 2014; Banerjee et al., 2018a) (see details in Appendix C). An alternating minimization algorithm similar in spirit to the classical Blahut-Arimoto algorithm (Blahut, 1972) has been proposed to compute this quantity (Banerjee et al., 2018b) . A deep neural network implementation of such an algorithm remains a challenge. In the limit β → 0, the VDB is a step forward towards estimating the unique information. This might be of independent interest in improving the practicality of the theory of information decompositions.

It follows from the data processing inequality that µ Y κ =⇒ µ mc Y κ. However, the converse implication is not true in general (Körner & Marton, 1975) .

The converse to the Blackwell's theorem states that if the relation Z Y X does not hold, then there exists a set of actions A and a loss function (y, a) ∈ R Y×A such that R(π Y , κ, ) < R(π Y , µ, ). Le Cam introduced the concept of a deficiency of µ w.r.t. κ to express this deficit in optimal risks (Le Cam, 1964) in terms of an approximation of κ from µ via Markov kernels. Definition 6. The Le Cam deficiency of µ w.r.t. κ is

EQUATION

where • TV denotes the total variation distance.

When the distribution of the common input to the channels is fixed, one can define a weighted deficiency (Torgersen, 1991 , Section 6.2).

Definition 7. Given Y ∼ π Y , the weighted Le Cam deficiency of µ w.r.t. κ is

EQUATION

Le Cam's randomization criterion (Le Cam, 1964) shows that deficiencies quantify the maximal gap in the optimal risks of decision problems when using the channel µ rather than κ.

Theorem 8 (Le Cam (1964) ). Fix µ ∈ M(Y; Z), κ ∈ M(Y; Z) and a probability distribution π Y on Y and write ∞ = max y,a (y, a). For every > 0,

δ π (µ, κ) < if and only if R(π Y , µ, ) - R(π Y , κ, ) <

∞ for any set of actions A and any bounded loss function .

Raginsky (2011) introduced a broad class of deficiency-like quantities using the notion of a generalized divergence between probability distributions that satisfies a monotonicity property w.r.t. data processing. The family of f -divergences due to Csiszár belongs to this class (Liese & Vajda, 2006) .

Definition 9. The f -deficiency of µ w.r.t. κ is

EQUATION

Many common divergences, such as the KL divergence, the reverse-KL divergence, and the total variation distance are f -divergences. When the channel µ is such that its output is constant, no matter what the input, the corresponding f -deficiency is called f -informativity (Csiszár, 1972) . The f -informativity associated with the KL divergence is just the channel capacity which has a geometric interpretation as an "information radius" (Csiszár & Körner, 2011) .

We can also define a weighted f -deficiency of µ w.r.t. κ.

Definition 10. The weighted f -deficiency of µ w.r.t. κ is

EQUATION

Specializing to the KL divergence, we have the following definition.

Definition 11. The weighted output deficiency of µ w.r.t. κ is

EQUATION

where the subscript o in δ π o emphasizes the fact that the randomization is at the output of the channel µ.

Note that δ π o (µ, κ) = 0 if and only if Z Y X, which captures the intuition that if δ π o (µ, κ) is small, then X is approximately output-degraded from Z w.r.t. Y . Using Pinsker's inequality, we have

EQUATION

From an information decomposition perspective, the original IB is actually minimizing just the unique information subject to a regularization constraint. This is a simple consequence of the fact that the synergistic information CI(Y ; X, Z) = 0 (see equation 25d) when we have the Markov chain condition Y -X -Z. Hence, one might equivalently call the original IB as the Unique information bottleneck. 

Appealing to classical:



:

Such a projection exists and is not necessarily unique. If nonunique, we arbitrarily select one of the minimizers as the Bayes act.

Figures and Tables:

Figure 2: Effect of the regularization parameter β. The upper left panel shows the accuracy on train and test data after training the VDB for different values of M . Here, M is the number of encoder output samples used in the training objective. L is the number of encoder output samples used for evaluating the classifier. The upper right panel traces the deficiency bottleneck curve for different values of β (see text). The curves are averages over 5 repetitions of the experiment. Each curve corresponds to one value of M = 1, 3, 6, 12. Notice the generalization gap for small values of β (towards the right of the plot). The lower right panel plots the corresponding information bottleneck curve. The lower left panel plots the minimality term vs. β. Evidently, the levels of compression vary depending on M . Higher values of M (our method) lead to a more compressed representation. For M = 1, the VDB and the VIB models coincide.

Figure3: We trained the VDB on MNIST with the basic encoder given by a fully connected network with two hidden layers of ReLUs generating the means and variances of 2D independent Gaussian latent representation. Ellipses represent the posterior distributions of 1000 input images in latent space after training with β = 10 0 , 10 -1 , 10 -3 , 10 -5 and M = 1, 3, 6, 12. Color corresponds to the class label.

Figure 4: Learning curves for MNIST, where the encoder is a MLP of size 784-1024-1024-2K, the last layer being a K = 2 dimensional diagonal Gaussian. The decoder is simply a softmax with 10 classes. The left panel plots the mutual information between the representation and the class label, I(Z; Y ), against the mutual information between the representation at the last layer of the encoder and the input, I(Z; X), as training progresses. The former increases monotonically, while the latter increases and then decreases. The right panel shows the test accuracy as training progresses.

Blackwell theory, Bertschinger et al. (2014)  defined a nonnegative decomposition of the mutual information I(Y ; XZ) based on the idea that the unique and shared information should depend only on the pairwise marginals (Y, X) and (Y, Z). Definition 12. Let (Y, X, Z) ∼ P , Y ∼ π Y and let κ ∈ M(Y; X ), µ ∈ M(Y; X ) be two channels with the same input alphabet such that P Y X (y, x) = π Y (y)κ y (x) and P Y Z (y, z) = π Y (y)µ y (z). Define∆ P = Q ∈ P Y×X ×Z : Q Y X (y, x) = π Y (y)κ y (x), Q Y Z (y, z) = π Y (y)µ y (z) , (28a) U I(Y ; X\Z) = min Q∈∆ P I Q (Y ; X|Z),(28b)U I(Y ; Z\X) = minQ∈∆ P I Q (Y ; Z|X),(28c)SI(Y ; X, Z) = I(Y ; X) -U I(Y ; X\Z), (28d) CI(Y ; X, Z) = I(Y ; X|Z) -U I(Y ; X\Z),(28e)where the subscript Q in I Q denotes that joint distribution on which the quantities are computed. The functions U I, SI, and CI are nonnegative and satisfy equation 24. Furthermore, U I and SI depend on the marginal distributions of the pairs (Y, X) and (Y, Z). Only the function CI depends on the full joint distribution P . U I satisfies the following intuitive property in relation to Blackwell's theorem 4. Proposition 13. (Bertschinger et al., 2014, Lemma 6) U I(Y ; X\Z) = 0 ⇐⇒ Z Y X. Proposition 2 follows from noting that δ π (d, κ) = 0 ⇐⇒ U I(Y ; X\Z) = 0 (Bertschinger et al., 2014, Theorem 22) and the fact that U I(Y ; X\Z) = I(Y ; X|Z) when Y -X -Z is a Markov chain.

Figure 5: Train and test accuracy of the VDB for L = 3 and L = 12. Similar to Figure 2.

Figure 6: Evolution of the mutual information between representation and output vs. representation and input (values farther up and to the left are better) over 200 training epochs (dark to light color) on MNIST. The curves are averages over 20 repetitions of the experiment. At early epochs, training mainly effects fitting of the input-output relationship and an increase of I(Z; Y ). At later epochs, training mainly effects a decrease of I(Z; X), which corresponds to the representation increasingly discarding information about the input. An exception is when the regularization parameter β is very small. In this case the representation captures more information about the input, and longer training decreases I(Z; Y ), which is indicative of overfitting to the training data. Higher values of M lead to the representation capturing more information about the target, while at the same time discarding more information about the input. M = 1 corresponds to the Variational Information Bottleneck.

Comparison of test accuracy values for different values of β and M . K is the size of the bottleneck and L =12. We see a slight improvement in the test accuracies for higher values of M .

Table content: <table><tr><td>β</td><td>K</td><td/><td>M</td></tr><tr><td/><td/><td>1</td><td>3</td><td>6</td><td>12</td></tr><tr><td colspan="5">10 -5 256 0.9869 0.9873 0.9885 0.9878</td></tr><tr><td/><td>2</td><td colspan="3">0.9575 0.9678 0.9696 0.9687</td></tr><tr><td colspan="5">10 -3 256 0.9872 0.9879 0.9875 0.9882</td></tr><tr><td/><td>2</td><td colspan="3">0.9632 0.9726 0.9790 0.9702</td></tr></table>

Peer review (iclr2019):
 The paper presents a method of learning representations that is based on minimizing "deficiency" rather than optimizing for information sufficiency. While perfect optimization of the sufficiency term in IB is equivalent to minimizing deficiency, the thesis of the paper is that the variational upper bound on deficiency is easier to optimize, and when optimized produces better (more compressed representations), while performing equally on test accuracy. The paper is well written and easy to read. The idea behind the paper (optimizing for minimizing deficiency instead of sufficiency in IB) is interesting, especially because the variational formulation of DB is a generalization of VIB (in that VIB reduces to VDB for M=1). What takes away from the paper is that while perfect optimization of IB/sufficiency is equivalent to perfect optimization of DB, it is not clear what happens when perfection is not achieved. Further, the authors claim that DB is able to obtain more compressed representations (But is the goal a compressed representation, or an informative one?). The paper would also benefit from evaluation of the representation itself, and comparison to other non-information bottleneck based algorithms.