MULTI-VIEW INFORMATION BOTTLENECK
Year: 2020

Abstract:
The information bottleneck principle provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label while minimizing the amount of other, excess information in the representation. The original formulation, however, requires labeled data to identify the superfluous information. In this work, we extend this ability to the multi-view unsupervised setting, where two views of the same underlying entity are provided but the label is unknown. This enables us to identify superfluous information as that not shared by both views. A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to common unsupervised approaches for representation learning.

heading: 1 INTRODUCTION
text: The goal of deep representation learning (LeCun et al., 2015) is to transform a raw observational input, x, into a, typically lower-dimensional, representation, z, that contains the information relevant for a given task or set of tasks. Significant progress has been made in deep learning via supervised representation learning, where the labels, y, for the downstream task are known while p(y|x) is learned directly (Sutskever et al., 2012; Hinton et al., 2012). Due to the cost of acquiring large labeled datasets, a recently renewed focus on unsupervised representation learning seeks to generate representations, z, that are useful for a wide variety of different tasks where little to no labeled data is available (Devlin et al., 2018; Radford et al., 2019).
Our work is based on the information bottleneck principle (Tishby et al., 2000) where a representation becomes less affected by nuisances by discarding all information from the input that is not useful for a given task, resulting in increased robustness. In the supervised setting, one can directly apply the information bottleneck principle by minimizing the mutual information between the data x and its representation z, I(x; z), while simultaneously maximizing the mutual information between z and the label y (Alemi et al., 2017). In the unsupervised setting, discarding only superfluous information is more challenging, as without labels the model cannot directly identify which information is relevant. Recent literature (Devon Hjelm et al., 2019; van den Oord et al., 2018) has focused on the InfoMax objective maximizing I(x, z) instead of minimizing it, to guarantee that all the predictive information is retained by the representation, but doing nothing to discard the irrelevant information.
In this paper, we extend the information bottleneck method to the unsupervised multi-view setting. To do this, we rely on a basic assumption of the multi-view literature – that each view provides the same task-relevant information (Zhao et al., 2017). Hence, one can improve generalization by discarding all the information not shared by both views from the representation. We do this by maximizing the mutual information between the representations of the two views (Multi-View
InfoMax objective) while at the same time eliminating the information not shared between them, since it is guaranteed to be superfluous. The resulting representations are more robust for the given task as they have eliminated view specific nuisances.
Our contributions are three-fold: (1) We extend the information bottleneck principle to the unsupervised multi-view setting and provide a rigorous theoretical analysis of its application. (2) We define a new model 1 that empirically leads to state-of-the-art results in the low-label setting on two standard multi-view datasets, Sketchy and MIR-Flickr. (3) By exploiting data augmentation techniques, we empirically show that the representations learned by our model in single-view settings are more robust than existing unsupervised representation learning methods, connecting our theory to the choice of augmentation strategy.

heading: 2 PRELIMINARIES AND FRAMEWORK
text: The challenge of representation learning can be formulated as finding a distribution p(z|x) that maps data observations x ∈ X into a representation z ∈ Z, capturing some desired characteristics. Whenever the end goal involves predicting a label y, we consider only z that are discriminative enough to identify y. This requirement can be quantified by considering the amount of label information that remains accessible after encoding the data, and is known as sufficiency of z for y (Achille & Soatto, 2018): Definition 1. Sufficiency: A representation z of x is sufficient for y if and only if I(x;y|z) = 0.
Any model that has access to a sufficient representation z must be able to predict y at least as accurately as if it has access to the original data x instead. In fact, z is sufficient for y if and only if the amount of information regarding the task is unchanged by the encoding procedure (see Proposition B.1 in the Appendix):
I(x;y|z) = 0 ⇐⇒ I(x;y) = I(y; z). (1)
Among sufficient representations, the ones that result in better generalization for unlabeled data instances are particularly appealing. When x has higher information content than y, some of the information in x must be irrelevant for the prediction task. This can be better understood by subdividing I(x; z) into two components by using the chain rule of mutual information (see Appendix A):
I(x; z) = I(x; z|y)︸ ︷︷ ︸ superfluous information + I(y; z)︸ ︷︷ ︸ predictive information . (2)
Conditional mutual information I(x; z|y) represents the information in z that is not predictive of y, i.e. superfluous information. While I(y; z) determines how much label information is accessible from the representation. Note that this last term is independent of the representation as long as z is sufficient for y (see Equation 1). As a consequence, a sufficient representation contains minimal data information whenever I(x; z|y) is minimized. Minimizing the amount of superfluous information can be done directly only in supervised settings. In fact, reducing I(x; z) without violating the sufficiency constraint necessarily requires making some additional assumptions on the predictive task (see Theorem B.1 in the Appendix). In the next section we describe the basis of our technique, a strategy to safely reduce the information content of a representation even when the label y is not observed, by exploiting redundant information in the form of an additional view on the data.

heading: 3 MULTI-VIEW INFORMATION BOTTLENECK
text: As a motivating example, consider v1 and v2 to be two images of the same object from different view-points and let y be its label. Assuming that the object is clearly distinguishable from both v1 and let v2, any representation z containing all information accessible from both views would also contain the necessary label information. Furthermore, if z captures only the details that are visible from both pictures, it would eliminate the view-specific details and reduce the sensitivity of the representation to view-changes. The theory to support this intuition is described in the following where v1 and v2 are jointly observed and referred to as data-views.
1Code available at https://github.com/mfederici/Multi-View-Information-Bottleneck

heading: 3.1 SUFFICIENCY AND ROBUSTNESS IN THE MULTI-VIEW SETTING
text: In this section we extend our analysis of sufficiency and minimality to the multi-view setting.
Intuitively, we can guarantee that z is sufficient for predicting y even without knowing y by ensuring that z maintains all information which is shared by v1 and v2. This intuition relies on a basic assumption of the multi-view environment – that the two views provide the same predictive information. To formalize this we define redundancy. Definition 2. Redundancy: v1 is redundant with respect to v2 for y if and only if I(y;v1|v2) = 0
Intuitively, a view v1 is redundant for a task whenever it is irrelevant for the prediction of y if v2 is already observed. Whenever v1 and v2 are mutually redundant (v1 is redundant with respect to v2 for y, and vice-versa), we can show the following: Corollary 1. Let v1 and v2 be two mutually redundant views for a target y and let z1 be a representation of v1. If z1 is sufficient for v2 (I(v1;v2|z1) = 0) then z1 is as predictive for y as the joint observation of the two views (I(v1v2;y) = I(y; z1)).
In other words, whenever it is possible to assume mutual redundancy, any representation which contains all the information shared by both views (the redundant information) is as predictive as their joint observation.
By factorizing the mutual information between v1 and z1 analogously to Equation 2, we can identify two components:
I(v1; z1) = I(v1; z1|v2)︸ ︷︷ ︸ superfluous information + I(v2; z1)︸ ︷︷ ︸ predictive information for v2 .
Since I(v2; z1) has to be maximal if we want the representation to be sufficient for the label, we conclude that I(v1; z1) can be reduced by minimizing I(v1; z1|v2). This term intuitively represents the information z1 contains which is unique to v1 and is not predictable by observing v2. Since we assumed mutual redundancy between the two views, this information must be irrelevant for the predictive task and, therefore, it can be safely discarded. The proofs and formal assertions for the above statements and Corollary 1 can be found in Appendix B.
The less the two views have in common, the more I(v1; z1) can be reduced without violating sufficiency for the label, and consequently, the more robust the resulting representation. At the extreme, v1 and v2 share only label information, in which case we can show that z1 is minimal for y and our method is identical to the supervised information bottleneck method without needing to access the labels. Conversely, if v1 and v2 are identical, then our method degenerates to the InfoMax principle since no information can be safely discarded (see Appendix E).

heading: 3.2 THE MULTI-VIEW INFORMATION BOTTLENECK LOSS FUNCTION
text: Given v1 and v2 that satisfy the mutual redundancy condition for a label y, we would like to define an objective function for the representation z1 of v1 that discards as much information as possible without losing any label information. In Section 3.1 we showed that we can obtain sufficiency for y by ensuring that the representation z1 of v1 is sufficient for v2 , and that decreasing I(z1;v1|v2) will increase the robustness of the representation by discarding irrelevant information. So we can combine these two requirements using a relaxed Lagrangian objective to obtain the minimal sufficient representation z1 for v2:
L1(θ;λ1) = Iθ(z1;v1|v2)− λ1 Iθ(v2; z1), (3)
where θ denotes the dependency on the parameters of the encoder pθ(z1|v1), and λ1 represents the Lagrangian multiplier introduced by the constrained optimization. Symmetrically, we define a loss L2 to optimize the parameters ψ of a conditional distribution pψ(z2|v2) that defines a minimal sufficient representation z2 of the second view v2 for v1:
L2(ψ;λ2) = Iψ(z2;v2|v1)− λ2 Iψ(v1; z2), (4)
By defining z1 and z2 on the same domain Z and re-parametrizing the Lagrangian multipliers, the average of the two loss functions L1 and L2 can be upper bounded as follows:
1
LMIB(θ, ψ;β) = −Iθψ(z1; z2) + β DSKL(pθ(z1|v1)||pψ(z2|v2)), (5)
where DSKL represents the symmetrized KL divergence obtained by averaging the expected value of DKL(pθ(z1|v1)||pψ(z2|v2)) and DKL(pψ(z2|v2)||pθ(z1|v1)) for joint observations of the two views, while the coefficient β defines the trade-off between sufficiency and robustness of the representation, which is a hyper-parameter in this work. The resulting Multi-View Infomation Bottleneck (MIB) model (Equation 5) is visualized in Figure 1, while the batch-based computation of the loss function is summarized in Algorithm 1.
The symmetrized KL divergence DSKL(pθ(z1|v1)||pψ(z2|v2)) can be computed directly whenever pθ(z1|v1) and pψ(z2|v2) have a known density, while the mutual information between the two representations Iθψ(z1; z2) can be maximized by using any sample-based differentiable mutual information lower bound. We tried the Jensen-Shannon IJS (Devon Hjelm et al., 2019; Poole et al., 2019) and the InfoNCE INCE (van den Oord et al., 2018) estimators. These both require introducing an auxiliary parameteric model Cξ(z1, z2) which is jointly optimized during the training procedure using re-parametrized samples from pθ(z1|v1) and pψ(z2|v2). The full derivation for the MIB loss function can be found in Appendix F.

heading: 3.3 SELF-SUPERVISION AND INVARIANCE
text: Our method can also be applied when multiple views are not available by taking advantage of standard data augmentation techniques. This allows learning invariances directly from the augmented data, rather than requiring them to be built into the model architecture.
By picking a class T of data augmentation functions t : X→W that do not affect label information, it is possible to artificially build views that satisfy mutual redundancy for y. Let t1 and t2 be two random variables over T, then v1 := t1(x) and v2 := t2(x) must be mutually redundant for y. Since data augmentation functions in T do not affect label information (I(v1;y) = I(v2;y) = I(x;y)), a representation z1 of v1 that is sufficient for v2 must contain same amount of predictive information as x. Formal proofs for this statement can be found in Appendix B.4.
Whenever the two transformations for the same observation are independent (I(t1; t2|x) = 0), they introduce uncorrelated variations in the two views, which will be discarded when creating a
representation using our training objective. As an example, if T represents a set of small translations, the two resulting views will differ by a small shift. Since this information is not shared, any z1 which is optimal according to the MIB objective must discard fine-grained details regarding the position.
To enable parameter sharing between the encoders, we generate the two views v1 and v2 by independently sampling two functions from the same function class T with uniform probability. As a result, t1 and t2 will have the same distribution, and so the two generated views will also have the same marginals (p(v1) = p(v2)). For this reason, the two conditional distributions pθ(z1|v1) and pψ(z2|v2) can share their parameters and only one encoder is necessary. Full (or partial) parameter sharing can be also applied in the multi-view settings whenever the two views have the same (or similar) marginal distributions.

heading: 4 RELATED WORK
text: The relationship between our method and past work on representation learning is best described using the Information Plane (Tishby et al., 2000). In this setting, each representation z of x for a predictive task y can be characterised by the amount of information regarding the raw observation I(x; z) and the corresponding measure of accessible predictive information I(y; z) (x and y axis respectively on Figure 2). Ideally, a good representation would be maximally informative about the label while retaining a minimal amount of information from the observations (top left corner of the parallelogram). Further details on the Information Plane and the bounds visualized in Figure 2 are described in Appendix C.
Thanks to recent progress in mutual information estimation (Nguyen et al., 2008; Ishmael Belghazi et al., 2018; Poole et al., 2019), the InfoMax principle (Linsker, 1988) has gained attention for unsupervised representation learning (Devon Hjelm et al., 2019; van den Oord et al., 2018). Since the InfoMax objective involves maximizing I(x; z), the resulting representation aims to preserve all the information regarding the raw observations (top right corner in Figure 2).
Concurrent work has applied the InfoMax principle in the Multi-View setting (Ji et al., 2019; Hénaff et al., 2019; Tian et al., 2019; Bachman et al., 2019), aiming to maximize mutual infor-
mation between the representation z of a first data-view x and a second one v2. The target representation for the Multi-View InfoMax (MV-InfoMax) models should contain at least the amount of information in x that is predictive for v2, targeting the region I(z;x) ≥ I(x;v2) on the Information Plane (purple dotted line in Figure 2). Since the MV-InfoMax has no incentive to discard any information regarding x from z, a representation that is optimal according to the InfoMax principle is also optimal for any MV-InfoMax model. Our model with β = 0 (Equation 5) belongs to this family of objectives since the incentive to remove superfluous information is removed. Despite their success, Tschannen et al. (2019) has shown that the effectiveness of the InfoMax models is due to inductive biases introduced by the architecture and estimators rather than the training objective itself, since the InfoMax and MV-InfoMax objectives can be trivially maximized by using invertible encoders.
On the other hand, Variational Autoencoders (VAEs) (Kingma & Welling, 2014) define a training objective that balances compression and reconstruction error (Alemi et al., 2018) through an hyper-parameter β. Whenever β is close to 0, the VAE objective aims for a lossless representation, approaching the same region of the Information Plane as the one targeted by InfoMax (Barber & Agakov, 2003). When β approaches large values, the representation becomes more compressed, showing increased generalization and disentanglement (Higgins et al., 2017; Burgess et al., 2018), and, as β approaches infinity, I(z;x) goes to zero. During this transition from low to high β, however, there are no guarantees that VAEs will retain label information (Theorem B.1 in the Appendix). The path between the two regimes depends on how well the label information aligns with the induc-
tive bias introduced by encoder (Jimenez Rezende & Mohamed, 2015; Kingma et al., 2016), prior (Tomczak & Welling, 2018) and decoder architectures (Gulrajani et al., 2017; Chen et al., 2017).
The idea of discarding irrelevant information was introduced in Tishby et al. (2000) and identified as one of the possible reasons behind the generalization capabilities of deep neural networks by Tishby & Zaslavsky (2015) and Achille & Soatto (2018). Representations based on the information bottleneck principle explicitly minimize the amount of superfluous information in the representation while retaining all the label information from the data (top-left corner of the Information Plane in Figure 2). This direction of research has been explored for both single-view (Alemi et al., 2018) and multi-view settings (Wang et al., 2019), even if explicit label supervision is required to train the representation z.
In contrast to all of the above, our work is the first to explicitly identify and discard superfluous information from the representation in the unsupervised multi-view setting. This is because unsupervised models based on the β-VAE objective remove information indiscriminately without identifying which part is relevant for teh predictive task, and the InfoMax and Multi-View InfoMax methods do not explicitly try to remove superfluous information at all. The MIB objective, on the other hand, results in the representation with the least superfluous information, i.e. the most robust among the representations that are optimal according to Multi-View InfoMax, without requiring any additional label supervision.

heading: 5 EXPERIMENTS
text: In this section we demonstrate the effectiveness of our model against state-of-the-art baselines in both the multi-view and single-view setting. In the single-view setting, we also estimate the coordinates on the Information Plane for each of the baseline methods as well as our method to validate the theory in Section 3.
The results reported in the following sections are obtained using the Jensen-Shannon IJS (Devon Hjelm et al., 2019; Poole et al., 2019) estimator, which resulted in better performance for MIB and the other InfoMax-based models (Table 2 in the supplementary material). In order to facilitate the comparison between the effect of the different loss functions, the same estimator is used across the different models.

heading: 5.1 MULTI-VIEW TASKS
text: We compare MIB on the sketch-based image retrieval (Sangkloy et al., 2016) and Flickr multiclass image classification (Huiskes & Lew, 2008) tasks with domain specific and prior multi-view learning methods.

heading: 5.1.1 SKETCH-BASED IMAGE RETRIEVAL
text: Dataset. The Sketchy dataset (Sangkloy et al., 2016) consists of 12,500 images and 75,471 handdrawn sketches of objects from 125 classes. As in Liu et al. (2017), we also include another 60,502 images from the ImageNet (Deng et al., 2009) from the same classes, which results in total 73,002 natural object images. As per the experimental protocol of Zhang et al. (2018), a total of 6,250 sketches (50 sketches per category) are randomly selected and removed from the training set for testing purpose, which leaves 69,221 sketches for training the model.
Experimental Setup. The sketch-based image retrieval task is a ranking of 73,002 natural images according to the unseen test (query) sketch. Retrieval is done for our model by generating representations for the query sketch as well as all natural images, and ranking the image by the euclidean distance of their representation from the sketch representation. The baselines use various domain specific ranking methodologies. Model performance is computed based on the class of the ranked pictures corresponding to the query sketch. The training set consists of pairs of image v1 and sketch v2 randomly selected from the same class, to ensure that both views contain the equivalent label information (mutual redundancy).
Following recent work (Zhang et al., 2018; Dutta & Akata, 2019), we use features extracted from images and sketches by a VGG (Simonyan & Zisserman, 2014) architecture trained for classification
on the TU-Berlin dataset (Eitz et al., 2012). The resulting flattened 4096-dimensional feature vectors are fed to our image and sketch encoders to produce a 64-dimensional representation. Both encoders consist of neural networks with hidden layers of 2048 and 1024 units respectively. Size of the representation and regularization strength β are tuned on a validation sub-split. We evaluate MIB on five different train/test splits and report mean and standard deviation in Table 5.1.1. Further details on our training procedure and architecture are in Appendix G.
Results. Table 5.1.1 shows that the our model achieves strong performance for both mean average precision (mAP@all) and precision at 200 (Prec@200), suggesting that the representation is able to capture the common class information between the paired pictures and sketches. The effectiveness of MIB on the retrieval task can be mostly attributed to the regularization introduced with the symmetrized KL divergence between the two encoded views. In addition to discarding view-private information, this term actively aligns the representations of v1 and v2, making the MIB model especially suitable for retrieval tasks

heading: 5.1.2 MIR-FLICKR
text: Dataset. The MIR-Flickr dataset (Huiskes & Lew, 2008) consists of 1M images annotated with 800K distinct user tags. Each image is represented by a vector of 3,857 hand-crafted image features (v1), while the 2,000 most frequent tags are used to produce a 2000-dimensional multi-hot encoding (v2) for each picture. The dataset is divided into labeled and unlabeled sets that respectively contain 975K and 25K images, where the labeled set also contains 38 distinct topic classes together with the user tags. Training images with less than two tags are removed, which reduces the total number of training samples to 749,647 pairs (Sohn et al., 2014; Wang et al., 2016). The labeled set contains 5 different splits of train, validation and test sets of size 10K/5K/10K respectively.
Experimental Setup. Following standard procedure in the literature (Srivastava & Salakhutdinov, 2014; Wang et al., 2016), we train our model on the unlabeled pairs of images and tags. Then a multi-label logistic classifier is trained from the representation of 10K labeled train images to the corresponding macro-categories. The quality of the representation is assessed based on the performance of the trained logistic classifier on the labeled test set. Each encoder consists of a multi-layer perceptron of 4 hidden layers with ReLU activations learning two 1024-dimensional representations z1 and z2 for images v1 and tags v2 respectively. Examples of the two views, labels, and further details on the training procedure are in Appendix G.
Results. Our MIB model is compared with other popular multi-view learning models in Figure 3 for β = 0 (Multi-View InfoMax), β = 1 and β = 10−3 (best on validation set). Although the tuned MIB performs similarly to Multi-View InfoMax with a large number of labels, it outperforms it when fewer labels are available. Furthermore, by choosing a larger β the accuracy of our model drastically increases in scarce label regimes, while slightly reducing the accuracy when all the labels are observed (see right side of Figure 3). This effect is likely due to a violation of the mutual
2These results are included only for completeness, as the Multi-View InfoMax objective does not produce consistent representations for the two views so there is no straight-forward way to use it for ranking.
redundancy constraint (see Figure 6 in the supplementary material) which can be compensated with smaller values of β for less aggressive compression.
A possible reason for the effectiveness of MIB against some of the other baselines may be its ability to use mutual information estimators that do not require reconstruction. Both Multi-View VAE (MVAE) and Deep Variational CCA (VCCA) rely on a reconstruction term to capture cross-modal information, which can introduce bias that decreases performance.

heading: 5.2 SELF-SUPERVISED SINGLE-VIEW TASK
text: In this section, we compare the performance of different unsupervised learning models by measuring their data efficiency and empirically estimating the coordinates of their representation on the Information Plane. Since accurate estimation of mutual information is extremely expensive (McAllester & Stratos, 2018), we focus on relatively small experiments that aim to uncover the difference between popular approaches for representation learning.
Dataset. The dataset is generated from MNIST by creating the two views, v1 and v2, via the application of data augmentation consisting of small affine transformations and independent pixel corruption to each image. These are kept small enough to ensure that label information is not effected. Each pair of views is generated from the same underlying image, so no label information is used in this process (details in Appendix G).
Experimental Setup. To evaluate, we train the encoders using the unlabeled multi-view dataset just described, and then fix the representation model. A logistic regression model is trained using the resulting representations along with a subset of labels for the training set, and we report the accuracy of this model on a disjoint test set as is standard for the unsupervised representation learning literature (Tschannen et al., 2019; Tian et al., 2019; van den Oord et al., 2018). We estimate I(x; z) and I(y; z) using mutual information estimation networks trained from scratch on the final representations using batches of joint samples {(x(i),y(i), z(i))}Bi=1 ∼ p(x,y)pθ(z|x). All models are trained using the same encoder architecture consisting of 2 layers of 1024 hidden units with ReLU activations, resulting in 64-dimensional representations. The same data augmentation procedure was also applied for single-view architectures and models were trained for 1 million iterations with batch size B = 64.
Results. Figure 4 summarizes the results. The empirical measurements of mutual information reported on the Information Plane are consistent with the theoretical analysis reported in Section 4: models that retain less information about the data while maintaining the maximal amount of predictive information, result in better classification performance at low-label regimes, confirming the hypothesis that discarding irrelevant information yields robustness and more data-efficient represen-
tations. Notably, the MIB model with β = 1 retains almost exclusively label information, hardly decreasing the classification performance when only one label is used for each data point.

heading: 6 CONCLUSIONS AND FUTURE WORK
text: In this work, we introduce Multi-View Information Bottleneck, a novel method for taking advantage of multiple data-views to produce robust representations for downstream tasks. In our experiments, we compared MIB empirically against other approaches in the literature on three such tasks: sketchbased image retrieval, multi-view and unsupervised representation learning. The strong performance obtained in the different areas show that Multi-View Information Bottleneck can be practically applied to various tasks for which the paired observations are either readily available or artificially produced. Furthermore, the positive results on the MIR-Flickr dataset show that our model can work well in practice even when mutual redundancy holds only approximately.
There are multiple extensions that we would like to explore in future work. One interesting direction would be considering more than two views. In Appendix D we discuss why the mutual redundancy condition cannot be trivially extended to more than two views, but we still believe such an extension is possible. Secondly, we believe that exploring the role played by different choices of data augmentation could bridge the gap between the Information Bottleneck principle and the literature on invariant neural networks (Bloem-Reddy & Whye Teh, 2019), which are able to exploit known symmetries and structure of the data to remove superfluous information.

heading: ACKNOWLEDGMENTS
text: We thank Andy Keller, Karen Ullrich, Maximillian Ilse and the anonymous reviewers for their feedback and insightful comments. This work has received funding from the ERC under the Horizon 2020 program (grant agreement No. 853489). The Titan Xp and Titan V used for this research were donated by the NVIDIA Corporation.

heading: A PROPERTIES OF MUTUAL INFORMATION AND ENTROPY
text: In this section we enumerate some of the properties of mutual information that are used to prove the theorems reported in this work. For any random variables w, x, y and z:
(P1) Positivity:
I(x;y) ≥ 0, I(x;y|z) ≥ 0
(P2) Chain rule:
I(xy; z) = I(y; z) + I(x; z|y)
(P3) Chain rule (Multivariate Mutual Information):
I(x;y; z) = I(y; z)− I(y; z|x)
(P4) Positivity of discrete entropy: For discrete x
H(x) ≥ 0, H(x|y) ≥ 0
(P5) Entropy and Mutual Information
H(x) = H(x|y) + I(x;y)

heading: B THEOREMS AND PROOFS
text: In the following section we prove the statements reported in the main text of the paper. Whenever a random variable z is defined to be a representation of another random variable x, we state that z is conditionally independent from any other variable in the system once x is observed. This does not imply that z must be a deterministic function of x, but that the source of stochasticity for z is independent of the other random variables. As a result whenever z is a representation of x:
I(z;a|xb) = 0,
for any variable (or groups of variables) a and b in the system.

heading: B.1 ON SUFFICIENCY
text: Proposition B.1. Let x and y be random variables with joint distribution p(x,y). Let z be a representation of x, then z is sufficient for y if and only if I(x;y) = I(y; z)

heading: Hypothesis:
text: (H1) z is a representation of x: I(y; z|x) = 0

heading: Thesis:
text: (T1) I(x;y|z) = 0 ⇐⇒ I(x;y) = I(y; z)

heading: Proof.
text: I(x;y|z) (P3)= I(x;y)− I(x;y; z) (P3)= I(x;y)− I(y; z)− I(y; z|x) (H1) = I(x;y)− I(y; z)
Since both I(x;y) and I(y; z) are non-negative (P1), I(x;y|z) = 0 ⇐⇒ I(y; z) = I(x;y)

heading: B.2 NO FREE GENERALIZATION
text: Theorem B.1. Let x, z and y be random variables with joint distribution p(x,y, z). Let z′ be a representation of x that satisfies I(x; z) > I(x; z′), then it is always possible to find a label y for which z′ is not predictive for y while z is.
Hypothesis:
(H1) z ′ is a representation of x: I(y; z′|x) = 0
(H2) I(x; z) > I(x; z ′)
Thesis:
(T1) I(x; z ′) < I(x; z) =⇒ ∃y.I(y; z) > I(y; z′) = 0
Proof. By construction.
1. We first factorize x as a function of two independent random variables (Proposition 2.1 Achille & Soatto (2018)) by picking y such that:
(C1) I(y; z ′) = 0 (C2) x = f(z ′,y)
for some deterministic function f . Note that such y always exists.
2. Since x is a function of y and z′:
(C4) I(x; z|yz′) = 0
Considering I(y; z):
I(y; z) (P3) = I(y; z|x) + I(x;y; z)
(P1)
≥ I(x;y; z) (P3) = I(x; z)− I(x; z|y)
(P3) = I(x; z)− I(x; z|yz′)− I(x; z; z′|y)
(C2) = I(x; z)− I(x; z; z′|y)
(P3) = I(x; z)− I(x; z′|y) + I(x; z′|yz)
(P1)
≥ I(x; z)− I(x; z′|y) (P3) = I(x; z)− I(x; z′) + I(x;y; z′)
(P3) = I(x; z)− I(x; z′) + I(y; z′)− I(y; z′|x)
(P1)
≥ I(x; z)− I(x; z′)− I(y; z′|x) (H1) = I(x; z)− I(x; z′)
(H2) > 0
Since I(y; z′) = 0 by construction and I(y; z) > 0, the y built in 1. satisfies the conditions reported in the thesis.
Corollary B.1.1. Let z′ be a representation of x that discards observational information. There is always a label y for which a z′ is not predictive, while the original observations are. Hypothesis:
(H1) x is discrete
(H2) z ′ discards information regarding x: I(z′;x) < H(x)

heading: Thesis:
text: (T1) ∃y.I(y;x) > I(y; z′) = 0
Proof. By construction using Theorem B.1.
1. Set z = x:
(C1) I(x; z) (P5) = H(x)−H(x|z) (H1)= H(x)
2. I(z′;x) < H(x) (C1) =⇒ I(z′;x) < I(x; z)
Since the hypothesis are met, we conclude that there exist y such that I(y;x) > I(y; z′) = 0

heading: B.3 MULTI-VIEW

heading: B.3.1 MULTI-VIEW REDUNDANCY AND SUFFICIENCY
text: Proposition B.2. Let v1, v2, y be random variables with joint distribution p(v1,v2,y). Let z1 be a representation of v1, then:
I(v1;y|z1) ≤ I(v1;v2|z1) + I(v1;y|v2)

heading: Hypothesis:
text: (H1) z1 is a representation of v1: I(y; z1|v2v1) = 0

heading: Thesis:
text: (T1) I(v1;y|z1) ≤ I(v1;v2|z1) + I(v1;y|v2)
Proof. Since z1 is a representation of v1:
(C1) I(y; z1|v2v1) = 0
Therefore:
I(v1;y|z1) (P3) = I(v1;y|z1v2) + I(v1;v2;y|z1)
(P3) = I(v1;y|v2)− I(v1;y; z1|v2) + I(v1;v2;y|z1)
(P3) = I(v1;y|v2)− I(y; z1|v2) + I(y; z1|v2v1) + I(v1;v2;y|z1)
(P1)
≤ I(v1;y|v2) + I(y; z1|v2v1) + I(v1;v2;y|z1) (H1) = I(v1;y|v2) + I(v1;v2;y|z1)
(P3) = I(v1;y|v2) + I(v1;v2|z1)− I(v1;v2|z1y)
(P1)
≤ I(v1;y|v2) + I(v1;v2|z1)
Proposition B.3. Let v1 be a redundant view with respect to v2 for y. Any representation z1 of v1 that is sufficient for v2 is also sufficient for y.

heading: Hypothesis:
text: (H1) z1 is a representation of v1: I(y; z1|v2v1) = 0
(H2) v1 is redundant with respect to v2 for y: I(y;v1|v2) = 0

heading: Thesis:
text: (T1) I(v1;v2|z1) = 0 =⇒ I(v1;y|z1) = 0
Proof. Using the results from Theorem B.2:
I(v1;y|z1) (ThB.2) ≤ I(v1;y|v2) + I(v1;v2|z1) (H2) = I(v1;v2|z1)
Therefore I(v1;v2|z1) = 0 =⇒ I(v1;y|z1) = 0
Theorem B.2. Let v1, v2 and y be random variables with distribution p(v1,v2,y). Let z be a representation of v1, then
I(y; z1) ≥ I(y;v1v2)− I(v1;v2|z1)− I(v1;y|v2)− I(v2;y|v1)

heading: Hypothesis:
text: (H1) z1 is a representation of v1: I(y; z1|v1v2) = 0

heading: Thesis:
text: (T1) I(y; z1) ≥ I(y;v1v2)− I(v1;v2|z1)− I(v1;y|v2)− I(v2;y|v1)

heading: Proof.
text: I(y; z1) (P3) = I(y; z1|v1v2) + I(y;v1v2; z1)
(H1) = I(y;v1v2; z1)
(P3) = I(y;v1v2)− I(y;v1v2|z1)
(P2) = I(y;v1v2)− I(y;v1|z1)− I(y;v2|z1v1)
(P3) = I(y;v1v2)− I(y;v1|z1)− I(y;v2|v1) + I(y;v2; z1|v1)
(P3) = I(y;v1v2)− I(y;v1|z1)− I(y;v2|v1) + I(y; z1|v1)− I(y; z1|v1v2)
(H1) = I(y;v1v2)− I(y;v1|z1)− I(y;v2|v1) + I(y; z1|v1)
(P1)
≥ I(y;v1v2)− I(y;v1|z1)− I(y;v2|v1) (PropB.2)
≥ I(y;v1v2)− I(v1;y|v2)− I(v1;v2|z1)− I(y;v2|v1)
Corollary B.2.1. Let v1 and v2 be mutually redundant views for y. Let z1 be a representation of v1 that is sufficient for v2. Then:
I(y; z1) = I(v1v2;y)
Hypothesis:
(H1) z1 is a representation of v1: I(y; z1|v1v2) = 0
(H2) v1 and v2 are mutually redundant for y: I(y;v1|v2) + I(y;v2|v1) = 0
(H3) z1 is sufficient for v2: I(v2;v1|z) = 0

heading: Thesis:
text: (T1) I(y; z1) = I(v1v2;y)
Proof. Using Theorem B.2
I(y; z1) (ThB.2)
≥ I(y;v1v2)− I(v1;y|v2)− I(v1;v2|z1)− I(y;v2|v1) (H2) = I(y;v1v2)− I(v1;v2|z1)
(H3) = I(y;v1v2)
Since I(y; z1) ≤ I(y;v1v2) is a consequence of the data processing inequality, we conclude that I(y; z1) = I(y;v1v2)

heading: B.4 SUFFICIENCY AND AUGMENTATION
text: Let x and y be random variables with domain X and Y respectively. Let T be a class of functions t : X→W and let t1 and t2 be a random variables over T that depends only on x. For the theorems and corollaries discussed in this section, we are going to consider the independence assumption that can be derived from the graphical model G reported in Figure 5.
Proposition B.4. Whenever I(t1(x);y) = I(t2(x);y) = I(x;y) the two views t1(x) and t2(x) must be mutually redundant for y.

heading: Hypothesis:
text: (H1) Independence relations determined by G

heading: Thesis:
text: (T1) I(t1(x);y) = I(t2(x);y) = I(x;y) =⇒ I(t1(x);y|t2(x)) + I(t2(x);y|t1(x)) = 0

heading: Proof.
text: 1. Considering G we have:
(C1) I(t1(x);y|xt2(x)) = 0 (C2) I(y; t2(x)|x) = 0
2. Since t2(x) is uniquely determined by x and t2:
(C3) I(t2(x);y|xt2) = 0
3. Consider I(t1(x);y|t2(x))
I(t1(x);y|t2(x)) (P3) = I(t1(x);y|xt2(x)) + I(t1(x);y;x|t2(x))
(C1) = I(t1(x);y;x|t2(x))
(P3) = I(y;x|t2(x))− I(y;x|t1(x)t2(x))
(P1)
≤ I(y;x|t2(x)) (P3) = I(y;x)− I(y;x; t2(x))
(P3) = I(y;x)− I(y; t2(x)) + I(y; t2(x)|x)
(P3) = I(y;x)− I(y; t2(x)) + I(y; t2(x)|t2x) + I(y; t2(x); t2|x)
(C3) = I(y;x)− I(y; t2(x)) + I(y; t2(x); t2|x)
(P3) = I(y;x)− I(y; t2(x)) + I(y; t2(x)|x)− I(y; t2(x)|t2x)
(P1)
≥ I(y;x)− I(y; t2(x)) + I(y; t2(x)|x) (C2)
≥ I(y;x)− I(y; t2(x))
Therefore I(y;x) = I(y; t2(x)) =⇒ I(t1(x);y|t2(x)) = 0
The proof for I(y;x) = I(y; t1(x)) =⇒ I(t2(x);y|t1(x)) = 0 is symmetric, therefore we conclude I(t1(x);y) = I(t2(x);y) = I(x;y) =⇒ I(t1(x);y|t2(x)) + I(t2(x);y|t1(x)) = 0
Theorem B.3. Let I(t1(x);y) = I(t2(x);y) = I(x;y). Let z1 be a representation of t1(x) . If z1 is sufficient for t2(x) then I(x;y) = I(y; z1).

heading: Hypothesis:
text: (H1) Independence relations determined by G
(H2) I(t1(x);y) = I(t2(x);y) = I(x;y)

heading: Thesis:
text: (T1) I(t1(x); t2(x)|z1) = 0 =⇒ I(x;y) = I(y; z1)
Proof. Since t1(x) is redundant for t2(x) (Proposition B.4) any representation z1 of t1(x) that is sufficient for t2(x) must also be sufficient for y (Theorem B.2). Using Proposition B.1 we have I(y; z1) = I(y; t1(x)). Since I(y; t1(x)) = I(y;x) by hypothesis, we conclude I(x;y) = I(y; z1)
C INFORMATION PLANE
Every representation z of x must satisfy the following constraints:
• 0 ≤ I(y; z) ≤ I(x;y): The amount of label information ranges from 0 to the total predictive information accessible from the raw observations I(x;y).
• I(y; z) ≤ I(x; z) ≤ I(y; z)+H(x|y): The representation must contain more information about the observations than about the label. When x is discrete, the amount of discarded label information I(x;y)− I(y; z) must be smaller than the amount of discarded observational information H(x)− I(x; z), which implies I(x; z) ≤ I(y; z) +H(x|y).
Proof. Since z is a representation of x:
(C1) I(y; z|x) = 0
Considering the four bounds separately:
1. I(y; z) ≥ 0: Follows from P1
2. I(x; z) ≥ I(y; z): Follows from:
I(x; z) (P3) = I(x; z|y) + I(x;y; z)
(P1)
≥ I(x;y; z) (P3) = I(y; z)− I(y; z|x)
(C1) = I(y; z)
3. I(y; z) ≤ I(y;x): Data processing inequality
I(y; z) (P3) = I(y; z|x) + I(y; z;x)
(C1) = I(y; z;x)
(P3) = I(x;y)− I(x;y|z)
(P1)
≤ I(x;y)
4. I(x; z) ≤ I(y; z) +H(x|y): For discrete x:
I(x; z) (P3) = I(x; z|y) + I(x;y; z)
(P3) = I(x; z|y) + I(y; z)− I(y; z|x)
(C1) = I(x; z|y) + I(y; z)
(P4)
≤ I(x; z|y) +H(x|yz) + I(y; z) (P5) = H(x|y) + I(y; z)
Note that the discreetness of x is required only to prove bound 4. For continuous x bounds 1, 2 and 3 still hold.

heading: D NON-TRANSITIVITY OF MUTUAL REDUNDANCY
text: The mutual redundancy condition between two views v1 and v2 for a label y can not be trivially extended to an arbitrary number of views, as the relation is not transitive because of some higher order interaction between the different views and the label. This can be shown with a simple example.
Given three views v1, v2 and v3 and a task y such that:
• v1 and v2 are mutually redundant for y • v2 and v3 are mutually redundant for y
Then, we show that v1 is not necessarily mutually redundant with respect to v3 for y.
Let v1, v2 and v3 be fair and independent binary random variables. Defining y as the exclusive or operator applied to v1 and v3 ( y := v1 XOR v3), we have that I(v1;y) = I(v3;y) = 0. In this settings, v1 and v2 are mutually redundant for y:
I(v1;y|v2) = H(v1|v2)−H(v1|v2y) = H(v1)−H(v1) = 0 I(v2;y|v1) = H(v2|v1)−H(v2|v1y) = H(v2)−H(v2) = 0
Analogously, v2 and v3 are also mutually redundant for y as the three random variables are not predictive for each other. Nevertheless, v1 and v3 are not mutually redundant for y:
I(v1;y|v3) = H(v1|v3)−H(v1|v3y)︸ ︷︷ ︸ 0 = H(v1) = 1 I(v3;y|v1) = H(v3|v1)−H(v3|v1y)︸ ︷︷ ︸ 0 = H(v3) = 1
Where H(v1|v3y) = H(v3|v1y) = 0 follows from v1 = v3 XOR y and v3 = v1 XOR y, while H(v1) = H(v3) = 1 holds by construction.
This counter-intuitive higher order interaction between multiple views makes our theory non-trivial to generalize to more than two views.

heading: E EQUIVALENCES OF DIFFERENT OBJECTIVES
text: Different objectives in literature can be seen as a special case of the Multi-View Information Bottleneck principle. In this section we show that the supervised version of Information Bottleneck is equivalent to the corresponding Multi-View version whenever the two redundant views have only label information in common. A second subsection show equivalence between InfoMax and MultiView Information Bottleneck whenever the two views are identical.

heading: E.1 MULTI-VIEW INFORMATION BOTTLENECK AND SUPERVISED INFORMATION BOTTLENECK
text: Whenever the two mutually redundant views v1 and v2 have only label information in common (or when one of the two views is the label itself) the Multi-View Information Bottleneck objective is equivalent to the respective supervised version. This can be shown by proving that I(v1; z1|v2) = I(v1; z1|y), i.e. a representation z1 of v1 that is sufficient and minimal for v2 is also sufficient and minimal for y.
Proposition E.1. Let v1 and v2 be mutually redundant views for a label y that share only label information. Then a sufficient representation z1 of v1 for v2 that is minimal for v2 is also a minimal representation for y.

heading: Hypothesis:
text: (H1) v1 and v2 are mutually redundant for y: I(v1;y|v2) + I(v2;y|v1) = 0
(H2) v1 and v2 share only label information: I(v1;v2) = I(v1;y)
(H3) z1 is sufficient for v2: I(v1;v2|z1) = 0

heading: Thesis:
text: (T1) I(v1; z1|v2) = I(v1; z1|y)
Proof.
1. Consider I(v1; z):
I(v1; z1) (P3) = I(v1; z1|v2) + I(v1;v2; z1)
(P3) = I(v1; z1|v2) + I(v1;v2)− I(v1;v2|z1)
(H3) = I(v1; z1|v2) + I(v1;v2)
(H1) = I(v1; z1|v2) + I(v1;y)
2. Using Corollary 1, from (H2) and (H3) follows I(v1;y|z1) = 0
3. I(v1; z) can be alternatively expressed as:
I(v1; z1) (P3) = I(v1; z1|y) + I(v1;y; z1)
(P3) = I(v1; z1|y) + I(v1;y)− I(v1;y|z1)
(Cor1) = I(v1; z1|y) + I(v1;y)
Equating 1 and 3, we conclude I(v1; z1|v2) = I(v1; z1|y), therefore z1 which minimizes I(v1; z1|v2) is also minimizing I(v1; z1|y). When I(v1; z1|y) is minimal, I(y; z1) is also minimal (see equation 2).

heading: E.2 MULTI-VIEW INFORMATION BOTTLENECK AND INFOMAX
text: Whenever v1 = v2, a representation z1 of v1 that is sufficient for v2 must contain all the original information regarding v1. Furthermore since I(v1; z1|v2) = 0 for every representation, no superfluous information can be identified and removed. As a consequence, a minimal sufficient representation z1 of v1 for v2 is any representation for which mutual information is maximal, hence InfoMax.

heading: F LOSS COMPUTATION
text: Starting from Equation 3, we consider the average of the losses L1(θ;λ1) and L2(ψ;λ2) that aim to create the minimal sufficient representations z1 and z2 respectively:
L 1+2 2 (θ, ψ;λ1, λ2) = Iθ(v1; z1|v2) + Iψ(v2; z2|v1) 2 + λ1Iθ(v1;v2|z1) + λ2Iψ(v1;v2|z1) 2 (6)
Considering z1 and z2 on the same domain Z, Iθ(v1; z1|v2) can be expressed as: Iθ(v1; z1|v2) = Ev1,v2∼p(v1,v2)Ez∼pθ(z1|v1) [ log
pθ(z1 = z|v1 = v1) pθ(z1 = z|v2 = v2) ] = Ev1,v2∼p(v1,v2)Ez∼pθ(z1|v1) [ log
pθ(z1 = z|v1 = v1) pψ(z2 = z|v2 = v2) pψ(z2 = z|v2 = v2) pθ(z1 = z|v2 = v2) ] = DKL(pθ(z1|v1)||pψ(z2|v2))−DKL(pθ(z2|v1)||pψ(z2|v2)) ≤ DKL(pθ(z1|v1)||pψ(z2|v2))
Note that the bound is tight whenever pψ(z2|v2) coincides with pθ(z1|v2). This happens whenever z1 and z2 produce a consistent encoding. Analogously Iψ(v2; z2|v1) is upper bounded by DKL(pψ(z2|v2)||pθ(z1|v1)). Iθ(v2; z1) can be rephrased as:
Iθ(z1;v2) (P2) = Iθψ(z1; z2v2)− Iθψ(z1; z2|v2) =∗ Iθψ(z1; z2v2)
= Iθψ(z1; z2) + Iθψ(z1;v2|z2) ≥ Iθψ(z1; z2)
Where ∗ follows from z2 representation of v2. The bound reported in this equation is tight whenever z2 is sufficient for z1 (Iθψ(z1;v2|z2) = 0). This happens whenever z2 contains all the information regarding z1 (and therefore v1). Once again, the same bound can symmetrically be used to show Iθ(z2;v1) ≥ Iθψ(z1; z2). Therefore, the loss function in Equation 6 can be upper-bounded with:
L 1+2 2 (θ, ψ;λ1, λ2) ≤ DSKL(pθ(z1|v1)||pψ(z2|v2))− λ1 + λ2 2 Iθψ(z1; z2) (7)
Where:
DSKL(pθ(z1|v1)||pψ(z2|v2)) := 1
2 DKL(pθ(z1|v1)||pψ(z2|v2)) +
1 2 DKL(pψ(z2|v2)||pθ(z1|v1))
Lastly, multiplying both terms with β := 2λ1+λ2 and re-parametrizing the objective, we obtain:
LMIB(θ, ψ;β) = −Iθψ(z1; z2) + β DSKL(pθ(z1|v1)||pψ(z2|v2)) (8)

heading: G EXPERIMENTAL PROCEDURE AND DETAILS

heading: G.1 MODELING
text: The two stochastic encoders pθ(z1|v1) and pψ(z2|v2) are modeled by Normal distributions parametrized with neural networks (µθ,σ2θ) and (µψ,σ 2 ψ) respectively:
pθ(z1|v1) := N ( z1|µθ(v1),σ2θ(v1) ) pψ(z2|v2) := N ( z2|µψ(v2),σ2ψ(v2)
) Since the density of the two encoders can be evaluated, the symmetrized KL-divergence in equation 4 can be directly computed. On the other hand, Iθψ(z1; z2) requires the use of a mutual information estimator.
To facilitate the optimization, the hyper-parameter β is slowly increased during training, starting from a small value ≈ 10−4 to its final value with an exponential schedule. This is because the mutual information estimator is trained together with the other architectures and, since it starts from a random initialization, it requires an initial warm-up. Starting with bigger β results in the encoder collapsing into a fixed representation. The update policy for the hyper-parameter during training has not shown strong influence on the representation, as long as the mutual information estimator network has reached full capacity.
All the experiments have been performed using the Adam optimizer with a learning rate of 10−4 for both encoders and the estimation network. Higher learning rate can result in instabilities in the training procedure. The results reported in the main text relied on the Jensen-Shannon mutual information estimator (Devon Hjelm et al., 2019) since the InfoNCE counterpart (van den Oord et al., 2018) generally resulted in worse performance that could be explained by the effect of the factorization of the critic network (Poole et al., 2019).

heading: G.2 SKETCHY EXPERIMENTS
text: • Input: The two views for the sketch-based classification task consist of 4096 dimensional sketch and image features extracted from two distinct VGG-16 network models which were pre-trained on images and sketches from the TU-Berlin dataset Eitz et al. (2012) for endto-end classification. The feature extractors are frozen during the training procedure of for the two representations. Each training iteration used batches of size B = 128.
• Encoder and Critic architectures: Both sketch and image encoders consist of multi-layer perceptrons of 2 hidden ReLU units of size 2,048 and 1,024 respectively with an output of size 2x64 that parametrizes mean and variance for the two Gaussian posteriors. The critic architecture also consists of a multi layer perceptron of 2 hidden ReLU units of size 512.
• β update policy: The initial value of β is set to 10−4. Starting from the 10,000th training iteration, the value of β is exponentially increased up to 1.0 during the following 250,000 training iterations. The value of β is then kept fixed to one until the end of the training procedure (500,000 iterations).
• Evaluation: All natural images are used as both training sets and retrieval galleries. The 64 dimensional real outputs of sketch and image representation are compared using Euclidean distance. For having a fair comparison other methods that rely on binary hashing (Liu et al., 2017; Zhang et al., 2018), we used Hamming distance on a binarized representation (obtained by applying iterative quantization Gong et al. (2013) on our real valued representation). We report the mean average precision (mAP@all) and precision at toprank 200 (Prec@200) Su et al. (2015) on both the real and binary representation to evaluate our method and compare it with prior works.

heading: G.4 MNIST EXPERIMENTS

heading: G.3 MIR-FLICKR EXPERIMENTS
text: • β update policy: The initial value of β is set to 10−3, which is increased with an exponential schedule starting from the 50,000th until 1the 50,000th iteration. The value of β is then kept constant until the 1,000,000th iteration. The same annealing policy is used to trained the different β-VAEs reported in this work.
• Evaluation: The trained representation are evaluated following the well-known protocol described in Tschannen et al. (2019); Tian et al. (2019); Bachman et al. (2019); van den Oord et al. (2018). Each logistic regression is trained 5 different balanced splits of the training set for different percentages of training examples, ranging from 1 example per label to the whole training set. The accuracy reported in this work has been computed on the disjoint test set. Mean and standard deviation are computed according to the 5 different subsets used for training the logistic regression. Mean and variance for the mutual information estimation reported on the Information Plane (Figure 4) are computed by training two estimation networks from scratch on the final representation of the non-augmented train set. The two estimation architectures consist of 2 hidden layers of 2048 and 1024 units each, and have been trained with batches of size B = 256 for a total of approximately 25,000 iterations. The Jensen-Shannon mutual information lower bound is maximized during training, while the numerical estimation are computed using an energy-based bound (Poole et al., 2019; Devon Hjelm et al., 2019). The final values for I(x; z) and I(y; z) are computed by averaging the mutual information estimation on the whole dataset. In order to reduce the variance of the estimator, the lowest and highest 5% are removed before averaging. This practical detail makes the estimation more consistent and less susceptible to numerical instabilities.

heading: G.4.1 RESULTS AND VISUALIZATION
text: In this section we include additional quantitative results and visualizations which refer to the singleview MNIST experiments reported in section 5.2.
Table 2 reports the quantitative results used for to produce the visualizations reported in Figure 4, including the comparison between the performance resulting from different mutual information estimators. As the Jensen-Shannon estimator generally resulted in better performance for the InfoMax, MV-InfoMax and MIB models, all the experiments reported on the main text make use of this estimator. Note that the InfoMax model with the IJS estimator is equivalent to the global model reported in Devon Hjelm et al. (2019), while MV-InfoMax with the INCE estimator results in a similar architecture to the one introduced in Tian et al. (2019).
Figure 7 reports the linear projection of the embedding obtained using the MIB model. The latent space appears to roughly consists of ten clusters which corresponds to the different digits. This observation is consistent with the empirical measurement of input and label information I(x; z) ≈ I(z;y) ≈ log 10, and the performance of the linear classifier in scarce label regimes. As the cluster are distinct and concentrated around the respective centroids, 10 labeled examples are sufficient to align the centroid coordinates with the digit labels.

heading: H ABLATION STUDIES

heading: H.1 DIFFERENT RANGES OF DATA AUGMENTATION
text: Figure 8 visualizes the effect of different ranges of corruption probabily as data augmentation strategy to produce the two views v1 and v2. The MV-InfoMax Model does not seem to get any advantage from the use increasing amount of corruption, and it representation remains approximately in the same region of the information plane. On the other hand, the models trained with the MIB objective are able to take advantage of the augmentation to remove irrelevant data information and the representation transitions from the top right corner of the Information Plane (no-augmentation) to the top-left. When the amount of corruption approaches 100%, the mutual redundancy assumption is clearly violated, and the performances of MIB deteriorate. In the initial part of the transitions between the two regimes (which corresponds to extremely low probability of corruption) the MIB models drops some label information that is quickly re-gained when pixel corruption becomes more frequent. We hypothesize that this behavior is due to a problem with the optimization procedure, since the corruption are extremely unlikely, the Monte-Carlo estimation for the symmetrized Kullback-Leibler divergence is more biased. Using more examples of views produced from the same data-point within the same batch could mitigate this issue.
H.2 EFFECT OF β
The hyper-parameter β (Equation 5) determines the trade-off between sufficiency and minimality of the representation for the second data view. When β is zero, the training objective of MIB is equiv-
alent to the Multi-View InfoMax target, since the representation has no incentive to discard any information. When 0 < β ≤ 1 the sufficiency constrain is enforced, while the superfluous information is gradually removed from the representation. Values of β > 1 can result in representations that violate the sufficiency constraint, since the minimization of I(x; z|v2) is prioritized. The trade-off resulting from the choice of different β is visualized in Figure 9 and compared against β-VAE. Note that in each point of the pareto-front the MIB model results in a better trade-off between I(x; z) and I(y; z) when compared to β-VAE. The effectiveness of the Multi-View Information Bottleneck model is also justified by the corresponding values of predictive accuracy.

Review summaries:
In this paper, the authors extend the Information Bottleneck method (to build robust representations by removing information unrelated to the target labels) to the unsupervised setting. Since label information is not available in this setting, the authors leverage multi-view information (e.g., using two images of the same object) , which requires assuming that both views contain all necessary information for the subsequent label prediction task. The representation should then focus on capturing the information shared by both views and discarding the rest. A loss function for learning such representations is proposed. The effectiveness of the proposed technique is confirmed on two datasets. It is also shown to work when doing data augmentation with a single view.