Learning Robust Representations via Multi-View Information Bottleneck
Authors: Marco Federici
Anjan Dutta
Patrick Forré
Nate Kushman
Zeynep Akata
Venue:  (2020)
Keywords: Information Bottleneck
Multi-View Learning
Representation Learning
Information Theory

Abstract: The information bottleneck principle provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label while minimizing the amount of other, excess information in the representation. The original formulation, however, requires labeled data to identify the superfluous information.  In this work, we extend this ability to the multi-view unsupervised setting, where two views of the same underlying entity are provided but the label is unknown. This enables us to identify superfluous information as that not shared by both views. A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and label-limited versions of the MIR-Flickr dataset.  We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to common unsupervised approaches for representation learning.

Published as a conference paper at ICLR 2020
LEARNING ROBUST REPRESENTATIONS VIA MULTI-VIEL INFORMATION BOTTLENECK
Marco Federici
Anjan Dutta
Patrick Forré
University of Amsterdam m. cefterrciiuuaa...
University of Exeter
University of Amsterdam p Eorre
Nate Kushmann Microsoft Research
Zeynep Akata
University of Tuebingen
nkushmanemicrroofft.cco
zeynep..
ABSTRACT
The information bottleneck principle provides an iiformation-thhoreee method for representation learning. by training an encoder to retain information which is relevant for predicting the label while minimizing the amount of other, excess information in the representation. The original formulation, however, requires la- beled data to identify the superftuous information. In this work, extend this ability to the multi-view unsupervised setting, where two views of the same un- derlying entity provided but the label is unknown. This enables us to identify superfluous information as that not shared by both views. A thcoretical analysis leads the definition of sults on the Sketchy dataset and label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of stan- dard data augmentation techniques, empirically showing better generalization ca- pabilities when compared to common unsupervised approaches for representation learning.
multi-view model that produces state-of-the-arr re-
INTRODUCTION
The goal of deep representation learning (LeCun et al. 2015 to transform raw observational input, x, into typically lower-dimensional representation, z, that contains the information relevant for a given task or set of tasks. Significant progress has been made in deep learning via supervised representation learning, where the labels, y. for the downstream task are known while p(y) x) is learned directly (Sutskever et al.) 2012 Hinton et al. 2012). Due to the cost of acquiring large labeled datasets, recently renewed focus unsupervised representation learning seeks generate representations, z, that are useful for a wide variety of different tasks where little to no labeled data available (Devlin al. 2018 Radford et al. 2019).
Our work is based on the information bottleneck principle (Tishby et al.) 2000) where a represen- tation becomes less affected by nuisances by discarding all information from the input that is not useful for given task, resulting in increased robustness. In the supervised setting, one can directly apply the information bottleneck principle minimizing the mutual information between the data and its representation z, [(x; z), while simultaneously maximizing the mutual information between z and the label y (Alemi al.) 2017). the unsupervised setting. discarding only superfluous infor- mation is more challenging, as without labels the model cannot directly identify which information relevant. Recent literature (Devon Hjelm et al. 2019 van den Oord al., 2018) has focused on the InfoMax objective maximizing [xx, z) instead of minimizing it, to guarantee that all the predictive information is retained by the representation, but doing nothing to discard the irrelevant information.
In this paper, we extend the information bottleneck method to the unsupervised multi-view setting. To do this, we rely on a basic assumption of the multi-view literature that each view provides the same task-relevant information (Zhao et al.) 2017). Hence, one improve generalization by discarding all the information not shared by both views from the representation. We do this by maximizing the mutual information between the representations of the views Multi-View

---

Published as a conference paper at ICLR 2020
InfoMax objective) while at the same time eliminating the information not shared between them, since it is guaranteed to be superfluous. The resulting representations are more robust for the given task as they have eliminated view specific nuisances.
Our contributions are three-fold: (1) We extend the information bottleneck principle to the unsu- pervised multi-view setting and provide rigorous theoretical analysis of its application. (2) We define a new model | that empirically leads to state-of-the-an results in the low-label setting on two standard multi-view datasets, Sketchy and MIR-Flickr. (3) By exploiting data augmentation tech- niques, we empirically show that the representations learned by our model in single-view settings are more robust than existing unsupervised representation learning methods, connecting our theory to the choice of augmentation strategy.
2 PRELIMINARIES AND FRAMEWORK
The challenge of representation learning can be formulated finding a distribution p(z]x) that maps data observations X into a representation E Z, capturing some desired characteristics. When- ever the end goal involves predicting a label y, we consider only z that are discriminative enough to identify y. This requirement can be quantified by considering the amount of label information that remains accessible after encoding the data, and is known as sufficiency of z for y (Achille & Soatto, 2018):
Definition 1. Sufficiency: representation z of is sufficient for y if and only if T(x; y/z) = 0.
Any model that has access to a sufficient representation z must be able to predict at least as accurately if it has access to the original data instead. In fact, is sufficient for y if and only if the amount of information regarding the task is unchanged by the encoding procedure (see Proposition B.I in the Appendix):
I ylx) =0 - T(x;y) = T(y:z).
=
(1)
Among sufficient representations, the ones that result in better generalization for unlabeled data instances are particularly appealing. When x has higher information content than y. some of the information in x must be irrelevant for the prediction task. This can better understood by subdi- viding I [xx; z) into two components by using the chain rule of mutual information (sce Appendix A)
I(x;z)= =
zxx) +
I(y; z
(2)
superiluous information
predictive information
Conditional mutual information I(x; z represents the information in that is not predictive of y, i.e. superfiuous information. While I(y;z) determines how much label information is accessible from the representation. Note that this last term is independent of the representation as long as z is sufficient for y (see Equation] D.. As consequence, sufficient representation contains minimal data information whenever (x; is minimized.
Minimizing the amount of superfluous information can be done directly only in supervised settings. In fact, reducing I(x: z) without violating the sufficiency constraint necessarily requires making some additional assumptions on the predictive task (see Theorem B the Appendix). In the next section we describe the basis of our technique, strategy to safely reduce the information content of representation even when the label y is not observed, by exploiting redundant information in the form of an additional view on the data.
3 MULTI-VIEW INFORMATION BOTTLENECK
As a motivating example, consider Vi and V2 to be two images of the same object from different view-points and let be its label. Assuming that the object is clearly distinguishable from both Vi and let V2, any representation z containing all information accessible from both views would also contain the necessary label information. Furthermore, if z captures only the details that are visible from both pictures, it would eliminate the view-specific details and reduce the sensitivity of the representation to view-changes. The theory to support this intuition is described in the following where vi and V2 are jointly observed and referred to as data-views.
available at htw.alawhccceettthtt.ooooeeeeeee
Informati inraatiin--ee 1
2

---

Published as a conference paper at ICLR 2020
3.1 SUFFICIENCY AND ROBUSTNESS IN THE MULTI-VIEN SETTING
In this section we extend our analysis of sufficiency and minimality to the multi-view setting. Intuitively,
guarantee that is sufficient for predicting y even without knowing y by suring that 2 maintains all information which is shared by Vı and V2. This intuition relies on a basic assumption of the multi-view environment that the two views provide the same predictive information. To formalize this we define redundancy.
Definition 2. Redundancy: V1 is redundant with respect to V2 for y if and only if [ v1(v2) = 0
Intuitively, a view Vi redundant for task whenever it is irrelevant for the prediction of y if v2 is already observed. Whenever Vi and V2 are mutually redundant (v1 is redundant with respect to V2 for y. and vice-versa), we can show the following:
Corollary 1. Let V1 and V2 be mutually redundant views for target y and let be repre- sentation of 2f is sufficient for v2 (I(v1:V)) V2|z1) = 0) then 2y is as predictive for y as the joint observation of the two views ((ViV22))) ))= ((y
In other words, whenever it is possible to assume mutual redundancy, any representation which contains all the information shared by both views (the redundant information) is as predictive as their joint observation.
By factorizing the mutual information between V1 and 2 analogously Equation we can identify two components:
I(v1:2))= 2(v1:211V2
+
[(v2 Z1
supertluous information
predictive information For
Since I(V2: 21) has to be maximal if we want the representation to be sufficient for the label, we conclude that (1) can be reduced by minimizing i(vi;zz] z2/22)). This term intuitively represents the information ž1 contains which is unique to V1 and is not predictable by observing V2. Since we assumed mutual redundancy between the two views, this information must be irrelevant for the predictive task and, therefore, it can be safely discarded. The proofs and formal assertions for the above statements and Corollary I can be found in Appendix B
The less the two views have common, the more [(11;2) can be reduced without violating suffi- ciency for the label, and consequently, the more robust the resulting representation. At the extreme, 1 and v2 share only label information, in which case we can show that is minimal for y and our method is identical the supervised information bottleneck method without needing to access the labels. Conversely, if v1 and V2 are identical, then our method degenerates to the InfoMax principle since no information can be safely discarded (see Appendix E
3.2 THE MULTI- MULTI-VIEW VIEW INFORMATION BOTTLENECK LOss FUNCTION
Given v1 and va that satisfy the mutual redundancy condition for a label y, would like define an objective function for the representation 21 of V1 that discards as much information as possible without losing any label information. In Section 3.1 we showed that y by ensuring that the representation 2 of v, is sufficient for V2 and that decreasing [(21; v vilv)) will increase the robustness of the representation by discarding irrelevant information. So we can combine these two requirements using relaxed Lagrangian objective to obtain the minimal suffi- cient representation z for V2::
obtain sufficiency for
(0; lo(znivlvv2) A₁ (V
where 0 denotes the dependency on the parameters of the encoder po(zilvi), and A represents the Lagrangian multiplier introduced by the constrained optimization. Symmetrically. define loss L2 to optimize the parameters t of a conditional distribution pv (zzlv)) that defines a minimal sufficient representation Z2 of the second view V2 for V1:
L2(1; A2) L (Z2; v2lv1) X2 L (viiz2),
(4)
By defining 2 and 22 on the same domain and re- parametrizing the Lagrangian multipliers, the average of the two loss functions Li and La can be upper bounded follows:
3

---

Published as a conference paper at ICLR 2020
Multi-View
Single-Ve View
Algorithm 1: LMIB 2/ B)
if Multi- View then
[(o]) 2/ )
~ p(vi, v2);
else
)E >x(x); {(t)) (6) ) u pt))::
for i 1 to B do
Po (2(())V1)
Po (azz2/
(2 \v1)
po (22ll22))
4 (ce (z) 1, (x
end for
end if
DsKL
DSKL
for i <- 1 to B do
(1) (1)
po (z) |0 ( Pv (22((12)
Lm < DsK (sKu(pa(Zi ) (22 (22
21
1 (21222))
22
21
I (21;222)
21
end for return
4
E cl [((((1] (9 ))
B
Figure 1: Visualization our Multi View Information Bottleneck model for both multi-view and single-view settings, where [ę(=;; z2) refers to the sample-based parametric mutual information estimation. Whenever p(v1) and p(v2) have the same distribution, the two encoders can share their parameters.
CMIB(0, v) Loy (lai(Zi22 (Z);22) DskL(po(zilvi)lle (2elv))))))))
(5)
where DSKL represents the symmetrized KL divergence obtained by averaging the expected value of (po 21 vv [ (2Pp (2) and DKL. (Po (22/v2)) l [ [v)) for joint observations of the two views, while the coefficient defines the trade-off between sufficiency and robustness of the repre- sentation, which is a apper-parameter in this work. The resulting Multi- View Infomation Bottleneck (MIB) model (Equation 5) is visualized in Figure while the batch-based computation of the loss function is summarized in Algorithm TT
V2)
The symmetrized KL divergence DSKL (pok(po(z) (2, [V) v22) can be computed directly when- ever (z [v1) and Pvl (z2|v2) have known density. while the mutual information between the two representations Low (21; Z22 can be maximized by using any sample-basee differentiable mutual formation lower bound. We tried the Jensen-Shannor Jıs (Devon Hjelm et al. 2019 Poole et al.) 2019) and the InfoNCE INCE (van den Oord et af., (2018) estimators. These both require introducing an auxiliary parameteric model Ce(=1, #2) which is jointly optimized during the training procedure using re- re-parametrized samples from pa (o(z][v) and (22 v2). The full derivation for the MIB loss function can be found in Appendix
3.3 SELF-SUPERVISIOX AND INVARIANCE
Our method can also be applied when multiple views are not available by taking advantage of stan- dard data augmentation techniques. This allows learning invariances directly from the augmented data, rather than requiring them to be built into the model architecture.
By picking a class of data augmentation functions X W that do not affect label information, it is possible to artificially build views that satisfy mutual redundancy for y. Let t; and to be two random variables over T, then V1 : t (x) and V2 ;= (x) must be mutually redundant for y. Since data augmentation functions in T do affect label information (T(vi:y) I(v2: y) [(x;y)), a representation 21 of v; that is sufficient for V2 must contain same amount predictive information Formal proofs for this statement can be found in Appendix B.4
Whenever the two transformations for the same observation are independent (I(t;; tz/x) = 0), they introduce uncorrelated variations in the two views, which will be discarded when creating
4

---

Published as a conference paper at ICLR 2020
representation using our training objective. As an example, if represents set of small translations, the two resulting views will differ by a small shift. Since this information is not shared, any 21 which is optimal according to the MIB objective must discard fine- tnn-grriiedd details regarding the position.
To enable parameter sharing between the encoders, we generate the two views V1 and V2 by inde- pendently sampling two functions from the same function class T with uniform probability. As result, t and L2 will have the same distribution, and so the two generated views will also have the same marginals (p(v)) p(v2)). For this reason, the two conditional distributions pa (zi v1) and Py (22 v2) can share their parameters and only one encoder is necessary. Full (or partial) parameter sharing can be also applied in the multi-view settings whenever the two views have the same (or similar) marginal distributions.
RELATED WORK
The relationship between our method and past work on representation learning is best described using the Information Plane (Tishby al. 2000). In this setting, each representation z of for predictive task y can be characterised by the amount of information regarding the raw observation [xx; z) and the corresponding measure of accessible predictive information [yy: z) (z and y axis respectively on Figure(). Ideally, good representation would be maximally informative about the label while retaining minimal amount of information from the observations (top left corner of the parallelogram). Further details on the Information Plane and the bounds visualized in Figure 2 are described in Appendix[C
Thanks to recent progress in mutual informa- tion estimation (Nguyen et al. 2008 Ishmael Belghazi et al. 2018 Poole et al.) 2019). the InfoMax principle (Linsker 1988) has gained attention for unsupervised representation learn- ing (Devon Hjelm et al.] 2019 van den Oord jet al.) 2018). Since the InfoMax objective in- volves maximizing (x;; z). the resulting repre- sentation aims to preserve all the information regarding the raw observations (top right corner in Figure 2]
Fassible region
[(xy)
SufTiciency
1(x:2)
5 VAB
InfoMax
MV- JnfoMxx
MIB (ours)
((xy))
((xvaa) H(x))) ((S)
Supervisol IB
1xx;2
Information Plane determined by [xx z) (x-axis) and I(y; z) (y-axis). Different ob- jectives are compared based on their target.
Figure 2:
Concurrent work has applied the InfoMax prin- ciple in the Multi- setting (Ji et al., 2019, Hénaff et al.] 2019: Tian et al. 2019: Bachman
jt al. 2019), aiming maximize mutual infor-
mation between the representation z of a first data-view and second one V2. The target repre- sentation for the Multi View InfoMax (MV-InfoMax) models should contain at least the amount of information in x that is predictive for v2. targeting the region I(z; x) I V2) tion Plane (purple dotted line in Figure 2). Since the MV-InfoMax has no incentive to discard any information regarding from z, representation that is optimal according to the InfoMax principle is also optimal for any MV-InfoMax model. Our model with ,3 = 0 (Equation 5 belongs to this family of objectives since the incentive remove superfluous information is removed. Despite their success, Tschannen et al.] (2019) has shown that the effectiveness of the InfoMax models is due to inductive biases introduced by the architecture and estimators rather than the training objective it- self, since the InfoMax and MV-InfoMax objectives can be trivially maximized by using invertible encoders.
the Informa-
On the other hand, Variational Autoencoders (VAEs) (Kingma & Welling, 2014) define train- ing objective that balances compression and reconstruction error (Alemi et al. 2018) through an hyper-parameter B3. Whenever ,9 is close to 0, the VAE objective aims for a lossless representation, approaching the same region of the Information Plane as the one targeted by InfoMax (Barber & Agakov) 2003). When /3 approaches large values, the representation becomes more compressed, showing increased generalization and disentanglement (Higgins et al. 2017: Burgess et al. 2018), and, as 3 approaches infinity, I [z; goes zero. During this transition from low to high B, how- ever, there are no guarantees that VAEs will retain label information ( (Theorem/B.. B. ii the Appendix).
The path between the two regimes depends on how well the label information aligns with the induc-
5

---

various domain Published as a conference paper at ICLR 2020
tive bias introduced by encoder (Jimenez Rezende & Mohamed 2015 Kingma et al., 2016). prior
Tomczak & Welling 2018) and decoder architectures (Gulrajani et al. 2017 Chen et al. 2017))
The idea of discarding irrelevant information introduced in Tishby et al. (2000) and identified as one the possible reasons behind the generalization capabilities of deep neural networks by
Tishby & Zaslavsky (2015) Achille & Soatto 2018). Representations based on the information bottleneck principle explicitly minimize the amount of superfluous information in the representation while retaining all the label information from the data (top-left corner of the Information Plane in Figure 2 This direction of research has been explored for both single-view (Alemi et al. (018)) and multi-view settings (Wang et al. 2019). even if explicit label supervision is required train the representation
In contrast to all of the above, our work is the first explicitly identify and discard superfluous information from the representation in the unsupervised multi-view setting. This is because unsu- pervised models based on the B- VAE objective remove information indiscriminately without iden- tifying which part is relevant for teh predictive task, and the InfoMax and Multi- InfoMax methods do not explicitly try to remove superfluous information at all. The MIB objective, on the other hand, results in the representation with the least superfluous information, i.e. the most robust among the representations that are optimal according to Multi-View InfoMax, without requiring any additional label supervision.
5 EXPERIMENTS
In this section we demonstrate the effectiveness of our model against state-of sute-offthhe-at baselines in both the multi-view and single-view setting. the single-view setting, we also estimate the coor- dinates on the Information Plane for each of the baseline methods as well as our method to validate the theory in Section
The results reported in the following sections are obtained using the Jensen-Shannon LJs (Devon Hjelm et al. 2019; Poole et al. 2019) estimator, which resulted in better performance for MIB and the other InfoMax- Masox models (Table in the supplementary material). In order to facilitate the comparison between the effect of the different loss functions, the same estimator is used across the different models.
5.1 -ILW TASKS
We compare MIB on the sketch-based image retrieval (Sangkloy et image classification (Huiskes & Lew 2008) tasks with domain specific and prior multi-vien learning methods.
al. 2016) and Flickr multiclass
5.1.1 SKETCH- -KETCH-BASED IMAGE RETRIEVAL
Dataset.
The Sketchy dataset Sangkloy al. 2016) consists of 12,500 images and 75,471 hand- drawn sketches of objects from 125 classes. As in Liu et al. (2017), we also include another 60,502 images from the ImageNet (Deng et 2009) from the same classes, which results in total 73,002
natural object images. As per the experimental protocol of ZZhang et al. (2018), total of 6,250 sketches (50 sketches per category) are randomly selected and removed from the training set for testing purpose, which leaves 69,221 sketches for training the model.
Experimental Setup. The aketch-based image retrieval task is a ranking of 73,002 natural images according to the unseen test (query) sketch. Retrieval is done for our model by generating represen- tations for the query sketch as well as all natural images, and ranking the image by the euclidean distance of their representation from the sketch representation. The basclines specific ranking methodologies. Model performance is computed based on the class the ranked pictures corresponding to the query sketch. The training set consists of pairs of image V1 and sketch V2 randomly selected from the same class, to ensure that both views contain the equivalent label information (mutual redundancy).
Following recent work (Zhang et al. 2018 Dutta & Akata, 2019), we use features extracted from images and sketches by VGG (Simonyan & Zisserman 2014) architecture trained for classification

---

Published as a conference paper at ICLR 2020
Vi E I2 4096
V2 E R 4006
E [225]
Method
mAP 6PP
Prec@2000
SaN (Yu et al. 2017
0.208
0.292
GN Tmplle (Sangkloy
2016
0.529
0.716
"cat"
Siamese CNN Q et 2016
0.481
0.612
Siamese-AlexNer (Liu al.
2017
0,518
0.690
Triplet-AleXeet (Liu et al. 2017
0.573
0.761
DSH (Liu et al. 2017
0.711
0.866
GDH (Zhang al. 2018
0.810
"apple"
MV-InfoMax
0.008
0.008
MIB MIB" (64-bits)
0.856=0.005
0.808
0.851 0.004
0.8341:0.003
Table 1: Examples of the two views and class label from the Sketchy dataset (on the left) and comparison between MIB and other popular models in literature on the sketch-based image retrieval task (on the right). denotes models that use a 64-bits binary representation. The results for MIB corresponds to 3
on the TU-Berlin dataset (Eitz al. 2012 The resulting flattened 4096-dimensional feature vectors are fed to our image and sketch encoders to produce a o4-dimensional representation. Both encoders consist of neural networks with hidden layers of 2048 and 1024 units respectively. Size of the representation and regularization strength are tuned on validation sub-split. We evaluate MIB on five different train/test splits and report mean and standard deviation in Table 1.1.1 Further details on training procedure and architecture are in Appendix
Results. Table 5.I.I shows that the our model achieves strong performance for both mean average precision (mAP@al) all) and precision at 200 (Prec @ 200), suggesting that the representation is able to capture the common class information between the paired pictures and sketches. The effective- ness of MIB on the retrieval task can be mostly attributed to the regularization introduced with the symmetrized KL divergence between the two encoded views. addition to discarding view-private information, this term actively aligns the representations of vı and V2, making the MIB model especially suitable for retrieval tasks
5.1.2 MIR-FLICKR
Dataset. The MIR-Flickr dataset (Huiskes Lew 2008) consists of 1M images annotated with 800K distinct user tags. Each image represented by a vector of 3,857 hand-crafted image features (v1). while the 2,000 most frequent tags are used to produce 2000-dimensional multi-hot encoding
(v2) for each picture. The dataset is divided into labeled and unlabeled sets that respectively contain 975K and 25K images, where the labeled set also contains 38 distinct topic classes together with the user tags. Training images with less than two tags removed, which reduces the total number of training samples to 749,647 69,647 pairs (Sohn et al.) 2014 Wang et al., 2016). The labeled set contains different splits of train, validation and test of size TOK/5K/TOK respectively.
Experimental Setup. Following standard procedure in the literature (Srivastava & Salakhutdinov 2014 Wang et al. 2016). we train our model on the unlabeled pairs of images and tags, Then a multi-label logistic classifier is trained from the representation of 10K labeled train images to the corresponding macro-categories. The quality of the representation is assessed based on the performance of the trained logistic classifier on the labeled test set. Each encoder consists of multi-layer perceptron of hidden layers with ReLU activations learning two (024-dimensional representations and 22 for images Vi and tags v2 respectively. Examples of the two views, labels, and further details on the training procedure are in Appendix G
Results. Our MIB model is compared with other popular multi-view learning models in Figure for 3 = (Multi-Vier InfoMax), 33 = 1 and 8 = 10-3 (best on validation set). Although the tuned MIB performs similarly to Multi-View InfoMax with a large number of labels, it outperforms when fewer labels are available. Furthermore, by choosing a larger 3 the accuracy of our model drastically increases in scarce label regimes, while slightly reducing the accuracy when all the labels are observed (see right side of Figure 3) This effect is likely due to a violation of the mutual
"hhese results are included only for completeness, the Multi-View InfoMax objective does not produce consistent representations for the views so there no forwight-forward way to use for ranking.

---

Published as a conference paper at ICLR 2020
Method
mAP
Original Inputs
0.48
0.75
CCA
0,529
Contrastive'
Herrmann Blunsom 2014
0,565
0.70
DCCA (Andrew ct al. (2013)
0,573
MVAE-var' (Ngiam et al. 2011
0.595
mAP
0.65
VCCA
Wang et al. 2016)
0,605
Model
VCCA-privvte"
0.615
MV IVV-Infoaax (
0.60
bi- bi-VCCA-privatee
0.626
MV-InfoMax
0.751-0.02
MIB ( 1)
MIB (33 = 10-3)
0.749-10000
0.55
MIB (3 = 1)
0.688+00.02
2 x010
2 x 10
10
Number of Labeled Examples
Figure Left: mean average precision (mAP) of the classifier trained on different multi-view representations for the MIR- task. Right: comparing the performance for different values of and percentages of given labeled examples (from 1% up to 100%). Each model uses encoders of comparable size, producing 1024 dimensional representation. results from Wang et al. (2016).
redundancy constraint (see Figure in the supplementary material) which can be compensated with smaller values of / for less aggressive compression.
A possible reason for the effectiveness of MIB against some of the other baselines may be its ability to use mutual information estimators that do not require reconstruction. Both Vuew VAE (MVAB) and Deep Variational CCA (VCCA) rely on a reconstruction term to capture cross-modal information, which can introduce bias that decreases performance.
5.2 SELF-SUPERVISED SINGLE-VIEN TASK
In this section, we compare the performance of different unsupervised learning models by measuring their data efficiency and empirically estimating the coordinates of their representation the Infor- mation Plane. Since accurate estimation mutual information extremely expensive (McAllester & Stratos 2018), we focus relatively small experiments that aim to uncover the difference be- tween popular approaches for representation learning.
Dataset. The dataset is generated from MNIST by creating the two views, V1 and V2, via the application of data augmentation consisting of small affine transformations and independent pixel corruption to each image. These are kept small enough ensure that label information is not effected, Each pair of views is generated from the same underlying image, so no label information used in this process (details in Appendix G).
Experimental Setup. To evaluate, we train the encoders using the unlabeled multi-view dataset just described, and then fix the representation model. A logistic regression model is trained using the resulting representations along with subset of labels for the training set, and we report the accu- racy of this model on a disjoint test set is standard for the unsupervised representation learning literature (Tschannenn et al. 2019 Tian et al. 2019 van den Oord et al. 2018). We estimate 7 7(x; 2) and I(y; z) using mutual information estimation networks trained from seratch on the final repre- p(x. y ()))()))))))))))))) All models are trained using the same encoder architecture consisting of 2 layers of 1024 hidden units with ReLU activations, resulting in 64-dimensional representations. The same data augmentation procedure was also applied for single-view architectures and models were trained for I million iterations with batch size 64.
sentations using batches of joint samples (x) (1), z
) 2
Results. Figure 4 summarizes the results. The empirical measurements of mutual information ported on the Information Plane are consistent with the theoretical analysis reported in Section 4 models that retain less information about the data while maintaining the maximal amount of pre- dictive information, result in better classification performance at low-label regimes, confirming the hypothesis that discarding irrelevant information yields robustness and more data-efficient represen-
8

---

Published as a conference paper at ICLR 2020
10
2.0
03-
Model
(8 00)
Bm B-21
B-2-4
R2:y1
1.5
VAE ($ 44)
0.
VAE 88
05-
InfoMax
0.5
2=
MV-InfoMax
8
2
MIB (8 = 1)
8.0
107 107
12f
12
(xx; 22)
Number of Labeled Examples
Figure 4: Comparing representations obtained with different objectives on MNIST dataset. The empirical estimation of the coordinates the Information Plane (in nats the left) is followed by the respective classification accuracy for different number of randomly sampled labels (from example per label up 6000 examples per label). Representations that discard more observational information tend to perform better in scarce label regimes. The measurements used to produce the two graphs reported in Appedix G.4.1
tations. Notably, the MIB model with / = 1 retains almost exclusively label information, hardly decreasing the classification performance when only one label is used for each data point.
CONCLUSIONS AND FUTURE WORK
In this work, introduce Multi-Vies Information Bottleneck, novel method for taking advantage of multiple views to produce robust representations for downstream tasks. In our experiments, we compared MIB empirically against other approaches in the literature on three such tasks: sketch- based image retrieval, multi-view and unsupervised representation learning. The strong performance obtained in the different areas show that Multi-Vies Information Bottleneck can be practically ap- plied to various tasks for which the paired observations are either readily available or artificially produced. Furthermore,
positive results on the MIR-Flickr dataset show that our model can work well in practice even when mutual redundancy holds only approximately.
There are multiple extensions that we would like to explore in future work. One interesting direction would be considering more than two views. In Appendix D we discuss why the mutual redundancy condition cannot be trivially extended to more than two views, but we still believe such an exten- sion is possible. Secondly, we believe that exploring the role played by different choices of data augmentation could bridge the gap between the Information Bottleneck principle and the literature on invariant neural networks (Bloem-Reddy & Whye Teh (019), which are able exploit known symmetries and structure of the data to remove superfluous information.
ACKNOWLEDGMENTS
We thank Andy Keller, Karen Ullrich, Maximillian Ilse and the anonymous reviewers for their feed- back and insightful comments. This work has received funding from the ERC under the Horizon 2020 program (grant agreement No. 853489). The Titan Xp and Titan V used for this research were donated by the NVIDIA Corporation.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of Invariance and Disentanglement in Deep Representations. JMLR, 2018.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep Variational Informa- tion Bottleneck. In ICLR, 2017.
Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy. Fixing Broken ELBO. In ICML. 2018.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In ICML, 2013.

---

Published as a conference paper at ICLR 2020
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning Representations by Maximiz- ing Mutual Information Across Views. arXiv, 2019.
David Barber and Felix Agakov. The im algorithm: A variational approach to information maxi- mization. In NIPS, 2003.
Benjamin Bloem- Bloem-Reddy and Yee Whye Teh. Probabilistic symmetry and invariant neural networks. arXiv, 2019.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in B-VAE. arXiv, 2018.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational Lossy Autoencoder. In ICLR, 2017.
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
R Devon Hjelm, Alex Fedorov, Samuel Marchilnnr, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Anjan Dutta and Zeynep Akata. Semantically tied paired cycle consistency for zero-shot sketch- based image retrieval. In CVPR, 2019.
Mathias Eitz, James Hays, and Mare Alexa. How do humans sketch objects? ACM TOG, 2012.
Y. Gong, S. Lazebnik, A. Gordo, and F. Perronnin. Iterative quantization: procrustean approach to learning binary codes for large-scale image retrieval. TPAMI, 2013.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. PixeIVAE: A Latent Variable Model for Natural Images. In ICLR, 2017.
Olivier Hénaff, Ali Razavi, Carl Doersch, M. Ali Eslami, and Aaron van den Oord. Data- Efficient Image Recognition with Contrastive Predictive Coding. arXiv, 2019.
Karl Moritz Hermann and Phil Blunsom. Multilingual Distributed Representations without Word Alignment. In ICLR, 2014.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, al. Deep neural networks for acoustic modeling in speech recognition. SPM, 2012.
Mark Huiskes and Michael S. Lew. The mir flickr retrieval evaluation. In ICMIR, pp. 39 39-43, 2008.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair. Yoshua Bengio, Aaron Courville, and R Devon Hjelm. MINE: Mutual Information Neural Estimation. In ICML, 2018.
Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant Information Clustering for Unsupervised Image Classification and Segmentation. In ICCV, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In ICML, 2015.
Diederik Kingma and Max Welling, Auto-Encoding Variational Bayes. In ICLR, 2014.
10

---

Published as a conference paper at ICLR 2020
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In NIPS, 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015. 2015.
R. Linsker. Self-organization in perceptual network. Computer, 1988.
Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, and Ling Shao. Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval. In CVPR, 2017.
David McAllester and Karl Stratos. Formal Limitations on the Measurement of Mutual Information. arXiv, 8.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. Multi- modal deep learning. In ICML, 2011.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. In NIPS, 2008.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On Varia- tional Bounds of Mutual Information. ICML, 2019.
Y. Qi, Y. Song, H. Zhang, and J. Liu. Sketch-based image retrieval via siamese convolutional neural network. ICIP, 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM TOG, 2016.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scall Scale Image Recognition. arXiv, 2014.
Kihyuk Sohn, Wenling Shang, and Honglak Lee. Improved multimodal deep learning with variation of information. In NIPS, 2014.
Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep boltzmann machines. JMLR, 2014.
Wanhua Su, Yan Yuan, and Mu Zhu. A relationship between the average precision and the area under the roe curve. In ICTIR, 2015.
Ilya Sutskever, Geoffrey E Hinton, and A Krizhevsky, Imagenet classification with deep convolu- tional neural networks. In NIPS, 2012.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Multiview Coding. arXiv, 2019.
Naftali Tishby and Noga Zaslavsky. Deep Learning and the Information Bottleneck Principle. In ITW, 2015.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. arXiv, 2000.
Jakub M. Tomezak and Max Welling. VAE with a VampPrior. In AISTATS, 2018.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On Mutual Information Maximization for Representation Learning. arXiv, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Pre- dictive Coding. arXiv, 2018.
Qi Wang, Claire Boudreau, Qixing Luo, Pang-Ning Tan, and Jiayu Zhou. Deep multi-view infor- mation bottleneck. In SIAM, 2019.

---

Published as a conference paper at ICLR 2020
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep Variational Canonical Corre- lation Analysis. arXiv, 2016.
Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. Sketch-a-Net that Beats Humans. IJC,, 2017.
Jingyi Zhang, Fumin Shen, Liu, Fan Zhu, Mengyang Yu, Ling Shao, Heng Tao Shen, and Luc Van Gool. Generative domain migrrtioon hashing for sketch-to-imes image retrieval. In ECCV, 2018.
Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-vier view learning overview: Recent progress and new challenges. IF, 2017.
12

---

Published as a conference paper at ICLR 2020
A PROPERTIES OF MUTUAL INFORMATION AND ENTROPY
In this section enumerate some of the properties of mutual information that used prove the theorems reported in this work. For any random variables w, x, y and z:
(i)) Positivity:
I(x;y) 0x1(xx 20
(P2) Chain rule:
(xxy:z) = 1(y:)) + ((x;2ly)
(P3) Chain rule (Multivariate Mutual Information):
yx ((y:z) 1o:0--ly::0 xy;
(P4) Positivity of discrete entropy: For discrete x
H(x) 2 0, Hxly)) 2
PPs) Entropy and Mutual Information
H(x) H(xy) I(x;))
B THEOREMS AND PROOFS
In the following section we prove the statements reported the main text of the paper. Whenever random variable is defined to be representation of another random variable conditionally independent from any other variable the system once x is observed. This does not imply that must be deterministic function of but that the source of stochasticity for is independent of the other random variables. As a result whenever z is representation x:
state that
I(2;a|xb) = 0,
for any variable (or groups of variables) and in the system.
B. ON SUFFICIENCY
Proposition B.1. Let x and be random variables with joint distribution p(x, Let be representation of x, then z sufficient for y if and only if I(x:y) = I(y:z)
Hypothesis:
(H1) a is a representation x:
x: T(y;2|x) = 0
Thesis:
(Ti) 1(x;y/z)=0 =0 + 1(x:y) = 1(y;2)
Proof.
(xxy)/z) (Pt) (xx;)) (asn))-(x;;y (xx;; (Pa) ((x:v) (y::)) ((::
(H1) ((x;y) - 1(y:z)
Since both 7(x;8) and T(y:z) non-negative (Pi), f(x: =0 - T(y;z) T(x:y) 1
13

---

Published as a conference paper at ICLR 2020
B.2 No FREE GENERALIZATION
Theorem B.1. Let x, z and be random variables with joint distribution p(x.y.z) v.z.. Let z' be representation of x that satisfies I [(x; [(x; z'), then is always possible to find label y for which v' is not predictive for y while is.
Hypothesis:
(H)) z' a representation of x: zxy
(H2) T(x:2) > 2(x:2)
Thesis:
(Ti) 1xx;))<<<x;;2> (x;;)) = 3y.((y;))> > I(y;2') =
Proof. By construction.
1. We first factorize x as a function two independent random variables (Proposition 2.1 Achille & Soatto 2018)) by picking such that:
(C1) 1yy
(C2) x= x= f(z',y)
for some deterministic function f. Note that such always exists.
2. Since x is function of y and z':
(C4) I(x; =
Considering zy;
(Pa) 1(y:z|x) + (xx;;;
1y;z)
(PP)
2 T(x:y;2)
(P)) 1(x; z) -(x;
(8) 1(x;z) -1 (xx:zlyz') - 1 (xx 2x;;
(C)) [(x; z) 1xx;,
(8)) T(x; 2) 1(x;2'ly) + 2(x2)
(P))
2 (xx;) (xx;z'ly)
(P) 1(x; z) 1(x;z') ) + +1xx;;;z
(P)) 7(x;2) T(x;2') T(y: z') 7(y;
(P)
2 1(x; z) 1(x;2') 1ly;
(11) 1(x; z) 1(x:2')
(Ha) >
Since I(y:z') = by construction and [ [y:; z) > 0, the y built 1. satisfies the conditions reported in the thesis.
14

---

Published as a conference paper at ICLR 2020
Corollary B.1.1. Let z' be representation of that discards observational information. There is always a label y for which z' not predictive, while the original observations are. Hypothesis:
(H1) x is discrete
(H2) z' discards information regarding x: I(z';x) < Hx)
Thesis:
(Ti) 3y.[(y;x) 1(yz2)=0
Proof. By construction using Theorem B.1
1. Set z = x:
(C1) 2(x;2) (1b) H(x) - Hxl2)) (111 H(x)
2. x(2;;)) < (x) (C I(z';x) 1(x;z)
Since the hypothesis are met, we conclude that there exist such that T(y:x) > T(y;2') =0
n
B.3 MULTI-VIEW
B.3.1 MULTI- VIEW REDUNDANCY AND SUFFICIENCY
Proposition B.2. Let V1, V2, y be random variables with joint distribution p(v1, V2, Let 21 be a representation of V1, then:
1(vi:ylzı) s [(viivalzz) ((viylvv)
Hypothesis:
(H1) 21 is representation of v:: 2(y;2 =0
Thesis:
(Ti) I(vi;y/zı) S v2/z)) + I(vi:y/v2)
Proof. Since z is representation of Vi:
(C1) 1(y:21/22v1) =
Therefore:
T(v1:N/z1) (Pa)
I(v1:y(21ve) + +[vi::223l2z)
I(Vi:y|V2) [(v1:y;2] + V2:1111)
(P))
[(V1:v(v2) -1y: v2) [ +
(P))
< I(V1:y|ve) 1(y:2//2vv) I(v::
(8)) I(V1:y V2) + I(v1:v2;ylan)
(P))
I(vi:y|v2) [(vi:valzz) 1 zi))
(P)
I(vi:y/V2) [(vi:v2/zz1
15

---

Published as a conference paper at ICLR 2020
Proposition B.3. Let be redundant view with respect for y. Any representation z1 of that is sufficient for V2 is also sufficient for y.
Hypothesis:
(H1) 21 is representation v1: T(y: 21 v2 =0
(H2) Vi is redundant with respect to f2 y: I(y;Vilvo) = 0
Thesis:
(Ti) I(Vi:velzı) =0 I(v1:y/zz)
Proof.
Using the results from Theorem B.2
I(vi;y|zı) (Th
[(vi:y[v2) + 7(v1; v2|z1) (Ha) I(vi;
Therefore 1(v15 =0
=
Theorem B.2. Lel Vi, V2 and y be random variables with distribution p(V1, V2. Let a be representation of V1, then
I(y:21) 2 I(y;viv2) - I(v1:v2lzz) I(v1sy/v2) I(V2;y/V1)
Hypothesis:
(H1) 21 is representation v1: T(y; =0
Thesis:
(Ti) 1(y:z1) 2 -wvivo))-wiiivoolaooV/vvvvvvvv I(v2;y/vi)
Proof.
I(y;2)) (13) [(y;zi|vive) Fly: v1
(H1)
(P) V(y:v T(y;
(2) 7(y:v T(y; v1lz1) 7(y:
( 1(y:viv)) 1ly: v1211 -
(PA) ((y:viV)) [yy:vilz))
(4)) T(y:Viv2) 7(y; V1 v1 T(y: 7(y;2/|v1)
(1))
1(y:viv2) (y:: v1 (2)) -(y:v2//v1)
(Progga
I(y:viv2) - I(vi:v/v2) I(vi:V2 - ((y:
Corollary B.2.1. Let V and V2 be mutually redundant views for y. Let z be representation of Vi that is sufficient for 22. Then:
T(y; 21) = T(viV2;))
Hypothesis:
16

---

Published as a conference paper at ICLR 2020
(H1) is representation v1 I(y; Zı 21/2) =0
(112) Vi and V2 are mutually redundant for y: [lyivilv2) 1(y:v2/vi)
(Ha) 21 is sufficient for V2: T(V2; V1(2)= =0
Thesis:
(Ti) Z(y: = ((viv2sy)
Proof. Using Theorem B.2
(Th..
21
1(y:viv)) - =(v1sylv2) (attaa)--((vviiiii) - (y: v2(v1)
(Ha) I(y; T(V1:V2/211
(Ha) I(y;
Since T(y: S T(y; is consequence of the data processing inequality, we conclude that T(y:z1) V(y:
B.4 SUFFICIENCY AND AUGMENTATION
Let and y be random variables with domain and respectively. Let T be a class of functions t: X + W and let t and tz be random variables over T that depends only on x. For the theorems and corollaries discussed this section, we are going to consider the independence assumption that can be derived from the graphical model G reported in Figure
y
1j
X
t2
tx
Z1
(x()
Figure 5: Visualization of the graphical model G that relates the observations x, label y, functions used for augmentation t1, t2 and the representation Z1.
Proposition B.4. Whenever [(t1(x);)) T(t2(x):)) T(x:y) the views t, (x) and tz(x) must be mutually redundant for y.
Hypothesis:
(H+) Independence relations determined by g
Thesis:
(Ti) I(ti(x);y) = 1(x;)) [(ti(x);xlt((x)) ((t())):)))x
Proof.
1. Considering we have:
17

---

Published as a conference paper at ICLR 2020
(C1) xxtxx)) =
(C2) I(y; (x))
x) =0
2. Since (x) is uniquely determined by x and tz:
(Ca) v(tx) = 0
3. Consider (x)x
x(txx))
((xx(x))xx
1(y:x(t2(x)) (ani))--((yyyy;; (xxxx
(8)
< x(y;x(tz(x))
( 1(yx) - (xx:x:t(xx)
16) I(y:) (2yit((x)) - (2y:t((x)(x)
(3)
I(;;x) ((y;t((x)) - (a:x))))-((ytt)))))))) ((y:t2(x)|txx)) + 12);
(C)) x(y;x) T(y:t2(x)) txy;
(2) I(y; x) - (2 = (2y;1(())|x - t(y;
(P))
2 1(y:x) ((y;t2(x)) - 1 (y: t2)))()))x))
(C))
2 I(y:x) t2xx))
Therefore 1(y:x) [(y;t((x)) ((2x))))) =
The proof for T(y;x) [(y;t((x)) = ((tx 0 is symmetric, therefore we conclude (x)) = I(ta(x):g) = T(x:y) = [(tt(x):)ltxxx)) ((x()))))))))
Theorem B.3. Let [(t)(x):y) T(tz(x);y) T(x:y). Let bn be a representation of fx If 2 is sufficient for (x() then T(x;y) 1(y;
Hypothesis:
(Hi) Independence relations determined by G
(H2) I(ti(x);)) (((()))))=(((((())))()))) (t2((x):))
(xx;)
Thesis:
(Ti) (x) = I(x:y) I(y:
Proof. Since (x) is redundant for (x) (Proposition B.4 any representation 2 of (x( that is
sufficient for t2 (x) must also be sufficient for y (Theorem B.2). Using Proposition B.1 we have (1) (x)).. Since 1(y:t((x)) = x(y;x) by hypothesis, conclude (x: I(y:21)
C INFORMATION PLANE
Every representation of must satisfy the following constraints:
18

---

Published as a conference paper at ICLR 2020
0 I I(y;z)= (xxy): The amount of label information ranges from 0 to the total predic- tive information accessible from the raw observations yx:
I(y: z) [(x: z) - [(:: z) + H (xly): The representation must contain more information about the observations than about the label. When x is discrete, the amount of discarded label information 1(x: y) T(y;2) must be smaller than the amount of discarded observa-
tional information (x) [xx: x), which implies T(x; 2) S T(y: 2) + Hx
Proof. Since z representation of x:
(C1) z(y: 0
Considering the four bounds separately:
1. I(y: z) 2 0: Follows from Pi
2. [(x; z) 2 I(y: z): Follows from:
[(x;z)
[(x:2ly) + [xx
(P) y(x:
(P) I(y:z) - [yy:
I(y:z)
3. I(y:2) S x): Data processing inequality
(Pa) I(y;z(x) + xx
1(y;z)
1(y;
(p) I(x:y) I(x:ylz)
I(x:y)
4. 7(x;2) T(y:)) + Hxxy)): For discrete x:
(Pa) [(x; z|y) + yx
[(x; z)
(Pa) T(x: z|y) T(y: z) - T(y:2lx)
(G) T(x; zly) + T(y:z)
(P))
< z(x; + (x(x))) + ((;;))
(B) H(xly) T(y:2)
Note that the discreetness of required only prove bound 4. For continuous x bounds 1, 2 and 3 still hold.
D NON-TRANSITIVITY OF MUTUAL REDUNDANCY
The mutual redundancy condition between two views V1 and V2 for label y can not be trivially ex- tended to arbitrary number of views, as the relation is not transitive because of some higher order interaction between the different views and the label. This can be shown with simple example.
Given three views V1, V2 and V3 and task y such that:
19

---

Published as a conference paper at ICLR 2020
V1 and V2 mutually redundant for y
V2 and are mutually redundant for y
Then, we show that V1 is not necessarily mutually redundant with respect to V3 fory.
Let V1. V2 and Va be fair and independent binary random variables. Defining y as the exclusive or operator applied to and V3 (y ;= Vi XOR V3), we have that 4(v1;y) = yv = 0. In this settings, V1 and V2 are mutually redundant for y:
(v2) H(vi/v2y) H(v)) H(v)) y(v2: = (v2lvi) (evvv>>H((V2VV)) (V222v (v2)) H =
I(v
Analogously, V2 and V3 are also mutually redundant for y as the three random variables are not predictive for each other. Nevertheless, V1 and V3 mutually redundant for y:
I(v1:y/v3) H(vilvs) H(vilvay) H(1)) =
1(v3:y/v1) H(v3lv1) H(v3lviy) (v3)) =1
Where H (V1 vvy) = H(v3lviy) = 0 follows from V1 V3 XOR y and V3 v, XOR y, while
(v11) H(v3) holds by construction.
This -ounter-intuitive higher order interaction between multiple views makes our theory non-trivial generalize to more than two views.
E EQUIVALENCES OF DIFFERENT OBJECTIVES
Different objectives in literature can be seen as a special case of the Multi-Vice Information Bot- uleneck principle. In this section show that the supervised version of Information Bottleneck is equivalent to the corresponding Multi- View version whenever the two redundant views have only label information in common. A second subsection show equivalence between InfoMax and Multi- View Information Bottleneck whenever the two views are identical.
E.1 MULTI- VIEW INFORMATION BOTTLENECK AND SUPERVISED INFORMATION BOTTLENECK
Whenever the two mutually redundant views and va have only label information in common (or when one of the two views is the label itself) the Multi-Vies Information Bottleneck objective is equivalent to the respective supervised version. This be shown by proving that 7(v1; V2) [(v1; Z ly), i.c. representation of vi that is sufficient and minimal for is also sufficient and minimal for y.
Proposition E.1. Let V1 and va be mutually redundant views for label y that share only label information. Then a sufficient representation 2, V, f V2 that is minimal for v2 is also a minimal representation for y.
Hypothesis:
(H1) and mutually redundant for y: I(vi:y/v2) + I(ve:ylv))
(112) Vi and va share only label information: ((11:22) 1(viy)
(Hz) 2i is sufficient for Vz: V(V1;V2/Z1) =
Thesis:
(Ii) 1(v1izılv2) ((v1;2yly)
Proof.
20

---

Published as a conference paper at ICLR 2020
1. Consider [v]:
(P)) 1(v1izi[v2) + (2v::
[(vizz)
(P%) 1(v1;ZI[V2) + ((V1:v2) (2v1v22)z))
(Ha) [(v1;z][V2) + V(v1:v2)
(#)) T(v1;2] V2) + [(v1:y)
2. Using Corollary from (H2) and (Hş) follows I(v1:y(21) =
3. [(v1;z) can be alternatively expressed as:
I(v1:21) (P4) I(vi:zıly) + ( [(V1:;
(P4) I(V1:zlly) [(v1:y) I(v1:y/zz)
(Com I(vi; + ((v1yy)
Equating and 3, we conclude /(v1;Zı\V2) = I(vi;zily), therefore 21 which minimizes [ 2i V2) also minimizing ( ly). When [(v1;zı|y) minimal, 1(y; 21) is also minimal (see equation 2
E.2 MULTI- VIEW INFORMATION BOTTLENECK AND INFOMAX
Whenever V1
V2, representation 1 of Vı that is sufficient for Va must contain all the orig- inal information regarding V1. Furthermore since T(v1;21|v2) 0 for every representation, no superfluous information can be identified and removed. As consequence, minimal sufficient rep- resentation 2 of v for V2 is any representation for which mutual information is maximal, hence InfoMax.
F LOSS COMPUTATION
Starting from Equation create the minimal sufficient representations ž₁ and 22 respectively:
we consider the average of the losses (0; and C2(1/; A2) that aim to
C L (6, th; A1, A2) To [ v2) + & (v2; vilv)) Xj A1 + (v: (6)
2
Considering 21 and 22 the same domain Z, Lo(viizi v2) can be expressed
log Po 1 z|Vi = v1)
To To(Vi;Zı/v3 E, v2) Bx-po
Po (0 2 2|V2 v2)
log Po z|vi = vi) Pv (22 = 2|V2 = 022
-o Ex (a
Po x|v2 22) po(zi 2/ U2)
= DDa, (v) < DKL (z, (z2)))
DRL.
Note that the bound is tight whenever Pe (Zg v2) coincides with (2(z) This happens when- ever Zn and Z2 produce consistent encoding. Analogously (2/vi is upper bounded by
DKL (po (122/22)) [22]lpp (21|v1)))
To (V22; 21) can be rephrased as:
(Pa)
[o(2;; V2)
Tow (Z1; 22V2) - (21;
Iew (z); Z2V22
(Z1: (21: (2) + 1eu(zi:vela
Tow (Z1;22) R2)
21

---

Published as a conference paper at ICLR 2020
Where follows from Zg representation of V2. The bound reported in this equation is tight whenever Z2 is sufficient for 21 (Low Z1: V2 V2/22) = 0). This happens whenever Zg contains all the information regarding 2 (and therefore v1). Once again, the same bound symmetrically be used to show To(22; v1) 2 Tox (2); Therefore, the loss function in Equation 6 can he upper-bounded with:
Ai + 2
Li (0,4); A1; S
Lo (21; 822
Where:
1 DKu (pw (2)o\v)))ll
(sKL(po(zalvii)lld = (
Lastly, multiplying both terms with 3 = A1+ and re-parametrizing the objective, we obtain:
LMIB 2 3) = -Lpy (21:22) + Dww.lpooeallviillpppp.oooooo
(8)
G EXPERIMENTAL PROCEDURE AND DETAILS
G.1 MODELING
The two stochastic encoders po(zi|vı) and Py (zelV2)) are modeled by Normal distributions parametrized with neural networks (me o and (Hi, o respectively:
po(zi|vi) = N (v)) Pv (Z2/v2) =- N (zaluu (V2), of (v2))
Since the density of the two encoders can be evaluated, the symmetrized KL--divergence in equa- tion can be directly computed. On the other hand, Tog (21; Z2) requires the of mutual infor- mation estimator.
To facilitate the optimization, the hyper-parameter / is slowly increased during training, starting from small value & 10-4 to its final value with an exponential schedule. This is because the mutual information estimator is trained together with the other architectures and, since it starts from a random initialization, it requires an initial warm-up. Starting with bigger / results in the encoder collapsing into fixed representation. The update policy for the has not shown strong influence on the representation, as long as the mutual information estimator network has reached full capacity.
during training
All the experiments have been performed using the Adam optimizer with learning rate of 10-4 for both encoders and the estimation network. Higher learning rate can result instabilities in the training procedure. The results reported in the main text relied on the Jensen-Shannon mutual information estimator (Devon Hjelm et al.) 2019) since the InfoNCE counterpart (van den Oord et al. 2018) generally resulted in worse performance that could be explained by the effect of the factorization of the critic network (Poole et al. 2019).
G.2 SKETCHY EXPERIMENTS
Input:
The two views for the sketch-based clasddlllsiiiie classification task consist of 4096 dimensional sketch and image features extracted from two distinct VGG-16 network models which were pre-trained on images and sketches from the TU-Berlin dataset Bitz et al. (2012) for end- to-end classification. The feature extractors are frozen during the training procedure of for the two representations. Each training iteration used batches of size B 128.
Encoder and Critic architectures: Both sketch and image encoders consist of multi-layer perceptrons of 2 hidden ReLU units size 2,048 and 1,024 respectively with an output of size 2x64 that parametrizes mean and variance for the two Gaussian posteriors. The critic architecture also consists of a multi layer perceptron of hidden ReLU units of size 512.
The initial value of /3 is set to 10-4, Starting from the 10,000th training iteration, the value of 3 is exponentially increased up 1.0 during the following 250,000 training iterations. The value of is then kept fixed to one until the end of the training procedure (500,000 iterations).
3 update policy:
22

---

Published as a conference paper at ICLR 2020
Evaluation: All natural images used both training sets and retrieval galleries. The 64 dimensional real outputs of sketch and image representation are compared using Eu- clidean distance. For having fair comparison other methods that rely binary hashing (Liu et al. 2017 Zhang et al. (2018), we used Hamming distance on a binarized represen- tation (obtained by applying iterative quantization Gong al. (2013) on real valued representation). We report the mean average precision (mAP@ @al@@lll all) and precision at top- rank 200 (Prec @ 200) (Su al. (2015) both the real and binary representation evaluate our method and compare it with prior works.
G.3 MIR-FLICKR EXPERIMENTS
Figure 6: Examples of pictures VI, tags V2 and category labels y for the MIR-Flickr dataset (Srivas- tava & Salakhutdinov 2014). As visualized is the second row, the tags are not always predictive of the label. For this reason, the mutual redundancy assumption holds only approximately.
V1 E 38357
V2 E (0, 1} 2000
y E (0, 1]
"watermelon", "hilarious", "chihuahua", "dog"
"animals", "dog",
"food"
"colors",
"cores".
"clouds", "sky", "structures"
"centro".
"comercial", "building"
Input: Whitening is applied the handcrafted image features. Batches of size B 128 are used for each update step.
Encoders and Critic architectures: The two encoders consists of multi layer perceptron of 4 hidden ReLU units of size 1,024, which exactly resemble the architecture used in Wang et al. (0016). Both representations 21 and Ze have a size of 1,024, therefore the architecture output total of 2x1 1,24 parameters that define mean and variance of the respective factorized Gaussian posterior. Similarly to the Sketchy experiments, the critic is consists of multi-layee perceptron of 2 hidden ReLU units of size 512.
The initial value of 8 is set to 10-8, Starting from I. th iteration, 3 is set to exponentially increase up to 1.0 (and 10-3) during the following 150,000 iterations.
3 update policy:
Evaluation:
Once the models are trained on the unlabeled set, the representation of the 25,000 labeled images is computed. The resulting vectors are used for training and eval- uating a multi-label logistic regression classifier on the respective splits. The optimal pa- rameters (such as :)) for our model are chosen based on the performance on the validation set. In Table 3 we report the aggregated mean the test splits as the final value mean average precision value.
G.4 MNIST EXPERIMENTS
Input:
The two views vi and V2 for the MNIST dataset are generated by applying small translation ([0-10]%), rotation ([-15,15] degrees), scale (090, shear ([-15,15) de- grees) and pixel corruption (20%). Batches of size = 64 samples are used during train- ing.
Encoders, Decoders and Critic architectures: All the encoders used for the MNIST experiments consist of neural networks with two hidden layers of 1,024 units and ReLU activations, producing 2x64-dimensional parameter vector that is used to parameterize mean and variance for the Gaussian posteriors. The decoders used for the VAE experiments also consist of the networks of the same size. Similarly, the critic architecture used for mutual information estimation consists of two hidden layers of 1,204 units each and ReLU activations.
23

---

Published as a conference paper at ICLR 2020
3 update policy: The initial value of 3 is set 10-3, which is increased with an exponen- tial schedule starting from the 50,000t until 1tee 50,000th iteration. The value of / is then kept constant until the 1,000,000th iteration. The same annealing policy is used to trained the different 3- V-Ss reported in this work.
The trained representation are evaluated following the well-know protocol described in T Tschannnn et al. 2 Tian et al. (2019); [Bachman et al. (2019); yan den Oord et al. (2018). Each logistic regression is trained 5 different balanced splits of the training set for different percentages of training examples, ranging from example per la- bel to the whole training set. The accuracy reported in this work has been computed on the disjoint test set. Mean and standard deviation are computed according to the 5 different subsets used for training the logistic regression. Mean and variance for the mutual informa- tion estimation reported on the Information Plane (Figure 4 are computed by training two estimation networks from scratch the final representation of the non-augmented train set. The two estimation architectures consist of 2 hidden layers of 2048 and 1024 units each, and have been trained with batches of size B 256 for total of approximately 25,000 iterations. The Jensen-Shannon mutual information lower bound is maximized dur- ing training, while the numerical estimation are computed using an energy- based bound (Poole et al., 2019 Devon Hjelm et al. (019)). The final values for [(x; z) and [y; z) are computed by averaging the mutual information estimation on the whole dataset. In order to reduce the variance of the estimator, the lowest and highest 5% are removed before av- eraging. This practical detail makes the estimation more consistent and less susceptible to numerical instabilities.
Evaluation:
G.4.1 RESULTS AND VISUALIZATION
In this section we include additional quantitative results and visualizations which refer to the single- view MNIST experiments reported in section 5.2
Table reports the quantitative results used for to produce the visualizations reported in Figure including the comparison between the performance resulting from different mutual information esti- mators. As the -Sanenon- estimator generally resulted in better performance for the InfoMax, MV-InfoMax and MIB models, all the experiments reported mator. Note that the InfoMax model with the Iss estimator is equivalent to the global model reported Devon Hjelm et al. (2019), while MV-InfoMax with the INCE estimator results in similar archi- tecture to the one introduced in Tian et al. 2019).
the main text make use of this esti-
Test Accuracy [%
Model
[(x; 2) [nats]
[2; y) [nats]
10 Ex
50 Ex
3750 Ex
60000 Ex
VAE (beta=0)
: 0,2
43.8 1 1.6
65.6 1 3.3
89.0 1 0.4
91.3 0.1
VAE (beta=4)
7.5
2.0 0.2
55.9 + 2.6
$1.4 4.0
94.2 0.3
96.0 + 0.2
VAE (beta=8)
3.0 0.5
1.1
43.8
288
61.1
4.8
81.9 1.1
87.2 2 0.6
InfoMax (TNCE)
12.8 1.7
0.2
25.4 1.9
39.6
6.72
74.6 14.6000 0.6
InfoMax (Iss)
2.2
35.0
52.5
1.14
78.2 1 1.2
MV-InfoMax (INCE)
12.2
0.2
50.2 3.6
75.8 3.8
94.6
96.5 0.1 0.1
MV-InfoMax (Tss)
11.1
2.2
$4.0 6.1
78.3
4.4
9.31
95.90 1 1.08 0.08
MIB (8- = 1, INCE)
4.6
2.1
81.8 - 5.0
92.7 0.9
9..19
97.75 + 0.05
MIB (8- Ls)
2.2
0.2
97.1 0.2
97.2 0.2
97.70
97.82 - 0.01
Table 2: Comparison of the amount of input information [(x; label information y), and accuracy of a linear classifier trained with different amount of labeled Examples (Ex) for the models reported in Figure 4 Both the results obtained using the Jensen-Shannel IISD (Devon Hjelm al. 2019
Poole et al.] 2019) and the InfoNCE INCE (van den Oord et al. 2018) estimators are reported.
Figure 7 reports the linear projection of the embedding obtained using the MIB model. The latent space appears to roughly consists of ten clusters which corresponds to the different digits. This observation is consistent with the empirical measurement of input and label information I(x: z) ss Izz ss log 10, and the performance of the linear classifier in scarce label regimes. As the cluster are distinct and concentrated around the respective centroids, 10 labeled examples are sufficient to align the centroid coordinates with the digit labels.
24

---

Published as a conference paper at ICLR 2020
Figure 7: Linear projection of the embedding obtained by applying the MIB encoder to the MNIST test set. The 64 dimensional representation projected onto the two principal components. Different colors are used to represent the 10 digit classes.
H ABLATION STUDIES
H.1 DIFFERENT RANGES OF DATA AUGMENTATION
Figure 8 visualizes the effect of different ranges of corruption probabily as data augmentation strat- egy to produce the two views V1 and V2. The MV-InfoMax Model does not seem to get any ad- vantage from the use increasing amount of corruption, and it representation remains approximately in the same region of the information plane. On the other hand, the models trained with the MIB objective are able to take advantage of the augmentation to remove irrelevant data information and the representation transitions from the top right corner of the Information Plane (no- (no-augmentation) to the top-left. When the amount of corruption approaches 100%, the mutual redundancy assump- tion clearly violated, and the performances tions between the two regimes (which corresponds to extremely low probability of corruption) the MIB models drops some label information that is quickly re-gained when pixel corruption becomes more frequent. We hypothesize that this behavior is due problem with the optimization proce- dure, since the corruption are extremely unlikely, the Monte-Carla estimation for the symmetrized Kullback-Leii Leibler divergence is more biased. Using more examples of views produced from the same data-point within the same batch could mitigate this issue.
MIB deteriorate.
the initial part of the transi-
MV-infoMax
(8
10
2.0
0.8
[z:y)
1.5
accurayy
Examples per Label
[|z;)
2.25 10
10-5
p= 10
23
0.5
p- 8.0-10
0.4
6000
MIB (8
0.0
255
5.0
75
10.0
12.5
15.0
0.2
0.4
0.4 Corruption Percentage
0.6
0.8
1.0
[x:
z)
Corruption Percentage
Figure 8: Visualization of the coordinates on the Information Plane (plot on the left) and predic- tion accuracy (center and right) for the MV-InfoMax and MIB objectives with different amount of training labels and corruption percentage used for data-augmentation.
H.2 EFFECT OF
The parpmeter 3 (Equation determines the trade-off between sufficiency and minimality of the representation for the second data view. When / is zero, the training objective of MIB is equiv-
25

---

Published as a conference paper at ICLR 2020
alent to the Multi- View InfoMax target, since the representation has incentive to discard any information. When 0 < 3 I 1 the sufficiency constrain is enforced, while the superfluous informa- tion is gradually removed from the representation. Values of 3 can result in representations that violate the sufficiency constraint, since the mininization of T(x: V2) is prioritized. The trade-off resulting from the choice of different is visualized in Figure and compared against B-VAE. Note that in each point of the pareto-front the MIB model results in a better trade-off between / [xx; z) and / (y; z) when compared 3-VAE. The effectiveness of the Multi-View Information Bottleneck model is also justified by the corresponding values of predictive accuracy.
B-VAE
2.5
10
2.0
0.8
2=
Examples per Label
1.5
0.4
6000
A=
MIB
NV-InfoMax
0.0
2.5
5.0
((x:
7.5
Z)
20.0
125
25.0
2- 2
2
2
2
Figure 9: Visualization the coordinates on the Information Plane (plot on the left) and predic- tion accuracy (center and right) for the 3-VAE, Multi-View InfoMax and Multi- View Information Bottleneck objectives with different amount of training labels and different values the respective hyperparameter B.
26


Reviews:
[{"review_id": "SkglAno0YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors extend the Information Bottleneck method (to build robust representations by removing information unrelated to the target labels) to the unsupervised setting. Since label information is not available in this setting, the authors leverage multi-view information (e.g., using two images of the same object) , which requires assuming that both views contain all necessary information for the subsequent label prediction task. The representation should then focus on capturing the information shared by both views and discarding the rest. A loss function for learning such representations is proposed. The effectiveness of the proposed technique is confirmed on two datasets. It is also shown to work when doing data augmentation with a single view.\n\nOverall the paper is well motivated, well placed in the literature and well written. Mathematical derivations are provided. Experimental methodology follows the existing literature, seem reasonable and results are convincing. I do not have major negative comments for the authors. This is however not my research area and have only a limited knowledge of the existing body of work.\n\nComments/Questions:\n- How limiting is the multi-view assumption? Are there well-known cases where it doesn't hold? I feel it would be hard to use, say, with text. Has this been discussed in the literature? Some pointers or discussion would be interesting.\n- Sketchy dataset: Could the DSH algorithm (one of the best prior results) be penalized by not using the same feature extractor you used?\n- Sketchy dataset: Can a reference for the {Siamese,Triplet}-AlexNet results be provided?\n- Sketchy dataset: for reproducibility, what is the selected \\beta?\n- I find it very hard to believe that the accuracy stays constant no matter the number of examples per label used. How can an encoder be trained on 10 images? Did I misunderstand the meaning of this number? Can this be clarified?\n- Again for reproducibility, listing the raw numbers for the MNIST experiments would be nice.\n- If I understood the experiments correctly, \"scarce label regime\" is used for both the MIR-Flickr and MNIST datasets, meaning two different things (number of labels per example vs number of examples per label), which is slightly confusing.\n\nTypos:\nPage 1: it's -> its\nPage 6: the the -> the\nPage 7: classifer -> classifier\nPage 8: independently -> independent\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx75ajTtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a good multiview representation learning paper with new insights. The authors propose to learn variables z_1 and z_2, which are consistent, contain view-invariant information but discard as much view-specific information as possible.\nThe paper relies on mutual information estimation and is reconstruction-free. It is mentioned in some previous works (e.g. Aaron van den Oord et al. 2018), that reconstruction loss can introduce bias that has a negative effect on the learned representation.\nComparing to existing multiview representation learning approaches that try to maximize the mutual information between learned representation and the view(s), this paper clearly defines superfluous information that we should try to throw away and figure out how to obtain sufficiency learned representation for output. The authors also draw clear connections between a few existing (multiview) representation learning methods to their proposed approaches.\nThe experimental results on the right side of Figure 3, deliver a very interesting conclusion. In low-resource case, robust feature (obtained by using the larger beta, discarding more superfluous information) is crucial for achieving good performance. While when the amount of labeled data samples is enough, vice-versa.\n\nHere are my major concerns:\n1.\tIn the paper, the authors said the original formulation of IB is only applicable to supervised learning. That is true, but the variational information bottleneck paper [Alexander A. Alem et al. 2017] already showed the connection of unsupervised VIB to VAE in the appendix.\n2.\tI would not consider the data augmentation used to extend single-view data to “pseudo-multiview” as a contribution. This has been done before (e.g. in the multiview MNIST experiment part of the paper \"On Deep Multi-View Representation Learning\").\n3.\tWhich MV-InfoMax do you really compare to? You listed a few of them: (Ji et al., 2019; Henaff et al., ´ 2019; Tian et al., 2019; Bachman et al., 2019) in the related work section.\n4.\tI think the authors should also make a more careful claim on their results in MIR-Flickr. \nI’d rather not saying MIB generally outperforms MV-InfoMax on MIR-Flickr, as MIB does not (clearly) outperform MV-InfoMax when enough labeled data is available for training downstream recognizers. But MIB does clearly outperform MV-InfoMax when scaling down the percentage of labeled samples used.\n5.\tRegarding baselines/experiments\na.\tIn Figure 4, it seems that VAE (with beta=4) outperforms MV-InfoMax. Why the ``\"pseudo-second view\" does not help Mv-Infomax in this scenario? Why VAE is clearly better than Infomax?\nb.\tIn Figure 3, you might also tune beta for VCCA and its variants, like what you did for VAE/VIB in a single view. \n6.\tDo you think your approach can be extended to more than two views easily? \nFor me, it seems the extension is not trivial, as it requires o(n^2) terms in your loss for n views.\nBut this is minor.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJxd8x7rqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends the information bottleneck method of Tishby et al. (2000) to the unsupervised setting. By taking advantage of multi-view data, they provide two views of the same underlying entity.  Experimetal results on two standard multi-view datasets validate the efficacy of the proposed method.\nI have three questions about this work.\n1. The proposed method only provides two views of the same underlying entity, what about 3 or more views?\n2. Can this method be used for multi-modality case?\n3. What about the time efficiency of the proposed method?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the information bottleneck method to the unsupervised representation learning under the multi-view assumption. The work couples the multi-view InfoMax principle with the information bottleneck principle to derive an objective which encourages the representations to contain only the information shared by both views and thus eliminate the effect of independent factors of variations. Recent advances in estimating lower-bounds on mutual information are applied to perform approximate optimisation in practice. The authors empirically validate the proposed approach in two standard multi-view settings.
Overall, the reviewers found the presentation clear, and the paper well written and well motivated. The issues raised by the reviewers were addressed in the rebuttal and we feel that the work is well suited for ICLR. We ask the authors to carefully integrate the detailed comments from the reviewers into the manuscript. Finally, the work should investigate and briefly establish a connection to [1].

[1] Wang et al. "Deep Multi-view Information Bottleneck". International Conference on Data Mining 2019 (https://epubs.siam.org/doi/pdf/10.1137/1.9781611975673.5)
Decision: Accept (Poster)