Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper extends the information bottleneck method of Tishby et al. (2000) to the unsupervised setting. By taking advantage of multi-view data, they provide two views of the same underlying entity.  Experimetal results on two standard multi-view datasets validate the efficacy of the proposed method.
I have three questions about this work.
1. The proposed method only provides two views of the same underlying entity, what about 3 or more views?
2. Can this method be used for multi-modality case?
3. What about the time efficiency of the proposed method?
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: In this paper, the authors extend the Information Bottleneck method (to build robust representations by removing information unrelated to the target labels) to the unsupervised setting. Since label information is not available in this setting, the authors leverage multi-view information (e.g., using two images of the same object) , which requires assuming that both views contain all necessary information for the subsequent label prediction task. The representation should then focus on capturing the information shared by both views and discarding the rest. A loss function for learning such representations is proposed. The effectiveness of the proposed technique is confirmed on two datasets. It is also shown to work when doing data augmentation with a single view.

Overall the paper is well motivated, well placed in the literature and well written. Mathematical derivations are provided. Experimental methodology follows the existing literature, seem reasonable and results are convincing. I do not have major negative comments for the authors. This is however not my research area and have only a limited knowledge of the existing body of work.

Comments/Questions:
- How limiting is the multi-view assumption? Are there well-known cases where it doesn't hold? I feel it would be hard to use, say, with text. Has this been discussed in the literature? Some pointers or discussion would be interesting.
- Sketchy dataset: Could the DSH algorithm (one of the best prior results) be penalized by not using the same feature extractor you used?
- Sketchy dataset: Can a reference for the {Siamese,Triplet}-AlexNet results be provided?
- Sketchy dataset: for reproducibility, what is the selected \beta?
- I find it very hard to believe that the accuracy stays constant no matter the number of examples per label used. How can an encoder be trained on 10 images? Did I misunderstand the meaning of this number? Can this be clarified?
- Again for reproducibility, listing the raw numbers for the MNIST experiments would be nice.
- If I understood the experiments correctly, "scarce label regime" is used for both the MIR-Flickr and MNIST datasets, meaning two different things (number of labels per example vs number of examples per label), which is slightly confusing.

Typos:
Page 1: it's -> its
Page 6: the the -> the
Page 7: classifer -> classifier
Page 8: independently -> independent

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This is a good multiview representation learning paper with new insights. The authors propose to learn variables z_1 and z_2, which are consistent, contain view-invariant information but discard as much view-specific information as possible.
The paper relies on mutual information estimation and is reconstruction-free. It is mentioned in some previous works (e.g. Aaron van den Oord et al. 2018), that reconstruction loss can introduce bias that has a negative effect on the learned representation.
Comparing to existing multiview representation learning approaches that try to maximize the mutual information between learned representation and the view(s), this paper clearly defines superfluous information that we should try to throw away and figure out how to obtain sufficiency learned representation for output. The authors also draw clear connections between a few existing (multiview) representation learning methods to their proposed approaches.
The experimental results on the right side of Figure 3, deliver a very interesting conclusion. In low-resource case, robust feature (obtained by using the larger beta, discarding more superfluous information) is crucial for achieving good performance. While when the amount of labeled data samples is enough, vice-versa.

Here are my major concerns:
1.	In the paper, the authors said the original formulation of IB is only applicable to supervised learning. That is true, but the variational information bottleneck paper [Alexander A. Alem et al. 2017] already showed the connection of unsupervised VIB to VAE in the appendix.
2.	I would not consider the data augmentation used to extend single-view data to “pseudo-multiview” as a contribution. This has been done before (e.g. in the multiview MNIST experiment part of the paper "On Deep Multi-View Representation Learning").
3.	Which MV-InfoMax do you really compare to? You listed a few of them: (Ji et al., 2019; Henaff et al., ´ 2019; Tian et al., 2019; Bachman et al., 2019) in the related work section.
4.	I think the authors should also make a more careful claim on their results in MIR-Flickr. 
I’d rather not saying MIB generally outperforms MV-InfoMax on MIR-Flickr, as MIB does not (clearly) outperform MV-InfoMax when enough labeled data is available for training downstream recognizers. But MIB does clearly outperform MV-InfoMax when scaling down the percentage of labeled samples used.
5.	Regarding baselines/experiments
a.	In Figure 4, it seems that VAE (with beta=4) outperforms MV-InfoMax. Why the ``"pseudo-second view" does not help Mv-Infomax in this scenario? Why VAE is clearly better than Infomax?
b.	In Figure 3, you might also tune beta for VCCA and its variants, like what you did for VAE/VIB in a single view. 
6.	Do you think your approach can be extended to more than two views easily? 
For me, it seems the extension is not trivial, as it requires o(n^2) terms in your loss for n views.
But this is minor.


review_title: Official Blind Review #2

Meta-review: This paper extends the information bottleneck method to the unsupervised representation learning under the multi-view assumption. The work couples the multi-view InfoMax principle with the information bottleneck principle to derive an objective which encourages the representations to contain only the information shared by both views and thus eliminate the effect of independent factors of variations. Recent advances in estimating lower-bounds on mutual information are applied to perform approximate optimisation in practice. The authors empirically validate the proposed approach in two standard multi-view settings.
Overall, the reviewers found the presentation clear, and the paper well written and well motivated. The issues raised by the reviewers were addressed in the rebuttal and we feel that the work is well suited for ICLR. We ask the authors to carefully integrate the detailed comments from the reviewers into the manuscript. Finally, the work should investigate and briefly establish a connection to [1].

[1] Wang et al. "Deep Multi-view Information Bottleneck". International Conference on Data Mining 2019 (https://epubs.siam.org/doi/pdf/10.1137/1.9781611975673.5)
Decision: Accept (Poster)