Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript provides an information theoretical perspective for multiview self-supervised learning (SSL). In specific, the authors show that a lower bound of mutual information is a useful objective function to learn informative representations. Additionally, optimizing such an objective function could be regarded as a generalization of both learning true latent variables (as in Zimmermann et al. (2021)) and separating content information from the irrelevant (as in Von Kugelgen et al. (2021)), from a generative model perspective.

Besides, the work also analyzes the loss functions of different SSL methods, e.g., contrastive learning based and projection reconstruction based ones, and demonstrates that these approaches do not maximize the mutual information lower bound exactly.

strength_and_weaknesses: **Strength**: 

It is an interesting idea to generalize previous model identification results of different generative models by using the mutual information lower bound. The authors also provide detailed analysis showing that many existing multiview SSL algorithms approximately maximize such a lower bound, which could help deepen the understanding of different SSL paradigms. However, the theoretical aspects as well as the empirical ones are not sufficiently clear in its current version.

**Weaknesses**:
1. The novelty seems not significant since a similar loss has been proposed and analyzed in Wang and Isola (2020) as uniformity and alignment terms. The uniformity is the entropy term here while the alignment corresponds to the reconstruction term (or cross entropy term). Could the authors spell out the differences and the advantages?
2. It is unclear that how Eq. (1) unifies both model identification theorems of Zimmermann et al. (2021) and Von Kugelgen et al. (2021). Specifically, contrastive loss (i.e., InfoNCE loss) is analyzed in Zimmermann’s work while $l_2$ matching loss is used in Von Kugelgen’s work. How is the equivalence between such a loss and (1) established? It is hard to tell how the proof techniques here differ from those of previous works.
3. The convergence rate results of the proposed methods are interesting but not very clear. Could the authors give more details on how the results are derived and perhaps more explanation?
4. It would help readers better understand if the detailed algorithm is listed.
5. It unclear how useful the proposed method is in practice since kernel density estimation is required for each iteration. Is there a wall time or complexity comparison?
6. In terms of experiment results, it might be more straightforward using synthetic experiments to showcase the desired theoretical properties. For example, how well the mutual information is maximized? How does that help identify the true latent variables and/or separate the content from the style, compared to the baselines? Is it necessary to learn good representations only if a tight bound is maximized?

Some minor comments and questions:
1. Both Zimmermann et al. (2021) and Von Kugelgen et al. (2021) assume that the generative function is the same for each view. As a comparison, in [1], it is shown that the generative function $g(\cdot)$'s do not need to be identical (in a more natural multivew setting) for each view in order to recover the true latent variables. Does the proposed criterion lead to similar model identification results under such a setting? 
2. The interpretation in the first paragraph on page 3 is not very clear to the reviewer. Could the authors be more specific about what information can be and is learned is? It seems the entropy term is simply a constrain (could be other alternatives) to avoid degenerated solutions.
3. Why is the proposed method more robust to changes of hyper-parameters, compared to other baselines?
4. How is the parameter of kernel band width selected? Or how the density is estimated?
5. In the Conclusion section, typo ‘or or isolating’.

[1] Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu. "Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective." In International Conference on Learning Representations. 2021.

clarity,_quality,_novelty_and_reproducibility: Overall, the manuscript is well-organized, but some parts can be made clearer (see weakness). The contributions are not significant since similar loss functions have been studied and the main theoretical results are not new. In terms of reproducibility, the detailed algorithm could be given for better clarity.
summary_of_the_review: This paper gives an interesting point of view for analyzing SSL using mutual information lower bound and proposes a method based on the criterion. However, both the theoretical and empirical results are not sufficiently clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a general framework to understand multi-view self-supervised learning (SSL) methods from a mutual information maximization perspective. Authors then use a naive lower bound of MI, i.e., entropy + reconstruction, as the objective function. The proposed methods, namely EntRecCont and EntRecDisc, achieve comparable performances to baselines.
strength_and_weaknesses: Strength:
1. The manuscript provides a good summary to previous multi-view SSL methods. 
2. It is interesting to see that a straightforward approximation of entropy (with traditional kernel density estimator or discrete entropy) and reconstruction can achieve comparable performances. 

Weaknesses:
1. The main result that multi-view (augmentation-based) SSL methods maximize MI has already been published in previous work, e.g., 
https://arxiv.org/abs/2010.02037. For acceptance, I would expect the authors to provide convincing arguments on why their work improve over previous work in this regard.
2. A large part of the paper reads as a report covering previous approaches to SSL, making it difficult to see the actual contribution of this work over previous work.
3. There are very few experiments included in the paper. Moreover, current result is very weak and does not demonstrate the advantage of EntRec:
3.1. The results show that the proposed methods perform worse than DINO (ICCV’21) which is already a year old. If the proposed methods are principled and direct approaches to the same objective which all others minimize, why do they perform worse?
3.2. The authors claim that the low bound in Eq. (1) helps obtain good representation and separate semantics from irrelevant information. 
However, it is not verified in detail.
clarity,_quality,_novelty_and_reproducibility: The manuscript is easy to follow. It seems not straightforward to implement EntRec. 
summary_of_the_review: This paper contributes some interesting ideas and also provides a good summary to previous literature. However, the theoretical contribution seems marginal. The experiment is also a bit weak.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes to unveil some relationships between mutual information maximization and multi-view self-supervised learning.
It claims to show that many SSL approaches can be framed as maximizing mutual information, and provide yet another approach that
considers maximizing mutual information as its objective (in an arguably 'more general' way). It then provides some empirical
results using two variants of this method. 

strength_and_weaknesses: **Strength:**
- The paper tackles a potentially interesting (though quite thoroughly studied) topic.
- The quality of the writing is above average. 

**Weaknesses:**

__Theory:__
- The theoretical contributions of the paper consists in showing that different self-supervised learning approaches all aim at maximizing the mutual information. This contribution falls short for several reasons:
  - Contrastive approaches are already known to approximate a lower bound on the mutual information even though this lower bound mostly holds when both views come from the same distribution. The paper provides slightly more precise, but rather straightforward analysis for when views come from different distributions.
  - The analysis for both BYOL and DINO is extremely handwavy: there is no clear explanation for why and how either of these algorithms would maximise the entropy term. In their current form, these paragraphs provide no precise statement relating BYOL's or DINO's objective to mutual information maximization, and I would argue in favor of removing them.
- I fail to understand what makes EntRecCont significantly different from CMC/SimCLR; in the case where $P_{Z_1} = P_{Z_2}$. When we add back the missing - in the definition of the entropy and remove the constant terms inside of the logarithm, we fall back to a standard NTXEnt loss (potentially up to normalization of the projections). Given how close the two are, it seems necessary to at least mention the equivalence/similarity. 
- I fail to understand the usefulness of mentionning Theorem 1, given both that the KDE approximation has clear limitations, probably making
 it a very bad approximation of Equation (1) (typically the provided lower bound cannot approximate Mutual Information higher than O(ln(k)), where k is the number of 'negatives' used, as mentionned in (McAlester et al. 2018)), and there are no experimental validation that any of the methods mentionned either verifies the hypothesis of the theorem, or matches their implications.
  
__Experiments:__

The experimental part requires a significant rework, as it currently gives a biased picture of the performance of the algorithms. More precisely:
  - The performance of BYOL in Table 1 is **significantly** lower than the performance reported in (Grill et al. 2020), namely the performance at 300 epochs is claimed to be 65.5% top-1, while the original paper mentions 72.5% **using the same architecture**. This changes significantly the story told by Table 1. While the difference may be explained by a change of hyperparameters, or image transformations, it seems disingenuous to place the algorithm in a setting that does not allow it to provide its best performance. I would suggest both putting back the original results in the table, and revising the claim of near SOTA performance.
  - Similarly, the results presented in Table 2 strongly contradicts results presented in (Grill et al. 2020), where BYOL loses less than 1% top-1 accuracy when going from batch size 4096 to 512, as opposed to 34.2%(??) in this paper. Again this significantly alters the conclusion provided by the paper.  
clarity,_quality,_novelty_and_reproducibility: Overall the paper is reasonably well written and flows quite well. However, it suffers a lot from juggling between precise and verifiable st
atements and loose/handwavy explanations. For instance:
- 'we show that many of these approaches are maximising an approximate lower bound...': while this may hold true in the case of contrastive methods, the content of the paper definitely does not **show** in any reasonable mathematical sense that BYOL or DINO maximises mutual information.
- 'In this section we demonstrate ...': Same here, it is still unclear after reading this section how BYOL and DINO relate to mutual informa
tion maximization. 

summary_of_the_review: The paper at hand provides a new take on the relationship between multi-view self-supervised learning and mutual information maximization. While the topic is interesting, the paper falls short in many directions. The main theoretical contributions of the paper are either already known (contrastive methods optimize a lower bound on the mutual information) or hardly made precise enough to be useful (non-contrastive methods e.g. BYOL and DINO optimize a lower bound on the mutual information).

I am more than doubtful of the empirical results: the results of at least one of the baselines provided are drastically below the results provided in the corresponding paper (BYOL @300 epochs is mentionned to achieve 65.8%, the original paper mentions 72.5%), and the batch size analysis displays the same trend (the paper claims that BYOL loses 34.2% accuracy when trained with batch size 512, while the original paper mentions that the loss in accuracy is about 1%).

Given both points, I strongly recommend rejecting this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper deals with the topic of mutual information (MI) maximization in multi-view self-supervised learning (SSL). The paper takes an information-theoretic perspective and shows that many current self-supervised learning methods maximize a lower bound on the MI between the representations of different views---a result that is not entirely novel. Based on a well-known decomposition of the MI into a cross-entropy and an entropy term, the paper argues that it can be beneficial to estimate these two terms individually. On the theoretical side, the authors draw a connection to two recent identifiability results related to nonlinear ICA. Empirically, the paper shows that models that estimate the individual (cross-)entropy terms perform on par with existing self-supervised learning methods.
strength_and_weaknesses: **Strengths:**
- The authors test two different estimators for the entropy terms, a KDE estimator and a plug-in estimator. It's also nice that the authors add information about the asymptotic behavior of these estimators, even if these results are not entirely novel.
- The authors reference critiques of the information-theoretic perspective on SSL objectives.
 


**Weaknesses**
- It is well known that many existing SSL methods can be connected via MI maximization and the contribution of the paper in this regard is not clear enough. For which methods is the MI maximization perspective actually new? This should be clearly stated in the abstract and/or introduction.
- Analogously, the decomposition of the MI into cross-entropy and entropy terms is well known and should not be stated as a contribution ("we show ...") in the abstract.
- The claim that the proposed objective "exactly optimises both the reconstruction and entropy terms" is misleading, because the model uses estimators.
- The "unification of current theoretical properties" is overstated. The two referenced papers (Zimmermann et al.; Kügelgen et al.) are quite similar in that they show identifiability results for multi-view nonlinear ICA and related settings. Specifically, Kügelgen et al. already discuss the connection of their work to the work from Zimmermann et al.
- The claimed "improved robustness" of one of the proposed models is not sufficiently clear from the experiments. 
clarity,_quality,_novelty_and_reproducibility: The quality is fair, but can be improved. The contributions should be stated more transparently and the empirical results could be more thorough to warrant the claims made in the abstract. The paper is mostly clear and easy to follow.

The novelty is limited and, in my opinion, insufficient (see Weaknesses).

summary_of_the_review: The decomposition of the MI and estimation of the two individual terms (cross-entropy and entropy) is an interesting research direction, which can lead to new insights into SSL. However, in its current form, the paper provides insufficient contribution compared to the results from previous work. In particular, the authors should state their contribution more clearly and verify the benefits of the proposed objective more thoroughly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This manuscript tries to unify several multi-view self-supervised learning algorithms into the same maximum mutual information framework, based on which a new variant, using KDE for entropy estimation, is proposed. 

Strength:

The authors did a good job connecting different multi-view SSL algorithms.

Weaknesses:

The reviewers think the paper does present insight on multi-view SSL algorithms, but the novel contribution is not crystal clear. The reviewers also found flaws in the experimental setup, which led to initial poor performance of the baselines. The authors shall take into account the reviewer comments to improve the paper.
justification_for_why_not_higher_score: Reviewers agree that the paper is not strong enough in terms of either theoretical analysis or empirical results.
Decision: Reject