Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a self-supervised learning with an information maximization criterion among alternative latent representations of the same input that naturally prevents dimensional collapse. It considers a second-order-statistics based mutual information measure, the log-determinant mutual information (LDMI), which is equivalent to Shannon mutual information under Gaussian distribution. A further first-order approximation to the log-determinant of the sum of two matrices is used to simplify the final objective to a Euclidean distance-based objective function regularized by the log-determinant of the feature covariance matrix. Consequently it  avoids the collapse problem establishes relevance among alternative representations by increasing the linear dependence among them.  Experiments on 4 image datasets show that the proposed approach gives better results than contrastive and non-contrastive methods.  
strengths_and_weaknesses: The paper is well written and easy to follow. However, there are several issues I'd like to address. The revised version has cleared my concerns.

1. This following statement is misleading: "A common self-supervision task is to match the latent representations that come from the distortions of the same input." It's true in computer vision, but not true in speech or natural language processing. 

2. The following statement is not quite right: "maximizing SMI between the representations of the same input is a challenging task whose implementation would require relatively large sample sizes [11, 12]." [11] demonstrates that when mutual information is large,  existing variational lower bounds degrade and exhibits either high bias or high variance. In fact McAllester and Stratos prove that serious statistical limitations are inherent to any lower bound method of measuring mutual information. More specifically, any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N). 

3. Collapse is a central concern for SSL with the same input in computer vision.

Reference:
David McAllester and Karl Stratos, Formal Limitations on the Measurement of Mutual Information, The Twenty Third International Conference on Artificial Intelligence and Statistics, 108:875-884, 2020.

questions: The proposed method maximizes Shannon mutual information under Gaussian distribution, and a further first-oder approximation is also used. 

Experiments on synthetic Gaussian data set in (Poole et al. 2019, McAllester and Stratos 2020) are need to see the effect of first-oder approximation when estimating mutual information, and the result should compare with the DoE method in McAllester and Stratos' paper. 

Finally are there any theoretical explanations to explain why it gets better results than contrastive and non-contrastive methods? The contrastive methods are essentially a lower bound method of Shannon mutual information. 
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper extensively addresses the collapse problem by proposing second-order statistics-based mutual information measure that reflects the level of correlation among the inputs. It claims that maximizing this correlative information measure between alternative representations of the same input serves two purposes: (1) it avoids the collapse problem by generating feature vectors with non-degenerate covariances; (2) it establishes relevance among alternative representations by increasing the linear dependence among them. All these are proved to simplify as a regularization term which acts as a natural barrier against feature space degeneracy. In general, the paper is well written and reasonably understandable. I found the underlying theory to be very strong, however the presented experiments aren't sufficient to show the strength of the work.
strengths_and_weaknesses: **Strengths**

(1) The paper is theoretically grounded on correlative information measure of representation. Additionally, it is very well reasoned, I like the way it is presented.

(2) On classification downstream task, the proposed methods has obtained state-of-the-art results one some datasets. Although, I found that the results are not robust and the comparison is not complete. Please also see my experiment related comments in the weaknesses section.

**Weaknesses**

(1) The experiments shown are bit limited and not very robust. Currently the experiments are only done on one downstream task i.e. classification. More experiments on other tasks, such as object detection, semantic segmentation, few-shot learning, representation transferability etc could have been interesting and could have strengthen the method. Moreover, ablation of the model on different hyperparameters should have been studied to understand the strength of the model. Furthermore, standard experimental strategies are not followed, as an example, for classification tasks, self-supervised models follow "linear" and "fine-tuning" strategies, which is not followed in this work.

(2) I wonder if there is any way to estimate the amount of mutual information maximized via the proposed CorInfoMax approach. It would be interesting to see some intermediate results or plots showing the evolution of estimated mutual information as the training progresses. Also it would have been interesting to see analysis on the performance behaviour of the model with evolution of mutual information. Does higher mutual information always guarantee a better trained model?
questions: (1) Currently, the abstract is quite long and puts a lot of unnecessary details and background information, which fades out the necessary information. It does not mention anything on the achieved results.

(2) Space missing in line 201 between "linear" and "(affine)"

(3) In line 330, the phrase "objective function" appears twice, which sounds repetitive.

(4) In equation (10), the dimensionality of the projector is bottleneck because of the correlation matrix computation. Could you please comment on that?

(5) In table 1, under the CIFAR-10 and ImageNet-100/ResNet-18 column, the boldfaced numbers are mistakenly written?

(6) It would be good to direct the main comparison table in the supplementary in the main paper.

(7) Appropriate direction to different sections of the supplementary material should be done from the main paper.

(8) In figure 2, the drawing of encoder and projector could have been in different colours. I mean if they share weights, they get the same colour, otherwise if they don't share weights, they get different colours.
limitations: The authors have not mentioned about the limitation of the work in the paper. A possible limitation could be the consideration of only the global information and not considering the local information, which might make this method only applicable for global task, such as classification. I suspect this type of model won't work very well on the task like involving fine-grained feature understanding.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduced CorInfoMax, a second order statistics based mutual information method to avoid collapse problem in self supervised learning (SSL). It used log-determinant mutual information (LDMI) as the measure between data pairs with different augmentations. Mathematically, the paper gave step-by-step formulation to derivate the final form of the objective via its first-order Taylor series approximation
Experimental results on CIFAR10/CIFAR100/ImageNet gives competitive results.

strengths_and_weaknesses: Strengths
- This work is novel and well intuited. Collapse issue is a well-known problem in SSL, traditionally Shannon Mutual Information (SMI) is applied to mitigate the collapse issue with several drawbacks, such as large batch requirement. The log-determinant mutual information (LDMI) naturally enforces latent distributions have non-degenerate distributions which can avoid collapse. Therefore, this method is well motivated and makes sense
- Clean and neat formula. Step by step formula gives clean loss format in Eq (8). It clearly indicated data under different augmentation should be closed to each other and log determinant formula enforces the model from model collapse. The result is elegent.
- This result shows state-of-the-art SSL performance. 

Weakness
- Mathematical formula is too heavy in the paper. It is a bit hard for audience to follow. I would strongly recommend a more detailed inference in any appendix to show how they are coming from
- More though experiments should be provided to demonstrate the effectiveness of the approach. For example, the paper argues the drawbacks for SMI and introduce LDMI measure as a better solution to avoid collapse. However, no evidence shown hot does it better avoids collapse. 

questions: As commented in strengths and weakness
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a new self-supervised learning method based on the log-determinant mutual information proposed earlier. Experiments on three small size datasets and one medium size dataset show the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1. The experimental results are good.
2. Overall, the article is easy to read.

Weaknesses:
1. The experimental part is relatively weak. 
2. The novel of this paper is limited. The main contribution of this paper is to directly generalize the earlier proposed log-determinant mutual information to the field of self-supervised learning.
3. Section 3 is superfluous.


questions: 1. This article only contains the results of classification tasks, and the performance of the proposed method on tasks such as detection, transfer, and semi-supervised learning also needs to be given.
2. The classification results on ImageNet-1000K should be reported.
3. The lines 159-162 are difficult to read. I don't understand that $h_{LD}^*\left( {x\left| y \right.} \right)$ can be treated as a measure of remaining uncertainty. I hope the author can give a detailed explanation.
4. This paper claims that the linear property of LDMI is the key to good performance. However, from " Self-Supervised Learning with Kernel Dependence Maximization", we can know that the nonlinear property of the Kernel Dependence is the key to good performance. I hope the author can give in-depth analysis and comparison.
5. Barlow-Twins is also proposed to explore the linear correlation between different augmentations of the same sample. I hope the author can give the difference and connection between the proposed method and Barlow-Twins, both experimentally and theoretically.
6. For large-scale dataset such as ImageNet-1000, the log-det cost is significant in practice.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper describes a self-supervised learning method based on an information maximization criterion that naturally prevents dimensional collapse. The authors consider the Shannon mutual information under the assumption that the data is Gaussian. A first-order approximation to the log-determinant of the sum of two matrices is used to simplify the final objective. Experiments on 4 image datasets show that the proposed approach gives better results than contrastive and non-contrastive methods.

Strengths:

1 - The paper is well written and easy to follow.
2 - The paper is theoretically grounded on correlative information measure of representation.
3 - Strong results on some downstream classification problems.
4 - Initially the experiments included only one downstream task regarding classification, but the paper has been updated to include also results for object segmentation and detection task.
5 - Novel and well motivated.
6 - state-of-the-art SSL performance.

Weaknesses:

- Some weaknesses are pointed out by reviewer GZwK, but these are not well justified.

Decision:

A majority of reviewers vote for acceptance. The only reviewer voting slightly towards rejection is GZwK, with a reasoning that is not well justified. For example, the main criticisms mentioned by reviewer GZwK

- The paper directly generalizes the earlier proposed log-determinant mutual information to the field of self-supervised learning. 
- this paper does not give a deep-going analysis that why the second-order statistics can play a important role in self-supervised learning

are not mentioned by any of the other reviewers.

Because of this, I have decided to accept the paper.


confidence: Certain
Decision: Accept