Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to learn generalizable representations without labels. To this end, this paper proposes Maximum Entropy Coding (MEC), inspired by the principle of maximum entropy in information theory. MEC uses the minimal lossy coding and the Taylor series approximation to make the maximum entropy estimation feasible. Extensive experimental results show that MEC consistently outperforms existing SSL methods under various downstream tasks. Furthermore, this paper demonstrates that MEC is robust to various hyperparameters (e.g., smaller batch sizes) and architectures (e.g., ViTs).

strengths_and_weaknesses: The proposed method has several strengths: (1) simplicity and scalability, (2) high performance across various downstream tasks, and (3) robustness to various hyperparameters and architectures. I think the extensive experiments well demonstrate these strengths. Furthermore, MEC can be interpreted as batch-wise and feature-wise SSL methods, so I feel this part is also interesting.

One major concern with this paper is that what information the proposed method can learn while existing SSL methods cannot is unclear. This paper mentioned that existing SSL methods introduce some "biases" while MEC does not. What are the biases exactly? Could you provide some empirical evidence supporting that MEC can learn less-biased representations? I think this paper does not explain why and how MEC can outperform other SSL methods.

Some minor concerns are provided in the **Questions** section.

To sum up, although some explanation seems insufficient, I feel the empirical results are strong. Hence, I vote for Weak Accept.

questions: - Too small top margins in p2-3. **This seems to violate formatting rules.** This should be modified in the rebuttal period. Also, other spaces (e.g., caption spaces) seem too narrow.
- Could you test more transfer learning experiments with fine-grained classification benchmarks? These benchmarks are also important to evaluate the transferability of the learned representations. For the transfer setup, I recommend seeing SimCLR and BYOL papers.

limitations: This paper well addressed the limitations and the potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a joint-embedding objective called Maximum Entropy Coding, which is close to MCR^2 [74]. This method directly optimizes the information content by minimizing the coding length function. This proposed method unifies batch-wise objectives (SimSiam) and feature-wise objectives (BarlowTwins). Practically, this is implemented as a Taylor series expansion. 
The proposed method shows strong performances (when combined with all existing techniques, including exponential moving average and asymmetric network) on a wide range of experiments, including ImageNet linear probe, semi-supervised classification, transfer learning on video tasks, and object detection. 
However, this paper actually uses all the tricks, including exponential moving average from BYOL, but only mentions it in the ablation study/supplementary material. It is obvious that the strong result of this method mostly comes from this mixture of existing design components.
strengths_and_weaknesses: Strengths:
1. The proposed method is strongly supported by theoretical motivation: the maximum entropy principle. 
2. The proposed method shows strong performances on a wide range of experiments (when combined with all existing techniques, including exponential moving average, asymmetric network), including ImageNet linear probe, semi-supervised classification, transfer learning on video tasks, and object detection.
3. The authors provided experimental details, including pseudo-code and all hyperparameters. 

Weaknesses:
1. The strong result mostly comes from a mixture of designing components, e.g., exponential moving average and asymmetric architecture. It is questionable how much advantage comes from the maximum entropy coding loss. In fact, the only contribution of the proposed method over existing methods is that it has a higher-order correction. Table 5e clearly shows that the extra two orders (2->4) only increase accuracy by 0.3%.
2. It's unclear what the main contribution is to this paper. 
(a) The "principle of maximum entropy" is no different from requiring independent features, which is already proposed in MCR^2, BarlowTwins, VICReg, and whitening SSL. 
(b) The higher-order approximation, as mentioned above, only has a 0.3% linear probe improvement. All other experiments fail to verify the advantage. 
(c) The practical advantage of this mixture model. The authors only compare their work to fundamental frameworks (BYOL, BarlowTwins, SwAV), but there are many similar mixture models, e.g., arXiv:2204.07141, arXiv:2104.14548, arXiv:2109.12909, arXiv:2012.13493, arXiv:2201.05119

====== after rebuttal comments =====

I am more convinced by the authors that the empirical advantage is misleading. The empirical advantage is exactly just 0.3%. Actually, in response point 1, Barlow Twins are supposed to reach 73.5%. So here, the extra two orders approximately indeed only provide a 0.1% improvement.

I think this is totally ok. A good theory unifies several ideas and provides incremental improvements. But it is unacceptable that the paper tries to hide this point.

The main figure and most of the results tables will mislead readers that the advantage of the proposed method over Barlow Twins / SimCLR is due to the proposed loss. In fact, leveraging other tricks like momentum encoders is the main reason but is only mentioned in L159 once. Not in the figure, not in the main text, not in the experimental detail.

questions: Do you consider n=2 (Taylor expansion order) as a novel model? If not, then how is the performance of this baseline across all experiments. If yes, how is it different from BarlowTwins?
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a self-supervised learning method dubbed Maximum Entropy Encoding (MEC) which leverages the principle of maximum entropy to learn unbiased representations of an image dataset (experiments done on ImageNet). The authors combine a maximum entropy with the augmentation-invariance objective of contrastive learning, which is justified as a view consistency prior which gives testable information. 
The exact resulting loss has a log determinant, which they approximate using a Taylor series. They show that various orders of this approximation are in fact equivalent to existing self-supervised methods such as SimSiam and Barlow Twins.
Experiments are conducted with pretraining on ImageNet and suitably diverse downstream tasks: linear evaluation on ImageNet, semisupervised classification on ImageNEt, transfer learning on object detection (VOC + COCO) and segmentation (COCO), as well as video tracking (Table 5).
strengths_and_weaknesses: Strengths:
Originality:
The community has been iteratively constructing contrastive or similar self-supervised learning methods for several years now. This paper subsumes several previous works by providing a novel unified view that allows for further adaption and exploration. While the resulting loss code is not markedly different from existing work, the theoretical approach it is derived from and the resulting flexibility/opportunity it yields constitute originality in my eyes.

Quality:
The experiments in this paper are very clean, with straightforward well-accepted experimental settings. Appendix C describes the methodology well without introducing any complicating bells and whistles. The results of the new method are consistently equal or superior to existing objectives all while being very motivated.
I cross-referenced numbers with original works and the authors match (or in some cases exceed) the original citations in table referenced numbers (example: the arxiv version of SimCLR lists 69.3 as its top1 imagenet acc while this work gives up to 70.4). I assume these discrepancies are from improved/additional facets in the training, but would appreciate clarification on this point.
I feel extremely comfortable that I could replicate the experiments of this paper with minimal effort and would see comparable results.

Clarity:
This paper is extremely well-written. I was able to understand all of the concepts on the first read-through. The order of presentation was logical and descriptions were enlightening without being overly wordy. As previously mentioned, I feel comfortable that I could replicate the results quickly, especially given the useful pseudocode in Appendix A.

Significance:
The direct tying of a family of objectives to a very grounded mathematical concept is highly significant in my eyes as it brings the iterative line of research to a convergence point from which more novel lines of research can be taken. There could have been several more iterations of self-supervised learning papers converging to this objective by empirical motivation but this work shortcuts that and opens up exciting possibilities for future work (especially given the prominence of contrastive learning objectives in current vision-language works).

Misc:
- Figure 4 was an extremely useful illustration
- Appendix was extremely informative and helpful
- The ablations in Table 5 (particularly (e)) largely match my intuition which is nice

Weaknesses:
Originality:
This paper consolidates existing lines of exploration and opens the door for further "contrastive" methods and as such the ultimate objective is not technically very different from existing work. Given the importance and significance of the consolidation though, this is not truly a weakness in my opinion.

Quality:
*I do wish that there were at least some pretraining experiments on datasets besides ImageNet (either uncurated web imagery or something like Places365). In my opinion this is the biggest weakness of this paper.*
As stated above, overall I was very impressed with the quality of this work. The only typo that jumped out to me while reading was "variaty" instead of "variety" in L60.

Clarity:
I am very familiar with the contrastive learning so am probably a biased estimator here, but I thought the explanations and experiments were extremely straightforward and non-confusing. I have no complaints about the clarity. My only related suggestion is that a pointer to Appendix H is placed in the main text (see Misc below)

Significance:
As stated in the above originality section, it's difficult for a consolidation paper to have earth-shattering impact as the value largely lies in the tying together of work that necessarily exists. I don't think that this by any means is a net negative; I would rank this paper as one of the more significant ones that I've come across in the last 2-4 weeks of arxiv, but it does cap it below what a Conference Best Paper might look like.


Misc:
- Appendix H (starting from L724) is extremely interesting and I was wishing for a similar section within the main text. While incorporating that many lines is impractical at this stage of the paper; I would suggest that a reference to Appendix H with a preview of what it contains is added to the main text 
questions: From Strengths-Quality: Can you clarify where the increase in metrics from cited sources comes from? Was I looking in the wrong place in the cited works or are there improvements stemming from more modern implementations?

From Weaknesses-Quality:
Are you able to provide any results with pretraining on moderate-to-large scale datasets besides ImageNet?
I'd also be curious if this objective can provide improvements for self-supervised learning on smaller-scale datasets (this latter point is an extra ask, if pressed for space in the rebuttal feel free to ignore)
limitations: Appendices E and F provide a succinct but accurate and precise description of limitations and the possible relation to societal impact
review_title: null

Meta-review: metareview: The paper in general received three positive feedbacks and ratings. The three reviewers all recognize the theoretical soundness of the paper, and the paper is also clearly presented with informative and strong experimental results. There is a few places making one reviewer less comfortable in terms of the exact effectiveness of the proposed theory. While overall the experimental results are comprehensive and basically suppor the claiming points made by the paper. The authors may furher clarify the points based on the comments.   
confidence: Certain
Decision: Accept