Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper makes two contributions to contrastive representation learning: 1. Using the more general f-mutual information rather than using Shannon mutual information for contrastive learning 2. Experimental results to compare the possible options given the new design freedom. 
main_review: This paper addresses an interesting problem, and demonstrates useful empirical results. While the paper has promise, I cannot recommend acceptance of the current state of the paper due to sometimes incoherent results, unclear statement of contributions, and some unjustified assumptions. Of course, I am happy to change my score if I mis-understood any parts of the paper or if the issues are improved in the rebuttal period. 

Theoretical: I find the assumptions quite odd. If the paper already assumes that the true distribution over features is a Gaussian, then naturally Gaussian kernels are the right choice. Therefore, I do not really see why the theorem provides interesting insight. Such a strong assumption might require experimental / theoretical justification, which is not provided in the paper. 

In section 4 the paper bounds the estimation error of the mutual information. However, I do not see how this is relevant to the goal of representation learning. If accurate estimation of the lower bound matters for representation learning, I think the paper needs to demonstrate this either theoretically or empirically. 

These are some claims in the intro that in my opinion are not adequately supported in the paper, and hence might benefit from being downplayed. One seeming contradiction is that the intro first claims that a tighter bound on the MI does not improve the representations, which is one of the motivations for this work. However, much of the theoretical results focus on how to obtain tighter bounds on the f-MI. This can be confusing as the paper did not justify why tighter bound to Shannon MI leads to worse representation learning performance, but it is desirable to obtain tighter bounds to the f-MI (which after all includes Shannon MI as a special case)? Also, the introduction claims that the newly proposed similarity function is ‘optimal for practical use’. However, I do not think such claims are adequately supported by theoretical or empirical results. 

Empirical: Given that the theoretical results are not fully convincing, the experiments become crucial. The paper would be a great contribution if it can convincingly demonstrate the empirical advantage of Gaussian similarity compared to cosine similarity, and the benefit of using another f-MI instead of Shannon MI. The experiments provide some evidence that some choices of f-MI (JS) is better on CIFAR, STL and TinyImagnet compared to baselines such as SimCLR. In addition, the experiments provide some evidence that using Gaussian similarity is better than using cosine similarity. 

The experiments have some short-comings. First, there is no experiment on the original setup of e.g. SimCLR which was applied to Imagenet. Second, on the STS dataset, compared to the cited paper, the reported baseline performance seems worse? While this is not a fatal issue, maybe the original paper used additional techniques that were difficult to reproduce, it is certainly something that needs to be discussed. Finally, there is not insufficient statistical rigor such as error bars, as some of the improvements are small enough to be possible to come from random fluctuations. 

That being said, I do think that the paper can become much stronger with a clearer statement of the actual contribution. In my opinion, the most interesting contribution would be proposing that f-MI lead to a new class of new objectives, that while not necessarily better than Shannon-MI, at least provides more options. In addition, experiments can be used to study which options are the best empirically. 

--------------

Post rebuttal: Thank you for the detailed rebuttal. After the rebuttal, I feel better about the experiments. There is a lot of subtlety with representation learning evaluation, so I believe the paper should include all the details of evaluation, including justification on why e.g. SBERT-base is chosen as a baseline (which is fine as the baseline does not necessarily have to be state-of-the-art with all the bells and whistles, but it should be clearly discussed and disclosed since the original paper contains methods with significantly better performance). The same goes for the newly added Imagenet experiments.

In addition, while the authors understandably had insufficient time to revise the paper during the rebuttal period to address the writing issues, I believe the claims of the paper should be more precise and well-supported by the actual experiments. A mediocre contribution that is well supported is better science and probably contributes to research more than amazing contributions that are weakly supported or questionable. 

Given the detailed rebuttal and the new experiments, I will raise my score. 
summary_of_the_review: The paper proposes an interesting family of new contrastive learning objectives, and some empirical comparison of the objectives. However, I think the paper needs a more clear and accurate statement of the contribution, better justified theoretical results, and better empirical evaluations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the f-mutual information objective for contrastive learning, which generalizes the existing mutual information framework. Theoretically, it makes a connection with alignment and uniformity in Wang & Isola (2020) when the joint feature distribution can be represented by a radial basis function, and derives its estimation error. Numerically, it demonstrates the efficacy of f-mutual information with Gaussian similarity.
main_review: Strength: 

The theoretical results show that the generalization from mutual information to f-mutual information is not merely a mathematical game but also has its own merits. Specifically:

1. The natural connection with alignment and uniformity is a neat observation. 

2. The finite-sample result on uniformity provides a counterpart of the existing results in the infinite-sample case, which is interesting because it demonstrates that not all f-divergences are equal.

3. The result of the estimation error is able to handle the dependent negative pairs, using a non-trivial application of Rademacher complexity theory.

Weakness:

1. The results in Table 9 in the Appendix seem not to demonstrate a clear advantage of the new framework compared with existing baselines. As such, I am worried if the results in the main body, such as  Table 2, are convincing enough to illustrate the advantage of the new framework.

2. For f-divergences that satisfy the condition (12) in Theorem 5, can you provide some guidelines on how to choose between them?



summary_of_the_review: The paper is well-motivated and clearly written. I enjoy reading the paper, and I believe the results are technically sound as I do not find major flaws in the proof. Although mathematically the new framework simply replaces mutual information with f-mutual information, I think it produces several interesting observations and draws connections with the literature. Due to computational efficiency, the authors are not able to provide confidence intervals of the objective in their numerical experiments, which raises the concerns whether the new framework is indeed better in some applications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes another variant of Contrastive Learning that uses a different lower bound of the mutual information based on f-divergence instead of InfoNCE.
main_review: Pros:
1) The paper is well written and easy to understand.
2) The experiment is quite comprehensive, and the results look positive.

Cons:
1) The paper has limited novelty compared to the InfoNCE framework.
2) The advantages/disadvantages of the proposed method compared to InfoNCE are not well analyzed in the paper.
3) The standard deviation of results is not provided, results of the baseline do not match those in the original paper.
summary_of_the_review: 1) Novelty:
- The main limitation of the paper is that the proposed idea is not novel. The variational lower bound of mutual information (MI) based on f-divergence was well discussed in previous works (e.g., [1, 2, 3]) so it is straightforward if we replace the InfoNCE bound with the new bound.

2) About the method:
- The main advantage of InfoNCE compared to other lower bounds of MI is its low variance which allows stable learning to achieve good representations. The low variance comes from the fact that InfoNCE uses multiple negative samples per positive sample and normalizes over them. On the downside, InfoNCE is biased and requires a large batch size. SimCLR has this drawback but it was overcome by MoCo. Therefore, I expect the authors to theoretically and empirically analyze the advantages and disadvantages of their proposed method compared to InfoNCE in terms of bias-variance trade-off. They should also compare their method with MoCo, which has been shown to perform better than SimCLR.

3) About experiments:
- The experiment results do not show the clear advantage of the new method compared to the existing baselines based on InfoNCE when using a linear classifier. The difference in performance in Table 3 is small which can be attributed to randomness in running. I think the authors should at least provide the standard deviation of their results to highlight the significance.

- The results in Table 3 in the paper are still very far from the results reported by SimCLR (Table 8 in the SimCLR paper). For example, SimCLR reports 95.3% accuracy on CIFAR10 with a linear classifier but in Table 3, the authors only show 89.7% accuracy. I think the authors should check their experimental settings again to make their results more comparable.

[1] On Variational Bounds of Mutual Information, Poole et al., ICML 2019

[2] Mutual Information Neural Estimation, Beghazi et al., ICML 2018

[3] Learning Deep Representations by Mutual Information Estimation and Maximization, Hjelm et al., ICLR 2019
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to combine contrastive learning with $f$-divergence, which naturally attains the alignment for positive pairs and the uniformity for data representations. The authors provided some theoretical results on choosing the correct function f and the upper bound of the estimation error. Finally, the authors carried out extensive experimental results to validate the effectiveness of the method.
main_review: It is novel to substitute the cosine similarity with $f$-divergence in contrastive learning, and the experiments validate that this is a very effective modification.

Theorem 5 is an important condition to help us choose the correct function which can avoid feature collapse.

The notation in Figure 1 is not easy to understand. I suggest the authors think of another way to express $-f^*f'$.

The description of the algorithm is not well written, according to Figure1, there is aug1, aug2, and aug. I assume aug is equal to either aug1 or aug2. And how to randomly sample augmentation functions t1 and t2?
summary_of_the_review: Theory 5 provides a way to tell whether an f divergence could be a good candidate, which attracts me a lot. But at the same time, the description of the main algorithm is not complete, some details are missing and the notations in the figure is not well-explained.
review_title: null

Meta-review: Most of the reviewers have concerns that the experimental results don’t show stronger enough improvements over baselines and that the theoretical contribution of the paper is not completely clear. These concerns make the paper a borderline paper for NeurIPS. Some reviewers have pointed out problematic or unsupported claims in the paper. With these in mind, I encourage the authors to revise the paper with more clarity and address the reviewers' comments on the exposition of the paper.
Decision: Reject