Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In regards to the contrastive learning framework, the paper claims that the use of mini-batches of negative samples instead of the computationally infeasible option of using the entire dataset leads to bias in the parameter gradients w.r.t. the loss function. The primarily arises due to the fact that the loss terms corresponding to the positive and negative pairs are coupled, which prevents the expected loss of a mini-batch to be equal to the actual expected loss. This in turn results in sub-optimal final parameters and performance. To address this issue, the paper proposes a Bayesian data augmentation approach that effectively decouples the loss terms arising from the positive and negative pairs, which makes the expected mini-batch gradient to be equal to the actual expected loss terms. The paper shows experiments on large scale vision and vision-language tasks.
strengths_and_weaknesses: Strengths:

- Contrastive learning has emerged as a successful mode of self-supervised representation learning. Thus any improvement to this framework will benefit the broader research community.
- The proposed algorithm is novel, and a generalization bound is provided.
- Experiments are conducted on large scale datasets.
- The experiments show improvement of the proposed method over the baseline, under the contrastive objective (Eq 1) considered in the paper.

Weakness:
- There seems to be a major technical flaw in the contrastive objective (Eq. 1), which is used as the baseline (SimCLR) and as a starting point on which the proposed algorithm is built on. See comment 1 below.
- The motivation behind the gradient bias problem, which is the primary subject of the paper, seems weak. See comments 2,3,4 below.
- The experimental results, while they show that the proposed algorithm outperforms SimCLR, may not be reliable because the SimCLR performance in the paper are far below the numbers reported in existing literature for the same experimental setting. The reason for this is related to the previous point regarding Eq. 1. See comments 5,6 below.

If there was something I misunderstood regarding the major problems mentioned above, I'll be happy to reconsider them upon proper clarifications.

questions: 1. Major technical flaw: The similarity scores in the contrastive loss function (like SimCLR) are typically in the exponent, but that is not the case in Eq. 1. This is problematic because similarity scores are typically the output of a cosine operator, which takes values between -1 and +1. Therefore, both the numerator and denominator can now take values between -1 and +1, and thus the combined term itself takes values between -inf and +inf. Applying the log function over such a term will cause numerical issues, and more importantly this is not a proper loss function. This problem creates issues in other places in the paper, which are discussed below.


2. Fig. 1: The toy experiment in Fig. 1 does not demonstrate the gradient bias problem that is introduced in the abstract/introduction, which questions the primary motivation of the paper. There are two problems:
    - Simulating the noise in mini-batches and showing that it causes divergence, does not necessarily show gradient bias, which is typically expected to lead to a sub-optimal solution. Divergence due to noise can happen even in unbiased gradient scenario, for instance, when the learning rate is too high. These details, which are not provided in the paper, are important. A more direct evidence would be when loss for both cases converge, but the biased gradient loss is sub-optimal.
    - There is a technical issue with this experiment. In the contrastive loss, the similarity scores are present in the exponent of e. But it seems the similarity scores described in Eq. 1, and what’s used for this toy experiment do not use the exponent. This is bound to lead to numerical issues.

3. Line 328-330 states: "The reason is that we find the normalization can make the impact of gradient noise less prominent because the representations are constrained to a much more compact sub-space”. This statement undermines the motivation of the paper, which is to mitigate the gradient bias problem in contrastive methods to improve its performance. As stated, if normalizing the latent representation to lie on the unit L2 ball (which is the de facto procedure for all contrastive methods) makes the gradient noise less prominent, then the gradient bias problem does not seem very concerning. In addition and importantly, removing the feature normalization step (i.e. using dot product instead of cosine as similarity) is not a technically sound thing to do, because the contrastive loss function is no longer meaningful.

4. In the opening paragraph of introduction, the authors state the gradient bias problem, which is the main subject of the paper. Following this, they rightfully acknowledge that non-contrastive methods like BYOL, do not need negative samples, which is the source of the gradient bias problem in contrastive methods. However, they dismiss such non-contrastive methods by stating that they are developed from a different perspective. But this does not rule out the possibility that non-contrastive methods cannot perform similarly to contrastive methods (in fact they sometimes report better performance), and may be addressing the gradient bias problem implicitly. Therefore to me, the claims here seems hand-wavy, and a deeper investigation is required to put non-contrastive methods in this context. This is also important to concretely establish that gradient bias is a real problem, meaning, no existing methods (not just SimCLR) solve it.

5. Experiments reporting accuracy: While the accuracy numbers after removing the normalization step are better for the proposed method, the numbers for SimCLR are much worse than what is reported in literature as well as on the GitHub repo page from where the code is used for the experiments in Table 1 in this paper. Even for the normalized feature setting, SimCLR numbers are worse by ~5% for batch size 512 compared to that reported in the GitHub repo (citation 42 in the paper). My guess is that this is because of the use of Eq. 1, which is different from the original SimCLR objective.

6. Experiments reporting accuracy: Once again, the performance of MoCo-v3 is far below the performance reported in their original paper. My guess is that the authors have use the loss function in Eq. (1), which does not put the similarity score in the exponent, which makes the loss function technically problematic.

Minor comments:

- This is more about convention, rather than a technical issue. Loss in Fig 1 converges to 1 for the unbiased case and is typically higher than the biased gradient case. However, typically, a lower value of loss is considered better.


- Line 78: The positive pair is stated as the pair of the original sample and its augmented version. However, a positive pair typically consists of 2 augmentations, and not one original and one augmentation.

- Line 129: Dimension of the matrix U is not specified on line 129.

#### POST REBUTTAL###
I thank the reviewers for their detailed response. Most of my concerns have been addressed. I have changed my score to weak accept. The reason for not providing a stronger score is because I am not fully convinced if the problem that the paper addresses has a very broad impact, since the proposed method is only significantly better compared to the original contrastive methods for certain batch sizes (typically small). Nonetheless, I think this paper proposes an interesting idea that does address an inherent problem in mini-batch based contrastive learning algorithms.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In order to address the gradient bias and sub-optimal solution problems in contrastive learning, this paper proposes a novel Bayesian data augmentation method to decompose contrastive loss with an efficient sEM algorithm. The authors develop a generalization theory and verify their approach with extensive experiments comparing with several contrastive learning baseline methods. The experimental results demonstrate the effectiveness of this approach on single-modal image representation learning and multi-modal vision-language representation learning tasks. 
strengths_and_weaknesses: Strengths:
1. The problem formulation is clear. The authors propose a novel approach to remedy two problems, i.e., gradient bias and sub-optimal solution, with a new contrastive loss applying the Bayesian data augmentation method. This paper points out a new research direction to improve the performance of contrastive learning. 
2. The proposed method only needs minimal modifications on most contrastive learning frameworks, e.g., SimCLR and MoCo-v3.  The authors provide a generalization theory and explain clearly how the proposed loss function can be used in most contrastive learning approaches. The authors show clear proofs for the proposed theorems. 
3. The experimental results (i.e., tables and figures) clearly demonstrate the effectiveness of this approach. Some improvements are clearly observed between the proposed method and the compared baseline methods on most evaluation metrics across all tasks. 
4. The paper is well written and easy to follow. The authors present a very good literature review. The notations, figures, and algorithms are easy to read.

Weaknesses:
1. The authors propose two methods to construct a sequence of rate parameters to approximate $E[s_{ik}-]$ of the Gamma posterior in lines 223-224. They should have provided sufficient analysis here. 
2. The authors provide some analysis and experimental results on the robustness of their method against gradient noise, it is not clear whether the gradient noise is equal to or the same as gradient bias. 
3. There are some minor issues. There is another meaningless red curve shown in Figure 1. To my understanding, $u_i$ in the negative sample part in Eq. (2) should also be boldface,  
questions: 1. In lines 235-236, you mentioned "For optimization convenience and algorithm stability, we marginalize out the auxiliary data $U$ from previous iterations, is it a good solution, or what is the problem behind this? 
2. in lines 328-330, since the normalization can mitigate the gradient noise problem, do you still need another approach to deal with the intrinsic gradient noise? 
3. What is the training time for your proposed methods? Is it much slower than SimCLR in your experiments?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors aimed to address the biased gradient, which is due to the non-decomposability of the loss, in the existing contrastive learning methods. A more generalized and decomposable loss function has been proposed by using Bayesian data augmentation technique. In the proposed loss, a joint distribution over the model parameters and the augmented parameters is defined, and can be optimized by stochastic EM algorithm. Extensive experiments were conducted on several datasets to confirm the efficiency of the proposed method.
strengths_and_weaknesses: Strengths

. A new loss function is proposed to address the gradient bias with theoretical analysis.

. Extensive experiments have been shown for the comparison between the proposed algorithm and state-of-the-art, eg., SimCLR and MOCO-V3.

#############

Weaknesses

The authors tried to address the gradient bias in contrastive learning. However, it's not convincing that the gradient bias is a big issue in existing contrastive learning methods. Especially usually a large number of negative samples are used along with some training tricks such as normalization. Such concern can also be confirmed from the experimental results. That is, generally, the proposed method has more advantage when the batch size is small. See Questions below for more detailed comments. 
questions: . Lines 94-118: The authors identified the gradient bias in contrastive learning as the current loss is non-decomposable. As mentioned later, this problem could be mitigated by using a large batch size. In Fig. 1, a motivation example is presented to show the consequence of the gradient bias. However, this example is not convincing as only one negative sample is used. In practice commonly hundreds or thousands negative samples are used along with some easy-to-implement training tricks, eg, normalization to mitigate the gradient bias. Overall, the gradient bias seems not to be a big issue in practical contrastive learning methods.

. What's the connection between equations (1) and (3), ie., the existing and proposed loss function? Are they equivalent in terms of optimality?

. Lines 180-194: From the analysis, the decomposibility is derived under the assumption of infinite negative samples. In practice, as there are only finite negative samples does the decomposibility still hold?

. In Algorithm 1, how to properly design the sequence of {\lambda_t} for convergence?

. Theorem 3: The generalization gap is shown to be upper bounded by two terms. Any comments on these two terms?

. From the provided experimental results, generally, the proposed method has more advantage when the batch size is small. When the batch size is large and the normalization is used the advantage is limited (eg, in Table 1: CIFAR-10/batch size 512, Table 2: batch size 512, Table 3: MoCo-v3-512 vs Ours-512). This again raises the question that, whether the gradient bias is a big issue for practical well-designed contrastive learning methods? The authors can provide comments on this for a better summary of comparison. 
limitations: Please see the Questions above for the detailed comments. 

No potential negative societal impact related.
review_title: null

Meta-review: metareview: This paper argues that using mini-batch updates for contrastive learning leads to a gradient bias problem. Authors take a probabilistic view point and propose an efficient Bayesian data augmentation technique which leads to decomposing the loss function. They further come up with an efficient sEM algorithm for optimizing the the new objective. Finally, their large-scale experiments indicate that their proposed method improves over existing contrastive learning techniques. 

Reviewers are in agreement that the ideas presented in the paper are novel. Furthermore, the presentation and arguments provided in the paper are reasonable. The main concern is that based on the provided empirical results, the gap between the proposed method and other techniques such as SimCLR shrinks significantly as the batch-size increases. That being said, the interesting discussions about gradient bias, the novelty and the improvement over SimCLR in small-batch setting is more than enough reason for accepting this paper for publication at NeurIPS.
confidence: Certain
Decision: Accept