Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an improved loss based on the infoNCE loss and show that SimCLR and MoCo with the proposed loss converge faster and require smaller batch sizes / memory banks.
main_review: My main concern about the paper is that given there are so many papers discussing the losses for contrastive learning, the authors did not cite any of them [1-3]. Some of these papers also focus on batch size sensitivity. I would recommend authors do a more comprehensive literature search and compare the proposed loss with these previous losses.

The thermotical analysis provides interesting insights but the authors seem not to claim it in the right way. Based on Figure 1 and my understanding, small batch sizes lead to a smaller chance to utilize hard negatives in each iteration, and consequently a simpler problem to optimize, i.e. $q_B \rightarrow 0$. Removing $q_B$ simply increases the signals provided by the simple negatives and positives. Instead, the authors claim the main issue to be a negative-positive coupling problem, which is not clear to me why that impacts the training.

The authors do not provide the code. Although it is simple to reproduce and the authors mention that this loss is insensitive to the hyperparameters, they do not provide any evidence to support this other than the claims in 4.4.

The authors state in the checklist that "they describe the limitations of the work in supplementary material", but they don't.

The empirical results of this paper are strong and the theoretical analysis is interesting. However, the paper lacks several relevant references. And there is evident space for the authors to improve in both experiments and writing. Therefore, I'm inclined to reject the paper at the current stage.

Minor issues:

1. line 14: Moco --> MoCo

2. Figure 1: $q_b$ --> $q_B$

3. line 211: rely --> relies

4. line 271-272: please delete the official template.


[1] Tsai, Yao-Hung Hubert, et al. "Self-supervised Representation Learning with Relative Predictive Coding." ICLR 2021.
[2] Hjelm, R. Devon, et al. "Learning deep representations by mutual information estimation and maximization." ICLR 2019.
[3] Ozair, Sherjil, et al. "Wasserstein Dependency Measure for Representation Learning." NeurIPS 2019.

limitations_and_societal_impact: The authors claim the limitations to be addressed in supplementary material, but I didn't find any. The potential negative societal impact is not discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper observes that in contrastive leanring the gradient contribution from positive pairs and negative pairs is scaled by a common multiplier that tends to be small in the small negative batch size N regime, and close to 1 for large N. The authors propose a minor modification to the InfoNCE loss function - removing the term on the denominator pertaining to the positive pair - that suffices to remove the multiplier (i.e. set it equal to 1 always). Although the change is minor, the authors observe an improvement in robustness to batch size in SimCLR (and queue size in MoCo) and find that fewer epochs of training are required to attain a certain level of performance (though in the large epoch regime the performance saturates at a similar optimal performance level as the vanilla InfoNCE loss). 
main_review: 
This paper has some merits. The motivation of reducing the coupling of gradient information from negative and positive samples is reasonable, the method sensible and very simple to implement, and the empirical results are encouraging. 

However, despite those merits, I also have concerns that make me unwilling to recommend acceptance.  The objective derived is similar in form to the alignment and uniformity loss [1]. In [1] the order of the expectation and exponential is swapped, but the *key feature* of removing the positive term in the denominator is there. If you computed the gradients of the alignment and uniformity loss you would also find that it avoids the cancellation effect that this work posits as its main goal to remove.  It is *critically important* in my mind that [1] is discussed in the related work, and added as a baseline in experiments (especially for Fig. 3 on sensitivity to batch size and Fig. 4 on convergence speed). I will not feel confident enough to recommend acceptance without that comparison. 

I strongly suspect that the objective of [1], if tuned correctly, would yield similar results to those in this paper (if I have missed some fundamental difference between the objective given here and that in [1] then please point it out - I will listen carefully and update my review on this point if needed). As a result of this, I would argue that the main contribution of this paper is not the proposed decoupled loss itself, but the observation that the loss can be motivated from the point of view of removing this gradient multiplier. While I think this is an interesting observation, on its own I do not feel it is sufficient contribution by the standards of NeurIPS. 

---

## Minor criticisms:

- Somewhat slapdash reporting of results - e.g. Figure 4 doesn’t even tell the reader which dataset corresponds to which figure. Figure 1(a) talks about something called the “coefficient of variation” - I still don’t know what this is since I could not find an explanation anywhere in the text. Even if the explanation is hidden away somewhere, it shouldn't bee so hard for a reader to find it (do you just mean variance/standard dev.?). Please do a review of the pedagogical basics and make sure all figures are properly labeled and explained. 
- In Figure 3 why use MoCo-v1 when you can just as easily run MoCo-v2 experiments and obtain more competitive baseline comparisons?
- Figure 3(b) has little practical consequence - in MoCo a larger memory bank is very cheap and generally no one feels a great need to shrink the size of the memory bank. Fig 3(a) on SimCLR is more interesting since batch size is a genuine constraint/consideration when using SimCLR.
- Table 2: baseline results are somewhat weak. SimCLR/MoCo-v2 results for CIFAR10 & STL10 results can, with a little care, be in the 90s (here they are generally in the low 80s) and CIFAR100 results in the high 60s or 70s (here they are in the 50s). This is not a major issue, and isn’t a deciding factor by any means, but stronger baselines are always preferable. 
- The reweighed loss version L_DCW is not much of an innovation. Previous work [2,3] apply almost identical re-weighting schemes (to positive and negative pairs). There is no mention of this prior work in the text. 

---

## Clarity: 

This section does not form a component of my final assessment of this work as a research contribution, but is intended to help the authors with improving clarity.
- Clarity is fairly poor in general. Ideas that should be easy and intuitive are written in a convoluted way and end up harder than necessary to understand. I would especially suggest revising section 3, which contains the main idea of the paper.
- l.92 “Simsam” —> SimSiam
- Fig1 is on page 2, but relates to formulae & discussions on Page 4. I would suggest moving the figure to a more logical position. 
- prop 2 is stated in a slightly odd (inverted) way. I would suggest a more linear sequence of logic: 1) define the new loss, 2) state that the new loss has gradients that are the same as the original loss, but with the “NPC multipliers” removed. 

---

**Possible directions for improvement:**

(This component does not factor into the assessment, but attempts to give constructive suggestions on possible directions)

The main goal of this work - making contrastive learning more efficient in terms of both training time and memory requirements - is an interesting one that many researchers have thought about. While perhaps the multiplier may play a role in this, there still remains the fundamental issue that the “alignment” loss and the “uniformity” are trying to enforce *very different* goals. The large epoch requirement for contrastive learning is (in my opinion) likely due to the conflict between these two learning objectives. Its would be very interesting to see an extension of this work to not just discuss scalar multipliers in front of gradient terms, but also discuss conflicting gradient *directions* (the alignment terms gradient likely points in a conflicting direction from the uniformity terms gradient). 

----

**Update 23rd Aug:**

Score moved from 5 to 6 based on information provided in the rebuttal.

---

**References:**

[1] Understanding contrastive representation learning through alignment and uniformity on the hypersphere, Wang and Isola 

[2] Contrastive learning with hard negative samples, Robinson et al.

[3] Contrastive Attraction and Contrastive Repulsion for Representation Learning, Zheng et al.

limitations_and_societal_impact: The authors do not discuss limitations or possible societal impact. Since the work is on fundamental methodology I am not hugely worried about the societal impact part. I would, however, appreciate a discussion of the limitations (especially since in the checklist they claim that they do include such a discussion in the appendix, for me only to find that there wasn't one). 

The main limitation I would put forward is one discussed above - that even with the proposed cancellation in the loss, there still remains a fundamental tension between the alignment and uniformity terms in all contrastive losses, and that this may well be the main factor driving the large training time requirements of contrastive learning. Note: I am not trying to say that the authors have to resolve this limitation, but the purpose of the limitations discussion is to give the reader a more honest idea of where the future directions for further improvement may lie.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study the problem of contrastive learning, and in particular consider the NT-Xent loss function used in popular contrastive learning methods, such as the SimCLR algorithm. Such algorithms typically require long training times and very large batch sizes to be effective, and the authors explore this phenomenon in more detail.

Concretely, they identify a term in the gradient of this loss that has a potentially undesirable effect (the negative-positive coupling term), supported through empirical analysis. They adjust the contrastive loss function to prevent this term appearing in the resulting gradient, with the goal of improving learning speed and robustness to small batches. 

In experiments on common SSL datasets including CIFAR10/100, STL10, and ImageNet, they find that by removing this coupling term, their method can improve significantly on competitive SimCLR and MoCo baselines, at much smaller batch sizes. Furthermore, they find that their method trains much faster with this adjustment to the loss.

Drawbacks: The authors do not appear to include a discussion of limitations in the paper or in the supplementary material. Furthermore, some of their baselines (in particular SimCLR) seems to be  weaker than other reported numbers in the literature, so this is worth examining in more detail. 
main_review: # Originality 
The paper contributes a nice mathematical argument as to what in the gradient of the  NT-Xent loss function might cause noisier learning (worse performance at small batch sizes and long training times). To the best of my knowledge, this is an original contribution, as is the specific adjustment to the loss function. 

# Quality 
The paper on the whole does a good job of identifying an important problem, hypothesising the cause (coupling term), providing some mathematical analysis and suggesting a solution, and then running empirical studies to verify their claims. 

The experiments are fairly thorough, and consider important benchmark datasets used in the literature (both small and large scale).  Applying the method to both SimCLR and MoCo is a good idea, and it is  interesting to observe benefits in both settings. The improvement in learning speed is a very nice benefit of the method, and this was good to see analysed.

My main concern is that the baselines are weaker than numbers that I have seen reported in the literature. For example, consider this open source, popular implementation of SimCLR on CIFAR10: https://github.com/leftthomas/SimCLR. The reported result with kNN eval, batch size 512, and 500 epochs of training is 89.1%, which is significantly higher than numbers reported in the paper (best comparison: 81.3% in Table 1. This is 200 epochs, not 500, but I am not sure if we should expect an ~8% improvement with extra training alone).

Although the results are impressive, I would like to see the proposed method improve on a stronger SimCLR baseline, rather than one that seems to be fairly weak, to be more confident that the method truly is effective.

Relatedly, I am interested to understand why the performance of the proposed method appears to worsen with increased batch sizes in some situations. Is this just variation across seeds, or something more fundamental?  

It would also help if the authors provided confidence intervals/measure of spread for the numbers reported. They mention running 5 repeats, and a measure of spread would be helpful (especially in understanding things like the above).

If computationally feasible, I am interested in the authors running an experiment with DCL with a large batch size (\~1024) and large number of training epochs (\~500) to understand if the performance degrades at these scales. It is good to understand if that might be a limitation of the method. I understand it is not a central focus of the paper, but it would be good to address this.

Furthermore, the authors mention they ran ablations in section 4.4. Can they provide more information about these? Was it just the reduction in learning epochs?

# Clarity
The paper is well written and laid out, and was very easy to read.

# Significance
Potentially high, but some outstanding questions regarding weak baselines, scalability to large batches and training times. 

# Summary
Overall, I think this is a good contribution, but the authors do not discuss limitations of the work and the quality of the baselines is somewhat uncertain. If the authors can address these points, I will consider acceptance. For now, I think this is slightly below acceptance.

# Update post author response
Thank you for responding to my questions and providing the additional results. I am more encouraged by the method's performance now, and appreciate the discussion on limitations. It would be good for the authors to add all the relevant extra discussion into the final version of the paper (better comparisons to related work and additional experimental results). Given this and also the responses to the other reviewer comments, I am increasing my score accordingly.
limitations_and_societal_impact: See above, but the authors do not mention limitations in their paper. This should be addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper gives a theoretical analysis of the contrastive loss to identify a "negative-positive coupling multiplier" term. By removing this term from the loss function, the authors achieved better performance over SimCLR and MOCO, particularly when using small batch sizes. 
main_review: **Pros:**

1. The authors identified a common multiplier term in the gradients of the well known contrastive loss, and observed that this multiplier (NPC) behaved differently for small vs large batch sizes. Since NPC fluctuates wildly for small batches, but converges to 1 for large batches, they propose to set NPC to 1 during training. The technique is very simple and does not require further tricks.

2. There is also an improved version that uses a weighting between positive and negative terms in the loss function. By choosing a weighting function cleverly they avoid having more hyperparameters to tune. (The temperature parameter needed to be re-tuned though.)

3. For small batch sizes, the performance and training efficiency on various datasets exceed SimCLR baselines. This is nice because these are two known disadvantages of contrastive learning (bad at small batches, slow training).

4. Writing is quite clear.

**Cons:**

I have some concerns with regards to the significance of the results. Contrastive learning is known to not do well at small batch sizes. Follow-up methods such as BYOL have achieved much better results at small batch sizes. While it is very nice that this paper has improved over SimCLR and MOCO at small batches, it still does not match the later SSL methods, and practical applications could be limited. The benefits may also disappear when models and training are scaled up.

**Nit:**

Line 127 - “Removing the positive pair from the denominator of Equation 2” - I think this should be Equation 1?

limitations_and_societal_impact: This is not applicable to this paper.
review_title: null

Meta-review: This paper presents an analysis of contrastive learning with the InfoNCE objective that identifies a strong dependence of the gradient on batch size, and proposes a new objective to remove this batch-size dependence by eliminating the positive pair from the denominator of the InfoNCE loss. This decouples the influence of the positive and negative pairs on the loss function (L_DC) in an additive manner, which is not true for the original InfoNCE loss. Through extensive experiments (in the original paper as well as rebuttal), the effectiveness over InfoNCE at small batch size is verified. 

Reveiewers raised several questions around the experimental setting and comparison to additional approaches (Barlow Twins, SimSiam, BYOL) that were largely addressed in the rebuttal. However, there remained concerns around the impact of yet another contrastive loss function given the success of non-contrastive losses like BYOL at small batch sizes. While the theoretical analysis and results in Fig 1 highlight the batch-size dependence of the gradient scale, there is not discussion of why removing this batch-size dependent impact should yield improved representation learning abilities, or how the new objective may be related to other proposed losses (but some of these concerns have been nicely addressed in the rebuttal, e.g. relation to alignment + uniformity losses). 

[[Addressing these concerns and incorporating more results from the rebuttal into the main paper would greatly improve the paper, but I cannot recommend it for acceptance in the current state.]]
Decision: Reject