Title: Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes
Venue: NeurIPS 2025
Accepted: yes
Keywords: diversity

contrastive learning

batch selection
Abstract: Contrastive learning thrives—or fails—based on how we construct \emph{positive} and \emph{negative} pairs. In the absence of explicit labels, models must infer semantic structure from these proxy signals. Early work on Siamese networks \citep{chopra2005learning,hadsell2006dimensionality} already showed that pair construction directly shapes learned representations. In modern contrastive frameworks, poor pair selection remains a primary failure mode: it either causes collapse, where all embeddings converge to a point, or wastes the representational capacity of the space \citep{chen2020simple,tian2020makes,khosla2020supervised}.

Contemporary methods typically generate positives via semantic-preserving augmentations (crop, jitter, view transform), while negatives are drawn from other elements in the mini-batch under the assumption that different images are semantically dissimilar. But this assumption breaks down in fine-grained, low-diversity, or high-resolution settings \citep{kalantidis2020hard,robinson2020contrastive,chuang2020debiased}, motivating techniques such as hard-negative mining and debiased losses \citep{bachman2019learning,tian2020makes}.

\paragraph{Beyond pairs: batch-level diversity.} 
While most prior work focuses on \emph{which} individual negatives to select, we study the geometry of the entire batch. Our central observation is this: the overall \emph{diversity} of the batch embedding space strongly governs both training dynamics and representational quality. If diversity is too low, the model sees nearly identical negatives and gradients vanish—leading to collapse. If diversity is too high, negatives become almost orthogonal, but the resulting gradients shrink in magnitude, and learning slows. Optimal training thus occurs within a \emph{moderate diversity window}: high enough to avoid collapse, low enough to preserve update strength.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies how to construct or identify batches for contrastive training which have certain gradient properties. They start by upper and lower bounding gradient norms. Then, they relate these bounds to effective rank, which can be computed over batches. This then gives the authors a way to control the gradient norm, by computing the effective rank of candidate batches and selecting the one which has the desired effective rank. They then show different convergence speeds for three different effective rank targets, and provide a greedy online method for building diverse batches. They show slight performance improvements relative to vanilla training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Using rank as a diversity proxy is both sensible and interesting.\n* Speedups seem strong. \n* I like the empirical verification of the theory. \n\nWeaknesses:\n* The main weakness I see is empirical evaluation. The evaluations are done at quite small scale model (ResNet-18), on CIFAR and ImageNet-100, both rahter small datasets. \n* Related to the above, overall accuracy gains on ImageNet-100 is very small, a fraction of a percent (Table 2 and Table 3) . \n* The writing is scattered, with key results in the appendix (eg. Table 2).\n* Figure 6 seems to demonstrate that training has not converged, which makes any claims about performance pre-mature. \n* Presentation of figures can be improved; for example Figure 2 is blurry, Title of Figure 3 is cut off, legend of Figure 8 is covered (this is minor).\n\nQUESTIONS:\n1. Does this result hold on problems outside of small-scale classification?\n2. Figure 6 does not seem trained to convergence, what happens with longer training?\n3. Self-supervised (contrastive) learning is often used to pre-train models with large-scale datasets before transfer. Does pre-training with batch selection give a meaninful performance boost for transfer?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a spectral framework for analyzing and controlling gradient magnitudes in contrastive learning. Using the matrix Bernstein inequality, the authors derive non-asymptotic upper and lower bounds on the squared InfoNCE gradient norm, relating it to alignment, softmax error, and the top eigenvalue of the batch covariance. Leveraging these theoretical insights, they propose several novel batch selection strategies, notably a fast Greedy Element-Wise Builder that outperforms random sampling on CIFAR-10 and ImageNet-100. They also demonstrate theoretically and empirically that in-batch whitening reduces gradient noise in SimCLR.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written, clearly motivated, and logically structured.\n\n- It offers strong theoretical motivation and justification for the proposed methodological approach, contributing to the understanding of SimCLR’s training dynamics.\n\n- Interesting maths, that theoretically justifies the limitations that users observe when training SimCLR.\n\n- The authors conduct well-designed synthetic and experimental simulations that convincingly validate their theoretical arguments in a way that is clear and interpretable.\n\nWeaknesses:\n- SimCLR is nowadays underused in Self-Supervised Learning and is not really SOTA anymore, so the benefits of this paper appears quite limited.\n\n- In line 94, the authors note that high alignment between anchor and positive leads to gradient vanishing and potential collapse. However, no related works are cited to support this claim, and SimCLR is generally considered collapse-proof compared to non-contrastive methods like BYOL or SimSiam. Stronger arguments and empirical validation of this specific phenomenon would strengthen the paper.\n\n- The experimental setups do not include datasets with many latent classes. Related work RankMe (Garrido et al., ICML 2023), which is not discussed in this paper, demonstrates in Proposition 5.1 that classification accuracy plateaus depending on the rank-to-number-of-classes ratio. Incorporating this discussion and conducting experiments with a higher number of classes would enhance the analysis. Specifically, examining the relationship between effective rank (or batch diversity) and downstream classification accuracy, as in RankMe, would be valuable.\n\n- While the authors assess the real-data spectral band on ImageNet-1k, they do not report speed or accuracy gains on this dataset. Including these results would strengthen claims of general utility.\n\n- The paper does not compare with MoCo v2+, which uses a queue mechanism to encourage batch diversity. A comparison with MoCo v2+, as well as a discussion or analysis of gradient magnitudes when a large portion of the batch has its gradient detached (as in queue-based contrastive learning), would situate the proposed method better within the literature.\n\n- It would be helpful to analyze whether the benefits of the greedy approach persist at larger batch sizes (>4096), as generally done in the literature.\n\n- The paper focuses on ResNet architectures; analysis on different architectures (e.g., Vision Transformers) would enhance generality claims.\n\nQUESTIONS:\nCould the authors provide empirical validation or stronger theoretical arguments for the claim that high alignment leads to collapse in SimCLR?\n\nHow does the proposed spectral selection perform on datasets with a large number of classes, and how does it relate to findings in RankMe regarding rank–accuracy relationships?\n\nWhy were speed and accuracy gains not evaluated on ImageNet-1k, despite the spectral band validation conducted there?\n\nHow does the proposed method compare with MoCo v2+, and what is its effect when the majority of the batch gradients are detached, as in queue-based contrastive learning?\n\nDoes the efficacy of spectral batch selection persist for very large batch sizes (e.g. 8192 or higher), and is there a diminishing return beyond certain sizes?\n\nHave the authors evaluated the method on different backbone architectures, such as Vision Transformers, and if not, do they anticipate any architectural dependencies?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a spectral-band framework to highlight the important role of batch diversity in contrastive learning. Based on their framework, they put forward batch selection strategies to enhance contrastive learning. Experiments on synthetic datasets, ImageNet-100, and CIFAR-10 demonstrate the effectiveness of their method.

STRENGTHS AND WEAKNESSES:
Strengths
1. The theoretical proof is solid and intuitively understandable.
2. This paper is readable.
3. Extensive experimental analyses demonstrate the effectiveness of their method.

Weakness
One of my major concerns is the issue of time overhead, because it is necessary to extract embeddings of multiple batches through the model when selecting the batch in each iteration. Could you provide the training time for each method based on the settings in Figure 8?

QUESTIONS:
1. In the experiments in Section 3, how are the augmentation parameters set? And do these augmentation parameters affect the effectiveness of your proposed strategy?
2. Would it be better to use a smaller network or embeddings from a certain intermediate layer of the model for batch selection? This would save more time, and the performance might not be significantly worse. 
3. What is the validation accuracy of each strategy in Section 3.5?
4. Why do the previous experiments use ImageNet-1k or ImageNet-100, but Section 3.6 switches to CIFAR-10?
5. The experiments in Appendix B are also of great importance, and I believe readers should be reminded in the main text to refer to Appendix B.

LIMITATIONS:
None.
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical and practical contribution to the understanding of contrastive learning. The authors introduce a spectral band framework that establishes sharp, non-asymptotic upper and lower bounds on the squared gradient norm of the InfoNCE loss.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe core of this paper is a novel and rigorous theoretical framework. Deriving explicit, non-asymptotic bounds on the InfoNCE gradient norm provides a formal understanding of phenomena (like collapse and gradient vanishing) that are typically understood through empirical observation. The decomposition of the gradient into alignment, variance, and spectral terms is particularly insightful.\n\nThe authors' claims are supported by a strong set of experiments. They first validate their theoretical bounds on synthetic data. They then demonstrate that these bounds are tight and predictive during real-world training of SimCLR on ImageNet. The evaluations of the proposed batch selection methods on both SimCLR and MoCo v2, and against strong baselines are good.\n\nThe proposed Greedy Element Wise Builder is good. It provides a lightweight method to construct spectrally diverse batches without the high computational cost of evaluating full-batch spectra, achieving comparable performance to the more expensive pool-based policy while offering significant runtime savings.\n\nWeaknesses:\n\nWhile the Greedy builder is efficient, I think it has the drawback that the full pool-based policies (P1, P3) require maintaining and evaluating a large candidate pool of batches (e.g., k=5120) at each step. This can introduce significant computational overhead. The paper demonstrates accelerated convergence in terms of epochs but is less clear about the net wall-clock time improvement when using these specific policies, which could be an important practical consideration.\n\nQUESTIONS:\nRegarding the pool-based policies, could you comment on the wall-clock time trade-off between the cost of evaluating the candidate pool and the reduction in training epochs? Is there a point where the selection overhead for P3 might negate the wall-clock time savings from faster convergence?\n\nYou show that your framework applies well to MoCo v2. How do you anticipate the dynamics changing with even larger negative queues or memory banks?\n\nThe performance of the Greedy builder is dependent on the probe size m, with m=16 showing degraded performance. Is there a theoretical principle or heuristic to guide the choice of m relative to the desired batch size n or embedding dimension d?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}