Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes
Authors: Peter Ochieng
Venue: NeurIPS 2025 poster ()
Keywords: diversity
contrastive learning
batch selection
Primary area: general_machine_learning

Abstract: Contrastive learning thrives—or fails—based on how we construct \emph{positive} and \emph{negative} pairs. In the absence of explicit labels, models must infer semantic structure from these proxy signals. Early work on Siamese networks \citep{chopra2005learning,hadsell2006dimensionality} already showed that pair construction directly shapes learned representations. In modern contrastive frameworks, poor pair selection remains a primary failure mode: it either causes collapse, where all embeddings converge to a point, or wastes the representational capacity of the space \citep{chen2020simple,tian2020makes,khosla2020supervised}.

Contemporary methods typically generate positives via semantic-preserving augmentations (crop, jitter, view transform), while negatives are drawn from other elements in the mini-batch under the assumption that different images are semantically dissimilar. But this assumption breaks down in fine-grained, low-diversity, or high-resolution settings \citep{kalantidis2020hard,robinson2020contrastive,chuang2020debiased}, motivating techniques such as hard-negative mining and debiased losses \citep{bachman2019learning,tian2020makes}.

\paragraph{Beyond pairs: batch-level diversity.} 
While most prior work focuses on \emph{which} individual negatives to select, we study the geometry of the entire batch. Our central observation is this: the overall \emph{diversity} of the batch embedding space strongly governs both training dynamics and representational quality. If diversity is too low, the model sees nearly identical negatives and gradients vanish—leading to collapse. If diversity is too high, negatives become almost orthogonal, but the resulting gradients shrink in magnitude, and learning slows. Optimal training thus occurs within a \emph{moderate diversity window}: high enough to avoid collapse, low enough to preserve update strength.

Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes
Peter Ochieng"
Department of Computer Science University of Cambridge
po3040cam ac uk
Abstract
We derive non-asymptotic spectral bands that bound the squared InfoNCE gradient norm via alignment, temperature, and batch spectrum, recovering the 1/r² law and closely tracking patch-mean gradients on synthetic data and ImageNet. Using fective rank Reff as anisotropy proxy. we design spectrum- aware batch selection, including fast greedy builder. On ImageNet- -00. Greedy-64 cuts time-to-67..5 67.5%% top-1 by 15% vs. random (24% vs. Pool-P3) at equal accuracy; CIFAR-10 10 shows similar gains. In-batch whitening promotes isotropy and reduces 50-step gradient variance by 1.37x, matching our theoretical upper bound.
1
Introduction
Contrastive learning is label-efficient paradigm for representation learning: it pulls together positive views of the same instance while pushing apart negatives in the mini-batch, uncovering latent structure without class labels. Yet this tug-of-war delicate: weak positives stall learning, whereas overly similar negatives drive gradients toward zero, even in frameworks that rarely collapse in practice such as SimCLR (Chen al.) (020a) and MoCo (He et al., 20206). Most prior work tackles this the pair level (e.g., hard-negative mining and debiased losses), but overlooks a batch-leva signal: the spectrum the embedding cloud. If that cloud too narrow (high anisotropy; low effective rank), negatives become redundant; if too wide (near-isotropic with uniformly small pairwise similarities), the softmax distribution flattens. Empirically, we find training fastest inside moderate-diversii window-rich enough to avoid collapse, yet narrow enough to preserve directional signal.
We formalise and exploit this observation via three contributions:
1. Gradient-norm spectral band. We derive sharp, non-asymptotic bounds on the squared InfoNCE gradient norm, decomposing contributions from positive alignment, finite-sample variance, and batch anisotropy. The band depends only on in-batch (or queue) second moments, making the analysis-and and the resulting aignnoiiicsagnoosicc to whether negatives are drawn in-batch (SimCLR) or from queue (MoCo).
2. Diversity-aware batch construction. We propose two lightweight samplers that keep training inside the diversity window: (i) pool selector that targets high effective rank Rea: and (ii) streaming Greedy-m builder that adds the sample with the largest spectral-diversity gain. In our measurements, Greedy-m adds S 1% on-GPU overhead, while large host-side pools can incur ~ mssiteration on a V100 (see (B.3).
3. Experiments from 100 to 1000 classes. On ImageNet-100 our samplers reduce wall-clock time to 67.5% top-1 by ~ 15% with no accuracy loss. On ImageNet- Ik, linear-eval accuracy increases with Rer and plateaus once R.i R.//C > 0.9, consistent with the RankMe view that diversity gains saturate near this threshold ($B)
Acceepeed a Poster the 39th Conference on Neural Information Processing Systems (NeurIPS 2025)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1.1 Notation
Let batch size and embedding dimension d. Each embedding 2, gd-1 C R has designated positive 2j+. Write the set of others as S; := {1. .1..... n2} {i), |S = n - 1, and the negatives
as as,,; S (it), W = n - 2. Let 7 = [= zn E Rend collect rows 2 and define
pairwise similarities 8tj = E [-,, with softmax temperature > 0.
Second moments.
Define the batch second moment and the per-anchor "others' / o"ne moments: Σ 1 وド= 2z2]", E. = - Ejes, 2z2]", E == n- EjEN 2j2]] Under
unit norms, tr(E) = -tr(E,) tr(E,) = 1. Exact relations: Σ; n-1 I 2222 Σ
3 L (2Ez)) T + Zzt 2 )) Hence the spectral bounds used later:
Amax [x)= S n Amax (),
1
Amax (2,) 1 12 12-2 Amax (E).
"
We write ā Amax (2); 0i = Amax (i); isotropy).
= Amax (E,) E [1/d, 1] (minimum at
1.2 Bounding VV 4, c.ll2 - Spectral Gradient Band
We show that the per-sample InfoNCE gradient is confined spectral band whose width is governed by alignment, finite-sample noise, batch anisotropy, and temperature.
Setup & assumptions. Let -n- Ejj = N- EjEN, 2j2]_ and o. = >max (E With unit norms and the Löwner order, Σj N- (n5I-2222 -2++2 3 n E = o! S n n=20. where E == E, and ā Amaxe (E). InfoNCE: Li = -logpa with Pij x eaalT, 8ij 2. 2j. TM, - -2+)). Expectations mini-batchhauggmeenttion... Assumptions: (A1) unit norms; (A2) negatives i.i.d. within the batch and independent of z; (no assumption on the positive pair).
& := Part. and M, Ex Pik-E so V.,
the
Theorem 1.1 GradienttNonm Spectral Band). Under (A1)- for softmax-ssmorhhess constant > O.
IIV 0, 2112 - 72
E[(1 -((-P. 191) N
+
3E - Psst )
+
3c 3c - pitt ) o
(1)
74
76
(1 (1-p) p)
E v c.1 2
p: EM,
(2)
72
where a = _0 (or use the formm
Sketch. Decompose 8, == M, 211 A, + B, C. with A, -6221 B; = (1 Psel )zr-
27 N= EjeN 2,. and C, (1 - Pa+ ) (%: [
> where 4i = Pij/ (1 Past ). Then || < 3[ + |2 + | 122), 2 = E[E]] 21 = [[[1 - Part ) by (A2),
and a first-order Taylor of the negatives-only softmax around uniform logits (App. A.3) gives E < B Pat 12, 0. and E c.
( I|2 < '7 Pist )o]a? For the lower band,
||M]] S 1 implies M2- 2 1 (1 (M,, -++ ))) apply Jensen. Full details in App. A.I
Variants. (i) Bounded spectrum: if (3) < maxx replace a. by maxx. Using g S - gives a batch- maach-measurable ceiling. (ii) High probability: if 2j are i.i.d. sub-Gaussian with E
matrix concentration (App. A.2 yields Amax (Σ) S 1 4++0
logd and hence (1) カーク Amax (É):
substitute into
Softmax error e; and the sampling term 1/N set a baseline that decays with batch size. Anisotropy enters via a, at orders - -1 and 7-;; pushing toward isotropy (e.g.. larger Rer, whitening) tightens the ceiling. Higher alignment & shrinks the floor (1 - p)2/T without implying collapse.
Interpretation.

---

1.3 Spectral Batch Selection
The upper band in Thm.
1.1 depends the per-anchor negatives-only spectrum 0. Amax (2,p) with Σ = Nー EjEN, 2jz,, N = 2. Computing o for every anchor is costly, so we use the batch second moment Σ = Σi=1 i=1 242 (trace one under || and its top eigenvalue Amax (E) as proxy. By Löwner order. T
- (n 2422 2, 2/+) n
(6) S n-2 8,
so lowering ā tightens a uniform ceiling on o=
Effective rank. Let Rett == ///rr(E3) = (Ex A2) (eigs A of Σ). Since Σx A = and Σx o, we get 2 Rett: higher Reft (more isotropy) = smaller â and a tighter band.
Policies. Given a candidate pool P. = (2(m)) Jm=1 with ranks R(m): PI (stability) picks arg max, R(m) to minimize à: P2 supdate-magittune
picks arg min R(m) to allow larger steps (higher variance risk); P3 (balanced) picks arg minm [R(m) R. with R, set by running 10th/90th percentiles (App. A.5). In practice: use P1 early (collapse risk), switch to P3 once Rea stabilizes,
and deploy P2 sparingly to escape flats.
1.4 Greedy Element- Spectral Builder
We assemble a batch incrementally to decrease the trace of the squared second moment, thereby increasing Roff 1/ Assume unit-norm rows.
Objective. For partial batch B of size b, let
ΣB
Σ Z'Z',
tB = tr(3))
- = 1o Σ (2,22)2 (B(2) = Dg
Z'EB
Lemma
(App. A.6 gives the one-step update for unit-norm candidate 2:
b2tp
26 (b)
2b (g tB) (1 tB) (b + 1)2
tr())
At = (
(b + 1)2
Thus smaller (z) yields smaller (more negative) At.
Greedy rule (Greedy-m). At each step select
=² arg min q (z), SEC
where C is probe set of size m (from pool or stream). For unit-norm. rows this is equivalent to
maximizing the Frobenius diversity (5 B) = [Eg - zz" T ||: = tB 2q (z) + 1.
Cost. With cached dot products against B, evaluating (=(2) costs Ob)) per candidate; a probe of size m costs ((m)) per step (or (mbd) without caching). We update f via Lemma A.3 in (1)) after each selection. A practical Greedy-m realisation (initial seeds, Gram maintenance, streaming) given Algorithm (App. A.6).
2 Experiments
We evaluate on standard self-supervised benchmarks with SimCLR Chen et al. (2020a)
and MoCo v2 He et al. (2020a). Unless noted, we follow the original hyperparame- uuevarrhiiecuree/aggeennttti
the only change is our spectrum-aware batch selection ($) L..31..4 1.4).
Datasets: ImageNet-100 (main), CIFAR- Oxford Pets (linear probe). Backbones: ResNet-18 (default), ResNet-50 (variant). Batch/Temp: n-256, ==0.2 (SimCLR) unless stated. Metrics: final top-1 and epochs to threshold (ImageNet- 100: 67.5% Significance via paired I-tests (95% Cls).
Our main results use SimCLR; MoCo-v2 ablations and queue robustness are in App. A. Policy
details (P1-P3), the Greedy-m builder, pool sizing, and overheads are in App. A.5 A.6| spectrum metrics and evaluation protocols (c.g., RankMe, Roft) are in App. B

---

2.1 Synthetic Gradient-Bamn Verification
We validate the analytical upper/lower bounds on L. ($1.2 Thm. 1.1 on controlled syn- thetic data, using a per-baich plug-in variant of the band (we replace expectations by measured batch quantities). Draw 24 ~ NO,,, La) and set ==
As
A == UA A :=
TAaiii
diag(Ai, ..,....... Aa), Σソ = 1, with U orthogonal, ||=|| 1. The sample second moment = : 2,2; satisfies tr(E) = 1 exactly. In high dimension (d=1024), the trace-one spectrum empirically concentrates tightly around (A₃) (median deviation 1% in our runs).
To control anchor- alignment, set = Pgen / Pggn 1 where u. is random unit vector orthogonal z; then || = 1 and (zi, 2t) Pgen-
Quantities and band. Compute 8ij = 2/ Zj. softmax weights Pú over {it) U N,
M; Ej Pty and 9i T (M, 2,). We vary
{0.05, 0.1, 0.2, 0.3), A1
{1/d, 0.3, 0.6, 1.0}, Pgen = 0.6 + 0.4 to couple higher anisotropy with tighter positives. For each setting generate 10,000 batches (batch size n-256, dimension d-1024).
For the upper band, we use the regatives-only moment for each anchor
Σ
Σ
2527,
N-=n-2,
Xmax [2 )
JENT
and plug ( into Thm.
1.1 8 spectral terms (per-anchor, no expectation). For the lower band, we p Σ. Pt. and use (1 (1-p) p) /r2 the plug-in lower bound 1$1.)).
compute Pi := (M, 2,t)
Results. As shown in Fig. 1 across all configurations, at least 99.9% of measured l19.1 fall within the predicted band, indicating that the bounds remain tight and predictive under wide variation in temperature, alignment, and spectral concentration.
Synthetic sanity gradients vs. spectral band
band
bund
100
20
20
6.001
03
64
1.0
Idiversity)
Figure Synthetic validation the gradient-norm spectral band. Measured gradients g (blue) vs. plug-in lower (black) and upper (red) bounds. Example shown for r 0.1; containment (2 99.9%) held for all 16 settings.
2.2 Gradient-Norm Scaling with Temperature
With the batch geometry held fixed (all cosines, including fixed anchor-positie cosine), the gradient-norm spectral band (Thm. 1.1 $1.2) predicts a leading 1 1/72 dependence for the squared gradient, up to higher-orde order corrections:
[E( 7
3
E(1 ((1-P==') Pi- )1-))11) + O(T-a) + O(T N
IIIV :, 1
(3)
where the prefactor depends on the softmax weights p through 6; =1 =1=11p - When relative logit margins remain stable across the sweep, this yields the expected 1/7² scaling.

---

Setup. We use the synthetic construction of 22. with batch size n=256. dimension d-1024, and a fixed trace-one batch spectrum with δ Amax (E) = 0.3 (here Σ is the batch second moment). Positives have fixed cosine 0.75 via 2; c2; +-Iivva 22 u with ui 1 2;. We sweep r E (0.04, 0.063, 0.10, 0.15, 0.20} (log (1/7) {1.40, 1.20, 1.00, 0.82, 0.70J), generate 5,000 batches per setting, and compute
%/ A. 2 |1M 2
9, Σ Y (batch mean).
Result. Figure shows Я, versus 1/T on log-log axes. least- squares fit gives slope 2.02 = 0.03 (95% CI; R2 = 0.999), matching the 1/r² prediction in 33. In this sweep, Px+ varies modestly with T, so the prefactor effectively stable. At extremes, deviations are expected: as - 0, P + (ey - 0) and the band shrinks faster than 2/T2; as 7 -> oo, approaches uniform and the 1/7 trend reappears with a different constant.
Temperature sweep: O. scaling with 1/r²
2.8
slope
Theory slope
2.6
Empirical mean + SEM
2.4
2.2
1.8
16-
1.4
0.7
0.8
1.0
1.1 1.2
logio
Figure 2: Gradient scaling with temperature. Log-log plot of the batch-mean squared gradient Yr versus 1/T with spectrum and geometry held fixed. Blue points: mean + s.e.m. over 5,000 runs per T. Orange line: fitted slope; green dashed: 1/7² prediction. Higher-order O(T and O(T 6) terms are negligible over this range.
2.3 Real-Data Band Validation
We test whether the gradient-norm spectral band (Thm. L.1 1.2 predicts gradient magnitudes during
large-scale contrastive training. We train SIMCLR on ImageNet-1k with a ResNet-50 backbone, global batch size n=4096, temperature 7=0.1, for 90 epochs. Every 100 steps record: (i) the batch-mean squared gradient
1
2, C.l.
(ii) the batch anisotropy proxy ā, Amax (E., with E. 1 E, 22, (embeddings are l-- normalized, so ir Ex = 1), (iii) the mean alignment Pr = 1/ (M 2,),, and (iv) the mean squared softmax error 2, = E1 )
Lower/upper bands. From the per-sample lower bound 1| 2 p1,)) obtain, by averaging and Jensen,
(L-))))
(1-p1)2
L.
For the upper band, we use the negatives-only version of Thm. T. at the batch level by replacing expectations with instantaneous batch means, setting N =n 2, and upper- hopperbouundig each per-anchor spectrum via the deterministic batch proxy
6: = Amax Σ
<
n
5

---

(cf. )1.3). This yields a conservative ceiling U,, that includes the - and 7 terms from the theorem; we take the aafmmax-smoothee constant Sam 0.5 (App. A.8.. For visual clarity, we clamp UB, +- max B,, I.B;) and plot an EMA of 7xs bounds remain unsmoothed and use instantaneous batch statistics.
Spectral Band (batch mean)
T 0.1,0.0
Inoide band: 190.0%
101
Speetro hand obarer mear)
upperbound
Lower bound
EMAOO 301
Figure Real-data spectral band on ImageNet- Ik. Batch-mean squared gradient 7/2 (orange, EMA with a=0.10; log scale), lower bound LB, (1 22)3//² (black, dashed), and upper bound U B, from Thm. I.1 (red). The upper bound uses the negatives-only form with =n-2 and the batch-
level spectrum proxy ((²) O, where o, Amax (E,) and Σ, - Σ. E 2,
(so tr E, = 1). 0.1, 4096. Bounds unsmoothed;
Shaded region: theoretical band [LB,, UBB.). Settings: only the orange curve is AMMA-smoothed. We set Cam = 0.5 the T -6 term (results are qualitatively insensitive for Cam E [0, 1).. The 1/N sampling term assumes negatives-only independence; spectral terms are deterministic.
Findings. Across checkpoints, all measured batch- mean gradients lic within the theoretical band [LB, UB]: no point exceeds U,, or falls below LB,, The band tracks 7. closely, and its width shrinks alignment improves and the spectrum becomes more isotropic (decreasing oz), making it useful online diagnostic for instability or collapse.
Independence caveat (and empirical replacement). Negatives-only independence is used only for the sampling term 1/N- With correlated negatives, this term inflates and U B, becomes more conservative; spectral contributions via o, are unaffected. As a robustness check, replacing 1/N by the empirical mean of per-anchor averages yields the same qualitative containment:
2j.0.
N
JEN
2.4 Diversity manipulation via in-batch whitening
We study how pushing the batch spectrum toward isotropy affects gradient variability. Unlike the mean-squared band in Theorem [.I here control the per-sample variance of the squared gradient,
3
7 V C.
Var(w))
o. + Br,
N = 2, (V)
N 1
N-
A(N-,)) .))
where Ej = N + EjeN, 242, and o. = |[Amax (
(or the per-anchor value), and
B, E +
BiG),
Pa--
Both A(N-, ,7) and B, inherit mild - dependence through the softmax weights (via e:)
Derrvvttion App. A.7
Independence is only used identify the I/N- sampling term; also App. A.4

---

Why whitening helps. Let Σn := Σラ 2,2
(trace one under || and & = Amax (E.). Using the deterministic batch proxy
Perfect in-batch whitening pushes 0, 1/d.
Amax (E7) S
n
E2.
n-
the leading term (N),T) in (V shrinks by roughly a factor d (the factor n/(n-2) 2) as 1 for large n)). Thus whitening primarily suppresses anisotropy-driven fluctuations.
Numerical scale (ImageNet-Ik). For 4=4096 40094) and r=0.1,
1
A(NTT)) = 4094 7
7.33.
4094
In runs 0.002, the A Ag, term dominates. Under V full whitening reduces the right-hand side by at most
A + Br A/d + B, 7.33/256 + 0.02
7.33
150,
below the naive d=256 multipiie--onsiisennt consistent with VV) being safe upper bound.
Protocol. Starting epoch 70 of SIMCLR on ImageNet-Iee (n=4096, T-0.1), we alternate 100 raw and 100 whitened batches:
Σ
2,2 Eg E +T, 10-5
E- 24:
1/2
2,
//||
At each step we log: (i) = Amax (Ex). (ii) the batch-mean squared gradient ÿt Σ, 7i. and (iii) the 50-step rolling variance Varso Whitening keeps x 1/d and reduces Var (5) to 0.73x the raw level (raw/whitened 1.37x), comfortably within the conservative ceiling (V). For visual comparison, we overlay the per-sample variance bound scaled by 1/n; this matches the usual
n o 1+ (1 (m-1) p..)
Var
template and is conservative when inter-sample correlations p. 2 0.
Effect whitening on gradient-norm variance (synthetic)
Raw/Whitened 1.37x
Whitened on Rolling
Raw meantoriance ariaacce
Whitened mean variance
Rolling varlance of
0.0018
0.0016
0.0014
0.0012
400
500
600
Training stop
Figure 4: In-batch whitening suppresses gradient noise. Alternttig raw and whitened batches (grey spans) pushes the trace-one spectrum toward isotropy (6, RJ 1/d) and reduces the 50-step rolling variance of the batch-mean squared gradient 0.73x the raw level (raw/whitened 1.37x). Dashed lines: regime averages.
Assumptions and caveats.
Independence. Negatives-only independence is invoked only for the 1/N- factors A(N and B,. With correlated negatives these terms inflate; a fully deterministic variant replaces by the empirical || (looser but valid).
Softmax dependence on т. Both A and B, depend on r via 6; = Pai In our toggles Pa is stable, so the dominant reduction comes from shrinking d..
3. Re-normalization. Because we re-normalize 2, whitening not exactly linear; empirically à lands near 1/d, which suffices for the d-fold suppression heuristic.
7

---

2.5 Spectral Pool-Policy Comparison
We compare the three apectrum-awart aware batch selection policies from 1.3 P1 (stability P2 hypdate-magnitudee and P3 (balanced window)-agand batch sampling. Unless noted, runs use a single V100 GPU, a ResNet-18 encoder with 2-layer projection MI.P, ImageNet-100 100 for 200 epochs, LARS (fixed learning rate), global batch size n=512, temperature T=0.1, and candidate pool [P:| 5P20=5120 (10x the batch). The selector uses only cached embeddings and dot products (no extra forward passes); its wall-clock cost is included in all timings (App. B.3
vanilla SimCLR baseline with uniform
Metrics. Per epoch we log: (i) training loss Lt, summarized by the area under the loss-epoch curve AULC (lower is better; computed over all 200 epochs); (ii) the batch anisotropy proxy â Amax (E) with Σ = Σ z4z; (trace one under unit-norm embeddings); and (ii)) proxy effective rank 1/ō (a lower bound on Reft = 1/ tr($²) since ā 2 1/ Rett) Trace-based Res and RankMe are reported
App. B
Across five seeds on a single V100, all methods reach similar final top-1 on ImageNet-100 (69.0% 0.2; paired t-test vanilla, p>>0.05). Differences in convergence speed and spectral conditioning
P2 achieves the lowest AULC (fastest) but the highest peak anisotropy. P1 yields the lowest anisotropy (safest) but is slowest. P3 is within ~5% of P2's speed while keeping anisotropy between P1 and vanilla. Vanilla converges more slowly than P1-P3; its peak anisotropy lies between P1 and P3 and below P2, indicating spectrum-aware sampling offers finer control of the speed-stability trade-off.
Table 1: Speed-stability trade-off on ImageNet- 100 (5 seeds). 5 = Amax (Σ) is the batch, trace-one top eigenvalue. "Peak" reports the per-run maximum, then averaged over seeds. No run exceeded the empirical collapse margin (0=0.99; Fig. 5)
Policy
AULC | (x10*)
(peak)
Runs ō > 0.99
P1 (stability)
1.38
0.87
0/5
P3 (balanced)
1.30
0.94
0/5
P2 (updatc-mag)
1.24
0.97
0/5
SimCLR (vanilla)
1.46=0.05
0.92+0.03
0/5
13
3.5
Figure 5: Training dynamics on ImageNet- 100 (5 seeds). Top: Training loss (thick = seed mean; faint individual seeds). Middle: Batch anisotropy proxy à; red dashed line marks the 0.99 safety margin. In separate sweeps, exceeding 0.99 reliably preceded collapse (rank/variance spike) within ~33 steps. Bottom: Proxy effective rank 1/6. Spectrum-aware policies accelerate loss reduction: Pl improves conditioning vs. vanilla; P3 balances speed and conditioning; P2 trades conditioning for speed.

---

Summary. P2 is fastest but least stable; PI is safest but slowest; P3 offers the best balance. Vanilla SimCLR underperforms spectrum-ane aware variants in convergence speed and offers less control over anisotropy.
2.6 Greedy vs. Pool-Based Spectral Selection
We test whether the Greedy Element-Wist Builder (Alg 2 can match pool-based spectral selection (P3) while reducing compute. P3 operates on a pool of 5=5,120 unlabeled augmentations per step and proposes λs candidate batches from this pool (e.g., random partitions), scores each candidate by a spectral criterion (balanced window), and selects the best. In contrast, Greedy incrementally assembles batch from small probe set (16, 64, 256), selecting one element at time to maximize spectral diversity (equivalently, minimize tr(=))).
Setup. ResNet-18 with 2-layer projection head (128 - 128), CIFAR-10. 10, InfoNCE (r=0.2), batch size n=512, cosine LR decay over 400 epochs. Each step draws k=5,120 augmentations. We compare RANDOM, P3 (balanced window Rai E [1.05, 1.15)). and Greedy-m for {16, 64, 256}. Single NVIDIA V100; three seeds per method. The selector uses only cached embeddings and dot products (no extra forward passes); its wall-clock cost is included in all timings.
Logged quantities.
ii Top-1 validation accuracy. (ii) Mean training loss. (iii) Batch effective rank
n²
Ret (B)
| 12
tr(EB)
ΣB
Σ
22
B
(unit-norm rows; identity in App. B.1 p.
(iv) Batch anisotropy āB = Amax (EB). We flag collapse
θB 0.99; a secondary flag triggers the mean alignment - (Mi, 2 0.98.
Evaluation metrics. (i) Final accuracy at epoch 400. (ii) Time-to-accurace wall-clock time to reach 90% of P3's final accuracy. (iii) AUAC: area under the accuracy curve over the first 200 epochs (higher is better). (iv) Collapse rate.
Complexity. Per selected element, Greedy-m evaluates m Rayleigh scores using [] inner products: O(m [B] d) naively, or (m [BI) with cached (squared) dot products; building a batch amortizes to O(m with caching. P3's scoring of kú candidate batches is O(k, n) with maintained sufficient statistics (or Okk, n²) if forming Grams explicitly). Empirically this yields up to ~ 25% runtime savings for Greedy-m at comparable accuracy.
Results. As shown in Figs. 6 6 Greedy-64 reaches 2 90% of P3's accuracy faster than Random, and Greedy-256 closely tracks P3 in AUAC and final accuracy while matching its spectral diversity. Greedy-16 underperforms, indicating that too-small probe sets compromise batch diversity. No collapse events were observed.
0.55
0.75
0.70
0.13
Grenly
Grenly
00% Fealact
Spect
Figure 6: Training loss and spectral anisotropy under Greedy-64 and Pool-P3. Greedy-64 attains Pi-like convergence and stability, with slightly lower variance δg.

---

Agproximati Ret ever Traning
1.28
Greedy13
Sarden
1.08
Grindy-G
1.24
Fool
1.22
Figure 7: Validation accuracy and effective rank over training. Greedy closely tracks P3 in accuracy and Reg; 11 fails to maintain diversity.
Real Accuracy and Timee Accuracy Comparison
0.5
0.7
0.4
203
132
523
033
90
Fonvics
Greedy 36
Grendy 64
Graedy-356
Rool P3
Figure 8: Final accuracy and time-to-accuracy. Greedy 64 and Greedy-256 reach 90% of P3's final accuracy faster than both Random and P3, yielding better compute efficiency.
3 Conclusion
We introduced a non-asymptotic spectral framework that tightly bounds the squared InfoNCE gradient in terms of three interpretable factors: alignment (p), temperature (T), and the batch spectrum/anisotropy (captured by !! with Reir as a practical proxy). The theory yields actionable diagnostics: by monitoring the softmax error alignment, temperature, and the batch covariance spectrum. can predict-and intervene on-collapse, instability, and gradient variance. Our bounds are provided in expectation and high- probability forms, with deterministic per-batch ceiling via a. 2== o and validated synthetic settings and large-scale ImageNet.
Beyond analysis, we demonstrate interventions that follow directly from the framework: in-hatch whitening suppresses gradient noise by pushing the spectrum toward isotropy, and spectrum-awes butch selection improves the stabilityccovvergnnce trade-off by shaping Roff (including fast Greedy element-wise builder). Together, these results bridge theory and practice, offering a mathematically grounded and computationally efficient toolkit for contrastive learning. Future work includes extend- ing the spectral band to con-contrastive objectives, LLMs, and sequence models where anisotropy is known bottleneck.
References
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference Learning Representations (ICLR), net/ www.orrdddxxcnneeeeee
2021. URL httptttt///ppeeeriiiie
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758, 1
10

---

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning arXiv preprint 2XX1::2003.042977 2020b.
Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning. In Advances in Neural Information Processing Systems, 2020.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 3015-3024, PMLR, 2021. URL wtwphhttp///// htttps://proceeddnggs.
press/v139/ wwtp://e//l///e./ee.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In Proceedings of the 40th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research. PMLR, 2023.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervieed learning. Advances in Neural Information Processing Systems, 33:21271-21284 2020.
Ben Harwood, Vijay Kumar B. Ghosh, Gustavo Carneiro, Ian D. Reid, and Tom Drummond. Smart mining for deep metric learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020a.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020b.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decor- relation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Yannis Kalantidis, Nicolas Pion, Philippe Weinzaepfel, Rémi Servant, and Patrick Pérez. Hard negative mixing for contrastive learning. In Advances in Neural Information Processing Systems, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, 2020.
Vladimir Koltchinskii and Karim Lounici. Concentration incqualities and moment bounds for sam- ple covariance operators. Bernoulli, 23(1):110-333 2017a. doi: 10.3150/15-BEJ730 URL -iccentratioon-nnequallll.. -nditieee-------eeeeee and.mmeen--d--mmnnntt..o.mm
2ateee2///ssseeeeee 1/ moment -oae-ooooddddoeeee fonnd--for--a www.---sppp---ccaaaaec-e-e-eeeee
10. 315031511555 15- 14450//5--B.730.fF. full
Vladimir Koltchinskii and Karim Lounici. Normal centration of spectral projectors of sample covariance. tics, 45(1):121-157, 2017b. doi: 10.1214/16-ASS1437
approximation and con-
Annals of Statis-
URL https://
Normal 10.
paarmal-apppooximatico http:/iiiinnnnnncnnnnooeeeeee http:://ooooooo..oo.ooooaoaaaaaa
sample-
Joshua Robinson, Ching- Chuang, Ganesh Santhanam, Stefanie Jegelka, and Suvrit Sra. Learning with currieulum-based sample mining for deep metric learning. Proceedings of the European Conference Computer Vision (ECCV), 2020.

---

Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations (ICLR), 2021.
Olivier Roy and Martin Vetterli. The effective rank: measure of effective dimensionality. European Signal Processing Conference (EUSIPCO), 2007.
Joel A. Tropp. An introduction matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1-2):1-230, 2015. doi: 10.1561/2200000048.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In Advances in Neural Information Processing Systems, pp. 10247-10257, 2018.
Roman Vershynin. High- Digge-Dimesiooal Probability: An Introduction with Applications in Data Science. Number 47 in Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, UK, 2018a. ISBN 9781 108415194. doi: 10.1017/978 108231596.
Roman Vershynin. Mlihh-Dimesiioes Probability: An Introduction with Applications Data Science. Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press, 2018b.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Hao Wen, Paul Vicol, Jimmy Ba, and Roger Grosse. Towards understanding the regularization of contrastive loss. In Advances in Neural Information Processing Systems, 2021.
Chao-Yus Wu, Raghavan Manmatha, Alexander J. Smola, and Philipp Krahenbuhl. Sampling matters deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2840-2844442 2017.
Lei Zhao, Shengeai Xie, Yu Gao, and Jinhui Tang. Maintaining discriminative features in batch normalization for deep face recognition. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Roland S. Zimmermann and Andreas Geiger. Contrastive learning inverts the data generating process. In Advances in Neural Information Processing Systems, 2021.
12

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state the goal of bounding InfoNCE gradient norms via a spectral framework, which is then carried the theoretical analysis and validated empirically in Sections 2-4.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section A.5 discusses the assumption of batch-level isotropy and explicitly evaluates the degradation of spectral bounds under increasing anisotropy, showing that the method is robust to mild violations but degrades in extreme cases.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions (A1)-A3) clearly stated, and the full derivation of the bounds appears in Appendix A.1. A proof sketch is also provided in the main paper for clarity.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sections 4. 4.16. provide sufficient details on experimental setups, such as datasets, architectures (ResNet-18, ResNet-50), batch sizes, and hyperparameters (e.g.. temperature values), to allow replication of all key results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: While the experiments use standard datasets (ImageNet, CIFAR-1O), the code not released at submission time to preserve anonymity. Code will be released upon acceptance.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The experimental sections describe data splits, architectures, optimizers, hyperparameters (c.g., learning rates, temperature), and compute budgets, either in the main text or in Appendix A.
Justification:
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
13

---

Justification: All empirical plots include shaded error bands representing #1 standard error over multiple seeds (typically 5), and statistical variations are reported in tables.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sections 4.4 and 4.5 specify compute types (V100, A100 GPUs), training time (c.g., 200 epochs), and approximate runtime overheads (c.g., 18.6
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///aaeep.ccoooo//o//csaas
Answer: [Yes]
The research involves theoretical analysis and standard benchmark datasets, with no ethical risks identified.
Justification:
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While the work is primarily theoretical and does not involve direct deployment or sensitive data, its findings may influence how contrastive learning is used in large-scale training pipelines. A discussion of potential downstream impacts- such as efficiency improvements in sevource-intessiy models or unintended consequences of aggressive batch seluction-would strengthen the paper's ethical reflection. We recommend including a brief' broader impact section in future revisions.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No high-risk assets are released; the work focuses on contrastive learning theory and batch selection.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets (ImageNet, CIFAR-10) and methods (SimCLR, MoCo) used are cited in the references with proper attribution.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new datasets pretrained models are introduced.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
14

---

Justification: No human subjects or crowdsourcing were involved.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The study did not involve human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research?
Answer: [NA]
Justification: LLMs were not used in the development evaluation of the proposed methods.
15

---

A Appendix Supplemental Material
A.1 Complete Proof of Theorem
Expectation convention. Unless stated otherwise, E(.] is over mini-batch sampling and data aug- mentations (conditioning on 2 when convenient).
Step I: Loss and weights. Let z; E s c Rd be the anchor and Zμ+ its positive. Define the set of others S == {1, (1,,,,,7)) m) {i) and the negatives-only N,; = S; \ {it) with size N The InfoNCE loss (van den Oord et al. 2018) is
exp(sa+ /T)
L; -log
s1 2, E 1-1,11 T>0,
EkE(ttJUN; exp(six/T)*
exp(sig/T)
with softmax weights Ptj Ene(i+)uN; exp($1/77)
and positive-miss == Part
Step Moments, spectrum, and assumptions.
Let
= 1 Σ
25
5 = N
Σ
2j3g
= Amax ($,) (1/,11] a!):=
1=1
JEA
Under unit norms, tr E tr 2; = We assume:
(AI) Unit norm: | for all
(A2) Zero mean & negatives-only independence: E[z] = 0; for j E N,, negatives are i.i.d. and independent of 2; (no independence is assumed for the pair (i, (i,
The spectral quantity o: is deterministic per batch/anchor. A useful bateh-level proxy is
Σ N n 2,2, 211 22+)
2 Σ
ô, Amax (2). (4)
n1
Step 3: Gradient.
Since V 2.j Zj²
V fe
Σ
Pai
2i+
= = (M, - 2++) = 7 M Σ PikZ
KEfEtJUN
V Li =
Step 4: Exact decomposition. Let qi = T-P
Py
for j E N, (so EjEN, 4j 1) and 27 =
N ESEN; 2/. Then
6, (pist 1)z + (1 Ditt
Σ
Pifzz A; A B Ci.
jEN;
with
Σ 4i N )
A -,
B; = (1 -Pat) Pa+t
C===
Post)
jEN,
We use || + + 2112 < 3 |0|1 + ||y| + 2| (source of the factor 3 below).
Step 5: Bounding the components.
(A) Softmax error. Since || = 1,
(B) Sampling noise. By (A2), B|| 1|
hence
E
1
E[ Pi+t )21.
16

---

Remark (independence vs. correlation). If negatives are correlated (mean zero, unit norm), then
N
B[z] zk. (i k))
Meorr,
Ppoor
so the sampling term inflates with positive correlation; the spectral terms below are deterministic in (8)
(C)
Adaptive fluctuations. Linearizing the negatives-only softmax around uniform logits N
(App.
A.3 Lem. A.1
gives
4i
915-2 N
Raj
8/
Σ
8ijs
JEAT
and C = -f c with (App. A.3
Cor.
BI $ 7
1
[
Pii )2 o
2 71
Cam
E Pai (0.
where Csm 0 is a doftmax-smoothee constant (using || |l S implies Cam }2
Step 6: Upper band (exact and proxy forms). Combining (A)(C) and |6,112 S 3(1|A,
| + [c).l2) yields
E[(1 Pitt )2]
V cil2 S
Ble
RT((1 - Pst [
N
74
(5)
3cem
[[ Pat
(0! (o.
Applying the deterministic proxy ceiling
per batch and then taking expectations gives the convenient
3 72
E Pii
pir-)
E
2 12 s
B[c +
N
3 [1 Pist (n(n(nn (////nn2111) -2))
+
(6)
71
3cam B((1 - Ditt) ) +
(n/(n 22)
76
Step 7: Lower band.
Since ||M||| -
[M, 2,+ 112 = ||Mal| 122 + 2(M, 21+) 2 (1 (M,, 2)
By Jensen,
(1 p)
V s, Ci ||
= [[ 2
define a spectral gradient band whose width scales with the positive-miss (e:), finite-sample noise (1/N-), batch anisotropy (via de or σ), and temperature (r).
Conclusion. The bounds
Remarks. (i) Independence vs. correlation. Assumption (A2) is used only obtain E| 1|==
1/N for the sampling term; with correlated negatives this term inflates as the remark in Step 5(B), while the spectral contributions and the lower band unaffected (see App. A.4).
(ii) Batch
proxy and concentration. The proxy provides a per-batch computable ceiling o High-
(4)
probability control of follows from standard matrix concentration for second moments (App. A.
(iii) Smoothness constants. The constant Cam comes from log-sum-exp smoothness (App. empirically, choices in [0, give nearly identical band tracking.
17

---

A.2 Matrix concentration for second moments
Assume (2:)-1 are ii.d. mean-zee zero sub Gaussian vectors in R with proxy K; that is, w|
Let Σ 2[ satisfy tr Σ 1, and define the (uncentered) em- pirical second moment Σ = : Ex=1 z,z," Since B(zi] 0, Σ the covariance.
||u 21 1l
There exists universal constant C such that, with probability at least -, 6,
3 - E112 s C d (og(1/6)
d log(1/6) Σ) + C n²
Xmaxx S
Amax
n
n
(See, e.g., Vershynin 2018a Thm. 5.39; see also Tropp 2015 for matrix Bernstein.)
Trace constraints and clamping. If, in addition, each sample is unit-norm (l2zl22 = 1), then tr E 1 Σ; || = 1 exactly, hence Amax (λ) s 1. More generally, when norms not fixed, ir E concentrates around 1 at dimension-free rate:
log(1/o)
tr - s C² C²R²
with probability least 1-6
n
and passing to the trace-one normalization : &. Lr E ensures Xmax ( < by construction.
Isotropic and effective-rank specializations. In the isotropic spherical case I/d, we have Amaxx [) = 1/d, giving
Amax ②) I min { 1, d
(15) cCx2
w.p. 21-6,
n
which is the brobbbbbobyblltyy clamp used in the corollary. More generally, bounds with effective rank r(E) = -<E)//[E12 = 1/Amax (E) S dl yield (up to constants)
(V log(1/5)
log(1/8)
E - El|2 x?
n
which can be substantially tighter when the spectrum is low-rank; see, e.g., Koltchinskii & Lounici (b)
A.3 Negatives-Only Softmax: First-Order Expansion & Remainder Bounds
Lemma A.1 (Negatives-only softmax linearization). Let qij be the negatives-only softmax weights over N₁ with logits lgg == S1g/T and E = - ΣjEN; lag. Then, for each anchor and j E N
fo- N- + Rej, |ayl S (ey 22
Equivalently, in vector form, with II = I - N- 111,
1 qi - N- nI; + R.
| S 1Te,
Proof. First-order Taylor of softmax = V lse around uniform logits; use 1 <2lse(u) 3 1/1 (spectral norm bound of the softmax covariance).
Corollary 1 (Bounds for c.0)) and c( With c.") = 1-PucL EjEN, (sig - 3, )a, and c.e) - (1-Pat) EjEN; Rezzj, we have
N
12 <
1 [[ Pis+ pis+) ) o..
Bl <
- Cam F( P (o!
72
where (00) = Xmax (Ep) and Cam
< 1. Sketch. For 1 apply Cauchy-Schward Schwarz across j
and Var jEN; (str) < 25 Eg 2, < ox
(8) For d.2) combine Lemma A.l with E, - E,)²
11 and ||1ع, < 1 N (
18

---

A.4 Effect of Correlated Negatives on the Sampling Term
This appendix refines Step S(B) of | There, the independence assumption is used only to obtain E|
1/N for the sampling term in the upper band (5) (6). Here quantify how correlation
among negatives alters that term; the spectral contributions remain deterministic in a (() and are
unchanged.
Let N₁ be the negatives for anchor i, N == | and & := - EjEN, Fj.
Lemma A.2 (Sampling term under pairwise correlation). Assume unit norm || ll2 = 1 and mean
zero E [|z]] 0. Define the (common batch-averaged) pairwise correlation
Σ j.E.V; jztk
E [
Pcorr
N-
1)
Then
N N-
N
E
N
Peom
B.
B[(1 Pii )²]
N-
N-
Moom
Relative to the independent case, the sampling term is inflated by factor + (N- 1eoor
1 ( Ej zb. take expectations, and group diagonal versus off- Proof. Expand || = (N
diagonal terms.
Corollary 2 (Operator-norm control). Let Cjk: = E[2yz]] for j + k and set 1 == / ju Cyuul = ||Cxxllo (common across pairs). Then Poorr N-(N--I Ejjt tr(Czx) S d) and hence
N -
dn.
N
N
Thus small cross-sample operator norm implies small inflation of the sampling term.
Synthetic validation. Let 9s == V., We generate correlated negatives via shared-component
model: 2j x vou+ VI-a with u Unif(Sd-1), E (N La), then renormalize to unit norm. This yields Moorr & a in high dimension. We sweep E 0.1). N E (62, 254, 1022].
256, - 0.1 and, for each batch, check coverage of ||g:|? between the lower bound (-
and the upper band in Thm. 1.1 (using per-anchor o())) Coverage remains within 5% tolerance up to peoor & 0.02 for all N Beyond this, deviations are driven exclusively by the sampling-term
inflation predicted by Lemma the spectral terms track as before.
MoCo queue note. sleue-based methods (e.g.. MoCo v2) can induce weak correlation among nearby keys due to momentum updates. practice, correlation decays with queue lag: averaging over all negative pairs in N,- yields small effective Peorr & 10-3, so the global inflation factor 1 + (N- 1) remains close to for N S 1024. Our coverage checks $A.11 match this prediction.
19

---

A.5 Balanced Spectral Picker (Policy P3)
Algorithm BBllanced Spectral Batch Selection (Policy P3)
Require: Candidate pool B,, target rank R. (e.g., running median/percentill;; cap by min mnn(n; d)): flag ROWSUNITNORME (TRUE, FALSE)
1: A" f- +00, Z* <-
2: for all batches Z E B, do if ROWSUUNTNORM then
Z E Raxd
3:
D Assume | = as (A1)
4:
G + ZZ
nxn Gram; cost (n²dd)
5:
8 |G]]
>= 1 Z p no eigendecomp needed > = 1/tr(E>) for Σ 1 Z Z with E =
6:
R - n²/5
7:
else
8:
H t Z
> dxd Gram; cost (dP
9:
| pp
t t (r(H)
10:
R 4 12/8
D 1/ tr(5²) with E = H/t (trace-one normalization)
11:
end if
12:
R
k clip(R, 1, min(n, d))
numeric guard; lies in this interval
13:
if R
R. < A then
14:
A+ + R R,b. +
15:
end if
16:
end for
17:
return Z*
Notes on cost. Use the G = zz branch when n < d (typical for vision), and the H = 2T z branch when d Both routes avoid eigendeccompostiion:; they require only Frobenius norms and traces.
Computational tip (choose Tz vs. ZT 2). Given Z E gaxd use the smaller Gram: TTZ).
Option A (rows unit-normalized), := = TZT E Ruxn Ref = n² n²/ /| (since | T 1 =
| (l Prefer when n < d.
Option (general norms). И Z E Raxd Rof (tr (tr 1/ tr tr((2) with Σ = H/ tr H. Prefer when d < n.
In both cases, form the Gram with BLAS and then Frobenius norm. Memory Ommnn d² }).
Sanity checks. (i) All rows identical G 11
|G n². Rot 1. (ii) Rows
near-orthogonal (and n < d) = G I,, |G Re n, so Rot R n.
A.6 Second-Moment Update and Trace Reduction
We analyze how adding single sample changes the batch second moment's trace-square, which controls the effective rank Raff = 1/ ir(2)). Here ΣB denotes the batch second moment (not negatives-only).
Lemma A.3 (One-star step trace update). Let B = f21....,20) 20} C Rd be unit vectors with Σg =
T For unit vector z, define
(3(2) == 2 TE 1 Σ 22,2))
tB = tr(E))
Then
12 LB + 2b (b)
2b (qn(2)) - (B) + (1 LB)
ta (n())U())))
to tr(2) - tg
(b + 1)2
(b+
20

---

If |f||=|| + 1, replace the terminal 1 by || Moreover, fifisttorder order expansion in 1/b gives
tr Bu(=) = LB + b [[((()) - [n] O))
(8)
Range. Since Eg = and |E 1,
tg = tr(20g) E 1/rank(EB)) 1],
so decreasing ty increases Rea = 1/tb.
Greedy-builder rationale. For fixed partial batch B (size b), the one-step change after adding
At tr Bu()) ts 2b (qx(2) (n) + +1 (1)1B) LB).
9=(2)= = Σ
(2, 2
(b+ 1)2
#'EB
Hence tr( < whenever
1 - tB
(B(2) < tB 2b
To minimize At a step, select
arg (B(2) arg min
Σ 22,2
z'en
Intuitively, qs (2(2) is the Rayleigh quotient of w.r.t. EB; low- choices spread mass away from current principal directions, reducing tp-
Relation Frobenius diversity. Define Ar(z] B) = [EB zz T 11 tr(E)) 22 E
||2z [| For unit 2, ||=2 so
arg max Ar(= | B) arg min (z
(If ||=||| * 1, replace the trailing by [|=
Algorithm 2 Greedy Spectral Batch Builder (balanced window)
Require: Pool D, target size n, probe size m,, window [Rmins Rmass] with 1 S Reain S Rmax S
min(n,d) d)
1: Initialize B with two seeds (ef. $1.4 compute tp tr(E.) R 1/tB
2: while [B] and R & [Rmin Rmaxl do
3: Sample m candidates C C DD> B
4: for all do
Cost per candidate (bd)), or O(b) with cached dot products
5:
q(2) <
22,
6: end for
7:
2" + arg minzec ((2)
8:
B B- BUBBU {2* update via Lemma A.3 set R clip(1/tn, 1, min(n, d))
9: end while
return B
Complexity & implementation. Evaluating (2)2) needs b inner products (O(bd).. With m candidates, step costs (mmdd). Maintaining the Gram of selected points Kg 12 2r) lu
allows (2) in (bb) per candidate and updates ty via running K ll (no cigendecooposstioon) Precomputing squared inner products avoids an extra square in the loop.
Centroid proxy (heuristic). The score A(2 2
B=:= = - -2, (2, with Eg = ".veB= 2' satisfies
1 M (3, S (B(=).
S'EB
so it correlates with gø but can select different points; therefore prefer qa when feasible.
21

---

MoCo compatibility. The same builder applies when candidates are drawn from a MoCo queue; diagnostics use the queue-aware proxies of App. A.11 (replace ē by min min(1,0q + Ex {x)), while the greedy objective qs and update in Lemma A.3 are unchanged.
A.7
Variance band for per-sample squared gradients
We bound Var(?4) with n. ;= s, Cil2. Recall V a; Li = =
"0,0; 0; ;= :- 2gt, and the decomposi-
tion 6;
A,+ B;+ C with A, - E, B, = (1 --P++)) 27 C = (1-Pas+ ΣjeN; (9is N- )zg
1.2
Then
T = = 1|0 - (114,1)= + |B + I[C:1l
Using Var(X) E[X²] and applying the same bounds termwise:
(A) Softmax error. | c so Var(l A, = [[4] - [[(]]-
(B) Sampling term. Under negatives-only independence (A2), 2, N= EjEN; 23 has B||2, 1|2-
I/N- and Var(1 (? N (1 - ) (unit-norm, mean-zero),
Var(I] [1 < E[( Pa+ )*] Var [a 1?) < (1 (1- xー N- [E((1 - Pist )
(C) Spectral fiuctuation. I Linearizing the negatives-only softmax (App. A.3) yields C₁ = -(')+ with
EIIC Thus Var[IC.:]1 Cill is controlled by -. and (o. (2 terms; grouping these into B, keeps the leading sampling-spectto term explicit.
1P - + B[(1 P =.
fam 7t
E[(1 ps+ )² (o. y3]
Combining (A)-(C) and the - prefactor yields
Var(n)) S
3
(1 (1-N-)
o. +
Br,
N- 1
N-
A(N-5r)
with o, = E[Amax (Σ,) ) (or per-anchor) and B, := Eie + Elel OrT 100=-211 2E - Pi+ Poul) +
O(T "0[[B[[[1- Pia )o..).. A deterministic proxy follows from Amax (2,) S n= (cf. $ $.3
A.8 Exact upper- expression for real-date validation
For batch { at temperature т, define
= 1 - Pal
2
N == n n
i=1
Let the negatives-only second moment for anchor i be Σ N= ESEN, 2j2] with (1))
Amaxe (E,).
Per-anchor (exact) upper band. yields
Averaging the negatives-oni form of Thm. across anchors
V
22l
3
3
n
3c
4 (o
(9)
7
76
where is softmax smoothness constant controlling the - remainder.
Batch- upper band (used in Fig. 3 If o. are not computed per-anchor, use the batch Xmax (E) and the conservative bound
second moment Σ - 2427 with
n
0+
<
δ for all i,
11 2
obtain
7
S
3
2 +
2
32
3
n
2
(10)
22
N-
74
n
76
n - 2
22

---

Implementation note. All quantities in (T0) come from the training batch: Pa+ from the InfoNCE softmax, 2² by averaging (1 Pa++)* ), and ā as the top eigenvalue of Σ (trace one under unit-norm embeddings). We clip Pu-- E [0, 1]. plots we set c=0.5; results are stable for [0, 1].
A.9 Empirical deviation from isotropy (diagnostic)
Our theoretical results in 1.2 rely only on a bounded-spectrun condition Xmax (E) S max (assump- tion (A3)); they not require isotropy. Nevertheless, it is informative to track how real mini-batches approach isotropy during training.
We train SIMCLR (ResNet-50) on ImageNet. After every optimization step we collect the é2- normalized projection-head outputs E IR 256 for a mini-batch (n=10244 and form the batch second moment
Σ = 1 2 Z.
= [ 2m
Here E, is the per-step analogue of used in the main text. Because || for all tr(E.) = 1 exactly. To quantify deviation from the isotropic matrix L1, we compute the relative Frobenius deviation
F - - 100valL= 27 2
& 100
since
/ = d-1/2
Figure plots 5, for the first 500 updates, averaged over ten independent seeds (solid line; shaded band- s.e.m.). Deviation starts just below 8% and decays rapidly, stabilizing around 3.7%. The
dashed horizontal line (8%) is visual guide only, not theoretical threshold.
Batches move quickly toward isotropy, but a negliggli anisotropy (3-4%) persists even after early convergence. This empirical behavior supports our choice of the more general bounded-spectrum assumption (A3) rather than assuming perfect isotropy.
Batch-lovel deviation from isotropy during early training
Drppncal 5, (rman of rums)
std. error threshold
200
200
300
400
500
Traning step
Figure 9: Deviation from isotropy during early training. Batch-level Frobenius deviation 5, over the first 500 steps (SimCLR ImageNet; 1024, d-256). Solid eurve: mean 10 seeds; shaded band: +1 s.e.m. The dashed line at 8% is a visual reference only.
A.10 Empirical sensitivity of the spectral band under anisotropy
The gradient-norm band in $ 1.2 depends on the negatives-only top eigenvalue a. = Amax (m))) where
Σ j
2
N =
For batch-level reporting also use the full-batch proxy Amax with E 1 Σ, 242 (trace-one under unit-norm embeddings); they satisfy
23

---

To sweep spectral skew, we synthesize unit-norm embeddings by sampling ~ (O, Ia) and setting Art
A with A = UA A = diag())....,,A.)) Ad), Ex d = L, and U orthogonal. This yields (r(E) exactly while letting the trace-one spectrum vary with (A&)-
We fix n=256, d-1024, and construct positives with target cosine E (0,1) via 2, = C2 V1 c² Tn Frram-Schhidt for uj). For each batch compute softmax weights Pijs M, Ej Pig the alignment == (M,, 2,-), and the negatives-only spectrum o! = max (E₂). Across spectral-spread settings (controlled by (Ax)) we generate 5,000 batches and, for every anchor, check whether ||9,1* falls within the theoretical band
UB(o!")] from Theorem
Oovougge Anisetrugy
Popp vola
Asiutran Food Phi
Figure 10: Spectral-band coverage batch anisotropy. Out-of-band rate for gradient norms function of Frobenius deviation 6(%) 100 [ Each marker aggregates 5,000 synthetic batches (n= 256, d= 1024, r= 0.1); bounds use per-anchor Ps and (1) (or the proxy S n-2 à when used). Dashed line: 5% tolerance. Coverage remains within tolerance up to 8 6%. with mild upper-bound overshoots beyond that level.
Across a wide range of anisotropy levels, coverage remains within tolerance. The few violations are mild upper overshoots at high o., consistent with conservative constants on higher-order terms (notably the r - piece). These results support replacing isotropy assumptions with top-cigenvalue control that better matches modern contrastive regimes.
A.11 MoCo v2: Queue-based evaluation and robustness
We assess the generality of the spectral band and spectrum aware selection in a nased con- trastive setup using MoCo v2 (He et al., (2200a] Chen et al.) (2200b). Unlike SimCLR, MoCo samples negatives from memory queue of momentum-encoded keys, which changes the negatives distribution and calls for queue-aware proxies of the band parameters.
Setup. Dataset: ImageNet- 100. Backbones: ResNet- 18 (default), ResNet-50 (variant). Batch:
n5256. Temperature: =-0.2, EMA momentum: m7 0.999. Projection: 128-1 1288. Queue size: K E (16,384, 32,768, 65, 65,536). Selection pool: k=2 Policies: RANDOM, POOL-P3 (balanced target R, E {0.3, 0.5, 0.7). default 0.5), and GREEDY-64. We report top-1 accuracy, epochs to 70% top-1, and paired t-tests over seeds (3 seeds to match our SimCLR setting; 10 seeds for Cls).
Queue-aware band proxies. For each anchor i, the MoCo softmax is over keys from the current mini-batch plus the queue. We estimate a negatives-only second moment from a random subset Q, of K, queue keys:
Σg
1 Σ 2 K.
ōQ
Amax -mmx
JEQ:
24

---

Let =g = E/22 ] denote the second moment of queue keys. By matrix concentration (App. A.2
with probability 2 1-10 d.
[ Eallz S EK, εK = C Ck22
Yost 1//
S d, Teff := 1
tr(ZZ()
K.
hence
0.00 Amax (2,) S Amax (2i) S min( 1, ōQ + t
Alignment P: (M, 21+) is computed with the MoCo softmax
queue queue-batch keys. These substitutions instantiate the spectral band of Thm. [.I the queue setting. Correlation note: localized queue correlations inflate only the sampling term in App. A.4 the spectral terms remain deterministic given ōQ-
Main result (ImageNNet-000, ResNet-18, K365,536). Table summarises convergence and accu-
racy. With small on-GPU screening pool (k=512), POOL-P3 accelerates time-to-70% by +9.2% (3 seeds) without harming final accuracy; GREEDY-64 yields 5..9. Across 10 seeds, POOL-P3 achieves statistically significant +9.44% 4% + 1.3% speedup (95% CI; paired (-test, p<00.01,, with equal or slightly higher top-1. Wall-clock profiling on an A100 shows I 1% selection overhead because screening uses small pool and runs on-GPU; thus runtime gains closely track the epoch reduction (contrast with BB3 where large host-side pools incurred nontrivial latency).
Table 2: MoCo v2 on ImageNet-100 (ResNet-18, queue K=65,536, n=256, k=512, r=0.2).
Means + s.e.m. over seeds. Speedup is the relative reduction in epochs to 70% top-1.
Method
Top-1 (%)
Epochs to 70% %
Speedup (%)
3 seeds Random
74.2 + 0.3
146.3 + 2.5
Greedy-64 (Ours)
74.0 + 0.4
137.7 = 1.8
+5.9
Pool-P3 (Ours)
74.5 + 0.2
132.8 = 2.1
+9.2
10 seeds (CIs and significance) Random
74.19 +: 0.23
146.3 + 2.7
Pool-P3 (Ours)
74.52 1 0.19
132.6 + 1.9
+9.4 + 1.3
Robustness. Across queue sizes K E {16k, 32k, 65k), the ResNet-50 backbone, and rank targets R. {0.3, 0.5, 0.7], POOL-P3 consistently reduces epochs target accuracy with no loss in final performance. Frozen-feature linear evaluation (SGD, 90 epochs. LR 0.03) shows no regression: relative to RANDOM, POOL-P3 improves top-1 by +0.5% on CIFAR-10 10 and +0.3% on Oxford Pets.
A.12 Comparison to stronger baselines
To ensure spectrum-aware selection improves over established curricula, we compare against three strong alternatives under the same MoCo v2 protocol as App.AA./1 (IageNNt-- 100, ResNet- 18, batch n=256, queue K=65,536, temperature 0-2.2). The selection pool size is k=512 and our greedy variant uses m=64.
Hard Negative Mixing (HNM) Kalantidis et al. 2020): for each anchor we mix its positive with the hardest queue negatives (cosine via the key encoder); the mix ratio fixed across runs.
Distance-Weighter Sampling (DWS) Wu 2017): negatives are drawn from the queue with probabilities inversely proportional to norm-adjusted distance, re-normalised each step.
SupCon Khosla et al. (2020): a supervised contrastive upper bound that uses labels; included as
an oracle reference.
methods share the same backbone and augmentations. SupCon fully supervised; others are unsupervised. As shown in Table 3 Poot.-P3 is both faster and slightly more accurate than HNM and DWS. SupCon attains the best absolute accuracy-as expected, given supervision but POOL-P3 closes substantial portion of the gap without labels.
25

---

Table 3: Strong baselines on ImageNet-100 (MoCo v2. ResNet- 18, n=256, K - 65,666, T=0.2. k=512). Means + s.e.m. over seeds. Bold indicates best among unsupervised methods. SupCon uses labels (oracle).
Method
Supervised?
Top-1 Acc. (%)
Epochs to 70%
Random Sampling
No
74.2 = 0.3
146.3 + 2.5
Hard Negative Mixing
No
74.4 - 0.3
139.6 = 2.1
Distance-Weighted
No
74.3 0.3
140.2 + 2.0
Greedy-64 (Ours)
No
74.0 - 0.4
137.7 + 1.8
POOL-P3 (Ours)
No
74.5 + 0.2
132.8 + 2.1
SupCon (Oracle)
Yes
75.9 + 0.2
125.1 + 1.9
Across five seeds, POOL-P3 outperforms both hard-negative curricula in convergence speed and final accuracy (paired t-test vs. Random; 0.05), while remaining fully unsupervised. SupCon provides an upper bound with label supervision.
A.13 Influence of augmentation strength
To verify that conclusions not sensitive to the augmentation recipe, ablate the strength of the two most impactful transforms-cole jitter and Gaussian blur-by +550% around the default SimCLR settings. We the same setup as BB.3 (ImageNet- 100, ResNet- 18, batch n=512, single V100), and report wall-clock time to reach 67.5% top-1. Selection overhead is included. (MoCo v2 shows the same trend; see App.[A.11]
Table 4: Wall-clock time (hours) reach 67.5% top-1 on ImageNet-100 (meands.e.m., 5 seeds). Augmentation strength scales the color-jitter and blur coefficients by -50%. 0% (default), or +50%; all other transforms unchanged. We report A
Kandon-Grees (positive = faster), Greedy-64
Random
consistently saves 14-15% wall-clock vs. Random (paired 1-test; < 0.05).
Augmentation
Random (h)
Greedy-64 (h)
(gain)
Weak (-50%)
7.05 + 0.10
6.05 + 01.88
+14%
Default (haseline)
7.20 + 0.11
6.10 + 01.09
+15%
Strong (+50%)
7.35 + 0.12
6.30 + 0.10
+14%
Across all three settings, Greedy-64 cuts time-to-acurray by roughly one-seventh, confirming that gains artifact of particular augmentation choice. The speed-up is slightly larger under stronger augmentations, suggesting spectral batch selection especially helpful when aggressive views amplify gradient heterogeneity.
A.14
Positive alignment and vanishing gradients
In $ 1.22 we noted that very high anchor-positive alignment can shrink contrastive gradients and slow optimisation. This is manishinng-rrdiicn effect, representational collapse; contrastive methods with negatives (e.g., SimCLR) are empirically robust in this regard (van den Oord et al. 2018 Chen al. (020aa (see also Grill al. 20200 Chen He 2021 for collapse-avoidance in non-negative settings). Prior work has discussed a similar tension between strong positive alignment
and optimisation dynamics (Wang Isola 2020 Robinson et 2021
From the par-sample squared lower bound,
(1 2)
2,12 >
Ps = (Mi
72
we obtain the & version
1 P:
|| c,
(11)
which makes the role of -nchor-positive alignment explicit (higher-order corrections are negligible in our regime).
26

---

Measurement. We train SimCLR on m0g (batch size n=512, temperature r=0.2, seeds). During the first 200 optimisation steps we record per-sample gradient norms 1, normalise them by their step-1 value, and bin by the anchor-positive cosine Pt cos(zi, 2, ) used here a proxy for p: (M;, >,)) Results in Fig.
Gradient norm vs. positive alignment
1944 p 0.3
p- 0.8
nermaiise by
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Anchor-positive alignment
as a function of the anchor-positive cosine p. Gradients remain well scaled for P S 0.3 (early training) and decrease approximately linearly in 1 - p; beyond p Z 0.8 they are effectively vanishing, consistent with the trend implied by (TT). Spectral diversity remains high throughout, indicating no representational collapse.
Figure 11: Normalised gradient scale vs. alignment. Mean Cill/lV
Take-away. High positive alignment does not induce collapse in SimCLR, but it does reduce gradient magnitudes roughly proportion to - p. hindering optimisation. Inequality (II) quantifies the effect; Fig. II validates the predicted trend empirically.
B ImageNet-1k: Effective Rank vs. Accuracy
We pre-train SimCLR with ResNet-50 backbone (global batch size 4096, temperature 0.1, 200 epochs) and log the batch second moment every 5 epochs. Unless stated otherwise, values in Table 5 are final-epoch means + S.c.m. over 5 seeds. The effective rank Roff is computed with the Σ, 2,2, after trace-one normalisation (tr Σ = under unit-norm embeddings), i.e., from the eigenvalues of E. After pre trainiann the encoder is frozen and linear classifier is trained for 90 epochs (SGD, learning rate 0.03).
RankMe estimator (Garrido et al. 2023) on the embedding second moment E
Table 5: Effective rank of embeddings and downstream linear evaluation accuracy ImageNet- 1k. Higher Ret CC (with C=1000) correlates with improved accuracy; Pool-P3 yields the strongest gains. Reported numbers final-epoch means + s.e.m. 5 seeds.
Policy
Rea
Re /C
Linear top-1 (%)
Random
790 1 6
0.79
69.8 1: 0.3
Pool-P1
925 1 5
0.93
71.1 1 0.3
Pool P3
960
0.96
71.4 0.3
Greedy 64
930
+
0.93
71.0 = 0.3
Table and Figure 12 reveal a clear monotonic link between spectral diversity and accuracy. Random batches achieve only Ret /C'00.79 and reach 69.8% linear top-1, while Pool-based selection raises the rank to 0.93-0.96 and improves accuracy by 1.2-1.6 points. Greedy 64 matches most of Pool-Pl's gains at lower cost, confirming it as practical alternative. Accuracy plateaus once Ro ( Z 0.9, consistent with the RankMe interpretation that diversity gains saturate beyond this threshold. Pool-P3 delivers the highest effective rank (0.96 C)) and the strongest linear accuracy (71.4%), but Greedy-64 achieves nearly identical performance once the batch rank is past the 0.9
27

---

Accuracy vs. Effective Rank on ImageNet-1k
RankMe plateau (0.9)
71.5
Policies
Poo.P3
Gready-ed
Accuracy
70.5
70.0
Ramdbm
0.800
0.625
0.950
Run
Figure 12: Accuracy effective rank on ImageNet-1k. Each marker shows mean s.c.m. over 5 seeds at the final epoch. Grey dashed line marks the RankMe plateau (Rett CC = 0.9 with C=1000).
plateau. In short, Retf is a strong predictor of downstream accuracy: lifting Ref up to the plateau yields significant improvements, while pushing beyond brings only marginal returns.
B.1
Equivalence of Two Effective-Rant Rank Estimators
Lemma B.1. Let Z Raxd collect a batch ) rows. Define the (uncentred) second moment $ 2 T Z and its trace-one normalisation E := Σ/ tr E. Then
1Zz 11 (TT)
(tr(zT2))²
tr(())
hence
tr tr(2²)
122 11
In particular, if [|:|||2 for all i (so Lr(ZT2) 2) nn), then
12 11?
n²
tr(
Rott =
n²
112 1
tr (2)
Proof. By definition,
tr($²)
n tr 2)²)
tr (r 2)²)
tr 5r2
(Ir {)
( (r(T 2))²
(ur(ZT 2))
Since tr tr(((2)) 2)2) tr (ZZ =| 1/ the stated identity follows. If additionaly ||=:|22 = for all i, then tr(Z Z) Σ, ||2:1|2 n, which yields the unit-nome corollary.
B.2 Bounding the second-order remainder
Lemma B.2. Let the mini-batch satisfy (A1)-(A3) and use the negatives-only set N₁ = S (it}
size -n= With the notation of Step 7, the softmax Taylor remainder satisfies
(8ij 5, )
Reg
Ris 5
Σ
Sijz
Sij := 2, 2z.
2N 72
JEN,
and
ci:2) = (1 (1-p
Σ
Rij 2j.
JEN,
Assume a bounded fourth moment for siz: EE [sss 3; ) universal constant C4. Then
CA of with a. Amax and
Ca
E [[c
< N-
[[ Pis 0z o..
(12)
28

---

Proof. By across and ||||| 1, 1,,, ayzyll S N N2 az with aj == (1
Patt) R.j. Hence
E I 12 I N E E a = N E { Pis+) )² >>>
Rij
Using
[Ray $ (8i) - (2) -²).
I S 4(N-) 24 Σ (8ig 87)²
Take expectations and apply the fourth-moment bound: IE E,(5is - -3, S N C4o2. Combining gives
E C| 1p I N B[(1 Pis+ )
1
N C,0
C4 B Pi+ )
4(N 727
T
Absorb the 1/4 into C4 to obtain
12p
Together with the leading term E ( = [ Pu+ P/o./T2. Lemma
B.2
justifies the Ö ([[(1-
Pot+) r)) remainder used in Step
B.3 Computational Efficiency and Wall-Clock Analysis
Spectrum-aware batch selection shortens training in epochs, but its practical value depends on wall- clock time. We therefore compare four strategies IMAGENET- (ResNet-18, global batch size n=512, five seeds, single V100 GPU): Pool-P3, which draws each batch from a candidate pool of k=5,120 images using the balanced policy (P3); and Greedy-64/Greedy-2
which construct the batch element-wise with Greedy-m 1.144 using probe sizes m E {64, 256} Pool-P3 evaluates candidates the host CPU while the GPU executes the current step. Although the extra Okd) dot products account for <3% of arithmetic FLOPs, we measure = 8 ms of host scheduling / kemnl- launch overhead per iteration on V100, which the primary source of the wall-clode penalty reported below.
Table 6: Wall-clock time to reach 67.5% top-1 (f-00% of Pool-P3's plateau). A Time is relative to Random; positive = faster, negative slower.
Method
Epochs
Time (hours)
!
Collapse
Random
1.7
7.20=0.11
0/5
Pool-P3 (k 5, 520)
110=
2.10-0.10
-11%
0/5
Greedy-64
114+0.9
6.10=0.09
+15%
0/5
G5c
112+00.7
6.80-0.10
+6%
0/5
Pool-P3 reaches the target in the fewest epochs, but is ss11% slower than Random in wall-clock time due to CPU-side overhead. Greedy-64 delivers the best overall efficiency: it needs 23 fewer epochs than Random yet reaches the target 15% faster wall-clock time, and 24% faster than Pool-P3. Greedy-256 lies in between, trimming epochs slightly further than Greedy-64 while yielding smaller time gain (+6% vs. Random). No run collapsed in any setting.
Practical recipe. A brief warm-up with Rool-P1/P3 during the first 15 epochs- when the isotropy gap d is largest-in is followed by switch Greedy-64 once 5 5%. Table 7 summarises the trade-offs
'The selector uses only cached embeddings and products (no extra forward passes). wall clock times reported here include selector overhead.
"Icons require busepackage(pifoont)
29

---

Diminishing Increasing
4.5
130
4.0
Epochd to 67.5% Acc.
3.5
Screering
125
120
2.0
1.5
110
1.0
1000
2000
3000
4000
5000
Candidate Pool Size
Figure 13: Diminishing returns with pool size k. Increasing & from 512 to 2,048 captures ~00 of the total epoch reduction obtained when raising k to 5,120, while roughly halving CPU screening cost; further increasing to 5,120 yields only marginal benefit.
Policy
Stage
Wall-Clow
Overhead
Stability
Pool-P1 (max Ran)
Warm-up (520
Med-Low
Pool-P3 (balanced)
Mid-late
High
***
Greedy-64
Post warm-up
Low
dz
Random
Ablations
Low *
Table 7: Qualitative trade-offs (* better). Greedy-64 offers the best speed-stability balance after warm-up.
Many-class setting. On IMAGENET- 1K (C=1000 classes) we observe the same pattern: Pool-P3 maximises Rer = 1/ tr(Š²) and lincar-eval accuracy (71.4%), but host-side overhead widens the time gap. Accuracy plateaus once Ret /C/ 2 0.9 (Fig. 12 App. B for details.
Pool-based selection delivers the strongest spectral conditioning per epoch, but large A can negate those gains in wall-clock time. Greedy-64 64 captures most of the convergence benefit at negligible runtime cost, making a practical default once early-collapse risk past.
C Related Work
Gradient Behavior in Contrastive Learning. Understanding gradient magnitudes and their sta-
bility has been central to preventing collapse in contrastive learning. Prior work has noted the vanishing- beahiingggradient problem when negatives are not sufficiently diverse (Chen et al. 2020a Wang & Isola 2020). Theoretical studies have explored gradient norms from statistical viewpoint (Wen al.] 2021), though most focus on loss curvature or optimization dynamics rather than bounding gradients explicitly. Our work provides the first tight non-asymptotic bounds per-sample InfoNCE gradient norms, linking them to spectral propertics of batch embeddings.
Spectral Views and Isotropy. Recent papers have highlighted the role of spectral geometry and isotropy in contrastive representations. Errmolov et al. (2021) and et al. (202T) advocate batch whitening to improve feature isotropy, while Cai al. (2021) show that local embedding distributions approach isotropy during training. Zimmermann & Geiger (202I) explores the alignmen--uiffrmity trade-off, yet do not provide spectral control mechanisms, Our spectral-band framework formalizes these intuitions and introduces a concrete tool (effective rank) for regulating batch anisotropy.
Batch Composition and Negative Sampling. Several works improve contrastive learning by modifying batch composition. Hard-negative mining (Robinson al. 2021 Kalantidis al. 2020
30

---

selects the most informative negatives, but can introduce instability or collapse. Debiased contrastive loss (Chuang al. 2020) reweights negatives avoid sampling bias. hisihtee-weighted sampling (Wu et al. 2017) and curriculum-base methods (Robinson et al. (020)) modulate learning dynamics lack a spectral perspective. Our approach unifies batch diversity and stability through a spectral lens and introduces principled batch samplers (Pool-P3 and Greedy-m) that adapt to gradient scale constraints.
Spectral Diversity and Effective Rank. The use of effective rank proxy for diversity has been studied domains like matrix estimation (Roy Vetterli] 2007), self-supervised learning, and Gaussian mixture recovery (Vershynin (018b). We extend its application to the InfoNCE setting by showing how effective rank tightly bounds gradient magnitudes. Moreover, our batch selection policies leverage this relationship optimize learning efficiency and avoid collapse without requiring supervision or loss reweighting.
Theory-Driven Batch Selection. Prior efforts to construct batches via theoretical surrogates include batch norm-awart sampling (Zhao et al. 2020), importance sampling in metric learning (Harwood et al. 2017), and entropy- based selection. Our work distinguishes itself by providing explicit upper and lower bounds on gradients, derived from spectral assumptions, and by constructing lightweight, label-free policies to maintain training in safe, stable zone.
31


Reviews:
[{"review_id": "4CCnzCBFhX", "reviewer": "Reviewer_NJiJ", "summary": "This paper presents a theoretical and practical contribution to the understanding of contrastive learning. The authors introduce a spectral band framework that establishes sharp, non-asymptotic upper and lower bounds on the squared gradient norm of the InfoNCE loss.", "questions": "Regarding the pool-based policies, could you comment on the wall-clock time trade-off between the cost of evaluating the candidate pool and the reduction in training epochs? Is there a point where the selection overhead for P3 might negate the wall-clock time savings from faster convergence?\n\nYou show that your framework applies well to MoCo v2. How do you anticipate the dynamics changing with even larger negative queues or memory banks?\n\nThe performance of the Greedy builder is dependent on the probe size m, with m=16 showing degraded performance. Is there a theoretical principle or heuristic to guide the choice of m relative to the desired batch size n or embedding dimension d?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe core of this paper is a novel and rigorous theoretical framework. Deriving explicit, non-asymptotic bounds on the InfoNCE gradient norm provides a formal understanding of phenomena (like collapse and gradient vanishing) that are typically understood through empirical observation. The decomposition of the gradient into alignment, variance, and spectral terms is particularly insightful.\n\nThe authors' claims are supported by a strong set of experiments. They first validate their theoretical bounds on synthetic data. They then demonstrate that these bounds are tight and predictive during real-world training of SimCLR on ImageNet. The evaluations of the proposed batch selection methods on both SimCLR and MoCo v2, and against strong baselines are good.\n\nThe proposed Greedy Element Wise Builder is good. It provides a lightweight method to construct spectrally diverse batches without the high computational cost of evaluating full-batch spectra, achieving comparable performance to the more expensive pool-based policy while offering significant runtime savings.\n\nWeaknesses:\n\nWhile the Greedy builder is efficient, I think it has the drawback that the full pool-based policies (P1, P3) require maintaining and evaluating a large candidate pool of batches (e.g., k=5120) at each step. This can introduce significant computational overhead. The paper demonstrates accelerated convergence in terms of epochs but is less clear about the net wall-clock time improvement when using these specific policies, which could be an important practical consideration.", "was_revised": true, "final_justification": "I remain with my original score which was already positive. I think that this paper is good but not a strong accept.", "rebuttal": "Response (computational overhead, Pool vs. Greedy policies).\nWe thank the reviewer for highlighting the practical importance of wall‑clock time. In the revision we added Section 3.7 (“Computational Efficiency and Wall‑Clock Analysis”), which benchmarks both epoch and time‑to‑accuracy. The results confirm the reviewer’s intuition:Pool‑P3 reaches the target accuracy in the fewest epochs (110 vs. 137 for Random) but incurs the highest wall‑clock cost due to CPU‑side candidate screening. Greedy‑64 achieves nearly the same epoch count (114) while completing training 15% faster than Random and 23% faster than Pool‑P3. To balance these trade‑offs, we now recommend a hybrid strategy: run Pool‑P1/P3 only in the first 15–20 epochs to avoid early collapse, then switch to Greedy‑64 for efficient long‑run training. We illustrate this trade‑off in a new figure (Fig. B.2), and show that a smaller pool size (k=2048) already delivers ~80% of the epoch gain at half the screening cost compared to k=5120. This analysis strengthens the practical relevance of our method and gives concrete deployment guidance.\n\nResponse (large queues in MoCo v2).\nOur spectral band extends naturally to queue‑based setups. As the queue size  𝑄\nQ grows, three terms behave differently: Noise floor (1/N) With N=batch+Q, the variance term shrinks hyperbolically once \nQ≫ batch size. Doubling the queue from 65k→130k halves this component again, but the total variance is already dominated by the spectral‑spread term at that scale. In fact, our MoCo v2 ablation (Fig. 7) shows speed‑ups plateauing once Q≈16k.\n\nSpectral spread (σ_{max} ).\nGrowth is governed by freshness, not sheer size. FIFO queues refresh ~25% per step, so covariance drift is modest. We observed only a 3–5% increase in σ_{max}  when scaling from 65k (default) to 262k.\n\nAlignment (ρ).\nThe lower bound in Eq. 1 depends only on anchor–positive alignment within the current mini‑batch; queues do not affect this term, so the collapse diagnostic remains unchanged. Finally, our samplers (Pool‑P, Greedy‑𝑚) operate on the candidate set for the next batch; whether negatives come from GPU or a memory bank only changes the cost of estimating 𝜎_{max}. . Empirically, even with a 262k queue we observe a +7% wall‑clock speed‑up over Random (vs. +9% at 65k). Thus, enlarging the queue beyond 65k yields only marginal gains, and diversity‑aware batch construction remains beneficial, albeit with smaller absolute improvements.\n\nResponse (choosing Greedy probe size 𝑚)\n\nThe Greedy‑𝑚 m builder samples  𝑚  candidates, scores them, and adds the one with the largest diversity gain. Its effectiveness depends on whether the probe “covers” enough directions to enlarge the batch’s effective rank 𝑅_{eff}. From the upper‑bound analysis, the variance contributed by the probe scales as  O(1/m). Thus: At the extreme m=1, Greedy reduces to random sampling; at 𝑚=𝑛 it is equivalent to Pool‑P. Empirically, the probe needs to cover at least a constant fraction of the space or the batch: m≳max( d ,n/4).\nThe dimension criterion (m≳ d ) ensures the probe spans principal covariance directions; the batch‑fraction criterion works when \nn≫d. Once  m≈O(d), marginal gains flatten (e.g. Greedy‑64 ≈ Greedy‑256). In practice, values below this regime (e.g. \nm=16 when d=128) under‑explore high‑diversity directions and degrade convergence. Our guidance is therefore to set   \n𝑚≈max(\\sqrt{d}}, ,n/4)"}, {"review_id": "OHYW561NlS", "reviewer": "Reviewer_wUiB", "summary": "This paper proposes a spectral-band framework to highlight the important role of batch diversity in contrastive learning. Based on their framework, they put forward batch selection strategies to enhance contrastive learning. Experiments on synthetic datasets, ImageNet-100, and CIFAR-10 demonstrate the effectiveness of their method.", "questions": "1. In the experiments in Section 3, how are the augmentation parameters set? And do these augmentation parameters affect the effectiveness of your proposed strategy?\n2. Would it be better to use a smaller network or embeddings from a certain intermediate layer of the model for batch selection? This would save more time, and the performance might not be significantly worse. \n3. What is the validation accuracy of each strategy in Section 3.5?\n4. Why do the previous experiments use ImageNet-1k or ImageNet-100, but Section 3.6 switches to CIFAR-10?\n5. The experiments in Appendix B are also of great importance, and I believe readers should be reminded in the main text to refer to Appendix B.", "limitations": "None.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The theoretical proof is solid and intuitively understandable.\n2. This paper is readable.\n3. Extensive experimental analyses demonstrate the effectiveness of their method.\n\nWeakness\nOne of my major concerns is the issue of time overhead, because it is necessary to extract embeddings of multiple batches through the model when selecting the batch in each iteration. Could you provide the training time for each method based on the settings in Figure 8?", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer: Weakness One of my major concerns is the issue of time overhead, because it is necessary to extract embeddings of multiple batches through the model when selecting the batch in each iteration. Could you provide the training time for each method based on the settings in Figure 8?\n\n Response: To address this, we have now included wall-clock training times for each method under the settings of Figure 8, where we compare Pool-P3 (with $k=5120$), Greedy-$m$, and the Random baseline on ImageNet-100 using ResNet-18. We report in Table 2 (Section 3.7) the total training time (in hours) to reach a representative accuracy threshold of 67.5% (90% of the plateau). While Pool-P3 achieves the fewest epochs, its screening overhead leads to a net increase in wall-clock time. In contrast, Greedy-64 provides a strong trade-off, requiring slightly more epochs than Pool-P3 but delivering the best wall-clock performance due to its low overhead (no candidate embedding evaluation). We emphasize that Pool-based screening is executed asynchronously on the host CPU while the model’s forward/backward pass runs on the GPU. Thus, even at $k=5120$, Pool screening adds only an ~8 ms delay per step (≈3% of total step time), and the memory footprint remains small (≈2.6 MB for $d=128$). We have added these results and discussions to the main text in Section 3.7 (Computational Efficiency and Wall-Clock Time Analysis), and provided the corresponding training times requested.\nReviewer: In the experiments in Section 3, how are the augmentation parameters set? And do these augmentation parameters affect the effectiveness of your proposed strategy?\nAugmentation settings (all of §3).\nWe adopt the canonical SimCLR (Chen et al., 2020) and MoCo v2 (Chen et al., 2020b) pipelines so that any observed gains stem solely from our batch‑selection strategy: Random resized crop: scale ∈ [0.2, 1.0], aspect ∈ [3/4, 4/3], then resize → 224²\nHorizontal flip: p = 0.5\nColor jitter: p = 0.8; {brightness, contrast, saturation} = 0.8, hue = 0.2\nRandom grayscale: p = 0.2\nGaussian blur: kernel = 23, σ ∈ [0.1, 2.0]; p = 0.5 (SimCLR) / 1.0 (MoCo v2)\nDoes augmentation strength change our conclusions?\nWe swept color‑jitter and blur coefficients by ±50% (see Appendix B.4). Even at the strongest setting, the Greedy‑64 policy still reduced time‑to‑67.5% top‑1 accuracy by 14 ± 1% vs. Random (compared to 15 ± 1% under default augmentations). Hence the relative benefit is robust. If anything, it tends to grow under stronger augmentations, since larger view‑to‑view variance amplifies gradient heterogeneity, which our spectral selector mitigates.\n\nReviewer :Would it be better to use a smaller network or embeddings from a certain intermediate layer of the model for batch selection? This would save more time, and the performance might not be significantly worse.\n\n Response: Intermediate layer features (ResNet‑18 conv4, 512‑d).\n• CPU cost: dot‑product screening drops from 8 ms → 6 ms per step, but this work is already overlapped with GPU compute, so the net wall‑clock gain is ≤ 0.7%.\n• Variance reduction: feature covariance is more anisotropic; the gradient‑norm band widens and Greedy‑64 yields only a +9% speed‑up vs. Random (compared to +15% with the final 128‑d projection).\n• Accuracy: unchanged in top‑1, but fewer epochs are saved.\nTiny “preview” network (2‑layer CNN, 64‑d).\n• CPU cost: screening becomes nearly free, but the extra forward pass on CPU adds ≈3% wall‑time (or competes with GPU cycles).\n• Spectral mismatch: preview features lag ≈1 epoch behind, causing batch‑ranking errors; this yields a 1.6 pp drop in top‑1 and eliminates the variance reduction benefit.\nWhy we use the final 128‑d projection.\n• Latency already overlaps with the next GPU step, so further CPU savings translate to < 1% wall‑clock gain.\n• Spectral fidelity is more important than FLOPs: using the exact representation seen by the loss produces the tightest gradient‑norm band and the largest variance reduction.\n• Simplicity and reproducibility: no auxiliary models or extra data loaders are needed.\nFuture work.\nAs noted in § 6, we see promise in distilling a lightweight predictor that tracks the full‑model spectrum more faithfully; we plan to release hooks to facilitate community exploration in this direction.\n\nReviewer: What is the validation accuracy of each strategy in Section 3.5?\nWe did track validation performance; the table below reports the final top‑1 on ImageNet‑100 (ResNet‑18, 5 seeds, mean ± s.e.m.):\nStrategy\tTop‑1 % (final)\nRandom\t69.2 ± 0.2\nPool‑P1\t69.4 ± 0.2\nPool‑P3\t69.5 ± 0.2\nGreedy‑64\t69.3 ± 0.2\nGreedy‑256\t69.4 ± 0.2\nAll methods reach statistically indistinguishable accuracy; the gain from spectral batch selection is faster and more stable convergence, not a higher ceiling. Notably, Greedy‑64 matches Random while cutting wall‑clock by 15 %, demonstrating efficiency without loss of generalisation.We have added this in text in pg 7 section 3.5.\n\nReviewer: Why do the previous experiments use ImageNet-1k or ImageNet-100, but Section 3.6 switches to CIFAR-10?\n\nResponse: Why § 3.6 uses CIFAR‑10 instead of ImageNet\n\nWe chose CIFAR‑10 intentionally:\n\nLow‑resource stress‑test.\n§ 3.6 asks whether our batch‑selection policies stay robust when data and model capacity are limited—precisely the regime where spectral diversity is hardest to maintain. CIFAR‑10 is the standard, compact benchmark for that question.\n\nPractical ablation budget.\nThe study sweeps multiple pool sizes and policies. Repeating those grids on ImageNet‑1k would require several GPU‑months; CIFAR‑10 lets us run exhaustive ablations in hours while preserving the qualitative dynamics.\n\nConsistent trends across scales.\nThe relative ordering of strategies (Random < Pool ≤ Greedy‑64) and the magnitude of the speed‑ups match what we observe on ImageNet‑100/1k. This shows the gains arise from the spectral criteria, not a dataset peculiarity.\nWe have added a short note in § 3.6 to make this rationale explicit\n\nReviewer :  The experiments in Appendix B are also of great importance, and I believe readers should be reminded in the main text to refer to Appendix B.\nResponse:  In the revised manuscript we have added explicit forward‑references to Appendix B from the main text. In particular, Section 3 now notes that full augmentation details are given in Appendix B.4, Section 3.5 highlights additional runtime and stability ablations in Appendix B.2, and Section 3.6 clarifies that complementary MoCo v2 results and robustness checks are included in Appendix B. We believe these reminders improve readability and ensure that the important supplementary experiments are not overlooked."}, {"review_id": "yCMry31bgJ", "reviewer": "Reviewer_8UG5", "summary": "This paper introduces a spectral framework for analyzing and controlling gradient magnitudes in contrastive learning. Using the matrix Bernstein inequality, the authors derive non-asymptotic upper and lower bounds on the squared InfoNCE gradient norm, relating it to alignment, softmax error, and the top eigenvalue of the batch covariance. Leveraging these theoretical insights, they propose several novel batch selection strategies, notably a fast Greedy Element-Wise Builder that outperforms random sampling on CIFAR-10 and ImageNet-100. They also demonstrate theoretically and empirically that in-batch whitening reduces gradient noise in SimCLR.", "questions": "Could the authors provide empirical validation or stronger theoretical arguments for the claim that high alignment leads to collapse in SimCLR?\n\nHow does the proposed spectral selection perform on datasets with a large number of classes, and how does it relate to findings in RankMe regarding rank–accuracy relationships?\n\nWhy were speed and accuracy gains not evaluated on ImageNet-1k, despite the spectral band validation conducted there?\n\nHow does the proposed method compare with MoCo v2+, and what is its effect when the majority of the batch gradients are detached, as in queue-based contrastive learning?\n\nDoes the efficacy of spectral batch selection persist for very large batch sizes (e.g. 8192 or higher), and is there a diminishing return beyond certain sizes?\n\nHave the authors evaluated the method on different backbone architectures, such as Vision Transformers, and if not, do they anticipate any architectural dependencies?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well written, clearly motivated, and logically structured.\n\n- It offers strong theoretical motivation and justification for the proposed methodological approach, contributing to the understanding of SimCLR’s training dynamics.\n\n- Interesting maths, that theoretically justifies the limitations that users observe when training SimCLR.\n\n- The authors conduct well-designed synthetic and experimental simulations that convincingly validate their theoretical arguments in a way that is clear and interpretable.\n\nWeaknesses:\n- SimCLR is nowadays underused in Self-Supervised Learning and is not really SOTA anymore, so the benefits of this paper appears quite limited.\n\n- In line 94, the authors note that high alignment between anchor and positive leads to gradient vanishing and potential collapse. However, no related works are cited to support this claim, and SimCLR is generally considered collapse-proof compared to non-contrastive methods like BYOL or SimSiam. Stronger arguments and empirical validation of this specific phenomenon would strengthen the paper.\n\n- The experimental setups do not include datasets with many latent classes. Related work RankMe (Garrido et al., ICML 2023), which is not discussed in this paper, demonstrates in Proposition 5.1 that classification accuracy plateaus depending on the rank-to-number-of-classes ratio. Incorporating this discussion and conducting experiments with a higher number of classes would enhance the analysis. Specifically, examining the relationship between effective rank (or batch diversity) and downstream classification accuracy, as in RankMe, would be valuable.\n\n- While the authors assess the real-data spectral band on ImageNet-1k, they do not report speed or accuracy gains on this dataset. Including these results would strengthen claims of general utility.\n\n- The paper does not compare with MoCo v2+, which uses a queue mechanism to encourage batch diversity. A comparison with MoCo v2+, as well as a discussion or analysis of gradient magnitudes when a large portion of the batch has its gradient detached (as in queue-based contrastive learning), would situate the proposed method better within the literature.\n\n- It would be helpful to analyze whether the benefits of the greedy approach persist at larger batch sizes (>4096), as generally done in the literature.\n\n- The paper focuses on ResNet architectures; analysis on different architectures (e.g., Vision Transformers) would enhance generality claims.", "was_revised": true, "final_justification": "The authors have adequately responded to my points during the rebuttal, and I am thus increasing my notation.", "rebuttal": "Reviewer Comment:\nSimCLR is nowadays underused in Self‑Supervised Learning and is not really SOTA anymore, so the benefits of this paper appear quite limited.\n\nAuthor Response:\n We agree that SimCLR itself is no longer state‑of‑the‑art; however, our contribution is not tied to SimCLR as a method. The core idea—using the spectrum of in‑batch embeddings to guide batch selection—depends only on the covariance structure of embeddings and applies to any contrastive or InfoNCE‑like framework. To clarify this, we emphasize in the revision that: \nModel‑agnostic principle. Our bound on the squared gradient norm (Eq. 1) involves only the batch covariance, regardless of how embeddings are produced. This makes the method applicable to newer SSL approaches such as BYOL, VICReg, SwAV, and DIN.\nValidation beyond SimCLR. We already demonstrate gains on MoCo v2 (§3.5, App. B.3), where Greedy‑64 yields a +9% wall‑clock speed‑up over Random. We also report competitive results on SupCon and Hard‑Neg baselines (Table 3).\nPractical relevance. Even if SimCLR itself is used primarily as a transparent test‑bed, efficient negative‑set management remains a bottleneck in large‑scale SSL. Greedy‑64 reduces training wall‑time by 15% on a single GPU and by 20% in distributed training, translating into meaningful energy and cost savings for any modern SSL recipe. Forward compatibility. Because the batch‑selection operates on cached embeddings, it is orthogonal to architectural advances (ViT, ConvNeXt, hybrid CNN–Transformer) and loss‑level innovations (InfoLOOB, CLIP‑style multi‑modal contrast).In summary, while SimCLR serves here as a simple and interpretable benchmark, the spectral batch‑selection idea is framework‑agnostic, validated on MoCo v2 and stronger baselines, and directly relevant to current and future self‑supervised pipelines.\nReviewer Comment:\nIn line 94, the authors note that high alignment between anchor and positive leads to gradient vanishing and potential collapse. However, no related works are cited to support this claim, and SimCLR is generally considered collapse‑proof compared to non‑contrastive methods like BYOL or SimSiam. Stronger arguments and empirical validation of this specific phenomenon would strengthen the paper.\n\nResponse:\n We agree that SimCLR is widely considered collapse‑resistant compared to non‑contrastive methods such as BYOL or SimSiam. Our intent at line 94 was not to suggest that SimCLR suffers from full representational collapse, but rather to highlight a gradient‑level degeneration: when anchor–positive pairs are overly aligned, the InfoNCE loss produces vanishing gradients, which can slow early training or amplify numerical instability. This is distinct from representational collapse and is observable even in SimCLR. To strengthen this claim, we have made the following changes in the revision: \nClarified wording in the main text to explicitly distinguish between gradient vanishing and representational collapse.\nAdded citations to prior works that analyze this effect:\n– Wang & Isola (NeurIPS 2020): show that excessive alignment hurts uniformity and reduces gradient quality.\n– Robinson et al. (ICLR 2021): discuss collapse of loss curvature when positives dominate.\n– Huang et al. (NeurIPS 2022): provide empirical evidence linking high alignment to shrinking gradients at decision boundaries.\nProvided empirical validation in Appendix A.9 we measure gradient norms during early training and show that gradients shrink sharply once $\\rho$ (anchor–positive cosine similarity) exceeds 0.8, while representations remain non‑collapsed. For the first 50 steps, $\\rho < 0.3$, confirming that the lower bound is informative in practical regimes. \nWe believe these clarifications and additions make our argument clearer and properly grounded in both the literature and our empirical evidence."}, {"review_id": "qjn9rwOFjf", "reviewer": "Reviewer_fanw", "summary": "This work studies how to construct or identify batches for contrastive training which have certain gradient properties. They start by upper and lower bounding gradient norms. Then, they relate these bounds to effective rank, which can be computed over batches. This then gives the authors a way to control the gradient norm, by computing the effective rank of candidate batches and selecting the one which has the desired effective rank. They then show different convergence speeds for three different effective rank targets, and provide a greedy online method for building diverse batches. They show slight performance improvements relative to vanilla training.", "questions": "1. Does this result hold on problems outside of small-scale classification?\n2. Figure 6 does not seem trained to convergence, what happens with longer training?\n3. Self-supervised (contrastive) learning is often used to pre-train models with large-scale datasets before transfer. Does pre-training with batch selection give a meaninful performance boost for transfer?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Using rank as a diversity proxy is both sensible and interesting.\n* Speedups seem strong. \n* I like the empirical verification of the theory. \n\nWeaknesses:\n* The main weakness I see is empirical evaluation. The evaluations are done at quite small scale model (ResNet-18), on CIFAR and ImageNet-100, both rahter small datasets. \n* Related to the above, overall accuracy gains on ImageNet-100 is very small, a fraction of a percent (Table 2 and Table 3) . \n* The writing is scattered, with key results in the appendix (eg. Table 2).\n* Figure 6 seems to demonstrate that training has not converged, which makes any claims about performance pre-mature. \n* Presentation of figures can be improved; for example Figure 2 is blurry, Title of Figure 3 is cut off, legend of Figure 8 is covered (this is minor).", "was_revised": true, "final_justification": "The authors provided additional experiments which substantially improved my confidence in the utility of the the authors findings.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces a spectral framework for analyzing and controlling gradient magnitudes in contrastive learning. 
This framework allows deriving non-asymptotic upper and lower bounds on the squared InfoNCE gradient norm, relating it to alignment, softmax error, and the top eigenvalue of the batch covariance. Leveraging these theoretical insights, the authors propose several novel batch selection strategies that outperforms random sampling on simple benchmarks. 

The authors response to the reviewers addressed the main concerns by adding substantial experimental results that would strengthen the paper. I believe this work could be useful to the SSL community as it provides a precise quantification of the effect of the choice of the batches and practical ways for selecting them to improve contrastive training.
Decision: Accept (poster)