On the Importance of Embedding Norms in Self-Supervised Learning
Authors: Andrew Draganov
Sharvaree Vadgama
Sebastian Damrich
Jan Niklas Böhm
Lucas Maes
Dmitry Kobak
Erik J Bekkers
Venue: ICML 2025 poster (2025)
Keywords: Representation Learning
Embedding Norm
Network Confidence
Self-Supervised Learning
Primary area: general_machine_learning->representation_learning
TL;DR: We show in theory, simulation and practice that the embedding norms have critical consequences for self-supervised learning.

Abstract: Self-supervised learning (SSL) allows training data representations without a supervised signal and has become an important paradigm in machine learning. Most SSL methods employ the cosine similarity between embedding vectors and hence effectively embed data on a hypersphere. While this seemingly implies that embedding norms cannot play any role in SSL, a few recent works have suggested that embedding norms have properties related to network convergence and confidence. In this paper, we resolve this apparent contradiction and systematically establish the embedding norm's role in SSL training. Using theoretical analysis, simulations, and experiments, we show that embedding norms (i) govern SSL convergence rates and (ii) encode network confidence, with smaller norms corresponding to unexpected samples. Additionally, we show that manipulating embedding norms can have large effects on convergence speed.
Our findings demonstrate that SSL embedding norms are integral to understanding and optimizing network behavior.

On the Importance of Embedding Norms in Self-Supervised Learning
Andrew Draganoy 12 Sharvaree Vadgama Sebastian Damrich Jan Niklas Böhm Dmitry Kobak" 4 Erik Bekkers* 3
Lucas Maes
Abstract
els (Girdhar et al., 2023; Bommasani et al., 2021). Indeed, prominent methods such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and Florence (Yuan et al., 2021) all rely on standard SSL objectives discussed in this paper.
Self-supervised learning (SSL) allows training data representations without a supervised signal and has become an important paradigm in ma- chine learning. Most SSL methods employ the cosine similarity between embedding vectors and hence effectively embed data on hypersphere. While this seemingly implies that embedding norms cannot play any role in SSL, few recent works have suggested that embedding norms have properties related to network convergence and confidence. this paper, we resolve this apparent contradiction and systematically establish the em- hedding norm's role in SSL training. Using theo- retical analysis, simulations, and experiments, we show that embedding norms (i) govern SSL con- vergence rates and (ii) encode network confidence, with smaller norms corresponding to unexpected samples. Additionally, we show that manipulat- ing embedding norms can have large effects on convergence speed. Our findings demonstrate that SSL embedding norms are integral to understand- ing and optimizing network behavior.
These algorithms work by representing similar inputs near each other and dissimilar inputs far apart in an embedding space normalized to hypersphere. This is done by optimiz- ing objective functions based on the cosine similarity be- tween embeddings: popular SSL frameworks like SimCLR (Chen et al., 2020a) and MoCo (He et al., 2020) optimize the InfoNCE objective, while SimSiam (Chen & He, 2021) and BYOL (Grill al., 2020) optimize the cosine similarity di- rectly. Thus, theoretical studies of SSL representations only consider the distribution of points on this latent hypersphere (Wang & Isola, 2020; Zimmermann et al., 2021).
At the same time, there has been some empirical evidence that the embedding norms prior to normalization contain meaningful information. For example, Kirchhof et al. (2022) noted that embedding norms are related to network confi- dence. No thcoretical explanation of this phenomenon was given. Separately, Zhang et al. (2020) and Wang et al. (2017) showed that training based on the cosine similarity loss af- fects the embedding norms and that the embedding norms affect the gradient magnitudes. However, the implications of this for SSL methods have not been fully explored. This implies a clear gap in the literature: several works suggest interactions between SSL models and the embedding norms but the extent of this relationship is unknown. Our work provides the first thorough analysis showing how the em- bedding norms interact with SSL training dynamics.
1. Introduction
Self-supervised learning (SSL.) has emerged as a powerful tool for learning representations from unlabeled data. This is because, when trained on large unlabeled datasets with contrastive objectives, SSL models often achieve perfor- mance levels comparable to those of supervised methods and can even induce emergent properties (Chen et al., 2020a; Caron et al., 2021). SSL has also substantially advanced molti-modal learning, particularly in languaee mod-
First, we prove theoretical bounds showing that the em- bedding norms impose a quadratic slowdown on SSL con- vergence and verify that this occurs in simulation and in practice. At the same time, we show that optimizing the cosine similarity grows these embedding norms across SSL methods. This leads to catch-22: small embedding norms are required to train SSL models but these norms grow dur- ing training. Put simply, effectively training SSL models requires managing the embedding norms. We offer and evaluate several mechanisms for doing this.
"Equal contribution Department of Computer Science, Aarhus University, Denmark PIAS-8, Jülich Forschungszentrum, Germany 3 "AMLab, University of Amsterdam, The Netherlands "Hertie In- stitute for AI in Brain Health, University of Tübingen, Germany Miila, Quebec Al Institute, Canada. Correspondence to: Andrew Draganov drraganovandrew @ gmail.com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
Second, we argue that, because the embeddings grow with each gradient update, their norms naturally correspond to

---

On the Importance of Embedding Norms in Selr-Supervieed Learning
ambient Rd
(0.0)
S
((+)= L(o) || [V2(O)]
Figure 1. Left: Network outputs (blue) lie in ambient I8 but only their projections (white) onto the hypersphere S enter the loss. However, the embedding norms (shades of blue) influence the norms of the SSL loss gradient. Middle: Gradient field of the cosine similarity with respect the orange direction. Highlighted gradients (black) illustrate the inverse relationship between the gradient and embedding norm (Proposition 3.1). The blue points trace an embedding's 's trajectory using gradient descent. Right: Because gradient updates are orthogonal an embedding point, they must increase the point's norm (Corollary 3.3).
the frequency of observed latent features. Since models are more certain in frequently observed data (Zhong al., 2021), we contend that SSL embedding norms encode model's confidence. We show that it consistently holds across SSL. methods.
2.1. Preliminaries & SSL methods
Perhaps the prototypical method in this family is Sim- CLR (Chen et al., 2020a), which appears among a broader line of contrastive methods for self-supervised learning on images. These all utilize repulsions from dissimilar (nega- tive) samples to prevent representation collapse (Oord et al., 2018; Henaff, 2020; He et al., 2020; Chen et al., 2020b; Misra & Maaten, 2020). In practice, this is performed by optimizing the InfoNCE loss, which we write with respect to embedding 2; as
Moreover, these two phenomena interact during SSL train- ing. For instance, since norms both encode uncertainty and scale gradients, SSL models implicitly learn unevenly over the samples- an effect we show must be mitigated when training on imbalanced datasets. Throughout our paper, draw attention to similar interactions between embedding norms and SSL dynamics by raising several open questions as directions for future work.
ExpSim(z; 2z)
(z)(Z) log Σ Exet
(1)
keti ExpSim (2,, 2x)
- 2, 2 + log (S.),
This paper supersedes our earlier workshop paper (Draganov al., 2024). Our code is available ht https: //ttpp///ttuub... Btt.aggeoooe..SEteeeeeeee...n.no
(Z (Z)(Z)
where 2= 2/ is the unit vector of ExpSim(a, b) exp (ā (/TT/r) is the exponent of the temperature-scaled co- sine similarity, and S; = Ex ExpSim (2,, =k) represents the sum over all & + i in the batch. For ease of interpre- tation, we split the loss term into attractive and repulsive components, resp. (Z) and C (Z). Note that 2 (Z) is simply the negative cosine similarity between positive pair zj and 2, scaled by r. We assume T = for simplicity; in SimCLR, 7 0.5.
2. Preliminaries and Related Work
Although this paper concerns general SSL techniques, we will use the familiar language of image representation learn ing. The typical SSL pipeline consists of obtaining two augmented variants x, and T, from an input image and ensuring that the corresponding embeddings 2; and zj have high cosine similarity. We refer to (zi, .I;) (resp. (#i, (Ri,2;)) as "positive" pairs of points (resp. embeddings). Since attract- ing all the points together will collapse the representation, each SSL method has an additional mechanism to prevent this. The most common approach enforces low similarity be- tween embeddings of dissimilar inputs: for unrelated inputs T, and Tg. the embeddings z; and zy should be far apart.
Another common contrastive objective function is the triplet loss, wherein one normalizes the embeddings to the hy- persphere and minimizes the mean squared error between positive samples while maximizing the mean squared error between negative ones (Schroff et al., 2015). Due to the normalization, this implicitly optimizes the cosine similarity
2

---

On the Importance of Embedding Norms in Self-Supervised Learning
between embeddings (Grill al., 2020).
their average magnitude unmanaged. Our paper therefore differs from the prior literature by (i) extending these results to the InfoNCE loss, (ii) evaluating the full extent to which large embedding norms affect real-world training, and (iii) providing principled mechanisms for addressing this effect.
Curiously, methods like BYOL (Grill et al., 2020) and Sim- Siam (Chen& & He, 2021) showed that one can simply opti- mize Li), and avoid collapse by applying only the gradients to embedding& 2 (rather than to both 2; and 2j). We refer to these as non-contrastive methods. Throughout this paper, we will use SimCLR with the InfoNCE loss to represent a prototypical contrastive SSL model and SimSiam with the negative cosine similarity to represent prototypical non-contrastive SSL model. As is standard (Oquab et al., 2023), we use the kNN classifier accuracy on the embedding space to evaluate the quality of a learned representation. We provide linear probe results in Table S3 in the Appendix to show the generality of our results.
Similarly, it has been suggested (Scott et al., 2021; Kirchhof et al., 2022; 2023b) that an SSL embedding's ' magnitude could serve as a measure for the model's certainty. This was evidenced in two ways: first, by qualitatively showing that samples with high-embedding norm are good representa- tives of the classes (see Figure S6) and, second, by finding that the embedding norm is smaller for out-of-distributte (OOD) samples. Importantly, these references only show the existence of this relationship but, to our knowledge, there has not been an explanation provided for why this occurs nor systematic evaluation of the settings in which norms encode model certainty. Thus, our work expands the litera- ture by establishing how and why embedding norms encode SSL model confidence and providing a thorough empirical analysis for the phenomenon.
2.2. Related Work
Analysis of SSL Embeddings Due to its reliance on the cosine similarity, the seminal work of Wang & Isola (2020) showed that contrastive learning must satisfy two require- ments: all positive pairs must be near one another (align- ment) and all negative samples must spread evenly over the hypersphere (uniformity). Expanding on this blueprint, subsequent works have sought to formalize the learning capacity of contrastive methods (Saunshi et al., 2019; Zim- mermann et al., 2021; HaoChen et al., 2021; Saunshi et al., 2022) while much of the research into non-contrastive meth- ods has focused on how their architectures help to prevent collapse (Jing et al., 2021; Tian et al., 2021a; Zhang al., 2022; Richemond et al., 2023).
3. The Properties of SSL Gradients
We begin by studying the gradients of the cosine similarity with respect to an arbitrary point 2,. Throughout this section, we assume T = and refer to Z as any set of points in Rd with no other assumptions over the distribution. The gradient acting on one of these points has the following structure:
Proposition 3.1. [Prup. 3 in Zhang et al. (2020);² proof in A Let Z be a set of points in Rd and let 2; and 2j be a positive pair in Z. Let oxj be the angle between z; and zj- Then the gradient of C.) (Z) with respect to 2, is
SSL on Imbalanced Data It is an active area of research to understand how contrastive learning performs over vari- ous data distributions (Zimmermann et al., 2021). Although Liu et al. (2022) showed that SSL training is relatively ro- bust to imbalanced classes, there are mechanisms which can improve its performance (Tian et al., 2021b; Van As- sel & Balestriero, 2024). Nonetheless, foundation models which use contrastive learning require balancing the data distribution before training (Xu et al., 2023).
1 T
2,4
2j
8g |
Ia
2
Les
where aLs is the component of orthogonal to b. This has sin()r))
magnitude ||
||:i||
This has an easy interpretation: I, 262 | projects the unit vector å onto the subspace orthogonal to z, This projected vector is then inversely scaled by || We visualize this in Figure 1. A similar result holds for the InfoNCE loss:
Relationship of Embedding Norms and SSL While SSL representation learning has largely focused on hyperspheri- cal embedding distributions, some work has examined the embedding norm's role. Wang et al. (2017); Zhang et al. (2018; 2020) all noted that embedding norms inversely scale gradients under the cosine similarity loss and sug- gested that the embeddings must grow. However, each paper largely brushed this interaction away: Wang et al. (2017) suggested that Eg-oormalization suffices to handle the embedding norms, Zhang et al. (2018) states that at- tempts to resolve this interaction were unsuccessful and Zhang et al. (2020) provides a regularization term which shrinks the variance of the embedding norms but leaves
Proposition 3.2. [Proof in A.2] Let Z be set of points in R&d, 2i and zj be a positive pair in Z, and V.A be in Prop. 3.1. Then the gradient of Cig with respect to 2; is
ExpSim 25. 2k)
V.
2
(2)
S
'Zhang al. (2020) also showed corresponding results under SGD with momentum and Adam optimization (Kingma, 2014).
3

---

On the Importance of Embedding Norms in Sel--Supervieed Learning
Let 2t, o i, be a sample in the denominator of L. (Z), then the gradient of Lig (Z) with respect to 21 is
4. Simulations
We now present a suite of simulations which allow us to characterize how the parameters in Section 3 influence SSL training under idealized conditions. Full implementation and experiment details can be found in Appendix B.
1
ExpSim(zi, 2i, 21) (Z)
V
(3)
|
St
In essence, because the InfoNCE loss is a function of the cosine similarity, the chain rule implies that its gradients behave similarly to the cosine similarity's. Specifically, just like those of C. the gradients of 23 have the properties that (1) they are inversely scaled by ||= and (2) they exist in 24's ss tangent space. Since the InfoNCE loss is the sum of L. and C, these properties all extend to the InfoNCE loss as well. Going forward, we refer any loss function or SSL. model as it exhibits these two properties. This orthogonality has also been noted in Wang et al. (2017) and Zhang et al. (2018). 8)..
4.1. Effect of the Embedding Norms on SSL Training
We start by evaluating to what extent the embedding norms and angles between positive samples slow down conver- gence, Specifically, we sampled 500 pairs of points directly on go0 We produce many such sets of samples while vary- ing their mean embedding norms and dij values. We then evaluate Theorem 3.4 by applying the cosine similarity gra- dients to all positive pairs of embeddings until convergence.
(cos.sim.-based) if
Figure 2a plots the number of steps until convergence and shows that, although the convergence rate depends on both parameters, having large embedding norms is significantly worse for optimization than having large angles between positive pairs. In essence, the embedding norm's 's unbounded nature allows it to induce arbitrarily large slowdowns. Mcan- while, the angle between positive samples only has non- negligible impact as the angle approaches its upper limit x. Because it is exponentially unlikely for the angle of every positive pair to be close to π, we ignore the angular com- ponent of Theorem 3.4 for the remainder of this paper and relegate its further discussion to Appendix C.2.
As direct consequence of this projection onto the tangent plane, applying the cosine similarity or InfoNCE gradients to point must grow its magnitude (visualized in Figure 1, middle and right):
Corollary 3.3 (First identified in Wang et al. (2017); Proof in A.3). Let E Rdd and let z' be the result of applying a step of gradient descent with respect to the cosine similarity (Prop. 3.1) or InfoNCE (Prop. 3.2) to z. Then || 2 ||2|].
The results in Propositions 3.1, 3.2 and Corollary 3.3 reveal an inevitable catch-22 for self- self-supereised learning: we re- quire small embeddings to avoid vanishing gradients but optimizing SSL. loss functions grows the embeddings. We refer to this the embealding-nom effect. This effect also holds for the mean squared error between normalized em- beddings and, by extension, the triplet loss.
To further investigate the interaction between weight de- cay and the embedding norm effect described in Theorem 3.4, extend this simulation by augmenting the objective with regularization term on the embedding norm. That is, we now optimize Lij = T + + -| where Y represents the weight decay strength. We initialize em- bedding pairs with different initial norms (1, 4, and 7) to systematically test the quadratic dependence predicted by our theory. The results, shown in Table 1, demonstrate that while weight decay can indeed accelerate convergence by controlling embedding growth, the fundamental quadratic dependence on initial embedding norm persists across all tested regularization strengths. For moderate weight decay values (0.5 < 7 S 1), we observe faster convergence com- pared to the unregularized case, with higher regularization providing greater speedup. However, when weight decay becomes too aggressive (? = 10), the representation col- lapses entirely, presumably because the embedding is more incentivized to shrink to the origin than to maximize the cosine similarity. These findings align with our practical experiments in Section 6, where excessive weight decay led to representation collapse.
Furthermore, Proposition 3.1 has direct implications for convergence rates under the cosine similarity. The gradient's s magnitude directly scales the learning rate, since 2/- = 2, +
>., 2; + (7 [ll V Thus, Proposition 3.1 can be interpreted as saying that the embedding norm and the sine of the angle parameterize the model's learning rate. Indeed, both quadratically slow down convergence:
Theorem 3.4 (Proof in A.4). Let 2, and zj be embed- dings with equal norm, i.e. ||
||=/|| = p. Let 2 + 2 ((2)),e, Ls, and 2/1 24 + p (24)1e,, be the em- beddings after maximizing the cosine similarity via a step of gradient descent with learning rate 7. Then the change in cosine similarity is bounded from above by:
27 sin? dii
z 2/3 2 2
(4)
Put simply, the change in the cosine similarity via a step of gradient descent scales quadratically with the embedding's 's norm and the sine of the angle to its positive counterpart.
4

---

On the Importance of Embedding Norms in Sel--Supervied Learning
Stops untill convergenc severrence
10000
12
Class
Class 2
8.0
5000
Embedding Norm
10
Class 4
Class 3
Embedding Norm
8
7.5
Embedding Norm
200
Steps until convergence
6
7.0
100
Prob. -1/2
Prab. ~1/8
4
Prob. -1/4
Prob. -1/16
0
0.1
0.2
0.3
0.4
0.5
20
40
Mean angle for pos. pairs (radians)
Distance to k-th Neighbor
Epoch
(a)
(b)
(c)
Figure 2. Simulations studying the relationship between SSL training and embedding norms. Left: Applying cosine similarity gradients to pairs of points converges slower as function the points' norm and the their angle. Middle: Training via InfoNCE to reconstruct latent classes induces higher norms in dense output regions. Right: Training via InfoNCE leads to larger norm for high-frequency classes.
network with the supervised InfoNCE loss function (which explicitly samples positive pairs belonging to the same class) on this dataset.
Table Steps to convergence for different initial embedding norms under varying weight decay strengths. The objective is modified Li (z') +7 7 || simulate weight decay effects.
Figure 2b plots each embedding's 's magnitude in the learned space as a function of (inverse) density in embedding space. We use the distance to an embedding's 's 10th nearest neighbor under the cosine similarity metric as a proxy for inverse density. We see that embeddings in dense regions of the embedding space tend to have higher norm. This follows from Corollary 3.3: dense regions of the embedding space receive the most gradient updates and, consequently, those embeddings will grow the most. We also modify the simula- tion for Figure 2c by providing a class imbalance parameter to the class distribution. Namely, class now has sample probability - 1/2 class 2 has sample probability -1/4, and We then see that over the course of training, the mean embedding norms for frequent classes grow to higher values than those for sparse classes.
Initial
0.5 (Steps)
y= 1.0
10.0 (Steps)
Embedding
(Steps)
Norm
64
49
divergence divergence divergence
4
526
318
7
1080
610
4.2. Effect of SSL Training the Embedding Norms
We now consider how SSL training affects the embedding norms via simplified training setting where the data is gen- erated from latent classes. Inspired by Zimmermann et al. (2021) and Kirchhof et al. (2023a), consider an SSL dataset as a set of latent class distributions {ž,,.., }. where each 2, probability distribution on the d-dimensional hypersphere 8d. Let the observations obtained via generating process g Sd DD. That is, our dataset is obtained by randomly choosing a distribution 2, drawing a sample , from it, and applying to ž. We are training a neural network f X' -> gd via contrastive learning to produce a learned latent embedding f(X).
X RD be
Takeaway Across these experiments, samples which are seen more often have higher embedding norm under the InfoNCE loss. This can occur either due to the network considering these samples to be prototypical (and therefore embedding them in dense regions of the learned space) or being otherwise over-represented in the dataset. We point out that these are precisely the settings in which expect network to be confident in the embedding. We leave formal quantitative analysis as an open question:
We analyze the relationship of parameterized SSL train- ing to the embedding norms by simulating the above see- choose centers for 4 latent classes uniformly at random from g10, We then sample 4K points around these centers and normalize them to the hypersphere. From this, we produce the dataset via generating process g S00 c II + 804 where g is given by multiplication by a random matrix, We finally train a 2-layer feedforward
nario. Specifically,
Open Question 4.1. What theoretical bounds can be made regarding a sample's embedding norm and (a) the accuracy with which it is classified or (b) the corresponding inpul' 's dissimilarity from the training data?
5

---

On the Importance of Embedding Norms in Self-Supervieed Learning
Normalized Embedding Magnitudes
1.0
0.93
1,0
1.0
0.91
0.72
0.74
0.65
0.42
0.37
0.43
Cilar 10 Trai
Cilae- 10
Cifar 100
Clar 100
Cilae110 Thain
Cifar- 10
Cifar 100
Cifar 100 Test
Cilar 10
Cifar- ID
Cifor- 100
Cifor- 100 Test
Cita-10
Cila--10 10 Test
Cifar 100 Test
Test
Truin
Test
Test
Triin
Tesk
Thiin
Triin
SimCLR
SimSiam
SimCLR
SimSiam
Trained on Cifarl0
Trained on Cifarl
Figure 3. Violin plot showing the distribution of embedding trained on. All values are normalized by the training set's ' mean emhedding magnitude. Black bars represent and are laheled by the mean value of each violin. We the same augmentations for the train and test sets for consistency.
each dataset split function of which dataset the model
1.0
Human Labeler Accuracy
1.0
Human Labeler Entropy
0.4
CIFAR-10
ANN Accuracy
SimSiam
0.5
0.2
SimSiam
CIFAR 10
0.6
CIFAN-L0U
SimCLR SimSiam
CIFAR-10 CIFAR 100
0.0
0.4
0.8
0.0
0.4
0.0
0.4
Relative Embedding Magnitude
Relative Embedding Magnitude
Relative Embedding Magnitude
Figure 4, Evaluation of how embedding norms correspond to measures of network confidence. In all cases, we normalize the embedding magnitudes by the in-dataset maximum and them into twenty buckets. For every bucket with more than embeddings, we evaluate the corresponding metric. We represent the number of embeddings bucket using the marker size in the left-hand plot.
5. Embedding Norm as Network Confidence
exhibit substantially smaller norms due to their greater distri- butional shift. This relationship holds symmetrically when training on Cifar- 100 and evaluating on Cifar- 10, as seen on the right side of Figure 3.
Given the simulations in Section 4.2, we expect that train- ing cos.sim.-basee SSL models should result in common input samples receiving higher norm. We therefore use this section to show the various ways in which embedding norms encode network's 's confidence in practice. All model implementation details can be found in Appendix C.I.
Embedding Norms Encode Classification Accuracy An- other measure network's 's confidence in an embedding is the accuracy with which that sample is classified. To this end, train SimCLR and SimSiam the Cifar-10, Cifar-100, and ImageNet100 datasets." We then normalize the embedding magnitudes by the maximum across the dataset and bucket the embeddings into ranges of 0.05, giving us 20 embed- ding buckets over the dataset. Figure (left) then shows the per-bucket accuracy &NN classifier which was fit on all the embeddings with respect to the cosine similarity metrie. Indeed, see that the INN classifier's accuracy shows clear monotonic trend with the embedding norms
the same experimental setup as above and
Embedding Norms Encode Novelty Figure demon- strates how embedding norms characterize sample's o "out- offddistribttoonesss. On the left side of the figure, we trained SimCLR and SimSiam models on the Cifar-10 train set for 512 epochs and then compared the embedding norms across different data splits, normalizing all embed- ding norms by the Cifar-10 10 train set mean. The results reveal clear pattern: embedding norms decrease progressively with increasing distributional distance from the training data. For example, the Cifar- 10 test set contains novel but dis- tributionally similar samples and therefore results in only slightly reduced norms. In contrast, the Cifar- 100 data splits
We default the ImageNet-100 split (Wang & Isola, 2020) huggi huggingface face co so/datasett//cll httmc//aaannmmennneeeeeeee immagmettt---- 100.
6

---

On the Importance of Embedding Norms in Sele-Supe Learning
across datasets and SSL models.
SimCLR
SimSiam
Embedding Norm
40
Embedding Norms Encode Human Confidence Interest- ingly, not only does the embedding norm provide a measure for the sample's 's novelty and its classification accuracy, but it also provides a signal for human labelers' confidence and their agreement among one another. Using the Cifar- 10-N and Cifar- 100-N labels from Wei et al. (2021), where each training sample is labeled by the consensus label over mul- tiple human annotators, Figure 4 (middle) shows higher embedding norms correspond to more accurate consensus labels. Similarly, the Cifar- 10-H dataset from Peterson et al. (2019) provides 50 human predictions for each image from the Cifar-10 10 test set, allowing us to evaluate the en- tropy of the label distribution. Figure 4 (right) shows that, as the embedding norms grow, the human labels have less entropy (i.e., they more likely to agree with one another).
20-
ANN Accuracy
80
60
40-
2
128
8
32
128
Train Epoch
Train Epoch
No docay
Standard weight deeay
High weight deeay
Figure 5. Effect of the weight decay on the embedding norms in SimCLR and SimSiam. Epochs are log-scale and go from to 256. Corresponding kNN accuracies are plotted in the bottom row.
Takeaways Under the common assumption that an SSL embedding's s direction represents its information, these ex- periments show that the embedding's norm represents how confident the network was in this information. Further- more, this measure of network confidence is inherent to cos.sim.-based loss functions and emerges naturally dur- ing training. Thus, an SSL latent space looks less like smooth sphere and more like spiky ball, with the spikes corresponding to regions of known data samples. This ob- servation has implications for few-shot learning settings, in which one has pre-trained on a large dataset and then wishes to adjust the model to a second, smaller dataset:
@torch.no (o_gchhno_ggad()) grad
def cut_init (model, c)
for paran in param, data param. data /
()
Listing I. PyTorch code for our cut-initialization layer.
embedding norm's 's impact on the ANN accuracy is much more pronounced in the attraction-only setting. We also see that, even without weight decay, the embeddings can shrink (as occurs for SimSiam). We attribute this to the embeddings being produced by shared weights: although the gradients require all embeddings to grow, shared weights matrix may not be able to independently update every point's position.
Open Question 5.1. By using the embedding norm as a measure for sample's novelty, can one more precisely guide the training process unseen inputs?
6. The Embedding-Norm Effect in Practice
Cut-Initialization As shown in Figure 5, weight decay gradually reduces emhedding norms over time, but doesn't 't control them at the start of training. To address this, we propose cut- inttiiiiiazzatinn- diviiiiatioodiiidiigi all network weights by constant at initialization. This ensures small embed- ding norms at initialization which then kept small via weight decay. We implement this uniformly across all lay- ers for simplicity (Listing 1). Interestingly, a variant of this can be found in HuggingFace's default image transformer code (Py (PyTorch Foundation, 2025) and we know at least one bos.sim.basee SSL model which uses it (Peng et al., 2022).
To understand how the embedding norm effect influences siiin--iiiilarty-bassee SSL. training, we investigate three distinct interventions which should mitigate it. These inter- ventions provide controlled settings to analyze the empirical relationship between embedding norms and SSL training. Our experiments are on the Cifar-10, 10, Cifar-100 Imagenet- 100 and Tiny-Imagent (Le & Yang, 2015) datasets.
Weight Decay Our first intervention mechanism - weight decay- - is already present in essentially all SSL models. The idea here is that adding a penalty on the weights implic- itly regularizes the embedding norms (Wang et al., 2017). Figure demonstrates this effect in SimCLR and SimSiam training: without weight decay (7 0), embeddings grow unconstrained, while excessive weight decay (1 = 3 causes collapse. With appropriate values (y = 10-5 for SimCLR, ? = 5 10 for SimSiam), norms decrease grad- ually, leading to improved kNN accuracy. Interestingly, the
We study the interplay between cut-initialization and weight decay values on SimCLR and SimSiam in Table 2. Specifi- cally, we report the LNN classification accuracy after 100 epochs on the Cifar- 100 and ImageNet-100 100 datasets and see that, in both the contrastive and non-contrastive settings, pairing aut-iniiialization with weight decay accelerates the training process. As was the case for the weight decay, the
7

---

On the Importance of Embedding Norms in Self-Supervised Learning
Table ANN accuracy at epoch 100 various values of cut- constant e and weight decay A. color-coded by value. Left: Sim- CLR on Cifar-100. Right: SimSiam on ImageNet-100 100. Default weight-decay is underlined.
Table 3. Imagenet- 100 ANN accuracy (epoch 100) for MoCo/Dino.
Top-1
Top-5
MoCo
c= = 1
38.7
69.7
SimCLR
SimSiam
V2
== 3
43.8
71.8
Weight Decay A
Weight Decay X
MoCo
C == 1
54.8
82.7
le-8
le-6
5e-6
le-5
5e-5
le-4
5e-4
1e-3
V3
= 3
58.8
86.0
1
40.8
40.5
40,9
41.5
36.7
38.5
40.5
44.7
Cut
0
2
42.7
42.3
42.8
41.4
42,9
42.0
41.1
42.2
40.2
41.1
41.2
44.0
48.8
49.8
49.1
42.1
Dino
C= 1
C = 3 29.0
43.7
55.6
76,6
37.1
36.8
37.9
37.3
44.4
46.4
50.2
53.2
difference stronger in the non-contrastive setting. c and c performed best for SimCLR, providing an addi- tional 1-2% in accuracy the default weight decay, while 8 led to about a 10% increase for SimSiam. Extrapolat- ing from the trends in Table 2, we use c = 3 for SimCLR and c = 9 for SimSiam going forward. These are futher ana- lyzed in Figure $4, which also includes BYO experiments. We show the ANN classifier accuracies at 500 epochs with and without cut-initialization in Table 4. Here we see that pairing SSL models with cut-initialization often helps the model reach higher final accuracies.
Table 4. kNN accuracies at epoch 1000 for Cifar-10/100 and at epoch 500 for ImageNe//Tnylmaaeen for default, cut-initialized and GradScale training on standard image datasets.
Cifar
Cifar
Imagenet
Tiny
100
100
Imagenet
Default
87.7
56.6
59.4
36.0
SimCLR
Cut (c = 3)
88.2
60.1
60.9
37.7
GradScale
88.2
58.2
01.0
38.1
Default
88.1
61.8
62.0
41.4
SimSiam
Cut (c = 9) 88.4
62.6
67.2
41.7
We also evaluate cut-initialization in imbalanced data settings. For Cifar- 10, we use the exponential split from Van Assel& & Balestriero (2024), where class é has n, = 5000 1.5- samples. Similarly for Cifar- 100, the i-th class receives n, 500 1.034-² samples. This way, all classes are represented and both imbalanced datasets contain roughly 15K samples. We also use Flowers 102's naturally long-tailed test set for training (Nilsback & Zisser- man, 2008). evaluating on its validation set. Table 5 then shows that, in ing with interventions on the embedding effect can provide double-digit accuracy improvements.
ents' dependence on the embedding norm. We visualize the resulting gradient fields with powers p 0 and p = 1 for a 2D embedding space in Figure 6. We refer to training with GradScale power p 1 simply as GradScale. This can also be interpreted as per-point adaptive learning rate that is scaled by each point's embedding norm.
Under GradScale, the gradient norms differ from those dur- ing default training, necessitating new learning rate sched- ule. Traditionally, SSL models are trained with 10 epochs of linear warmup followed by a cosine- annealing schedule (Chen et al., 2020a). However, the schedule has an implicit division by the embedding norms over the course of train- ing. We therefore simulate this effective learning rate for the GradScale setting. Namely, we choose base learning rate y = 7/6 with 100 linear warmup epochs followed by cosine annealing, where 7 is the default learning rate. This was the first relatively stable schedule which we found for the 1 setting and we performed no additional tuning.
settings, pairing SSL train-
Lastly, Table 3 shows that cut-initialization also provides the expected boost in performance for MoCov2 and MoCov3 (Chen et al., 2020b; Fan et al., 2021), which are cos.sim.- based but use a transformer backbone (Dosovitskiy et al., 2020). We point out that Dino's 's objective function does not use the cosine similarity (Caron et al., 2021). Consequently, it does not benefit from intiittializziinn.
Table 4 demonstrates that, when training remains sta- ble, SimCLR's s ANN accuracy benefits from GradScale's embedding norm cancellation. Consistent with our cut- initialization experiments, this improvement becomes par- ticularly pronounced on the class-imbalanced datasets in Ta- ble 5: GradScale provides a roughly 5% accuracy increase across the imbalanced datasets. While these results are promising, we observed that GradScale with p failed to converge on Imagenet- 100 and for non-contrastive models. This limitation aligns with these models' known sensitivity issues (Zhang et al., 2022; Richemond et al., 2023).
GradScale Layer Perhaps the cleanest way to overcome the embedding-nomn effect is to simply rescale the gradient directly. We achieve this by introducing a custom Py ToTch aut ograd. Functicn on which we refer to as GradScale (for full implementation, see Listing 2 in the Appendix). This layer accepts a power parameter p and is simply the identity function in the forward pass. However, the back- wards pass multiplies each sample 2,'s 2 contribution to the gradient by ||= Thus, choosing power = 0 gives the default setting while choosing p = will cancel the gradi-
8

---

On the Importance of Embedding Norms in Selt-Supervised Learning
||
TET
||V4(2]|| sin(a)
Table 6. KNN accuracies at epoch 100 using the Adam optimizer for SimCLR and SimSiam under embedding norm interventions.
Cifar- 10
Cifar-100
Default
79.6
44.3
SimCLR
Cut
79.9
45.3
GradScale
80.5
46.6
Default
73.7
36.4
SimSiam
Cut
79.8
44.9
Figure 6. Left: the default gradient field of the cosine similarity with respect the north-pointing line (orange). Right: the gradient field using the GradScale layer with 1.
In addition to sceing that the embedding noom effect is more pronounced in attraction-only settings, we have found that the embeddings can shrink even in the absence of weight decay. We attribute the latter phenomenon to the network's shared weights: while our theory predicts uniform embed- ding growth, producing these embeddings via single set of weights creates competition between different regions of the latent space.
Table KNN accuracies at epoch 500 for default, aut-initialized and GradScalc training class-imbalanced image datasets.
Cifar-10
Cifar- 100
Flowers Unb.
Unb.
Unb.
Default
56.5
24.3
42.6
SimCLR
Cut (c 3)
61.1
26.3
61.6
Finally, we have been careful to not describe the embedding- norm effect strietly negative phenomenon. Consider the common transfer-learning setting in which prototypical class examples should anchor the learned representation (Pan et al., 2019; Lee et al., 2022). Our findings suggest the embedding effect may naturally support this goal: prototypical examples get large embedding norms and con- sequently receive smaller gradients.
GradScale
61.3
29.1
47.2
Default
47.0
21.6
22.5
SimSiam
Cut (c 9)
61.7
31.5
39.9
Embedding-norm Effect Under Adaptive Optimizers A potential concern is that our theoretical analysis, which focuses on standard gradient descent, may not extend to adaptive optimizers. To address this, we evaluate our embed- ding norm mitigation strategies using the Adam optimizer (Kingma, 2014). As shown in Table 6, the embedding- norm effect persists under Adam optimization: both cut- initialization and GradScale continue to provide consis- tent improvements across SimCLR and SimSiam both Cifar-10 10 and Cof Table S4 reveals that our mitigation strategies have the expected effects on embedding magni- tudes even under adaptive optimization: cut-initialization ensures small norms [|| = 2.1) while GradScale allows them to grow (||lI = 175) without affecting convergence.
Open Question 6.2. Are there SSL training schemes in which the embedding effect is beneficial?
7. Discussion
In this work, we investigated a fundamental aspect of co- sine similarity-based self-supervised learning: embedding norms serve dual role, both inversely scaling gradients and encoding model certainty. These characteristics to standard SSL models and our analysis showed how they alter the expected training dynamics in both standard and hlass-imbllanced settings.
intrinsic
Takeaways These results make it clear that the embed- ding norm effect impacts SSL training- laaiing-partiinnn particularly in non-contrastive models and can be mitigated using our proposed strategies. The effect appears most detrimental in class-imbalanced settings, aligning with our results on SSL confidence: imbalanced data creates variance in embedding norms, destabilizing training. Nonetheless, there remain questions which are beyond the scope of this work:
Given that these properties are natively present in widely- used SSL approaches, they suggest several research direc- tions beyond those already stated in the paper. First, em- bedding norms could serve as simple yet effective reliability metrics during inference. Second, the embedding norms provide new lens for studying the modality gap in multi- modal representation learning (Liang et al., 2022). Lastly, the interplay between the embedding norm't 's roles - as both confidence metric and gradient scalar - suggests training pipelines that explicitly leverage both mechanisms.
Open Question 6.1. Why are non non-contrastive architectures more sensitive to the embedding-ne effect?

---

On the Importance of Embedding Norms in Self-Supervised Learning
Impact Statement
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, TT., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
The goal of our paper is to advance the field of machine learning. We do not see any potential societal consequences of our work that would be worth specifically highlighting,
Acknowledgements
Draganov, A.,
Vadgama, S., and Bekkers, J. The hidden pitfalls of the cosine similarity loss. ICML Workshop on High- Biggh-Dimessional Learning Dynamics, 2024.
Andrew Draganov is partially supported by the Independent Research Fund Denmark (DFF) under a Sapere Aude Re- search Leader grant No 1051-00106B, by a StiboFund IT travel grant for PhD students and by project W2/W3-108 Impuls und Vemnezzungsfonds der Helmhhltz--Gemeinshhat Sharvarce Vadgama is supported by the Hybrid Intelligence Center, 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research (NWO). This work was partially funded by the Gemeinnützige Cyber Valley Research Fund (D 30.33.88739 and the National Institutes of Health (UM ). The content is solely the responsibility of the authors and does not necessarily rep- resent the official views of the National Institutes of Health. Dmitry Kobak is member of the Germany's 's Excellence cluster 2064 "Machine Learning - New Perspectives for Science" (EXC 390727645). The authors thank the Interna- tional Max Planck Research School for Intelligent Systems (IMPRS-IS for supporting Jan Niklas Böhm.
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6824-6835, 2021.
Girdhar, R., El-Nouby, A., Liu, Z.,, Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Stiftunn, the
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent- new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271- 21284, 2020.
HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, Provable guarantees for self-supervised deep learning with spec- tral contrastive loss. Advances in Neural Information Processing Systems, 34:500005011 2021.
References
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.072 07258, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Caron, M., Touvron, H., Misra, I.. Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729- 9738, 2020.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020a.
Henaff, O. Data-efficient image recognition with contrastive predictive coding. In International Conference on Mu- chine Learning pp. 4182-4192. PMLR, 2020.
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parckh, Z., Pham, H., Le, Q., Sung. Y.-H., Li, Z., and Duerig, T. Scaling up visual and vasion-language representation learning with noisy text supervision. In International Conference Machine Learning, pp. 4904-4916. PMLR, 2021.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv preprint ar0iv:2003.00 04297, 2020b.
Jing, L., Vincent, P., LeCun, Y., and Tian, Y. Understand- ing dimensional collapse in contrastive supf-supe learning. arXiv preprint arXiv:2/ 2021.
10

---

On the Importance of Embedding Norms in Self-Supervised Learning
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Pan, Y., Yao, T., Li, Y., Wang, Y., Ngo, C.-W., and Mei, T. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2239- 2247, 2019.
Kirchhof, M., Roth, K., Akata, Z., and Kasneci, E. non- isotropic probabilistic take on proxy proxy-based deep metric learning. In European Conference on Computer Vision, pp. 435 454. Springer, 2022.
Peng. Z., Dong, L.,, Bao, H., Ye, Q., and Wei, F. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint 2tt::2200.006666 2022.
Kirchhof, M., Kasneci, E., and Oh, S. Probabilisticc con- trastive learning recovers the correct alcatoric uncertainty of ambiguous inputs. In International Conference on Machine Learning, pp.
Peterson, J. C., Battleday, R. M., Griffiths, L., and Rus- sakovsky, O. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision pp. 9617-9626, 2019.
PMLR, 2023a.
Kirchhof, M., Mucsányi, B., Oh, S. J., and Kasneci, D. E. Url: A representation learning benchmark for transferable uncertainty estimates. Advances in Neural Information 2023b.
Processing Systems,
PyTorch Foundation. Pytorch default image trans- former, 2025. URL https://gittub..em com/ husgiinnfaceeppproorc----n----oo
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 23/N, 7(7):3, 2015.
transfformer.pyt...0
Lee, G., Eom, C., Lee, W., Park, H., and Ham, B. Bi- directional contrastive learning for domain adaptive se- mantic segmentation. In European Conference on Com- puter Vision, pp. 38-55. Springer, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8766 PMLR, 2021.
Liang V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022.
Richemond, P. H., Tam, A., Tang, Y., Strub, F., Piot, B., and Hill, F. The edge of orthogonality: simple view of what makes byol tick. In International Conference on Machine Learning, pp. 29063-29081. PMLR, 2023.
Liu, H., HaoChen, J. Z., Gaidon, A., and Ma, T. Self- supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2022.
Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., and Khan- deparkar, H. A theoretical analysis of contrastive unsu- pervised representation learning. In International Con- ference on Machine Learning pp. 5628-5637. PMI.R, 2019.
Misra, I. and Maaten, L. v. d. Self-supervised learning of pretext-invariant representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Saunshi, N., Ash, J., Goel, S., Misra, D., Zhang, C., Arora, S., Kakade, S., and Krishnamurthy, A. Understanding contrastive learning requires incorporating inductive bi- ases. In International Conference on Machine Learning, pp. 19250-19286. PMIR, 2022.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In Indian Con- ference Computer Vision, Graphics and Image Pro- cessing, Dec 2008
Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815-823, 2015.
Oord, A. d., Li, Y., and Vinyals, O. Representation leamn ing with contrastive predictive coding. arXiv preprint 2018
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V. Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Scott, T. R., Gallagher, A. C., and Mozer, M. C. von mises- fisher loss: An exploration of embedding geometries for supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 10612- 10622, 2021.

---

On the Importance of Embedding Norms in Self-Supervised Learning
Smith, I., Ortmann, J., Abbas-Aghababazadeh F., Smirnov, P., and Haibe-Kains B. On the distribution of cosine similarity with application to biology. arXiv preprint mrXhv:2310.13994, 2023.
Zhong, Z.,, Cui, J., Liu, S., and Jia, Improving calibra- tion for long-tailed recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16489-16498, 2021.
Tian, Y. Chen, X., and Ganguli, S. Understanding self- supervised learning dynamics without contrastive pairs. In International Conference Machine Learning, pp. 10268-10278. PMLR, 2021a.
Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive learning inverts the data generating process. In International Conference Machine Learning, pp.
PMLR, 2021.
Tian, Y., Henaff, O. J., and Van den Oord, Divide and
contrast: Self-supervised learning from uncurated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. (W063-10074, 2021b.
Van Assel, and Balestriero,
graph matching ap- proach balanced data sub-sampling for self-supervised learning. In NeurlPS 2024 Workshop: Supervied Cearning-Theon and Practice, 2024.
Wang, F., Xiang, X., Cheng, J., and Yuille, Normface: L2 hypersphere embedding for face verification. Pro- ceedings of the 25th ACM International Conference Multimedia, MM 17. ACM, October 2017.
Wang, T. and Isola, Understanding contrastive represen- tation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020.
Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint 10. 2021.
Xu, H., Xie, S., Tan, X. E., Huang, P.-.,, Howes, R., Sharma, V., Li, S. Ghosh, G., Zettlemoyer, L., and Feicht- enhofer, C. Demystifying clip data. arXiv preprint 1oXiv:2309.16677 2023.
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. Florence: A new foundation model for computer vision. arXiv preprint 1rXiv:2111.1144 2021.
Zhang, C., Zhang, K., Zhang, C., Pham, X., Yoo, C. D., and Kweon, S. How does simsiam avoid col- lapse without negative samples? unified understanding with sulf-supe contrastive learning. arXiv preprint arXiv:2203. 16262, 2022.
Zhang, D., Li, Y., and Zhang, 7. Deep metric learning with spherical embedding. Advances in Neural Information Processing Systems, 33:18772-18783, 2020.
Zhang, X., Yu, F. X., Karaman, S., Zhang, W., and Chang, S.-F. Heated-up up softmax embedding. arXiv preprint arXiv: 1809: 04109, 2018.
12

---

On the Importance of Embedding Norms in Self-Supervieed Learning
A. Proofs
A.1. Proof of Proposition 3.1
Proposition 3.1. [Prop. 3 in Zhang et al. (2020);² proof in A.1J Let be set of points in IR and let 2, and 2j be a positive pair Z. Let Øiz be the angle between 2; and 2y. Then the gradient of C2j (Z) with respect to 2;
244 |Zs
2j
2g ||
la
|
1ey
where a h is the component of orthogonal b. This has magniude
||A
sint Φx
Proof. We are taking the gradient of LA² a function of 2. The principal idea is that the gradient has term with direction 2j and term with direction -2, We then disassemble the vector with direction 2j into its component parallel to 2 and its component orthogonal to 2;. In doing so, we find that the two terms with direction cancel, leaving only the one with direction orthogonal to
Writing it out fully, we have C = -2, 2 24 ||2]]]).
Taking the gradient amounts to using the quotient rule, with 2 2j- Taking the derivative of each, we have
f - 2j and g = || || / 2 21
f -Zj
g' = ||
2i ||زح||
Zi
2 2i
||2
l'g Js
(z) ||
|||z II) +
||>,|]
|
2 2g)
g²
|| 1 ||
-2
2,2 2j
||a||]
| 1313 1
where we use boldface emphasize which direction each term acts along We now substitute cos()r)) 2, 2// (11=/ II)) in the second term get
f'g g'f
-2g
2, cos(Ø)
+
(5)
g²
It remains to separate the first term into its sine and cosine components and perform the resulting cancellations. To do this, we take the projection of ê 2,/ ///lz onto z; and onto the plane orthogonal to z,, The projection of ε onto is given by
2
COs dij
||
while the projection of "fz/|==]]] onto the plane orthogonal to z; is
2
Zj
I
2
|z
is easy to assert that these components sum to 2 by replacing the cos diy by
ZZhang et al. (2020) also showed corresponding results under SGD with momentum and Adam optimization (Kingma, 2014).
13

---

On the Importance of Embedding Norms in Self-Supervised Learning
6
2
Figure Cosine similarity with respect the direction indicated by the blue line. Three circles radii 0.5, 1, and are superimposed to show that, for higher norms, the cosine similarity is less steep. Left: 3D Surface plot, right: 2D topview plot.
We plug these into Eq. 5 and cancel the first and third term to arrive the desired value:
I'g 58
COS Φ
g2
||
||
2iZ4
Zj
I
||=
|2
2
||
2, cos( (e)
|| 12
-1
(I
2
2j
||2/||
|| ||
||>
We visualize the loss landscape of the cosine similarity function in Figure S1.
A.2. InfoNCE Gradients
Proposition 3.2. [Proof in A.2] Let Z be of points in Rd, 2; and zj be positive pair in Z, and V.A be as Prop. 3.1.
Then the gradient of Lig (Z) with respect to 2, is
ExpSim (2,, 2k)
V.
24
(2)
2
S
Let 21, 1 4 i, be a sample in the denominator of L (Z), then the gradient of (Z) with respect to 2 is
1
ExpSim 21)
V.
(2x) (24)
(3)
1z
S
14

---

On the Importance of Embedding Norms in Sel--Supervied Learning
Proof. We are interested in the gradient of L. with respect z4. By the chain rule, we get
Enzi ExpSim 2i, 2x)
Ucc
Enze ExpSim (2,, 2k)
3
c6
Eizi ExpSim
21, 2k)
82.
S
It remains substitute the result of Prop. 3.1 for à We sum this with the gradients of the attractive term obtain the full InfoNCE gradient for positive sample. Similarly, for we obtain
2 Ak
/dz..
=
2 21
V
V R
ExpSim 2)
12 11
ExpSim (2,, 21)
(2:)
(6)
Exye ExpSim(2;; 2k)
02
S |2|
where the last equality follows again from Prop. 3.1.
We note that the gradients of the InfoNCE loss positive and negative samples are always tangent to the respective points.
A.3. Proof of Corollary 3.3
Proof. First, consider that we applied the cosine similarity's gradients from Proposition 3.1. Since 2; and (zz) 1s are orthogonal, | = ||> + | ==, The second term is positive if sin Φij > 0, which is the case save for identical or antipodal points, because then, we have dig E (0,
The same exact argument holds for the InfoNCE gradients. The gradient is orthogonal to the embedding, step of gradient descent can only increase the embedding's 's magnitude.
A.4. Proof of Theorem 3.4
We first restate the theorem:
Let 2; and zj be positive embeddings with equal norm, i.e. ||>:|| = || = p. Let z! and z be the embeddings after step of gradient descent with learning rate Then the change in cosine similarity is bounded from above by:
Y sin dxj
7 cos ø
&T - 2T <
-
Ø2
/2
We now proceed to the proof:
Proof. Let 2; and 2y be two embeddings with equal norm", i.e. || = ||> = p. We then perform step of gradient descent v
to maximize 2 2z. That is, using the gradients 3.1 and learning rate 7, we obtain new embeddings x/ = z; +
| (2,) 1
and o/} = 2, + |=
7 (2:) (z)) Going forward, we write d.j- ($)) and o5: = (Z) so 2 = 24 + 7 day and 2/$ 2j 2
Notice that since 2; and 61y are orthogonal, by the Pythagorean theorem we have | || 11? 2 |1 112 z || Lastly,
we define p' ||>/|| |zE|l-
*We assume the Fuclidean distance for calculations.
15

---

On the Importance of Embedding Norms in Self-Supervised Learning
We interested in analyzing 2 23
2, 2z.
To this end, begin by re-framing #/=
T
+
2 +
2 2)
p
P
2,
85
33 dūj
0
2i
2j
+
7
7
+
7
p²2
p
p
p²²
We now consider that, since dig is the projection of 2, onto the subspace orthogonal 24. we have that the angle between 2; and ōz 17/2 Øig. Plugging this in and simplifying, obtain
2: 6,z
| |1öh||l cos cos(11/2 o6g)
|=-|| |ōzzر|| sin dij
= psin" Φ55-
By symmetry, the same must hold for 2] 2...
Similarly, we notice that the angle this between dig and dzs is Vas = π - Dig- The reason for this is that we must have a quadrilateral whose four internal angles must sum to 2nr, i.e. this + Øis + 2 2m. Thus, we obtain 65 =
[|6t,|] |I ||ó (o)( sin? 2 Øi cos dij.
We plug these back into our equation for 2/ 8/ and simplify:
2J JJs
TT
2) +
72
p²2
P
p osin
2 Oij
psin 0i
sin 2 Diy cos Øij
2/ +
p²2
x²
1
2g 2j + 27 sin? Øi
T
72
sin
2
Φi
cos Øz
p²z
We now consider the original term in question:
Φis y2 sin²
dey cos dis
2. Zj
2 -
23
2, 2j 2 sin²
p²
p²
p²
1
7² sin? tis cos dis
24 Zg
$
2s 23 + 2 sin² Øij
p²
D2
1
sin? Øig cos dij
27 sin?
Φij 7?
1
22
7 sin" 2 dij
7.00s Ø1j
2
Ø²
27 sin? dig
p2
This concludes the proof.
B. Simulations
B.I. Nonparametric Simulations
For the simulations in Section 4.1, we produced two datasets, X, and X2, independently by randomly sampling points
in 20 from standard normal distribution and normalizing them to the hypersphere. The i-th point in dataset X, the
16

---

On the Importance of Embedding Norms in Self-Supervised Learning
(a)
(b)
Figure S2. Left: A depiction of 8 latent classes in 3D obtained via the description in Section B.2. Dashed lines represent vectors from the origin to the mean the distribution. Right: A depiction the learned latent space (unnormalized) using the supervised InfoNCE loss after 50 epochs of training.
positive counterpart for the i-th point in dataset X2- The first dataset is then set to be static while the second is modified in order control for the embedding norms and angles between positive pairs.
We optimize the cosine similarity by performing standard gradient descent the embeddings themselves with learning rate 10. We consider a dataset "converged"" when the average cosine similarity between positive pairs exceeds 0.999.
Controlling for angles. In order to control for the angle between positive pairs, we use an interpolation value a E [-1, Let #1 be static embedding in X; and 22 be the embedding in X₂ whose angle wish to control. In expectation, 6(11, 22) will be 2/2. We therefore define the embedding 22 whose angle has been controlled as
T2 (1 |az)) + x1 a. 2/2
Specifically, the mean angle among positive pairs is controlled using the a parameter. In essence, when a = 0, 2/2 D2. However, when a (resp. a 1), 2²2 = 21 (resp. x/2 = -I))
Controlling for embedding norms. the desired value.
This setting is simpler than the angles between positive pairs. We simply scale X, by
B.2. Parametric Simulations
We restate the entire implementation for the simulations in Section 4.2 for completeness. We choose centers for 4 latent classes {c1, C2. C3, } uniformly at random from g10 by randomly sampling vectors from a standard multivariate normal distribution and normalizing them to the hypersphere. We then obtain the latent samples around center e via z Nc, 0.1 I) and re-normalizing to the hypersphere. For each center, we produce 1000latent samples; these constitute such latent classes (in dimensions) in Figure S2a. We finally obtain the
our latent classes. We depict an example dataset by generating random matrix in ig 11x64 and applying it to the latent samples.
We train the InfoNCE loss via 2-layer feedforward neural network with the ReLU activation function in the hidden layer. The network's 's output dimensionality is R so that, after normalization, it can reconstruct the original latent classes. We train the network using the supervised InfoNCE loss with a batch size of 128. Each data point's 's positive pair is simply another data point from the same latent class.
We visualize the learned (unnormalized) embedding space Figure $2b.
17

---

On the Importance of Embedding Norms in Self-Supervieed Learning
C. Further Discussion and Experiments
C.1. Experimental Setup
Unless otherwise stated, we use ResNet-50 backbone (He et al., 2016) and the default settings outlined in the SimCLR (Chen et al., 2020a) and SimSiam (Chen & He, 2021) papers. We use 1e-6 as the default SimCLR weight decay and 5e-4 the default SimSiam one. When running on Cifar-10 and Cifar-100 100, we amend the backbone network's first layer as detailed in Chen al. (2020a). We embedding dimensionality 256 in SimCLR and 2048 in SimSiam. When reporting embedding norms, we use the projector's output in SimCLR and the predictor's output in SimSiam: these are the spaces where the loss function is applied and therefore where our theory holds.
Due to computational constraints, we run with batch-size 256 in SimCLR. Although each batch is still 256 samples in SimSiam, we simulate larger batch sizes using gradient accumulation. Thus, our default batch-size for SimSiam is 1024, Our base learning rate 10-epoch linear warmup followed by cosine scaling. For all cases, we use 200 for the k-NN classifier and apply on the normalized embeddings.
0.18 BatchSize /256 for SimCLR and 0.12 BatchSize 256 for SimSiam. We employ a
C.2. Opposite-Haue Halves Effects
We devote this section of the Appendix studying the role of the angle between positive samples on the cosine similarity's convergence under gradient descent. Referring back to Figure 2a, we see that the effect is most impactful when the angle between positive embeddings is close to π, i.e. Qij unlikely for a single pair of embeddings in high- -igendimensional space:
for e + O. The following result shows that this is exceedingly
Proposition C.1. Let Ty ~ NO, I) be d-dimensional, i...d. random variables and let 0 < < 1. Then
P 2 1-
2d(1 - 8)2
Proof. By Smith al. (2023), the cosine similarity between two i.i.d. random variables drawn from (0, has expected value μ 0 and variance o² 1/d, where d is the dimensionality of the space. We therefore plug these into Chebyshev's inequality:
Pr
2
ko
5
|
12
x
k
Pr
|
|
1
We now choose ls vd(1 =), giving us
2 Dz
1
P
1
S
|
||
|x>]]
d(1 - E)2
It remains to remove the absolute values around the cosine similarity. Since the cosine similarity is symmetric around 0, the likelihood that its absolute value exceeds E is twice the likelihood that its value exceeds 1 - E, concluding the proof.
We note that this is actually an extremely optimistic bound since we have not taken into account the fact that the maximum of the cosine similarity is 1.
The above proposition represents the likelihood that one pair of embeddings has large angle between them. is therefore exponentially unlikely for every pair of embeddings in a dataset to have angle close π, would require Proposition C. to hold across every pair of embeddings. Thus, the opposite- effect is exceedingly unlikely to occur.
In accordance with this, Table SI shows that, after one epoch of training, positive pairs of embeddings have angle greater than x/2 at a rate of around 5% and 25% for SimCLR and SimSiam/B YOL, respectively. So even if the 'strongest' variant of the opposite-hale -alves effect is not occurring, a weaker one may still be. However, very early into training (epoch 16), every method has a rate of effectively 0 for the opposite-halwee halves effect. Furthermore, the rates Table S1 measure how
18

---

On the Importance of Embedding Norms in Self-Supervised Learning
Table S1. The at which embeddings are on opposite sides of the latent space (angle between positive pair is greater than x/2) for various datasets and SSL models.
Table S2. kNN accuracies for SimSiam trained with various batch sizes. We performed training for both default and cut-initialized variants and reported KNN accuracies at 100 and 500 epochs.
Effect Rate
Effect Rate Epoch 16
Model
Dataset
Epoch
Batch Size
SimCLR
Imagenet-10e
2%
0%
Epoch
256
512
1024
Cifar- 100
11%
1%
Default
46.1
41.2
32.6
SimSiam
Imagenet-100
26%
1
100
Cut (c = 9)
43.1
46.5
44.3
Cifar-100
21%
0%
Default
59.1
60.4
61.3
BYOL
Imagenet-100 100
28%
1
500
Cut (c 9)
59.4
58.9
61.5
Cifar-100
20%
0%
Table $3. Linear probe accuracies epoch 500 for default, cut- initialized and GradScale training on subset of our image datasets.
Table S4. Final embedding norms epoch 100 for SimCLR trained with Adam optimizer on Cifar-10. demon- strating the effect of different mitigation strategies on embedding magnitude.
Cifar-100 Tiny Imagenet
Default
59.8
41.9
SimCLR
Cut (c = 3)
63.2
42.8
Method
Embedding Norm
GradScale
62.2
43.2
Default 81.0
SimSiam
Default
63.7
Cut GradScale
2.1
Cut (c = 9)
64.2
174.8
often dij > 5. This is the absolute weakest version of the opposite-halves effect. Thus, while some weak variant of the aaposite-halves effect may occur the beginning of training, it does not have strong impact the convergence dynamics and, in either case, disappears quite quickly.
C.3. Weight Decay
We evaluate the effect of weight decay in the imbalanced setting in Figure S3, which is an analog of Figure 5 for the imbalanced Cifar-10 10 dataset detailed in Section 6. We again see that using weight decay controls for the embedding norms and improves the convergence of both models. In correspondence with the other results on imbalanced training, we find that stronger control over the embedding norms leads to improved convergence: the high weight decay value does not perform as poorly on SimCLR as in Figure and, on SimSiam, outperforms the other weight decay options.
C.4. Cut-Initialization
We plot the effect of the cut constant on the embedding norms and accuracies over training in Figure S4. To make the effect more apparent, we use weight-decay À 5e in all models. We see that dividing the network's weights by > 1 leads to immediate convergence improvements all models. Furthermore, this effect degrades gracefully: as >> becomes < 1, the embeddings stay large for longer and, as a result, the convergence is slower. We also see that cut-initialization has more pronounced effect in attraction-only models trend that remains consistent throughout the experiments.
We also show the relationship between cut-initialization and the network's 's batch size on SimSiam in Table S2. Consistent with the literature, we see that training with large batches provides improvements to training accuracy. However, we note that larger batch sizes also significantly slow down convergence. However, cut-initialization seems to counteract this and accelerate convergence accordingly. Thus, training with cut- out-initialization and large batches seems to be the most effective method for SSL training (at least in the non-contrastive setting).
19

---

On the Importance of Embedding Norms in Sel--Supervied Learning
SimCLR
SimSiam
SNN accuracy embodding nomn
60
40
20
50-
40-
20-
10
10
10
2-
2
Train Epoch
Train Epoch
No weight decay
Standard weight decay
High weight decay
Figure S3. An analog to Figure 5 performed on the exponentially imbalanced Cifar- 10 dataset. Weight decays are [0. 1e-5, 5e-2] for SimCLR and [0, 5e-1, 5e-2] for SimSiam. We plot the effective learning rate the bottom row. This calculated by scaling learning rate by the inverse of the mean embedding norm.
SimCLR
SimSiam
BYOL
k-nn acc. (%)
80
60
60
60
40
40
40
20
20
20
2
32
128
2
8
32
128
2
8
32
128
Embedding Norm
10a
10a
10a
10a
101
101
10a
10a
10a
2
32
128
32
128
8
32
128
Training epoch
Training epoch
Training epoch
Weights /= 9
Weights /= 3
Default
Weights == 3
Weights
Figure $4. The effect of cut-initialization on Cifar-10 SSL. representttt,,.nnn z-axis and emhedding norm's 's y-axis are log-scale. À = 5 all experiments.
D. More details on gradient scaling layer
An implementation of our GradScale layer can be found in Listing 2 and pseudocode in Algorithm 1. We note that this layer is purely Py PoTch optimization trick and does not amount to implicitly choosing different loss function:
Proposition D.1. Let Rn be unit vector, p I2 {0) - [-1, 2+ tT 2/||²|| the cosine similarity with respect to t. R. and n IR, || Then the vector field Vp has a potential g, i.e. Vq o Vvp, only for 0.
+
Proof. Suppose Vp has potential. Consider two paths with segments $1, and $3, $1 going t + 2t - 22t and 21, where the segments 81, 84 scaling + +2t are straight lines and the other segments 82, 83 follow great
20

---

On the Importance of Embedding Norms in Sel--Supervised Learning
Listing 2. PyTorch code for gradient scaling layer (torch.aut.orra)) Function):
class scale_grad_by_nnt @staticmethod
def forwardictx, power=U)
etx. for backward
etx-power power return @staticmethed
def backward (ctx, grad :
sttxdsaee_teesorss
power ctx.
norm = torch. linalg vector_norm. (2, dim=-1, keepdin=True)
return put norm+*power, None
Algorithm Pytorch-like pseudo-code using the gradient scaling layer
Input: Encoder network model, gradient scaling power
model(batcht
grad scaling toyer.apply(2.p)
sim | T
loss Infooll InfoNCE(sim)
loss bassba))ward)))
circles on 5m-1, By Proposition 3.1, we know that Vp(z) 0 for = E RRy0 a Vp is zero on 81 and 84. Moreover, we have
VVpdz
1/2|" Vpd 2a
Vpdz = 2th (p(2t)
- p(-)t)) 22d2=22(((21)) 2
(8)
and similarly
pdz = 1a 2 2 2.
(9)
Since we assume the existence of a potential, we can use path independence to conclude
Vpdz =f Vvpdz = VVpdz= [Vpd== =2
20+1_
(10)
81.82
13.00
Thus, a = and a does not perform any scaling.
E. Additional figures
We provide bar plot analogous to Figure 3 in Figure S5.
We also show each Cifar-10 10 class's 10 highest and 10 lowest embedding norm samples in Figure S6. These obtained after training default SimCLR on Cifar- 10 for 512 epochs. We see that the high-norm class representatives are prototypical examples of the class while the low-norm representatives are obscure and qualitatively difficult to identify. This property was originally shown by Kirchhof et al. (2022).
21

---

On the Importance of Embedding Norms in Self-Supervised Learning
1.0
0.91
0.8
0.72
0.74
0.65
0.6
0.42
0.43
0.4
0.37
0.2
0.0
Cifar10
Citarto
Cibrl00 Cifa1100
Gitr10
Citar100 Test
Citar100
Citar100 Test
Train
Test
Tain
Test
Train
Train
Test
Train
Test
Train
Trin
Train
SimCLR
SImCLR
SimSiam
Trained on Cifar- 10
Trained on Cifar-100
Figure Bar plot which analogous Figure showing embedding magnitudes on each dataset split function of which dataset the model was trained on, All values are normalized by training set's mean embedding magnitude. Normalized means are represented by black bars, We use the same data augmentations for the train test sets for consistency.
High Norm Exemplars
Low Norm Exemplars
Figure S6. Left: Highest-norm representatives (top 10) per class. Right: Lowest-norm representatives (bottom 10) per class. All from default SimCLR trained on Cifar-10.
22


Reviews:
[{"review_id": "WSIVWctI2G", "reviewer": "Reviewer_FWxF", "summary": "This submission explores the embedding norms interaction with SSL training dynamics, where cosine similarity is commonly used to map the data to a hypersphere. This paper studies the gradients of the cosine similarity loss (and the InfoNCE loss) revealing that while gradients are inversely scaled by the norm of the embedding (large norms cause vanishing gradients), norms after gradient step get larger (causing a vicious cycle). This also results in an the expression for cosine similarity change after one gradient step, showing quadratic slow down in convergence wrt the norm and angle between the pair of embeddings.\n\nThese findings are then validated via a set of simulation experiments by controlling the angle and the norm. Alternatively, the final embedding norms are themselves influenced by optimization of cosine similarity - dense regions get higher norms. This leads to authors making a conclusion (though not formally treated, but empirically demonstrated) that embedding norm is descriptive of the network confidence in the data point, potentially encoding novelty (OOD samples), downstream performance (classification accuracy), human labelers' confidence and agreement.\nManipulating the embedding norm is later studied in the form of three methods (weight decay, cut-initialization and gradscale layer), showing gains over default settings and in imbalanced datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and in-depth review! Below, we address your concerns by adding experiments on Tiny-ImageNet and testing additional models (BYOL, MoCo v3, Dino), which confirm the generality of our theoretical and experimental results. Furthermore, we clarify the novelty of our analysis: while some of our methods and ideas have precedent, we are the first to consolidate them and show unambiguously how the embedding norms interact with self-supervised learning. We believe that knowledge of these interactions is valuable to the ICML community.\n\nDetailed responses are below:\n\n> \"[The paper] could benefit from experiments on larger dataset like Imagenet\"\n\nWe agree that it would be nice to run on ImageNet. However, due to computational constraints, this is infeasible for us. We therefore refer to our response to Reviewer wGR1, where we include experiments on Tiny-Imagenet. The results are consistent with what we see on other datasets.\n\n> \"I wonder if **other methods** also exhibit similar behaviour.\"\n    \nTo address both this and the broader question of our results' generalizability, we have trained BYOL (which optimizes the cos. sim.), MoCo v3 (which optimizes InfoNCE) and Dino (which is not cos.sim.-based) on ImageNet-100 for 100 epochs with and without cut-init. We find that cut-init improves BYOL's accuracy at 100 epochs from 26\\% to 54\\% and MoCo v3's accuracy from 54\\% to 58\\%. These are in accordance with the results in our paper (in particular, Figure S4 from the paper). Our theory only applies to losses with normalized embeddings, thus not to Dino, VicReg or Barlow Twins. Indeed, we find that cut-init drops Dino's accuracy from 44\\% to 29\\%. This implies that trying to resolve the embedding norm effect in settings where it does not occur hurts performance.\nNevertheless, many other prominent SSL methods do rely on normalized embeddings (e.g., W-MSE, NNCLR, ProtoNCE, SWAV, CLIP, DCL, and BEiTv2), highlighting how widely applicable our analysis is. We will discuss this in the revision.\n    \n> \"it is unfair to say there has been no explanation for network confidence via norm\"\n    \nWhile we agree Kirchhof et al. (2022) have discussed the embedding norm as a concentration parameter for the vMF distribution, they only *stated* that it behaves this way and gave brief empirical evidence of this. In fact, Kirchhof et al. reference Scott et al. (2021) who explicitly raise why this holds as an open question: ``Why does the COSINE embedding convey a confidence signal in the norm?'' Our work therefore gives a clear *explanation* for this phenomenon for the first time. Furthermore, Scott et al. was in the supervised setting.\n    \n> \"...weight decay and cut-init are already known methods...\"\n    \nWhile weight-decay is a known method, its standard discussion in the literature is entirely different from ours and focuses on overfitting. We only know of one reference, Wang et al. (2017), which suggests that weight decay may help with the embedding norm effect but this was not tested.\n\nSimilarly, cut-init only exists in transformer architectures and was proposed to manage the attention. We are not aware of it in any convolutional architectures or for managing embedding norms. Thus, the idea of dividing the weights at initialization has not been discussed with regards to the convergence rate and in this sense is a novel contribution of our paper.\n\nBoth of these are complementary to the regularization term for the variance from Zhang et al. (2020).\n\n> \"It is not clear whether GradScale is a novel contribution of this paper or not.\"\n    \nGradScale is a novel contribution, but our goal with it was not to propose a new SOTA method. Instead, we use it as a way to analyze how embedding norms affect SSL training. That is, by using GradScale we can show that removing the gradient's relationship to the embedding norm can improve generalization, especially on imbalanced datasets.\n\n> \"When describing GradScale, p=0 and p=1 are switched.\"\n    \nThe $p$ parameter in GradScale is the power of the embedding norm that gets additionally multiplied to the original gradient. Thus, for $p=0$, the gradient is multiplied by $\\|z\\|^0 = 1$, leaving the gradient untouched.\n\n> \"the authors argue that negative contribution is averaged across many samples and should be smaller than the positive contribution. I'm not sure this is true\"\n    \nWe agree with this point and will remove this from the paper. However, we note that this has no effect on the correctness of Prop 3.2. The main point is that *all* gradients of the InfoNCE loss are orthogonal to the points they act on, thus increasing the embedding points' norm, independent of the sizes of the attractive and repulsive contributions.\n\n-------\n\nWe hope we have addressed your concerns. Please let us know if there is anything we can do to convince you further.", "claims_and_evidence": "For the most part evidence is convincing. However, the 'Embedding Norm as Network Confidence' section is mainly an empirical study (although theoretically grounded via norm size dependency on the sample frequency during training for cosine similarity loss), it could benefit a lot from experiments on larger dataset like Imagenet, since CIFAR datasets are relatively simple and small. While Imagenet has 1K classes, it is possible to take subsets of classes for the purposes of the study (e.g. Figure 3). The paper uses one such subset (Imagenet-100) in Figure 4, so this should not be a problem. Adapting different subsets of Imagenet for experiments on network confidence would help make the results more convincing.\n\nFor the InfoNCE loss proof, the authors argue that negative contribution is averaged across many samples and should be smaller than the positive contribution (cosine similarity loss). I'm not sure this is always true, especially at the start of training and for complex datasets and network architectures and without assuming the network is smooth enough (similar augmented images land closer in latent than negatives).\n\nThe authors also discuss 3 ways in which this submission differs from the related work on embedding normalization for deep metric learning literature, which is very commendable. I agree with the first, however, the rest do not seem to be faithfully validated. The second - evaluating how large embedding norms affect real world training - brings us back to the choice of datasets raised earlier. The third - while weight decay and cut-init are already known methods, it is not entirely clear whether GradScaler is a novel contribution of this paper or not. If yes, there should have been better empirical confirmation of it effectiveness (now it states that the models are sensitive and GradScaler fails to demonstrate gains on larger dataset). In addition, it is not verified/discussed whether addressing embedding norm affect is more beneficial than the other techniques from DML (i.e. regularization term for variance from [1]) or if they are complimentary.", "methods_and_evaluation_criteria": "The formulated experimental setups do make sense, but it would be more convincing if Imagenet-like datasets were used (see Claims and Evidence).", "theoretical_claims": "I looked through the proofs in Appendix they seem alright, but I haven't checked everything very carefully (especially since the results are obtained in previous work [1]).", "experimental_designs_or_analyses": "Yes, the synthetic experiments and confidence experiments description seem valid.", "relation_to_broader_scientific_literature": "The submission addresses current gaps in understanding how self-supervised representation learning (SSRL) methods work, specifically those based on contrastive learning. While there has been a track of works connecting modern SSRL methods, especially SimCLR/Contrastive Predictive Coding to deep metric learning (DML), this work seem to use results obtained in earlier DML literature to describe the training dynamics of common contrastive learning-based methods based on data augmentation and provide ground for connecting embedding norms and network confidence in cosine-similarity-based methods.\n\nWhile earlier discussion on network confidence via norm in the previous literature is discussed in the related work section, it seems unfair to say there has been no explanation for this phenomenon, especially when referencing work that treats norms as concentration parameter in von Mises-Fischer-based models.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Theoretical findings largely depend on (and restate) the results from previous work (especially [1] which they acknowledge). This is not per se a serious weakness, since the authors formalized the quadratic inverse dependence on the norm in Theorem 3.4. However, this diminishes the conceptual contribution of the work.\n\n[1] Zhang, Dingyi, Yingming Li, and Zhongfei Zhang. \"Deep metric learning with spherical embedding.\" Advances in Neural Information Processing Systems 33 (2020): 18772-18783.\n\nSee also earlier sections (Claims and Evidence, Relation to Broader Scientific Literature)", "other_comments_or_suggestions": "When describing GradScaler, it seems p=0 and p=1 are switched. Having scalar to power 0 will eliminate the scaling effect, while p=1 leaves the norm scaling intact, for now it is written the other way around if I'm not confused.\n\nThere is also broken reference to a Table in the supplementary.", "questions_for_authors": "While I understand that SimCLR, SimSiam and BYOL are picked based on their use of cosine similarity, I wonder if other MSE-based methods also exhibit similar behaviour. Have you tried similar experiments of VICReg/W-MSE?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ewINnCjpiR", "reviewer": "Reviewer_5PH5", "summary": "This paper investigated the relatively overlooked area of embedding normalization in self-supervised learning (SSL), as most prior works default to using cosine similarity between embedding vectors -- which normalizes by the product of magnitude of both vectors -- and effectively projects data onto a unit hypersphere. Inspired by some empirical evidence that the pre-normalization embedding norms contain meaningful information, the authors aimed to systematically establish the embedding norm’s rule in SSL. The authors claimed that embedding norms (1) govern SSL convergence rates, and (2) encode network confidence, and validated the claim with theoretical analyses, simulations and empirical results. \n\nThe first main result is on how embedding norms affect convergence. The authors first proved theoretical bounds showing that embedding norms impose a quadratic slowdown on SSL convergence, and further validated in simulation and real experiments, demonstrating the benefit of small embedding norms. Then they showed embedding norms grow during training when cosine similarity is optimized. These two results indicated that effective and efficient training of SSL requires managing the embedding norms, and they provided methods for doing this.\n\nThe second main result is on how embedding norms encode confidence. The authors argue that since the embeddings grow with each gradient update, their norms naturally correspond to the frequency of observed latent features, and this naturally correspond to model confidence. They also provided methods for studying this and validated this claim.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the kind words regarding our paper. We respond to your questions below.\n\n> \"The weakness would be the proposed methods, especially the cut-initialization, are relatively brutal and less innovative\"\n\nRegarding the cut-initialization, we agree with this point and it is something we deliberated on for quite some time. However, we could not come up with an alternative method for ensuring that the embedding norms are small at initialization. An alternative option is to apply cut-initialization at only the final layer, but this performs roughly equivalently. We also note that cut-initialization seems to accomplish the desired task very well, as evidenced by Tables 1-3 in the paper.\n\nIf the reviewer has suggestions as to what we may do as an alternative to cut-initialization, we would be happy to try them out.\n\n> \"Besides k-NN accuracy, have you considered other measures of latent space quality?\"\n\nWe recognize that there are several probes that can be used. We focused on kNN as it is known to lower-bound the other probes [1] and is a good indicator of model performance [2]. To address the reviewer's question, we have run the linear probe on various models at the 500 epochs mark and see the same performance improvements:\n\n|SimCLR|Default|Cut|GradScale|\n|---|---|---|---|\n|Cifar100|59.8|63.2|62.2|\n|Tiny Imagenet|41.9|42.8|43.2|\n\n|SimSiam|Default|Cut|\n|---|---|---|\n|Cifar100|63.7|64.9|\n\nWe will include linear probe evaluations in the revision of the paper.\n\n[1]: Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" TMLR 2023.\n\n[2]: Marks, Markus, et al. \"A closer look at benchmarking self-supervised pre-training with image classification.\" arXiv preprint 2024.\n\n------\n\nPlease let us know if there is anything else which we can address.", "claims_and_evidence": "Yes. The claims are clearly organized and supported by evidence.", "methods_and_evaluation_criteria": "The proposed methods (weight decay, cut-initialization, and GradScale) are reasonable and directly address the identified embedding norm effect. Even though the cut-initialization is a little too simple and brutal, in my opinion the main contribution is the insight rather than the specific novelty of the method itself.\n\nEvaluation criteria, such as k-NN classification accuracy on standard datasets (Cifar-10, Cifar-100, ImageNet-100, Flowers102), appropriately measure SSL representation quality. The methods and evaluation criteria used in the experiments are sensible for studying the stated problems and phenomena.", "theoretical_claims": "I reviewed the correctness of the main theoretical claims (Proposition 3.1, Proposition 3.2 and Theorem 3.4). The proofs seem correct to me, or at least I did not identify issues. But I am not particularly good at proofs so I wouldn't rely on this analysis.", "experimental_designs_or_analyses": "The experimental designs and analyses conducted are sound and valid. I reviewed the experimental setup for results shown in Figures 2-5 and Tables 1-3. The approach is thorough and methodologically sound.", "relation_to_broader_scientific_literature": "The paper builds effectively upon existing literature regarding embedding norms and SSL. It clearly positions itself relative to foundational works such as SimCLR, SimSiam, and prior studies on embedding norms (Wang et al., 2017; Zhang et al., 2020). It provides deeper empirical insights compared to earlier works. The paper is well situated within the broader SSL literature.", "essential_references_not_discussed": "The paper appears comprehensive in its referencing of relevant literature. All critical prior works directly related to embedding norms in SSL seem adequately discussed, and I did not identify any essential missing references.", "other_strengths_and_weaknesses": "Overall, I find this paper to be insightful. In my opinion, the most interesting contribution is that it highlighted how embedding norms affect convergence. Specifically, the authors demonstrated both theoretically and empirically that smaller norms are preferred, and meanwhile related that to the observation that embedding norms generally increase while optimizing for cosine similarity. The conclusion that effective and efficient SSL training relies on managing the embedding norms is very insightful.\n\nThe weakness would be the proposed methods, especially the cut-initialization, are relatively brutal and less innovative. With that said, I find the knowledge and insight brought to the community by this paper to outweigh this weakness.", "other_comments_or_suggestions": "Nothing at the moment.", "questions_for_authors": "Great work. One minor question is, besides k-NN accuracy, have you considered other measures of latent space quality (linear probing and/or finetuning performance for classification)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pO8cNuZEHp", "reviewer": "Reviewer_Jmax", "summary": "The article examines the structure of the gradient expression for the InfoNCE SSL objective. Building on a previous result, this gradient expression is reformulated to emphasize that:\n\n- The gradient involves a projection onto a subspace orthogonal to the embedding vector with respect to which the gradient is computed.\n- The gradient is inversely proportional to the norm of this embedding vector.\n\nThese properties are utilized to derive learning characterizations, such as the continuous growth of embedding vector norms and an upper bound on the improvement of the cosine similarity between positive pairs. These characterizations are  then applied to analyses, including the use of embedding norms for addressing class imbalance and assessing network confidence, as demonstrated through numerical examples.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. Regarding Theorem 3.4's applicability, we appreciate your insights about potential mitigations - weight decay, scaling gradients by embedding norms, and adaptive optimizers. We note that our paper systematically analyzes how the first two affect SSL training in practice. For completeness, we include adaptive optimizer experiments below.\n\nWe now respond to individual comments:\n\n> \"The extension to the broader InfoNCE objective is trivial\"\n    \nWe agree that the extension from cosine similarity to the InfoNCE objective is straightforward. However, we believe that this is a strength of our paper.\n\nThe InfoNCE objective function is used in countless settings (see reply for reviewer FWxF) and our work shows that, in each of these, it has a dependence on the embedding norms. Although a few of these ideas were known in the deep metric learning literature, they had not been experimentally validated or extended to standard contrastive losses. In short, it is relevant for the SSL community to be aware of how the embedding norms interact with the InfoNCE loss function.\n    \n> \"the core analysis reaffirms known results about the cosine-distance gradient\"\n\nQuick clarification: we believe that our core analysis lies in studying the relationship between SSL training and the embedding norms with (i) new theory (Thm 3.2 and 3.4), (ii) new experiments and (iii) new mitigation strategies. Such a comprehensive analysis was previously absent from the literature.\n\n> \"Could we not... use adaptive learning rate rules to prevent the slow down\"\n    \nWe believe the reviewer is implying that the results in Theorem 3.4 do not apply under, for example, the Adam optimizer. If so, we disagree. The gradient under Adam optimization still inversely depends on the embedding norm and, consequently, the embedding norms slow down the training process. To test this, we have trained SimCLR and SimSiam on the Cifar datasets using the Adam optimizer for 100 epochs and find consistent results:\n\n|SimCLR w/ Adam|Default|Cut|GradScale|\n|---|---|---|---|\n|Cifar10|79.6|79.9|80.5|\n|Cifar100|44.3|45.3|46.6|\n\n|SimSiam w/ Adam|Default|Cut|\n|---|---|---|\n|Cifar10|73.7|79.8|\n|Cifar100|36.4|44.9|\n\nWe also note that the Adam optimizer is not common in cos.sim.-based SSL models.\n\n> \"Could we not choose the learning rate proportional to the embedding norm?\"\n    \nWe agree. In fact, this is *precisely* what our GradScale does: it multiplies the gradient on each sample by its embedding's norm. We find this improves training but can lead to instability due to a positive feedback loop which is introduced: the embedding norm grows with the magnitude of the gradient and the gradient is now multiplied by the embedding norm. We include the embedding norms from the SimCLR w/ Adam Cifar10 training runs as an example of how the norms are affected by our mitigation strategies:\n\n|Default|Cut|GradScale|\n|---|---|---|\n|81.0|2.1|174.8|\n\n> \"How do regularization terms in the overall loss (such as weight decay) impact the conclusions drawn?\"\n\nWe politely note that we extensively analyzed how weight-decay improves SSL convergence in Section 6 and the appendix, particularly Table 1 and Figures 5, S3.\n\nAdditionally, weight decay would not change the correctness of Theorem 3.4: each individual step still depends on the square of the embedding norm. Nonetheless, we see the reviewer's point that weight decay will interact with the bound in Theorem 3.4, possibly leading to faster convergence. Practically, however, it is difficult to find a regularization strength large enough to mitigate the slow-down in Theorem 3.4 without training diverging.\n\nTo evaluate this, we have extended the simulation from Section 4.1 by augmenting the objective with a regularization term on the embedding norm to simulate how weight-decay affects convergence. Thus, we now optimize $\\mathcal{L} = -\\hat{z}_i^\\top \\hat{z}_j + \\gamma \\cdot ||z_i||_2$. The results are listed in the tables below. We find that while weight-decay can speed up convergence, the number of required steps still depends quadratically on the initial embedding norm. However, if the weight decay is made too large, convergence never occurs. This is in line with our experiments in Section 6.\n\nFor $\\gamma = 0.5$:\n|Initial Norm|Steps to Converge|\n|---|---|\n|1|64|\n|4|526|\n|7|1080|\n\nFor $\\gamma = 1$:\n|Initial Norm|Steps to Converge|\n|---|---|\n|1|49|\n|4|318|\n|7|610|\n\nFor $\\gamma = 10$, the simulation diverges.\n    \n> \"Would it be possible to constrain embeddings to the sphere?\"\n    \nThis is something we considered: it would mean forcing the outputs onto $\\mathcal{S}^{d-1}$ and may work well with the loss from Koishekenov et al. [1]. We chose to implement the GradScale layer as an alternative as it accomplishes a similar effect in principle.\n\n### References\n\n[1]: Koishekenov, et al. \"Geometric contrastive learning.\" VIPriors Workshop, CVPR 2023.\n\n------\n\nPlease let us know if we have addressed your concerns.", "claims_and_evidence": "- The claims regarding the special structure of the cosine distance gradient—specifically, its projection onto the tangent space and its inverse dependence on the embedding norm—are extensions of previous work and are well-supported by straightforward derivations.\n\n- The claim about the increase in embedding norm and the dependence of cosine angle improvement is analytically grounded in the gradient structure.\n\n- The claims concerning class imbalance and confidence are motivated by the earlier theoretical findings but are primarily supported by numerical experiments.\n\nThese claims are specific to SSL loss functions that involve cosine distance and optimization schemes that use a constant or non-adaptive learning rate.", "methods_and_evaluation_criteria": "The setups for numerical experiments based on synthetic data and classical ML datasets mainly make sense as well as the conclusions drawn from these examples.", "theoretical_claims": "The analytical results are mainly in Section 3 (and their proofs are in Appendix A). These are extensions of the cosine angle gradient result by Zhang et al (2020) and appear to be correct. The probability bound (Proposition C.1) in Appendix C appears to be correct.", "experimental_designs_or_analyses": "The experimental designs seem appropriate, although the presentation clarity for Figure 2b (Section 4.2 ) and Figure 3 (Section 5) needs improvement.", "relation_to_broader_scientific_literature": "The article analytically extends the structural interpretation of the cosine distance gradient and provides simulation-supported insights into SSL training.", "essential_references_not_discussed": "I believe the article offers essential references including  Zhang et al. (2020) which provides the structural form of the  cosine distance gradient, which is extended and further elaborated in this article.", "other_strengths_and_weaknesses": "Strengths:\nThe article provides useful insights into the learning process of cosine-distance-based self-supervised learning (SSL), particularly focusing on embedding-norm normalization and projection onto the tangent space of the embedding vector.\n\nWeaknesses:\nThe main results build upon prior work on the structure of the cosine-distance gradient. Extending this analysis to the entire InfoNCE objective is relatively trivial, as it naturally follows from the use of cosine distance for negative samples.\n\nThe analysis is specific to the use of SGD-based updates for cosine-distance-based SSL with a constant or non-adaptive learning rate. However, in practice, learning rates can be chosen adaptively—for instance, using learning rate rules that scale with the norm of the gradient—thereby eliminating the quadratic slowdown claimed in Theorem 3.4. Moreover, SSL algorithms often incorporate $\\ell_2$ -normalization or regularization on embeddings or projector outputs during training, which fundamentally alters the dynamics of embedding norms.\n\nDespite the fact that  the article provides interesting insights about the impact of embedding-norm and tangent-space projection components of the  cosine-distance based SSL gradient, the overall contribution is weakened by multiple factors. Most notably, the core analysis essentially reaffirms known results about the structure of the cosine-distance gradient, making its extension to the broader InfoNCE objective feel rather trivial. Furthermore, the reliance on a strictly SGD-based framework with a fixed learning rate neglects the practical reality of common SSL training, where adaptive learning rate schedules—and often $\\ell_2$-normalization or related regularization strategies—impact the claimed quadratic slowdown feature. Therefore, while there is value in the article’s focus on embedding-norm dynamics, the limited novelty in analytical extension and practical applicability diminish the strength of the overall contribution.", "other_comments_or_suggestions": "Presentation improvement suggestions: There appears to be various typos:\n\n- Line 802: R^20->R^{20}?\n- Line 895: variables->vectors\n- Line 926: Table ??", "questions_for_authors": "- Based on Equation (3), the authors claim that embedding norms cause a slowdown proportional to their magnitude. In this expression, the learning rate  $\\gamma$ is a tunable algorithm parameterthat can be adjusted to compensate for this effect. Could we not simply choose $\\gamma \\propto \\rho^2$, or use adaptive learning rate rules to prevent the  slow down? Alternatively, would it not be possible to constrain embeddings, to $S^{d-1}$ for example?\n\n- How do regularization terms in the overall loss (such as weight decay) impact the conclusions drawn about the embedding norms based on the special structure of the cosine-distance metric?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "goXRJy6kdO", "reviewer": "Reviewer_wGR1", "summary": "The paper proposes that the norm of the embeddings play an important role that may affect both optimization (convergence) and generalization properties of self-supervised learning methods. The paper makes an analytical observation about how embedding norm can slow down convergence. The paper argues about a model's confidence on seen/unseen data can be linked to embedding norm empirically. The paper uses SimCLR as an example of contrastive method and SimSiam as an example of non-constrastive method in its (empirical) analysis.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the extensive analysis and suggestions for how to improve our work! Among other things, our rebuttal includes experiments on the Tiny-ImageNet dataset, on additional models, additional probes and on how the embedding's rank corresponds to the embedding norm. We respond to individual points below:\n\n> \"I found the lack of analysis/experiments on **ImageNet** a concern.\"\n    \nUnfortunately, due to computational constraints, this is infeasible for us. However, to address this concern, we trained SimCLR for 500 epochs on the Tiny-Imagenet dataset, which contains 200 classes, each with 500 training samples. The results are consistent with our other experiments:\n\n|Probe|Default|Cut|GradScale|\n|---|---|---|---|\n|kNN probe|36|37.7|38.1|\n|Lin. probe|41.9|42.8|43.2|\n\nWe will include these when updating the paper.\n    \n> \"I have seen linear probe... favored in SSL literature\"\n\nWe focused on kNN as it is known to lower-bound the other probes [1] and is a good indicator of model performance [2]. However, to address this concern, we have also run linear probes on several datasets:\n\n**Tiny-ImageNet** linear probe results are presented in the table above.\n\n**CIFAR-100** linear probe results can be found below:\n\n|SimCLR|Default|Cut|GradScale|\n|---|---|---|---|\n|Cifar100|59.8|63.2|62.2|\n\n|SimSiam|Default|Cut|\n|---|---|---|\n|Cifar100|63.7|64.9|\n\nIn both cases, the linear probe results are in line with the kNN ones. We will include linear probe evaluations in the revision.\n    \n> \"using popular SSL methods would make the paper interesting to readers as well as the authors\"\n\nWe agree that adding more methods would be beneficial. To this end, we have run additional experiments on BYOL (resnet backbone; optimizes the cosine similarity), Mocov3 (ViT backbone; optimizes InfoNCE) and Dinov2 (ViT backbone; not cos.sim.-based) on ImageNet-100. The results can be found in the response to Reviewer FWxF and are consistent with our other experiments.\n\n> \"the use of embedding norm makes it harder to interpret (why is 0.74 considered a good indicator of \"OOD\" in Figure 3)\"\n    \nThe absolute embedding norms will differ between models and would be difficult to use directly as a confidence metric. Instead, we propose using the *relative* embedding norms. Put simply, if an embedding norm is smaller than most norms seen on the training set, then the sample is likely OOD. Thus, in Figure 3, we have normalized the values by the training set's mean embedding norm, implying that the expected in-distribution norm is 1. The closer to 0 the value is, the more likely the sample is to be OOD. We will make this more clear in the revision.\n    \n> \"lack of discussion on the uses of embedding rank and relationship to downstream performance\"\n\nWe thank the reviewer for these references, we will include them in the related work section. Although the references address a different aspect of SSL embeddings than we do (they discuss the rank of all the embeddings whereas we discuss the norm of individual embeddings), we agree that these topics are related. Specifically, our work shows that the embedding norms grow in regions of the latent space which have high density (Section 4.2). This implies that these regions may induce large eigenvalues on the covariance matrix or be clusterable as in CLID.\n\nAs a preliminary test, we include below a table which shows the rank of the Cifar10 embedding space over training epochs and the corresponding mean embedding norm. We calculate rank as the number of principal components required to capture 99 percent of the latent space's variance. Interestingly, we find that the rank starts growing at roughly the same epoch where the norms stop growing, implying a potential correlation. We will include this in the discussion section at the end of the paper.\n\n|Epoch|1|2|4|8|16|32|64|128|\n|---|---|---|---|---|---|---|---|---|\n|Rank|23|13|10|8|9|12|17|21|\n|Norm|1.4|2.5|5.5|12.9|24.1|22.7|23.5|22.1|\n\nWe also note that this is likely related to the study on dimensional collapse in SSL methods, such as [3].\n    \n> \"the new interventions need testing at a larger scale\"\n    \nWe emphasize that these interventions were included to study the behavior of the embedding-norm effect in practice and were not intended to obtain SOTA results. However, we agree that further testing would be helpful. To this end, we hope the analysis on additional models (BYOL, MoCov3 and Dino on ImageNet-100) and the additional experiments on Tiny-Imagenet have addressed this concern.\n    \n> Figure 3\n\nWe will amend the figure.\n\n### References\n\n[1]: Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" TMLR 2023.\n\n[2]: Marks, Markus, et al. \"A closer look at benchmarking self-supervised pre-training with image classification.\" arXiv preprint 2024.\n\n[3]: Tian, Yuandong, et al. \"Understanding self-supervised learning dynamics without contrastive pairs.\" ICML 2021.\n\n------\n\nPlease let us know if these have addressed your concerns.", "claims_and_evidence": "Convergence: The paper makes a claim on convergence and supports it via analysis (P 3.1, P3.2, C3.3, T3.4) and uses a toy setting for empirical support. Note that P3.1 is a result from prior work (clearly noted in the paper) while the other proprositions/theorems are new work proposed in the paper (IMO)\n\nSeen/Unseen Data: The paper uses an empirical approach to argue about embedding norm and seen/unseen data. A problem that I see  with the analysis is that the paper uses norm values that aren't clearly interpretable. I find it hard to figure out what constitutes a small norm vs big norm value", "methods_and_evaluation_criteria": "The paper uses a mix of analysis, toy models and small-scale experiments. Datasets include CIFAR-10, CIFAR-100 and ImageNet-100 (a subset of ImageNet). The paper uses SimSiam and SimCLR as stated earlier for as model prototypes. \n\nWhile the above is good, I found the lack of analysis/experiments on ImageNet a big concern. This is especially concerning as the newly proposed methods/interventions need to be tested at least at ImageNet-1K scale as is standard practice in current SSL literature", "theoretical_claims": "The theoretical claims appears to be fine. I checked P3.1, P3.2, C3.3 and T3.4 and read/skimmed proofs in the appendix", "experimental_designs_or_analyses": "- I found the lack of analysis/experiments on ImageNet a big concern. This is especially concerning as the newly proposed methods/interventions need to be tested at least at ImageNet-1K scale as is standard practice in current SSL literature \n\n- Additionally, the use of embedding norm makes it harder for this reader to interpret (why is 0.74 considered a god indicator of \"OOD\" in Figure 3 on the right)\n\n- While kNN classifier is a good probe, I have seen linear probe and more recently attentive probe being favored in SSL literature. This omission is a concern for this reader/reviewer\n\n- While SimCLR and SimSiam are good prototypes for SSL there are other recent methods like I-JEPA, DinoV2 that needs to be considered if the authors are interested in testing non-contrastive methods. While I understand that jumping off point is InfoNCE loss analysis, the authors already consider SimSiam so using popular SSL methods would make the paper interesting to readers as well as the authors.", "relation_to_broader_scientific_literature": "One area that's a significant concern is the lack of discussion on the uses of embedding rank and relationship to downstream performance. See the following works:\n\n- \\alpha-Req: https://openreview.net/forum?id=ii9X4vtZGTZ \n- RankMe: https://arxiv.org/abs/2210.02885\n- LiDAR: https://arxiv.org/abs/2312.04000\n- CLID: https://openreview.net/forum?id=BxdrpnRHNh\n\n(see section on essential references on suggested changes to manuscript)", "essential_references_not_discussed": "One area that's a significant concern is the lack of discussion on the uses of embedding rank and relationship to downstream performance. See the following works:\n\n- \\alpha-Req: https://openreview.net/forum?id=ii9X4vtZGTZ \n- RankMe: https://arxiv.org/abs/2210.02885\n- LiDAR: https://arxiv.org/abs/2312.04000\n- CLID: https://openreview.net/forum?id=BxdrpnRHNh\n\nSpecifically, I believe the authors should check the conclusions CLID paper makes wrt kNN classifiers. Ideally, all of the above papers should be discussed by the authors as they appear to be related work. Rank is different from embedding norm but the fact that rank has been shown to correlate well with downstream performance appears to be a related part WRT generalization.", "other_strengths_and_weaknesses": "- The paper is well written. I really mean this as this has been an enjoyable read\nA weakness I see is that the new interventions (Cut-Initialization and GradScale) needs thorough testing at a larger scale than what the authors used to ensure the observations may be useful to a broader group of readers in SSL", "other_comments_or_suggestions": "- Figure 3 could be improved\n  - (a) the axes would reader better if transposed \n  - (b) the colors make it hard to distinguish classes", "questions_for_authors": "Please see my comments and observations made above. I look forward to authors' response as that may help clarify my questions and/or confusion about the work reported in the paper.\n\n## Updated score to indicate my support for the paper", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZbOPXEDiwK", "reviewer": "Reviewer_8y1U", "summary": "This paper explores the role of embedding norms in self-supervised learning (SSL). It shows that embedding norms are crucial for SSL models in two key aspects: they influence convergence rates and encode model confidence. The study demonstrates that smaller embedding norms are associated with unexpected samples, and manipulating embedding norms can affect training efficiency. The paper includes theoretical analysis alongside simulations and experiments to establish how embedding norms influence the dynamics of SSL training. The results reveal that embedding norms govern how SSL models train and evolve, offering insights into the optimization process and network behavior. The study highlights the importance of controlling these norms for better performance in SSL tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful suggestions and the in-depth review! We include experiments and discussion in response to your questions on the SSL methods used in the paper, the training epochs and the applicability/novelty of our analysis. Responses to individual questions are below:\n\n> \"while the paper makes a significant contribution to understanding the role of embedding norms in SSL, the study is limited to just two SSL methods\"\n    \nWe agree with the reviewer that adding more methods would be beneficial. To address this concern, we have run additional experiments on BYOL (resnet backbone; optimizes the cosine similarity), Mocov3 (ViT backbone; optimizes InfoNCE) and Dino (ViT backbone; not cos.sim.-based) on ImageNet-100. The results can be found in the response to Reviewer FWxF and are in accordance with the results in the paper. Please let us know if this has helped to address the concern.\n\n> \"I am curious whether the same phenomenon would occur with embedding norm in clustering and whitening-based methods, such as SwAV and W-MSE.\"\n\nOur key theoretical contribution (norms grow, gradients decrease by norm) holds generally for loss functions based on normalized embeddings. As a result, it applies to SwAV, as well as the similar ProtoNCE, and W-MSE. The latter two directly use the cosine similarity, so Prop. 3.1 and Thm. 3.4 apply directly. Additional methods which use normalized embeddings for which our results apply include NN-CLR, ProtoNCE, BYOL, CLIP, DCL, and BEITv2, to name a few. We will make this clearer when revising the paper and will include a formalization of the general statement (that our findings hold for loss functions which depend on normalized embeddings).\n\n> \"Methods like Barlow Twins[1] and VICReg[2] do not adopt Embedding Norm; incorporating it leads to collapse in these methods.\"\n\nWe agree that this is an interesting direction for future work. As the reviewer points out, several SSL models perform worse when the embeddings are normalized. However, our analysis does not apply to these methods and we are not able to test why Barlow Twins and VICREG fail in the normalized setting in the limited rebuttal period. We will clarify this during the revision.\n\n> \"theorem 3.4 is unrealistic in my opinion, as it only involves the cosine similarity of positive pairs and neglects the interactions of negative samples\"; \"the theoretical contributions, while useful, lack novel insights\"\n\nWe maintain that theorem 3.4 is a novel theoretical contribution and note that it holds immediately for the non-contrastive models SimSiam and BYOL, both of which only optimize the cosine similarity between positive samples. While it does not directly apply to the InfoNCE loss, we believe that our extensive analysis in Section 6 describes how embedding norms and convergence interact in contrastive settings. We also refer to our rebuttal to reviewer Jmax, where we extended the simulations in Section 4.1 to further analyze how weight-decay affects convergence.\n\n> \"The lack of original theoretical contributions (such as new theorems or proofs) limits the novelty of the paper’s theoretical aspect.\"\n\nWe recognize that our theoretical contributions build upon existing work in deep metric learning. However, our paper's novelty lies not in creating new mathematical frameworks, but in bringing theory and observations together and showing that they are causally related. The theory about gradient norms had not previously been experimentally verified and the experimental observations about SSL confidence did not have a theoretic underpinning. Our theory, simulations and experiments were designed to validate and explore these connections.\n\n> \"the training duration used in the experiments (128–256 epochs) is relatively short\"\n\nTo address this concern, we have run on Cifar10/100 for 1000 epochs and report the kNN accuracies below:\n\n|SimCLR|Default|Cut|GradScale|\n|---|---|---|---|\n|Cifar10|87.7|88.2|88.2|\n|Cifar100|56.6|60.1|58.2|\n\n|SimSiam|Default|Cut|\n|---|---|---|\n|Cifar10|88.1|88.4|\n|Cifar100|61.8|62.6|\n\nThe performance improvements of cut-initialization and GradScale persist after 1000 epoch training and even grow in the case of Cifar100. We thank the reviewer for the suggestion.\n\nThe ImageNet experiments in Tables 2 and 3 were already run for the requested 500 epochs.\n\n------\n\nWe hope we've addressed your concerns but please let us know if there is something else we can do to convince you further.", "claims_and_evidence": "The claims in this paper are generally supported by evidence, but the study's limited scope—focused on just a few SSL methods (SimCLR and SimSiam)—means that these findings may not universally apply to other SSL methods. This is not explicitly stated in the paper, which could lead to questions about the generalizability of the results. Therefore, while the empirical evidence is convincing, a broader range of methods and further analysis could provide stronger and more universally applicable results.", "methods_and_evaluation_criteria": "The paper uses both theoretical analysis and empirical experiments, which are appropriate for exploring the relationship between embedding norms and SSL dynamics. The theoretical component provides mathematical bounds and insights into how norms influence convergence rates and gradient magnitudes, while the empirical experiments validate these claims under practical conditions.", "theoretical_claims": "I have checked the correctness of proofs in this paper. However, most theoretical analyses in this paper build on prior work.\n\nIn addition, the theorem 3.4 is easy but unrealistic in my opinion, as it only involves the cosine similarity of positive pairs and neglects the interactions of negative samples which are the important part to avoid feature collapse.\n\nAs a result, the proofs lacks theoretical novelty.", "experimental_designs_or_analyses": "1. The experiments are conducted on CIFAR-10, CIFAR-100, and ImageNet-100, which are widely accepted SSL benchmarks. The models are trained for 128–256 epochs, with SimCLR and SimSiam used as the primary SSL models. However, standard benchmarks[1] require 1000 epochs on CIFAR-10/100 and 400 epochs on ImageNet-100 for proper convergence.The limited training time raises concerns about whether the observed effects persist in fully trained models.\n\n2. The paper only evaluates a few SSL methods, specifically SimCLR and SimSiam, while omitting other prominent frameworks like Barlow Twins, VICReg, SwAV, and DINO. Since Barlow Twins and VICReg do not use embedding norms and actually experience collapse when norms are introduced, it is crucial to analyze their behavior to confirm whether the findings hold across SSL paradigms. The lack of diversity in SSL models weakens the generalizability of the study.\n\n[1] solo-learn: A library of self-supervised methods for visual representation learning.", "relation_to_broader_scientific_literature": "The contributions of this paper are rooted in and extend the broader literature on embedding norms in SSL. By mainly offering new empirical results, and practical techniques for manipulating embedding norms, the paper provides contributions to the understanding of SSL dynamics and introduces methods to improve the training and performance of SSL models. The work builds on and extends prior findings while addressing gaps in the literature related to convergence speed, confidence encoding, and embedding norm manipulation.", "essential_references_not_discussed": "The author emphasizes the importance of Embedding Norm in self-supervised learning. However, it is worth noting that methods like Barlow Twins[1] and VICReg[2] do not adopt Embedding Norm; rather, incorporating it leads to collapse in these methods. The author should delve deeper into analyzing these phenomena.\n\n[1] Self-supervised learning via redundancy reduction. In ICML, 2021.\n\n[2] Vicreg: Varianceinvariance-covariance regularization for self-supervised learning. In ICLR, 2022.", "other_strengths_and_weaknesses": "Significance:\n\nThe insights into embedding norm manipulation could have important implications for optimizing SSL models, especially in terms of convergence efficiency. By providing methods like cut-initialization and weight decay to control norms, the paper offers practical techniques that could be applied to a wide range of self-supervised methods to improve training speed and stability. This is a significant contribution to the SSL field, as improving training dynamics remains a crucial challenge in modern machine learning.\n\nWeaknesses:\n1. Limited Scope of Methods: while the paper makes a significant contribution to understanding the role of embedding norms in SSL, the study is limited to just two SSL methods: SimCLR and SimSiam. These models represent only a small subset of the diverse family of SSL techniques. The paper does not discuss or evaluate methods like Barlow Twins, VICReg, or DINO, which could have provided a more comprehensive analysis. Including these methods would have made the findings more generalizable and would have strengthened the argument that the observed effects of embedding norms are consistent across different SSL paradigms.\n\n2. Theoretical Depth: the theoretical contributions, while useful, lack novel insights and mostly build on prior work, especially in terms of embedding norm behavior. The mathematical analysis of how norms affect convergence rates and gradient scaling is solid, but the lack of original theoretical contributions (such as new theorems or proofs) limits the novelty of the paper’s theoretical aspect. The paper could have been more impactful if it had introduced new theoretical findings that directly contribute to advancing the understanding of norm manipulation in SSL.\n\n3. Training Epochs: the training duration used in the experiments (128–256 epochs) is relatively short, especially for datasets like CIFAR-10/100 and ImageNet-100, which typically require much longer training times (e.g., 1000 epochs for CIFAR-10/100). This raises concerns about whether the effects of embedding norm manipulation hold during the full training process or if they only manifest after longer periods of training. A more extended evaluation would provide more robust evidence of the claims, particularly regarding long-term stability and model performance.", "other_comments_or_suggestions": "See Other Strengths And Weaknesses.", "questions_for_authors": "I am curious whether the same phenomenon would occur with embedding norm in clustering and whitening-based methods, such as SwAV and W-MSE.\n\nI would be happy to raise the score if authors' rebuttal can solve my concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates the role of network’s feature norms in some aspects of self-supervised learning specifically for the case of a loss with the cosine similarity metric. It first formally studies the feature norm’s role in learning dynamics and convergence through the lens of its gradient (update steps) and empirically verifies the results for a simulated and controlled setup, arriving at the conclusion that feature norms need be directly treated for improved training dynamics.  Then, it argues that high-norm features could be due to the occurrence frequency and therefore the norm can be a proxy measure for out-of-distribution detection or “confidence” of the model. They thoroughly examine this hypothesis. Finally, they empirically investigate some implications of these two aspects interacting during training and propose and evaluate some mitigations for the adverse effects.

Five expert reviewers evaluated the paper which initially had divided scores; with two on the reject side (ratings 2, 2) and three on the positive side (ratings 3, 3, 4). 
The reviewers appreciated the relevance and importance of the study and (some of) the insights it offers regarding convergence and the connection of SSL feature norm and OoD.	
The main raised issues consist of (1) the reliance on prior theoretical framework and results, (2) the purely hypothetical and empirical nature of some results which are also not examined within a large-scale benchmark (relatively small datasets, short training time, low number of SSL methods), and (3) limited novelty of the proposed mitigation techniques.

The authors provided a thorough rebuttal in which (1) they present additional results covering larger datasets, more SSL methods which are trained with more epochs, (2) they agree with the simplicity of the formal results considering the prior work but argue that the formal connection is new (and important) and that the corresponding empirical results are extensive, consistent and convincing, and (3) proposed mitigations are either entirely novel or used for a novel purpose.

All reviewers attended the rebuttal which led to an increase in rating by two reviewers with the final ratings arriving at 2,3,3,4,4. 

The AC agrees with the reviewers that the study is relevant, of wide interest, and brings some interesting insights and mitigation practices. The AC also agrees that the novelty is limited in a few aspects but despite the limited novelty, the AC finds the message of the paper, on the role of feature norm in training dynamics and OoD of the main family of SSL methods, considerably more solidified than any prior work and worth dissemination. Therefore, the AC agrees with the absolute majority of the reviewers and recommends acceptance.
Decision: Accept (poster)