Exploring Structural Degradation in Dense Representations for Self-supervised Learning
Authors: Siran Dai
Qianqian Xu
Peisong Wen
Yang Liu
Qingming Huang
Venue: NeurIPS 2025 poster ()
Keywords: Self-Supervised Learning
Dense representation learning
Primary area: general_machine_learning

Abstract: In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense prediction tasks (e.g., semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across sixteen state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge.
To tackle this issue, we introduce a Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DSE-based regularization method. Experiments on sixteen SSL methods across four benchmarks confirm that model selection improves mIoU by $3.0\\%$ on average with negligible computational cost. Additionally, DSE regularization consistently mitigates the effects of dense degradation. Code is available at \url{https://github.com/EldercatSAM/SSL-Degradation}.

Exploring Structural Degradation in Dense Representations for Self-supervised Learning
Siran Dai
Qianqian Xu Peisong Wen
Yang Liu Qingming Huang 5,3x
Institute Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences State Key Laboratory of Al Safety, Institute of Computing Technology. CAS
Peng Cheng Laboratory
School Computer Science and Tech., University of Chinese Academy Sciences daisiran0iie ialiran@iie.ac.. ac. cn xuqiangian@ict.eet ac.
(venpeisong, quhuang)Oucas.ccm cn
liuyang2320maile ucas. an
Abstract
In this work, observe counterintuitive phenomenon in self-supervised learn- ing (SSL): longer training may impair the performance of dense prediction tasks (e.g., semantic segmentation). We refer to this phenomenon Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across six- teen state-of-the-ar SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, mea- suring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge. To tackle this issue, we introduce Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensional- ity measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and DSE-based regularization method. Experiments sixteen SSL methods across four benchmarks confirm that model selection improves mloU by 3.0% average with negligible computational cost. Additionally, DSE regulariza- tion consistently mitigates the effects of dense degradation. Code is available at https:// thub. com/ Ett///Eeerratt.M.mem Das.ss/AtttSSS--aaaaaatioo
Introduction
Self-Supervised Learning (SSL) has greatly benefited from advancements in training algorithms, larger datasets and models, and extended training periods, leading significant success in image-level representation learning [10, 16, 3, 29]. However, dense (patch or pixel-level) representation learning remains challenging with only slight improvements [70, 5].
While training models for extended periods to extract high-quality representations has been a common practice in SSL, identify and study counterintuitive phenomenon, named Self-supervised Dense Degradation (SDD), which helps explain the challenges in self-supervised dense representation learning. As illustrated in Fig. 1, although the training loss converges and classification performance steadily improves, the dense performance declines at the later stages of training. Consequently. the final checkpoint exhibits significant performance gap compared to the best performance observed during training.
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Extensive experiments on sixteen state-of satte-of-hhe the art methods across four benchmarks confirm that the SDD phenomenon consistently appears across diverse training approaches and evalu- ation protocols. More importantly, it persists even when training and evaluation are conducted on the same dataset. This demonstrates that SDD highlights the performance inconsistency between different tasks rather than overfitting to the data distribution, introducing a new chal- lenge to the SSL community.
Loss
Cls Accuracy
Seg mloU
306
150
(a) MoCo v3 [16]
(h) DINO [10]
Figure I: The Self-supervised Dense Degradation (SDD) phenomenon. While the training loss con- verges and classification performance gradually increases, the segmentation performance declines and the final checkpoint is not optimal. The met- rics are normalized to 1 for better illustration.
Because of SDD, the common practice of train- ing until convergence results in suboptimal dense performance. Finding metric to predict downstream dense performance thus becomes essential for both understanding the cause of SDD and reducing its negative impact. Typically, models are evaluated however, the cost of evaluating even a single checkpoint can exceed that of one full epoch of pre- training, making this impractical. Moreover, SSL typically lacks access to downstream data labels. Although previous studies have tried estimating SSL downstream performance in an unsupervised manner [2, 24, 69, 68, 36, 60], these metrics mainly focus image-level tasks and are negatively correlated with dense-level performance in our experiments.
labeled validation set;
To guide the development of a better metric, we first provide theoretical analysis based error rate decomposition. We show that the downstream error rate is small if 1) intra-class representation radius is smaller than inter-class distance, and 2) effective dimensionality of representations is large.
Based on this analysis, we propose Dense Representation Structure Estimator (DSE), which consists of measures for class staassseprrability and effective dimensionality, directly corresponding to our theoretical findings. Using the DSE metric, we propose two strategies to counteract SDD. In the off-the-shelf setting, we suggest a simple yet effective method of selecting the checkpoint with the highest local DSE. In the online training setting, we integrate the DSE metric as a regularizer.
Empirical evaluations demonstrate that the proposed DSE accurately predicts downstream perfor- mance, significantly outperforming existing metrics. By visualizing components of the DSE, explain that the cause of the SDD phenomenon is either a loss of class separability or dimensional collapse. For example, in Fig. 1, MoCo v3's performance drop due to dense dimensional collapse, while DINO's degradation arises from decreased class separability. Moreover, our proposed check- point selection approach improves the mloU by 3.0% on average, and incorporating DSE into training further improves both DSE scores and downstream dense performance, effectively eliminating SDD's negative effects.
We summarize our contributions as follows:
We identify the SDD phenomenon in SSL, revealing inconsistency between image-level and dense-level performance, which is prevalent in satt--of-the-al methods and negatively impacts dense tasks.
We introduce the Dense Representation Strueture Estimator (DSE), which accurately and efficiently predicts downstream dense performance without relying on downstream data.
Using the DSE metric, we propose model selection and regularization strategies that effec- tively reduce the negative impact of the SDD phenomenon.
Extensive experiments on sixteen leading SSL methods across four benchmarks demonstrate the precision of DSE and the effectiveness of the proposed approaches in addressing the SDD phenomenon.

---

2 Related Work
2.1 Self-supervised Learning
Contrastive Learning. Contrastive self-supervised methods have shown significant progress in recent years [11, 73, 28, 61, 62]. These methods typically construct positive examples (different augmented views of an image) and negative examples (samples from other images), then use contrastive loss [49] to train models to differentiate between them [12, 15, 78, 4, 32, 27, 20J.
Non-contrastive Learning. Recent advances in self-supervised learning avoid explicit negative examples. These methods often employ siamese networks and aim to achieve cross-view consisteney by aligning representations between teacher and student networks [26, 10, 83, 50, 14, 9].
Masked Image Modeling. Masked image modeling can be framed as a generative task. Models are trained to reconstruct original images from masked inputs [29, 74, 3, 13]. Recent works [83, 50] achieved great success by combining latent-space reconstruction with self-distillation.
Dense Representation Learning. Another line of work focuses on learning dense representations. Using techniques such as dense alignment [86, 70, 31, 71], clustering [84, 58, 40, 59, 5]. and reconstruction [83, 50], these methods achieve strong results on dense prediction tasks. However, despite optimizing dense representations directly, performance degradation still observedduring training.
2.2 Unsupervised Transferability Estimation
Recently, a-REQ [2] uses the parameter of power-law distributions of covariance matrix singular values as metric. RankMe [24] employs the effective rank [53] estimate the transferability performance, and Lidar [60] further improves it by introducing linear discriminative analysis. Other approaches analyze feature activation statistics [36], coding rate reduction [77] or model memorization effects [69, 68]. Despite progress in image-level tasks, evaluating dense representations remains an open challenge. Due to space limitation, more related works are discussed in Appendix B.
3 The Self-supervised Dense Degradation Phenomenon
In self-supervised learning, models are typically trained for long periods. It is widely recognized that downstream classification performance generally improves as training loss converges [10, 14, 16]. However, we observe that dense performance actually degrades during pretraining, with the performance at the final checkpoint being significantly worse than that of the best model. This observation contradicts previous intuitions. We term this phenomenon Self-supervised Dense Degradation (SDD), and empirical and theoretical analyze this phenomenon this section.
3.1 Empirical Observations of the SDD Phenomenon
SDD is a General and Harmful Phenomenon. To investigate whether SDD occurs broadly, we conduct extensive experiments. The main findings are summarized in Fig. 2. These experiments confirm that SDD occurs consistently across 1) Various Pre-training Approaches: SDD exists in sixteen state-of-the-art methods across different types of training loss, model architecture and optimization strategies, and Various Evaluation Protocols: SDD is evident in both linear probing and transfer learning scenarios (where the backbone is not frozen), spanning diverse downstream datasets, evaluation hyperparameters, and tasks. Detailed results can be found in Appendix E. SDD is Not Caused by Overfitting the Training Data. We further investigate whether SDD stems from memorizing the training data. To test this, we train and evaluate DINO [10] on the same COCO dataset. The trend of dense performance mirrors that observed for DINO trained on ImageNet, with the final checkpoint experiencing a significant degradation of 4.0% mloU, This result indicates that SDD is not due to overfitting. Further details presented in Appendix E.3.
Since SDD occurs broadly and is not related to dataset overfitting, identifying a suitable metric to predict downstream performance would be valuable for understanding and mitigating this degradation. However, defining such metric challenging. the following subsection, we seek theoretical insights to help develop such performance measure.

---

139
E075
20
453
302
632
230
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
OF1-0 A1-43
350
Faoth
Soorh
Parch
Fooch
(c) EsVIT [41]
(f) MEC [44]
(g) VICRegL [5]
(h) DINO [10]
17.3
130
315
200
100
553
200
800
200
463
266
200
-00
650
500
(i) iBOT [83]
Mugs [84]
(k) MAE [29]
(I) I-JEPA [3]
COCO-Stuff
PASCAL VOC
ADE20K Last
Cityscapes
Method Type
Method
Architsccuree
Contrastive
MoCo v3 [16]
DenseCL, [70]
ViT-Small-16
ResNet-50
40.1
37.1
38.3
15.1
-22.0
-1.8
51.1
57.5
56.2
5.9
-15.2
-1.3
20.9
18.3
18.1
3.9
-14.4
-2.8
44.3
36.4
41.7
21.9
-11.5
-2.6
BYOL [26]
ResNei-50
37.1
30.7
-6.4
52.1
45.4
-6.7
18.8
10.9
-7.9
42.4
34.9
-7.5
Non Contrastive
SimSiam [14]
ResNei-50
35.1
34.5
-0.6
47.5
46.6
-0.9
15.9
14.4
-1.5
39.4
39.1
-0.3
EsVIT
Swin-Tiny-7
42.1
33.4
-8.7
60.2
51.3
-5.9
24.1
19.4
-5.0
51.1
47.8
-3.5
MEC [44]
ResNet-50
36.1
Volume-Base Based
Barlow Twins [78]
ResNei-50
37.6
36.9
-0.7
51.8
51.4
-0.4
20.2
19.6
-0.6
43.22
42.2
-1.0
VICReg [1]
ResNei-50
37.3
36.7
-0.6
53.0
52.7
-0.3
20.0
19.6
-0.4
43.1
42.5
-0.6
VICRegL [5]
ReaNei-50
-0.4
SwAV [9]
ResNei-50
41.3
40.8
-0.5
56.4
55.6
-0.8
22.8
22.1
-0.7
47.8
47.6
-0.2
DINO
ViT-Small-16
40.4
36.0
-4.4
57.1
45.8
-11.3
23.4
19.2
-4,2
44.4
44.3
-0.1
Clustering-
IBOT [83]
[83]
ViT-Small-16
VTT Base-16
48.9
47.0
-1.9
71.5
69.4
-2.1
31.3
29.9
-1.4
51.3
49.9
-3.2
-14
Mugs [84]
VIT UTT-Small-i 16
45.4
43.7
-1.7
68.1
67.6
-0.5
29.7
28.6
-1.1
47.0
44.9
-2.1
ReSA
ResNei-50
36.6
-0.4
18.4
18.2
39.6
-088
Masked Modeling
MAE [2]
VIT-Small-Je
36.8
36.4
-0.4
49.2
47.9
-1.3
18.2
17.5
-0.7
37.8
35.7
-2.1
IJEPA []]
NiT- 16
39.6
34.0
-5.6
60.2
52.6
-7.6
22.4
17.9
-4.5
40.1
36.2
.3.9
Figure 2: Top:
The change in dense performance throughout the pretraining process, assessed via linear segmentation the COCO-Stule dataset. Performance degradation is consistently observed across all methods. Bottom: A performance gap between the best and the last models is present across all datasets and methods.
3.2 Theoretical Analysis of the SDD Phenomenon
Our goal is to establish theoretically grounded metric for downstream performance. To achieve this, analyze the downstream performance and identify two crucial factors influencing it: 1) class separability, quantified by the difference between inter-class distance and intra-class radius (formally presented in Thm. 2), and 2) the dimensionality of the representations (Cor. 5). With these insights, readers interested primarily in methodology and experiments may skip the remainder of this section without any loss continuity.
3.2.1 Problem Formulation
Given that SDD appears across various methods, analyzing their training processes within a single unified framework is challenging. Therefore, mainly focus the linear probing approach, which aims to train classifier using fixed (dense) representations. Specifically, given downstream dataset D = {X,), consisting of N images, a fixed encoder fe X + Nxd produces N dense representations [ for each image, where is the dimension the representations. To simplify, formulate dense linear probing classification problem, where each representation

---

2; is assigned to one of K latent classes, represented as y(zz). The aim of linear probing is to train a classifier G(z) (for example, a linear head) that accurately maps each 2 to correct latent class. Following the analysis in [34], we choose simple Nearest Neighbor (NN) classifier:
((z) arg min x - Mells kE[K]
where μ4 = =2:;(=)=<((* denotes the center of the representations for the k-th class. The error rate of the fixed encoder so the downstream dataset is given by: (fo) BeED ((())) (ı(z) # G(=)]]
Since the NN classifier can be viewed as a special case any linear classifier, its error rate naturally serves upper bound for all classifiers.
3.2.2 Decomposing the Downstream Error Rate
Next, we decompose downstream performance and identify the factors that influence downstream accuracy. For the NN classifier G, representation can be correctly classified if the following condition holds:
My(z)ll
min Be [K] (x))
Ma O.
Intra-class distance
Inter-class distance
Inspired by this relationship, downstream performance be expressed in terms of intra and inter- class distances. However, directly measuring these distances is not feasible. The main challenge arises from the fact that, in sulf-suuereieed setting, class labels y(z) are unavailable. While techniques such as k-means clustering can be utilized to generate pseudo labels, we find that instance-wise distance measures still face critical issue, resulting in meaningless predictions:
The instance-wise intra-class distance is always smaller than the inter-class distance when using k-means pseudo- labels. Thus, the estimated accuracy is always 1, regardless of the actual situation.
Proposition 1.
Proof can be found in the Appendix A. The fun- damental reason for this issue lies in the fact that the instance-wis wise distance measure tends to un- derestimate the intra-class distance for those ex- amples near the decision boundary. We present an illustration of this issue in Fig. 3. To address this issue, we replace the instance-wise measure with a class-wise radius. As a result, we refor- mulate the condition that could be correctly classified as:
Instance-wise (Eq. (1))
Class-wisc (Eq. (2)
Rytn
| min I|z Rell
Ryon
min 11z Mell
|| Pscoll S
|| Pall
Ryco
||2
Pell
exyts
key())
Err(ff) =
(rr(()) = 8 P(8)
Figure 3: Left: Instance-wise condition (Eq. 1) predicts all examples in the intersection area as correctly classified, leading to an inaccurate eπтоr rate estimation. Right: Our class-wise condition (Eq. 2). Examples in the intersection area are accurately predicted misclassified.
Rey(()))
min kE(K)vo(=)
||2 111--el1- (2)
Intra-cian radius
Inter-class distance
Building this idea, further demonstrate that under the assumption that the representations within each class are concentrated (e.g., following a sub-Gaussian distribution), the downstream performance can be guaranteed.
Theorem rellvvtt Measure for Downstream Performance). Let = {2 y(Z) = j} be the set of examples in the j-th class with [Z]| Ny- Assume that for all j [K]. the examples
{ Me in Z are i..d.
random vectors in Rdd. Denote zí N, ΣN z and
Z] z 23
#1 as the centered embedding matrix for Zś, Then, for any 8 > 0:
1 (2((()))
Erro(Jo)
D min
Cs
(3)
(Ny(z)
Downstream crror rate
Inter-class dissunce
Estimated intra class radiax
Here, oz(.) represents the i-th singular value, Cs is margin term that jointly determined by R, 8, and N; (please refer to Appendix A. 2.3 for exact formulation), and D min minke((]]e(e)) ||= pp
denotes the minimal inter-class distance.
5

---

Batch Sampling
Intra-class Consistency Measure
Cluster View
B
a
M Mintra
K-means Features 2
Pd
Trace
Minter
Separability
Mintra
I Bill
Miteer
DSE
M Minter
M intraa
+
M Maam
Z IN Bxxxxxx
--maans Features 2
of
Clustering
Image Representations
(c) Dimensional Collapse Measure
Dimension View
Representations
Erunk(2)
Dimensional Collapse
M
Collapsed Dimension
Matm
Cluster Centers
Patch Samples
Shuffled Features Z
Figure 4: The proposed Dense representation Structure Estimator (DSE) consists of three components. The intra-class consistency measure and inter-class distance measure jointly conduct the class- separability measure, which is motivated by the results in Thm 2. The dimensional collapse measure estimates the effective dimensionality, which corresponds the analysis in Cor.
Remark 3. In this theorem, we estimate the intra-class radius Ryy(s) with the normalized trace of the representation matrix. When this radius (plus a margin term Cş) smaller than the inter-class distance, a simple NN classifier would be enough to separate these representations.
Remark For simplicity, we ignore the label estimation error in this theorem. A complete version, including the effect of k, is provided in Appendix A.4. Briefly, the bound is tightest when k equals the true number of classes, but generally remains valid when & exceeds the actual number of classes.
Next, we reveal another key factor affecting the downstream error rate: the dimensionality of representations.
Corollary 5 (Error Rate Decay with Dimensionality). Under Thm. 2's assumptions with v3). for any ō >> 0:
mink4y(x) [IAxy(*) - ma van + lng(8/8)
Erro(Jo) 5 2K exp (-Cs.d)
where Ö3 a constunt.
Remark 6. This corollary establishes connection between analysis and the broadly studied dimensional collapse phenomenon SSL [24, 85]. When is small, the downstream performance experiences significant degradation. These findings highlight the critical need to jointly assess the dimensionality of representations.
Addressing SDD via Dense Representation Structure Estimator
4.1 Dense Representation Structure Estimator
Inspired by the analysis in the previous section, we propose a metric called Dense representation Structure Estimator (DSE) with the following formulation:
Σi or (e(())))
DSE Eg D
+
dim..
(N
Class separability messure
Effective dimensionality
The first term is derived from the result of Thm. Since the cumulative density function in Eq. decreases monotonically with the deviation, the deviation between two terms can be treated

---

Algorithm DSE-based Model Selection
Input: Training dataset X, checkpoints 1 maximum number of candidates T. for i = to N do
Sample a batch of data X from X and calculate the dense representations (x) Calculate the metric P. based on Eq. 5
end for
Select the local maximum points by (i: arg jeljili-224tt Pj
Keep the indices in C with the top-7" metric and obtain C = (Sb : C}. Output: Model candidates C.
measure of downstream performance. The second term corresponds to the analysis in Cor, 5. For intuitive understanding, readers can refer to Fig. 4.
Measuring Class Separability. Given a batch of dense representations Z {2i BxN we first calculate the k-means on all B x N representations to obtain a pseudo-label ğ(z) [k] for all
representations 2. Let (z E Z y(2) j} denotes the representations in the j-th cluster and N, |Z represents the number of representations, the intra-class radius and inter-class distance are calculated as:
Mia (2( i=l
Mintro
Minter
min || p0:l22 itj
jm1 (N, 1)
N₂
#E
where Z Zi - 1 ΣE is the centered representation matrix, and jú
-i=
1 |20|
Σxe2i
represents the center of the j-th cluster.
Mcasuring Dimensional Collapse. As discussed in Cor. 5, the dimensionality of representations affects their separability, and thus should also be considered. Building on previous work [24], we first randomly sample B' dense representations from different images (ensuring their independence) and concatenate them to B' x matrix Z. By setting B' >> d, the rank Z reflects the number of non-collapsed dimensions of representations, Thus, compute its effective rank [53]:
M Erank Z) exp (-Σ p. log
Σ
(2) is the i-th normalized singular value of Z. where Pı [o(
Final Formulation of DSE. DSE is calculated by:
DSE Minter Mintra + A M
(5)
where À is a parameter that rescales the measure of effective dimensionality to the same amplitude of class-separability statisties. In practice, it taken as:
A Std(Meeter - Mintra)
Std(Maim)
where Std(-) denotes the standard deviation calculated across all checkpoints.
4.2 Mitigating SDD Phenomenon with the DSE Metric
DSE-Guided Off-the-shelf Model Selection. When modifying the training process is not feasible, we propose selecting the best model from the saved checkpoints using the DSE metric to reduce the negative impact of the SDD phenomenon. When comparing two models, the one with higher DSE indicates better class separability and effective dimensionality, and is therefore expected to perform better according to our theory. Based on this idea, we first compute the DSE metric and select checkpoints corresponding to local maxima as potential candidates. To reduce computational costs, we then choose the top 7 (T 3) checkpoints with the highest. DSE values from this candidate set as final models. The complete procedure is shown in Alg. 1.
7

---

mloU on COCO-Stuff
The proposed DSE metric
10:
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
3K 6K
(e) VVV [41]
(l) MEC [44]
(g) VICRegI. [5]
(h) DINO [10]
120
(i) iBOT [83]
Mugs [84]
(k) MAE [29]
I-JEPA [3]
Figure 5: The proposed DSE metric precisely predicts the downstream performance.
DSE-regularized Online Optimization. Since the DSE metric represents lower bound on perfor- mance and all operations involved in computing DSE are differentiable, we also consider directly optimizing the DSE an explicit regularizer. Specifically, we add the negative DSE metric the original loss function for each learning framework:
Coriginal 3 DSE.
In our experiments, we À in DSE and 3 to 0.001. Empirically, we find that training for 10 epochs starting from the checkpoint with the best initial performance effectively mitigates the SDD phenomenon and enhances downstream performance. More details are presented Appendix
Empirical Studies
5.1 DSE Metric is Precise Estimator for Dense Performance
Experiment Setup. To assess the effectiveness of the proposed DSE metric, we examine correlation between DSE and downstream segmentation performance across sixteen SSL methods and four datasets. We leverage linear transfer learning as the downstream task, with settings remain the same introduced in the previous section. To save time, only 2048 images (~ 0.16%) of the training dataset (ImageNet-Il are used to compute the metric, and both the metric and dense performance evaluated every 10 epochs.
To validate the correlation between the proposed metric and downstream performance, the results are evaluated using Kendall's coefficient [37]. The coefficient ranges from [-1, 1]. When = 1, the metric is perfectly aligned with the downstream performance, and r represents that they are inversely correlated. We provide a detailed explanation in the App. D.
Analysis of the Empirical Results. We draw two main conclusions from Fig. and Tab. 1. 1) The DSE metric accurately reflects downstream performance. The metric curve consistently aligns with downstream performance across different datasets and methods. Hypothesis tests yield an average Kendall's 7 coefficient of 0.57, confirming the reliability of DSE. 2) Compared to existing estimators, DSE better suited for dense tasks. Current estimators are ineffective for dense performance evaluation due to the SDD phenomenon. Even if these estimators are adapted for dense representations (specifically,
adapted RankMe [24] equivalent to our Maim), DSE still significantly outperforms them. The advantage of DSE stems from its ability to more comprehensively characterize dense performance, as analyzed in detail the following subsection. Additional empirical analyses, omitted here due space constraints, are provided in Appendix F.

---

Table 1: Left: Kendall's coefficient of the DSE metric. We denote the insignificant results with (p > 0.05), otherwise, the results are significant with p < 0.005. Right: Comparison of Kendall's coefficients for different estimators. Methods with are adapted to dense representations.
Method
CocO
PVOC
ADE20K
Citsscapes
McCo [16]
0.55
0.58
0.60
0.45
DenseCL [70]
0.70
0.81
0.52
0.63
BYOL [26]
0.61
0.48
0.79
0.78
SimSiam [14]
0.82
0.84
0.51
0.76
Estimator
Images
COCO
VOC
ADE
City
Avg.
a-ReQ
25600
0.09
MEC [44]]
0.68
0.65
0.15"
0.63
RankMe [24]
25600
-0.10
-0.09
-0.14
0.00
-0.08
Barlow Twins [78]
0.55
0.57
0.53
0.61
Lidar
10000
-0.37
-0.36
-0.26
-0.21
-0.30
SwAV [9]
VICReg [4]
VICRegL. [S]
0.72
0.90
0.73
0.75
0.74
0.91
0.74
0.89
0.72
0.76
0.72
a-ReQ! [2]
RankMel [24]
2048
2048
0.17
0.25
0.19
0.26
0.11
0.10
0.14
DINO [10]
0.00
0.07
0.15'
0.42
Lidar [60]
2048
0.38
0.37
0.33
0.23
0.33
IBTT
(Ours)
2048
0.60
0.49
Mugs [84]
0
0.47
0.46
0.20
ReSA
0.74
MAE [29]
0.38
0.44
0.46
0.21
IJEPA
0.49
0.38
0.59
0.28
is noteworthy that the DSE metric is theoretically derived from the class-relevance downstream tasks like semantic segmentation. To see if DSE can accurately predict the dense performance beyound segmentation, we present more results on depth estimation task in the Appendix F.I.
5.2 DSE Metric Demystifies the SDD Phenomenon
Based these observations, provide in- sights into the causes of the SDD phenomenon from two perspectives: 1) SDD arises from in- sufficient class separability, reduced effective dimensionality, both. While SSL aims to balance semantic alignment and the effective dimensionality of representations, these two ob- jectives often involve a trade-off. For instance, fully uniform representations have high effective dimensionality but may lack separability; con- versely, focusing too much semantic align- ment can lead to dimension collapse. We argue that the degradation seen in different methods is due the failure to maintain this trade-off during training, which causes a bias toward one objective. 2) The reasons for degradation depend specifically on the method. As illustrated in Fig. 6, MoCo v3's performance degradation is due dimensional collapse in dense features, which reduces representation separability. Additionally, the unusual performance drop observed in DINO at around 300 epochs corresponds to a slower reduction in intra-class distances relative to inter-class distances, decreasing overall class separability.
Mintra DSE
Minter
Mam
VOC mloU
wywwh
02
(a) MoCo v3 [16]
(b) DINO [10]
Figure 6: Different components of DSE metric.
These insights also explain why DSE improves over other estimators. Although other estimators accurately measure effective dimensionality, they fail to predict collapses caused by reduced class separability. DSE metric addresses this by providing more complete measure of dense performance.
5.3 DSE-based Approaches Effectively Mitigate the Negative Impact of SDD
DSE-b Model Selection is Accu- rate and Efficient. We validate the effectiveness of our proposed model selection method on four benchmark datasets. As shown in Tab. 2, our model selection consistently improves mloU and accuracy across all meth- ods and datasets, achieving an average improvement of 3.0% in mloU. Com-
Table 3: Our approach achieves a performance gain compa- rable to supervised oracle, without requiring testing data or labels, and with negligible computational cost.
Estimator
Data
Label
A mIoU t
GPU hours 1
Loss Supervised
-1.0
0.0
v
43.6
2.43
DSE
+3.0
0.025 (~> 0.01
pared the previous srate-of-the-ar method, iBOT, our approach further improves the best mIoU by an average of 2.5%.

---

Table 2: The overall performance of DSE-based model selection.
Method
Architecture
mloU
COCO-Stull'
Acc
mloU)
PASCAL VOC
mloU
ADEMI
Acc
miol
Chtyeegess
Acc
[16]
Smal--
53.9
49.7
24,9
+MS
VIT-Small-
30.9 (+15.8)
69.1++15.5)
42.0(-36.1)
(-1000
(412))
63.2
(+135)
(+9.9)
(e3.3)
DenseCL. [1]
Reshier-50
38.3
72.0
562
89.2
18.1
64.5
41.7
87.1
+MS
ResNict-0 50
39.7
728 (40.8)
(697)
89.5 (+0.3)
20.5 (12.4)
(4.5)
(3.4
87.8
[
ResNect50 50
30.7
65.2
45.4
85.8
10.9
57.8
34,9
$4.5
ResNer-50
(16.4)
(15.))
(4577
(42.1)
18.7 (17.8)
63.5 (45.7)
(17.()
87.3 (128)
SumSiam [14]
Reshect50
67.8
46.6
14,4
59.7
39,1
+MS
Reshice50
35.0
68.1 (+0.3)
47.0(-0.4)
86.9 (40.))
(+1.2
60.5 (+0.8)
(403
(-06)
EVVIT (4))
Swin- Ting-
3344
66.3
$4.3
87.6
19.4
61.3
47,8
58.9
MSS
Swis- Tiny-7
(1.6
(17.))
(6.8
(9.7()2.1)
24.4 (15.0)
67.5 (16.2)
50.8
89.5
MEC [4]]
ResNect50 50
35.4
67.8
48.0
$3.1
16.3
60.3
29,8
854
-MS
ResNer-50)
(40.)))
(10.)
(4.5()055)
37.2
(2130)
60.7 (40.4)
(200)
Barlow Twins (78)
ResNet-50
36.9
69.0
514
19.6
62.4
42.3
+MS
50
(0.5)
(40.))
(1.6()02)
88 (e.2()
19.9 (40.3)
(2.2)
(25((055)
VICReg (-)
Resher-50)
36.7
69.2
$2.7
68.1
19.5
62.3
42.5
867
MMS
ReiNer-50
(10:5
09.5 (10.))
$3,(3033)
88.3 (.0.2)
19.8 (10.2)
(2.))
(2.(3003)
869 (
VICRegL
Se
38.1
71.5
54.5
58.7
20.4
64.6
44.6
87.9
+MS
Re
38.3 (+0.))
(40))
(4.6(+0.))
(e0.1)
20.7 (40.3)
(40.2)
(4.2(002)
(-.))
SWAV [9]
ResNes-50
40.8
72.5
55.6
89.0
22.1
65.6
47.6
88.7
+MS
56
(1.0
72.5 (10.0)
(6.()044)
(900
22.8
(40.)
47.6(000)
88.7 (-0.0)
DINO [[O]
V.T. 16
36.0
69.7
45.8
873
19.2
64.8
44.3
$9.1
LMS
WT--mmil-16
40.1 (14.1)
74.5
89 (4225)
22.7 (13.5)
68.3
(43.5)
44.3 (4.00000))))
89.1 (-00)
iBOT
IT-8mmal-16
43.2
73.6
64.4
91.7
24.1
69.6
44.6
89.2
Small111
(22.2)
76.5 (429)
66.844244
(+0.8)
27.5 (43.4)
(+19)
(6.5(19))
(-05)
MAE [1)
WT--mmal-16
36.4
71.3
47.9
17.5
65.1
36.7
87.0
MS
WT--mmll-116
(10.))
(10.2)
(8.0()
88.2 (20.0)
18.2 (10.7)
65.5 (G0.4)
572((115)
(1.6)
Mugs| [S]]
Small-
43.7
24,9
67.6
92.3
28.5
70.7
44.9
89.2
MMS
1T--mmil-16
(40
(41.
(700)
92.3 (e0.0)
28.6 (40.0)
70.7
(40.0)
45 (506))
(00(20.))
ResNer-50
36.2
68.2
49.1
18.2
61.2
38.8
847
50
(104)
68.5 (10.3)
(9.5()0.4)
87.4 (40.4
18.3 (40.))
61.2
(9,6
(5.0-033)
[JEPA [1]
VTT-Base-16
34.0
68.6
$2.6
58.3
17.9
63.4
362
863
MSS
WT--Bae-166
(456))
72.6
(16.7)
(417()14)
(14.5)
66.9 (43.5)
(62(424)
(-1.)
With DSE Regularization
Baseline
07.50
1.526
61.
-273
-0 -0.009
67.30
-1.442
-0.28
+0.2
40.2
67.30
8 1.550
+0.013
60.1
0.292
00)
-1.575
-0.302
66.70
150
1.591
59.1
800
00:311
600
(a) iBOT
(b) 1-JEPA
Figure 7: Effect of DSE Regularization on iBOT [83] and I-JEPA [3].
In Tab. 3, we compare our model selection method with two baseline estimators: training loss and supervised downstream performance. Loss-based selection fails to track dense performance under the SDD phenomenon, and supervised selection is impractical due to high computational cost. In contrast, our DSE-based selection achieves competitive results with approximately 97. 97.2x speed-up. highlighting both efficiency and effectiveness.
DSE Regularization Improves Dense Performance. As shown in Fig. 7, DSE regularization consistently enhances model performance and the DSE metric. More importantly, it reverses the trend of dense degradation, demonstrating its fundamental capability to mitigate the SDD phenomenon. Results for additional methods and component ablation studies are provided in the Appendix F.9.
6 Conclusion
This paper identifies a widespread and detrimental phenomenon in self-supervised learning, termed Self-supervised Dense Degradation (SDD), Specifically, SDD occurs when dense task performance degrades over the course of training, causing the final checkpoint to exhibit a significant performance gap compared to the best intermediate checkpoint. To address SDD, we propose a Dense Repre- sentation Structure Estimator (DSE) that evaluates representation quality for downstream tasks by quantifying class separability and effective dimensionality. Our DSE is theoretically justified and empirically shown to correlate strongly with downstream task performance. To eliminate the harm of SDD, based on DSE, we introduce a checkpoint selection method for the off-the-shelf setting, and also optimize DSE directly as a regularizer. Experiments of sixteen SSL methods across four benchmark datasets confirm that these approaches effectively mitigate the negative effects of SDD.
10

---

Acknowledgments
This work supported in part by National Natural Science Foundation of China: 62525212. 62236008, 62441232. U21B2038, and U23B2051, in part by Youth Innovation Promotion Association CAS, part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680201, , in part by the China National Postdoctoral Program for Innovative Talents, Grant No. BX20250377.
References
[1] Andrea Agostinelli, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. Transferability metries for selecting source model ensembles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1936,7946, 2022.
[2] Kumar K Agrawal, Arnab Kumar Mondal, Arma Ghosh, Blake Richards. a-ReQ: Assessing rep- resentation quality in self-supervised learning by measuring eigenspectrum decay. Advances in Neurul Information Processing Systems, 35:17626- 17638, 2022.
[3] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas, Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619-15629, 2023.
[4] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance- Variance-invvariance invariance-covariiaee regularization for self-supervised learning. arXiv preprint wxXi:2205.04900 2021.
[5] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. Advances in Neural Information Processing Systems, 35:8799-8810, 2022.
[6] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
[7] Daniel Bolya, Rohit Mittapalli, and Judy Hoffmann. Scalable diverse model selection for accessible transfer lcarning. Advances in Neural Information Processing Systems, 144199301-19311 2021.
[8] Holger Cacsar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1209-1218, 2018.
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 13:9912-9924, 2020.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[12] Ting Chen, Simon Korblithh, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances in neural information processing systems, 23:22243-22255. 2020.
[13] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo. Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autocncoder for self-supervised representation learning. International Journal of Computer Vision, 132(1):208-22) 223-,, 2024.
[14] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 55750-15755 2021.
[15] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive lcarning, arXiv preprint arXiv:2003. 2020.
[16] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640-9649, 2021.

---

[17] Marius Cordts, Mohamed Omran. Sebastian Ramos, Timo Rehfeld. Markus Enzweiler. Rodrigo Benenson. Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscupes dataset for semantic urban scene understanding. In IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 3213-3223, 2016.
[18] Siran Dai, Qianqian Xu, Peisong Wen, Yang Liu, and Qingming Huang. Exploring non-contrastive self-supervised representation learning for image-based profiling. arXiv preprint arXiv:2506. 14265, 2025.
[19] Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona, A linearized framework and new benchmark for model selection for fine-tuning, arXiv preprini arXiv:2102.00000 2021.
[20] Kshitij Dwivedi and Gemma Roig. Representation similarity analysis efficient task taxonomy & transfer learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12387-12396, 2019.
[21] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-338, 2010.
[22] Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2): 179 188, 1936.
[23] Keinosuke Fukunaga. Introduction to statistical puttern recognition. Elsevier, 2013.
[24] Quentin Garrido, Randall Balestricro, Laurent Najman, and Yann LeCun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning, pages 10929-10974, PMLR, 2023.
[25] Mohsen Gholami, Mohammad Akhari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. Etram: Energy-based transferability estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[26] Laan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[27] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34: 5000-5011, 2021.
[28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pages 16000-16009, 2022.
[30] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International conference machine learning, pages 4182 4192. PMLR, 2020.
[31] Olivier J Hénaff, Skanda Koppula, Jean- Bran-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10086-10096, 2021.
[32] R Devon Hjelm, Alex Fedorov, Samuel and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint a80iv:1808.0 06677 2018
Karan Grewal, Phil Bachman, Adam Trischler,
[33] Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In International conference on machine learning, pages 9201-9225. PMLR, 2022.
[34] Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:2111.0074: 2021.
[35] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishman, and Samy Bengio. Fantastic general- ization measures and where to find them. arXiv preprint 11X 2019.
12

---

[36] Neha Kalibhat, Kanika Narang, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Measuring self-supervised representation quality for dowastream classification using discriminative features. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13031-13039, 2024.
[37] Maurice Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
[38] Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900 2014.
[39] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[40] Tim Lebailly, Thomas Stegmüller, Behzad Bozorgtabar, Jean-Philippe Thiran, and Tinne Tuytelaars. Cribo: Sulf-supe learning via cross-image object-level bootstrapping. arXiv preprint arXiv:2310.07855. 2023.
[41] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. International Conference on Learning Representations (ICLR), 2022.
[42] Xiaotong Li, Zixuan Hu, Yixiao Ge, Ying Shan, and Ling- Yu Duan. Exploring model transferabilityy through the lens of potential energy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5429-543 5438. 2023.
[43] Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2663-2673, 2021.
[44] Xin Liu, Zhongdao Wang, Ya-Li Li, and Shengjin Wang. Self-supervised learning via maximum entropy coding. Advances Neural Information Processing Systems, 35:34091-34105, 2022.
[45] Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, and Qingming Huang. Not all pairs are equal: Hicrarchical learning for average-precision preciiionnooiinteee vidco retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3828-38 3837, 2024.
[46] Yang Liu, Qianqian Xu. Peisong Wen, Siran Dai, and Qingming Huang. When the future becomes the past: Taming temporal correspondence for self supersue video representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24033-24044, June 2025.
[47] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[48] Cuong Nguyen, Tal Hassner, Matthias Seeger. and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pages 7294-7305. PMLR, 2020.
[49] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 10X.v:8807.03745 2018.
[50] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
[51] Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Transferability estimation using bhattacharyya class separability.. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9172-9182, 2022.
[52] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arheláez, Alex Dorkine-Homung and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 1704.00655 2017.
[53] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEEE, 2007.
[54] Ana Finchez-Fernandee Elisabeth Rumetshofer, Sepp Hochreiter, and Günter Klambauer. Cloome: contrastive learning unlocks bioimaging databases for queries with chemical structures. Nature Communi- cations, 14(1):7339, 2023.
13

---

[55] Jonathan Sauder and Bjarne Sievers. Self-supervised deep learning on point clouds by reconstructing space. Advances in neural information processing systems, 32, 2019.
[56] Wenqi Shao, Xun Zhao, Yixiao Ge. Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models equal: Predicting model transferability in a self challehalln fisher space. In European Conference on Computer Vision, pages 286-302. Springer, 2022.
[57] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, al. Dinov3. arXiv preprint arXiv:2508, 10104, 2025.
[58] Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars. and Jean- -han-Philippe Thiran. Croc: Cross-view online clustering for dense visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7000-7009, 2023.
[59] Qing Su, Anton Netchaev, Hai Li, Shihao Ji. Fisl: Feature-level self-supervised learning, Advances in Neural Information Processing Systems, 36:6568-6535 16881,65622222 2023.
[60] Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint embedding ssi architectures. arXiv preprint arXiv:2312 2,0000000, 2023.
[61] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[62] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
[63] Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-suuer video pre-training. Advances in neural information processing systems, 35: 10078-1099,, 2022.
[64] Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1395-1405, 2019.
[65] Roman Vershynin. High-dimensional probubility: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[66] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019,
[67] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International conference on machine learning, pages 9929 9939. PMLR, 2020.
[68] Wenhao Wang, Adam Dziedzic, Michael Backes, and Franziska Boenisch. Localizing memorization in ssl vision encoders. arXiv preprint arXiv:2409. 19069, 2024.
[69] Wenhao Wang. Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes. Nicolas Papernot. and Franziska Boenisch. Memorization in supervupe learning improves downstream generalization. arXiv preprint XxXii:2401.1222 2024.
[70] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, Lei Li. Dense contrastive learning for self-supervised visual pre mee-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3024-3033, 2021.
[71] Peisong Wen, Qianqian Xu, Siran Dai, Runmin Cong, and Qingming Huang. Semantic concentration for self-supervised dense representations learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-17, 2025. doi: 20110/9/PA...225....... 206573697778....
[72] Xi Weng, Jianing An, Xudong Ma, Binhang Qi. Jie Luo, Xi Yang. Jin Song Dong, and Lei Huang. Clustering properties of self-supervised learning. arXiv preprint arXiv:2501.18-4 2025.
[73] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 2018.
14

---

[74] Zhenda Xie. Zheng Zhang, Yue Cao. Yutong Lin. Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 9653-9663, 2022.
[75] Huiwen Xu and U Kang. Fast and accurate transferability measurement evaluating intra-class feature variance. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11474- 11482, 2023.
[76] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long, Logme: Practical assessment of pre-trained trained models for transfer learning. In International Conference on Machine Learning, pages 12133-12143. PMLR, 2021.
[77] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in neural information processing systems, 33:9422 9434, 2020.
[78] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised lcarning via redundancy reduction. In International conference machine learning, pages 12310-12320 PMIR, 2021.
[79] Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Y Zhang. Yuxuan Liang, Guansong Pang, Dongjin Song, et Self supervised learning time series analysis: Taxonomy, progress, and prospects. IEEE transactions on pattern analysis and machine intelligence, 46(10):6775- 6794, 2024.
[80] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitikk Self-supervised contrastive pre-training for time series via time-freyuency consistency. Advances in newral information processing systems, 35:3988-4003, 2022.
[81] Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features any point-cloud. In Proceedings of the IEEE/CVF international conference computer vision, pages 10252-10263, 2021.
[82] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 633-641.20 2017.
[83] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:21 2021.
[84] Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: multi-granular self-supervised learning framework. arXiv preprint arXiv:2203. 144115, 2022.
[85] Zhijian Zhuo Yifei Wang, Jinwen Ma, and Yisen Wang. Towards unified theoretical understanding of non-contrastive learning rank differential mechanism. arXiv preprint arXiv:2333..02.330 2023.
[86] Adrian Ziegler and Yuki M Asano. Sclf-supervised learning object parts for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14502-
14511, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The contributions and scope of the paper are accurately summarized in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix. G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

The assumptions are provided in Sec. 3.2 and the proofs are provided in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All details are introduced in Sec. Se.Sec. 5, and Appendix D. The code is provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We open-source datasets for evaluation, the code is provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (attps://nips)
Staia//guidde//ooeSSuoeeee oooooiisissoooccroi.ooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cccccc///oo ccc/public/gee /gtcc//g//s//////eoo /Cons//omdembiiim ississsies ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We following the default data splits for the datasets used. Other details are provided in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Though we report statistical significance tests in the paper, we do not report error bars due to the computational cost of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /neurisuus..p// cc cc/ccc/paao
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: In this paper, we focus on foundational research for self-supervised learning, societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit and mention the license and terms of use of the assets used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve LLMs as any important, original, or non- non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: pettps://neur/i wttp//rpps..c//oeeeeeeeeeeeeeeee for what should or should not be described.
22

---

Appendix Contents
Proofs
24
A.1 Proof of Proposition
24
A.2 Proof of Theorem 2
24
A.2.1
Preliminaries
24
A.2.2
Important Lemmas for the Proof
25
A.2.3 Proof of the Main Theorem
28
A.3 Proof of Corollary 5
29
A.4 Analysis of the Effect of
30
B Additional Related Works
32
Detailed Methodology for DSE-regularized Online Optimization
33
D Detailed Settings
34
D.I Pretraining
................................
34
D.2 Evaluation
35
D.3 Kendall's Coefficient
35
D.4 Details in DSE Calculation
................................
35
D.5
Computational Resources
E
Additional Experiment Results of the SDD Phenomenon
36
E.1
The SDD Phenomenon Exists Across Datasets..
36
E.2
The SDD Phenomenon Exists Under Varying Evaluation Protocols
38
E.2.1
The SDD Phenomenon Exists when Backbone is Not Frozen.
38
F.2.2
The SDD Phenomenon Exists in Varying Evaluation Hyperparameters.
38
E.3
The SDD Phenomenon not Caused by Dataset Overfitting
39
E,4
The SDD Phenomenon Exists in Varying Downstream Tasks
39
Additional Experiment Results of the DSE Metric
40
El
DSE Metric is Precise Estimator Depth Estimation Task
40
F2
DSE Precisely Indicates Performance Trends Across Unseen Datasets
40
F3
Ablation Study of Different Components.
42
F.4
Sensitivity Analysis on Number of Images and Clusters
F55
Analysis of Bias Introduced by Data Distribution Shift and Label Estimation Error Understanding the Reason behind SDD Phenomenon
44
F6
46
F.7
Extending the DSE Metric toward Image-level Performance Estimation
46
F88
Visualizations
47
F99
Ablation Study of DSE Regularization
49
Additional Discussions
50
23

---

A Proofs
A.1 Proof of Proposition
12..22..... 2n be set of points in R., and let C {ci, C2 ..., C be the set of cluster centers obtained by the K-means algorithm. For each point E 2, let e(z) E C denote the cluster center to which z is assigned by k-means. Then, for every and for all C, C with G; + c(z), the Euclidean distance satisfies
Proposition (Restate of Prop. 1). Let 2
|| - -())22 I 1| c.ll2-
As a result, the estimated error rate of the NN classifter always 0:
Erro(fo) EEED PrefO(a))
||2 c(2)]l
|| c:ll =0 cx#c(=)
min
Proof. By the definition of the K-means clustering algorithm, each point E Z is assigned to the cluster with the nearest center. Specifically, c(z) is chosen to minimize the squared Euclidean distance to z among all cluster centers in C. Formally,
c( arg min ||
eyec
Assume, for contradiction, that there exists point Z and cluster center c; C with c; c(z) such that
= c(z)]l2 | ca|l2-
Squaring both sides (since the Euclidean distance is non-negative), we obtain
|| c(2) | c:
However, this contradicts the definition of c(z) as the cluster center that minimizes the squared distance to z. Therefore, our assumption must be false, and must hold that
|| c(())l2 < || ccll2
for all a e with o, # c(z). It yields that:
PreSe(m) || ||2- c(=)ll
||2 call = 0. c.fc(2)
min
This completes the proof.
A.2 Proof of Theorem 2
Proof Scratch. We first present proof scratch for better understanding. The idea straightforward. by assuming representation in each class is R-suub-aausian by the property of concentration, the representations lie the main part the distribution with probability of least for any > If the radius is smaller than the minimal inter-class distance, all the representations in the main part would be correctly classified, leading to a final error rate smaller than 6.
A.2.1 Preliminaries
We first list some important properties used in the derivation. Lemma Sub-Gaussian Property). random variable X R-sub-Gaussian if its moment generat- ing function satisfies:
12
E(exp(AX)) S exp
VAER.
Lemma 9 (Sub-Exponential Norm Bound). X RRd is R-sub-Gaussian vector with independent coordinates, then
|| Ei X? is sub-exponential. Specifically, the sub-exponential norm
||X satisfies:
|| CRRd,
where C' > 0 is an absolute constant.
24

---

Proof. Since each X; is R-sub-Gaave Gaussian, X is sub-exponential with || S CR (by [65], Prop 2.7.1). The sum of d independent sub-exponential variables has norm of at most CR L
Lemma 10 (Chernoff Bound for Sub-Exponential Tails). Let Y = ||||| where X E Rd is an R- Bab-Gaussian vector with independent coordinates. Then is sub-exponential with [Y S CR For any >> 0:
P( 2 B[Y] Ci Red(++ vi) se
Proof. From Lemma 9, || S CR a. Using the cubbeexpoential tail bound:
-emin (o= a )))
P(Y - E
2 t) S exp
a2
Set / a(s + )2/2) for > Then:
t
= min (5*+202/2 +s, s s s²
min
a2 K
Thus P(Y > F[Y] CR&d(s V8)) c0 Rename 1/c for absolute constant C C' C'mx 61 Vol yields the proof.
Lemma 11 (Bernstein's 's Inequality [65]). Let Y1, Yn be be independent sub-exponential random variables with | I K. For any 2 0,
Σor (v. - E(TI) 21 < 2exp (-Cam min (244)) 4 K2' (R )
1=1
where C2 > 0 is an absolute constant.
Lemma 12 (Norm Concentration for Sub-Gaussian Variables [65]). If E Rd is R-suuaGaussian, then for all >> O,
P 2IXII CsR(Va+ eep((-t).
where C3 0 an absolute constant.
A.2.2 Important Lemmas for the Proof
Next, we proof some crucial lemmas for the proof.
Lemma 13 (Trace Concentration for Sub-Gaussian Random Vectors). Let X E RRd be a mean-zero Gaussian random variable; i.e., for all a E Rd
R-sub-Gaussian
E T S exp R² [all
Let [ and Σ = Cov(X). Suppose {X;} are i.i.d. copies of X, and define
N == Σ
1
(X (X, X
X
N
Then with probability at least 1 -
|ir(5) (r(())(r(())) S Ò R2 (ay log(8/8 (8/8)
d + log 8)
N
where Ò > is a constant.
Proof. We know
tr(2) [[ 11.112
25

---

N N
tr(2)
|x
|
N
Rearrange the terms, we have
N
tr(E)-tr(Z)
Σ |
-II
(IIX 2 ||
N
Term B
Term
where the factor N-I only introduces a constant factor. Next, we bound Term A and Term separately. Bounding Term Define
N N
Term A
|X - B|I
Since X is R-sub-Gaussian, Lem. 9 iimply that || is sub-exponential with || S SCR. Let Y, =x=x3, K CR², applying Bernstein's inequality (Lem. 11) gives:
t2
P -11 21) s expl -C2N min (CRed Ciea
Next, solve for to achieve the desired confidence 1 - 5 1-
Set the right-hand side equal /:
2 exp( (-22 min
(CRPd)2 CRPd CkPd)) 4'
t
t
This equation has two regimes:
2 Small t: (C
C.N log (g) = t= CRdd
log(8/6)
C2N
C2N log =1= CR²d
Large t: CIe
CaN
To unify both regimes, we define
log(8/8) C2N
t= CR
C2N
Thus, with probability at least
4. we conclude
log(8/5)
CRddlog(8/0)
Term A| s CR
C2N
C2N
2. Bounding Term B. Since X is zero-mean, we write
Term B = | ||al = 18
Applying Lem. 12 and substituting Y = X and R R/VN:
p
| 2 C3 VN
R
(v S exp(-12).
Set c = 5/4 = / Vog (48)), substituting into above inequality:
| C3 R
8
Vd + /lo(4//5) with probability at least
4
26

---

Using vd + Vlog(4/6) (4/6) - 2 + log (by Scuchy-Schwarz), we simplify:
|| S V 2(d + log 4/6))
8 4"
with probability of at least 1
N
Then immediately with probability at least
4'
d + log(#)
Term B ||| - 223 R²
N
Combining the bounds. With probability at least
2 (by a union bound), both Term and
Term B satisfy their respective bounds. Hence
N
(ur(S)-ure - 5
(/Term A + Term B|)
N
log(8/8)
ClFdlog(8/0) C2N
+ log(
2
CR
20g n²
C2.V
N
By selecting = max
C
C
20%], have
VC²
C2
1og(8/5) N
d + log(
tr(E)
tr(E) S C
R
dv
N
Choosing 8 small enough or letting N grow large makes the term less significant, so the main deviation is typically on the order of
ddv (8/(8/6)
This completes the proof.
Lemma 14 (Sub-Gaussian Radius Bound). Let X E IRd be mean-zero R-sub-Gaussian random vector with covariance matrix Σ E Rdxd For any >> with probability at least 1 2.
| S traco(E)) + CiRed (1og(2/8) + Vlog(2/8))
Proof. Since is B-sub-Gaussian, by Lem. 10, denote Y = X2 and E[Y] = μa for any > P vi) set
By identifying μ = trace(E), we arrive
p( (1111122 trace(E) + C2R vt) s et
Putting = log(?) yields
p(IXI| V trace(E) C1 CRd (log(2/8) + vlog(2/0)
8
2 1
2
This completes the proof.
Lemma 15 (Bounded Radius of Sub-Gaussian Variables). Let Z Rdd be an R-sub-Gaussian
random variable, and {2; 1== i.i.d copies Z. Denote 2 Σim 2; and
N
Ze [2] 2, Z2 Z, NN z
as the centered embedding matrix. Its singular values are d1, 02 .... 0a 2 0. For any 0 < < 1 with probability at least 8:
Σi= (Z.)
2| - E|2||
<
V(N -1)
(a)
log(8/5)
+ log( ())
C Red (log(2/8) + vlog(2/8) + +0 R2
(dy
N
N
27

---

Proof. By Lem. 14, we have:
p(1Z -[1Z-E(21] V trace(Z) + C,Rdd (log(2/0) + Vlog(2/6))
2
Denote Σ NZ Ze as the centered representation matrix. By Lem. 13, with probability of at least 1 - (with the union bound), we have:
log(8/o)
d + log( (g)
||--||Z11 <
trace Σ) C.Rdd (log(2/0)+ vlog(2/6)) + np
dv
N
With va + S Va + Vb, rearrange the terms:
|| - E|2I & v (raco($)
G RRRd (log(2/6) vlog(2/8)) + Ò R? ( (av
[og(8/0) (a/1os(s10)
d + log(
N +
Since
1 Σ
trace (2) N o. (Z.)²
o. (Ze))
Thus, with probability at least 6, we have:
|| B|2l| < Σ (Z()
(N -j
Cieted (log(2/0) (1B82(10)(2/0)+ + vlog(2/8))
+ Ô 122
(a/
dy
log(8/0)
N + + log( 3)
This completes the proof.
A.2.3 Proof of the Main Theorem
Theorem 16 (Formal version of Thm 2). Let Zi = {Z v(2) = {}} be the examples in j- th class with |Z'| Nj. Assume that for all j E [K]. Z) E Rdd an R-sub-Gaussiad random variable, and (2/) ii.d copies of Z Denote ží N EN 2 and 7 1 2) = z z 23] as the centered embedding matrix for Zd. Then, for any
N,
320:
Errp(fo) 5 P(6).
With or(.) represents the i-th singular value
P(6) P Ed (Zb(()))
((() + -C((x)
|| - mell)
min ke(KT(v(=)
Here,
(a log(8/0) No(e)
+ + log(?)
G CiRed (log(2/6) + vlog(2/5)) + Ö R2
Nyix)
is positive class-irrelevant bias term and C1, Ò are positive constants.
28

---

Proof. Using NN classifier, a representation 2 could be correctly classified if: || - My(z))l - || -
holds for all & [K])(zz)
Using the result of Lem. 14, for any class j and 5 >> 0, the expected distance of the intra-class distance is bounded with probability at least 8:
Σ (Z))
[z:y(=)=)[lz= Mu(=) |l| -
N
log(8/o) N₂
d + log
CillPd (log(2/0) + vlog(2/8))
CIP
d
N₂
For the sake of simplicity, define
1og(8/8) N
d + (
C% - Ci (og((/0)) + vlog(2/0)) + C H²
(1
N₂
For the representations in the j-th class, we separate it into two parts: the main part Z in which all examples lic in the radius, and the outside part Z = Z Zi From the concentration property, we have Z² Z Zg and Pzezs (z (%,) 1 6.
For the main part, the accuracy the NN classifier on the j-th class could be calculated by:
(NN, II) + +. S min Σ (.(5)
P PZEZi
|| - px11)
kE[K]]
Thus, the error rate of the j-th class should be least (assuming all the representations in the outside part are misclassified):
E (..) V(N, 1) KE|K]V
E 5 PZEZ)
+ +
min ||= rexll)
Rearrange the terms and take expectation on j yields the result.
A.3 Proof of Corollary 5
We first introduce some useful lemmas for the proof.
Lemma 17 Bub--Gassiian Norm Concentration [65])). For R-sub-Gaussian vectors X Rd:
t
P (ixl RVd Rvd+1) exp
2R
Lemma 18 (Intra-class Concentration (Adapted from [66])). For any R-sub-Gaussian class j with Nj samples:
1
Σ (222 < RVdd 1+
log(8/5) N,
log(8/6)
==1 VN,
N₂
holds with probability 2 5/4.
Lemma 19 (Dimensional Scaling of the Concentration Term). The concentration term Cg. admits the dimensional scaling:
d
C < R Cdloog(2/8) +
C'd
+
R
N₂
Sub-GGuuszian
Covariance
For d 2 log(8/8), this simplifies to Cg. < V3Rvd with probability 2 1 8/4
29

---

Corollary 20 (Formal Version of Corollary 5). Under Thm 2's assumptions with the condition
mink/)(()) [| p > vdR 2 + log(8/8) V3).fon for any ā > 0:
Efrp(fe) 2 2K exp (-Ōs.d)
where Cs
is a constant.
Proof. By the triangle inequality
|
uall 2 (12)(2) - Mall | - Mb(
would be correctly classified when:
|| Mu(x))l S min (o(x)) pall Ei=1 0z (z((())) kzg
CM(2)
v(No(=)~ -
Using Lemma 18 and Lemma 19, and denote A min ||2y(z) Mell/ Vd, we have:
| - Mwcell S v A R(11 (1+1))
log(8/0)
v3
For the sake simplicity, denote R (1+ Gaussian (17), know:
lo2(8/6) N,
+ v3)
By the concentration of sub-
(1= Ms(z)]l 2 RVd+1) s 2exp
2
Selecting t (A - R R) gives:
log(8/8)
V3
IP ||= M 2 Vd(A -i)) 2exp
N
lon(s/6) +v3
Denote Ö3
and apply the union bound across all classes yields:
P()) < 2K exp (-Co- d).
Immediately,
(foo 2K exp (-Ōs.d).
which completes the proof.
A.4 Analysis of the Effect of k
In the previous analysis, we assumed the pseudo-label be accurate for simplicity. In this subsection, we analyze the eтror introduced by l * C. From the original error decomposition:
Er((k) < 5 Pa(6) = 6 (a) + (Po(8) - Pa(O))
A(k)
It is easy to tell that Err(k) < 5 + Pk( - Err(k) < + Po( when A 2 0. In the next theorem, we model the relationship between A (Is) and k. Theorem 21 (Error Bound with Clustering Deviation). For any d > 0. the error bound satisfies:
Err < ê Pa(6) A(k)
where A(k) Po(8) Po(6) exhibits the following properties:
30

---

( 2 0 when k > C.
As k decreases from C to 1, (kk first increases to positive peak, then decreases to negative values.
Proof. Intuitively, the proof starts by h == C and discusses the change of ( with two cases: I) Increasing k leads to the splitring of original clusters, and 2) Decreasing merges two adjacent clusters.
Case 1: (Over--clustering) When C, since over clustering converts some false positives into true positives without affecting negative predictions, ((k) 2 naturally holds.
Case 2: k < C' (Under-clustering) Denote the class center and radius as M4 E[z|= Cyl, R, = se, || - -Hyll respectively. The distance between class centers is then defined as d, = 14 Myll. When merging C₁ and C2 into C", define the merged center and radius as μ4, and
R,, respectively.
The impact of merging depends on the separation between classes. Let C₁ and C2 be two classes with separation ratio p
d1a
We distinguish two regimes:
Ri+R,
Overlapping Merging (p 1): Classes poorly separated, with d12 small relative their radius.
Separated Merging (p 2 1): Classes are distinct, with d12 comparable to or larger than radius.
When merging two overlapping classes (p < 1), define the center and radius of the merged cluster + and R. I max(R1, R2) +
C = C, C2 as: μ. = N1+N2
min(N. N1+Na d12, respectively. For
$ C*, the minimal distance to other classes improves due to the merged center's shift:
D (k) 2 D (C) + || - Mproj]|
Gain From center shift
where Moro is the nearest original center. This increases the margin D Dim (k) R, for non-merged classes, reducing (5).
For z E C*, the radius R. remains comparable to original radius since d12 is small. The dominant effect the elimination of misclassification between C₁ and Cz. Thus, Px( < Po(o), resulting in (kk) > O.
When merging well vell-separated classes (p 2 1), the merged radius R. max(R1, R2) + d12 becomes significantly larger. For z E C":
N₂
D min (k)) R. S 1| -- -mll R. S R1
d12 d12 < 0,
N + N₂
increasing (8). For z & C", the minimal distance may decrease slightly, but the dominant effect is the inflated R., leading to (5) > (o) and (k) < 0.
From above analysis,
that as & decreases from C to 1, the trajectory of (kk) follows:
Overlapping classes are merged first (since k-means prioritizes reducing within-cluster variance). This reduces Px (8), causing A(k) 0.
Initial Mergers:
Late Merges:
Remaining classes are better separated, and merging them inflates R. significantly. (6) increases to 1, making A(k) < 0.
Since (kk) could possibly be smaller than 0 when & < C, in practice, suggest taking k equal to or slightly larger than the real number clusters to ensure the tightest bound.
31

---

B Additional Related Works
Supe-super Learning Approaches Beyond Images. In recent years, self-supervised learning has been applied to multiple modalities, including video [46, 45, 63], point clouds [81, 55], time- series data [79, 80], and cell images [18, 54]. While SSL has achieved strong results in these areas, examining whether similar fineggriieed performance degradation occurs during training is a valuable direction for future work.
Studies on Class Separability, In this work, our theoretical analysis uncovers the connection between class separabilitty and downstream performance. We clarify that this connection study; it has been widely explored in the machine learning community [22, 6, 23]. In the context of self-supervised learning, several studies have examined this relationship through the lens of alignment and uniformity in contrastive learning [67]. or through coding rate reduction [77].
unique to our
Although we do not introduce the concept of class separability, our work makes non-trivial contribu- tions by bridging the downstream performance with measurable factors. Furthermore, we propose practical methods for evaluating the quality of dense representations.
Supervised Transferability Estimation. Due to the high computational cost of transfer learning, large number of studies have emerged to estimate downstream performance without fine-tuning [33, 35, 76, 20, 19, 7. 1, 56, 42]. Early approaches focus on approximating the posterior distribution of target datasets [64, 48, 433. Many works also leverage the energy score [25] the class separability [75, 51] as the metric. While effective in supervised settings, these methods require labeled data, limiting their applicability to self-supervised scenarios.
Concurrent Studies on SSL Degradation. We also acknowledge concurrent study, DINO v3 [57], which investigates degradation in self-supervised learning. In this paper, we show that the SDD phenomenon widespread across sixteen state-of-the-art methods, datasets, and tasks, and we propose theoretically grounded metric for performance estimation, model selection, and regularization. In contrast, DINO v3 focuses on degradation within the iBOT/DINO v2 family and introduces gram-matrix distillation to address it. The findings in DINO v3 strongly support the scalability of the SDD phenomenon. Their gram-matrix loss selects an early model with hand-crafted iteration steps as the teacher for correlation distillation, which could be improved by integrating our DSE-based model selection. Thus, the techniques in DINO v3 and those in this paper are likely complementary. and DINO v3 provides valuable scaling evidence for the SDD phenomenon that we do not include here due to resource limitations.
We also note that Wen et al. [ performance degradation during training from the perspective of insufficient semantic concentration. Their semantic concentration framework effectively mitigates the degradation by improving intra-class compactness.
32

---

C Detailed Methodology for DSE-regularized Online Optimization
As mentioned earlier, the DSE metric provides a lower bound on dense performance. Since all operations involved in computing DSE are differentiable, directly optimizing DSE can potentially address the SDD problem effectively. In practice, we include DSE explicitly as regularizer in the training process:
= Coriginal - 3 DSE.
Although calculating DSE itself is model-agnostic, integrating into the training procedure requires certain model-specific adjustments. Specifically, all baseline methods in this study use Joint- Embedding Self- Supervieed Learning (JE-SSL) framework, which employs a siamese network with two global views of size 224 x 224 during pretraining. Thus, we use the student model (the encoder) extract dense representations from these global views, and then compute the DSE based on these representations. For approaches involving masked modeling, we additionally feed the unmasked views through the student model obtain dense representations.
To illustrate how DSE can be integrated into the training process, we take the training procedure of DINO [10] as example. We provide the code for the DSE regularizer class in the Supplementary Material. This allows DSE regularization to be easily integrated into any JE-SSL framework by extracting dense representations from the student model and including the DSE regularizer in the loss function.
Algorithm 2 An example Py PyTorch pseudocode of DINO with DSE-regularized training.
Is, It: student and teacher encoder
gt: atudent and teacher heads
center (K)
tps, tpt: student and teacher temperatures
1, network and center nomentun rates
a: weight DSE regularization loss
Itrears parears
gt. parems garpars
for in loader: # load minibatch with samples
auguent (x)
vievs
zi, 22 =f(xx), fe(x2) student output z-by-(p:1)-by-d
zl_cls, zl-patch = =1[:,0], 21[:,1:]
=2_cls, z2_patch =2];,0), 22[:,1:]
extract cls and patch tckens
2_patch Concat (21.pacch, 22_patch)
output n-by-X t1, t2 (x(xt (xt((xx)) teacher output n-by-X
s2 ga(z1_cls), ga(z2_cla)
(SE(z_patch)
1osa #(t1, #2)/2 H(t2, a1)/2 das loss backvard bacx-propagate
student, teacher and canter updates update(gs) # SGD gt.parans >lgtt.paran ==C (1-n) mea)
(t-1))gg..prrac
def #(t, a): detach # stop gradient softmax tps. din-1) softmax ((t. -)
dim=1)
sharpan
return (t log(s)). sun (din-i-))) uean))).uean()
33

---

D Detailed Settings
D.1 Pretraining
For pretraining, we reimplement all methods based on their original settings, but we disable automatic mixed precision (AMP) in I-JEPA [3] to prevent training instability. All models are trained for 800 epochs on ImageNet- [39], except for SwAV [9], VICReg [4], VICRegL [5], and DenseCL [70]. We observe model collapse when training SwAV for more epochs, and take the default settings for
DenseCL, VICReg and VICRegL. The full pretraining hyperparameters are listed in Tab. 4.
Table 4: The hyperparameters used for pretraining.
Method
Architecture
Learning Rate
Optimizer
Warm Epochs
Epochs
Batch size
Imsge size
[16]
VIT-Small-Bb 16
1.5c4
224
DenseCL. [70]
ResNer-50
0.03
SGD
200
256
2 224
BYOL [26]
ReaNat-50
0.2
LARS
10
800
1024
2 22
SimSium [14]
ResNer.50
10
224
EsVIT 1411
Swin Swin-Tiay-7
Se-4
AdamW
800
1024
2 22
MEC
ResNer-50
0.5
SGD
10
800
512
224
Barlow Twins [78]
ResNet-50
0.2
LARS
10
800
2048
2 224
VICReg [4]
ResNer-50
0.2
LARS
10
1000
2048
2 224
VICRgL
ResNct-0
0.2
224
SwAV []]
ResNer-50
0.6
SGD
400
512
2 2142
6-6 96P
DINO [10]
VIT-Small-16 16
5c-4
AdamW
10
800
1024
2 221"
10 x 96
iBOT
VTT-Small-16
2 224
Muga [8]]
VIT-Small-16 16
Rc-4
AdomW
10
800
1024
2 221
10 x 96
ReSA [22)
ResNet.50
0.5
SGD
800
1024
2 224
MAE [29]
VIT-Small-16 16
1.5e4
AdamW
40
800
4096
221
LJERA [3]
VIT-Base- 16
le-3
AdamW
15
800
2048
224
For readers' convenience, we provide a brief introduction to these methods as follows:
MoCo v3 [16]: Improves contrastive learning stability and performance by combining
MoCo [28] with ViTs.
DenseCL. [70]: Enhances self-super learning by applying contrastive loss [49] the dense feature level, enabling better dense-level representation learning.
Explores non-contrastive SSL with an asymmetric design including an extra prediction head and a stop-gradient mechanism to avoid collapse.
BYOL [26]:
SimSiam [14]: Proposes simple siamese network for celf-supervised learning without negative samples, relying on stop-gradient and predictor mechanisms to avoid collapsing solutions.
ESVIT [41]: Enhances self- supervieedd ViT training by combining masked image modeling with contrastive learning for improved efficiency and scalability.
MEC [44]: Introduces the idea of maximum entropy encoding that explicitly optimizes on the structure of the representations, leading to generalizable representations.
Barlow Twins [78]: Uses an invariance term and a redundancy-reduuction term to optimize representation structure, effectively preventing collapse.
VICReg [4]: Builds on Barlow Twins by adding variance regularization loss to keep the representation space well spread.
VICRegL [5]: Extends VICReg local features, learning strong dense representations while maintaining image-level quality; it also shows a trade-off between coarse and fine-grained performance.
SwAV [9] Combines online clustering with SSL, using swapped prediction between cluster assignments to learn meaningful representations without pairwise comparisons. Assigning representations clusters naturally enhances the class separability.
DINO [10]: Leverages self-distillation with ViTs and teacher-student framework to achieve strong instance-level representations.
iBOT [83]: Integrates masked image modeling with self-distillation in features space of ViTs, enabling joint learning of global and local visual representations.
For BYOL, the pytorch implementation from tattts::/ggtt com/stthalles/PPYorcet
34

---

Mugs [84]: Proposes multi granularity discriminative framework that performs well on both instance-level and dense prediction tasks.
ReSA [72]] Improves clustering based SSL by learning cluster assignments directly from encoder features, removing the need for prototypes.
MAE [29]: Introduces simple and scalable masked autoencoder framework for self- supervised learning, reconstructing masked patches from visible ones.
I-JEPA [3] Proposes a joint-embedding predictive architecture for self-supervised learning,
focusing predicting representations of masked regions in abstract latent space.
D.2
Evaluation
Linear Semantic Segmentation. To assess dense representation quality, we adopt the standard linear evaluation protocol standard in SSL [50, 86, 83, 84]. We evaluate four benchmarks: COCO-Stuf127 [8]. PASCAL VOC [21], ADE20K [82], and Cityscapes [17]. We remove projectors, and train only a lightweight classifier dense features. For all models, we use the last layer's patch embeddings. Input images are resized to 336 x 336 (except 896 x 896 for Cityscapes to preserve detail) and the classification head is trained on 100,000 images. The head is optimized using batch size of 256 (64 for Cityscapes due to GPU memory limitation) and learning rate of 0.01 v batch size/25c using an Adam [38] optimizer.
In the linear probing setup, the backbone is kept fixed, and only a small classifier is trained using the dense features. the linear transfer learning setup, the backbone fine-tuned with lower learning rate.
Semi-supervised Video Object Segmentation. We evaluate the semi-supervised video object segmentation on the DAVIS 2017 dataset [52] following the [10]. During testing, the labels of the first frame are given, and we propagate predictions to subsequent frames via k-NN similarity matching of dense features.
k-NN Image Classification. (for the motivation) Following the standard k-NN setting in DINO [10], we apply k--N classification using class tokens (or averaged patch tokens if no class token exists) mapped to ImageNet-Ik labels.
D.3 Kendall's r Coefficient
We measure the correlation between our metric and downstream performance using Kendall's coefficient, which is a standard metric for transferability analysis [76, 25, 75]:
2
Σ ISICIEN
sign(M, My)siin((P P2).
N(N-N)
Here, N is the number of checkpoints, M,, P. denote the value of metrics and downstream perfor- mance calculated using i-th checkpoint, respectively. When r = 1, the proposed metric is perfectly aligned with the downstream performance, and -1 represents that the proposed metric is inversely correlated with the downstream performance.
D.4 Details in DSE Calculation
For metric calculation, 1k), and no extra data is introduced for performance estimation. As standard testing protocol, the images first resized to 256 x 256, and then center cropped into 224 224 and normalized. For pscudo-label generation, the k-means clustering is applied. To improve robustness, we compute Mintra by averaging the results obtained by B 1 and B 8. We set k and A 24 for B = 1 and B = 8, respectively. Sensitivity analyses of these parameters are discussed in later sections.
2048 randomly sampled images from the pertaining dataset (ImageNet-
D.5
Computational Resources
All pretraining are conducted on 8 NVIDIA A A100 GPUs, the evaluation is done on 8 NVIDIA 4090 GPUs, and the DSE metric computed on single NVIDIA 4090 GPU. The results in Tab. are obtained by averaging the time cost of 5 runs the 4090 GPU.
35

---

E
Additional Experiment Results of the SDD Phenomenon
E.1
The SDD Phenomenon Exists Across Datasets
To comprehensively analyze the SDD phenomenon, we visualize training curves for the state-of- the-art methods the PASCAL VOC, COCO-Stuff, ADE20K, and Cityscapes datasets in Fig. Fig. 9, Fig. 10, and Fig. 11, respectively. The SDD phenomenon persists consistently across all datasets, with training curves exhibiting similar degradation patterns. These results confirm that SDD is dataset-agnostic and closely linked to dense representation quality.
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(e) SWAV [9]
(f) DINO [10]
EsVIT [41]
(h) iBOT [83]
(i) MEC [44]
(i) VICRegL [5]
(k) Mugs [84]
(I) ReSA [72]
(m) MAE 29]
(n) I-JEPA [3]
Figure 8: The SDD phenomenon on PASCAL VOC.
Crin
Eown
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(c) SwAV [9]
( DINO [10]
(g) EsVIT [41]
(h) iBOT [83]
MEC [44]
(i) VICRegL. [5]
(k) Mugs [84]
(1) ReSA [72]
(m) MAE [29]
(n) I-JEPA [3]
Figure 9: The SDD phenomenon on COCO-Stuff.
36

---

fax
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(c) SWAV [9]
(f) DINO [10]
(g) EsVIT [41]
(h) iBOT [83]
(i) MEC [44]
(i) VICRegL [5]
(k) Mugs [84]
(1) ReSA [72]
(m) MAE [29]
(n) I-JEPA [3
Figure 10: The SDD phenomenon ADE20K.
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(e) SwAV [9]
(f) DINO [10]
EsVIT [41]
(h) iBOT [83]
(i) MEC [44]
(j) VICRegL [5]
(k) Mugs [84]
(I) ReSA [72]
(m) MAE [29]
(n) I-JEPA [3]
Figure 11: The SDD phenomenon on Cityscapes.
37

---

E.2
The SDD Phenomenon Exists Under Varying Evaluation Protocols
E.2.1
The SDD Phenomenon Exists when Backbone is Not Frozen.
To examine how changes to the backbone during fine-tuning influence downstream performance, we evaluated checkpoints where the backbone was unfrozen and present the results in Tab. 5. The findings reveal a pattern similar to that seen in the linear probing setting. with an average decrease of 2.9% in mloU. While fine-tuning the backbone notably improves downstream performance, the decline becomes less pronounced as the backbone continues to adapt during training. Nevertheless, similar trend to the fixedbbacboone setting remains, further supporting the existence of the SDD phenomenon.
Table 5: A performance gap between the best and the last models is present across all datasets and methods when backhone is not frozen.
COCO-Stuff
PASCAL VOC
ADE20k
Method Type
Method
Architecture
Best
Last
Diff
Best
Last
Diff'
Best
Last
Diff'
MoCo v3 [16]
VIT-Small-16
39.6
35.4
-4.2
57.3
18.1
-39.2
21.0
6.1
-14,9
Contrastive
DenseCL [70]
ResNet-50
43.2
42.4
-0.8
62.3
61.8
-0.5
25.0
24.8
-0.2
MEC [44]
ResNet-50
41.4
41.2
-0.2
57.8
57.6
-0.2
22.0
22.0
0.0
SimSiam [14]
ResNet-50
42.7
42.6
-0.1
60.0
59.9
-0.1
22.5
22.5
SwAV [9]
ResNet-50
38.8
55.3
55.1
-0.2
20.1
20.0
Non-Contrastive
DINO
ViT-Small-116
42.6
41.0
-1.6
62.0
60.6
-1.4
25.6
25.5
EVVIT [41]
Swin-Tiny-7
42.9
39.5
-3.4
62.5
56.0
-6.5
25.4
22.2
-3.2
BOT [33
VIT-Small-16
46.9
44.9
-2.0
69.7
68.2
-1.5
29.1
28.1
-1.0
Masked Modeling
MAE 22
VTT-Small-16
39.1
39.0
-0.1
55.3
54.9
-0.4
19.9
19.6
-0.3
I-JEPA
VIT-Base-16
46.1
44.8
-1.3
70.0
67.5
-2.5
29.5
28.1
-1.4
E.2.2 The SDD Phenomenon Exists in Varying Evaluation Hyperparameters.
To investigate whether SDD stems from specific evaluation settings, we test different learning rates and fine-tuning durations for downstream tasks. As shown in Fig. 12 and Fig. 13, performance degradation trends remain consistent across hyperparameter configurations. This demonstrates that SDD is not an artifact of specific evaluation hyperparameters but reflects a general correlation with representation quality.
1m=0.005
1r=0.01
Ir=0.00
20
Coach
Epeen
Epech
Opoch
(a) MoCo v3 [16]
(b) DenseCI. [70]
(c) MEC [44]
(d) SimSiam [14]
60.0
3.4
50.0
Ep0c1
Epach
(c) SWAV [9]
(f) DINO [10]
(g) ESVIT [41]
(h) iBOT
From
(i) MAE [29]
(i) I-JEPA [3]
Figure 12: The SDD phenomenon consistently exists across different learning rates.
38

---

50k training images
100k training images
150k training images
E100
teeth
t50ch
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) MEC [44]
(d) SimSiam [14]
2.40
50.0
21.0
tpoc
(e) SwAV [9]
(1) DINO [10]
(g) VsV [41]
(h) iBOT [83]
Epec1
MAE [29]
(i I-JEPA [3]
Figure 13: The SDD phenomenon consistently exists across different fine-tuning iterations.
E,3 The SDD Phenomenon not Caused by Dataset Overfitting
To examine whether SDD is caused by dataset overfitting, we train DINO [10] and evaluate its performance on the same COCO-Stuff dataset. All settings are kept identical to those used for training on ImageNet, except for the training dataset.
As shown Tab. 6, the model exhibits similar performance degradation when pretrained on ImageNet. This results in large gap between the best and final checkpoints, suggesting that SDD is due to dataset overfitting.
Table 6: The SDD phenomenon when training and evaluating on the same COCO dataset.
Method
Best
Last
Diff
DINO [10] 36.3
32.3
-4.0
E.4 The SDD Phenomenon Exists in Varying Downstream Tasks
We evaluate semi-supervised video object segmentation on DAVIS-2011 [52] using the protocol from [10]. Due to the limited performance of non-ViT architectures, we focus VIT models. Results in Tab. 7 reveal persistent performance gaps between optimal and the last checkpoints, confirming that SDD generalizes to diverse dense downstream tasks (More results depth estimation are provided in Appendix F.1).
Table 7: SDD phenomenon of Video object segmentation task DAVIS-2017 dataset.
J&F
Imean Last
F Fmenn
Method Type
Method
Architecture
Best
Last
Diff
Best
Diff
Best
Diff
Contrastive
MoCo [16]
VIT-Small-16
61.5
60.7
-0.8
59.6
59.0
-0.6
63.3
62.5
-0.8
DINO [0]
VrT-Small-16
61.9
61.8
40.1
60.2
59.8
-0.4
63.8
63.7
-0.1
Non-Contrastive
iBOT [83]
ViT-Small-16
62.4
61.6
-0.8
61.1
60.3
-0.8
64.3
62.9
-1.4
MAE [29
ViT-Small-16
49.2
41.2
-8.0
48.7
39.7
-9.0
49.7
42.8
-6.9
Masked Modeling
I-JEPA [3]
VIT-Base-16
59.6
54.5
-5.1
58.7
53.6
-5.1
60.6
55.4
-5.2
39

---

Additional Experiment Results of the DSE Metric
F.1 DSE Metric is Precise Estimator for Depth Estimation Task
We further examine whether SDD is unique to semantic segmentation or affects other dense tasks. First, We conduct an additional experiments on depth estimation. Similar to linear probing setting, we freeze the model and train linear layer to predict the depth value on NYU-deethh v2 dataset [47]. As shown in Tab. 8, the SDD phenomenon persists in depth estimation task. The results validates that SDD is a general phenomenon that not specific to segmentation task.
Next, to see if the proposed DSE metric works well on depth estimation task, we also compute the Kendall's т coefficient between the DSE metric and depth estimation performance. As shown Tab. 9, metric positively correlates with the depth estimation performance, and DSE-based model selection consistently improves model performance. While these results demonstrate the effectiveness of DSE across different downstream tasks, note that the Kendall's coefficient is relatively lower compared with the segmentation task. The reasons are two-fold: 1) Our DSE metric is derived from the class-relevant performance, the flass-seperabiliee may not be fully useful for depth estimation task. 2) As a regression task, the RMSE curve quakes, making hard to predict. We would like to further imporve the DSE metric strengthen its capability depth estimation task.
Table The SDD phenomenon in depth estimation task. We report the RMSE metric (lower better) on NYU-depth v2 dataset.
Method
Best |
Last +
Difference
MoCo v3 [16]
0.638
1.589
0.951
DenseCL [70]
0.547
0.559
0.012
SimSiam [14]
0.597
0,608
0.011
SWAV [9]
0.705
0.725
0.020
DINO [10]
0.515
0.553
0.038
MAE [29]
0.680
0.775
0.095
I-JEPA [3]
0.460
0.487
0.027
Table 9: Left: Kendall's r coefficient between DSE and RMSE on depth estimation. Right:
DSE-based model selection results. We report RMSE metric with the improvement compared the last epoch.
Method
Kendall's
Method
RMSE
MoCo v3 [16]
0.452
MoCo v3 [16]
0.638 (-0.951)
DenseCL. [70]
0.752
DenseC1. [70]
0.547 (-0.012)
SimSiam [14]
0.659
SimSiam [14]
0.598 (-0.010)
SwAV [9]
0.247
SwAV [9]
0.714 (-0.009)
DINO [10]
0.071
DINO [10]
0.532 (-0.021)
MAE [29]
0.107
MAE [29]
0.709 (-0.066)
I-JEPA [3
0.190
I-JEPA [3]
0.479 (-0.008)
F.2 DSE Precisely Indicates Performance Trends Across Unseen Datasets
To comprehensively analyze the relationship between DSE and downstream performance, we present full results across all datasets and methods. These results confirm that DSE consistently identifies performance trends and peaks, establishing it as reliable metric for model selection.
40

---

mIoU on PASCAL VOC
The proposed DSE metric
6001
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(e) SWAV [9]
(f) DINO [10]
(g) EsVIT [41]
(h) iBOT [83]
ii MEC [44]
i VICRegL [5]
6001
(k) Mugs [84]
(1) ReSA [72]
(m) MAE [29]
(n) I-JEPA [3]
Figure 14: The proposed DSE metric precisely predicts the downstream performance on the PASCAL
VOC dataset.
mloU on COCO-Stuff
proposed metric
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(e) SwAV [9]
(1 DINO [10]
(g) EsVIT [41]
(h) iBOT [83]
MEC [44]
i VICRegL [5]
(k) Mugs [84]
(1 ReSA 722
(m) MAE [29]
(n) I-JEPA [3]
Figure 15: The proposed DSE metric precisely predicts the downstream performance on the COCO-
Stuff dataset.
41

---

mloU on ADE20k
The propused DSE metric
6001
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(e) SWAV [9]
(1) DINO [10]
(g) EsVIT [41]
(h) iBOT [83]
MEC [44]
(i VICRegL [5]
6001
(k) Mugs [84]
(I) ReSA [72]
(m) MAE [29]
(n) I-JEPA [3]
Figure 16: The proposed DSE metric precisely predicts the downstream performance on the ADE20k dataset.
mloU on Cityscapes
The proposed DSE metric
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) BYOL [26]
(d) SimSiam [14]
(e) SwAV [9]
DINO [10]
(g) ESVIT [41]
(h) iBOT [83]
MEC [44]
ii VICRegL [5]
(k) Mugs [84]
(1) ReSA [72]
(m) MAE [29]
(n) I-JEPA [3]
Figure 17: The proposed DSE metric precisely predicts the downstream performance on the
Cityscapes dataset.
F.3
Ablation Study of Different Components.
As shown in Tab. 10, both the class-separability and dimensionality measures exhibit a positive correlation with downstream performance, supporting our theoretical findings. Combining these components further enhances Kendall's coefficient. Notably, when only Maim is used (Line 2 in Tab. 10), is equivalent adapting the RankMe metric [24] dense representations. These results highlight DSE's superior ability to assess dense representation quality.
F.4 Sensitivity Analysis on Number of Images and Clusters
While DSE itself does not explicitly depend on hyperparameters, pseudo-label generation introduces unavoidable parameters. We analyze their sensitivity assess whether DSE's accuracy is contingent on specific choices.
42

---

Table 10: Ablation study of different components of DSE. We report the average Kendall's
coefficient across different methods.
Minter Mintra
Mdäm
COCO
VOC
ADE
City
Avg
0,45
0.42
0.33
0.37
0.39
0,25
0.26
0.22
0.23
0.24
0.58
0.60
0.56
0.49
0.57
Effect of the number of images. By default, we only use small amount data (2048 images. 0.16% of the training dataset). This is the minimum batch size B' to meet the requirement of B' d in order obtain B' independent representations. In this part, we test whether this limited sample size suffices for robust DSE estimation. Fig. 18 shows no significant change in DSE with larger datasets, confirming that 2,048 images suffice for accurate performance estimation.
2048 images
7168 images
13312 images
26624 images
0.8
0.0
1.4
0.0
080
20
105
(a) MoCo v3 [16]
(b) DenseCT. [70]
MEC [44]
SimSiam [14]
2230
1007
Eachh
Epac
(c) SWAV [9]
(f) DINO [10]
(g) ExVIT [41]
(h) iBOT [83]
300
30
150
MAE [29]
I-JEPA [3]
Figure 18: Sensitivity analysis different numbers of images used for the DSE calculation.
Efffet of the number of clusters. In this part, we examine the impact of the choice of k. The number of clusters is a key parameter for estimating representation quality because the actual number of classes in an image is unknown without labels. As result, the estimated class-related metrics may be biased the number of clusters does not match the true number of classes. Based on the earlier theoretical analysis, we recommend choosing & slightly larger than the expected number of classes. Since ImageNet is curated, we assume that the average number of classes per image (including the background one class) is about 2 or 3. Therefore, we explore the effect of setting l between 3 and Fig. shows that while absolute DSE values vary with k. performance trends remain consistent, preserving DSE's utility of predicting the relative performance between models.
43

---

3 clusters
4 clusters
clusters
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) MEC [44]
(d) SimSiam [14]
(c) SWAV 19]
DINO [10]
(g) ESVIT [41]
(h) iBOT
(i) MAE [29]
(i I-JEPA [3]
Figure 19: Sensitivity analysis of different numbers of clusters k.
F55
Analysis of Bias Introduced by Data Distribution Shift and Label Estimation Error
When estimating downstream performance, two types of bias are introduced: 1) Data distribution shift-DSt is computed on the training dataset (i.e., ImageNet), while the target performance is evaluated on testing sets (e.g., COCO, VOC). 2) Label estimation error-DSE relies pseudo-labels generated by the k-means algorithm, which may introduce bias during label assignment.
We conduct two groups of experiments to analyze their impact: 1) DSE calculated on the training set, which serves as the default setting for metric computation, and DSE calculated on the testing set using ground truth labels. As shown in Fig. 21, Fig. 20, and Fig. 22 although systematic errors are present, these biases act as shifting factors rather than altering the overall trend of Minter, Mintras and M Consequently, the estimated DSE metric remains reliable for comparing model performance.
Estimated Minter metric
Real inter-class distance calculated with testing data and label
2600
00
088
696
(a) MoCo v3 [16]
(b) MEC [44]
(c) SimSiam [14]
(d) DINO [10]
3.0
430
20
(0) ESVIT [41]
(f) IBOT [83]
(g) MAE [29]
I-JEPA [3]
Figure 20: Comparison between the estimated Minter and the real inter-class distance.
44

---

Estimated Mintra metric
Real intra-class radius calculated with testing data and label
0.9
20 20
40
400
30
200
190
20
(a) MoCo v3 [16]
(b) MEC [44]
(c) SimSiam [14]
(d) DINO [10]
1.20
0.4
14
(e) ESVIT [41]
IOOT [83]
( MAE [2]]
(h) I-JEPA
[3]
Figure 21: Comparison between the estimated Mintra and the real intra-class radius.
Estimated Maim metric
Real effective rank calculated with testing data
6.2
0.4
300
805
(a) MoCo v3 [16]
(b) MEC [44]
(c) SimSiam [14]
(d) DINO [10]
0.00
6.00
6:00
6300
300
(e) ESVIT [41]
(i) IBOT [83]
(
MAE [29]
(h) I-JEPA [3]
Figure 22: Comparison between the estimated Maim and the real effective rank.
45

---

F.6
Understanding the Reason behind SDD Phenomenon
DSE not only measures downstream performance but also helps explain the causes of SDD. The behavior of different components of DSE offers insights into SDD. For instance, Fig. 23 shows that DINO's performance drop at 300 epochs coincides with slower reduction in intra-class distance compared to inter-class distance, which decreases class separability. In contrast, MoCo v3's degra- dation is associated with a collapse the dimensionality of dense features, making downstream separation more difficult. These findings offer deeper understanding of the training dynamics in SSL methods and provide useful guidance for improving algorithm design.
Mintra
Minter
Mdmi
DSE
PASCAL VOC mloU
(a) MoCo v3 [16]
(b) DenseCL [70]
(c) MEC [44]
(d) SimSiam [14]
03
52
(c) SWAV [9]
(f) DINO [10]
(g) EsVIT [41]
(h) iBOT [83)
(i) MAE [29]
I-JEPA [3]
Figure 23: Visualization of different components of DSE metric, which reveals the reason behind
SDD phenomenon.
F.7 Extending the DSE Metric toward Image-level Performance Estimation
Technically, our analysis should hold for both dense and image-level settings, as image-level learning can be viewed special case where an image is only split into one patch. To investigate how DSE performs at the image level, we conducted the following experiments. DSE originally accepts input aattn__ach_ttoeeeee
tensors shaped as "Num_images first extract the class token (or averaged dense tokens), obtaining a tensor of shape (Num images, Dimensionality). Then, we resize this tensor to (Num images / 200, 200, Dimensionality) and caleu- late the DSE metric. For evaluation, we computed Kendall's 7 coefficient between the adapted DSE and ImageNet k-NN performance, comparing with the state-of-the -ott----ttt--tttt unsupervised transferability estimation method RankMe. The results are shown in the following table:
Dimensionality). For image-level tasks, we
Table 11: Kendall's т coefficient between metrics and ImageNet l-NN performance.
Method
DINO
MEC
Es
iBOT
I-JEPA
Average
RankMe
0.57
0.81
0.71
0.93
0.90
0.79
DSE(Ourr)
0.87
0.90
0.92
0.86
0.73
0.86
46

---

F88 Visualizations
(a) MoCo v3 [16] last model
(b) MoCo v3 [16] selected model
(c) I-JEPA [3]
last model
(d) I-JEPA [3]
selected model
Figure 24: The selected model achieves better intra-class alignment and intra-class separatebility compared with the last model.
Furthermore, we visualize the t-SNE plot and representation similarity matrix of the and the selected model. The inputs are sorted by class labels thus ideal similarity matrix should be block diagonal. As spotted Fig. 24 and Fig. 25, the selected models have better intra-class alignment and larger inter-class distance, which means that it would be easier for the downstream classifier to separate the representations. The results well support our theoretical and empirical conclusions.
47

---

(a) MoCo v3 [16] last model
(b) MoCo v3 [16] selected model
(c) DINO [10] last model
(d) DINO [10] selected model
Figure 25: The selected model achieves better intra-class alignment and intra-class separatebility compared with the last model.
48

---

F9
Ablation Study of DSE Regularization
To evaluate the effectiveness of different components of DSE, we conduct an additional ablation study in Tab. 12. We draw three main conclusions from these results:
Different components DSE separately contribute to improving class separability and effective dimensionality. Consequently, this improves downstream performance, which aligns with our theoretical analysis of the DSE metric.
The performance improvement mainly comes from addressing model-specific degradation. For example, DINO's performance improves primarily due to enhanced class separability, whereas resolving dimensional collapse notably boosts dense-task performance in MoCo v3. These observations align with our model-specific analysis presented in Sec. 5.2.
When both factors are used together, class separability often dominates optimization. Thus, introducing hyperparameter to balance these factors may further enhance performance.
Table 12: Ablation studies of different components of DSE regularization. We report Minter Mintra Mdim, and mloU on VOC dataset.
Method
M Minter Mintro
M
VOC mloU
DINO [10]
-1.221
0.863
56.6
DINO [10]+ Maim
-1,210 (+0.011)
0.884 (+0.021)
56.9 (+0.3)
DINO [10]+ Mais
-1.103 (+0.118)
0.851 (-0.012)
57.4 (+0.8)
DINO [10]+ Maim + Mots (DSE)
-1.115 (+0.106)
0.865 (+0.002)
57.8 (+1.2)
MoCo v3 [16]
-0.958
0.744
49.1
MoCo v3 [16]+ Masm
-1.021 (-0.063)
0.892 (+0. 148)
52.3 (+3.2)
MoCo v3 [16]+ Mats
collapsed
MoCo v3 [[6]+ Maim Mets (DSE)
-0.867 (+0.091)
0.752 (+0.008)
52.1 (+3.0)
49

---

G
Additional Discussions
Limitations. This paper's theoretical analysis mainly focuses on the linear probing setting and does not account for potential distributional shifts during transfer learning. When the backbone is fine-tuned for more iterations on data that differs significantly from the pre-training distribution, the estimated performance may become inaccurate.
Future Works Several prior studies supervised transferability estimation have addressed related issues [33, 35, 76, 20, 19, 7, 1, 56, 42]. Extending the proposed DSE metric to such scenarios would be interesting direction for future research.
would also be valuable to investigate model-specific causes of dense degradation. For example, understanding why dimensional collapse occurs in MoCo v3 or what leads to separability degradation in DINO could provide deeper insights.
Extensions. Although the proposed DSE metric is primarily designed to address the SDD phe- nomenon, and we introduce two methods to reduce its negative impact, its applications are limited to this context. For example, image-level tasks can be viewed as special case of dense tasks, where the number of patches is one. such cases, our theoretical analysis and the DSE metric can be applied directly.
50


Reviews:
[{"review_id": "TbCsXsy7NU", "reviewer": "Reviewer_fV6N", "summary": "This paper introduces a degradation phenomenon in self-supervised learning called Self-supervised Dense Degradation. It shows that longer pretraining often reduces performance on dense prediction tasks. To address this, the authors propose the Dense Representation Quality Estimator. This metric combines class separability and feature dimensionality to estimate representation quality. It is used for model selection and training regularization. The method is tested on multiple SSL models and shows consistent improvements in semantic segmentation.", "questions": "- Can you provide component-wise ablations showing the effect of each DQE term in regularization?\n\n- Have you evaluated DQE on other dense tasks beyond segmentation?\n\n- Can you provide qualitative feature visualizations to confirm that DQE values align with actual feature structure?\n\n- How does DQE behave when applied to image-level representations such as classification?\n\n- Why do collapse and separability degradation occur in dense tasks but not in image-level tasks under SSL?\n\n- What do you believe causes the remaining uncertainty in DQE’s correlation with downstream performance?", "limitations": "The authors do not discuss key limitations. While it proposes a metric to detect and mitigate SDD, it does not explore the underlying causes of SDD. While societal impacts are not discussed, no notable concerns are identified.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper identifies an underexplored phenomenon in dense SSL training.\n\n- The proposed metric is simple, theoretically motivated, and easy to apply.\n\n- DQE enables effective model selection and training-time regularization.\n\n- Experimental results show consistent improvements in segmentation performance.\n\nWeaknesses\n\n- The paper lacks ablation studies showing the effect of each DQE component when used as a regularizer.\n\n- All downstream experiments are limited to semantic segmentation. The metric's generality to other dense tasks such as object detection or depth estimation remains untested.\n\n- The metric's effectiveness is evaluated only through numerical scores. Since DQE assumes structure via spectral norms and class separability, qualitative visualizations such as t-SNE or PCA are needed to validate its interpretability, especially under collapse.\n\n- Although DQE is positioned as a dense-specific metric, its formulation may apply to image-level representations. The paper does not explore this possibility or analyze why degradation is observed only in dense tasks.", "was_revised": true, "final_justification": "The paper clearly identifies and addresses the SDD phenomenon in semantic segmentation, proposing a theoretically grounded and empirically validated metric (DQE) that improves performance through model selection and regularization. While the authors demonstrate potential for broader dense tasks, limitations remain for regression-based and certain instance-level settings. With the scope explicitly limited to semantic segmentation, the contributions are solid, the results are convincing, and the work merits a borderline accept.", "rebuttal": "We sincerely appreciate your detailed and constructive comments, which significantly help us improve the quality of our manuscript. Below are our detailed responses:\n\n**Q1:** The paper lacks ablation studies showing the effect of each DQE component when used as a regularizer.\n\n**R1:** Thank you for your insightful suggestion. Below are the ablation results for the DQE regularization:\n\n**Table 1:** Ablation studies of different components of DQE regularization.\n\n| Method    |$M_{inter} - M_{intra}$ | $M_{dim}$| VOC mIoU | \n|-|-|-|-|\n| DINO| -1.221 | 0.863| 56.6 | \n| DINO + Dim | -1.210 (+0.011) | 0.884 (+0.021) | 56.9 (+0.3) |\n| DINO + Cls | -1.103(+0.118) | 0.851 (-0.012)  | 57.4 (+0.8) |\n| DINO + Dim + Cls (DQE)| -1.115 (+0.106)| 0.865 (+0.002) |57.8 (+1.2)| \n| MoCo v3 |-0.958 |0.744| 49.1 | \n| MoCo v3 + Dim | -1.021 (-0.063)| 0.892 (+0.148)| 52.3 (+3.2)  |\n| MoCo v3 + Cls | collapsed | collapsed | collapsed |\n| MoCo v3 + Dim + Cls (DQE)|-0.867 (+0.091)| 0.752(+0.008) |52.1 (+3.0) |\n\nWe draw three main conclusions from these results:\n\n1. Different components of DQE separately contribute to improving class separability and effective dimensionality. Consequently, this improves downstream performance, which aligns with our theoretical analysis of the DQE metric.\n2. The performance improvement mainly comes from addressing model-specific degradation. For example, DINO's performance improves primarily due to enhanced class separability, whereas resolving dimensional collapse notably boosts dense-task performance in MoCo v3. These observations align with our model-specific analysis presented in Sec. 5.2.\n3. When both factors are used together, class separability often dominates optimization. Thus, introducing a hyperparameter to balance these factors may further enhance performance.\n\n**Q2:** Evaluate on other dense tasks.\n\n**R2:** We conducted additional experiments on the depth estimation task. Similar to our linear probing setup, we froze the model and trained a linear layer to predict depth values on the NYU-depth v2 dataset. Results are shown in Table 2.\n\n**Table 2:** The SDD phenomenon in the depth estimation task, we report the RMSE metric (lower is better) on the NYU-depth v2 dataset.\n\n| Method | Best $\\downarrow$ | Last $\\downarrow$ | Difference |\n|-|-|-|-|\n| MoCo v3 | 0.638 | 1.589 | 0.951 |\n| DenseCL | 0.547 | 0.559 | 0.012 |\n| SimSiam | 0.597 | 0.608 | 0.011 |\n| SwAV | 0.705 | 0.725 | 0.020 |\n| DINO | 0.515 | 0.553 | 0.038 |\n| MAE | 0.680 | 0.775 | 0.095 |\n| I-JEPA | 0.460 | 0.487 | 0.027 |\n\n**As shown in Table 2, the SDD phenomenon persists in depth estimation tasks.** The performance trends are similar to segmentation tasks, validating that SDD is a general issue, not specific to segmentation. Although we cannot show the performance curves due to NeurIPS 2025 policy, we promise that these results will be included in the final version.\n\nNext, to evaluate if the DQE metric is effective for depth estimation, we computed Kendall's $\\tau$ coefficient between the DQE metric and depth estimation performance. Results are shown in Table 3:\n\n**Table 3:** Kendall's $\\tau$ coefficient between DQE and RMSE metrics, since RMSE is lower the better, we multiply the RMSE by -1 before computing Kendall's $\\tau$.\n\n| Method  | Kendall's $\\tau$|\n|-|-|\n| MoCo v3 | 0.452|\n| DenseCL | 0.752|\n| SimSiam | 0.659|\n| SwAV    | 0.247|\n| DINO    | 0.071|\n| MAE     | 0.107|\n| I-JEPA  | 0.190|\n\n**As shown in Table 3, the DQE metric positively correlates with depth estimation performance.** Although the results confirm DQE's effectiveness across various downstream tasks, the Kendall's $\\tau$ coefficient is lower compared to segmentation tasks. We attribute this to two reasons:\n\n1. The DQE metric is derived from segmentation performance, the class separability measure may be less relevant for depth estimation.\n2. As a regression task, the RMSE curve fluctuates, making predictions challenging. We aim to further refine the DQE metric for better performance on depth estimation.\n\nWe also study the effectiveness of DQE-based model selection on the depth estimation task:\n\n**Table 4:** DQE-based model selection results for the depth estimation task.\n\n| Method  | RMSE $\\downarrow$|\n|-|-|\n| MoCo v3 | 0.638 (-0.951) |\n| DenseCL | 0.547 (-0.012) |\n| SimSiam | 0.598 (-0.010) |\n| SwAV    | 0.714 (-0.009) |\n| DINO    | 0.532 (-0.021) |\n| MAE     | 0.709 (-0.066) |\n| I-JEPA  | 0.479 (-0.008) |\n\nAs shown in Table 4, DQE-based model selection consistently enhances model performance, demonstrating DQE's generalizability across different tasks.\n\n**W3, Q3:** Qualitative results.\n\n**R3:** Thank you for this valuable suggestion. To visualize the dimensional collapse, we project the representations to the principal components. Then, we divide the dimensions into 8 segments based on the eigenvalues and perform t-SNE visualization for each segment separately. We observed that the top principal components align well with class separability, while lower components rapidly collapse into single points as the dimensionality decreases. These results validate the consistency between class-separability and dimensionality.\nComparing the last and selected models, the selected models showed more compact intra-class and larger inter-class distances, indicating better class separability. Due to NeurIPS 2025 policy restrictions, figures cannot be presented here, but we will include them in the final version.\n\n**W4, Q4:** How does DQE behave with image-level representations, such as classification?\n\n**R4:** We appreciate your constructive comment! Technically, the analysis should hold for both dense and image-level settings, as image-level learning can be viewed as a special case where an image is only split into one patch. To investigate how DQE performs at the image level, we conducted the following experiments. DQE originally accepts input tensors shaped as (Num_images, Num_patch_tokens, Dimensionality). For image-level tasks, we first extract the class token (or averaged dense tokens), obtaining a tensor of shape (Num_images, Dimensionality). Then, we resize this tensor to (Num_images // 200, 200, Dimensionality) and calculate the DQE metric.\nFor evaluation, we computed Kendall's $\\tau$ coefficient between the adapted DQE and ImageNet $k$-NN performance, comparing it with the state-of-the-art unsupervised transferability estimation method RankMe. The results are shown in the following table:\n\n**Table 5:** Kendall's $\\tau$ coefficient between DQE and ImageNet $k$-NN performance.\n| Method    | DINO | MEC | EsViT | iBOT | I-JEPA | Average |\n|-|-|-|-|-|-|-|\n| RankMe    | 0.57 | 0.81| 0.71  |**0.93** | **0.90**  | 0.79    |\n| **DQE(Ours)** | **0.87** | **0.90**|**0.92**  | 0.86 | 0.73   | **0.86**    |\n\nAs demonstrated, **the adapted DQE metric is a precise estimator for classification, achieving an average Kendall's $\\tau$ coefficient of 0.86, significantly surpassing RankMe**. Additionally, DQE consistently performs well across methods (with lower variance), confirming its generalizability and suitability for predicting downstream task performance.\n\n**Q5:** Why does SDD occur in dense tasks but not image-level tasks under SSL?\n\n**R5:** Thank you for your constructive comment! The primary reason is likely the misalignment between SSL objectives and dense tasks. In image-level SSL, by aligning different parts of objects, the different objects within a class are pulled together [1]. While alignment between dense representations primarily happens in the same position [2,3,4], e.g., parts of the same object might not be sufficiently aligned. Hence, compared with image-level SSL, the dense representations of different objects within the same class are more likely to be dispersed in the representation space.\n\n**Q6:** Causes of remaining uncertainty in DQE correlation?\n\n**R6:** We appreciate the valuable comment. While the DQE metric could precisely estimate the trend of downstream performance with an average Kendall's $\\tau$ coefficient of 0.54, we believe the remaining uncertainty arises from: \n1. In our experiments, the downstream performance is primarily evaluated by training a classification head on dense representations. In the derivation of the DQE metric, the head is simplified to a nearest neighbor classifier for the sake of analysis. **Simplifying the classification head to a nearest neighbor classifier may introduce bias compared to more complex heads in practice.**\n2. Since we do not have the downstream data or labels, we estimate the performance on the training set by k-means clustering. **Estimating performance with k-means clustering on training data might introduce additional noise.** For a more comprehensive analysis, please refer to Appendix E.\n\nIn future work, we aim to address these issues through refined theoretical analysis for more specific classification heads and more precise pseudo-labeling techniques.\n\n[1] Huang, Weiran, et al. \"Towards the generalization of contrastive self-supervised learning.\" arXiv preprint arXiv:2111.00743 (2021).\n\n[2] Ziegler, Adrian, and Yuki M. Asano. \"Self-supervised learning of object parts for semantic segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[3] Zhou, Jinghao, et al. \"ibot: Image bert pre-training with online tokenizer.\" arXiv preprint arXiv:2111.07832 (2021).\n\n[4] Li, Zhong-Yu, Shanghua Gao, and Ming-Ming Cheng. \"SERE: Exploring feature self-relation for self-supervised transformer.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 45.12 (2023): 15619-15631."}, {"review_id": "gRyMlz8K6y", "reviewer": "Reviewer_HMLT", "summary": "This paper identifies a phenomenon in self-supervised learning (SSL), termed Self-supervised Dense Degradation (SDD) — where longer pretraining on SSL tasks leads to degraded performance on dense prediction tasks (e.g., segmentation), even while image-level classification performance improves. To address this, the authors introduce a theoretical and empirical metric, Dense representation Quality Estimator (DQE), which combines class-separability and feature dimensionality. DQE is used for two practical purposes: (1) selecting optimal checkpoints (unsupervised model selection), and (2) as a regularizer to prevent degradation. Experiments on 10 SSL methods across 4 datasets are used to validate these claims.", "questions": "Please refer to the weakness above. \n\nI would be willing to adjust my score, provided that my key concerns are appropriately resolved.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## **Strengths**\n- The paper tackles a non-trivial and under-discussed problem — how dense task performance behaves under prolonged SSL pretraining.\n\n- DQE is a reasonable metric that reflects separability and collapse, grounded in theoretical error decomposition and backed by empirical mIoU correlation.\n\n- The proposed solutions (checkpoint selection, regularization) are simple and effective.\n\n- Component-level attribution of degradation (e.g., MoCo vs. DINO) is insightful.\n\n## **Weaknesses**\n1. Incomplete and Biased Empirical Coverage (**major concern**) \n\nDespite aiming to generalize across SSL methods, the empirical study is highly selective and omits major paradigms:\n      \n- No coverage of volume-maximization methods (e.g., VICReg[1], Barlow Twins[2], VICRegL(cited in Intro)), which are especially relevant as they explicitly optimize for variance and non-collapse — directly tied to DQE’s dimensionality component.  \n- Omission of BYOL[3]. a landmark method in SSL and the foundation of modern non-contrastive (or implicit contrastive) approaches, is unjustifiable. Given its popularity and unique properties (no negative samples, momentum encoder), it should be included at minimum for completeness.\n- **Dense-optimized methods** like Mugs (only cited), Selfpatch[4], DetCon[5], CroC[7], CrIBo[6] and FLSL[8] are excluded. Their omission is especially serious given that they aim to solve the very degradation problem this paper highlights. \n  - Mugs is a strong SSL method that takes a multi-level clustering approach. It performs well on both instance-level and dense prediction tasks. Should be included in the baselines.\n  -  CroC, CrIBo, FLSL shares DQE’s motivation (modeling intra-/inter-cluster structure) and warrants at least discussion or adapted evaluation.\n\nOmitting these methods weakens the claim that SDD is broadly unaddressed and makes the DQE gains harder to contextualize. It also risks overstating the novelty.\n\n2. Misrepresentation of Prior Work  \n\nThe paper claims that SDD is \"largely unaddressed,\" which is inaccurate. Multiple SSL works explicitly target dense representation degradation due to misalignment of pretraining and downstream tasks, including DenseCL, PixPro, DetCon, iBOT, Mugs, FLSL, and others. This mischaracterization undermines the novelty claim.\n\n3. Mismatch Between SSL Paradigm and DQE Assumptions (**major concern**) \n\nDQE assumes that dense features are well-approximated by $k$-means clustering, but SSL methods differ significantly in their latent structure:\n\n- DINO, SwAV, iBOT use explicit clustering (65K or 2K prototypes/centroids).\n\n- MoCo, SimSiam are contrastive and non-parametric.\n\n- MAE is generative, not clustering-based at all.\n\nApplying a fixed-$k$ clustering uniformly across these paradigms without justification introduces a major confounder. Methods that induce natural clusters (e.g., DINO) may align well with small-$k$ k-means, while others may not, e.g. MAE may produce smooth or manifold-like features that are poorly approximated by any hard clustering with small $k$. This makes DQE scores across models non-comparable unless the authors calibrate $k$ per paradigm or show robustness to this choice.\n\n4. Ambiguity in DQE's Scope of Comparison\n\nWhile DQE is clearly used for intra-method checkpoint selection, the paper implicitly uses DQE for cross-method comparison, but never explicitly discusses or clarifies whether this is valid., e.g. plot of DQE curves for different SSL methods side by side (e.g., Fig. 2–13), which could imply cross-method comparison, and highlighting certain methods (e.g., MoCo v3) as degrading more, based on DQE trends — again suggesting cross-method interpretation. This creates confusion and overextension of the metric’s intended scope.\n\n5. No Visual Validation of Clustering Geometry (**major concern**) \n\nThe paper provides visualization of similarity matrices of MoCov3 and DINO in Appendix. However, given the core claim is about cluster quality (separation and compactness), it is surprising that the paper provides no qualitative feature visualizations (e.g., t-SNE or UMAP) across checkpoints or methods. These are standard tools in SSL and would support the metric’s interpretability and degradation diagnosis.\n\n6. Lack of Efficiency and Overhead Analysis (**major concern**) \n\nThe paper presents DQE as lightweight and practical, but provides:\n- No runtime or FLOP estimates for DQE-based checkpoint selection, which involves feature extraction, k-means, and rank computation.\n- No training-time overhead analysis for DQE regularization. It is unclear how it scales with dataset size and model size.\n- No comparison to other low-cost model selection or early stopping heuristics.\nWithout this, it's unclear whether DQE is actually deployable at scale or just an academic tool.\n\n7. Lack of Scaling Study and Model Diversity. \n\nThe experiments are limited to small to medium models only: ResNet-50, ViT-Small, Swin-Tiny, and a single ViT-B model used only in the I-JEPA experiment. There are no results on ViT-Base or larger models across other methods, nor any discussion of how model capacity interacts with SDD or DQE. Since larger models are standard in current SSL pipelines (e.g., ViT-B, ViT-L in DINOv2, iBOT, MAE), this raises concerns about generalization. Representational degradation and rank collapse may behave differently at scale — and without including such models, the findings may not extrapolate to practical settings.\n\n> [1] Bardes, A., Ponce, J. and LeCun, Y., 2021. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906.\n\n> [2] Zbontar, J., Jing, L., Misra, I., LeCun, Y. and Deny, S., 2021, July. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning (pp. 12310-12320). PMLR. \n\n> [3] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M. and Piot, B., 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33, pp.21271-21284.\n\n> [4] Yun, S., Lee, H., Kim, J. and Shin, J., 2022. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8354-8363).\n\n> [5] Hénaff, O.J., Koppula, S., Alayrac, J.B., Van den Oord, A., Vinyals, O. and Carreira, J., 2021. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10086-10096).\n\n> [6] Stegmüller, T., Lebailly, T., Bozorgtabar, B., Tuytelaars, T. and Thiran, J.P., 2023. Croc: Cross-view online clustering for dense visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7000-7009).\n\n> [7] Lebailly, T., Stegmüller, T., Bozorgtabar, B., Thiran, J.P. and Tuytelaars, T., 2023. Cribo: Self-supervised learning via cross-image object-level bootstrapping. arXiv preprint arXiv:2310.07855.\n\n> [8] Su, Q., Netchaev, A., Li, H. and Ji, S., 2023. Flsl: Feature-level self-supervised learning. Advances in Neural Information Processing Systems, 36, pp.6568-6581.", "was_revised": true, "final_justification": "Most concerns are adequately addressed. Accept.", "rebuttal": "We sincerely appreciate your time reviewing our paper! Following your detailed comments, we have conducted additional experiments to further support our claims. Our detailed responses are provided below:\n\n**Q1:** Incomplete and Biased Empirical Coverage **(major concern)**\n\n**R1:** Thank you for your constructive feedback. We acknowledge the importance of including these baselines to strengthen our claims. However, for each baseline, we need to reproduce the results and evaluate the per-epoch dense performance and the DQE metric, which is computationally intensive (requiring thousands of GPU hours per baseline). As a first step, we have included key baselines such as VICReg, VICRegL, and Mugs.\n\n**Table 1:** The SDD phenomenon on additional baselines.\n\n| Method | COCO Best| COCO Last | COCO Diff | VOC Best | VOC Last | VOC Diff | ADE Best | ADE Last | ADE Diff | \n|-|-|-|-|-|-|-|-|-|-|\n| VICReg | 37.3 | 36.7 | -0.6 | 53.0 | 52.7 | -0.3 | 20.0 | 19.6 | -0.4 |\n| VICRegL | 38.4 | 38.1 | -0.3 | 54.6 | 54.5 | -0.1 | 20.7 | 20.4 | -0.3 |\n| Mugs | 45.4 | 43.7 | -1.7 | 68.1 | 67.6 | -0.5 | 29.7 | 28.6 | -1.1 |\n\n**Table 2:** The Kendall's $\\tau$ coefficient between the DQE metric and dense performance.\n\n| Method | COCO | VOC | ADE20K |\n|-|-|-|-|\n| VICReg | 0.73 | 0.75 | 0.74 |\n| VICRegL | 0.72 | 0.74 | 0.72 |\n| Mugs | 0.01* | 0.47 | 0.46 |\n\n**Table 3:** The DQE-based model selection results on additional baselines.\n\n| Method | COCO | VOC | ADE20K |\n|-|-|-|-|\n| VICReg| 37.2(+0.5) | 53.0(+0.3) | 19.8(+0.2) |\n| VICRegL| 38.3(+0.2) | 54.6(+0.1) | 20.7(+0.3) |\n| Mugs| 45.0(+1.3) | 67.9(+0.3) | 29.2(+0.6) |\n\nBased on the results in Tables 1 to 3, we conclude:\n\n1. **The SDD phenomenon appears in all additional baselines.** In particular, the volume-maximization design, especially in VICRegL, can effectively reduce the SDD phenomenon. While Mugs achieves the strongest dense performance among all baselines, its last performance is still much lower than its best performance. Thus, we could expect an improved baseline if the SDD phenomenon is effectively addressed.\n2. **The proposed DQE metric is effective for these baselines**, showing strong generalization to models trained with different paradigms. Consequently, model selection effectively improves the downstream performance.\n\nWith these new results, our empirical analysis is more complete. We also agree that it is necessary to present the results for BYOL and Barlow Twins. Although we cannot provide these results now due to limited computational resources, we promise to include them in the camera-ready version.\n\n**Q2:** Misrepresentation of Prior Work\n\n**R2:** Thank you for your valuable feedback. We have carefully checked the manuscript but it appears that we did not make any statements like \"SDD is largely unaddressed.\" Nonetheless, we acknowledge that previous dense SSL approaches have significantly improved the dense performances. We will add a new subsection in the related work to discuss the idea of these dense SSL approaches. In addition, we agree that it is important to clarify the differences and contributions of our work compared to previous studies. The main distinctions are as follows:\n\n1. Previous works such as DenseCL, PixPro, DetCon, CroC, CrIBo, and FLSL mainly focus on addressing misalignment within specific methods, often overlooking the generality of the SDD phenomenon. In contrast, we introduce a general metric (DQE) to quantify the severity of SDD.\n2. Despite the advancements in methods like iBOT and Mugs, noticeable SDD still exists, indicating that the problem remains unsolved and requires further investigation.\n3. Dense representation degradation can take various forms, such as those caused by domain shift or suboptimal training objectives. These types of degradation may occur throughout training. In contrast, we specifically target the degradation induced by the training dynamics of separability and dimensionality, which typically emerge in the later stages. This also explains why SDD still appears in iBOT and Mugs, i.e., the degradation we focus on is of a different nature.\n\nWe will revise the Related Work section to more clearly describe the connections with prior work and discuss the contributions of the aforementioned methods.\n\n**Q3:** Mismatch Between SSL Paradigm and the DQE Assumptions **(major concern)**\n\n**R3:** Thank you for your insightful comment. In this paper, we propose the DQE metric to address the negative impact of the SDD phenomenon, **focusing on comparisons between checkpoints trained with the same method**. We acknowledge that as you pointed out, the current version of DQE is incapable of cross-paradigm comparison, such as between MAE and DINO. In the current design, cross-model comparison can be achieved by first selecting the best checkpoints with DQE, then comparing the downstream performances of these checkpoints. This would also significantly speed up the evaluation compared to simply testing all the checkpoints.\n\nTo further test the robustness of DQE to the choice of $k$, we conducted additional experiments:\n\n**Table 4:** The Kendall's $\\tau$ coefficient between DQE metric and dense performance with different $k$.\n\n| Method| k=3 (default)| k=4 | k=5|\n| -| -|-|-|\n| MoCo v3 | 0.55|0.49 |0.45|\n| MEC| 0.53|0.57 |0.58|\n| SimSiam | 0.73|0.73 |0.72|\n| DINO| 0.16|0.19 |0.20|\n| EsViT| 0.63|0.59 |0.55|\n| iBOT| 0.47|0.46 |0.46|\n| MAE| 0.37|0.34 |0.31|\n\nFrom the results in Table 4, we see that the Kendall's $\\tau$ coefficient is insensitive to the change of $k$, regardless of whether the pretraining paradigm is clustering-based, contrastive, or generative. This demonstrates that our fixed $k$ strategy is sufficient for our needs.\n\n**Q4:** Ambiguity in DQE's Scope of Comparison\n\n**R4:** We apologize for the confusion. You are correct that DQE is used for intra-method comparison. We will add an explicit explanation in the methodology section and incorporate the discussions in R3 to our paper.\n\n**Q5:** No Visual Validation of Clustering Geometry **(major concern)**\n\n**R5:** Thank you for your valuable comment. To visualize dimensional collapse, we first project the representations onto principal components. We then divide the dimensions into eight segments based on the eigenvalues and perform t-SNE visualization for each segment. We observe that the structure of the top-k principal components matches the class separability, while the bottom components quickly collapse to a single point as the dimensional index decreases, confirming consistency between the two. We also compare the t-SNE plot between the last model and our selected model. The selected models have more compact intra-class representations and larger inter-class distances, indicating a better class-separability compared with the last models.\nAlthough we cannot present figures here due to NeurIPS 2025 policy, we promise to include these qualitative results in the final version.\n\n**Q6:** Lack of Efficiency and Overhead Analysis **(major concern)**\n\n**R6:** Thank you for your constructive comment. Indeed, efficiency analysis is important to determine whether the proposed method is practical at scale. We present two experiments below:\n\n1. **DQE-based model selection balances efficiency and effectiveness.**\n\n**Table 5:** Effectiveness and efficiency analysis of different model selection methods.\n\n| Method| Data| Label | $\\Delta$ mIoU $\\uparrow$ |Additional GPU hours for each checkpoint|\n| -| -|-|-|-| \n|Loss-based model selection|||-1.0|0|\n|Supervised model selection|$\\checkmark$|$\\checkmark$ |+4.8| 2.43|\n|DQE-based model selection||+4.0| 0.025($\\sim$ 0.01 $\\times$)|\n\n**Table 6:** Runtime analysis of different components of DQE (measured in seconds on a single 4090 GPU).\n| Feature Extraction| $k$-means| $M_{inter}$ | $M_{intra}$ | $M_{dim}$ | Total |\n| -| -|-|-|-|-| \n| 4.9|45.2|9.8|10.9| 4.3| 75.1|\n\nLoss-based model selection or early stopping does not reliably reflect dense performance due to the SDD phenomenon, while supervised model selection is too expensive in practice. In contrast, the proposed DQE-based model selection achieves strong performance with minimal overhead. We will also include these results in Table 3 of our paper.\n\n2. **DQE regularization improves the best performance with an acceptable cost.**\n\n**Table 7:** Runtime analysis of DQE regularization with 50 epochs of training on DINO.\n| Method|mIoU |Total Training GPU hours|\n|-|-|-|\n|DINO| 56.6 | ~133 |\n|DINO + DQE| 57.8 | ~162|\n\nFrom Table 7, DQE regularization introduces about 22% additional training time, which we believe is acceptable. Importantly, **DQE regularization improves the best performance achieved by DINO**. This demonstrates the potential of integrating DQE into training, and we will further explore ways to speed up DQE regularization by improving the clustering and metric calculation.\n\n**Q7:** Lack of Scaling Study\n\n**R7:** Thank you for your constructive comment. To study the effect of model size on the SDD phenomenon and the DQE metric, we conducted an additional experiment on iBOT with ViT-Base-16. The results are shown below:\n\n**Table 8:** The SDD phenomenon on the larger backbone.\n\n|Method|COCO Best|COCO Last|COCO Diff|VOC Best|VOC Last|VOC Diff|ADE Best|ADE Last|ADE Diff| \n|-|-|-|-|-|-|-|-|-|-|\n|iBOT (ViT-Base)| 48.9 | 47.0 | -1.9 | 71.5 | 69.4 | -2.1 | 31.3 | 29.9 | -1.4 |\n\n**Table 9:** The Kendall's $\\tau$ coefficient between DQE metric and dense performance.\n\n|Method| COCO|VOC|ADE|\n|-|-|-|-|\n| iBOT (ViT-Base)|0.66|0.52|0.57|\n\n**Table 10:** The DQE-based model selection results.\n\n| Method|COCO|VOC|ADE|\n|-|-|-|-|\n| iBOT (ViT-Base) | 48.9(+1.9)|71.5(+2.1)| 31.1(+1.2)|\n\nThe main conclusions still hold at this larger scale: **The SDD phenomenon persists and the DQE metric can still accurately predict the trend of dense performance**. Due to limited computational resources and the timeline for the rebuttal, we will conduct more experiments on larger models in the updated version."}, {"review_id": "NeG96KT7nb", "reviewer": "Reviewer_riT6", "summary": "This paper identifies a general and critical issue affecting a wide range of image self-supervised learning algorithms, termed Self-supervised Dense Degradation (SDD). This issue reveals that extended SSL training degrades performance on dense tasks (such as segmentation), even as it improves image-level classification. This finding highlights a fundamental limitation of modern image self-supervised learning approaches. To address this, the authors propose a theoretically grounded metric for evaluating dense representations without the need for labels or downstream data, called the Dense Representation Quality Estimator (DQE). The DQE is based on two key factors: class separability and effective dimensionality. Notably, the DQE outperforms existing unsupervised metrics (e.g., RankMe, α-ReQ), demonstrating a strong correlation to downstream mIoU with a Kendall’s τ value of 0.54. When integrating DQE into the SSL training process, it reverses degradation trends and boosts performance with negligible computational overhead.", "questions": "1. Regarding the class separability factor in DQE, is it measured at the image level or the pixel level? If it is at the image level, how does this factor accurately capture the performance trends of downstream dense tasks?\n2. As discussed in section 5.3, different methods experience the SDD phenomenon for various reasons. Could you explain why this variation occurs? For instance, MoCo v3’s performance degradation is attributed to dimensional collapse in dense features. Does this imply that increasing the dimensionality of MoCo v3 could lead to better performance?", "limitations": "1. The theoretical analysis of DQE is based on sub-Gaussian distributions assumptions, which might not be true for real-world cases.\n2. The performance for other dense tasks beyond segmentation is under-explored in this work, for instance, depth estimation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. **Novel Phenomenon Discovery**: This paper identifies SDD—a counter-intuitive degradation in dense tasks during prolonged SSL training—which challenges the established belief that \"longer pretraining always helps.\" Additionally, the authors demonstrate SDD’s consistency across 10 SSL methods, 4 benchmarks, and diverse architectures (ViTs, CNNs).\n2. **Theoretical Grounding**: The authors provide a robust theoretical foundation by deriving a principled link between downstream error and two factors: class separability and effective dimensionality. This theorem ultimately leads to the development of the DQE.\n3. **Practical Solution for SDD**: The evaluations indicate that DQE serves as an effective and practical solution for addressing SDD. As a label-free metric, when applied to model selection, DQE results in a +4.0% mIoU gain with negligible computational cost. Furthermore, when used as a training regularizer, it reverses SDD, thereby improving both DQE scores and task performance.\n\nWeaknesses\n1. **Limitations discussion**: It would be beneficial to further discuss the limitations of proposed DQE, for example, the failure mode, the SDD causes (e.g., dimensional collapse vs. separability loss) might offer some architecture-specific fixes etc.", "was_revised": true, "final_justification": "Most of my concerns are well addressed. I'm satisfied with its current form. Thus I will maintain my score. Thank you!", "rebuttal": "Thank you very much for your valuable feedback and suggestions. We appreciate your positive comments on the SDD phenomenon and our proposed DQE metric. Below are our detailed responses:\n\n**Q1:** Limitations discussion\n\n**R1:** Thank you for your valuable comment. As discussed in Appendix G, our DQE metric primarily focuses on off-the-shelf performance estimation. Therefore, when the backbone is fine-tuned for an extended period or with a large prediction head, the estimated performance may become inaccurate. In addition, the DQE is designed as a general performance estimator. Explaining the model-specific causes of the SDD phenomenon, such as the degradation in MoCo v3 or DINO, is an interesting and important future direction. We will move the limitations discussion into the main paper in our revision.\n\n**Q2:** Regarding the class separability factor in DQE, is it measured at the image level or the pixel level?\n\n**R2:** We apologize for the confusion. Both the class separability and dimensionality measures are calculated on **patch/pixel-level representations**. We will explicitly clarify this at the beginning of the method section.\n\n**Q3:** Why do different methods experience the SDD phenomenon for various reasons? Does increasing the dimensionality of MoCo v3 lead to better performance?\n\n**R3:** Thank you for your insightful comment. In general, self-supervised learning aims to achieve both semantic alignment and high effective dimensionality in representations. These two objectives often require a trade-off. For example, a fully uniform representation ensures high effective dimensionality but may reduce class separability, while emphasizing semantic alignment can lead to dimensionality collapse. We argue that the degradation seen in different methods results from an inability to maintain this trade-off throughout training, which causes the model to become biased toward one side.\n\nAddressing your question about MoCo v3, we conduct an experiment by incorporating our $M_{dim}$ term into the loss function of MoCo v3. We resume MoCo v3 from epoch 40 (when it starts to collapse) and train it for another 10 epochs. The results are shown in the following table:\n\n**Table 1:** The performance of MoCo v3 with increased dimensionality.\n| Method | $M_{dim}$   | VOC mIoU    |\n| -| -| -|\n| MoCo v3      | 0.744 | 49.1 |\n| MoCo v3 + Dim| 0.892 (+0.148) | 52.3 (+3.2) |\n\nAs shown in this table, the effective dimensionality increases from 0.744 to 0.892. Consequently, the downstream performance is significantly improved by 3.2\\% mIoU. The results demonstrate that mitigating the spotted cause of SDD could effectively improve the performance.\n\n**Q4:** The sub-Gaussian assumptions might not hold for real-world cases.\n\n**R4:** Thank you for your constructive comment. We would like to restate the assumption:\n\n**Assumption:** For all $j \\in [K]$, given the representation set $Z^j$ of the $j$-th category, the representations $\\\\{z_i^j\\\\}_{i=1}^{N_j}$ in $Z^j$ are i.i.d. $R$-sub-Gaussian distributed, or formally:\n$\\forall \\lambda \\in \\mathbb{R}, \\mathbb{E}[\\exp(\\lambda Z)] \\leq \\exp(\\lambda^2R^2 / 2)$.\n\nIntuitively, this means the tail of $Z$ decays at least as fast as that of a Gaussian distribution. Since \"sub-Gaussian\" does not refer to a specific distribution family, standard tests like the Kolmogorov–Smirnov or Shapiro–Wilk test are not applicable. Therefore, we use the Orlicz norm to assess whether the tail decay of $Z$ is sufficiently fast, following the theorem below:\n\n**Theorem (Equivalence of a sub-Gaussian Variable, Proposition 2.5.2 in [1]):** Let $\\psi_2(x) = \\exp(x^2) - 1$ be the Orlicz function of sub-Gaussian. The $\\psi_2$ norm of random variable $Z$ is defined as: $\\|Z\\|\\_{\\psi\\_2} = \\inf\\\\{t > 0\\~|\\~\\mathbb{E}[\\psi_2 (|Z|/t)]\\leq 1\\\\}$. We have $Z$ follows a sub-Gaussian distribution if $\\|Z\\|_{\\psi_2} < \\infty$.\n\nBased on this theorem, we assess conformity to the sub-Gaussian assumption by computing the corresponding $\\psi_2$ norm as a quantitative measure. Specifically, we select models trained with EsViT, iBOT, and SwAV, extract the dense representations on COCO, and compute the $\\psi_2$ norm for each dimension, reporting the mean values as follows:\n\n**Table 2:** The $\\psi_2$ norm of dense representations on COCO.\n\n| Method | Class 0         | Class 5         | Class 10        | Class 15        | Class 20        | Class 25        |\n| - | - | - | - | - | - | - |\n| EsViT  | 1.6094 ± 0.0491 | 1.6100 ± 0.0449 | 1.6048 ± 0.0453 | 1.6134 ± 0.0507 | 1.6101 ± 0.0459 | 1.6078 ± 0.0470 |\n| iBOT   | 1.6105 ± 0.0460 | 1.6137 ± 0.0517 | 1.6102 ± 0.0456 | 1.6093 ± 0.0501 | 1.6119 ± 0.0489 | 1.6097 ± 0.0457 |\n| SwAV   | 1.6117 ± 0.0493 | 1.6126 ± 0.0517 | 1.6098 ± 0.0478 | 1.6067 ± 0.0439 | 1.6147 ± 0.0498 | 1.6112 ± 0.0499 |\n\nAs a reference, the norm of the standard Gaussian distribution is $\\sqrt{8/3} \\approx 1.633$, while the norm for the actual representations is around 1.61 in most cases. This indicates a faster tail decay than the standard Gaussian distribution, thus satisfying the sub-Gaussian assumption.\n\n**Q5:** The performance for other dense tasks like depth estimation is under-explored.\n\n**R5:** Thank you for your constructive comment. We conducted additional experiments on depth estimation. Following our linear probing setup, we freeze the model and train a linear layer to predict depth values on the NYU-depth v2 dataset. The results are reported in the table below:\n\n**Table 3:** The SDD phenomenon in depth estimation. We report the RMSE metric (lower is better) on the NYU-depth v2 dataset.\n\n| Method | Best $\\downarrow$ | Last $\\downarrow$ | Difference |\n|-|-|-|-|\n| MoCo v3 | 0.638 | 1.589 | 0.951 |\n| DenseCL | 0.547 | 0.559 | 0.012 |\n| SimSiam | 0.597 | 0.608 | 0.011 |\n| SwAV | 0.705 | 0.725 | 0.020 |\n| DINO | 0.515 | 0.553 | 0.038 |\n| MAE | 0.680 | 0.775 | 0.095 |\n| I-JEPA | 0.460 | 0.487 | 0.027 |\n\n**As shown in Table 3, the SDD phenomenon also occurs in the depth estimation task.** The performance trend is similar to that observed in segmentation tasks. These results validate that SDD is a general phenomenon and not specific to segmentation. While we cannot present the full performance curves here due to NeurIPS 2025 policy, we assure you that these results will be included in the final version.\n\nTo further evaluate the effectiveness of the DQE metric on depth estimation, we also compute Kendall's $\\tau$ coefficient between the DQE metric and depth estimation performance. The results are shown below:\n\n**Table 4:** Kendall's $\\tau$ coefficient between DQE and RMSE.\n\n| Method | Kendall's $\\tau$|\n|-|-|\n| MoCo v3 | 0.452 |\n| DenseCL | 0.752 |\n| SimSiam | 0.659 |\n| SwAV | 0.247 |   \n| DINO | 0.071 |   \n| MAE | 0.107 |    \n| I-JEPA | 0.190 | \n\n**As shown in Table 4, the DQE metric positively correlates with depth estimation performance.** Although these results show the effectiveness of DQE across different downstream tasks, we note that Kendall's $\\tau$ coefficient is relatively lower compared with segmentation. There are two main reasons: 1) Our DQE metric is derived from segmentation performance, so class separability may not fully benefit the depth estimation task. 2) As a regression task, the RMSE curve fluctuates, making prediction more difficult. We plan to further improve the DQE metric to enhance its capability for depth estimation.\n\n**Table 5:** DQE-based model selection results on depth estimation.\n\n| Method  | RMSE $\\downarrow$ |\n|-|-|\n| MoCo v3 | 0.638(-0.951) |\n| DenseCL | 0.547(-0.012) |\n| SimSiam | 0.598(-0.010) |\n| SwAV |    0.714(-0.009) |\n| DINO |    0.532(-0.021) |\n| MAE |     0.709(-0.066) |\n| I-JEPA |  0.479(-0.008)|\n\nIn Table 5, we further examine whether the selected models perform better on the depth estimation task. As shown, DQE-based model selection consistently improves model performance, demonstrating that DQE is generalizable to different downstream tasks.\n\n[1] Vershynin, R., 2018. High-dimensional probability: An introduction with applications in data science (Vol. 47). Cambridge University Press."}, {"review_id": "JNvQKZ1LUV", "reviewer": "Reviewer_pWRt", "summary": "This paper identifies Self-supervised Dense Degradation (SDD), a phenomenon where longer self-supervised learning (SSL) pretraining improves image-level tasks (e.g., classification) but degrades dense task performance (e.g., semantic segmentation). The authors demonstrate SDD’s prevalence across 10 diverse SSL methods (contrastive, non-contrastive, masked modeling) and 4 benchmarks. To address SDD, they propose the Dense Representation Quality Estimator (DQE), a theoretically grounded unsupervised metric combining class separability (inter-class distance vs. intra-class radius) and effective dimensionality.", "questions": "1. Figure 5 and Table 1 show that there is not a strong correlation between DINO, the proposed DQE metric, and downstream performance. What could be the reason for this?\n2. The class-separability measure in Section 3 is interesting. A relevant paper published at ICML 2025 introduces a novel clustering-based SSL method called ReSA[1], which demonstrates the ability to achieve more fine-grained representations and better class-separability compared to previous methods like DINO and MoCoV3. I am curious whether this method might also exhibit the SSD phenomenon.\n\n[1] Clustering Properties of Self-Supervised Learning.", "limitations": "The authors do not mention potential limitations in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The concept of Self-supervised Dense Degradation (SDD) is a valuable contribution, highlighting a gap in SSL research that was previously underexplored.\n2. The DQE metric is validated through extensive experiments, showing its high correlation with downstream dense performance.\n\n## Weaknesses\n1. Experiments focus solely on semantic segmentation. While the authors briefly mention video object segmentation (Table 6), broader validation (e.g., object detection, depth estimation) would strengthen claims about \"dense tasks.\" \n2. Lack of Deep Insight into Causes of Degradation. While the paper presents a broad theoretical understanding of SDD, it could delve deeper into the underlying reasons for the degradation across different SSL methods, providing more clarity on the specific mechanisms at play.\n3. The paper lacks an analysis of the evaluation setup, which affects the credibility of the experimental results. The authors state, \"For ViT models, we use the last layer’s patch embeddings; for ResNet50 and Swin Transformer, we extract features from the fourth layer.\" However, they do not explain why this setup was chosen, nor do they analyze whether using intermediate layers in ViT for segmentation embeddings might yield better results than using the last layer. The experimental results presented in the paper show that contrastive learning-based methods, such as DINO and MoCoV3, perform poorly with ViT, while SimSiam and DenseCL perform well with ResNet. This suggests that there might be an issue with the evaluation setup for ViT. I recommend that the authors provide a detailed analysis to explain this phenomenon.", "was_revised": true, "final_justification": "Most of my concerns are adequately addressed.", "rebuttal": "Thank you for your time and effort in reviewing our paper! Please find our detailed responses below.\n\n**Q1:** Broader validation on other tasks.\n\n**R1:** We have conducted additional experiments on the depth estimation task. Following our linear probing setting, we freeze the model and train a linear layer to predict depth values on the NYU-depth v2 dataset. The results are presented in the table below:\n\n**Table 1:** The SDD phenomenon on the depth estimation task, reporting RMSE (lower is better) on the NYU-depth v2 dataset.\n\n| Method | Best $\\downarrow$ | Last $\\downarrow$ | Difference |\n|-|-|-|-|\n| MoCo v3 | 0.638 | 1.589 | 0.951 |\n| DenseCL | 0.547 | 0.559 | 0.012 |\n| SimSiam | 0.597 | 0.608 | 0.011 |\n| SwAV | 0.705 | 0.725 | 0.020 |\n| DINO | 0.515 | 0.553 | 0.038 |\n| MAE | 0.680 | 0.775 | 0.095 |\n| I-JEPA | 0.460 | 0.487 | 0.027 |\n\n**As shown in Table 1, the SDD phenomenon also appears in the depth estimation task.** The performance trend is similar to that observed in the segmentation task. These results confirm that SDD is a general phenomenon and is not limited to segmentation. While we cannot present the full performance curves due to NeurIPS 2025 policy, we will include them in the final version.\n\nTo further evaluate the effectiveness of the proposed DQE metric for depth estimation, we also compute Kendall's $\\tau$ coefficient between the DQE metric and depth estimation performance. The results are shown below:\n\n**Table 2:** The Kendall's $\\tau$ coefficient between DQE and RMSE metric.\n| Method | Kendall's $\\tau$|\n|-|-|\n| MoCo v3 | 0.452 |\n| DenseCL | 0.752 |\n| SimSiam | 0.659 |\n| SwAV | 0.247 |   \n| DINO | 0.071 |   \n| MAE | 0.107 |    \n| I-JEPA | 0.190 | \n\n\n**As shown in Table 2, the DQE metric is positively correlated with depth estimation performance.** Although the DQE metric works across different downstream tasks, the Kendall's $\\tau$ coefficient is lower compared to the segmentation task. There are two main reasons: 1) The DQE metric is derived from segmentation performance, and class separability may not fully reflect the needs of depth estimation. 2) As a regression task, the RMSE curve is less stable, making prediction more difficult. We plan to further improve the DQE metric to enhance its effectiveness for depth estimation.\n\n**Table 3:** The DQE-based model selection results on the depth estimation task.\n| Method  | RMSE $\\downarrow$ |\n|-|-|\n| MoCo v3 | 0.638(-0.951) |\n| DenseCL | 0.547(-0.012) |\n| SimSiam | 0.598(-0.010) |\n| SwAV |    0.714(-0.009) |\n| DINO |    0.532(-0.021) |\n| MAE |     0.709(-0.066) |\n| I-JEPA |  0.479(-0.008)|\n\n**Table 3 shows that DQE-based model selection consistently improves model performance**. This demonstrates that DQE is generalizable to different downstream tasks.\n\n**Q2:** Lack of deep insight into causes of degradation.\n\n**R2:** Intuitively, SSL aims to balance semantic alignment and the effective dimensionality of representations. These two objectives often involve a trade-off. For instance, fully uniform representations have high effective dimensionality but may lack separability; conversely, focusing too much on semantic alignment can lead to dimension collapse. We argue that the degradation seen in different methods is due to the failure to maintain this trade-off during training, which causes a bias toward one objective.\n\n**Q3:** Issue with the evaluation setup.\n\n**R3:** Thank you for your detailed review. In fact, **we use only the last layer's embeddings for evaluation** as is standard in SSL [1,2,3,4]. For ResNet50 and Swin Transformer, the fourth layer is exactly the last layer, as both have only four layers (stages). We apologize for any confusion caused by our previous description and have revised the experimental setup section for clarity.\n\nBeyond this clarification, we also find it interesting and important to explore the SDD phenomenon at different representation layers. We evaluated the dense performance of various layers of DINO on the COCO-Stuff dataset. The results are shown below:\n\n**Table 4:** SDD phenomenon in different layers.\n| Method    | Best | Last | Diff |\n|-|-|-|-|\n| 6th layer  | 30.0 | 29.7| -0.3  |\n| 9th layer  | 39.1 | 37.3| -1.8  |\n| 12th layer | 40.4 | 36.0| -4.4  |\n\nWhile the SDD phenomenon is weaker in shallower layers, the overall performance trend remains similar. At the 6th layer, the SDD phenomenon is almost negligible, but the performance is much worse, and it is not suitable for downstream tasks.\n\n**Q4:** Analysis of failure case in DINO.\n\n**R4:** Thank you for your insightful comment. The main reason is that DINO uses an EMA update strategy, where the momentum coefficient $\\beta$ increases from 0.996 to 1 during training. In the later training stages, the teacher model stops updating, making the student more likely to experience dimension collapse when aligning to a fixed target. As shown in Fig. 5, this leads to a significant increase in the dimensionality metric. While our metric captures this trend, the large value of $\\lambda$ gives too much weight to the dimensionality metric, causing a mismatch between the final DQE score and actual performance. Therefore, for DINO, it may be necessary to dynamically adjust $\\lambda$ together with the EMA momentum coefficient to better account for different training phases.\n\n**Q5:** More discussion about clustering-based SSL method ReSA [5]\n\n**R5:** Thank you for pointing out this related study. We find the idea of clustering-based optimization in ReSA [5] interesting and novel, and it achieves strong experimental results. To investigate whether clustering-based optimization affects SDD and the DQE metric, we reproduced ReSA and tested its dense performance. The results are shown below:\n\n**Table 5:** The SDD phenomenon in ReSA.\n| Method | COCO Best| COCO Last | COCO Diff | VOC Best | VOC Last | VOC Diff | ADE Best | ADE Last | ADE Diff | \n|-|-|-|-|-|-|-|-|-|-|\n| ReSA | 36.6 | 36.2 | -0.4 | 49.7 | 49.1 | -0.6 | 18.4 | 18.2 | -0.2 |\n\n**Table 6:** The Kendall's $\\tau$ coefficient between the DQE metric and dense performance.\n| Method | COCO | VOC | ADE20K |\n|-|-|-|-|\n| ReSA | 0.74 | 0.72 | 0.71 |\n\n**Table 7:** The DQE-based model selection results on ReSA.\n| Method | COCO | VOC | ADE20K |\n|-|-|-|-|\n| ReSA | 36.6(+0.4) | 49.5(+0.4) | 18.3(+0.1) |\n\n\nAs shown, the SDD phenomenon also occurs in ReSA. This may be because clustering optimization is not directly designed for dense representations in ReSA. The DQE metric still aligns well with ReSA’s performance. Therefore, **the main contributions of our paper remain valid for ReSA.**\nWhile we have tried to reduce the SDD phenomenon by directly optimizing the DQE metric, we believe further work exploring clustering-based optimization for dense representations, or studying how SDD behaves on encodings and embeddings, would be valuable future directions. We will add the ReSA results to the final version as an important baseline and provide more discussions in our paper.\n\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[2] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n[3] Zhou, Jinghao, et al. \"ibot: Image bert pre-training with online tokenizer.\" arXiv preprint arXiv:2111.07832 (2021).\n\n[4] Chen, Xinlei, Saining Xie, and Kaiming He. \"An empirical study of training self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[5] Weng, Xi, et al. \"Clustering Properties of Self-Supervised Learning.\" arXiv preprint arXiv:2501.18452 (2025)."}]

Author rebuttal:

Meta-review: This paper received 4, 3, 3, 5 scores originally. Reviewers agree that "The concept of Self-supervised Dense Degradation (SDD) is a valuable contribution", but are concerned that the proposed method is mainly constrained to semantic segmentation and/or small to medium models. After the rebuttal and discussions, reviewers' concerns are mostly addressed, and the scores become 5, 4, 5, 5. Furthermore, even benefits are cosntrained to only semantic segmentation, the proposed SDD concept and method are valuable.
Decision: Accept (poster)