Title: Exploring Structural Degradation in Dense Representations for Self-supervised Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-Supervised Learning

Dense representation learning
Abstract: In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense prediction tasks (e.g., semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across sixteen state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge.
To tackle this issue, we introduce a Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DSE-based regularization method. Experiments on sixteen SSL methods across four benchmarks confirm that model selection improves mIoU by $3.0\\%$ on average with negligible computational cost. Additionally, DSE regularization consistently mitigates the effects of dense degradation. Code is available at \url{https://github.com/EldercatSAM/SSL-Degradation}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies Self-supervised Dense Degradation (SDD), a phenomenon where longer self-supervised learning (SSL) pretraining improves image-level tasks (e.g., classification) but degrades dense task performance (e.g., semantic segmentation). The authors demonstrate SDD’s prevalence across 10 diverse SSL methods (contrastive, non-contrastive, masked modeling) and 4 benchmarks. To address SDD, they propose the Dense Representation Quality Estimator (DQE), a theoretically grounded unsupervised metric combining class separability (inter-class distance vs. intra-class radius) and effective dimensionality.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The concept of Self-supervised Dense Degradation (SDD) is a valuable contribution, highlighting a gap in SSL research that was previously underexplored.\n2. The DQE metric is validated through extensive experiments, showing its high correlation with downstream dense performance.\n\n## Weaknesses\n1. Experiments focus solely on semantic segmentation. While the authors briefly mention video object segmentation (Table 6), broader validation (e.g., object detection, depth estimation) would strengthen claims about \"dense tasks.\" \n2. Lack of Deep Insight into Causes of Degradation. While the paper presents a broad theoretical understanding of SDD, it could delve deeper into the underlying reasons for the degradation across different SSL methods, providing more clarity on the specific mechanisms at play.\n3. The paper lacks an analysis of the evaluation setup, which affects the credibility of the experimental results. The authors state, \"For ViT models, we use the last layer’s patch embeddings; for ResNet50 and Swin Transformer, we extract features from the fourth layer.\" However, they do not explain why this setup was chosen, nor do they analyze whether using intermediate layers in ViT for segmentation embeddings might yield better results than using the last layer. The experimental results presented in the paper show that contrastive learning-based methods, such as DINO and MoCoV3, perform poorly with ViT, while SimSiam and DenseCL perform well with ResNet. This suggests that there might be an issue with the evaluation setup for ViT. I recommend that the authors provide a detailed analysis to explain this phenomenon.\n\nQUESTIONS:\n1. Figure 5 and Table 1 show that there is not a strong correlation between DINO, the proposed DQE metric, and downstream performance. What could be the reason for this?\n2. The class-separability measure in Section 3 is interesting. A relevant paper published at ICML 2025 introduces a novel clustering-based SSL method called ReSA[1], which demonstrates the ability to achieve more fine-grained representations and better class-separability compared to previous methods like DINO and MoCoV3. I am curious whether this method might also exhibit the SSD phenomenon.\n\n[1] Clustering Properties of Self-Supervised Learning.\n\nLIMITATIONS:\nThe authors do not mention potential limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a general and critical issue affecting a wide range of image self-supervised learning algorithms, termed Self-supervised Dense Degradation (SDD). This issue reveals that extended SSL training degrades performance on dense tasks (such as segmentation), even as it improves image-level classification. This finding highlights a fundamental limitation of modern image self-supervised learning approaches. To address this, the authors propose a theoretically grounded metric for evaluating dense representations without the need for labels or downstream data, called the Dense Representation Quality Estimator (DQE). The DQE is based on two key factors: class separability and effective dimensionality. Notably, the DQE outperforms existing unsupervised metrics (e.g., RankMe, α-ReQ), demonstrating a strong correlation to downstream mIoU with a Kendall’s τ value of 0.54. When integrating DQE into the SSL training process, it reverses degradation trends and boosts performance with negligible computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. **Novel Phenomenon Discovery**: This paper identifies SDD—a counter-intuitive degradation in dense tasks during prolonged SSL training—which challenges the established belief that \"longer pretraining always helps.\" Additionally, the authors demonstrate SDD’s consistency across 10 SSL methods, 4 benchmarks, and diverse architectures (ViTs, CNNs).\n2. **Theoretical Grounding**: The authors provide a robust theoretical foundation by deriving a principled link between downstream error and two factors: class separability and effective dimensionality. This theorem ultimately leads to the development of the DQE.\n3. **Practical Solution for SDD**: The evaluations indicate that DQE serves as an effective and practical solution for addressing SDD. As a label-free metric, when applied to model selection, DQE results in a +4.0% mIoU gain with negligible computational cost. Furthermore, when used as a training regularizer, it reverses SDD, thereby improving both DQE scores and task performance.\n\nWeaknesses\n1. **Limitations discussion**: It would be beneficial to further discuss the limitations of proposed DQE, for example, the failure mode, the SDD causes (e.g., dimensional collapse vs. separability loss) might offer some architecture-specific fixes etc.\n\nQUESTIONS:\n1. Regarding the class separability factor in DQE, is it measured at the image level or the pixel level? If it is at the image level, how does this factor accurately capture the performance trends of downstream dense tasks?\n2. As discussed in section 5.3, different methods experience the SDD phenomenon for various reasons. Could you explain why this variation occurs? For instance, MoCo v3’s performance degradation is attributed to dimensional collapse in dense features. Does this imply that increasing the dimensionality of MoCo v3 could lead to better performance?\n\nLIMITATIONS:\n1. The theoretical analysis of DQE is based on sub-Gaussian distributions assumptions, which might not be true for real-world cases.\n2. The performance for other dense tasks beyond segmentation is under-explored in this work, for instance, depth estimation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a phenomenon in self-supervised learning (SSL), termed Self-supervised Dense Degradation (SDD) — where longer pretraining on SSL tasks leads to degraded performance on dense prediction tasks (e.g., segmentation), even while image-level classification performance improves. To address this, the authors introduce a theoretical and empirical metric, Dense representation Quality Estimator (DQE), which combines class-separability and feature dimensionality. DQE is used for two practical purposes: (1) selecting optimal checkpoints (unsupervised model selection), and (2) as a regularizer to prevent degradation. Experiments on 10 SSL methods across 4 datasets are used to validate these claims.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths**\n- The paper tackles a non-trivial and under-discussed problem — how dense task performance behaves under prolonged SSL pretraining.\n\n- DQE is a reasonable metric that reflects separability and collapse, grounded in theoretical error decomposition and backed by empirical mIoU correlation.\n\n- The proposed solutions (checkpoint selection, regularization) are simple and effective.\n\n- Component-level attribution of degradation (e.g., MoCo vs. DINO) is insightful.\n\n## **Weaknesses**\n1. Incomplete and Biased Empirical Coverage (**major concern**) \n\nDespite aiming to generalize across SSL methods, the empirical study is highly selective and omits major paradigms:\n      \n- No coverage of volume-maximization methods (e.g., VICReg[1], Barlow Twins[2], VICRegL(cited in Intro)), which are especially relevant as they explicitly optimize for variance and non-collapse — directly tied to DQE’s dimensionality component.  \n- Omission of BYOL[3]. a landmark method in SSL and the foundation of modern non-contrastive (or implicit contrastive) approaches, is unjustifiable. Given its popularity and unique properties (no negative samples, momentum encoder), it should be included at minimum for completeness.\n- **Dense-optimized methods** like Mugs (only cited), Selfpatch[4], DetCon[5], CroC[7], CrIBo[6] and FLSL[8] are excluded. Their omission is especially serious given that they aim to solve the very degradation problem this paper highlights. \n  - Mugs is a strong SSL method that takes a multi-level clustering approach. It performs well on both instance-level and dense prediction tasks. Should be included in the baselines.\n  -  CroC, CrIBo, FLSL shares DQE’s motivation (modeling intra-/inter-cluster structure) and warrants at least discussion or adapted evaluation.\n\nOmitting these methods weakens the claim that SDD is broadly unaddressed and makes the DQE gains harder to contextualize. It also risks overstating the novelty.\n\n2. Misrepresentation of Prior Work  \n\nThe paper claims that SDD is \"largely unaddressed,\" which is inaccurate. Multiple SSL works explicitly target dense representation degradation due to misalignment of pretraining and downstream tasks, including DenseCL, PixPro, DetCon, iBOT, Mugs, FLSL, and others. This mischaracterization undermines the novelty claim.\n\n3. Mismatch Between SSL Paradigm and DQE Assumptions (**major concern**) \n\nDQE assumes that dense features are well-approximated by $k$-means clustering, but SSL methods differ significantly in their latent structure:\n\n- DINO, SwAV, iBOT use explicit clustering (65K or 2K prototypes/centroids).\n\n- MoCo, SimSiam are contrastive and non-parametric.\n\n- MAE is generative, not clustering-based at all.\n\nApplying a fixed-$k$ clustering uniformly across these paradigms without justification introduces a major confounder. Methods that induce natural clusters (e.g., DINO) may align well with small-$k$ k-means, while others may not, e.g. MAE may produce smooth or manifold-like features that are poorly approximated by any hard clustering with small $k$. This makes DQE scores across models non-comparable unless the authors calibrate $k$ per paradigm or show robustness to this choice.\n\n4. Ambiguity in DQE's Scope of Comparison\n\nWhile DQE is clearly used for intra-method checkpoint selection, the paper implicitly uses DQE for cross-method comparison, but never explicitly discusses or clarifies whether this is valid., e.g. plot of DQE curves for different SSL methods side by side (e.g., Fig. 2–13), which could imply cross-method comparison, and highlighting certain methods (e.g., MoCo v3) as degrading more, based on DQE trends — again suggesting cross-method interpretation. This creates confusion and overextension of the metric’s intended scope.\n\n5. No Visual Validation of Clustering Geometry (**major concern**) \n\nThe paper provides visualization of similarity matrices of MoCov3 and DINO in Appendix. However, given the core claim is about cluster quality (separation and compactness), it is surprising that the paper provides no qualitative feature visualizations (e.g., t-SNE or UMAP) across checkpoints or methods. These are standard tools in SSL and would support the metric’s interpretability and degradation diagnosis.\n\n6. Lack of Efficiency and Overhead Analysis (**major concern**) \n\nThe paper presents DQE as lightweight and practical, but provides:\n- No runtime or FLOP estimates for DQE-based checkpoint selection, which involves feature extraction, k-means, and rank computation.\n- No training-time overhead analysis for DQE regularization. It is unclear how it scales with dataset size and model size.\n- No comparison to other low-cost model selection or early stopping heuristics.\nWithout this, it's unclear whether DQE is actually deployable at scale or just an academic tool.\n\n7. Lack of Scaling Study and Model Diversity. \n\nThe experiments are limited to small to medium models only: ResNet-50, ViT-Small, Swin-Tiny, and a single ViT-B model used only in the I-JEPA experiment. There are no results on ViT-Base or larger models across other methods, nor any discussion of how model capacity interacts with SDD or DQE. Since larger models are standard in current SSL pipelines (e.g., ViT-B, ViT-L in DINOv2, iBOT, MAE), this raises concerns about generalization. Representational degradation and rank collapse may behave differently at scale — and without including such models, the findings may not extrapolate to practical settings.\n\n> [1] Bardes, A., Ponce, J. and LeCun, Y., 2021. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906.\n\n> [2] Zbontar, J., Jing, L., Misra, I., LeCun, Y. and Deny, S., 2021, July. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning (pp. 12310-12320). PMLR. \n\n> [3] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M. and Piot, B., 2020. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33, pp.21271-21284.\n\n> [4] Yun, S., Lee, H., Kim, J. and Shin, J., 2022. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8354-8363).\n\n> [5] Hénaff, O.J., Koppula, S., Alayrac, J.B., Van den Oord, A., Vinyals, O. and Carreira, J., 2021. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10086-10096).\n\n> [6] Stegmüller, T., Lebailly, T., Bozorgtabar, B., Tuytelaars, T. and Thiran, J.P., 2023. Croc: Cross-view online clustering for dense visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7000-7009).\n\n> [7] Lebailly, T., Stegmüller, T., Bozorgtabar, B., Thiran, J.P. and Tuytelaars, T., 2023. Cribo: Self-supervised learning via cross-image object-level bootstrapping. arXiv preprint arXiv:2310.07855.\n\n> [8] Su, Q., Netchaev, A., Li, H. and Ji, S., 2023. Flsl: Feature-level self-supervised learning. Advances in Neural Information Processing Systems, 36, pp.6568-6581.\n\nQUESTIONS:\nPlease refer to the weakness above. \n\nI would be willing to adjust my score, provided that my key concerns are appropriately resolved.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a degradation phenomenon in self-supervised learning called Self-supervised Dense Degradation. It shows that longer pretraining often reduces performance on dense prediction tasks. To address this, the authors propose the Dense Representation Quality Estimator. This metric combines class separability and feature dimensionality to estimate representation quality. It is used for model selection and training regularization. The method is tested on multiple SSL models and shows consistent improvements in semantic segmentation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper identifies an underexplored phenomenon in dense SSL training.\n\n- The proposed metric is simple, theoretically motivated, and easy to apply.\n\n- DQE enables effective model selection and training-time regularization.\n\n- Experimental results show consistent improvements in segmentation performance.\n\nWeaknesses\n\n- The paper lacks ablation studies showing the effect of each DQE component when used as a regularizer.\n\n- All downstream experiments are limited to semantic segmentation. The metric's generality to other dense tasks such as object detection or depth estimation remains untested.\n\n- The metric's effectiveness is evaluated only through numerical scores. Since DQE assumes structure via spectral norms and class separability, qualitative visualizations such as t-SNE or PCA are needed to validate its interpretability, especially under collapse.\n\n- Although DQE is positioned as a dense-specific metric, its formulation may apply to image-level representations. The paper does not explore this possibility or analyze why degradation is observed only in dense tasks.\n\nQUESTIONS:\n- Can you provide component-wise ablations showing the effect of each DQE term in regularization?\n\n- Have you evaluated DQE on other dense tasks beyond segmentation?\n\n- Can you provide qualitative feature visualizations to confirm that DQE values align with actual feature structure?\n\n- How does DQE behave when applied to image-level representations such as classification?\n\n- Why do collapse and separability degradation occur in dense tasks but not in image-level tasks under SSL?\n\n- What do you believe causes the remaining uncertainty in DQE’s correlation with downstream performance?\n\nLIMITATIONS:\nThe authors do not discuss key limitations. While it proposes a metric to detect and mitigate SDD, it does not explore the underlying causes of SDD. While societal impacts are not discussed, no notable concerns are identified.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}