Title: Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
Venue: NeurIPS 2025
Accepted: yes
Keywords: computer vision

DINOv2

regularization

representation learning

domain adaptation

interpretability

medical imaging

efficient learning
Abstract: Vision Transformers (ViTs), such as DINOv2, achieve strong performance across domains but often repurpose low-informative patch tokens in ways that reduce the interpretability of attention and feature maps. This challenge is especially evident in medical imaging, where domain shifts can degrade both performance and transparency. In this paper, we introduce Randomized-MLP (RMLP) regularization, a contrastive learning-based method that encourages more semantically aligned representations. We apply RMLP when fine-tuning DINOv2 to both medical and natural image modalities, showing that it improves or maintains downstream performance while producing more interpretable attention maps. We also provide a mathematical analysis of RMLPs, offering insights into its role in enhancing ViT-based models and advancing our understanding of contrastive learning in this context.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a regularization technique called Randomized-Multi-Layer Perceptron (Randomized-MLP). Randomized MLP is a contrastive learning-based method that \nThis work conducts the experiment on top of DINOv2, and compares Randomized MLP trained model versus regular MLP fine tuned model, and shows Randomized MLP improves downstream tasks including semantic segmentation and depth estimation. Randomized MLP fine-tuned also show cleaner attention maps among nature and medical images. Authors conduct theoretical analysis to prove Randomized MLP impacts the contrastive learning paradigm by turning point embeddings into probability balls with the help of KoLeo regularizer.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n* The methodology is intuitive and mathematical details are well explained. Authors explain the impact of each component, e.g., contrastive losses encourage similarity between views, KoLeo promotes uniformity, and GELU restricts the space to mostly positive coordinates. This paper also provides detailed visualization to explain the impact of different values of \\lambda. \n* The proposed method is not bound to a specific architecture and in theory it can be generalized to other domains, although the actual movement needs to be evaluated.\n* Theoretical foundation: authors provide theoretical analysis on how the embedding is projected and how it provides regularization effect.\n* Straightforward and comprehensive experiment setup. This paper evaluated Randomized MLP across different downstream tasks on different datasets.\n\nWeakness\n* It would be a plus if authors can show how this regularization works with other models, since it looks very generalizable and it should be easy to apply to other models. Does it only help DINO style models, or vision models, or can be generalized to NLP tasks?\n* Although authors present the negatively skewed proportion curve reflecting the sparse representation, I don’t find quantitative results on to what extent the attention maps become cleaner or better aligned with human perception.\n\nQUESTIONS:\n* In contrast with previous work, our results show that this behavior also emerges in smaller versions of the model => Do you mean storing global information in patch tokens? Do you mean DINOv2-S is worse?\n* Line 87 “sparcity” => I believe you mean sparsity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve the interpretability of the attention maps and feature maps of DINO ViTs, especially for medical images. They propose to replace the MLP heads of DINO with randomized MLPs that are fixed and add Gaussian noise to the intermediate features. They prove that adding Gaussian noise parameterizes the features to probability balls, and KoLeo promotes feature uniformity. They show that the scale of the noise should not be too large (Fig. 2). In experiments, the method shows better attention maps and improved performance on some benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper considers improving the interpretability of pre-trained DINO features for medical usage, which is beneficial for social good.\n- The paper provides mathematical proof for some properties of DINO and the proposed RMLP.\n- The paper also provides visualizations to depict the properties of DINO and the proposed method (eg, Fig. 6). \n\n### Weaknesses\n- I noticed that in the provided code, the implementation of both DINOHead and the modified version omitted weight normalization (on the last layer) and also feature normalization. These are crucial designs of DINO to ensure that self-distillation is performed on normalized *prototypes*, and removing them has shown notable performance drop in the original paper. This could raise a critical concern about whether DINO is faithfully re-implemented and whether the comparison is fair. Why did the authors remove these operations?\n- The comparisons, especially visualizations of attention maps and PCA visualizations, are always provided in very low resolutions. In DINO, DINOv2, and Darcet et al., they provided higher-resolution visualizations that are much clearer for comparison between methods. It is strongly suggested that the authors also update these visualizations to higher-res ones, as currently it is hard to identify important parts, e.g., vessels in medical images.\n- Many conclusions in the proof (Sec. 4) are unsurprising and this part could be moved to later sections for clarity. For example, KoLeo regularizer is not an important part of DINOv2, and removing it does not affect model behaviour notably. The main regularizers are the centering and sharpening operations, and I wonder why KoLeo is picked for analysis. Also, the fact that adding Gaussian noise turns deterministic inputs into stochastic \"probability balls\", and the scale of the noise should be small, are trivial.\n- The structure and formatting could also be improved. Currently, the introduction section is overly simple, and the related work section is also missing. Besides, I would also suggest adding a formal introduction of DINO in the main text.\n\nQUESTIONS:\nCan the authors provide more details on how DINO is re-implemented, and why weight normalization and feature normalization are omitted?\n\nLIMITATIONS:\nThe limitations are discussed on page 9.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe  basis of the manuscript is that the the authors state with quite a bit of evidence that ViT models trained with DINOv2 repurpose low-information patch tokens with global information.  This is particularly problematic for tasks that are segmenting in nature.  The specific domain of concern is  ophthalmologic images where the chief task is segmenting OCT and eye fundus images. The results also show improved performance on natural image datasets.\n\nThe solution proposed is to replace the MLP layer with a novel Randomized-Multi-Layer Perceptron (RMLP). RMLPs replace the standard learnable MLP heads in DINO and iBOT with fixed, randomized linear operators that preserve structure.  The authors believe this guards against shortcut learning by preventing token-level overfitting.  The goal is also to avoid major archetectural changes. \n\nThe results show qualitative improvements in depth estimation and segemenation on natural images.  Heatmaps on ophthalmologic images suggest that there is improved attention to the objects of interest and less attention attributed to the non-informative parts. This is most compelling in the OCT images. Quantitative results for segmentation and classification tasks are also provided in conclusions with some mixed results.  Most dramatic performance improvement is in \"pathology classification\" using Nearest Neighbor clustering.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nReplacing MLP with RMLP might correct for hypothesis (with fair amount of evidence) that ViT models trained with DINOv2  store global information in low information patch tokens. \nThe theoretical proofs are very thorough and only have minor flaws.  I have some clarifying questions below \n\nWeaknesses:\nThe heatmaps should have a legend to be considered interpretable. Simply stating \"Brighter maps indicate more attention.\" is not adequate for full interpretation of the figures. \nAs a person not familiar with the domain of OCT and fundus image, I am not fully able to understand the goal of analyzing these image. Broadly defining these as \"medical images\" is also in appropriate because the goals for many other medical imaging tasks seem to be quite different from these ophthalmologic tasks. \nResult section are largely qualitative which limits evaluation. Quantitative results are in the Conclusion section.\n\nQUESTIONS:\nPlease better define \"probability balls\". It is a confusing terminology and while I think I conceptually understand, it is hard to be certain. \n\nI understand that you initialized with DINOv2-S trained on natural images and further trained on the different dataset after replacing the MLP heads.  Is there any advantage to this over training from scratch with RMLP only?\n\nIf the fundamental break through of the proposed algorithm is in improved segmentation, how does one understand the improved performance in classification task in Table 2a? \n\nDINOv2 is the basis of many SOTA models in histopathologic image classification. Do you think that incorporating RMLP would improve performance in this domain?\n\nLIMITATIONS:\nLimitations section is perfunctory at best. This needs to be enhanced to be acceptable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Randomized-MLP(RMLP) to improve domain adaptation interpretability and reduce attention map artifacts in DINOv2 by replacing learnable MLP heads with randomized, non-trainable operators. The method aims to address artifacts where DINOv2-S repurposes low-informative patches for global information storage, particularly problematic in medical imaging. The work combines theoretical analysis with empirical validation across  both natural and medical image domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n \n(1) Lightweight component: RMLP requires no architecture modifications and trainable parameters that can be applied during fine-tuning with minimal computational overhead. The method is theoretically motivated and simple to implement.\n(2) Solid theoretical framework: Theorems 1-4 provide comprehensive mathematical analysis of how RMLP regularization influences contrastive learning geometry, offering valuable insights into the mechanism underlying the observed improvements.\n(3) Comprehensive experiments: the experiments show multiple domains and tasks such as classification, segmentation and depth estimation with proper statistical significance testing using Mann-Whitney U tests.\n \nWeakness:\n \n(1) The paper shows experiment results only on DINOv2-S based models, experiments on other relevant models such as DINOv2-base and large are untested.\n(2) Apart from Figure 6,this paper relies heavily on visual results to show the enhanced interpretability of attention maps. Providing more quantitative evaluation results would strengthen the claims.\n(3) RMLP shows performance degradation on some downstream tasks, indicating the method seems to sacrifice linear separability for neighborhood structure.\n\nQUESTIONS:\n(1)Do you have a principled method for selecting λ for new domains, or is manual tuning required?\n \n(2) Could RMLP be implemented on other self-supervised ViTs or the method is specific to DINOv2's architecture?\n\nLIMITATIONS:\nThe authors have properly addressed its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-07"}