Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
Authors: Joel Valdivia Ortega
Lorenz Lamm
Franziska Eckardt
Benedikt Schworm
Marion Jasnin
Tingying Peng
Venue: NeurIPS 2025 poster ()
Keywords: computer vision
DINOv2
regularization
representation learning
domain adaptation
interpretability
medical imaging
efficient learning
Primary area: deep_learning

Abstract: Vision Transformers (ViTs), such as DINOv2, achieve strong performance across domains but often repurpose low-informative patch tokens in ways that reduce the interpretability of attention and feature maps. This challenge is especially evident in medical imaging, where domain shifts can degrade both performance and transparency. In this paper, we introduce Randomized-MLP (RMLP) regularization, a contrastive learning-based method that encourages more semantically aligned representations. We apply RMLP when fine-tuning DINOv2 to both medical and natural image modalities, showing that it improves or maintains downstream performance while producing more interpretable attention maps. We also provide a mathematical analysis of RMLPs, offering insights into its role in enhancing ViT-based models and advancing our understanding of contrastive learning in this context.

Randomized-MLP Regularizatioo Improves Domain Adaptation and Interpretability in DINOv2
Joel Valdivia Ortega 1
Lorenz Lamm 1,3,4
Franziska Eckardt
Benedikt Schworm 5
Marion Jasnin 2,6,
Tingying Peng 1.*
Helmholtz AI, Helmoltz Munich, Neuherberg, Germany
Helmholtz Pioneer Campus, Helmholtz Munich, Neuherberg, Germany Schooo of Computation, Information and Technology, TUM, Garching, Germany Biozeentrum, University of Basel, Basel, Switzerland Department of Ophthalmology. LMU University Hospital, LMU Munich, Munich, Germany 6 Department of Chemistry, TUM, Garching, Germany.
[joel. wallelivvviii lorenz lamm, marion. jasnin, tingying. pengg)peeggggee @helmhol www.iiichh.mmm....mmmmmmmmmmmccc de (franziska sckardd,beeedikt schworm @med. uni wwi-mmuunceen..m
Abstract
Vision Transformers (ViTs), such as DINOv2, achieve strong performance across domains but often repurpose low-informative patch tokens in ways that reduce the interpretability of attention and feature maps. This challenge especially evident in medical imaging, where domain shifts degrade both performance and transparency. In this paper, we introduce Randomized-Mi ization, contrastive learning- based method that encourages more semantically aligned representations. We use RMLPs when fine-tuning DINOv2 to both medical and natural image modalities, showing that it improves maintains downstream performance while producing more interpretable attention maps. We also provide mathematical analysis of RMLPs, offering insights into its role in enhancing ViT-based models and advancing our understanding of contrastive learning
(RMLP) regular-
Introduction
Learning robust visual representations remains a central challenge in computer vision. Transformer- based models, such as Vision Transformers (ViT's [1I) have emerged as powerful backbones, especially when trained with self-supervised methods like contrastive reconstruction-basee learning 32 20). These approaches yield generalizable models, which can be fine-tuned for specialized tasks, particularly in domains with limited labeled data, like medical and biological
330 5 15
imaging [21 9 25).
However, despite their empirical success, ViTs exhibit persistent issues in how they encode and distribute semantic information. Previous studies have observed that large ViTs often store global context within semantically weak background regions [4] 38 9]. These structural artifacts can
"Corresponding authors
'Code and pre-trained models are available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Attention Maps
PCA Visualization
DINOv2-S DINOv2-S
DINOv2-S DINOv2-S
Input
DINOv2-S
DINOv2-S
+ MLP
R3-MLP
+MLP R5-MLP
Fundus OCTID BSDS300
Glaucoma
Figure 1: Left: Input image from BSDS300 27 OCTID [1] and Glaucoma Fundus 2 datasets from top to bottom. Middle: Second- (BSDS300) and first-order (OCTID/Glaucoma Fundus) attention maps (see Section (T) over grayscale image. Right: Visualizations of top-3 PCA of patch tokens using DINOv2-S, DINOv2 fine-tuned on selected modality, and DINOv2 fine-tuned on selected modality using our regularizer. Colors assigned in the RGB regime as the norm of the principal components. More qualitative results are available in Appendix A.l
degrade performance on dense prediction tasks and compromise interpretability. a critical concern in biomedical domains [16 46).
Our analysis reveals that these artifacts are not limited to large ViTs. Even though the small variant of DINOv2 (30] performs well on natural image tasks (Table [T). still exhibits anomalous attention behavior. In medical imaging, DINOv2-S struggles to generalize to dense tasks such as segmentation (Table D. despite achieving strong performance on classification tasks (Table Bay. This supports the hypothesis that it sacrifices local detail in favor of encoding global information through uninformative tokens. By analyzing patch token norms and their principal components, we identify first-order Prrifacts-where high-norm tokens align with background regions in ophthalmological images and second-order artifacts-when where the variance captured by the top three principal components is misaligned with semantic relevance in natural images (Figure D First order artifacts are more pronounced in medical images, where DINOv2-S statistically attends more to void or anatomically irrelevant regions than to biologically meaningful tissue (Table[4a).
To address these issues, we propose the Randomized Multi- Layer Perceptron (RMLP), a lightweight, theory-driven regularization module that improves semantic alignment in ViT representations. RMLP motivated by the geometry of contrastive learning 351 and enhances interpretability without requiring retraining or architectural changes. When applied to DINOv2, RMLP yields significant improvements both natural and biomedical domains, achieving state-of-the the- classification and segmentation performance ophthalmology datasets with minimal computational overhead. Additionally, we demonstrate that RMLP generalizes beyond DINOv2 by fine-tuning SwAV model trained under different self-supervised paradigm- and showing it consistently improves performance across modalities, indicating the broader applicability of our approach.
2 Related Work
Artifacts in Transformer Representations. Transformers are known to exhibit uneven attention allocation across input tokens. In NLP, Xiao, et al. [4]] showed that disproportionate attention, regardless of their semantic importance. Sun, et al. [38] attributed such behavior to sparse, high-norm activations. Extending these observations to vision, Darcet, et al. 9 found that ViTs often produce a small set of high-norm patch tokens concentrated in background regions, which act as "registers" for global context. While these patch repurposing have been mostly studied in scage ViTs, work shows it also emerge in smaller models like DINOv2-S.
tokens receive

---

Mitigating Attention Artifacts. Prior attempts to address attention artifacts have focused on heuristic or architectural solutions. Darcet, et al. proposed adding learnable tokens and retraining the model on proprietary datasets. Jiang, et al.
[8] showed that even untrained tokens can mitigate
high-norm anomalies by absorbing excess activation. Others introduced auxiliary loss terms [40] or attention-smoothing -moothing modules 422 to encourage more uniform feature distributions. While these methods show empirical improvements, they lack theoretical grounding, require expensive retraining. or cannot be used as domain adaptation techniques. In contrast, proposed RMLP module is easy to integrate, requires no retraining, and is grounded in the geometry of contrastive representation spaces. It also improves both semantic fidelity and interpretability across modalities.
3 Randomized-Multi-layyy Perceptron (RMLP)
Central to our approach is the observation that the structure of the representation heads neadlynamely,, the DINO [5] and iBOT [44) heads-play a key role in how information is encoded across patch and class tokens. In DINOv2, the contrastive framework aligns student and teacher global representations via the DINO head (operating on the class token), while the iBOT head introduces masked image modeling by aligning patch tokens through a cross-entropy objective. Both heads are implemented as MLPs.
Building the findings of Darcet, et al. (9, we hypothesize that the representation heads a key enabler of this hehavior. To counteract this, we replace the learnable MIP heads with a randomized, non-trainable operator designed preserve the topology of the representation space. This encourages the backbone to learn more robust and interpretable features, while preventing the heads from exploiting token-level classification shortcuts.
To avoid modifying the architecture, we first express the structure of the DINO and iBOT heads. A standard MLP [30] can be written f = 6, ay .... a1 o Ø1, where oi are linear layers and ag activations. We replace each d, with a randomized map (P: : RR - RR" as
P 2m)) (Z1. zn,) Ir (
where (z) ( )n can be truncated pero-p version of the input vector, T is Gaussian matrix with i.i.d. entries drawn from (0, A/n) and À is a tunable amplitude. This operator also be seen residual connection. Thus, we formally define Rx-MLP
= Fr Qr-1 .... o 01 p1-
RMLPs introduce no trainable parameters, yet remain fully compatible with end-to-end contrastive training objectives like DINO [5] and iBOT
Theoretical Analysis
We now develop a topological framework to analyze how ViT embeddings evolve while passing through transformer blocks (Theorem (T) the way it generalizes from its training data into bigger domain (Theorem 2 and Corollary B and how our RMLP regularizer impacts the contrastive learning paradigm (Theorem 4.. This enables the characterization of RMLPs as random operators which turn point embeddings into probability balls (Corollary 5). improving robustness and promoting sparcity without altering the topology of learned representations when an adequate amplitude is chosen.
Theorem 1. Let Q be an image space and a latent space. A VIT : + vi can be decomposed as a tokenization function C Ω V followed by sequence transformer blocks T T V, which are defined by local orthonormal bases generated in the attention heads by the queries, keys, and values layers along with the input data itself.
Proof. ViTs use the attention mechanism to decompose the data using queries and keys layers, creating a field that data follows during the representation process. The value layers act as embeddings within the space defined by queries and keys. For a complete proof, refer to Theorem TT in Appendix A.2
Training with the KoLeo regularizer
35 ensures that V remains injective since the loss, namely,
LKOLOO (fx1,...,,
Σ log
min j+t
||
xz x512
3

---

would diverge otherwise. Continuity of C. along with KoLeo and augmentations, guarantees that T is locally injective (see Def. 6 on C[S2]. These properties allow T to generalize from its training data into V while preserving its embedding ability as shown in Corollary
Theorem 2.
Being an homeomorphism an invertible continuous function with continuous inverse, assuming T is locally injective and taking V be a metric space, there exists & > 0 such that T is an homeomorphism on {ar E V 0(x, [[]]) < <)}, where ó is the distance on V.
Proof. T is continuous from Theorem | Since images can only take values from a finite set, Ω is bounded and by construction, C. is continuous. Further, taking the codomain of and using again that images take discrete values, we can assume C[92 is compact because of being finite union of closed sets, namely singletons of images. Thus, for Lemma [ Appendix A.2 an E > 0 exists such that I is injective in {I y o(x, ((2]) < E). i.c., is invertible in that subset. Furthermore, since V is metric, the result follows from Lemma 13 Appendix A.2
metric space
Corollary 3. If S Q the training data, and T locally injective on A, the following holds:
There exists 0 such that T is an homeomorphism s-cloud containing A (see Def.
Let P.Q Q. [[])C[Q] disconnected & if and only iTV[P] is disconnected in V (see Def. 8 which can contribute to the batch effect.
If D C V is dense in v (see Def. 9). then V[D] is dense in V(v..
Proof. A complete proof is in Appendix A.2 in Theorem 18
Theorem 4. Let (pi...., .PN.. S R", O, and A 0. with I matrix of size with
i...d normal entries N'0,, >) Then, for each x E R", [[[ = mAn 1|1:118 Moreover,
I produces an s-distortion (see Def. 14) on the E probability if
p: P Alliiii = N NJ with high
p
An
In N
Proof. (Sketch) Using concentration results for Gaussian matrices, we follow 28 can be found at Appendix AA.2 Theorem 19
A complete proof
Corollary 5. If (21...., NN) = E8" mm An-1 < 8² 22(811 N), then for all e (1, (1....,Ny
O, and yp: R R defined as Eq. with
[ Is,n) - (xx))112 <E
with high probability.
Proof. This follows directly from the definition of oo and Theorem
We now analyze the impact of RMLP heads on image embeddings. Let us take an RMLP g as in Equation 2 with amplitude À and GELU activation acting the embedding V(z) ((zug) ((si))ssaacs jsd)isu. Applying Corollary 5 the first random layer yo1 turns each token z; into a ball of radius e, determined by the embedding's is dimension, X, and the number of tokens v.
By Theorem 4 the output of ypi (V(z)) lies on sphere of radius d'An;' on expectation. The con- trastive losses encourage similarity between views, while the KoLeo regularizer promotes injectivity, However, the GELU activation restricts the space spherical region with mostly positive coordi- nates, where dispersion is optimized via approximate orthogonality among embeddings. Successive layers P2,, iPs, (P4 expand the radius of probability balls further. Thus, go V(x) becomes point cloud creating probability ball, rather than individual points, regularizing the learning by applying cross- entropy over neighborhoods. If the RMLP amplitude X too large, the topology of the embedding space may be distorted, as shown in Figure Remarkably, the topological properties of ViTs remain unaffected whether ML.ss or RMLPs heads are used during training, as our analysis independent of it.

---

0.1
0.3
0.5
Figure 2: Visualization of randomized operator following Equation | applied the unit circle in _ (A/2 = 0) for different values of A. For A/2 = 0.1, the point cloud retains a circular structure, preserving the topology of the circle. For A/2 0.5, the point cloud no longer exhibits circular topology, as it transitions to disk-like shape.
Implementation and Training Details
Model Variants. We use two ViT- self celf-supervised models: DINOv2-S (30 and SwAV [
Fine-tuned variants include standard MLP heads and our regularized version using RX-MLP. Fine- tuning follows the DINOv2 protocol with the original head architecture retained. Hyperparameters and external code can be found Tables
Training Modalities. Each backbone is fine-tuned separately three modalities: ImageNet- Ik (natural), Colour Fundus Photography (CFP), and Optical Coherence Tomography (OCT) (Table (7a). Medical datasets sourced from public subsets aggregated in RETFound [43 ensuring geographic diversity (Table 7b).
public
Hybrid Architecture for Dense Tasks. For pixel-level predictions, we pair the VIT encoder with UNet-style decoder 13 Patch and class tokens are projected and fused with early UNet features, enabling multi-scale feature integration. This ViT-UNet hybrid is used for segmentation and depth estimation tasks.
6 Experimental Setting
Evaluation Setup. We fine-tune DINOv2-S and SwAV with - at amplitudes [0.1, 5, 10, 20), running 10 trials per modality and setting. Baselines use MLP heads. Downstream tasks employ either linear probes or the hybrid decoder depending on task type. Results are averaged over runs and tested for significance using the Mann-Whimne U test (26). appropriate under distributional assumptions. Qualitative results for both natural and ophthalmological modalities can be found in Appendix A.I
Natural Image Tasks. On ImageNet- 1k 34 we perform image classification training with cross- entropy. For semantic segmentation of 45 and depth estimation of [36), we apply a linear head and the ViT-UNet hybrid. Segmentation is trained with linear combination of focal 24 and dice loss (29]. Depth maps are rescaled to [0, 50) and trained with focal loss. Table reports performance
compared to baselines
40
OCT and CFP Tasks. We assess disease classification and retinal layer segmentation. Segmenta- tions emphasize the ophthalmologically relevant Outer Nuclear Layer (ONL). UNet output biases initialized (-2 for background, 2 for others) for stable training. We benchmark against RETFound
43 and domain-specific model 12 (Tables
Cross-Modal Attention Analysis. We visualize attention maps across modalities: second-order for natural images (27) and first-order for OCT 14 and CFP (2. High vs. low-information patches are separated using 2-component Gaussian Mixture Model smoothed gradient at patch level.
5

---

Ground Truth
DINOv2-S + MLPP
DINOv2-S + R3-MLP
OCT
RETFound
DINOv2-S
Figure 3: Semantic segmentation of retinal layers the dataset from Bckardt, al. 12] (OCT) using different backbones (columns) in a ViT-UNet hybrid. Black arrows highlight cases where fine-tuning DINOv2-S with R3-MLP outperforms other models.
DINOv2-S R5-MLP Imput
DINOv2-S M5-MLP Input
Figure 4: Top: OCTID 14 (left) and Glaucoma Fundus 2 (right) images. Middle: First-order attention maps from model fine-tuned with R,MMLP overlaid on grayscale version input image. Bottom: First-order attention maps from DINOv2-S overlaid on grayscale version of input image. Brighter maps indicate more attention. While DINOv2-S focuses void regions, DINOv2-S with R5-MLP attends to key anatomical features in the OCTs suchh as the foveal pit, macular hole, and epithelial detachment and highlights the rim and blood vessels in Fundus images.
7 Results and Discussion
Effectiveness of RMLP Regularization. Across natural and medical image domains, fine-tuning with RMLMPP consistently outperforms MI.P baselines. natural images, better preserves performance compared to other regularizers. In the medical domain, despite limited data and single-GPU training. -MLP achieves state-of-the-art OCT segmentation (Figure 3 and improves classification, especially with I-NN, indicating enhanced representation quality (Table 3). While linear-head classification with R,-MLP slightly trails state-of-the-art or DINOv2-S, this reflects differing latent space structures since RMLPs promote neighborhood-basee similarity (Theorem Corollary 5). whereas MLPs enforce linear separability. Larger À values degrade performance by altering representation topology (Tables 3 Figure 2}. while smaller amplitude values do not consistently improve downstream results, highlighting a tradeoff between retaining information and preserving topological structure VITs need overcome during fine-tuning.
Enhancement on Cross-Paradigm Learning. RMLPs enable effective adaptation of models trained under different learning paradigms, such as SwAV 4, when combined with DINOv2 learning algorithm (Tables 2b 3b)). While applying DINOv2 to standard MLPs also yields gains, RMLPs consistently deliver superior performance across tasks.

---

Table Performance across downstream tasks on natural images. Semantic segmentation on ADE20k 45 depth estimation on NYU-Depth V2 [36] and image classification on ImageNet- 1k [34]. Bold indicates best mean, second-best best on for each base model (DINOv2-S, DINOv2-G and SwAV). Underlined values are significantly better than DINOv2-S (p < 0.001); and denote significance over base model fine-tuned with MI.P at p < 0.1 and p < 0.001 (Mann-Whitney U test 26 show std from ten independently fine-tuned backbones.
Results
Model
Semantic Segmentation 1
Depth Estimation
Image Classification
ViT-UNNet Hybrid
Linear Head
ViT-UNet Hybrid
Linear Head
I-Nearest Neighbor
Random Forest
Linear Head
DINOV2-S
DINOv2-S 5D
0.81:0.1
'0.7611.11"
7+:"
9=3"
0.70
0,18
0.77:10.01
DINOv2-S MLP
0.7810.1
0.75
1813
'10.13
0.64.10.01
0.14
0.73.10.01
1o.1-MLP
0.66--0.1
84:1
0.64-:00001
0.15:00.11
0.74-40.06
DINOv2-8 MLL
10.8010.11
0.77
71:3""
91:3"
"0.65:10011"
00.16:0.011
00.76:10:01
DINOv2-8 R1o-MLP
10.80 "
0.77.+0.1
74:3"
9=3""
10.65+0.01"
0.15+00.11 "*
0,77=10011
S Rgo MLP
10.2
0.77 0.11
71.3"
9.13"
0.64
0.14
0.75
Registers D
0.72+0.2
0.67
71:1
9
0.28
0.05
0.74
DVT 12
0.4510.2
0.64
7.11
911
0.11
0.64
DINOv2-G
Sinder
0.51+0.2
0.56+0.1
6+3
8+1
0.78
0.12
0076=00.11
SwAV
SWAV 4
0.37:0.1
12:11
0.69:10.01
SwAV MLP
0.29++0.08
113+1
0.69-+0.01
SWAV Ry-MLP
0,34+0.09
113=2
0.69
SwAV R1L-MLP
0,271001
12:1
0.69110.01
SwAV R20-MLP
10,36+0.1
133
0.69+t0.01
Table 2: Performance metrics for semantic segmentation on Eckardt, et al. [12 dataset using a ViT- UNet hybrid. Results show std of DICE scores from independently fine-tuned backbones using DINOv2 and AW as base models. Bold numbers indicate best performance, the second- best, and underlined results outperform RETFound (p < 0.001, Mann- Whittey U test). Statistical significance vs. base model fine-tuned with MLP is shown by (p 0.1).
(a) Base model: DINOv2.
(b) Base model: SWAV.
Model
Averaged DICE
DICE on ONL
Model
Averaged DICE
DICE on ONL
RETFound 43
0.92=0.06
0.59=0.10
SWAV f
10.8910.15
0.5911..14
Registers
0.9210.1
0.62#0.21
SWAV MLP
0.88
10.6.0.18
Sinder 40
0.90.10.21
0.61
SWAV 1-LP
0.92:10.12
0.64:10.1
42
0.78-20.25
0.43-20222
SwAV
Ra MLP
0.83.10.27
0.581002
DINOv2-S 30
0.7910.20
0.5420.20
SWAV R1o-MI.P
0.71 0.333
0.42--0.23
DINOv2 ML.P
0.86:10.20
0.55
SWAV R20-MLP
0.84=0.26
0.58+0.21
DINOv2 + Rg, 1-MLP
'0.9420.12
00.682021
RG MLP
0.72:10.9
DINOv2 R10-MLP
0.87 0.27
0.61 0.20
DINOx2 R20-MLP
0.70-20.30
0.47+0.20
Eckardi, 12]
0.92.10.03
0.44
Attention Artifacts on Natural Images. Fine-tuning ImageNet-Ik slightly reduces the CorLoc score for both MIP and RMLP heads (Table 4b). a minor drop consistent with other regularization methods and attributable to fine-tuning rather than the RMLPs. While regularized models from liter- ture maintain global context and classification accuracy comparable to DINOv2-S, they degrade patch token quality in dense prediction tasks (Table []. Notably, DINOv2-S encodes global information in patch tokens from low-information regions, a tendency amplified by regularized models. Fine-tuning with MLPs RMLPs reduces this artifact, equalizing patch token behavior, with RMLPs better preserving downstream performance post fine-tuning (Table 4).

---

Table Performance metrics for pathology classification across OCT and CFP modalities. Results show std from ten independently fine- tunettuned backbones. Bold numbers indicate best perfor- mance, ? the second-best, and underlined results outperform RETFound (p < 0.001, Mann- Mann-Whitney U test). Statistical significance vs. base model fine-tuned with MLPP is shown by (p<0.1). (p
0.01), and (p < 0.001).
(a) Accuracy for pathology classification using DINOv2-S base model.
Dataset
DINOv2.8 2
DINOv2-5 MLP
DINOv2-S
DINOv2-S RR-MLP
DINOv2-S
DINOv2-5 + R2u
Ro. 1MLP
+ MLLP
I-Neareat Neighbor classification
OCTID !E!
(0.8)
10.75"
0.69-0002
0.68+0001
0.78=0.01 "M
0.7340.01 '
0.68+0.01 "
Glaucoma Fundus
(0.78)
0.67
0.64.10.01
10.01
0.71110.09
10.6910.009
0.6310.07
IDRID S1
(0:44)
0.42
0.43=0.03
0.36=0001
0.47=00011
"0,44:0.02"
0.43=0.04
JSTEC x
(0.45)
0.59
0.51+-0.22
0.51
0.67-+0.01
10.65+0.00""
0.59+0008
MESSIDOR-2
1
10
(0.56)
00.53
0.47=0.01
0.43=0.01
0.55==0.01
10.53++0.08"
0.52+0.08
PAPILA A2
(0.63)
'0.71 "M
0.65.10.03
0.65110.01
0.74 0.002
1..7110.001
0.65
Retina 6
(0.5)
0.54
0.54=00.2
0.39
0.58
0.58:+0.011
0.57
Random Forest classification
OCTID
(0.81)
0.75
0.54+0.01
0.58-0001
"0.74=0.01"
0.73+
0.69++0.22
Glaucoma Fundus
(0.73)
0.69
0.65:1008
0.67:00.1
"0.6810.08"
0.67:10.08
1..68
IDRID [1]
(0.4)
"0.49"
0.46=0.01
0.47=0.01
0.47=0.01
0.46=0.01
0.5+0.05
JSIEC X
(0.28)
0.28
0.26:0.08
0.25=:001
0.32=00.8
0.29::00.06
0.2810006
MESSIDOR-2 00
(0.58)
0.58
0.58=0001
0.58=0001
0.58
0.58++0.01
0.58-0.01
PAPILA AB
(0.69)
0.72
0.68 0..81
10.01
0.71 0.01
0.71
0.69 10.09
Retina 6
(0.59)
0.59
0.55=0007
0.55=0.01
0.59
0.05
0.58
Linear classification
00.99++00055
0.88++0088
0.76=0002
0.82:40011
"0.87:00011"
10.87
0.86+0.01 "
Glaucoma Fundus
3
(0.83:10.06)
0.79
0.71:1009
0.73:10.2
0.75:10.01
0.75
00.7610.011
IDRID 31
(0.44.10.04)
0.04
'0.44.10.03
0.08
0.42,10.03
0.40
"0.44_0.03
JSIEC x
(0.72::0.01)
0.73=2007""
0.63=0.01
0.68=0002
0.78=:0.01
1..76+00.01
0.73+0.03"
MESSIDOR-2 2 10
(0.56++0.03)
0.46++0.08
*0.52=00.03
0.51
0.54
T..52+0.08
0.49-0.09
PAPILA
22
(0.67.10.007)
0.71
0.66:10.04
0.66
10.69.10.03
0.710.03
'0.69
Retina 6
00.51_1...03)
0.52.10.04
0.53
0.51 10.02
*0.54_10.03
1..04
0.55 0.02
(b) Accuracy for pathology classification using SwAV as base model.
Dataset
SwAV 4
SwAV+ MLP
SwAV- Ro.1-MLP
SwAV+ R5-MLP
SwAV+ R10-MLP
SwAV+
Rao-MLP
OCTID [4]
(0.93=0.055
0.84=0.02
10.83=0.02
0.84:1002
0.84-0.02
0.82+:0.22
0.84+:0.02
Glaucoma Fundus
2
(0.83=0.06)
0.76:10.01
0.76:10.11
10.75::0.01
0.76:10001
0.76:1:0011
0.76:t0.02
IDRDD
(0.44=00004)
0.45+0003
0.49:10.22
0.47++0002
0.49:10.04
0.47+00.22
T0.48:+0.04
JSEEC
7
(0.72:10.01)
0.73:1001
'0.72:10.02
10.72:10.01
0.7311001
0.73:10.01
10.72.10.11
MESSIDOR-2 [0.000.56+0.033
0.55=20.4
0.55+0.03
0.55::0.02
*0.54+00.22
0.55:1:.02
0.55
22
(0.67:0007)
0.63:0.05
10.6520.04
0.62::0.3
0.66
10.65:10.44
0.64
Retina 0
(0.51
0.52+0.03
10.53+0.03
10.53+0.02
0.54+0002
0.52+0.02
0.54+0002
Attention Artifacts on Ophthalmological Modalities. Both RETFound and DINOv2-S, along with its regularized variants from literature, achieve strong classification accuracy ophthalmology datasets using patch tokens alone (Table 4c). even from void regions, indicating excessive global information leakage. Moreover, these models exhibit weak negative correlation with retinal layer presence (Table[4a) Figure In contrast, models fine-tuned with RMLPs (using suitable À) show better anatomical alignment and reduced attention artifacts well better performance in dense tasks (Table 2). This is crucial, as interpretable pathology detection relies on attending to anatomically relevant regions.
ViT-UNet Hybrids. Combining VIT backbones with UNet-style decoders consistently improves performance on dense prediction tasks (Tables 8). highlighting the benefit of integrating global context with local spatial detail.
8

---

Table 4: Evaluation patch tokens repurposing.
(a) Pearson correlation between patch tokens on top ranked by norm and presence of retinal layers in Eckardt, et al. 12
(b) Corl lac scores applying LOST [3 with default parame- ters on VOC07 L3 accuracy on ImageNet- 1k using 1- Nearest Neighbors on class tokens and patch tokens of mini- mum/maxinum norm of second-order attention maps from low and high information patches respectively (scc Section I. Bold and underline indicate best and second-bor -bstt accuracy per model.
Model
Correlation
RETFound
4N
0.0+0.011
Registcos
-0.19=0.13
Model
Corl.nc
Classification accurecy
Sinder 10
0.0:10117
DVT A2]
0..77.0.009
Registers
35.20
0.71/0.38/0.34
DINOv2-S (50
-0.13-4.144
Sinder AN
33,97
37770.39/..17
DVT 22
31.23
0.71/0.42/0..1
DINOv2 MLP
0.19.00.3
DINOv2-S
077/0.311/0.26
DINOv2 Ro. MMLP
0.15+:0.02
DINOv2-S ML.
31.18:00.03
0.33=0.0100.200-0000
2,000000 18 +0.01
DINOv2 Rs-MLP
0.21=0.033
DINOV2-S Ru, -MLP
31.20++0.04
0.63=0.0110000000000
+0.01
DINOv2 R10-MLP
0.2+0.04
DINOv2-S- Rs-LPP
31.17:10.05
004-200010221100001999000000
DINOv2 R20-MLP 0.16+0.001
DINOv2 S R10 MLP
31.17.10.05
014_100118211000000111
DINOv2-S R20-ML.P
31.19:0007
065-:00010200000000000000000
(c) Pathology classification accuracy on Eckardt, et al. 12] OCTID 14,, and aggregated CFP datasets 22 [1 (Z) 10 22 (6)) using Random Forest with class tokens and patch tokens of ciiimuum/maximnn
norm from low and high information patches respectively (see Section [T. Bold and underline indicate best and second-best accuracy per model per dataset.
Model
Fekardr,
PP
OCTID 14
CFP
RETFound
1,00,922
0.77/0
0.53-00.1700551+0 17/0
Registers 9
0.83908779.86
0.67/0.55/0.62
0.54=0..600,5+00000000 1.0.
Sinder 40
0,76/088/0.6
0,78/0.56/) 55
1.54:0.160.35001660.6
DVT 22
0,77/0, 7/0.4
0.76/0.610.557
0.55=00.40.:11 12000555110.000002100
DINOv2-S
50
0.74/08849.77
0.73/0.48/0449
DINOv2 + MLP
0.94-008006700007000000
0.55+
0.S-:0.5500477007774777
DINOv2-S + Ro. 1MLP
0.95100.200.88:000000000
0.55 110:0000200.600000000:0000
DINOv2 + MLP
0.96 0.02/ 0.67
1.0p:/0066_..000.... 0.07/0.66........ 0.02
0.55 1 0.000.3..66666. 11..0...6..........1111...
0.53 10..7...0.........
DINOV2-S R1o-MPP
DINOv2.S + R20 MLP
MLP
0.95 005-100020011100707770000
0.55 11510000.00331100110..77.111
0.3::0..10.4111..7077...000
8 Conclusions
Our work shows that RMLP regularization enhances the interpretability and robustness of VIT representations across both natural and ophthalmological domains. By inducing sparsity in patch tokens and mitigating first- and second-order artifacts, fine-tuning using R, -MLPs produces more structured embeddings. Despite being trained on limited data and using small computational resources, Rx-MLPs help ViT's achieve strong performance in classification and dense prediction tasks both on natural and ophthalmological images. These results highlight randomized-Mo -Lrsss as a lightweight and effective approach for regularizing representation geometry, pointing to promising direction for developing more semantically grounded and interpretable vision transformers.
Limitations. The optimal regularization strength may depend the data modality or the dimension- ality of the ViT's latent space. However, this work does not propose principled method for selecting it, relying instead on heuristic tuning. Additionally, our experiments are currently limited to small- and mid-scale datasets. Future work should investigate performance at larger scales and across broader range domains.
Acknowledgments and Disclosure of Funding
J.V.O. and L.L. received support from the Helmholtz Association under the joint research school "Munich School for Data Science MUDS". T.P and B.S. received support from the DFG grant 51 513025799. We want to thank Salome Kazeminia and Sophia J. Wagner for the valuable inputs and conversations during the making of this paper.

---

References
[1] Abrämoff, M. D. et al.: Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol. 131, 351-357 (2013).
[2] Ahn, J. M. et al.: A deep learning model for the detection of both advanced and early glaucoma using fundus photography. PLoS ONE 14, c0207982 (2018).
[3] Aptos 2019
blindness
detection (2019) Kaggle.
Available
at:
https://www.kaghhe hatpa//mmeeettinaaaoooooo/laaaee
(Accessed:
30 April 2025).
[4] Caron, M., Misra, I., J., Goyal, P., Bojanowsky, P., Joulin, A.: Unsupervised Learning of Visual Features by Contrastive Cluster Assignments. NeurIPS. a7X1v:200609882255 (2021)
[5] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A.: Emerging properties in self-supervised vision transformers. ICCV (2021).
[6] Cataract dataset (2019) Kaggle. Available at: http://www..aaggec.oommmmmmmrrpp blindnes-ddttcctionnaaaa (Accessed: 30 April 2025).
[7] Cen, I.-P. et al.: Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nat. Commun. 12, 4828 (2021).
[8] Chen, C., et al.: MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Seg- mentation. arXiv. xrXiv:2309.08842 (2024)
[9] Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv. arXiv:2309. 16588 (2024)
[10] Decencière, et al.: Feedback on a publicly distributed image database: the Messidor database. Image Anal. Stereol. 33, 231-234 (2014).
[11] Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint 1:Xiv:2010.1192) (2020)
[12] Eckardt, F., Mittas, R., et al.: Deep Leamnin-Baased Retinal Layer Segmentation in Opti- cal Coherence Tomography Scans of Patients with Inherited Retinal Diseases. Klin Monbl Augenheilkd. doi: (0.1055/a-2227-377
(2024)
[13] Everingham, M., Van Gool, L., Williams, C.K.I. al.: The PASCAL Visual Object Classes (VOC) Challenge. Int J Comput Vis 88, 303-338. https://bb..rgg/000077/11166666 (2010)
[14] Gholami, P. et al.: OCTID: optical coherence tomography image database. Comput. Electr. Eng. 81, 106532 (2020)
[15] He. K. et al.: MAE: Masked Autoencoders Are Scalable Vision Learners. CVPR. arXiv:211 1.06377 (2022)
[16] Huang, H., Zou, J., Meng, L., Yue, X., Zhao, Q., Li, J., Song, C., Jimenez, G., Li, S., Fu, G.: Comparative Analysis of ImageNet Pre- Trained Deep Learning Models and DINOv2 in Medical Imaging Classification. arviv. arXiv:2402.07559 (2024)
[17] Isensee, F., Jaeger, P.F. Kohl, S.A.A. et al.: nnU-Net: a self-configuring method for deep learning-basee biomedical image segmentation. Nat Methods 18, 211-211. https://doi.org// 1038//4 1ttp://////1111//1111111111 1500444419200000000000 (2021)
[18] Jiang, N., et al.: Vision transformers don't need trained registers. NeurIPS. arXiv:2506.08010 (2025)
[19] Khattak, M., Kunhimon,S., Naseer, M., Khan, S., Khan, F: UniMed-CLIP: Towards
Unified Image-Tes Text Pretraining Paradigm for Diverse Medical Imaging Modalities. arXiv. 10X1v:2412.10372 (2024)
10

---

[20] Kirillov, A. et al.: SAM: Segment Anything, arXiv. arXiv:2304.02643 (2023)
[21] Koch, V., Wagner, S., Kazeminia, S., Sancar, E., Hehr, M., Schnabel, J., Peng, T., Marr, C.: DinoBloom: A Foundation Model for Generalizable Cell Embeddings in Hematology. arXiv. arXiv:2404.05022 (2024)
[22] Kovalyk, O. et al.: PAPILA: dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment. Sci. Data 9, 291 (2022).
[23] Lin, T., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. ICCV (2017)
[24]
Loshchilov, Hutter, F:: Decoupled weight decay regularization. ICLR (2019)
[25] Ma, C., Tan, W., He, R. et al.: Pretraining a foundation model for generaliz- able fluorescence microscopy bassed image restoration. Nat Methods 21, 1568-1567. 10944/055000//41552022422
(2024)
[26] Mann, H. B., Whitney, D. R.: On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1), 50-60. (1947)
http://www.ssorro/ttteeee222
[27] Martin, D., Fowlkes, C., Tal, D., Malik, J.: A Database of Human Segmented Natural Images and its Application Evaluating Segmentation Algorithms and Measuring Ecological Statistics. ICCV (2001).
Martinsson, P.-G. and Tropp. J.A.: 'Randomised embeddings', in Randomized Numerical [28]
Linear Algebra: Foundations & Algorithms. arXiv preprint arXiv:2002.01387 (2020)
[29] Milletari, F. Navab, N., Ahmadi, S.: V-Net: Fully convolutional neural networks for volumetric medical image segmentation. 3DV, (2016)
[30] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
[31] Porwal, P. et al.: IDRiD: diabetic retinopathy segmentation and grading challenge. Med. Image Anal. 59, 101561 (2020)
[32] Radford et al.: CLIP: Learning Transferable Visual Models From Natural Language Supervision. ICML. a1Xii:210000000000000 (2021)
[33] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv. w:x.iv11505.45555 (2015)
[34] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla A., Bernstein, M., Berg, C., and Fei-Fei, L.: Imagenet large scale visual recognition challenge. IJCV, arXiv:1409.0577 (2015)
[35] Sablayrolles A., Douze M., Schmid C., and Jégou H.: Spreading vectors for similarity search. ICLR (2019)
[36] Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. ECCV (2012)
[37] Siméoni, O., al.: Localizing objects with self-supervised transformers and no labels. BMVC. arXiv:2109. 14279. (2021)
[38] Sun, M., et al.: Massive activations in large language models. COLM. arXiv:2402. (2024)
[39]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L.,, Polosukhin, I. Attention is all you need. arXiv preprint
(2017)
[40] Wang, H., Zhang, T., Salzmann, M.: SINDER: Repairing the Singular Defects of DINOv2. ECCV. 16822 (2024)

---

[41] Xiao, G., et al.: Efficient streaming language models with attention sinks. ICLR. EXXi::330917453: (2024)
[42] Yang, J., et al.: Denoising Vision Transformers. ECCV, arXiv:2401.02955 (2024)
[43] Zhou, Y., Chia, M.A., Wagner, S.K. et al.: A foundation model for generalizable disease http:///....///1118188000000
detection from retinal images. Nature 622, 156-163
x (2023)
[44] Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T:: ibot: Image bert pre-training with online tokenizer. ICLR (2022)
[45] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A.: Scene parsing through ade20k dataset. CVPR (2017)
[46] Zu. W., Xie, S., Chen, H., Ma, L.: Pre-trained Models Succeed in Medical Imaging with Representation Similarity Degradation. arXiv, arXiv:5503.07958 (2025)
12

---

A Technical Appendices and Supplementary Material
A.1
Supplementary Figures
Figure 5
shows fine-tuning with RMLPs create tokens with smaller norm and that it assigns bigger norms patch tokens coming from high-information regions unlike when fine-tuning with MIPP.
Glaucoma Fundus
BSDS300
OCTID
proportion
proportion
10-2
10-2
10-3
30-
1000 10
10
10
20
40
nom
norm
norm DINOV2-5
RI-VIT
R:o-VIT
Ric-VTT
C-VIT
(a) Proportion curves of attention maps' norms.
Glaucoma Fundus
BSDS300
OCTID
DINOv2-8
MLP
R5-MLP
0.0 25 0.5 0.75 1.0
0.0 0.5 0.5 0.75 1.0
0.0 0.25 05 0.75 10
daraset-nomallzze nom
dataset-normalzzes norm dataset-normaltzed norm
low informatioaionn patch
high-information patch
(b) Normalized probability density functions of attention maps' norms for low- and high information patches across models (rows) and datasets (columns). Low- and high-content patches are classified by two-component Gaussian Mixture model fitted on smoothed pixel gradients averaged per patch. Dashed lines stand for expected value.
Figure 5: Visualization second- (BSDS300 27 and first-order (OCTID [44/Glacooma Fundus ( attention maps' norm statistics. Subfigure (a): Proportion eurves. Subfigure (b): Normalized probability density functions for low- and high-information patches.
The following visualizations for the first- and second-order attention maps as well as the PCA visualizations were computed following sliding window approach.
Here we present some extra examples on the performance of DINOv2-S, DINOv2-S+MLP and DINOv2-S+Ry-MLA in natural, OCT and CFP modalities, in addition of RETFound [ for the ophthalmology modalities mentioned before.
13

---

DINOv2-S + MLP
DINOv2-8 + R&-MLP
OCT
Target
RETFound
DINOv2-S
Figure 6: OCT segmentation on the dataset from Eckardt, et al. [12 using different backbones (columns) a ViT-UNet hybrid
14

---

Ground Truth
DINOv2-S DINOv2-S DINOv2-S DINOv2-S
Input
DINOv2-S
MLP
+ R5-MLP + R1o-MLP R2
ViT-UNet Linear Head
VIT-UNet Linear Head
ViT-UNet Linear Head
ViT-UNet Linear Head
ViT-UNeet Linear Head
Figure Depth estimation of NYU-Depth V2 [36 dataset using various backbones (last five columns) with linear head or ViT-UNet hybrid.
15

---

Ground Truth
DINOv2-S DINOv2-S DINOv2-S DINOv2-S MLP
Input
DINOv2-S
+ Rs-MLP + Rio-MMP R20
ViT-UNet Linear Head
VIT-UNet Linear Head
ViT-UNet Linear Head
ViT-UNet Linear Head
ViT-UNee Linear Head
Figure Semantic segmentation of ADE20k 45 dataset using various backbones (last five columns) with linear head or ViT-UNet hybrid.
16

---

Input
DINOv2-S
DINOv2-S MML
DINOv2-S
Registers
+ +R-MMLP
Figure 9: Second-order attention maps on the BSDS300 27 dataset. Patch tokens were extracted using different backbones (columns).
17

---

Input
DINOv2-S
DINOv2-S .ML
DINOv2-S
Registers
+ R5-MLP
Figure 10: PCA visualization of embedding of BSDS330 (columns).
27
dataset using different backbones
18

---

Input
DINOv2-S
DINOv2-S MML
DINOv2-S R5-MLP
Registers
Figure 11: First-order attention maps on Glaucoma Fundus dataset using different backbones (columns).
19

---

Input
DINOv2-S
DINOv2-S MLP
DINOv2-S
Registers
+ +
Figure 12: PCA visualization of embeddings of Glaucoma Fundus 2 bones (columns).
dataset using different back-
20

---

Input
DINOv2-S
DINOv2-S MLL
DINOv2-S
Registers
R5-MLP
Figure 13: First-order attention maps on OCTID 14
dataset using different backbones (columns).
21

---

Input
DINOv2-S
DINOv2-S MML
DINOv2-S
Registers
+ R5-MLL
Figure 14: PCA visualization of embeddings of OCTID (columns).
14
dataset using different backbones
22

---

A.2
Mathematical Definitions and Proofs
Given the interdisciplinary of the theoretical analysis in this paper, not only across scientific fields, but also across mathematical domains, first provide list of useful mathematical definitions results to analytically understand the results in this paper.
A.2.1 General Mathematical Background
We present now collection of topological definitions and result. Definition 6. Given a function f XX - Y, we will say that f is locally injective on a subset A = X if. Va E A, 3Ug S X open such that flu, is injective.
Definition 7. Given metric space (X, 8). a subset A = X and s > 0, we call s-cloud to the set {a X 8(z, A) < E}.
Definition 8. Given topological space X, we will say X is disconnected if there are A, S X such that they are open and non-empty, A e Ø and {A, B3 disconnection.
X. In addition, we will call
Definition Given a topological space X and D D X, we will say is dense X is for whatever D Ø
open subset U C. X, U
Remark 10. f:
is contimuous and is disconnected, is disconnected.
Proof. Let {A, B} be disconnection for Y. Then [[A] and [ are open and non-empty due being continuous and a function resp. Furthermore, f [ u [ X since f is function and {A, B} a disconnection. Lastly, using that {A,, B} is a disconnection and knowing that the pre-image opens finite intersections, f [[] n [ f B f f Ø.
A.2.2 Topological Analysis of Vision Transformers
In this subsection, we will present a mathematical model describing ViTs and proving some results on their topological properties. Notably, the only assumption are making in this subsection is that the corresponding ViT was trained using the KoLeo 335 regularizer.
Theorem 11. VET's be decomposed translations. Said translations are defined by local orthonormal bases generated by the modulation of the data via the queries, keys and values layers.
tokenization function followed by composition of
Proof. Let Ω be image domain and V = U Raxd be the VIT's latent space and d its token nEN
dimension. Further, let V : Ω - V denote the ViT as learnable function and T' : W - V the transformer encoder, following the structure from Dosovitskiy al. [1I. We decompose these maps as V T o C and T = A T o o To, respectively where is a continuous map from Ω to V M a normalization layer and for each E {0. ....... L 1}, we define
T + W
(1
with 9t described below.
Leaving the normalization N aside, residual steps sy. Following Vaswani et al. [3), 8t can be further decomposed as
see T' as a translation defined as composition of the
81 : x - ft (2 + (x(Ni,(x))))
where h; represents the multi-head self-attention mechanism, fi is the MLP block and Mi, and N2.1 are layer normalizations.
Assuming linear layers have bias for simplicity, the attention hy can be expressed
(e )Ck. Lv.x
hy x
23

---

where a is the softmax function, r is scaling factor and Lv., CK,,, Lq. are learnable linear maps for queries, keys, and values.
Since zrr" is symmetric, it can be diagonalized, allowing hy to be rewritten
: (+ Lv.ex
where D, is diagonal and L and Lxx incorporate the change of basis.
Thus, understand 7 map creating field for the embeddings follow, where the self- attention layers dynamically generate local orthonormal bases over Yi that the queries, keys and value matrices then modulate while the MLPs fi refine the resulting directions.
D
Lemma 12. If a continuous function f: Y locally injective on compuct set K X and the topology on X is induced by metric 8, then 3e 0 such that f is injective on
f x: 8(x, K) <...
Proof. Let U. := (a E X 8(x, K) < E) and let us work by contradiction, i.e. we will assume de > 0, 3/2 Ue) = Ug such that T. Us and f(x)) = [Ve)) Given this, we can construct two sequences such that for n e Nt, (In yn) S Unjns In * yn and f(x.) = f(yn).
Since K is compact, it is easy to that for every positive natural number U1jn is compact too. Thus, given is metric and (mn)nsa, } U1, we can extract convergent subsequences {In, Jiso, (un hiso to x and respectively.
Let us notice that K = n U1jn- Therefore { n>0
f(y) since f is continuous and
injective in K, having x y as a consequence. Also, since is locally injective. 3V, open such that r E Vj and f is injective on it and since {In, Ni [yn, both converge to x, 3.V N such that
Vi N, (Inn yo. = Vz f(Xn.) f'(nn. Vi N, reaching this way the contradiction we were looking for.
Lemma 13. function f X - Y is injective and continuous with compact domain and Hausdorff codomain, f an homeomorphism on [[X].
Proof. Assuming the hypothesis from the lemma, the only thing remaining to show is that f is open.
Thing being this way, let us take an open set U = X. Then XUU is closed which turns it into a compact set since X is compact. Thus, by continuity, [[X compact, which makes it closed since Y is Hausdorff, forcing [U] to be open.
A.2.3 RMLPs as Stochastic Regularizer for VITs
We now introduce some usefuu definitions and rigorous proofs on the way RMLPs regularize the representation's 's topology by making the ViT perceive points in the representation space as ball of certain radius with high probability.
Definition 14. (28) Let E 8" a set and e (0. 1) be a distortion parameter. We say that a linear map S F - Fm produces an e-distortion of E if, Vx E E.
(1 (1- [===]|=12 < Sxl|2 & (1+t)|æ42.
Remark 15. If Rd and I is d matrix with iid entries and such that F N(0,03) then T [3] do |x]]
24

---

Proof. Computing
T T E| (EyF |1 E 1yy))
Since the entries of T are independent and [i MV,0 -²) then have
= =
do
Definition 16. Let S & Fm be linear map and E s (F) subset of the unit sphere in
F". Then, Martinsson and Tropp [28] define the minimun and maximum restricted singular values respectively as
Omin (S, E) == min
Sxll
and
m (S, E) = max |Sx|l-
ze
Lemma 17.
28 Let us consider {a....., aN) R2, TE paxd and build
E
|a ااlه :l S -IEi<jENN N s
a jj
Then have the following probability bounds:
P min ET, <
1 2v log( N/2)
- t & -d
Vn
and
2 log(N
(
P Tmax 1+
+ 1
ddt//2
vd
Thus, it is sufficient that n Be 8 log N for being able to guarantee I' has a distortion of 14 with high probability.
Corollary 18. A Q is the training data, and T is locally injective on A, the following holds:
There exists
0 such that T is an homcomorphism
s-cloud containing A (see Def.
Let P.Q C[P is disconnected in V if and only [V is disconnected which can contribute the batch effect.
(see Def. K
3) If D = V is dense in V (see Def. 9p. then V[D] is dense in V(V).
Proof. Assuming the corollary hypothesis, let us prove the statements.
Ub The existence of the s-cloud is direct result of theorem
:) The result follows directly from Theorem 2 since homcomorphisms preserve connections and disconnections.
Let D be dense in We note is continuous for being defined as multiplication of matrices and addition of vectors. Thus C[D] is dense in C[R]. Therefore, V[D] is dense in V[ since T' is an homeomorphism because theorem
Theorem 19. Being (p1. .P.....PN' PN) C RT
>> 0, À > 0 and I' a matrix of size whose entries are iid and following a normal distribution N An -1), I' will have an e distortion on the set
E
pi Pj
iEi<jEN;
[p: - Pj
25

---

with high probability if
8²
An <
8mnN
In addition,
E| T 2 = mAn 1x1
Proof. To see |[[[<=|]] mnAn-1Ia it is enough use remark
On the other hand, to compute the distortion of T, let assume the hypothesis and define the restricted singular values following Martinsson and Tropp 228 as
mmin (T, E) == min T
max E) == max [x],
wEE
zEE
let us realize the statement is equivalent to having
- E < mmin E) & [Tx] & Omaz (T, (raa((TB
1
Thus, having V A e/v8ln it follows A/n (1 2 2/ln(/22) and then
V/nn (1 2 /n( This way, we can take and write
IP ma (T, E) & - t) < IP min (T, < - v\/n (1 + (1+2/i(N/22) -t)
Similarly, V/n < <E/V N implies 1 1 2V An
Thus, being > 0,
P maxx (T, E) 8 & (mmax (T, E) 7 1 -n 1m +1).
Therefore, using that the Gaussian width of E, (F), satisfies w(E) < 2 (n(N22 and the theorem for restricted singular values and Gaussian matrix from Martinsson and Tropp (28 for every / we have
PP (amin (T, & - - 1) -
and
mo E) >1+++) = 1n19/2
:. has an e-distortion with high probability.
A.3 Technical Details
Backbone Training. All training performed single GPU (Quadro RTX153 8000 NVIDIA A100-SXM4-40GB) and required approximately 15 hours per trained backbone, reflecting the low computational cost of our approach and its reduced environmental footprint. Models were trained on randomly sampled mini-batches until validation performance plateaued. For datasets without predefined splits, we manually partitioned them into 70% training, 15% validation, and 15% testing.
All models were trained using the AdamW optimizer fine-tuning stages as well as during training of downstream linear heads and UNet decoders.
24 which we employed consistently across
Table Table shows the main hyperparameters used when fine-tuning DINOv2-S natural, OCT and CFP modalities. Further implementation details can be found in our code.
Downstream Tasks Training. For classification tasks using DINOv2, we constructed the input repre- sentations for 1 I-Nearest Neighbor, Random Forest, and linear probing classifiers by concatenating the class token with the mean of the patch tokens. In contrast, for SwAV, which does not produce patch tokens, used its output directly. The lincar classifier was trained using the cross-entropy loss.
26

---

Table 5: Hyperparameters used for fine-tuning DINOv2-S obtain C- ViT and R ViT.
Hyperparameter
Value
Oplimizer
W 24
Platean size for early stop
10 spochs
Token's 'een's dimension
384
DINO coefficient
iBOT cocfficient
KoLeo coeflicient
Initial learning
le:7
Patiemco/factor for learning rate scheduler
3/.4
learning
Brtkien/Bottlenecc/ee dimensions for MI.Ps Ps and RMI.Ps
1536/251/5536
Number of transforner blocks
Patch size
Crop size
224
Steps per epoch
Warm up epochs
To integrate the VIT and UNet architectures for dense prediction tasks, we first projected the output of the ViT through a linear layer and then concatenated it with the features from the encoder branch of the UNet. This combined representation was subsequently fed into the UNet's decoder branch. The ViT outputs were handled differently depending on the task: for segmentation, we concatenated the class token with the patch tokens, while for depth estimation, only the patch tokens were used. Segmentation heads were trained using a weighted combination of focal loss and Dice loss, whereas depth estimation heads were trained using focal loss.
Evaluation of Patch Token Quality. First-order attention maps were computed by evaluating the norm of the patch token embeddings. For the second-order attention maps, independently performed principal component analysis on the patch tokens of each image and calculated the norm of the top three principal components. To distinguish between low- and high-information patches within an image, we first computed the image gradient, applied a smoothing operation, and then averaged the resulting gradient values within each patch. Gaussian Mixture Model with two components was subsequently used to classify the patches based on their average gradient magnitudes.
A.4 Information External Sources
We used two pre- pre-trained models in this work. Their licenses and repositories be found in Table 6 A recollection all the datasets used in this work can be found in Table [7a] both for natural and medical domains. To show the geographic diversity from our geographical dataset, show the country of origin of the medical datasets used in this paper in Table 7b
Table 6: External code and weights used as baselines.
Repository
DINO22 20
Apache-2.0 CC BY-NC-40
https: h/gg://///ttbbb/// con/ http:///ccooooorrrrrrr///aaaaaaa
RETFound
httpe: //gtthb.. ruaphhh
DVT
X2
MIT License
bttps: http//ttttttbtteeee con/ Jiaves Yang/ Www./singoonnnooeeee ViT
Registers
Apache
kttps: Pttps://gitaul com/ htt.:ttt........oooooooarorrron.
SwAV A
License:
CC BY-NC
Sinder
40
attps://githus con/ hwwphhaagggggaeee..c.. sinder
27

---

Table Used datasets, licenses and country of origin.
Used datasets and their licenses.
(h) Country of creation for medical
dataset.
Name
License
Repository
ImageNet-1k 1k
CCU: Public Domain
ImageNet-
Dataset
Country
NYU-Depih
36
MIT
NYL-Dephh
OCTID 14
India
ADE20 45
BSDD-3-Clause Clause
South
BSDS300 27
Non-commercial research
TDRID 31
India
VOC07 18
Custom
PASCALL VOC 2007
JSIEC 2
China
TCTTD
MESSIDOR-2
France
Glaucoma Fundos
CCO 1.0
Glancoena Fundus
PAPILA
Spuin
IDRID 31
Open Access
LDKID
Retina
Unspecified
JSTEC 7
Open Access
JSTEC
Eckardi, et
12
Germany
MESSIDOR-2
Non-commercial research
MESSIDOR-2
PAPILA
GPL 3.0+
Retina
8
Open Access
Kelaa
Aptos
Custom
Aptos
Eckardt, et
12
Property LMU University Hospital
A.5
Supplementary Results
Table DICE scores for segmentation on the Eckardt, et al. Nuclear Layer (ONL).
12 dataset with emphasis on the Outer
ViT-UNer bybrid
Lincar Head
Model
Averaged DICE
DICF on ONI.
Averaged DICE
DICF on ONI.
RETFound
0..05
0.59.10.10
0.10.10.22
0.01 0.00_10.02
DINOv2-8 30
0.79:t.20
0,54:0.20
'0.16:0.08
0.12++0.05"
DINO
0.86+0.20
0.55+0.20
0.20-0.05
0.03++0.03
DINOx2-S+Ro,1
10.94+0.12
10.68+0.21
0.20=0.03
0.06++0.04
DINO+2-S+R;.N MI.P
0.97
0.72-10.09
0.13:10.01
0.00
DINOv2-S+Rio
0.8720.20
0.611:020
0.20
0.07
DINOv2-S+Rap MLP
0,7010.30
0.47:10.0
0.13
0.005
Eckardt, et [
0.92+0.03
0.44+0.03
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect
paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we claim we developed a theoretical grounded regularizer that improves DINOv2-small's interpretability, mantains performance when fine-tuning on natural images and improves performance when trained in medical data. Data backing up this claims can be found in Figure Sauurerandd and Tables T respectively. In addition, discussed in Section g
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss limitations of the paper in Section on a specific subsection.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
29

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All our theorems are properly proved in the main paper (Section 4 or in Appendix (..)). Same for needed mathematical definitions and related results.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe in detail the regularization method in Section 3 specifying that the training algorithm is the same the from DINOv2 (3]
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The models presented in this paper were trained public datasets, with the exception of the dataset presented by Eckardt, et al. [12]. This dataset was used to trained the OCT segmentation task, but cannot be released due to data privacy laws. Regarding code, well-documented code for reproducing our experiments will be released after acceptance to preserve anonymity. Same for the trained weights of the models presented here.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
public/guidess/Codeeiiee Soaedd/es//ooeSSuubbssssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips.com cttpsscccppbbb////eeeeeee wttp://wweee/eeeeeeee /
www.siiiccciiiiiccooooooooiiiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings are described in Section Further, code will be released after acceptance.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Description of significance test and its assumptions are described on Section 6 Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
is described Section A.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttps://neuriie htt.://ppppp....pp..../tccccccaa
Answer: [Yes]
Justification: Authors read NeurIPS Code of Ethics and made adhere to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
32

---

Justification: The work presented in this paper is general algorithm for optimizing models trained in a contrastive learning fashion, so there it direct potential societal impacts. On top of this, we mention how our method requires relatively small training times, reducing the environmental impact. We also mention the dataset medical dataset used in this paper is geographically diverse, but we did not elaborate on that since the model itself is not the main point of the paper. Nevertheless, information abut the countries of origin of the datasets is shown in Appendix A.4
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We are not releasing any dataset and the downstream tasks trained in this paper does not pose direct risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Citations clear and correct and licenses be found in Appendix A.4
33

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: mentation.
Code and trained models will be released after acceptance with proper doeu-
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowd-sourcing research with humans since all used datasets already existed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
34

---

Answer:
[NA]
Justification:
We had no study participants since this was an entirely computational work.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLM was used beyond rephrasing in the making of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy [ctppe///hacripscccc//oooeeeeees for what should or should not be described.
35


Reviews:
[{"review_id": "xv81GLjhVn", "reviewer": "Reviewer_wDm7", "summary": "This paper proposes Randomized-MLP(RMLP) to improve domain adaptation interpretability and reduce attention map artifacts in DINOv2 by replacing learnable MLP heads with randomized, non-trainable operators. The method aims to address artifacts where DINOv2-S repurposes low-informative patches for global information storage, particularly problematic in medical imaging. The work combines theoretical analysis with empirical validation across  both natural and medical image domains.", "questions": "(1)Do you have a principled method for selecting λ for new domains, or is manual tuning required?\n \n(2) Could RMLP be implemented on other self-supervised ViTs or the method is specific to DINOv2's architecture?", "limitations": "The authors have properly addressed its limitations.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n \n(1) Lightweight component: RMLP requires no architecture modifications and trainable parameters that can be applied during fine-tuning with minimal computational overhead. The method is theoretically motivated and simple to implement.\n(2) Solid theoretical framework: Theorems 1-4 provide comprehensive mathematical analysis of how RMLP regularization influences contrastive learning geometry, offering valuable insights into the mechanism underlying the observed improvements.\n(3) Comprehensive experiments: the experiments show multiple domains and tasks such as classification, segmentation and depth estimation with proper statistical significance testing using Mann-Whitney U tests.\n \nWeakness:\n \n(1) The paper shows experiment results only on DINOv2-S based models, experiments on other relevant models such as DINOv2-base and large are untested.\n(2) Apart from Figure 6,this paper relies heavily on visual results to show the enhanced interpretability of attention maps. Providing more quantitative evaluation results would strengthen the claims.\n(3) RMLP shows performance degradation on some downstream tasks, indicating the method seems to sacrifice linear separability for neighborhood structure.", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nWe are very glad to read that you find our theoretical framework solid and our method properly evaluated across modalities and tasks. We also value very much that you find our work highly original and appreciate the constructive feedback. In the following sections, we would like to address your remaining questions.\n\n**Principled method for selecting** $\\lambda$. While we do not yet provide an automated procedure for determining $\\lambda$, we offer both an intuitive visualization and a theoretical foundation to guide its choice. As shown in Figure 2, if $\\lambda$ is too small, the resulting permutations are insufficient to act as an effective regularizer; if too large, $\\lambda$ can distort the embedding topology, negatively impacting performance. To support this principle, Theorem 4 establishes a mathematical connection between $\\lambda$ and the dimensionality of the latent space, offering a theoretical basis that may inform more systematic approaches in the future. In our experiments, we selected $\\lambda$ via manual grid search. We evaluated performance across various downstream tasks and image modalities, consistently finding that $\\lambda =5$ yielded the best results. While this suggests a possible heuristic, we acknowledge that further empirical validation across a wider range of datasets and architectures is needed to establish a generalizable selection method.\n\n**RMLP implementation on other architectures**. Our theoretical analysis suggests that RMLPs can be applied broadly to any contrastive learning setup, independent of the underlying ViT architecture. To support this, we conducted some preliminary tests using the DINOv2 base model and observed a similar performance improvement by RMLPs on downstream tasks. Nevertheless, we decided not to include these results in this manuscript’s final version because we wanted to focus on the low-resource setup, to improve computational efficiency in the medical imaging field. Exciting follow-up projects could include investigating the effects of RMLPs not only on architectures like MoCo and Swin Transformers used in vision tasks, but also on those used for natural language processing, such as SimSCE or BERT.\n\n**Qualitative assessment in ophthalmological models**. We fully agree that interpretability is a critical aspect in understanding model behavior. At the same time, we would like to emphasize that in ophthalmology, quantifying interpretability remains challenging. As highlighted in RetFound (Zhou et al. Nature (2023)) and other prior work (e.g., Eckardt et al. (2024)), interpretability in this domain is often best evaluated through spatial correlation with anatomical structures and clinical relevance as judged by domain experts, rather than through scalar metrics alone. Nevertheless, following the reviewer's suggestion, we also conducted additional quantitative analysis on a dataset where ground-truth annotations of anatomical structures are available (see below), to complement our qualitative analysis.\n\n**New quantitative results (Correlating high-norm patches and anatomical structures)**. We recognize the importance of complementing qualitative insights with quantitative analysis and have therefore conducted additional experiments to better support our interpretability claims. Precisely, we evaluated the alignment between patch token norms and retinal layer annotations using the dataset introduced by Eckardt et al. (2024), which includes OCT scans with expert-labeled retinal layers from patients with and without Inherited Retinal Disorders (IRDs). To do this, we measured the correlation between the spatial positions of the top 25% highest-norm patch tokens from the models’ embeddings and the annotated retinal layer regions. Results below indicate no alignment with anatomically relevant structures for DINOv2-S, whereas R$_5$-ViT ($\\lambda=5$) suggests improved localization of clinically relevant structures by our RMLP attention map. These findings provide quantitative support for our interpretability claims, particularly in the ophthalmological domain.\n\n|    Model    | Patients with IRD | Patients without IRD |\n|:-----------:|:-----------------:|:--------------------:|\n|  DINOv2-S   |       -0.06       |         0.09         |\n| R$_5$-ViT   |        0.36       |         0.33         |\n\n**New quantitative results (Information in patch tokens)**. We further investigated model behavior by analyzing the diagnostic information contained in individual patch tokens. Following a protocol similar to Darcet et al. (2024), we performed a 1-nearest neighbor classification task to distinguish patients with and without IRDs from the Eckardt et al. (2024) dataset. Specifically, we compared classification accuracy when using: (1) the class token, (2) the highest-norm patch token, and (3) the lowest-norm patch token.\n\nThe results (see table below) reveal that while DINOv2-S achieves reasonable accuracy using the class token (0.76), its highest-norm patch token performs poorly (0.48)—even worse than the lowest-norm token (0.76). This suggests that DINOv2-S attends to non-specific or potentially irrelevant regions. In contrast, R$_5$-ViT maintains high class token accuracy (0.88) and achieves strong performance (0.76) using only the highest-norm patch token, while the lowest-norm token performs poorly (0.40), as expected. These findings support our claim that RMLP encourages attention to diagnostically meaningful areas, rather than repurposing void regions.\n\n\n|   Model   | Class Token | Highest-Norm Patch Token | Lowest-Norm Patch Token |\n|:-------------:|:---------------:|:-----------------------------:|:----------------------------:|\n|   DINOv2-S    |      0.76       |             0.48             |            0.76             |\n|  R$_5$-ViT    |      0.88       |             0.76             |            0.40             |\n\n**Additional comments on expert evaluation**. We would like to emphasize that our approach to interpretability extends beyond raw attention visualizations in two important ways. First, in Figure 5 and in Appendix Figures 15, 17, and 19, we present qualitative examples where R-ViT models attend to disease-relevant anatomical structures, whereas DINOv2-S does not. These examples were carefully reviewed by ophthalmologists and serve as task-relevant, clinically informed evidence of improved interpretability. Second, we provide a statistical analysis of attention map distributions in Figure 6. For example, Subfigure 6b illustrates how the norm distributions shift when comparing low- and high-information patches—capturing structure-aware changes that reflect the semantic properties of the image modality. These analyses go beyond visual intuition, offering quantitative and anatomically grounded insights into how model attention behavior changes with RMLP.\n\n\n**Linear separability vs. neighborhood-based classification**. We would like to clarify that, on average (as reported in Table 2a. Look at table below), DINOv2-S and R₅-ViT perform equivalently on linear classification tasks, while R$_5$-ViT consistently outperforms DINOv2-S in 1-Nearest Neighbor classification. This distinction aligns with our design: as discussed in lines 193–203, MLP heads in DINOv2-S tend to promote global linear separability, whereas our randomized MLPs (RMLPs) encourage local neighborhood preservation in the latent space—a hallmark of contrastive learning. Therefore, the performance differences observed in specific tasks reflect the intended trade-off between linear discriminability and semantic locality, with RMLPs offering advantages in tasks that benefit from structured, neighborhood-aware representations, like 1-NN classification.\n\nNotably, R₅-ViT outperforms RetFound (Zhou et al. Nature (2023), a specialized retinal foundation model trained on hundreds of thousands of OCT scans, in both 1-NN classification (can be used for image retrieval) and segmentation tasks (See Table 2b). This demonstrates the strong transfer learning potential of our method from natural images to medical images, even when fine-tuned on a relatively small dataset of hundreds of OCT images.\n\n|   Model   | 1-Nearest Neighbor | Random Forest | Linear classification |\n|:-------------:|:---------------:|:-----------------------------:|:----------------------------:|\n|   DINOv2-S    |      0.60       |            0.58             |            0.64 $\\pm$ 0.06             |\n|  R$_5$-ViT    |      0.64 $\\pm$ 0.02      |             0.58 $\\pm$ 0.03             |            0.64 $\\pm$ 0.02             |\n\n**References**\n\n​​Eckardt, F., Mittas, R., et al.: Deep Learning-Based Retinal Layer Segmentation in Optical Coherence Tomography Scans of Patients with Inherited Retinal Diseases. Klin Monbl Augenheilkd. doi: 10.1055/a-2227-3742 (2024)\n\nZhou, Y., Chia, M.A., Wagner, S.K. et al. A foundation model for generalizable disease detection from retinal images. Nature 622, 156–163 https://doi.org/10.1038/s41586-023-06555-x (2023)\n\nDarcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv. arXiv:2309.16588 (2024)"}, {"review_id": "ZQ44P2UGzQ", "reviewer": "Reviewer_FxJh", "summary": "The  basis of the manuscript is that the the authors state with quite a bit of evidence that ViT models trained with DINOv2 repurpose low-information patch tokens with global information.  This is particularly problematic for tasks that are segmenting in nature.  The specific domain of concern is  ophthalmologic images where the chief task is segmenting OCT and eye fundus images. The results also show improved performance on natural image datasets.\n\nThe solution proposed is to replace the MLP layer with a novel Randomized-Multi-Layer Perceptron (RMLP). RMLPs replace the standard learnable MLP heads in DINO and iBOT with fixed, randomized linear operators that preserve structure.  The authors believe this guards against shortcut learning by preventing token-level overfitting.  The goal is also to avoid major archetectural changes. \n\nThe results show qualitative improvements in depth estimation and segemenation on natural images.  Heatmaps on ophthalmologic images suggest that there is improved attention to the objects of interest and less attention attributed to the non-informative parts. This is most compelling in the OCT images. Quantitative results for segmentation and classification tasks are also provided in conclusions with some mixed results.  Most dramatic performance improvement is in \"pathology classification\" using Nearest Neighbor clustering.", "questions": "Please better define \"probability balls\". It is a confusing terminology and while I think I conceptually understand, it is hard to be certain. \n\nI understand that you initialized with DINOv2-S trained on natural images and further trained on the different dataset after replacing the MLP heads.  Is there any advantage to this over training from scratch with RMLP only?\n\nIf the fundamental break through of the proposed algorithm is in improved segmentation, how does one understand the improved performance in classification task in Table 2a? \n\nDINOv2 is the basis of many SOTA models in histopathologic image classification. Do you think that incorporating RMLP would improve performance in this domain?", "limitations": "Limitations section is perfunctory at best. This needs to be enhanced to be acceptable.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nReplacing MLP with RMLP might correct for hypothesis (with fair amount of evidence) that ViT models trained with DINOv2  store global information in low information patch tokens. \nThe theoretical proofs are very thorough and only have minor flaws.  I have some clarifying questions below \n\nWeaknesses:\nThe heatmaps should have a legend to be considered interpretable. Simply stating \"Brighter maps indicate more attention.\" is not adequate for full interpretation of the figures. \nAs a person not familiar with the domain of OCT and fundus image, I am not fully able to understand the goal of analyzing these image. Broadly defining these as \"medical images\" is also in appropriate because the goals for many other medical imaging tasks seem to be quite different from these ophthalmologic tasks. \nResult section are largely qualitative which limits evaluation. Quantitative results are in the Conclusion section.", "was_revised": true, "final_justification": "Showing the change of the model induces attention to the biologically meaningful structures provides strong evidence that their simple model modification has solved what appears to be a valid problem with DINOv2 training.", "rebuttal": "Dear reviewer,\n\nWe are glad that you found our theoretical thorough and appreciate that you recognized the potential of our method to tackle the norm-related artifacts from DINOv2. Following your constructive feedback, we would like to address your questions on the following sections.\n\n**Proper definition of “probability balls”**. A mathematical definition can be proposed as follows: Given a point $x \\in \\Omega$, $\\|\\cdot\\|$ a norm  and a random matrix $\\varphi$, we would say that $B(x, r) :=$ {$ z \\in \\Omega : \\|z - x\\| < r$ } is a probability ball if $\\varphi x\\in B(x,r)$ with high probability. In other words, a probability ball characterizes  the region in the latent space containing perturbations of a point under random linear transformations.. It reflects the typical spatial distortion induced by $\\varphi$ and serves as a geometric intuition for analyzing local neighborhoods under randomized projections. We will be happy to include this definition in the final version of the paper to help formalize our terminology.\n\n**Fine tuning DINOv2 vs. training from scratch**. The numerical experiments we ran showed improved results on downstream tasks when fine-tuning DINOv2-S compared to training from scratch using RMLPs. This aligns with prior findings, such as Chettaoui et al. (2025), which similarly report that fine-tuning pretrained vision transformers tends to yield better performance than training from scratch. . Moreover, we decided to focus on low-resource setups (as commonly present in medical imaging), where both time and computational resources are limited. Initializing DINOv2 with weights pretrained on natural images has been shown to be beneficial for both of these factors while not falling short in performance metrics, like in Roth et al. (2024).\n\n**Improvement on classification tasks**. Although we observe strong performance improvements in segmentation across multiple modalities when using our proposed regularizer, we emphasize that these gains reflect broader enhancements in representation learning with R$_\\lambda$-ViT, rather than isolated effects. By incorporating RMLPs, we promote more reliable and semantically robust image representations, which lead to consistent improvements across downstream tasks—most notably segmentation and classification. As shown in Table 2, R$_5$-ViT achieves better performance in 1-Nearest Neighbor (1-NN) classification than DINOv2-S, and comparable performance in random forest and linear classification settings.\n\nNotably, R$_5$-ViT also outperforms RetFound (Zhou et al., Nature, 2023)—a specialized foundation model trained on nearly a million of OCT scans—in both 1-NN classification (a proxy for image retrieval) and segmentation tasks. This result underscores the strong transfer learning potential of our method from natural images to medical imaging, despite being fine-tuned on a relatively small dataset of hundreds of OCT images.\n\n**Quantitative interpretation of attention maps**. We would like to begin by pointing out that in AI applied to ophthalmology, visual inspection remains a valuable tool to assess a model’s performance due to the challenging nature of interpretability in the field. Thus, similar to what RetFound (Zhou et at. Nature (2023)) did, the goal of Figure 5 is to show the different regions from the OCT and Fundus images that the considered models attend to, noticing how DINOv2-S focuses on void regions while R$_5$-ViT locates anatomically relevant structures such as retinal layers for the OCT or blood vessels in Fundus scans. \n\nTo further support our interpretability claims, we conducted new experiments that quantitatively evaluate the alignment between model attention and anatomical structures, using the OCT dataset introduced by Eckardt et al. (2024), which includes patients with and without Inherited Retinal Disorders (IRDs). For each model, we identified the top 25% of patch tokens with the highest embedding norms and assessed how well their spatial positions aligned with the annotated retinal layer regions. As shown in the table below, DINOv2-S shows little to no alignment with anatomically meaningful structures, while R₅-ViT exhibits significantly improved localization, suggesting better semantic grounding in clinically relevant image regions.\n\n|    Model    | Patients with IRD | Patients without IRD |\n|:-----------:|:-----------------:|:--------------------:|\n|  DINOv2-S   |       -0.06       |         0.09         |\n| R$_5$-ViT   |        0.36       |         0.33         |\n\nWe also conducted an experiment inspired by Darcet et al. (2024) to examine the type of information encoded in individual patch tokens. Using the dataset from Eckardt et al. (2024), we performed a classification task to distinguish patients with and without IRDs based on three inputs: the class token, the highest-norm patch token, and the lowest-norm patch token. The results, shown below, provide further quantitative evidence that R$_5$-ViT focuses on semantically meaningful, diagnostically relevant regions, whereas DINOv2-S appears to store classification-related information in less informative or void areas.\n\n|   Model   | Class Token | Highest-Norm Patch Token | Lowest-Norm Patch Token |\n|:-------------:|:---------------:|:-----------------------------:|:----------------------------:|\n|   DINOv2-S    |      0.76       |             0.48             |            0.76             |\n|  R$_5$-ViT    |      0.88       |             0.76             |            0.40             |\n\nWe will be happy not only to make a more descriptive legend of Figure 5, but also to include these new experiments to make our manuscript more accessible to readers without familiarity with OCT or Fundus modalities.\n\n**Using RMLPs on histopathology**. Although we have not performed systematic experiments on histopathology images, the theoretical formulation of our method does not include any particularity from any modality. Given our experiments show improvements on classification tasks for microscopy images, it would be an exciting follow-up work to apply our regularizer to histopathology data, where improvements can be expected.\n\n**Used modalities**. We acknowledge that calling “medical images” to our ophthalmological data might be ambiguous. We will be happy to modify the manuscript to specify we restricted our experiments on retinal microscopy images into the ophthalmological field.\n\n**Position of quantitative results**. We will happily rearrange the paper to make sure that both quantitative and qualitative results are in the corresponding section.\n\n**Limitations**. We will be glad to extend our limitations’ section.\n\n**References**\n\nChettaoui, T., Damer, N., Boutros, F. FRoundation: Are foundation models ready for face recognition? arXiv:2410.23831 (2025)\n\nRoth, B., Koch, V., Wagner, S., et al. Low-resource finetuning of foundation models beats state-of-the-art in histopathology. doi: 10.1109/ISBI56570.2024.10635695. (2024)\n\nZhou, Y., Chia, M.A., Wagner, S.K. et al. A foundation model for generalizable disease detection from retinal images. Nature 622, 156–163 https://doi.org/10.1038/s41586-023-06555-x (2023)\nEckardt, F., Mittas, R., et al.: Deep Learning-Based Retinal Layer Segmentation in Optical Coherence Tomography Scans of Patients with Inherited Retinal Diseases. Klin Monbl Augenheilkd. doi: 10.1055/a-2227-3742 (2024)\n\nDarcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv. arXiv:2309.16588 (2024)"}, {"review_id": "9kaOII8NpH", "reviewer": "Reviewer_UqKd", "summary": "This paper aims to improve the interpretability of the attention maps and feature maps of DINO ViTs, especially for medical images. They propose to replace the MLP heads of DINO with randomized MLPs that are fixed and add Gaussian noise to the intermediate features. They prove that adding Gaussian noise parameterizes the features to probability balls, and KoLeo promotes feature uniformity. They show that the scale of the noise should not be too large (Fig. 2). In experiments, the method shows better attention maps and improved performance on some benchmarks.", "questions": "Can the authors provide more details on how DINO is re-implemented, and why weight normalization and feature normalization are omitted?", "limitations": "The limitations are discussed on page 9.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper considers improving the interpretability of pre-trained DINO features for medical usage, which is beneficial for social good.\n- The paper provides mathematical proof for some properties of DINO and the proposed RMLP.\n- The paper also provides visualizations to depict the properties of DINO and the proposed method (eg, Fig. 6). \n\n### Weaknesses\n- I noticed that in the provided code, the implementation of both DINOHead and the modified version omitted weight normalization (on the last layer) and also feature normalization. These are crucial designs of DINO to ensure that self-distillation is performed on normalized *prototypes*, and removing them has shown notable performance drop in the original paper. This could raise a critical concern about whether DINO is faithfully re-implemented and whether the comparison is fair. Why did the authors remove these operations?\n- The comparisons, especially visualizations of attention maps and PCA visualizations, are always provided in very low resolutions. In DINO, DINOv2, and Darcet et al., they provided higher-resolution visualizations that are much clearer for comparison between methods. It is strongly suggested that the authors also update these visualizations to higher-res ones, as currently it is hard to identify important parts, e.g., vessels in medical images.\n- Many conclusions in the proof (Sec. 4) are unsurprising and this part could be moved to later sections for clarity. For example, KoLeo regularizer is not an important part of DINOv2, and removing it does not affect model behaviour notably. The main regularizers are the centering and sharpening operations, and I wonder why KoLeo is picked for analysis. Also, the fact that adding Gaussian noise turns deterministic inputs into stochastic \"probability balls\", and the scale of the noise should be small, are trivial.\n- The structure and formatting could also be improved. Currently, the introduction section is overly simple, and the related work section is also missing. Besides, I would also suggest adding a formal introduction of DINO in the main text.", "was_revised": true, "final_justification": "The discussion has resolved most of my concerns and the remaining ones can be resolved in a revision.", "rebuttal": "Dear reviewer,\n\nWe thank you for your feedback and would like to use the chance to address your comments in the following sections.\n\n**Distinction between adding noise and RMLPs**. We would like to clarify a potential misunderstanding in the reviewer’s summary of our proposed method. Punctually, RMLPs are not implemented as fixed randomized MLPs that add Gaussian noise to intermediate features. Instead, as described in Section 3 and visualized in Figure 2, our approach involves multiplying the ViT’s output by a sequence of Gaussian matrices—i.e., matrices whose entries are independently drawn from a Gaussian distribution at each forward pass. This distinction is critical: the stochastic nature of the Gaussian matrices (with fresh samples at each call) prevents the model from overfitting to a static transformation, which would undermine the regularization effect. Moreover, unlike additive noise, the use of multiplicative random projections introduces specific structural properties to the representation space. For instance, the induced sparsity in token activations shown in Figure 6.a emerges as a direct consequence of this multiplicative mechanism. We believe this foundational difference —between additive perturbations and stochastic multiplicative transformations— plays a central role in f the representational and interpretability benefits observed in R$_\\lambda$-ViT.\n\n\n**Further details on DINO re-implementation**. We confirm that our re-implementation of ViT was carefully validated by ensuring that it reproduced the same outputs as DINOv2-S when using pretrained weights, prior to any fine-tuning. \n\n**Omission of normalizations**. We thank the reviewer for their comment regarding the normalization layers in the DINO head. In our experiments, we compared downstream performance of both C-ViT and R$_\\lambda$-ViT using DINO-style heads with and without the normalization layers, and observed no significant difference between the two configurations. Based on these findings, and in the interest of architectural simplicity, we chose to exclude them.\n\nWe believe one contributing factor is that our models are fine-tuned from pretrained DINOv2 checkpoints, whereas the original DINO paper trains the vision transformer from scratch—making normalization more critical in the original context. Additionally, for R$_\\lambda$​-ViT, Corollary 5 ensures that the token representations remain close to the unit sphere after passing through the randomized head. In the case of C-ViT, since the ViT backbone already produces well-structured token embeddings and the weights of the final linear layer are initialized within a bounded range, meaningful representations can still be learned in the absence of explicit normalization.\n\nThat said, we acknowledge that we did not conduct a dedicated ablation to isolate the role of these normalization layers, and we appreciate the reviewer for emphasizing their relevance in the original DINO framework. \n\n\n**Resolution of attention maps and quantitative evaluation**. Our qualitative figures aimed to illustrate general alignment between attention distributions and semantically meaningful regions, not to serve as the primary evidence for interpretability and thus, were kept at low-res. The main support for our claims comes from statistical analyses (e.g., Figure 6).\nWhile we intentionally avoided high-resolution delineation of small structures to prevent over-interpretation, we acknowledge the importance of quantitative interpretability evidence. To that end, we conducted an additional experiment inspired by Darcet et al. (2024), using 1-NN classification on the OCT dataset from Eckardt et al. (2024). We compared performance when classifying patients with and without IRDs using the class token and patch tokens with the highest and lowest norms.\nResults show that R$_5$​-ViT performs well when using the highest-norm token, suggesting it attends to clinically relevant structures. In contrast, DINOv2-S performs the best with the lowest-norm token, indicating it may focus on anatomically void regions. \n\n|   Model   | Class Token | Highest-Norm Patch Token | Lowest-Norm Patch Token |\n|:-------------:|:---------------:|:-----------------------------:|:----------------------------:|\n|   DINOv2-S    |      0.76       |             0.48             |            0.76             |\n|  R$_5$-ViT    |      0.88       |             0.76             |            0.40             |\n\nMoreover, we evaluated the alignment between patch token norms and retinal layer annotations using the dataset introduced by Eckardt et al. (2024). To do this, we measured the correlation between the spatial positions of the top 25% highest-norm patch tokens from the models’ embeddings and the annotated retinal layer regions. Results below indicate no alignment with anatomically relevant structures for DINOv2-S, whereas R$_5$-ViT ($\\lambda=5$) suggests improved localization of clinically relevant structures by our RMLP attention map. These findings provide quantitative support for our interpretability claims, particularly in the ophthalmological domain.\n\n|    Model    | Patients with IRD | Patients without IRD |\n|:-----------:|:-----------------:|:--------------------:|\n|  DINOv2-S   |       -0.06       |         0.09         |\n| R$_5$-ViT   |        0.36       |         0.33         |\n\nThis supports our claim that R$_\\lambda$-ViT produces more interpretable representations. We would be glad to include these quantitative results in the final manuscript.\n\n**Impact of KoLeo regularizer on training**. We politely disagree with the reviewer’s assessment that the KoLeo regularizer is unimportant in DINOv2. In DINOv2, Oquab et al. report clear performance gains in 1-NN classification and image retrieval when using KoLeo (Tables 1 and 3). In our work, KoLeo is not only retained for its empirical value, but also because it is mathematically essential for enabling Theorem 2, which assumes the injectivity of the ViT backbone. \n\n**Significance of theoretical results**. We respectfully disagree with the reviewer’s assessment regarding the theoretical contributions in Section 4. To the best of our knowledge, there is currently no existing mathematical formulation of ViTs that characterizes their representational and generalization properties in the manner presented in Theorems 1 and 2. \nWe would also like to clarify that Theorem 4 and Corollary 5 go beyond illustrating that deterministic inputs become stochastic or that the involved noise should be small. They establish a rigorous connection between the dimensionality of the latent space and the magnitude of noise required to preserve topological properties, a relationship that is foundational for understanding the behavior of our RMLP regularizer, as discussed in lines 193–203 of the manuscript. These mathematical derivations provide a novel lens on how randomization and latent space geometry interact in contrastive learning setups. We appreciate the reviewer’s engagement with our theoretical analysis and hope this clarifies the motivation and significance of our formal contributions.\n\n**Manuscript’s formatting**. While we understand the reviewer’s concern regarding the manuscript structure, we would like to clarify that the current organization was a deliberate choice made in service of both clarity and depth. Although we do not include a standalone Related Work section, the Problem Formulation serves a similar role by reviewing key prior art relevant to our approach. More importantly, this structure allowed us to allocate sufficient space to our theoretical analysis, which we view as a central contribution of the paper. Our Theorems 1–4 offer a comprehensive mathematical foundation that explains how RMLP influences the geometry of contrastive learning—a contribution that would have been difficult to present in full had we followed a more conventional layout.\n\n**Including a formal introduction to DINO**. Given the widespread adoption and extensive documentation of DINO in the literature, we initially chose to reference and cite the original work rather than provide a detailed reintroduction in the main text. However, we agree that briefly summarizing the core components of DINO would improve the accessibility of the paper, especially for readers less familiar with self-supervised ViT training. We are happy to include a concise overview of DINO in the revised manuscript.\n\n**References**\n\nDarcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv. arXiv:2309.16588 (2024)\n\nEckardt, F., Mittas, R., et al.: Deep Learning-Based Retinal Layer Segmentation in Optical Coherence Tomography Scans of Patients with Inherited Retinal Diseases. Klin Monbl Augenheilkd. doi: 10.1055/a-2227-3742 (2024)\n\nOquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)"}, {"review_id": "JJqJsnHG7A", "reviewer": "Reviewer_WhNa", "summary": "This paper introduces a regularization technique called Randomized-Multi-Layer Perceptron (Randomized-MLP). Randomized MLP is a contrastive learning-based method that \nThis work conducts the experiment on top of DINOv2, and compares Randomized MLP trained model versus regular MLP fine tuned model, and shows Randomized MLP improves downstream tasks including semantic segmentation and depth estimation. Randomized MLP fine-tuned also show cleaner attention maps among nature and medical images. Authors conduct theoretical analysis to prove Randomized MLP impacts the contrastive learning paradigm by turning point embeddings into probability balls with the help of KoLeo regularizer.", "questions": "* In contrast with previous work, our results show that this behavior also emerges in smaller versions of the model => Do you mean storing global information in patch tokens? Do you mean DINOv2-S is worse?\n* Line 87 “sparcity” => I believe you mean sparsity.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n* The methodology is intuitive and mathematical details are well explained. Authors explain the impact of each component, e.g., contrastive losses encourage similarity between views, KoLeo promotes uniformity, and GELU restricts the space to mostly positive coordinates. This paper also provides detailed visualization to explain the impact of different values of \\lambda. \n* The proposed method is not bound to a specific architecture and in theory it can be generalized to other domains, although the actual movement needs to be evaluated.\n* Theoretical foundation: authors provide theoretical analysis on how the embedding is projected and how it provides regularization effect.\n* Straightforward and comprehensive experiment setup. This paper evaluated Randomized MLP across different downstream tasks on different datasets.\n\nWeakness\n* It would be a plus if authors can show how this regularization works with other models, since it looks very generalizable and it should be easy to apply to other models. Does it only help DINO style models, or vision models, or can be generalized to NLP tasks?\n* Although authors present the negatively skewed proportion curve reflecting the sparse representation, I don’t find quantitative results on to what extent the attention maps become cleaner or better aligned with human perception.", "was_revised": true, "final_justification": "This work proposes a regularization method that improve downstream tasks and make the attention map more meaningful. Authors added results to generalize this method to non DINO models and also provided quantitive analysis on attention map alignment, which make the evaluation more comprehensive, thus I think the technical contribution is solid.", "rebuttal": "Dear reviewer, \n\nWe are glad to read that you found our experiment setup straightforward, our methodology intuitive and our mathematical results properly explained. We also want to thank you for your constructive feedback and would like to address your comments and questions in the following sections.\n\n**DINOv2-S artifacts on ophthalmological data**. We appreciate the opportunity to clarify this point. In stating that this behavior emerges in a smaller version of the model, we were referring to our observation that DINOv2-S places uninterpretable attention on void or low-informative regions in ophthalmological images (as shown in Figure 5). This phenomenon has been linked by Darcet et al. (2024) to global information being stored in patch tokens, but their analysis primarily noticed it on larger models. Our results indicate that this artifact also appears in DINOv2-S, suggesting that the issue is not limited to high-capacity models. We agree that this could be stated more clearly and will update the manuscript accordingly to avoid confusion. \n\n**Typo**. Good catch! Thank you. We will correct it.\n\n**RMLP implementation on other architectures**. Our theoretical analysis suggests that RMLPs can be applied broadly to any contrastive learning setup, independent of the underlying ViT architecture. To support this, we conducted some preliminary tests using the DINOv2 base model and observed a similar performance improvement by including RMLPs. Nevertheless, we decided not to include these results in this manuscript’s final version because we wanted to focus on the low-resource setup, to improve computational efficiency, a common challenge in the medical imaging field. Exciting follow-up projects could include investigating the effects of RMLPs not only on architectures like MoCo and Swin Transformers used in vision tasks, but also on those used for natural language processing, such as SimSCE or BERT.\n\n**New quantitative results (Correlating high-norm patches and anatomical structures)**. We acknowledge the importance of supporting qualitative observations with quantitative evidence and therefore conducted additional experiments to strengthen our interpretability claims. Using the dataset from Eckardt et al. (2024)—which contains OCT scans with expert-annotated retinal layers for patients with and without Inherited Retinal Disorders (IRDs)—we assessed the alignment between patch token norms and retinal structures. Punctually, we calculated the correlation between the spatial positions of the top 25% highest-norm patch tokens and the annotated retinal layer regions. The results show that DINOv2-S exhibits little to no alignment with anatomical structures, while R$_5$-ViT ($\\lambda=5$) demonstrates a stronger correspondence, indicating improved localization of clinically meaningful features. These results offer quantitative backing for our interpretability claims, particularly within ophthalmology, and we will happily include them in the final version of our manuscript.\n\n|    Model    | Patients with IRD | Patients without IRD |\n|:-----------:|:-----------------:|:--------------------:|\n|  DINOv2-S   |       -0.06       |         0.09         |\n| R$_5$-ViT   |        0.36       |         0.33         |\n\n**New quantitative results (Information in patch tokens)**. We further examined model behavior by assessing the diagnostic relevance of individual patch tokens. Using a protocol similar to Darcet et al. (2024), we conducted a 1-nearest neighbor classification task on the Eckardt et al. (2024) dataset to distinguish patients with and without IRDs. Explicitly, we compared classification accuracy using: (1) the class token, (2) the highest-norm patch token, and (3) the lowest-norm patch token.\n\nAs shown in the table below, DINOv2-S achieves decent accuracy with the class token (0.76), but its highest-norm patch token performs poorly (0.48)— worse than the lowest-norm token (0.76). This indicates that high-norm tokens may correspond to irrelevant or void regions. In contrast, R₅-ViT performs well with the class token (0.88) and retains high accuracy using the highest-norm patch token (0.76), while the lowest-norm token performs poorly (0.40), as expected. These results support our claim that RMLP promotes attention to diagnostically relevant regions, rather than repurposing low-information areas, and we will gladly include them in the final version of our paper.\n\n|   Model   | Class Token | Highest-Norm Patch Token | Lowest-Norm Patch Token |\n|:-------------:|:---------------:|:-----------------------------:|:----------------------------:|\n|   DINOv2-S    |      0.76       |             0.48             |            0.76             |\n|  R$_5$-ViT    |      0.88       |             0.76             |            0.40             |\n\n\n**References**\n\nDarcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. arXiv. arXiv:2309.16588 (2024)\n\nEckardt, F., Mittas, R., et al.: Deep Learning-Based Retinal Layer Segmentation in Optical Coherence Tomography Scans of Patients with Inherited Retinal Diseases. Klin Monbl Augenheilkd. doi: 10.1055/a-2227-3742 (2024)"}]

Author rebuttal:

Meta-review: This paper aims to improve the domain adaptation interpretability of the attention/feature maps of DINO, and applies the approach for medical image analysis. The manuscript is well written and clearly structured. The rebuttal process was effective. The authors successfully addressed major concerns of some reviewers by providing detailed explanations and additional evaluation with quantitative results. This paper overall received reviewers' ratings of borderline reject, borderline accept, borderline accept, accept. The AC takes the majority votes and would like to recommend a final acceptance of this submission.
Decision: Accept (poster)