Simplifying DINO via Coding Rate Regularization
Authors: Ziyang Wu
Jingyuan Zhang
Druv Pai
XuDong Wang
Chandan Singh
Jianwei Yang
Jianfeng Gao
Yi Ma
Venue: ICML 2025 poster (2025)
Keywords: Self-supervised learning
DINO
Primary area: deep_learning->selfsupervised_learning
TL;DR: We propose a methodology that simplifies and improves DINO, a widely used self-supervised learning algorithm.

Abstract: DINO and DINOv2 are two model families being widely used to learn representations from unlabeled imagery data at large scales. Their learned representations often enable state-of-the-art performance for downstream tasks, such as image classification and segmentation. However, they employ many empirically motivated design choices and their training pipelines are highly complex and unstable --- many hyperparameters need to be carefully tuned to ensure that the representations do not collapse --- which poses considerable difficulty to improving them or adapting them to new domains. In this work, we posit that we can remove most such-motivated idiosyncrasies in the pre-training pipelines, and only need to add an explicit coding rate term in the loss function to avoid collapse of the representations. As a result, we obtain highly simplified variants of the DINO and DINOv2 which we call SimDINO and SimDINOv2, respectively. Remarkably, these simplified models are more robust to different design choices, such as network architecture and hyperparameters, and they learn even higher-quality representations, measured by performance on downstream tasks, offering a Pareto improvement over the corresponding DINO and DINOv2 models. This work highlights the potential of using simplifying design principles to improve the empirical practice of deep learning. Code and model checkpoints are available at https://github.com/RobinWu218/SimDINO.

Simplifying DINO via Coding Rate Regularization
Ziyang Wu' Jingyuan Zhang? Druv Pai 1 Xudong Wang' Chandan Singh? Jianwei Yang²
Jianfeng Gao Yi Ma 124
Abstract
(Hadsell et al., 2006; Oord et al., 2018; Wu et al., 2018;
Grill et al., 2020; He et al., 2020; Bardes et al., 2021; Chen & He, 2021; Caron et al., 2021; Zhou et al., 2021; He et al., 2022; Assran et al., 2023; Oquab et al., 2023), captioned images (Radford et al., 2021), videos (Feichtenhofer et al., 2022), and text (Radford et al., 2018; Devlin, 2018; Radford et al., 2019; Brown et al., 2020), among other modalities. In the context of image SSL. there reconstructive (He et al., 2022), where the goal is to recon- struct some function of the true image data from a "view", i.e., corruption or augmentation, and confrastive (Hadsell et al., 2006), where the goal is, for each image, to have the features of different views of the image all close, and features of views of different images be far.
DINO and DINOv2 are two model families be- ing widely used learn representations from unlabeled imagery data at large scales. Their learned representations often enable state-of-the- art performance for downstream tasks, such as image classification and segmentation. However, they employ many empirically motivated design choices and their training pipelines are highly complex and unstable - many hyperparameters need to be carefully tuned to ensure that the rep- resentations do not collapse which poses con- siderable difficulty improving them or adapting them to new domains. In this work, we posit that we can remove most such-motivated idiosyn- crasies in the pre-training pipelines, and only need to add an explicit coding rate term the loss func- tion to avoid collapse of the representations. As result, obtain highly simplified variants of the DINO and DINOv2 which we call SimDINO and SimDINOv2, respectively. Remarkably, these simplified models are more robust to different de- sign choices, such as network architecture and hyperparameters, and they learn even higher- quality representations, measured by performance on downstream tasks, offering Pareto improve- ment over the corresponding DINO and DINOv2 models. This work highlights the potential of using simplifying design principles to improve the empirical practice of deep learning. Code and model checkpoints are available at https: / //gi figithub... com http//Robiniu21111 8 10/51/mmmmmmm /mD INO.
main approaches:
Within contrastive SSL, key challenge lies in preventing representation collapse, where models learn trivial solu- tions that map all inputs to the same output. One common approach to address this is through the use of negative sam- ples, which explicitly encourages representations of differ- ent images to be dissimilar. Thus far, the success of using negative samples depends on having a large batch size (Wu et al., 2018; He et al., 2020), which poses computational challenges at scale. Methods which attempt to avoid this bottleneck by using negative samples in more implicit and indirect ways to avoid collapse (Caron et al., 2021) can cope with smaller batch sizes, but often require training pipelines with many components and hyperparameters carefully tuned to avoid collapse, making them difficult to train.
The state-of-the-an art for image SSL is generally considered to be the DINOv2 model family (Oquab et al., 2023). It is built on the DINO model family (Caron et al., 2021). Both classes of models are trained using contrastive SSL and thus run into the representation collapse issue. While DINOv2 explicitly and directly uses negative samples to avoid col- lapse, it inherits much of its training pipeline from DINO, which uses negative samples more indirectly. As such, both model families' training pipelines are highly complex and unstable, requiring many tweaks and careful hyperparame- ter selection in order for the training to converge for a given architecture. Despite this capriciousness, the trained models' representations are highly useful for downstream tasks, and are widely used (Baharoon al., 2023; Wei al., 2024).
1. Introduction
Self-supervised learning (SSL) is the toolkit of choice to learn representations for large datasets of unlabeled images
'CC Berkeley 'TranscEngram *Microsoft Rescarch *HKU. Correspondence to: Ziyang Wu <zywu@ bww.uberrcloy..du....
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Simplifying DINO via Coding Rate Regularization
(a): DINO
(b): SimDINO
(X,)
V/g
X
CDINO
X
SimDDNO
Va
V2
X
(X.)
dae Ps (X.)
X
zal (X,)
(c): DINOv2
(d): SimDINOv2
Zpaida (X,)
Zaaic (X,)
V
X
X
DDNNOV
X
Vind
(X...
Fe (Xamx)
Uync
(Xmc)
me
Piuiky (Xi.c)
(X
Figure 1. The DINO and DINOv2 pipelines are substantially simplified to the respective SIMDINO and SimDINOv2 pipelines. (a) In the DINO pipeline, an input image is turned into patches. Then a global view Ug and a local view v. are randomly sampled. The global view is pushed through the teacher encoder, while the other view is through the student encoder. (b) The SimDINO pipeline removes the need for expensive post-pros processsin operations present in DINO, such as cumeension-nncranse softmax. (c) The DINOv2 pipeline adds masking (here masked patches are denoted by x) and an additional loss on image patch features (d) The SimDINOv2 training operates directly on the learned representations, simplifying the pipeline.
linear layer and a high-dimensions
to the DINO pipeline.
Our contributions. In this work, we remove many tweaks and hyperparameters from the DINO and DINOv2 training pipelines, replacing them with term in the objective which explicitly uses negative samples. We show empirically that this term, which involves the total coding rate regularizer (Ma et al., 2007; Yu et al., 2020; Li et al., 2022), enables much more simple, robust, and computationally efficient training pipelines, as shown in Figure 1. We show that the resulting models, named SimDINO and SimDINOv2, learn representations that achieve even higher oftte--ffthee- art performance as those learned by DINO and DINOv2 across a variety of downstream tasks. Our work underscores the value of understanding and simplifying pipelines to improve performance in vision SSL.
tures (i.e., a representation for each input patch). The net- work is a Vision Transformer (Dosovitskiy, 2020; Touvron et al., 2021) with appended multi-layer perceptrons (MLPs) to post-process each feature followed by L f--normalizations.
2. Methods: Simplifying DINO and DINOv2
2.1. Recap of the Original DINO Pipeline
The goal of DINO is to learn an aggregate representation of the input image which contains information about large- scale semantics of the input (e.g., the locations and proper- ties of different objects in the image). They do this via pre-training pipeline (Caron et al., 2021) which is depicted in Figure (a), and we also describe it throughout this sec- tion. The main idea is to take multiple views (i.e., different crops) of the data, and ensure that the features generated by these views are consistent with each other (in sense which will be made precise shortly) as much as possible. If the views each contain a salient part of the input such as central object, the feature corresponding to any view would then contain information about this central object. The end goal is that the feature of any large-enough view contains information about all relevant objects in the input image, which can then be extracted for use in downstream tasks such as image classification or image segmentation.
Notation. Let C, H. W. D, .N, d2 1 be positive integers. For a set A, let the space of finite sequences of elements of A be denoted as A = U A Our data will be images X E RCXHXW We consider different augmentations, or views, of the input data X, such as rotations or crops; we can represent view a function RCx#xW + DD. where N,, is the number of tokens in the view. By an abuse of vocabulary we also call 2 e(X) E DDXN, a view.
Let Sa-1 S Rd be the (d - di-dimnnsionnl e-sphere For the purpose of representation learning, we will con- sider an encoder neural network parameterized by weights 0 E ④ (i.e., the weight space), as a function fo: PR Sa-1 x S We factor fo = [ sfatch where folh: : Dx* - S outputs the so-called class token fea- ture (i.e,, an aggregate representation of the input patches) and pp spatch; (RD)* + S outputs the patch tokens' fea-
In the rest of the section, we will discuss the pre-training pipeline. As is common in contrastive SSL, the DINO framework uses two networks: a so-called teacher network parameterized by 0, E O, and a so-called student network parameterized by 0g E O. During pre-training, the loss
2

---

Simplifying DINO via Coding Rate Regularization
will encourage the student's representation align with the teacher's representation, even as the teacher is simultane- ously updated using student weights; this is self-distillation, and can be viewed as an optimization strategy or even im- plicitly regularizing the objective (Chen & He, 2021).
where the expectation is over X, the (local global) view Ve, and the global view dgg. and the function dcE is defined via the cross-entropy as
m dCE (p. 7) == Σ
Ps log 4:-
(5)
In the pipeline, we process each image X in the following way. First, we sample at random a view, or crop, Ve., inde- pendently of X; the view can either be a "global view" (i.c., large crop)
When training, DINO estimates the expectation in (4) by stratified plug-in estimator over a batch of sample images. That is, to estimate the expectation, we condition on X then estimate the conditional expectation E(dcr(-.-) X] via plug-in using several different global views (usually two global views, which play the role of the arbitrary view ve and the global view v/y) and several different local views, and finally average over X obtain the estimate. The optimization of this estimated loss, too, is done in an ad-hoc way; while all five parameters 8g. 8t, V//- DINO PINO are updated at each iteration, they update in different ways:
"local view" (i.e., small crop). We denote X. == vx ( E (RDxNoe U DDDNNg)). We also sample a global view Ug. and denote X, == v/ (X) E RDxNNto. Views are implemented in the same way as in DINO; they are formally described in Appendix A for completeness.
The (local or global) view X is fed to the student network2 fole get =5'=( (X.)= E Rd, while the global view Xg is fed to the teacher network for to get zi (X,) E Rd, i.e.:
cle/
scls J6,
(Xe),
(Xy) == fa. (X (L)
The student parameters 0₂ and VINO are updated via an iteration of a stochastic gradient descent (SGD)-type algorithm, such as Adam, on the loss (4). The back- propagation for the loss gradient is computed assuming the teacher parameters ft, 1i PINOO and J are "frozen" or constants (i.e., "stop-gradient").
Now, it is certainly possible to directly compare and evaluate these features. However, DINO adds post-processing steps, arguing that they improve performance and prevent collapse:
They add weight-normalized linear layers (Salimans & Kingma, 2016) h,,DiNo, h p Rd - IRm where > d, called the "DINO heads" and parameterized by DINO, PINO, appended to the end of the student and teacher networks respectively.
The teacher parameters 9G, 7 DINOO, and μ are updated via exponentially moving averages (EMAs) of the student weights 9z. the student DINO head ng DINO, and the average output of teacher the DINO head Elhh, (2.3 (X,))] (in practice estimated over minibatch), respectively. Formally, for decay parame- ters X, // E [0, 1]. at each iteration we compute 6 + A0 + (1 - 2) 1% DINO +- Anp DINO + (1 A) DINO and μ + vp + (1 - v) Elh,,pino (z (X,))
They center the teacher-computee features using a learned vector μ E RT
They take a seapeeratur-weighher softmax of both fea- tures to compute probability vectors in RTh, sometimes called prototype scores, which they then can compare using cross-entropy.
The decay parameters X, v/ and the temperature parameter T change along the optimization trajectory, and their schedules are design decisions which impact convergence.
Mathematically, the post-processing steps to get probability vectors for each view are as follows:
cla Xx) = softmax((1,pme (2 (Ta) Ps
(2)
As previously mentioned, many of the ad-hoc methods and choices described above are due to a tension: trivial solu- tion to optimizing (4) is to enforce that fe, and fe, collapse, i.e., become or approximate the constant function, which map each local and global view to the same feature = or even to the same probability vector p. To explain why DINO does not collapse, we wish to highlight the centering opera- tion in (3), which computes batch statisties during its EMA update, hence using negative samples and implicitly pushing different samples' features apart, even though the precise conceptual mechanism by which this occurs is not clear and involves a careful interaction between the centering vector and temperature scaling (Caron et al., 2021). In- deed, Caron et al. (2021) shows that collapsed solutions are common without very carefully tuning the EMA schedule
P.
cls X g) = noftmax((h,,p))
(Xg)) /L)/T.)
(3)
where Ta, T. are temperature parameters for the student and teacher respectively. Finally, the loss (to be minimized) encourages pcle( (X.) and p.la((X (X,) to be close using a cross- entropy functional dcF, which effectively distills the teacher into the student by aligning the predicted outputs:
DINO := Eldce(p;" cl (Xg). P (X.))l
(4)
'More precisely, let be random vector containing the bound- aries of the crop. so that Us crops exactly the region supplied by Analogous notation can be defined for g and V/g-
NNoe that the parameters 0g and 0, each contain a positional encoding over all patches; when view is fed through the network, receives an interpolated positional encoding of the view's 'sss crop.
3

---

Simplifying DINO via Coding Rate Regularization
and temperature schedule, and arguing that the remaining hyperparameters and choices would severely degrade the performance if perturbed. A more in-depth discussion of the tension, and the added complexity required to train model in spite of it, is in Appendix B. As we will see, if this tension is alleviated in an alternative way, many hyperparameters can be removed and the rest can be changed robustly.
where 7 > 0 is hyperparameter. Note that dea (2
2 20la = 1 (zghh) z since 25th 2/ E Sa-1-
When training, similar to DINO, we estimate the expecta- tion and covariance in (8) by type of plug-in estimator. Namely, the expectation is estimated similar to DINO, just using de= instead of dcE. To estimate the coding rate, we sub sub-sample several > (Xe) over both X and Vey, estimate Cov[z ( (Xx) Ve E Vglol on that sub-sample via plug-in, estimate R. of the population covariance by calculating it on the sample covariance, then average the estimates over all sub-samples. We conjecture that the latter estimator has lower variance compared to the naive plug-in estimator for Cov(2 (X.) de E Vglol as it is similar to variance- reduction methods in statistics (Kahn & Marshall, 1953), which we hypothesize might be factor as why SimDINO can handle smaller batch size than other contrastive SSL methods that explicitly use negative samples but avoid col- lapse using higher-variance or more implicit regularizers.
2.2. From DINO to SimDINO
To go from DINO to SimDINO, we ask the question:
Can we just compare 2g Xxx) and 2 (X))?
If we could do this, then we could avoid the large DINO head, the centering operation, the softmaxes, and the cross- entropy based loss. However, the mechanism in DINO for avoiding representation collapse via negative samples would therefore be removed. Thus, we have a second question:
The overall pipeline is shown in Figure ( Note that it is much simpler than DINO. We provide pseudocode for the training pipeline in Algorithm in Appendix D.
Can we efficiently use the negative samples' features explicitly to enforce non-collapse?
After training, we use the teacher network for evaluation.
For the first question, we argue that the most simple squared Euclidean distance, namely
2.3. From DINOv2 to SimDINOv2
dra (x, 3) : 1 2112 (z.
(6)
The pipeline of the DINOv2 framework (Oquab al., 2023). as shown in Figure 1(c), is built upon the DINO pipeline, and has two main goals: first, learn aggregate representation which contains large-scale semantics of the input (i.e., the goal of DINO); second, learn patch-based representations which have graineai semantic information about each patch and its local neighborhood. The main new ideas to achieve this, drawn from the iBOT pipeline (Zhou et al., 2021), that the imput the student has some masked patches, and that the loss also computes similarity of the patch-based features. To see why this works, consider if some patches are masked, and the model is able to predict masked patches using their unmasked neighbors, then from each patch the model can extract strong information about the semanties of nearby patches, which is an idea similar in spirit masked autoencoding (He al., 2022). Thus, these two ideas from iBOT would furnish our model with informative patch-based representations.
functional (5) applied to an affine transformation of the features, as in (4). For the second question, we argue that we may directly penalize the covariance of the features in order to avoid collapse, as follows. For a hyperparameter E >> O, the (total) coding rate (Ma et al., 2007; Yu et al., 2020; Li et al., 2022) of a symmetric positive semidefinite matrix I Rdxxd is
works at least as well as the cross- cross-entroopy-biee
1 (T) == logdet I+ d r 2
In words, R₂ is an approximation to the rate distortion with quantization error & of a Gaussian random variable with covariance (and this approximation is perfect in the limit E - 0). More concretely, it is a measure of size of non-
the covariance, even if the underlying variables Gaussian. Thus one way to ensure non-collapse is to add We E Velo)) as a regularizer (where Ue E Velo means that ve is a global view).² yielding the loss
- Re (Cov[=
We now discuss the DINOv2 pipeline, before discussing our modifications. While we have the same ("base"" views de and ii as before, we also consider masked view Ome, which computes Ve but, Uf E Vglo. subsequently replaces a frac- tion E [0 of the tokens in the view output with learn- able mask token Tmask (as in (He et al., 2022), the mask token is shared across all views). Similarly to previous nota- tion, we denote Xme == umo E DRDx Mis U DxD)) Now that we have this setup, we do similar operations to
LSinDINo == [[de=(z( cls (2)): 25l (X.))]
(8)
-e (Cov(2, (X.)) ve E Velol)-
'We only use global views" features for the sake of efficiency. If de in (8) is small, then the local and global views' student features close since they are both close to the global views' teacher features, so Cov (2. (X.) Rs Cov(2e 2 (Xx) I Ve E Vgoo).
4

---

Simplifying DINO via Coding Rate Regularization
DINO pipeline, with some changes:
the softmaxes, and compute the Euclidean satancee-based loss directly on normalized features. We obtain the loss
There are additional "iBOT heads" for the student and teacher, processing the patch-baser features column- wise (i.e., patch-wises wise), with weights n/. 1/BOT Th. iBOT (cf
LiiDDNOOe == Eldra (2 (x
(Xg), 2 (Xme))]
(16)
Naw
the "DINO head" with weights 1/2 PINO .]t PINO).
+ 2
Nglo de (x (X Us V/g
E
The centering operation on teacher-output features is performed on both the aggregate features and (column- wise) on the patch-wise features.
- 7 RR cls (X Ue E Velal)
The same caveats as in SimDINO apply with respect to how the expectations and covariances are estimated, and the optimization and evaluation procedures carry over. We provide pseudocode for the training pipeline in Algorithm 2 in Appendix D. In the sequel, we will show that these greatly simplified designs actually help the model performance.
The centering operation uses three iterations of the Sinkhom-Knopp algorithm (Cuturi, 2013; Caron et al., 2020), denoted below by SKC, instead of an EMA, and is parameter-free but more expensive than simple subtraction. Note that the Sinkhorn- Sinkhorn-Knopp algorithm uses features from all images each minibatch.
Optimal value for 7 In both the SimDINO loss (8) and the SimDINOv2 loss (16), in order to aid learning while making sure neither the distance term nor the regularizer term dominates, we choose y up to an absolute constant fac- tor so that it balances the asymptotic order of the gradient (Frobenius) norms of both terms. By the Cauchy- Schwarr inequality, it suffices to equalize the norms of the gradients of each term w.r.t. the features Z. Since the features are nor- malized on the sphere, the gradient (Frobenius) norm of the (10) distance term is (1). For the second term, assuming that (11) we use a batch size B, the gradient norm of the second term is Ol min (d, BJ/B). To make these equivalent, we (12) take 7 = @(v OVVB/(d BBddd BJ)). The same rate holds for SimDINOv2. While this choice of y is ultimately a heuristic, and the constant factor needs to be tuned, it helps to scale SimDINO and SimDINOv2 in practice. Formal calculations, including prescriptive choice for y taking all parameters into account, are provided in Theorems C.I and C.2.
Let geatch and ppatch be the patch-wise representations and prototype scores respectively, and let 2' E Rd be the gt column of Zaatch (and similar for p' - ppatch). Then we have (where 1 sis Neto):
(x (m 2.aacc/ mmm)) == fo. (Xmo)
(9)
(=thh (X 2, 2,aach (Xg) == fo. (Xg)
P= (Xmo) = coftmax(h,,pm)) (2 (Xme)) mTs)
p [ (Xmm)) == softmax(A (2)(( (mm))) )) Ts)
pele (Xg) = Doftmax(SKC(tt,pp) (z (X,) ()))
(13)
pi(X,, g) := softmax SKC (z. (X /)))
(14)
We then compute the loss using all such probability vectors:
1
DDINOV2- = Elder (1. (Xg), Pe (X mc))
2
(15)
3. Experimental Verification
Na.
In this section, we empirically investigate and evaluate proposed SimDINO and SimDINOv2 models and compare them to the original DINO and DINOV2 model families. In particular, we examine their differences in training dynamics and learned representation both quantitatively and qualita- tively. Overall, our experiments show that our proposed SimDINO model families can achieve better performance and learn representations of higher quality than the original DINO families while being significantly simpler and more robust to variations in hyperparameters and architecture
dcE(pl(Xg), g) (mm))) Ne ===00
y Entropy(2 2gla(( Ve Veloo,
where m,me is 1 if patch of Xe is masked by Tn and O otherwise, and the Entropy functional is the differen- tial entropy: it plays similar role as the coding rate Rg in SimDINO (and shortly SimDINOv2) in ensuring non- collapse. It is estimated by Oquab et al. (2023) using the KoLeo estimator (Delattre & Fournier, 2017)) which explic- itly uses negative samples. However, the Koleo estimator is a non-parametric estimator of the expectation of a function of a nigh-dimensional probability density (Beirlant al., 1997), and so it has relatively poor sample efficiency (i.e., the required batch size to converge in practice is large).
3.1. Experimental Setup
Model architecture. Since our method is directly built upon DINO and DINOv2, we adopt settings as close as pos- sible to the original method for fair comparison. Specifically, for all inputs we set patch size to be 16; we use the small, base, and large models of the ViT (Dasovitskiy, 2020) archi- tecture as the backbone, which is connected to a projector
We now greatly simplify the above pipeline using the same ideas as introduced in SimDINO. Namely, we dispense with the DINOABOT heads, the Sinkhorn- BaakhornKKnoppp centering, and
5

---

Simplifying DINO via Coding Rate Regularization
composed of three MLP layers with a hidden size of 2048 and output dimension of 256. The output features after the projector are ee normalized. Specifically for original (i.e., unsimplified) DINO models, these normalized features are then fed to a weight-normalized linear layer that outputs a high-dimensional (e.g., 65536) vector, before computing the softmax and then the cross-entropy loss.
75.0
74.5
SIMDINO
KNN Accuracy [%
74.0
73.5
73.0
72.5
Datasets and optimization. For pretraining, we use the ImageNet- IK dataset across all methods. For fair compari- son, we closely follow the original works (Caron et al., 2021; Oquab et al., 2023). We choose AdamW (Loshchilov, 2017) as the optimizer and adopt the same optimization strategies (e.g.. learning rates, warm-up schedules). For multicrop augmentation, we use 10 local views of resolution 96 x 96 and 2 global views of resolution 224 x 224 for all experi- ments. We provide more details on hyperparameter choices in Appendix E. We also consider several downstream tasks. Specifically, we evaluate our pretrained models on 1) un- supervised object detection and segmentation on COCO val2017 (Lin et al., 2014), 2) semantic segmentation on ADE20K (Zhou et al., 2017), and 3) video object segmenta- tion on DAVIS-2017 (Pont-Tuset et al., 2017).
71.5
60
70
80
90
100
Epoch
Figure 2. Evolution of k-NN accuracy of ViT-B trained for 100 epochs using DINO and SimDINO paradigms on ImageNet- K. We omit earlier epochs of similar metries better visual clarity.
Table I. Performance comparison on ImageNet-IK: IK, SimDINO and SimDINOv2 consistently outperform the original DINO and DINOv2 model families. They also more stable, while training of DINO on ViT-L diverged (row 3).
Method
Model
Epochs
k-NN
Linear
DINO
ViT-B
100
72.9
76.3
SimDINO
ViT-B
100
74.9
77.3
DINO
VIT-L
100
SImDINO
VIT-L
100
75.6
77.4
3.2. Experimental Results
DINOv2
ViT-B
100
76.0
77.2
ImageNet Classification. We report the classification ac- curacies on ImageNet-Ik in Table 1. Following (Caron al., 2021), we evaluate both k-NN and linear accuracy on the ViT backbones pretrained by the DINO model fam- ilies and our simplified variants. We observe that under both DINO and DINOv2 paradigms, our simplified methods are able to outperform the original pipelines. Furthermore, we observe that applying identical hyperparameter settings from ViT-B to ViT-L results in instability and divergence in DINO, while the same setup yields a steady improvement for SimDINO. To better understand the optimization dynamics of SimDINO, visualize the evolution of accuracy during training in Figure 2. It can be observed that performance of SimDINO steadily improves as training progresses, while optimization of DINO noticeably slows down, with even a slight performance drop near the end of training. Together, these results demonstrate our simplified pipelines' stability and ease of optimization compared the originals.
SimDINOv2
ViT-B
100
78.1
79.7
DINOv2
VIT-L
100
80.8
82.0
SimDINOv2
ViT-L
100
81.1
82.4
SWAV
VIT-S
800
66.3
73.5
MoCov3
VIT-B
300
76.7
emerging properties similar to the original DINO when us- ing our simplified algorithm. More qualitative results available Appendix F.6. To quantitatively evaluate these representation, we perform MaskCut on the COCO val2017 dataset and report our results in Table 2. These results show SimDINO achieves much stronger performance on segmen- tation and detection tasks than DINO when trained on the same network (row 2 vs 3), and overall even outperforms DINO trained on smaller patch size" (row 2 4).
Semantic Segmentation on ADE20K. We evaluate our proposed methods on the ADE20K semantic segmentation task and report the results in Table 3 (columm 3 & 4). Specif- ically, we follow the linear evaluation protocol of (Zhou et al., 2021), where we fix the pretrained backbone and only finetune linear layer on top of it. From the results, observe that proposed SimDINO consistently out- performs the original algorithms. In particular, on ViT-B, SimDINOv2 is able to improve DINOv2 by 4.4@mloU,
Object Detection and Segmentation. To better under- stand the learned representation, we evaluate the pretrained models on segmentation and object detection tasks. Specif- ically, we adopt MaskCut (Wang et al., 2023), an effec- tive unsupervised approach of extracting features from frozen vision backbone for object detection and instance segmentation. In Figure 3, we present qualitative segmen- tation results by applying MaskCut on models trained with both DINO and SimDINO. Both methods are observed to produce meaningful segmentation results, confirming the
'When trained using DINO, ViT models with smaller patch sizes tend outperform those with larger ones on various tasks including segmentation (Wang et al., 2023; Caron et al., 2021).
6

---

Simplifying DINO via Coding Rate Regularization
Table Unsupervised object detection and segmentation via MaskCut evaluated on COCO val2017 under COCO's official evaluation protocol. SimDINO conclusively performs better than the DINO at detection and segmentation metrics, comparable with DINO with smaller path size (16 vs 8).
Table 3. Semantic segmentation on ADE20K and video object segmentation on DAVIS-2017. For semantic segmentation, we train linear layer on the frozen pretrained backbone. On DAVIS, we segment scenes between video frames using nearest neighbor search. On both tasks, SimDINO(v2) consistently outperforms their original counterparts.
Detection f
Segmentation f AP
Method
Model
AP50
AP7s
AP
APau
AP7s
Lin. Seg. mAcc
Vid. Seg. t Im
SimDINO
ViT-L/16
5.4
1.9
2,4
4.5
1.4
1.9
Method
Model
mloU
(J&F) m
Fm
SImDINO VIT-B/I6
5.2
2.0
2.5
4.7
1,5
2.0
DINO
ViT-B/16
33.1
41.9
63.0
61.5
64.4
DINO ViT-B/16
3,9
1.5
1.8
3.1
1.0
1.4
SImDINO
VIT-B/16
33.7
42.8
63.0
61.6
64.4
DINOv2
VIT-B/16
32.5
41.4
53.2
52.7
53.7
DINO
VIT-B/8
5.1
2.3
2.5
4.1
1,3
1.8
SimDINOv2
VIT-B/16
36,9
46.5
60.9
60.4
61.4
DINOv2
VIT-I/16
41.0
50.8
62.0
61.7
62.3
These results suggest that our simplified methods lead to representations favorable to dense prediction tasks.
SimDINOv2
ViT-I/16
41.8
52.2
62.6
61.9
63.3
to encode different augmentations (i.e., views) of the same input, and pushes the features of these augmentations to- gether, similar to SimDINO. SimCLR uses explicit negative samples in the loss, while SimSiam manipulates the loss gra- dient structure using stop-gradients to avoid collapse. Both methods' losses measure alignment or difference via the squared Euclidean distance (cquivalently the dot product) of the features. In contrast, SimDINO uses two separate networks - the teacher and student - that update via self- distillation. Furthermore, SimDINO uses the inner product of features in the loss, but it also uses coding rate regu- larizer instead of implicitly contrasting negative samples or using the more bespoke contrastive loss in SimCLR.
DAVIS Video Object Segmentation. In Table 3, we also provide evaluation results on DAVIS-2017 video instance segmentation benchmark. We follow the same evaluation protocol in (Caron al., 2021) and segment scenes be- tween consecutive video frames with nearest neighbor. We observe that our proposed SimDINO(v2) outperforms the original methods on this task. One interesting observation is that despite achieving much better NN accuracy, DI- NOv2 generally underperforms the original DINO in this task (and similarly for the simplified variants). A similar phenomenon is noted in (Zhou et al., 2021), where this discrepancy is found to be caused by the sensitivity of the evaluation protocol itself (e.g., image resolution). In our evaluation, we do not tune these individual factors and sim- ply adopt the same setting across all models we consider.
Explicit covariance regularization in SSL. There have also been works that use explicit penalization of the first- and second-order statisties of the features, such as VICReg (Bardes al., 2021). VICReg uses completely separate networks to encode two augmentations of the same input batch, and then explicitly penalizes the aligmment of those features (via Euclidean distance) as well as the features' variance and covariance within the batch, aiming to whiten the features as much as possible. In spirit, this is similar to SimDINO, which also penalizes the alignment and the features' covariance, albeit using different covariance reg- ularizer and not penalizing the features' variance. Also, SimDINO uses self-distillation to train the teacher network, while VICReg uses two separate networks.
More on Stability and Robustness. Apart from the ob- served divergence on ViT-L. in Table 1, we note that DINO is sensitive to its pipeline-specific hyperparameters. as ev- idenced in Table 6 (in Appendix F). To further verify the stability of SimDINO, we experiment with training both algorithms on a different dataset than ImageNet- kk. Speci- fielly, we train them on COCO train2017 (roughly 1/10-th the size of ImageNet-Ik) and report the results in Figure 4. Under this setting, SimDINO vastly outperforms DINO. We provide additional ablations on other factors (e.g. batch sizes) in Appendix F. Together, these results demonstrate the superior stability and robustness of SimDINO.
4. Related Work
Self-distillation in SSL. Several works such as MoCo (He et al., 2020) and BYOL (Grill et al., 2020) train two networks, teacher and a student, via self-distillation by setting the teacher weights to be an exponential moving average of the student weights. While MoCo uses explicit negative samples from previous batches in its InfoNCE loss computed on a given batch, BYOL does not use negative samples but instead manipulates the gradient structure (akin to SimSiam) in order to prevent collapse, and it uses an extra ("prediction") module appended to the student network,
In this section, we identify several previous works which the SimDINO and SimDINOv2 methodologies are similar to or build on. We have already discussed similarities to DINO and DINOv2 in depth we omit this comparison.
Siamese contrastive SSL. Siamese contrastive learning, archetyped by SimCLR (Chen et al., 2020) and SimSiam (Chen & He, 2021) among others, uses the same network
7

---

Simplifying DINO via Coding Rate Regularization
Figure 3. Visualization of MaskCut segmentation results from DINO ViT-B/16 (row 1), SimDINO ViT-B/16 (row 2) and SimDINO
ViT-L/16 (row 3) selected images.
powerful measure for non-collapse or expansion of the fea- tures from a given batch. Other regularizers to accomplish this include the VICReg-type regularizers and the MMCR regularizer (Yerxa et al., 2023; Schaeffer et al., 2024).
DIND SimDINO
k-NN Accuracy (%)
20
5. Conclusion
In this work, we identify that the reasons for many empir- ically motivated design choices in the original DINO and DINOy2 are to avoid collapse of the learned representa- tion. We show that these complicated design choices can be significantly reduced or simplified by adding a coding-rate- related regularization term. The resulting simplified models, called SimDINO and SimDINOv2, even better in terms of performance for downstream tasks, and their pretraining pipelines are much more robust to different settings and hyperparameters, offering a Pareto improvement against the DINO and DINOv2 model families. Our work demonstrates the value of simplifying deep learning pipelines as well as making tradeoffs as explicit as possible when designing high-perffrmance vision SSL models.
30
Epoch
Figure 4. k-NN accuracy on ImageNet-iK of ViT-B trained on COCO train2017 using DINO and SimDINO paradigms.
making the teacher and student asymmetric. SimDINO uses self-distillation with the same architecture for teacher and student, explicitly uses the simple Euclidean distance in the loss, and explicitly uses the coding rate to prevent collapse.
Patch feature prediction in SSL. While most contrastive SSL methods pick a single feature vector (say, of the cls
token) as the representation, recent contrastive learning ap- proaches such as DINOv2, I-JEPA (Assran al., 2023), and C-JEPA (Mo & Tong, 2024) compute losses on the features corresponding to each patch. In I-JEPA, there is one local and one global view, whose crops are nested, and the (Euclidean distance) loss is only computed on the patch features. C-JEPA adds VICReg-esque variance and co- variance penalty to the objective of I-JEPA. In contrast, in SimDINOv2, there are multiple local and global views, the loss incorporates both patch-based and aggregate features, and collapse is prevented by using coding rate term.
In light of these overarching contributions, there are several possible opportunities for future work. On the theoreti- cal side, our simplified framework provides an entry point for studying the geometric properties of the global optima of suff-sueervieed learning losses. Further study in Ap- pendix F.4 shows that in the framework of the paper, is possible to set up a self-supervised objective that does not require oolf-ditillation to optimize, making a theoretical analysis much easier, while the resulting model is still quite powerful and practically usable. On the empirical side, one can apply the paradigm of making implicit design choices more explicitly present in the loss to more self-supervised learning frameworks, making existing pipelines more stable and the resulting models of better performance.
Coding rate, and related regularizers. Several works have used coding rate-related terms in the objective (Ma et al., 2007; Yu et al., 2020: Dai et al., 2022; Tong et al., 2022) as well as a way to evaluate quality of representations (Yu et al., 2023; Pai et al., 2023; Wu et al., 2024; Yang et al., 2024). The coding rate has thus been shown provide
8

---

Simplifying DINO via Coding Rate Regularization
Impact Statement
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
This paper aims to advance the field of Machine Learning. Since DINO and DINO-2 are very popular methods for un- supervised learning of visual representations, we believe that the methods presented in this work will significantly ad- vance the practice of visual representation learning. Except for values to academics, we do not anticipate any significant social or ethical implications.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Acknowledgment
Chen, X. and He, K. Exploring simple siamese represen- tation learning. Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Yi Ma would like acknowledge support from the joint Simons Foundation-NSF DMS grant 22001199, the ONR grant N00014-22-1-2102,, the NSF grant #2402951, and also support from and the HKU startup, the Hong Kong Center for Construction Robotics Limited (HKCRC) Award 052245, and JC Club of Hong Kong. Druv Pai acknowl- edges support from a UC Berkeley College of Engineering Fellowship.
Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Dai, X., Tong, S., Li, M., Wu, Z., Psenka, M., Chan, K. H. R., Zhai, P., Yu, Y., Yuan, X., Shum, H.-Y., et al. Ctrl: Closed-loop transcription to an ldr via minimaxing rate reduction. Entropy. 24(4):456, 2022.
References
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619- 15629, 2023.
Delattre, S. and Fournier, N. On the kozachenko-leonenkoo entropy estimator. Journal of Statistical Planning and Inference, 185:69-93, 2017.
Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint urXiv:1810.04805 2018
Baharoon, M., Qureshi, W., Ouyang, J., Xu, Y., Phol, K., Aljouie, A., and Peng. W. Towards general purpose vi- sion foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks. arXiv preprint 23xvv2312.002666 2023.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition scale. arXiv preprint wrXiv:2010.11929 2020.
Feichtenhofer, C., Li, Y., He, K., et al. Masked autoen- coders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958 2022.
Bardes, A. Ponce, J., and LeCun, Y. Vicreg: Variance- invariance-covariame regularization for supervipe learning. arXiv preprint 105.0:20 2021.
Grill, J.-B., Strub, F., Altché, F,, Tallec, C., Richemond, P.,
Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z. Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Beirlant, J., Dudewicz, E. J., Györfi, L., Van der Meulen,
E. C., et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
reduction by learning an invariant mapping. In 2006 IEEE computer society conference computer vision and pattern recognition (CVPR (CVPR'06), volume 2, pp. 1735- 1742. IEEE, 2006.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9999 2924,, 2020.

---

Simplifying DINO via Coding Rate Regularization
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Kahn, H. and Marshall, A. W. Methods of reducing sam- ple size in monte carlo computations. Journal of the Operations Research Society of America, 1(5):263-278, 1953.
Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information process- ing systems, 29, 2016.
Li, Z., Chen, Y., LeCun, Y., and Sommer, F. T. Neu- ral manifold clustering and embedding. arXiv preprint 2Xiv:2201.10000 2022.
Schaeffer, R., Lecomte, V., Pai, D. B., Carranza, A., Isik, B., Unell, A., Khona, M., Yerxa, T., LeCun, Y., Chung, S., et al. Towards an improved understanding and utilization of maximum manifold capacity representations, arXiv preprint arXiv:2406.00 09366, 2024.
Lin, T.-Y., Maire, M., Belongie. S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12. 2014, Proceedings. Part V 13, pp. 740- 755. Springer, 2014.
Tong, S., Dai, X., Chen, Y.. Li, M., Li, Z., Yi, B., LeCun, Y., and Ma, Y. Unsupervised learning of structured repre- sentations via closed-loop transcription. arXiv preprint arXiv:2210.1677 2022.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Touvron, H.,, Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- ers & distillation through attention. In International con- ference on machine learning, pp. 10347-10357. PMLR, 2021.
Ma, Y., Derksen, H., Hong, W., and Wright, J. Segmenta- tion of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 29(9):1546-1562 2007.
Wang, T. and Isola, P. Understanding contrastive represen- tation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929 9939.9 PMLR, 2020.
Mo, S. and Tong. S. Connecting joint-embedding predic- tive architecture with contrastive self-supervised learning- arXiv preprint wxXiv:2410.19566 2024.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: 03748, 2018
Wang. X., Girdhar, R., Yu, S. X., and Misra, I. Cut and learn for unsupervised object detection and instance seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3124 3134, 2023.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Wei, Z., Chen, L.,, Jin, Y., Ma, X., Liu, T., Ling, P., Wang. B., Chen, H., and Zheng, Stronger fewer & superior: Har- nessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28619-28630, 2024.
Pai, D., Wu, Z. W., Buchanan, S., Yu, Y., and Ma, Y. Masked completion via structured diffuusion with white-box trans- formers. International Conference on Learning Represen- tations, 2023.
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimina- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine- Hornung, A., and Van Gool, I.. The 2017 davis challenge on video object segmentation. arXiv: 00675, 2017.
Wu, 7.., Ding, T., Lu, Y., Pai, D.,, Zhang, J., Wang, W., Yu, Y., Ma, Y., and Haeffele, B. D. Token statistics transformer: Linear-time attention via variational rate reduction. arXiv preprint arXiv:2412. 12.412..77/// /7810, 2024.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre- training. 2018
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Yang, J., X., Pai, D., Zhou, Y., Ma, Y., Yu, Y., and Xie, C. Scaling white-box transformers for vision. arXiv preprint arXiv:2405.20222 2024.
10

---

Simplifying DINO via Coding Rate Regularization
Yerxa, T., Kuang, Y., Simoncelli, E., and Chung. S. Learn- ing efficient coding of natural images with maximum manifold capacity representations. Advances in Neural Information Processing Systems, 36:24103-24122 2023.
Yu, Y., Chan, K. H. R., You, C., Song, C., and Ma, Y. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in neural information processing systems, 33:9422- -9434, 2020.
Yu, Y., Buchanan, S., Pai, D., Chu, T., Wu, Z., Tong, S., Haeffele, B., and Ma, Y. White-box transformers via sparse rate reduction. Advances in Neural Information Processing Systems, 36:9422-9457, 2023.
Zbontar, J., Jing, L., Misra, LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learn- ing. pp. 12310-12320 PMLR, 2021.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 633-641, 2017.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre- training with online tokenizer. arXiv preprint arXiv:2111.0778. 2021.

---

Simplifying DINO via Coding Rate Regularization
A. Formal Description of Local and Global Views
Each local view, say vy acts as follows, given an input image X of shape (C, H, W). First, for a hyperparameter Ploe E 00 crops rectangular component from X of shape (C, He, We), where Hę and We are chosen such that H,W, HW. i.e. the crop is fraction Ploe of the whole image. Then the component is resized shape (C, Stoc Stoe). where Stoe hyperparameter, and then divided into Mic == / P square patches of shape (C, P), where the patch size P is hyperparameter. Each patch is unrolled into a vector of length D == CP2, and the Nic unrolled vectors are placed in raster order as columns to get the output X₂ E IR DDeM. Each global view Vg acts the same as a local view, except that the corresponding hyperparameters Pglo, Sglo are larger than their local counterparts plec: Stoe (hence also Nglo vs. N1oe), while the patch size P (hence dimension D) remains the same. 5
We use these local and global views for training. For evaluation or inference, we do a similar procedure: given X of shape (C, H, W). we resize X proportionally so that its shorter edge is length Leval, then take square crop from the center of shape Seval: Seval). This sequence is divided into Neval == / P2 square patches of length (C, P. P); each patch is unrolled into vector of length D == CP2, and the Neval unrolled vectors are placed in raster order as columns to get the output X, E D
B. Complex Interactions in DINO and Their Removal
We wish showcase finer point about why the DINO pipeline is so unstable. Notice that
(cE(p.g) q)
Σ
Pi log q
(17)
i=1
= Σ
ps log(p:/(:)) -
Σ p. log
(18)
1=1
dkr.(p. 9) + H(p)
(19)
where dKL. is the KL divergence, and H is the entropy of a probability distribution. In other words, this objective is minimized whenever p q and both are one-hot vectors. Now consider the DINO objective:
DIIOO = E[dcE(P] rls (Xg): P cls (Xx) x))] = E[dKL (Xg), p. (Xc)) + H cl (X,)))
(20)
Suppose that, for example, hypINo and hyp had ranges multiple of the all-ones vector, and were a constant multiple of the ones vector. Then the first term in the loss would be minimized, but the second term would become as large as possible (since both p would be just 1 m 1m i... probability vectors corresponding to the uniform distribution), so this would not be the optimal solution in general. This implies that the learned fyNINo and hypINo in general would not both be degenerate. This enables the tradeoff between the EMA parameter and the temperature parameters Tg T which enables non-collapse. If the objective just vnvolved the KI. divergence and not the entropy term, or else had hygo be degenerate (manually set and frozen, for instance), or else didn't 't have carefully set tradcoff between X, Ta, Tr, then the model would collapse. However, SimDINO removes all of this complexity and replaces it with an explicit coding-rate- term.
C. Theory for Hyperparameter Scaling
To develop estimates for how different terms the loss scale with different batch sizes, we first introduce an empirical version of the SimDINO loss. For given optimization step, we:
Sample minibatch {X,.... X
For each sample X,, draw M global views uio and Mice local views ofor. Define X biglo == udoo and X := (X,).
Compute the features 2 cls (Xi x (X and x5 (Xb
*Of course, we also need the patch size P to divide both the image sizes Stoc and Sglo
12

---

Simplifying DINO via Coding Rate Regularization
Form the tensors z E RBxxMxd Zels
RBxadgh xd where M
Malo
Mioe by
2s (X) heln
if sissss Mglos
(Zgh)) bx
(2 = 21 ( (xbglo)
(21)
2 (X) Mal0)
6,100
otherwise
Compute the loss
B Mgin
Mglo
(Z ),r ( (2 ):.
SSMDIIN MgM glo
Σ
ΣΣΣ
(Z(h))) bi (Zi
)bb
Re
(22)
Mglo
where for 1 iis Mglo have
Ig 1,i: (Zgh) B,,l E RBBxd
Our main theorem for SimDINO the following.
Theorem C.I. The (Frobenius) gradient norm of (22) is
v /dminnd BJ/B
V n Simminoll
+ +
(23)
VM
28
Proof. For the first term, call upon Lemma C.4 which says that
[1 1 Ma
{
V
(2eh))
[( Jos
(Z 1b l2
Mglo VM.
(24)
This allows to compute the gradient norm of the first term in (22) as
B Malo
M
V Z
(Zgh))0 10gl1
(25)
2BM glo
b=1
B Mgln
V Z
(l )oo (Z )ay)
(26)
MM glo
2
hml
1=1
Malo
<
BMM glo
V
Zel
(( )oi (Z
(27)
b=1
B
B
Mato M
V (2gb))
(Z (Z )o,1
(28)
BMM glo
W=1
1=1
Moto
V (2cb)))
|| (Z(s))) (2r
(29)
MM 1=1
< MM
Σ Mgto M
(30)
(31)
VM
For the second term, we call upon Lemma C.6 which says that
cls )..
min (d, B\/B
(zab),. Re
(32)
B
45
13

---

Simplifying DINO via Coding Rate Regularization
This allows to compute the gradient norm of the second term in (22) as
Mgla
(Zal))
Mgla
(Z T
Zols
7
Re
V
Z
R
(33)
Mgg
B
Mglo
B
F
Malu
(Zahh) : (
7
ZztaRa Rs
(34)
Mglo
FF
Mgto
Meto
(Zg ):. (Zela..,
V (2st)) Rs
(35)
glo
B
Mato
(Z ):00
(Z
7
V (Zgh) R&
(36)
glo
B
Mgto
dmin {d, B}
(37)
M
glo
28
B
minn (d, B3
= 2e
(38)
B
Putting these terms together via triangle inequality, have that
V z CimmINll
&
1
+ Y
dmnn B}
(39)
VM
2s
B
as desired.
We can also come up with another result for SimDINOv2. This requires a slightly revised pipeline, as well as a different loss. For a given optimization step, we:
Sample a minibatch (X1....,_p
For each sample X,, draw Mglo global views Vooo and Mic local views ufoc. Define X biglo == uooo and X x1o = ufoc (X)).
For each sample Xs, draw Mglo masks mi and apply them to each global view. Define X X b.glo.mack = m/ (igloo) and define bb be the indicator variable whether patch is masked by mi-
Compute the features
(X (aago, Zpatch (X mask. 2016 (X (ooo))): 20 (X: ( and 2 patch X b.gto))
Form the tensors Z E RBxarxd Zels E RBxMgaxd Z Zaatch E RBxMMixXNooxd zpatch RBxdfga where M == Mgto + MMocs by
2/ (X
mask) 1-
if ii Mglos
(Z )m
(Z = = zola (X (.oo)
(40)
cle
x-
Malo
otherwise
b,
Agllo )ii Zpatch X
mask))
(Zpatch) )( = patch (Xi.gta)
(41)
Compute the loss
B Mato M
2SimDINOv2
Σ
(Z )ms (Z c/o )os11 (2
(42)
2
2BM M gin
j=1
B Mato Nato
(2 oon (Z Join 1221 b,n
patch
2BMslo Velo
bml
m=1
14

---

Simplifying DINO via Coding Rate Regularization
Male
( (Zgh) (Z.
)
R
1:
Mglo
where for 1 i S Malo we have (Z ):d = [Z 1,
(cle
(Z {
E RRxdd
Our main theorem for SimDINOv2 is the following:
Theorem C.2. The (Frobenius) gradient norm of (42) is
V Z Dinovll
1
7
d minnd, B}
(43)
2
VM
V
glo
2e
B
where (recall) [0, is the fraction of patches of the input global view which is masked out.
Proof. By the proof of Theorem C.I, the gradient norm of the first and third terms are known. So we study the second term. We can invoke Lemma C.5, which tells us that for every b and we have
Ngto
}
V
(
patch
Join
(Zaato " Join 1121 bolóon
= V( NN)
(44)
2
n=
using the identity that for each (z, b) pair that
glo
Σ R=1
1/ = - ag)NNoo
(45)
Then, holds overall that
Mato Nito
V Z
Σ
[12 patch
Jbi - siglo
16in1 - Join 1121 b
(46)
2BM
Noto
Rlo
b=1
s=l n=1
M Ngto
1
Vz.
(Z natch Jon - -Zanato Join 101m p,m
(47)
BM
11 ==1
B Ma
Neto
V Z.
11 (1 bin - -Zaaato Joinl 11:11
(48)
BMgloN Nglo
2 ==1
b=1
B Mglo
B
Mglo
Ngto
gla BM glo N
V (Zppnti Yorr
Σ
||
[[(patch
Jain
patch sili Join
b.n
(49)
b=1
6
n=1
B Mglo
Ngto
MMM N
glo
paacchh Joi
(2patcch) Join (Zaatch )bin 11211
(50)
b=1
n=l
B
Ma
1
V
a)
(51)
M.Mgoo Nglo
V1 1 a
(52)
/Ngoo glo
Using the gradients computed in Theorem I.I and the triangle inequality, it holds that
d min (d, B}
|| 2 E S
a
2 V B
Y
(53)
VM
Nglo
as desired.
15

---

Simplifying DINO via Coding Rate Regularization
Remark C.3. In the main body, in order obtain a prescription for how scale y with the batch size B, choose 7 to make the different terms in (22) and (42) have equal magnitude. Given the explicit rate in Theorem C.I, the value of chosen in SimDINO is
B dM min B}
7SNDDINO 2c
(54)
Meanwhile using Theorem C.2, the value of y chosen in SimDINOv2 is
(VM
1 VM
l-a a Nglo
B dimin B)
7SimDINOv2 =
V
(55)
If, for instance, we are just interested in scaling with the batch size B (so that o: M, No d are held constant), then 7SinDINO and YSimDINOv2 have the same asymptotic order. In practice, we take these prescriptions for y up to multiplicative constant, which is tuned on a single setting and can then be transferred to different settings.
C.1. Auxiliary Lemmas
Lemma C.4 (Scale of Gradient of Pairwise Distance Term). Let d, m, n be positive integers. Let A E Ruxd and B E Rexd have rows ag and by which are unit- unit-P-noomm -norm, i.e,, ||alI22 = ||bحl| = 1 for E
j E {l...., n... Then
V
la, 6,13
S <mvn.
(56)
1=1 1=
Proof. Since all rows are normalized,
aa - b,1 =
bj =
Σ a
bj
(57)
$=1 ==
1=1
-1 -1 A) B)
(58)
- AB AB 1n
(59)
matrix calculus computation shows that
B
AB 1n = -11,1
(60)
m
To bound the norm of this term, we have
1
1 Al
1,, Allr S 11 lop |
(61)
It is easy to show by matrix algebra that and that, since the rows of A are normalized,
| 1 Vmn,
(62)
|
1al12 mm. Vn.
(63)
1=1
Putting these together it holds that
m
n
B
|a; - -b,103
$ mvn
(64)
2
j=l
as desired.
16

---

Simplifying DINO via Coding Rate Regularization
Lemma C.5 (Scale of Gradient of Patchwise Distance Term). Let d, n. be positive integers. Let A, B E [Ruxd have rows a, and b which unit-fe mnrmm iee, ||a:||2
|by 2 1 for i.j (1 n... Then
V B
a4
bill2
vn.
(65)
Proof. Since all rows are normalized,
1
Σ 1=1
a.
bull2
-
a b,
(A) ()); (B)
(66)
i=1
i=
matrix calculus computation shows that
V tr tr A.
(67)
The Frobenius norm of this term can be explicitly calculated as
|
F
1al12 = vn Vn.
(68)
==1
Putting this together, we obtain
V
a.
b
= -vn,
(69)
2
as desired.
Lemma C.6 (Scale of Gradient of Coding Rate Term). Let d, be positive integers. We have
Z Z
d dminfd n
max ZER || | V4
VzR
(70)
n
11
where 2; is the ith row of Z.
Proof. Let a = d/(n=²) and let f: [Ruxd R be defined by
1
f(Z) logdet (I aZ Z),
(71)
i.e., ((2) R2 (ZTTZ)//). Now, let == min(d, (d, n). For any matrix M, let (M(M) be its jth largest singular value, for i I,..., d. First, note that since ||z,||2 = for all i, holds
d > 2 2)
Σ (2) Σ o. (2)
Σ i=1
(
o. Z 2) tr 2)
Σ 2 = d.
(72)
1=1
Now, we can simplify the gradient. holds
VS(Z) aZ aZ 2Z-1
(73)
Thus, holds that
(74)
= +az T -2)
(75)
17

---

Simplifying DINO via Coding Rate Regularization
Using that the trace is the sum of singular values, it holds by taking the SVD of Z that
tr( (1Z(++oZZZ)=22]
(Z
aZ 2)-
(76)
==1
(77)
[] + ao, Z ??p
In this case we directly optimize over the singular values, obtaining the problem
ZEREXA ==
max
|V](Z <
max in EER 24 8,20 vi
+ ar,)) )2
(78)
Eis aind
The function + (1+0r]= on [0, oo) has global maximum at
6
1 and the value 1. Therefore it follows that la
max EE
(79)
MER 3:20 E m/=dd
+ 0.0. )
4a
i=1
4:20 vi t=
Unpacking this notation, we obtain
|V saa.
or
d dmnn n)
(80)
4a
4
4ng²
Taking square roots, it holds
1
dmin d n}
|[][(2)| S
(81)
2e
Therefore,
<T2 Z
dmin nd n)
z Rsg
= =V][(Z)| S 25
(82)
F
71
as desired.
Remark C.7. is possible that the inequality
max
[Vf(Z)]l
max WER
32 az+,
(83)
ZeRuxd 1=
2. 20 D.
is met with equality; proving this would require exhibiting Z fulfilling the constraints of the first problem such that it has the prescribed singular values which solve the second problem. We do not need to do so here for the purposes of using the bound (e.g., for learning rate scaling).
Remark C.8. While the quick-and-din dirty bound
max "ER" m.20 vi
Σ
32 (1 + ax;)²
S
(84)
4a
i=1
E a.=d
by way of ignoring the constraint Σ =; d seems like it could significantly loosen the bound, we do not believe this is the case. In particular, when 1/a S d/r, note that setting 2 = = _,- = 1/a and x, = d - (r-1))/ 1)/a sandwiches the objective between (r (r-1)/(4a)) 1)4(4a) and r/ r4(44),, so the maximum is at least the same asymptotic order, in the very reasonable case that is small enough that 1/a & d/r, i.e., using the definition of a, such that
1 d
d d²
x² max
(85)
n min n}
n n²
Similar strategies should hold if we allow for an absolute constant c 2 such that 1/a S od/ rr etc, relaxing the requirement while preserving the asymptotic order of the LHS of (84).
18

---

Simplifying DINO via Coding Rate Regularization
D. Training Pipeline Pseudocode
this section we provide pseudocode for the training pipelines SimDINO and SimDINOv2.
Algorithm 1 SimDINO training pipeline.
fs, it: student and teacher networks, this time outputting ONLY the cls toxen feature eps: coding rate reqularization quantization hyperparameter
gamma: coding rate reqularization strength hyperparaneter
lam: teacher network EMA rate
ft. ftrparams fs.params
for x in Icader: # load a minibatch x of samples
xg, xl hiebal_viewsli (x), vocal_views(e & (B, Mglo, N_glo, D), (B, M_ccc, N_loc, D)
2sg,
(xg), fs (x1) student output (B, Mglo, d), (B, Mioc, d)
2tg (xg) # teacher output (B, Kglo,, d)
2s cat ([sg, zsli, dim=1) # (B, M, d) where M M Mggl
sq_ ststs sum (zs. vie. M,, d) vig.view (8, (Mglo, d)) 2, dim-3) (8, M, M_glo)
zsg bdim = zsg. transpose (0, 1) # (M (M, B, d)
covs = zag bdim. transpose (1, 2) @ zsg bdim / & (M glo, d d)
e_ps batch logdet (I unsqueeze (0) d/ (eps**2) covs) (M gio))
loss = mean (istt gamma mean (R_pps)
bossbbacwward)) I back-propagate
student and teacher updates
update (fs) SGD or Adam
ft. itrams
1am * ft.params + (1 1am) * fs..sppa params
E. Implementation Details
The training codes and hyperparameters for SimDINO and SimDINOv2 are derived from the released official settings in DINO and DINOv2 separately, see Table for detailed comparison. Notes that for SimDINOv2, we choose to use bfloat 16 dtype in student backbone parameters and reductions for better numerical stability while other modules uses the same FSDP mixed precision settings from DINOv2.
F. Additional Experiments
E1. Ablations on Stability of DINO Training
In Table 6, we study the optimization behavior and stability of DINO by varying hyperparameters that are specific to its pipeline. Specifically, we select teacher momentum, whether to apply normalization for the last layer, and teacher temperature. We vary each of them and study their impact on DINO training. As shown in Table 6, moderate adjustments for each component leads to divergence (during early training stages). These results suggest DINO training can be highly unstable and requires careful tuning efforts.
F.2. Ablation Studies on Batch Sizes
We vary the batch sizes when training ViT-S using SimDINO and report the results in Table We observe that SimDINO robust the choice of batch sizes and can converge to reasonably good performance with smaller batch size of 256.
F.3. Experiments on Longer Training
More training epochs SSL typically lead to better performance. We provide the performance of SimDINO when doubling the number of epochs in Table Clearly, these results show the efficacy of longer training for SimDINO.
19

---

Simplifying DINO via Coding Rate Regularization
Algorithm 2 SimDINOv2 training pipeline.
[s, ft: student and teacher networks, this time outputting BOTH the cls token feature and patch token features eps: coding rate regularization quantization hyperparameter
gamma: ceding rate regularization strength hyperparameter
lam: teacher network EMA rate
alpha: proporticn of patches that get masked
st params = fs.params
for in Loader: # Load minibatch of samples
generate_mask (x, alpha) boolean mask (B, Mgglo, N_glo)
xg, x1 gglcba_ (x), vocal_views (x) (B, Mgllo, N_glo, D), (B, Mlooc, N_loc, D)
xmg apply_mase (xg, (B, M_glo, Nglo,
28g, Zsg = fs (xmg) student on masked global views (B, M_glo, d) (B, Mgll, N_glo, d)
2sl, Zsl = fs student output on local views (B, M_loc, d) (B, Mloc, N_loc, d)
ztg, 2tg (xg) teacher output on global views (B, M_glo, d) (B, Mgllo, N_glo, d)
2s cat ([zsg, zal], dim-1) # (R, M, d), M - Molc + M_glo
sq dists sum (zs. view MB, 1, d) ztg. view (B, 1,,1,M glo, d) 2, dim=3) (B, M_gio)
patch sq dists mean (sum (Zsg Ztg) ** 2, din=3) m, dim=2) (B, M glo)
2sg_bdim = zsg. transpose (0, 1) I (M_llo, B, d) covs = wwg.bdim.transppneooooooooooo (-2, -1) asg [sg_bdim 3 A (Mglo, d, d) unsqueeze d/ (eps. cevs) (Mgio)
R_pps batch_logdet
loss = (mean { mean (patch_ss_ /2 ganma mean (R_eps)
loss. backwadc)drd # back-propagate
student and teacher updates update (fs) SGD or Adam
ft farams 1am * parmss + (1 1am) * rs.params
20

---

Simplifying DINO via Coding Rate Regularization
F.4. DINO without Self-Distillan
Due to the explicit coding rate regularization, it is possible to train SimDINO without self-distillation. To validate this, train VIT-S models ImageNet-Ik 1k setting the teacher network to be the student network at each iteration, effectively removing the EMA operation. Results presented in Table 9. We see that the original DINO collapses under this setup for reasons discussed in Appendix B, while SimDINO is able to yield non-trivial performance. It is worth noting that compared to training with full self-distillation, this variant primarily lags behind in terms of k-NN performance while the gap in linear probe is significantly smaller.
E5. Ablations on Loss Functions
To prevent representation collapse, we adopt the coding rate objective R& in SimDINO and SimDINOv2. In this part, we examine the effectiveness of the coding rate formulation and compare it with other loss functions that aim to prevent collapse. Specifically, we swap the coding rate function with the following three widely-used objectives: (1) the vanilla contrastive loss feuntrastive based on InfoNCE in SimCLR (Chen et al., 2020), (2) the uniform loss Lumiiorm proposed in (Wang & Isola, 2020) that encourages the representations be uniform the unit sphere, (3) the Barlow Twins loss Ou proposed in (Zbontar et al., 2021) that penalizes off- off-diagonal terms while promoting the on-diagonal terms on the covariance matrix of learned representations. Results are presented in Table 5. We observe that the coding rate objective consistently performs better than the other choices considered, validating our design.
F.6. Visualization of Attention Maps
Following (Oquab et al., 2023; Caron et al., 2021), we provide visualizations of self-attention maps of different models for qualitative comparison. We use test images that do not appear during pretraining. More concretely, we compute and visualize the average of self-attention maps across all attention heads from the last layer in Figure 5. It is clear from the attention maps that all methods studied in our paper lead to prominent segmentation properties that emerge from vision sele-supe learning.
21

---

Simplifying DINO via Coding Rate Regularization
Image
DINO VIT-B/16
8/16
DINOV2 VIT-8/16
SImDINOV2 VIT-8/16
SimDINO
DINOV2 VIT-L/16
SimDINOv2 VIT-L/16
Figure 5. Visualization of average self-attention maps obtained from both DINO(v2) and SimDINO(v2) algorithms.
22

---

Simplifying DINO via Coding Rate Regularization
Hyperparameter
SimDINOv2
DINOv2
SimDINO
DINO
Patch size
16
Register tokens
Pos-embedding anti-alias
True
False
Model
Init layer scale
0.1
le-5
Drop path rate
0.3
0.1
Weight normalize last layer
removed
True
removed True
Output prototypes K
removed
65536
removed
65536
Init EMA momentum
0.9
0.992
0.996
Centering temperature
removed
0.07
removed
0.07
Warm-up temperature
removed
0.04
removed
0.04
Warm-up temperature epochs
removed
30
removed
30
Pipeline
iBOT sample prob.
0.5
iBOT mask ratio
0.1-0.5
iBOT head untying
False
Koleo loss weight
removed
0.1
Global crops scale
0.4 1
Local crops scale
0.05 0.4
Data
Local crops number
10
Global crops size
224
Local crops size
96
Batch size
128x8
64x8
Epochs
100
Warm-up epochs
10
Freeze last layer epochs
removed
removed
Optim.
Learning rate
0.004
0.002
Layerwise Ir decay
0.9
Weight decay
0.04
Weight decay end
0.4
Gradient clip
3.0
0.3
Table 4. Training hyperparameters used in the experiments
Objectives to prevent collapse
k-NN
Linear
contrastive loss Lcontmsttvve uniform loss Funiform Barlow Twins loss Ext coding rate loss Rg (ours)
79.4
81.0
77.2
82.1
81.1
82.4
Table Ablations Loss Functions. We evaluate VIT-L pretrained ImageNet- Ik for 100 epochs by swapping the coding rate function in SimDINOv2 with other choices of collapse-prevention objectives. Barlow Twins loss causes representation collapse in our experiments.
23

---

Simplifying DINO via Coding Rate Regularization
Table Sensitivity of DINO selected hyperparameters. We pick three DINO-specific hyperparameters (i.e. teacher momentum, last-layer head normalization, teacher temperature) of the official configuration (Caron et al., 2021) to study their impact. Varying each leads to divergence in early training.
Config
Mom
Norm.
Temp.
k-NN
official (400ep)
0.996
0.04 0.07
76.1
0.90
v
0.04
0.07
NaN
0.996
0.04 0.07
NaN
0.996
V
0.07
NaN
Batch size
256
512
1024
k-NN
68.3
69.7
69.6
Table 7. Effect of batch sizes. We evaluate k-NN accuracy of VIT-S pretrained on ImageNet- 1k for 100 epochs.
Method
Epochs
k--N
Linear
DINO
100
72.9
76.3
SimDINO
100
74.9
77.3
DINO
200
73.6
77.1
SimDINO
200
76.0
77.7
DINO
400
76.1
78.0
Table Efffect of training epochs. We evaluate ViT-B pretrained ImageNet-1k Ik for 100, 200, and 400 epochs. (DINO* 400 epochs evaluated on provided checkpoint)
Method
Model
self-distillation
Epochs
k-NN
Linear
DINO
ViT-S
X
100
SimDINO
VIT-S
X
100
58.6
68.0
SimDINO
VIT-S
100
69.7
73.6
Table 9, Performance ImageNet-1K 1K without self-distillation.
24


Reviews:
[{"review_id": "g4XxksFbb5", "reviewer": "Reviewer_S9wV", "summary": "The authors aim to improve the stability of self-supervised models DINO and DINOv2, while simplifying the training process. Collapse is a common problem in self-supervised learning (SSL), and it is usually avoided with careful hyperparameter tuning and architectural adaptations. In particular, to avoid collapse, DINO uses softmax temperature and centers the Teacher’s features, while DINOv2 adds KoLeo regularizer to make the span of features more uniform. The authors propose the following two main changes, (1) remove the heads, and replace cross-entropy loss with MSE on the $l_2$ normalized features of the encoder, (2) add rate distortion regularization to reduce the correlation between feature dimensions, which mitigates a constant output. \n\nThe authors experiment with ViT-S, B, and L models, which they pre-train on ImageNet-1K (IN-1K), and test on multiple downstream tasks, including classification, object detection, semantic segmentation, and video object segmentation. The proposed models, SimDINO and SimDINOv2, consistently outperform the DINO and DINOv2 baselines trained by the authors. They also show that hyperparameters from training ViT-B can be transferred to ViT-L, allowing for stable training. Finally, the authors provide ablations in the Appendix to study the effect of different hyperparameters and design choices, e.g., batch size.\n\n## update after rebuttal\nBecause of the discrepancy in the description of the DINO loss, which is even involved in derivations, I can not recommend the paper for publication. However, the authors clarified/addressed a number of comments from my initial review, so, I increased my score from 1 to 2.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer S9wV,\n\nThank you for the detailed review. Many comments, especially about presentation or typos, are helpful and will be incorporated in the camera-ready version; we do not address them here. Others require clarification or even stem from misunderstandings on your end, and we reply to these as follows.\n\n## Claims and Evidence\n\n> Training duration\n\nLonger training usually leads to better performance, which is demonstrated in Appendix F.3. SimDINO still outperforms DINO with more training epochs. We are limited by compute and cannot adopt the exact same configurations as the original DINO(v2). With that said, we have since trained a ViT-B SimDINO model for 400 epochs with better performance in $k$-NN accuracy (76.6) than the official DINO model (76.1). This is still unfair comparison since the official setting uses full fp32 precision while ours only uses fp16 due to memory constraints. We will include more such results in our final version.\n\n> Additional baselines\n\nPlease see our response to Reviewer ubYD (5-1).\n\n> Training stability\n\nIt may seem contradictory but is actually not. For example, checkpoints at 100-th epoch in a 100-epoch run are very different from those in an 800-epoch run, due to difference in learning rate, teacher momentum, etc., and are incomparable. We also notice a similar performance dip in DINO trained for 200 epochs. \n\n> Computational efficiency\n\nThe coding rate computation is efficient since covariance matrices are PSD. We will include quantitative analysis in the final version. \n\n## Methods and Evaluation Criteria\n\n> Definition of $\\mathbb{R}^{D \\times *}$\n\n$\\bigcup_{t = 1}^{\\infty}\\mathbb{R}^{d \\times t} = \\{x \\colon \\exists t\\ \\text{s.t.}\\ x \\in \\mathbb{R}^{d \\times t}\\}$ is the set of all finite-size matrices with $d$ rows. This is a standard definition for the symbol $\\bigcup_{t = 1}^{\\infty}$.\n\n> Definition of $\\mathbb{S}_{d - 1}$\n\n- The sphere in $\\mathbb{R}^{d}$ is a $(d - 1)$-dimensional submanifold of $\\mathbb{R}^{d}$. The definition including \"$\\mathbb{S}_{d - 1} \\subseteq \\mathbb{R}^{d}$\" disambiguates this notation.\n\n> Definition of $\\Theta$\n\nHere $\\Theta$ is the weight space. We will make this more explicit.\n\n> Multiple definitions of view \n\nAs is relatively common in SSL, we identify the function $v$ and the object $v(X)$ with each other for conceptual ease. We will make this more explicit.\n\n> Expectations over $X$\n\nThe expectation is over $X$; the variables $z, p$, etc., are functions of $X$. We plan to streamline this notation to show dependencies.\n\n> Typo in the loss definition\n\nThank you very much for pointing it out. We emphasize that **this is only a typographical error, and we use the official DINO(v2) losses in experiments**. We will fix this in the paper. However, it makes _very little_ difference in the main body and Appendix B, as the symmetry is not important for our argument.\n\n> Parameter $\\epsilon$\n\nIn this work, we only care that a smaller $\\epsilon$ makes the regularizer larger. However, for completeness we will add a short description of the information-theoretic role of $\\epsilon$.\n\n> MSE and similarity\n\nSince the features are normalized on the sphere, an MSE term _is_ a similarity term.\n\n> Covariance approximation \n\nThe provided argument applies only when the MSE term is small. Since we choose $\\gamma$ to balance the terms’ relative sizes, a well-trained SimDINO(v2) model _will_ have a small MSE term (induced by the feature alignment loss term), so the argument will apply.\n\n## Supplementary Material\n\n> Section name choice\n\nWe use this section label precisely because there is no self-distillation in this setting; one just tries to minimize the distance between two views' features, instead of taking one model as a privileged teacher from which the other network is distilled.\n\n## Experimental Designs and Analyses\n\n> Performance at scale\n\nOur experiments (up to ViT-L and on ImageNet-1k) use very popular/standard settings in vision SSL. They are by-no-means considered to be limited scale. We are bound by our compute resources, and leave further scaling-up of our approach as future work.\n\n> Teacher/student weights\n\nYour assertion is false; both DINO and our work use the teacher weights for evaluation. This is shown in the original DINO paper (Fig.6), where the teacher model consistently outperforms the student.\n\n> CNN backbones\n\nAs you point out, DINO(v2) perform strongly with ViT models, which are our focus as well. In particular, DINOv2 does not provide training on CNN backbones either. \n\n## Overall\n\nThanks again for your valuable feedback. We will fix many raised issues about prose and typos in the camera-ready version. We hope that our responses clarify the strength and correctness of our work. Given that we have attempted to address all of your main concerns and clarified several critical misconceptions of yours, we humbly request that you reconsider your recommendation for this work, as we believe it has good value to the community.", "claims_and_evidence": "The authors make two main claims, (1) SimDINO and SimDINOv2 allow for simpler, more robust, and more computationally efficient training compared to DINO and DINOv2, (2) learned representations are of higher quality because of performance on downstream tasks.\n\n- Performance: In the conducted experiments, the proposed models outperform the baselines across different downstream tasks, and different pre-training datasets. However, I see a number of issues with the experimental design:\n    - Training duration: In Table 1, all models are trained for 100 epochs, while in the DINO paper [1] they report performance after 800 epochs, and they train for 100 epochs only for ablations. Having said that, the proposed method may be able to outperform the baselines if all models are trained longer, but this should be verified experimentally, especially since SSL methods benefit from extended training.\n    - The provided experiments are adequate to show relative differences between DINO and SimDINO models, but additional baselines are required to be able to estimate the impact of the proposed method in the literature.\n- Simplicity:\n    - One of the experiments in support of the increased training stability is provided in Fig. 2, where SimDINO performance increases as training progresses, while DINO saturates and even experiences a decrease in accuracy. However, these results contradict the results reported in the original DINO paper, e.g., in Appendix Section D in [1], ViT-S improves its k-NN accuracy by a big margin as it is trained for 100, 300 and 800 epochs. This raises concerns about the training of the baselines, or about the adequacy of the training duration, since in Fig. 2 the models are trained only up to 100 epochs.\n    - The authors claim that training is more efficient, however, training compute is not reported. The proposed pipeline is indeed simplified because the heads are removed, but a new loss term is introduced, which requires non-trivial computations. So, I think any gains in training efficiency should be quantified.\n\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" *Proceedings of the IEEE/CVF international conference on computer vision*. 2021.", "methods_and_evaluation_criteria": "I think there are a number of issues with the description of the methods.\n\n- ln 92, col 1: $R^{D \\times *}$ represents the space of finite sequences, but in $U_{T=1}^{\\infty}$, $T$ goes to infinity.\n- What is the reason for defining a $d-1$ dim space instead of a $d$-dimensional one? For example, I could see how something like that would make sense for the sequence dimension $N$, since $N-1$ is the number pf patches, and there is an additional cls token for a total sequence length of $N$, but why for $d$? Also, in ln 133, col 1, the heads are defined as functions $R^d \\rightarrow R^m$, but for their inputs $z$, in ln 166, col 2, we have $z \\in S_{d-1}$.\n- ln 99: $\\Theta$ is not defined.\n- ln 101: In the definition of $f_{\\theta}^{\\text{cls}}$, I guess it should be $S_{d-1}$ instead of $S^{d-1}$.\n- ln 113-116, col 1: I think it would be good to make a distinction between a transformation that augments an image, and the output of such a transformation for a given image, which can be called a view. In ln 92-93, col 2, the term “view” is treated as the output of a transformation, i.e., “the feature corresponding to any view”, and in ln 113-116, col 1, as the transformation itself. The same in ln 121, col 1, where $X_c$ is called a “view”, while in the paragraph before, $v_c$ is defined as a “view””, and $X_c = v_c(X)$. SimCLR [2] has simple and clean notation for this issue.\n- Eq. 2, 3: The temperature values should be different, because they are not assumed the same in DINO [1].\n- ln 112, col 2: It is mentioned “the expectation is over $X$”, while there is no $X$ in Eq. 4.\n- If I am not mistaken, the definition of DINO loss in Eq. 4, 5, 6 is incorrect. As can be seen in Algorithm 1 in [1], loss = H(t1, s2)/2 + H(t2, s1)/2 which is equivalent to $1/2 (CE(p_t(t_1), p_s(s_2)) + CE(p_t(t_2), p_s(s_1)))$, while based on Eq. 5 should hold $p_t(t_1) = p_s(s_1)$ and $p_s(s_2) = p_t(t_2)$, which is not true.\n    - This affects derivations in Appendix B, where Eq. 5 is used.\n    - Is this the loss used for DINO in the experiments?\n- Fig 1: In (a), it seems that only the teacher weights are updated through EMA, but this is true for the head weights as well.\n- Eq. 8 is not adequately explained. For example, in [3], which is cited by the authors, it is explained that “rate distortion $R(z, \\epsilon)$ is the minimal number of binary bits needed to encode $z$ such that the expected decoding error is less than $\\epsilon$”. This is the core contribution of the work, and the authors introduce $\\epsilon$ as just a hyperparameter, without explaining its meaning.\n- Footnote 3, starts at ln 215, col 2:\n    - The authors refer to a similarity term in Eq. 9. I guess they refer to the 1st term, however it is an MSE term, not similarity term.\n    - Their argument makes sense when the MSE is small, however, what if it is large?\n- ln 245-246, col 1: $Z^{\\text{patch}}$ and $P^{\\text{patch}}$ are mentioned, e.g., “$i^{\\text{th}}$ column of $Z^{\\text{patch}}$”, before they are defined.\n\n[2] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" *International conference on machine learning*. PmLR, 2020.\n\n[3] Yu, Yaodong, et al. \"Learning diverse and discriminative representations via the principle of maximal coding rate reduction.\" *Advances in neural information processing systems* 33 (2020): 9422-9434.", "theoretical_claims": "The authors provide theoretical results in Appendix Section B and C. I commented about Section B in the previous section. I went through the proof in Section C, but not in enough detail to be absolutely certain that is correct.", "experimental_designs_or_analyses": "- One of the main contributions of DINOv2 paper is performance at scale, both in terms of model and training data size. Based on that, the scale of the conducted experiments doesn’t allow for direct comparisons with the main claims of the DINOv2 paper.\n- After pre-training, the authors keep the Teacher model and discard the Student (ln 188-189, col 2), however, in DINO paper the opposite holds true. Which model is kept for the reported baselines? Are there any noticeable differences in the performance of the Teacher and Student models?\n- It would be interesting to have experiments with CNN backbones, similar to the DINO paper, to see the effect of the proposed regularizer, however, I don’t think this is a must-have, since DINO models are reported to excel with Transformer backbones.", "relation_to_broader_scientific_literature": "The proposed method builds directly on the DINO line of models, and offers increased stability and simplicity to the training process by introducing a rate distortion regularizer. As I mentioned earlier, this regularization term aims to reduce the correlation between feature dimensions, which is an idea already explored in SSL [4]. As a result, the proposed method is a combination of existing ideas in a modern setting, which I think has value, since SSL is an important research direction.\n\n[4] Zbontar, Jure, et al. \"Barlow twins: Self-supervised learning via redundancy reduction.\" *International conference on machine learning*. PMLR, 2021.", "essential_references_not_discussed": "There are references that could be added, e.g., [4], but in general, I think the authors provide adequate references.", "other_strengths_and_weaknesses": "No additional comments.", "other_comments_or_suggestions": "There are some minor typos, for example:\n\n- ln 163, col 1: “receives a interpolated”, should be “an”.\n- ln 112, col 1: “During the pipeline”, this phrase doesn’t seem right.\n- ln 113-116, col 1: It is mentioned “we sample at random a view”, and then “selected randomly”, which seems unnecessary.", "questions_for_authors": "No additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Pq5t8BXa4", "reviewer": "Reviewer_nrJd", "summary": "The work aims to simplify the architectures of DINO and DINOv2 family of models by adding an explicit coding rate term in their loss. The simplified architectures (referred as SimDINO and SimDINOv2) are more robust to different design choices and offer a Pareto improvement over the DINO model families. The simplification, if effective, could be useful for the larger field of AI and encourage more computationally efficient approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nrJd,\n\nThank you for your insightful comments and suggestions.  We are encouraged by your compliments on the simplification of the proposed method as well as the empirical evidence and presentation quality of our work. Here we attempt to address the concerns raised in your review.\n\n> Including more variants of regularizers and coding rate terms\n\nWe appreciate your suggestion. First, we use coding rate as the anti-collapse loss here due to its simplicity and theoretical grounding (also refer to our response to Reviewer 5gcy for further theoretical justification). In terms of its computation, we would like to clarify that there do not exist many variants (i.e. all the cited works compute the exact same quantity). Other than coding rate, we agree it is possible to prevent collapse with other choices of regularizers and is an interesting direction to explore. We have experimented with directly using contrastive loss, but the results lag behind our choice. We will include these ablations in the camera-ready version of the paper.\n\n> Other variants of DINO and DINOv2 pipelines\n\nTo the best of our knowledge, there aren’t any major variants of DINO and DINOv2 pipelines. Generally people either directly use the pretrained checkpoints or rely on the official implementations. We believe this can also be largely attributed to the complexity and fragility of the DINO pipeline, as we have demonstrated in Table 5 in the appendix. To compare our methods with the official pipeline, Figure 1 directly compares and summarizes the relevant changes.\n\n> Some issues on presentation\n\nThank you for your comments. Regarding the use of \"pareto improvement\", we respectfully assert that a \"Pareto improvement\" is when the whole Pareto curve is moved forward (i.e., when improvements are gained without any cost). You may have been thinking of moving along the Pareto frontier, which involves tradeoffs. For \"vision SSL\" (page 2, line 88), we want to suggest that simplification can be very valuable to the larger field of vision SSL, and our work pushes this broader envelope. Rather than adding more tricks and complexities, we show that identifying the necessary components (i.e. feature alignment while avoiding collapse) naturally leads to a greatly simplified design and improved performance. We believe such practices are useful for the larger field of vision SSL. If this causes confusion, we are happy to revise it. We hope these clarifications can resolve your concerns.\n\nWe again thank you for your time and effort in evaluating our work, which will no doubt improve its quality. We welcome your feedback and will be happy to engage in further discussion.", "claims_and_evidence": "The contribution of this work can be briefly summarized as: methods to reduce complexity of DINO and DINOv2 families by reducing complexities in training pipelines and adding a coding error rate regularizer to enable simple, robust and efficient training pipelines which result in an improvement performance in vision SSL.", "methods_and_evaluation_criteria": "- In Figure 1, it is stated that the simplification lies in removal of a linear layer and a softmax layer in DINO, and in the removal of the same layers along with one component of loss in DINOv2.\n- The DINO pipeline is recapped (this is good for the readers) to highlight z^{cls}_c and z^{cls}_g obtained from local (student network) and global views (teacher network). It is argued that these features can be compared directly. However, the post-processing steps that are applied to prevent collapse result in a Loss highlighted in Equation 4. The complexities in updates of student and teacher parameters are further highlighted. It is explained how the collapse is avoided during EMA update while mentioned that the exact mechanism is not clear. \n- The two main questions addressed are 'how to use z^{cls}_c and z^{cls}_g directly' and 'how to enforce non-collapse by efficient use of negative samples?'. The two questions, if addressed, could lead to removal of linear and softmax layers along with EMA update structure.\nEquation (9) is proposed as a solution (for DINO) here which applies squared Euclidean distance and a rate distortion term with a hyperparameter γ and similarly Equation (17) has been proposed for DINOv2. Guidance on the hyperparameter γ has also been provided.\n- The overall problem is well posed and the problem is formulated here. \n- For evaluation, The ImageNet-1K classification dataset has been used for all methods for ViT backbones pretrained on DINO and SimDINO variants and COCOval2017, ADE20K and DAVIS-2017 for downstream detection and segmentation tasks.", "theoretical_claims": "Theoretical details have been listed for hyperparameter tuning in the Appendix C provided specifically to justify the choice of γ. I have not carefully verified the proof but the steps are consistent and I have not noticed any discrepancies in a brief look.", "experimental_designs_or_analyses": "- Experiments compare the DINO approaches with their SimDINO alternatives on\n    - Classification: Improvements in the performance have been highlighted in Table 1.\n    - Object Detection and Segmentation: Tables 2 and 3 have highlighted the performance improvements for SimDINO. \n- Segmentation maps for DINO and SimDINO in the supplementary material are comparable.", "relation_to_broader_scientific_literature": "DINO families have been introduced with the background of contrastive SSLs and the issue of representation collapse is discussed in the context of contrastive SSL.  The use of negative samples (directly) in DINOv2 and (indirectly) in DINO models is highlighted along with the complexity and instability of the models along with the requirement of 'many tweaks' and careful hyperparameter selection for convergence.  The references to the corresponding sections  and/or papers should be cited while mentioning the issues with complexity of these models and the claim that all these are required for convergence. For instance, the appendix F.1 highlights some ablation studies on the stability of DINO training could be referred, etc. This work removes the 'many tweaks and hyperparameters' (figures, discussion could be referred accordingly) from DINO families by adding a coding error rate regularizer in the optimization.", "essential_references_not_discussed": "I am not aware of any missing essential references.", "other_strengths_and_weaknesses": "The manuscript is well-written and maintains high standards for readability. The problem and framework are well posed. Experiments have been conducted to highlight the value of proposed solutions. The design designs could potentially be further elaborated on by extending the experiments to include more variants of regularizers (including choices for coding rate terms), methods for improving efficiency, etc. This would help in establishing the value of proposed solution over the existing choices which could motivate the users for adoption.", "other_comments_or_suggestions": "- The last part 'vision SSL' (page 2, line 88) could perhaps be replaced by DINO and DINOv2 families or more details can be provided on how it would apply to the larger vision SSL field.\n- The coding rate regularisation has been significantly used in the literature as mentioned in the Subection 'Coding rate, and related regularizers'. Since, the coding rate term (in Equation 9, 17) is central to the contribution, it would be good to compare variants of coding rate terms in literature and how they differ in the implementation, evaluation, etc.\n- The improvements have been proposed over the existing DINO and DINOv2 pipelines. It would be good to summarize all the framework, training style variants of DINO and DINOv2 families which differ from the standard frameworks chosen for comparison. The contribution can be split into efficiency and improvement. Both of these are valuable. However, discussion and extension of experiments on design choices could strengthen the work. The literature and experiments could potentially be improved to include other common strategies that avoid representation collapse and how the proposed strategy is not only more efficient at avoiding the collapse but also more effective.\n- Discussion on tradeoffs and efficiency could be provided when using alternate strategies for representation collapse.", "questions_for_authors": "- The use of 'pareto improvement' term in the abstract is a slightly confusing to me. It is generally used if the improvement is coming at a cost of something (like a decrease) else (based on the Pareto definitions). In this work, I see that efficiency and performance are argued to improve together.\n- I have shared my comments in other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e92EumdmZm", "reviewer": "Reviewer_5gcy", "summary": "The paper reveals that many comonents in the pipelines and models of DINO and DINOv2 are used to avoid collapse of the learned representations, and thus attempts to simplify the training pipelines, architectures, and hyperparameters of DINO and DINOv2 by introducing a $\\ell^2$ distance based on the loss function plus a total coding rate based regularization to replace the cross entropy based loss function.  Such a simplification leads to more simple, efficient, and robust training pipelines and models---SimDINO and SimDINOv2, without need to use several manually designed components in models and tricks in training pipelines.  Extensive experiments demonstrate that SimDINO and SimDINOv2 outperform the original DINO and DINOv2 on the downstream tasks, including image classification, object detection and segmentation, semantic segmentation, and video object segmentation.  Moreover, the experimental results also demonstrate the stability and robustness against hyper-parameters.\n\n## update after rebuttal: \nThe reviewer read the responses from the authors and the other review comments, and believes that this submission is an excellent work and should be accepted.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5gcy,\n\nWe deeply appreciate your insightful review and am pleased to see that you find our work backed by solid empirical studies, with potential to improve practice of visual representation learning. Here we attempt to address the concerns you posed.\n\n> Theoretical justification for coding rate regularization\n\nThanks for your suggestion. It actually can be shown that when coding rate is maximized, it will induce the feature $Z$ to be full-rank. This can be proved by a spectral analysis as $\\log\\det(I+ Z^\\top Z) = \\sum_{i=1}^{\\min(n,d)}\\log(1+\\sigma_i^2)$, where $\\sigma_i$ is the $i$-th singular value of $Z$. Maximizing coding rate then leads to uniform non-zero singular values (as columns of $Z$ are $\\ell^2$ normalized), making $Z$ full-rank. This means the learned features are spread on the unit sphere, avoiding a collapsed solution. On the other hand, the $\\ell^2$ distance encourages the student features to be close to the teacher features on the same image, ensuring that we do not arbitrarily project different crops of the same image to large distances in the feature space. This is the extent of our added theoretical analysis for now; note that a full optimization theory-esque landscape analysis is very challenging and/or unreflective of practice due to the self-distillation (teacher/student) aspect, and we leave such analysis for future work. We hope these clarifications alleviate your concerns and we will put the relevant theoretical analysis in the final version of the paper.\n\n> The connection between MSE and dot product similarity\n\nThank you for pointing out this minor algebraic error. Note that in in eq.(7) we have included a factor of $\\frac{1}{2}$, so the correct additive constant is +1. We will amend this in the final version of the paper.\n\nWe greatly appreciate your time in evaluating our work and your suggestions will certainly help improve the quality of our work. Please let us know if you have more questions or concerns and we are happy to address them.", "claims_and_evidence": "The claims made in the paper is supported by extensive experiments on the downstream tasks, including image classification, object detection and segmentation, semantic segmentation, and video object segmentation.  Moreover, there is also partial theoretical analysis in the appendix to  reaveal the possible reasons of the training instability in DINO.", "methods_and_evaluation_criteria": "Yes. Simplifying DINO and DINOv2 does make a lot of sense for wide applications of unsupervised representation learning.", "theoretical_claims": "Yes, the proofs in the appendix are correct.", "experimental_designs_or_analyses": "Yes. The experimental designs are fair and sound.", "relation_to_broader_scientific_literature": "The papers simplified two very popular methods DINO and DINOv2 in unsupervised learning of visual representation and thus proposed two promising models: SimDINO and SimDINOv2. Since that DINO and DINOv2 are widely applied in visual representation learning and have broad connections to deep learning literature, the reviewer believe that the simplified models again have signifiant potential to advance the practice of visual representation learning.", "essential_references_not_discussed": "Essential references are discussed.", "other_strengths_and_weaknesses": "Strengths: \n+ Extensive experiments are provided to demonstrate the SimDINO and SimDINOv2 outperform the original DINO and DINOv2 on image classification, object detection and segmentation, semantic segmentation, and video object segmentation.  \n+ Experimental results also confirm the stability and robustness of  SimDINO and SimDINOv2 against hyper-parameters. \n\nWeaknesses:\n- There is no theoretical justification on the mechanism why adding the total coding rate maximization term leads to avoid the feature collapse. For example, what is the landscape of the optimal solutions when using the simplified loss function based on $\\ell_2$ distance and the rate reduction regularization in SimDINO?", "other_comments_or_suggestions": "At L166 in right column: \"Note that $d_{\\ell^2}(z_c^{cls}, z_g^{cls}) = - (z_c^{cls})^\\top z_g^{cls}$ since ....\". This is not correct----a constant 2 is missing.", "questions_for_authors": "Regarding to the mechanism of adding a total coding rate maximization term to avoid the collapse, is there possible to have some theoretical results? For example, what is the landscape of the optimal solutions when using the simplified loss function based on $\\ell_2$ distance and the rate reduction regularization in SimDINO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxsHlJUmHF", "reviewer": "Reviewer_ubYD", "summary": "The authors proposed the coding rate regularization technique to improve the training method of DINO and DINOv2, enhancing their performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ubYD,\n\nWe deeply appreciate your insightful review and am pleased to see that you find our work intuitive to follow, and well-justified by extensive empirical studies. Here we attempt to address the concerns you have raised in the review.\n\n3-1: This is a great suggestion. It is possible to apply coding rate to other representation learning methods, as it is a principled measure of non-collapse (we provide a sketch of theoretical justification in our response for Reviewer 5gcy). As you rightfully point out, we focus on simplifying DINO and DINOv2 in this work as they are widely used and attain SOTA performance. The study of extending the coding rate to other learning frameworks is interesting, but out of scope for this paper and we leave it for future work.\n\n5-1: Thanks for your comment. As discussed in the previous point, this work focuses on the DINO family of models as they are very widely used and typically best-performing across various downstream tasks. In ImageNet classification results, we also have included additional baselines (e.g. MoCov3) for reference. With that said, we are happy to incorporate more baselines in the final version of the paper.\n\n5-2: We would like to respectfully note that few-shot learning is not really necessary here, since we already obtain linearly separable features from DINO/SimDINO training (demonstrated by the superior $k$-NN performance). Due to this property, few-shot learning was not evaluated in the original DINOv2 paper. We hope this helps clarify and resolve your concern.\n\n10-W1: Thank you for your question. We can attach training plots in the final version of the paper. As long as the balancing coefficient $\\gamma$ is selected appropriately, the coding rate value will steadily increase, meaning that the features are spread out without collapse. If it’s too small, then feature alignment loss will dominate and lead to collapse (coding rate approaches 0). If it’s too large, the opposite happens, where the coding rate dominates. In this case, the features don’t collapse but feature alignment between global and local views is no longer enforced. We will follow your suggestion and put these analyses in the paper.\n\n10-W2: We appreciate your suggestion. In our preliminary test, using regularization such as direct contrastive loss leads to worse performance than coding rate regularization. We will run and include additional experiments with other regularization methods in the revised version of the paper.\n\nWe are grateful to your suggestions and comments, which will no doubt improve the quality of our work. Please let us know if you have further questions or concerns, and we are happy to address them.", "claims_and_evidence": "2-1. The careful review of the original DINO and DINOv2 provides an intuitive and well-explained reasoning for why collapse occurs.\n\n2-2. It is impressive that a method for effectively selecting the necessary hyperparameters for the proposed regularization is presented, along with a well-supported justification.", "methods_and_evaluation_criteria": "3-1. As stated in the title, it is clear that simplifying DINO is the main focus of the paper. However, the proposed coding rate regularization seems applicable to other representation learning methods that use contrastive learning. Are there any experimental results demonstrating this?", "theoretical_claims": "4-1. The proof provided in the Appendix has been reviewed, and no issues were found.", "experimental_designs_or_analyses": "5-1. The intention to improve upon DINO and DINOv2 is clear, but why is there no comparison with other self-supervised learning methods?\n\n5-2. Is there a reason why few-shot learning performance, which is commonly evaluated in most representation learning papers, is not included?", "relation_to_broader_scientific_literature": "7-1. Self-supervised representation learning is an actively researched field. While various methods have been proposed, representation collapse remains a critical issue that needs to be addressed. This work provides a simple yet effective solution to this problem.", "essential_references_not_discussed": "8-1. There do not appear to be any missing key references.", "other_strengths_and_weaknesses": "10W-1. How does the actual regularization value change? Is there an analysis comparing cases where collapse occurs and where it does not?\n\n10W-2. Is there a performance comparison when applying other regularization methods?", "other_comments_or_suggestions": "I have no further comments.", "questions_for_authors": "Please refer to the comments provided in Questions 2-11.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents SimDINO and SimDINOv2, simplified variants of the widely used self-supervised learning methods DINO and DINOv2, by introducing a coding rate regularization term to replace complex collapse-prevention mechanisms. The paper is well-motivated and makes a strong case for reducing architectural and training complexity while achieving better or comparable performance across a range of downstream tasks.

Strengths:
* clear motivation and contribution: the authors propose a principled simplification of DINO/DINOv2 via a theoretically grounded regularizer, yielding improved stability and performance.
* extensive empirical validation: evaluations across classification, detection, and segmentation tasks, including ablations and comparisons to alternate regularizers, support the claims.
* robust rebuttal: the authors proactively addressed reviewers’ concerns (e.g., experimental scale, baseline comparisons, and loss formulation discrepancies) and provided new results, including comparative tests against SimCLR, uniformity, and Barlow Twins losses.

Some concerns:
* reviewer S9wV raised concerns about loss notation, derivations, and experimental scale. While the authors provided clarifications, the reviewer remained unconvinced despite acknowledging the paper’s direction is valuable.
* reviewer ubYD appreciated the idea but found the evidence around the proposed regularization insufficient, though authors provided new experiments and comparisons in response.

Despite two reviewers maintaining weak rejects, their critiques center around presentation and experimental thoroughness rather than flaws in core methodology. The authors’ detailed rebuttals and new experiments significantly mitigate these concerns. This paper delivers a well-executed, impactful simplification of an important line of SSL work with promising theoretical and empirical support. The proposed simplification is practical, effective, and of clear interest to the self-supervised learning community.
Decision: Accept (poster)