DINO as a von Mises-Fisher mixture model
Authors: Hariprasath Govindarajan
Per Sidén
Jacob Roll
Fredrik Lindsten
Venue: ICLR 2023 notable top 25% (2023)
Keywords: self-supervised learning
vision transformers
mixture models

Abstract: Self-distillation methods using Siamese networks are popular for self-supervised pre-training. DINO is one such method based on a cross-entropy loss between $K$-dimensional probability vectors, obtained by applying a softmax function to the dot product between representations and learnt prototypes. Given the fact that the learned representations are $L^2$-normalized, we show that DINO and its derivatives, such as iBOT, can be interpreted as a mixture model of von Mises-Fisher components. With this interpretation, DINO assumes equal precision for all components when the prototypes are also $L^2$-normalized. Using this insight we propose DINO-vMF, that adds appropriate normalization constants when computing the cluster assignment probabilities. Unlike DINO, DINO-vMF is stable also for the larger ViT-Base model with unnormalized prototypes. We show that the added flexibility of the mixture model is beneficial in terms of better image representations. The DINO-vMF pre-trained model consistently performs better than DINO on a range of downstream tasks. We obtain similar improvements for iBOT-vMF vs iBOT and thereby show the relevance of our proposed modification also for other methods derived from DINO.

Published as a conference paper at ICLR 2023
DINO AS A VON MISES-FISHER MIXTURE MODEL
Hariprasath Govindarajan 1.2 Per Sidén 1.2 Jacob Roll? Fredrik Lindsten
'Linköping University, Sweden
Qualcomm Technologies, Inc.
[hargov, seirgo,,psiden, jroll/ 1 eqti. . qualcomm. com, fredrik. wiidstennniinnn se
ABSTRACT
Self-distillation methods using Siamese networks are popular for self-super pre-training. DINO is one such method based on a cross-entropy loss between K K-dimensional probability vectors, obtained by applying a softmax function to the dot product between representations and learnt prototypes. Given the fact that the learned representations are n.2norrallzeed we show that DINO and its derivatives, such as iBOT, can be interpreted as a mixture model of von Mises--ishee compo- nents. With this interpretation. DINO assumes equal precision for all components when the prototypes are also n.-norralizee. Using this insight we propose DINO- vMF, that adds appropriate normalization constants when computing the cluster assignment probabilities. Unlike DINO, DINO-vMF is stable also for the larger ViT-Base model with unnormalized prototypes. We show that the added flexibility of the mixture model is beneficial in terms of better image representations. The DINO-vMF pre-trained model consistently performs better than DINO on a range of downstream tasks. We obtain similar improvements for iBOT-vMF vs iBOT and thereby show the relevance of our proposed modification also for other methods derived from DINO.
INTRODUCTION
Self-supervised learning (SSL) is an effective approach for pre-training models on large unlabeled datasets. The main objective of SSL pre-training to learn representations that are transferable to range of, so called, downstream tasks. Early SSL methods achieved this through handcrafted pretext tasks that act as inductive biases the representation learning process (Komodakis Gidaris 2018
Noroozi & Favaro] 2016 Kim et al.) [018]] Doersch et al., 2015 Larsson et (0 et al.
2016). Contrastive methods (Tian al., 2020: Wu et al. 2018 using supervisory signals in the form of augmentation invariances and instance discrimination, have produced strong performance benchmarks. In practice, contrastive methods require large batch sizes (Chen et al. 2020a) or
specialized techniques like memory banks (He et al., 2020 Misra & Maaten 2020) to achieve the best performance. Self-distillation methods based on the Mean Teacher (Tarvainen & Valpola 2017) framework are effective representation learners that do not require such large batch sizes for pre-training. A trivial solution where the network learns to output the same representation irrespective of the input is known as representation collapse. The negative samples in contrastive learning prevent representation collapse. In the absence of negative samples, self-distillation methods use explicit
approaches to avoid collapse, such as asymmetric model architecture (Grill al. 2020
Chen & He
2021) and whitening (Ernolov et al. 2021 Zbontar et al. 2021).
Transformers /Vaswani al. 2017 originally introduced NLP, have emerged as a strong model
architecture for vision tasks as well Dosovitskiy et al.) 2220: Liu et al.
2021). Current state-of-the-an
SSL methods leverage the highly flexible Vision Transformers (ViTs) (Caron et al. 2021 Bao et al.
2021 Li et al. 2021 Zhou et al. 2021; Xie et al. [021] Chen et al.
2021). DINO (Caron et al.
1021 is a non-contrastive SSL method that is effective for pre-training ViT models. Interestingly,
ViTs pre-trained using DINO outperformed ResNets (He et al. (016) by significant margin at kNN classification based on learned representations. DINO is an influential SSL method with several -tate-of-the-art derivatives: MSN (Assran et al.] 2022) adapts DINO to produce strong sew- performance with enhanced training efficiency; iBOT (Zhou et al. 202T) extends DINO by adding an additional masked image modeling task; Es VTT extends DINO by adding patch-level tasks that also

---

Published as a conference paper at ICLR 2023
DINO-like formulation. These methods are all trained using the Mean Teacher framework by learning to produce consistent outputs in the probability simplex. The networks output softmax-logit scores based on an inner product between learned representation and set of prototypes.
We provide a better understanding of DINO, and its derivatives, by taking a closer look at its Mises-Fisher mixture model under certain
inner-produet formulation. We interpret DINO assumptions. Based on this interpretation, we propose DINO-vMF, as a modified version of DINO, that adds flexibility in the learned latent space while keeping the training stable. DINO-vME pre- training consistently improves performance on similar downstream tasks as DINO. We also show that the larger ViT models achieve significantly improved few-shot classification performance with our pre- training. By incorporating our vMF modification in iBOT, we achieve significantly improved performance that suggests that our method is applicable to DINO-derived methods as well.
DINO
The oeff-distillation learning framework in DINO considers teacher network go, and a student network ge,, with parameters 0, and f.. respectively. In DINO, the student network is formulated to predict vector in the (K" 1) --dimensional probability simplex using softmax function. The student probability distribution is obtained as follows:
(x) & exp 90, (xx
(k)
where & indicates that the right-hand side is normalized w.r.t. the index & (i.e. the equation above corresponds to a softmax). The teacher probability distribution P;(xr) is computed analogously.
Given an unlabeled image dataset I, consider uniform samples æ ~ I and two random augmentations Ag As: A, ~ Ar- By applying these augmentations, we get two views æ; = A.(x)) and a; = (xx). The student network is trained using gradient updates produce outputs P=(x,) that are consistent with those of the teacher network (xz) by minimizing a cross cross-entropy loss given by: ming, k== (x( log ((A) (xz). The teacher network parameters 0, are only updated as an exponential moving average (EMA) of 0g.
SSL methods based on Siamese networks (Bromley et al. 1993) face the representation collapse problem, where the network learns to produce the same output irrespective of the input. One approach to address this is by introducing an asymmetry between the teacher and student networks. DINO uses the same model architecture for the student and teacher networks but instead shows that adding asymmetry through centering and sharpening operations sufficient avoid collapse. The targets produced by the teacher network are centered to remove bias towards a cluster and sharpened using a temperature T₁ as go, (xx) + (ga, (xx) - c)/Te TT- On the other hand, the student outputs are only sharpened as ge, (x,) + ge. (/e.)TT. The centering operation prevents one of the probability components from dominating but the solution could collapse to uniform distribution instead. This is avoided by the sharpening operation, where the teacher uses a lower temperature value than the
student, Tt < T. 0.1. The overall schematic of DINO is illustrated Figure Ia
METHOD
3.1
DINO: A CLOSER LOOK AT THE FINAL LAYER
The teacher and student networks are designed by combining a backbone network that can be transferred to downstream tasks and a prediction head that is specific to the SSL pre-treiinnn trainining. We take closer look at the prediction head as shown in Figure [b] which is found to be important to achieve good performance in ablation experiments (Caron et al. 202T). The weight-normalization in the last linear layer (Salimans & Kingmal 2016) refers to reparameterization of the weights W as w = (()) where (5) is the keth column of W, (*)) is scalar magnitude, and ((()) is unit vector. Optionally, the weights are 2 normalized by fixing = 1.
Whether not to L n'onomzalizee the prototypes is subtle, and in our opinion ad hoc, design choice which is not discussed enough in the literature, Most prior clustering- SSL methods do use
normalized prototypes (Caron et al. 2018 2019 2020 [i al. 2020 Asano al. 2020), and this
2

---

Published as a conference paper at ICLR 2023
((x)) (axz) log (z,)
go(at)
Weight-normaizee layer (Optionallyy normalized
B
P. (A
Prediction head how
sg
softmax
softmax
B 256
sharpening Ta
sharpening
centering
B 256
EMA
3-layer MLP
student (6,(-)
teacher (-)(-)
B 384
fo(-)
Backbone
Z1
B x x 224 224
Figure 1: Overview of DINO. (a): High-level architecture of DINO; (b): A closer look at the networks go, modeled as a combination of backbone fa and a prediction head hg.w, where { v/, W). The prediction head contains MLP layers, an L'=normalizaatin bottleneck and a weight-normalized (Salimans & Kingma, 2016) linear layer. The weights of the weight- verght-normalized linear layer are L L'-nomalizeed the larger ViT-Base models to ensure stable training.
not mentioned as a design choice the paper by Caron al. (202T), nor their supplementary material. However, from the public code repository of DINC it can be noted that not L'-normalizing W leads to a performance boost for ViT-Small models (these are the results reported in their paper). However, the larger ViT-Base models require the weights to be 7,2 I.2-noraalzeed since the training is unstable otherwise. Furthermore, comparing the ViT-Small and ViT-Base models at patch size surprisingly the ViT-Small model achieves better kNN top-1 validation accuracy ImageNet (Caron et al., 2 Table 2). This suggests that L=-normalization is an important design choice and that the representations learned by the ViT-Base model (i.e. with (²=normalization) are sub-optimal and has room to improve. In the next section we provide better understanding of this from mixture model perspective and investigate how we can better leverage prototypes that are L -o-ommalleedd.
3.2 DINO A MIXTURE MODEL
DINO learns predict soft distribution over K components. Let the intermediate output in the prediction head after the [2=normalization of the representationf] be denoted as yt see Figure Ib Then, DINO can be interpreted as performing clustering in the latent space of Specifically, since a unit vector, DINO performs clustering on the unit hypersphere.
Clustering is closely related mixture modeling. On the unit hypersphere, mixture model can be defined using von Mises-Fishhe Fisher (vMF) components. For a random p-dimensional unit vector y, the vMF probability density function is given by f(y: μ4, x)) Cp(K) exp(141" T y), where μ is mean vector with ||ب = 1, Æ is a scalar concentration parameter that measures isotropic precision, and (x) is a normalizing constant defined as: (,) = 2/2-1 (<(())))=////(((2>>>>> /[(2m)p/21p 4/ (x)], where I, denotes the modified Bessel function of the first kind and order v. Assuming mixture model containing K vMF components with mixture component ratios (8), the probability of assigning sample yı to mixture component i (also known as the responsibility cluster k) is given by:
(k) n (b) Pr(ys 2, k) (4) Cpp((()) exp (ma(*) (k) .yx))
where (u, v) = u" denotes inner product.
"https///ithub... httpp://ggthhh.c///tcooooooeeee com/fucchwokressecchhiin
Note that this is different from the j.mmoormaizzatio of the prototypes.
3

---

Published as a conference paper at ICLR 2023
In DINO, the probability distributions for the student and the teacher can be rewritten as:
(x)) x exp
(tw! yyy)) /yaa/Ta]
(xx) (xx) & exp ( (ww) ye) - ch)) /m| exp [-c exp
/To]
(w) (k)
/////1 (4)
Note that centering is only applied on the teacher outputs and that different sharpening temperatures are used for the teacher and the student. By comparing Eq. (3) with Eq. (2). we observe that the DINO formulation resembles that of a mixture model, given some assumptions. We explain these
assumptions below to establish a more concrete connection. This interpretation also applies to other
recent works that use similar inner- prueer--rouuc-bseeddd prototype formulation (Caron et al. 2020: Assran ct al. 2022 Li et al. 2021 2020).
The exponential term in DINO from Eq. corresponds to the unnormalized probability density of a vMF distribution if we identify x(()) |//" and (x(*) = w w ||. Similar to prior clustering- SSL works, DINO avoids collapse by encouraging a uniform distribution of the data over the prototypes, which explains the absence of T(k) in Eq. (3). In the teacher model, Eq. 4 we obtain an additional term through the centering operation which further encourages uniformity. We discuss more about the centering operation in section 3.4
Following the mixture model interpretation, we note that missing term in the DINO formulation is the normalization constant, Cpp ()))))) However, this is inconsequential when the prototypes (()) are ne-normalized as it will result in constant n²(R) 1/T. Hence, the constant Cp (1) would vanish in the softmax. However, constant x**) implies the assumption that all clusters should have a similar isotropic precision. This constraint reduces the flexibility of the mixture model, and this lack of flexibility in the latent space translates the backbone mapping well. Indeed, as discussed above, unnormalized prototypes help in improving the performance achieved by DINO (for ViT-Small models) by enabling better representations (Caron al. 2021). However, larger ViT-Base models affected by training instabilities when trained with unnormalized prototypes. We hypothesise that this due to the "missing" normalization constant and propose to modify DINO by including appropriate normalization according to a vMF mixture model.
3.3 NORMALIZING VON MISES- -ISSER (VMF) COMPONENTS
In DINO, a large [| || scales the logit scores of mixture component proportionally. In a properly normalized formulation, a large [()] instead scales the (k) parameter proportionally and results in sharper vMF distribution for the component. Hence, the model cannot naively increase | (8) to increase the responsibility of a component to a data sample-it also needs to map the data samples close to the component's 's prototype. If the images assigned mapped close to its prototype in the latent space, only then it is beneficial to increase (*))
component can be consistently
The vMF normalization constant C₂ (k) includes the modified Bessel function of the first kind and order v, denoted as (K). It is easy to obtain x- = ||w|]/T, but since (k)) is a complicated function, we propose to use differentiable approximation thereof. Let = p/2 and K = vr. Then, 4n can be approximated by the following uniform expansion for large (DLMF Eq. 10.41.3):
en
Ui(s)
(vr) (2Tv) 2211/4
== + x 1/2 log
12) (5)
where ~ denotes Poincaré asymptotic expansion w.r.t. v, (1--=1++))) 1 and polynomials (s). Here, the bottleneck dimension in the prediction head is p = 256, and this results in v = 127, which is large in this context. Empirically, evaluating only the first term in the sum, Ua(s) = seems to be sufficient, see Appendix A.2 for details. With this approximation. we compute (x())) ) for the k-th component and modity the logit scores for component for both teacher and student from (k) //) to ((w(()) //)/T- + log (n))) That is, we add log (1((a)) to the logit scores in (3) and (4) to obtain the following expressions for DINO-vMF:
(x)) & exp
(wn(+) yy)//T, + log Cp (n!+)).
+og
(6)
(x)))) (M) & exp [c(b]] exp (1o) y.)/ T. + log Cp ( [kt [=
(7)
4

---

Published as a conference paper at ICLR 2023
3.4 AVOIDING COLLAPSE
The probability distributions are computed based on the teacher and student network outputs using softmax function. With the motivation of avoiding collapse, centering and sharpening operations are applied to the network oulputs to obtain the logit scores for the probability distribution. The centering parameter in DINO is computed as an EMA of the teacher network outputs with forgetting parameter m S 1 as follows:
(1) mk) + +(1 m)
(k) m
yr.b)
(8)
However, when centering is done in the logit space as in with our modified logit scores (ww) (k) y,))/ log Cy (x())) EMA of the log normalization constant will be added to c(t), When the centering operation is performed on the teacher outputs, the impact of the log normalization constant is dampened. To avoid this, we propose instead compute the probability distributions for the images in the batch, and then average the probability distributions instead. This also closer to how mixture proportions (k) are computed in the M-step of the EM algorithm for a mixture model, although the estimated probabilities play different role in DINO; see Appendix A.4 for discussion. Our proposal for computing the centering parameter c is thus:
B
(R) ( + (1 mmmooo
1
Σ
(softmax (r,/////T)) (M)
4 RELATED WORK
Clustering- based and prototypical methods: SSL has advanced significantly in recent times and line of work based clustering have also evolved. [Caron al. (2018) proposed predictive task using "pseudo-labels" generated by K-means clustering in the representation space. Following
its success, several methods based on clustering have been proposed (Caron al. 2020: Asano
eet al., 2020: Caron et al.] 2019 Li et al., 2020 /Zhuang et al. 2019 Assran et al. 2022 2021).
These methods learn set of prototype vectors or cluster centroids that correspond with the clusters. When combined with ViTs and self-distillation, Caron et al. (2021) demonstrated that the learned representations possess useful properties strong nearest neighbor embeddings and rich spatial semantic features. While the DINO formulation was only based on the [CLS] representation, several recent works have extended DINO to utilize patch-level representations in the objective formulation (Li et al. 2021 Zhou al.) 2021). A common aspect among most recent methods is that they utilize I.-nnomalizeed prototypes. Our work is based on the DINO formulation. By incorporating our work in iBOT, we show that our insights from prior works by proposing a way to remove the - no-oomallzzaiinn constraint on the prototypes and hence, enable flexible mixture model in the representation space. Assran et al. (2022) uses an entropy regularization to avoid collapse and our centering computation shares similarity with it, in terms of computing intra-batch averages of probability distributions. However, we only average teacher probability distributions to compute the centering parameter instead of applying an explicit regularization of the student outputs.
prove beneficial to other methods as well. Our method differs
Mixture of von Mises-Fisher distributions: Banerjee et al. (2005) proposed a mixture of vMF (movMF) distribution to perform clustering on the unit hypersphere. The EM updates are often done using an approximate repurameterization of the vMF precision parameter x (Banerjee et al., 2005 Barbaro & Rossi 202T). Instead, we learn к directly through the prototype magnitudes. Some image segmentation methods have used movMF formulations (Yang et al. 2020 Hwang et al.) 2019) but
assume a fixed k for all mixture components to simplify computations. Taghia et al. (0014): )): Gopal & Yang (2014) use complete Bayesian approach based on variational inference to model data as
movME. Recent works relate the cosine similarity formulation commonly used in contrastive SSL to vMF density (Shwartz- Ziv et al.) 2022) Lee al. [021] Wang & Isola 2020).. Our mixture model interpretation is closest to Hasnat et al. (2017) that proposes a movMF loss to train a supervised face verification model, but they make similar uniformity assumptions DINO.
5

---

Published as a conference paper at ICLR 2023
Table 1: ImageNet KNN classification
Table ImageNet classification accuracy using linear and kNN classifiers. ([C) [CLS], [D] = DINO)
accuracy ablating on the impact of I normalization of prototypes, vMF nor- malization and probability centering. Av- erage over runs are reported. Refer A.6.1 for results of individual runs.
Method
KNN
Lin. [C]
Lin. [D]
ViT-Base/16
iBOT
77.10
79.33
79.50
iBOT-vMF
78.66
80.20
80.27
Centering space
DINO
76.11
77.88
78.17
Normalization
logit
probability
DINO-vMF
77.40
78.67
78.81
BelT-v2
80.10
None
70.18
70.49
MSN
73.33
75.84
74.80
L
69.51
69.86
vMF
70.21
70.87
VIT-Smal/16
default setting for DINO/VIT-S,
DINO
74.44
76.09
76.98
$
default setting for DINO/ViT-B.
DINO-vMF
74.74
76.
76.97
MSN
74.86
76.20
76.62
iBOT
75.20
77.90
EXPERIMENTS
We closely follow the training settings for different backbones specified in the public repository for D in order to ensure fair comparison. The models are pre-trained on the ImageNet dataset (Deng al. 2009) with the adamw optimizer (Loshchilov Hutter 2018). Weight decay with cosine schedule from 0.04 to 0.4 is used. The student temperature 7 is set to 0.1 and the teacher temperature is linearly scaled from 0.04 0.07 over some initial epochs (50 epochs for ViT-Small/16 and 30 epochs for ViT-Base/16). We follow the same data augmentations as DINO and BYOL Grrill jtt al. 2020): color jittering, Gaussian blur and solarization. The multi-crop approach used with same definition of global and local crop sizes as DINO. The model trainings are done on a single 110 node, consisting of 8 GPUs. The batch sizes are adapted to fit the node and adjusted based on the model architecture (batch size=64 per GPU for ViT-Base/16 16 and 128 for ViT-Small/16). The vMF normalization computation does not add any noticeable overheads to standard DINO and iBOT. Due to computational reasons, we train ViT-Smalle for only 40 epochs (referA.6.2) and do not consider ViT' larger than the Base variant. For comparison of results, we use the publicly released DINO and MSN pre-trained models available in their respective repositories.
5.1
IMAGENET CLASSIFICATION
5.1.1
ABLATION STUDIES
We conducted ablation experiments to study the impact of our proposed modifications DINO. We consider small-scale experiments by training a ViT-Smal//16 backbone for 100 epochs with all other settings maintained similar to the standard training. We report the kNN top- classification accuracy on ImageNet in Table by averaging over 2 runs. We observe that the performance is better when the cluster prototypes are not n.2-normal We find that vMF normalization and probability centering individually lead to performance improvements. Best performance is achieved when both vMF normalization and centering in probability space are used simultaneously. On this basis, we decide to vMF normalization along with centering in the probability space in all other experiments. Note that maximal improvement is observed when compared to the default setting of DINO/ViT-B.
5.1.2 IMAGENET CLASSIFICATION WITH FULL DATASET
The quality of representations learned by self-supervised models are usually evaluated by linear classification benchmarks on top of frozen pre-trained backbone model. DINO is also shown to be an efficient nearest neighbor classifier. For linear evaluation, we follow the same protocol as DINO. Additionally, we also evaluate the performance using only the [CLS] representation. For
'https///githu... hhttp://gttub..co//fcchhceeo http://hhhh..mmiiiccccccccccccca http::///gthhh..cco//cccchocrres

---

Published as a conference paper at ICLR 2023
Table 3: ImageNet few-shot evaluation (as in MSN)
ViT-Base/16
ViT-Small/16
Dataset
DINO
MSN
DINO-vMF
iBOT
iBOT-vMF
DINO
MSN
DINO-vMF
1 img/cls
41.8
49.8
50.3
46.0
51.6
38.9
47.1
39.2
2 imgs/cls
51.9
58.9
59.3
56.0
61.1
48.9
55.8
49.4
5 imgs/cls
61.4
65.5
66.1
64.7
68.3
58.5
62.8
59.1
I % imgs
67.2
69.1
70.4
69.9
72.3
64.5
67.2
65.0
Table 4: Prototype utilization based cosine similarity threshold of 0.9.
# unique prototypes
Largest duplicate prototype set
Architecture
DINO
DINO-vMF
DINO
DINO-vMF
ViT-Small/16
961
1122
2087
1226
ViT-Base/16 775
928
54877
1208
KNN evaluation, consider a straightforward sweep over range of values for nearest neighbors and generally find A: = 10 to work best with DINO-vMP models. The accuracy achieved by linear and kNN classifiers ImageNet reported in Table [ (more baselines in(A.6.2). With ViT-Base/16 6, both DINO-vMF and iBOT-vMF clearly perform better than their standard counterparts. With ViT- Small/16, the performance improvement is found to be marginal. MSN (Assran al. [2022) shows competitive performance on ViT-Small/16 but is significantly worse on the larger ViT-Base/16.
5.1.3 IMAGENET FEW-SHOT EVALUATION
Few-shot classification an important application area for SSL pre-training. MSN (Assran et al.) 2022) pre-training is found to be effective at few-shot classification tasks. We compare our DINO, iBOT, their vMF versions and MSN pre-training using the same evaluation protocol as MSN. A linear classifier is trained features obtained using center crop of the image from the frozen pre-trained model. L'-regularization with a constant strength of 0.075 is used. We consider different levels of labelled data availability: 1, 2 and 5 images per class, and 1% of all the training images. We report the top-1 accuracy in Table (3 For the 1, 2 and 5 images per class experiments, we use three different splits and report the mean of the accuracy (refer[A.6.44 [A.6.4 standard deviations). With ViT-Base/16 model, we observe a large improvement by adding the vMF modification and even surpassing the performance achieved by MSN (Assran et al. 2022), which specifically targets the few-shot learning problem. In comparison, the ViT-Small/16 model with DINO-vMF pre-training only improves marginally over DINO and is found be lagging behind MSN by a large margin.
5.2 ANALYSIS OF LEARNED vMF MIXTURE MODEL
In this section, we take deeper look at the mixture modeling task, which acts as the supervisory signal to train the self-supervised model.
Inspecting the prototype vectors learned by DINO, we observed that many of them were almost identical. We consider two prototypes (1) and w to be duplicates if their cosine similarity greater than threshold value. We identify unique prototypes by removing such duplicates. Recall that DINO uses a large number of prototypes set to 65536 for all models. Assran al. (2022) does not observe a benefit in using more than 1024 prototypes, but fewer prototypes lead to worse performance. In Table 4 we show the number of unique prototypes and the size of the largest set of duplicate prototypes based on a cosine similarity threshold 0.9 (in A.3 we show that our observations hold at other threshold choices as well). Interestingly, when pre-trained with DINO, the VíT- Base model produced one large duplicate set containing x 83% of the prototypes. Based highest probability, We refer to this as the void prototype set. Interestingly, the void prototype has a cosine similarity of exactly -1 1 with the mean of y over all the training data. This means that the void prototype vector
Prototype utilization:
data samples assigned to the corresponding mixture components.
7

---

Published as a conference paper at ICLR 2023
points in direction exactly opposite to the data mean, y. This occurs because the model minimizes the logit scores unused prototypes i.c. void prototypes, by moving them as far away as possible from the data representations to minimize their effect on the loss. With DINO-vMF pre-training, do not observe that any such void prototype set is formed and more unique prototypes are utilized than standard DINO pre-training. Utilizing more prototypes increases the difficulty of the SSL task and we expect this to result in better image representations.
Interpreting learnt vMF precision: In the DINO-vMF formu- lation, the norm of the prototypes is directly related to the vMF precision as A = [|//T_ We investigate if the preci- sion values be indicative of image classification difficulty by evaluating downstream performance. We associate an image with the component that has the maximum logit score under the DINO(-vMF) formulation. For each model, we divide the data based on percentile ranges of the precision values the associated components. In Figure 2 we show the kNN top-1 validation accuracy on ImageNet for subsets of data restricted to lie in the different precision percentile ranges. For DINO-vMF, we observe that the kNN classification accuracy is increasing with increasing precision values. That is, data points associated with higher precision components appear to be easier to classify, and vice versa. The prototype magnitudes learned by DINO (only in ViT-Small/16) do not exhibit such a clear association.
S/16, DINO
KNN val. top-1 accurncy
VII-S/16, DINO-MM
VIT-B/16, DINO-VME
70
65
60
(0,20)
[20,40]
(40 601
[60 80]
[8n, 100]
Percentile (based on prototype precision)
Figure 2: kNN accuracy for data sorted based percentile ranges of associated [|))
5.3 TRANSFER LEARNING
An important benefit of SSL is that the learned model can be transferred to downstream tasks to achieve competitive performance with limited computational cost or limited training data. The tasks we consider are: linear classification on small datasets, image retrieval and video object segmentation.
Linear classification on small datasets: We conduct linear classification experiments on suite of datasets trained on features extracted from a frozen pre-trained model. The implementation details for this experiment are explained in A.6.3 In Table we report the linear classification accuracy on the test or validation dataset, depending on which is publicly available. First, we observe that the DINO-vMF model performs par or better than the standard DINO model most of the datasets with both ViT-Small/16 and ViT-Base/16 architectures. The iBOT-vMF model consistently performs best for most datasets. We also observe that the MSN model performs worse than both the DINO and DINO-vMF models on all datasets. Additional fine-tuning results are given in A.6.5
Image Retrieval: We consider the blurred versions (v1.0) of Oxford Phillin et al. 2007) and Paris datasets (Philbin et al.) 2008) and use the medium (M) and hard (H) data splits. We perform nearest neighbour based image retrieval using ImageNet pre-trained frozen features and report the mean average precision (mAP). We follow similar evaluation protocol to DINO but we use newer
Table 5: Linear classification accuracy when transferred to other datasets
Method
Acft.
Cal101
C10
C100
DTD
Flwrs.
Food
Pets
SUN
Avg.
ViT-Base/16
DINO
57.7
94.5
96.9
86.3
74.8
96.0
81.6
93.9
68.5
83.4
DINO-vMF
57.3
94.5
97.1
86.3
74.8
95.7
82.5
94.6
68.7
83.5
iBOT
55.9
94.7
97.6
87.4
74.4
95.4
82.7
93.8
69.4
83.5
iBOT-MMF
58.1
95.5
98.0
88.0
74.7
94.8
83.6
93.9
70.2
84.1
MSN
51.0
92.8
96.9
85.3
73.7
92.8
80.0
93.9
66.8
81.5
ViT-Smal//16
DINO
53.8
93.1
96.2
83.5
74.7
94.7
79.7
93.8
66.8
81.8
DINO-vME
56.0
93.7
96.0
83.9
74.1
95.0
80.1
93.9
66.6
82.1
MSN
51.6
93.1
95.9
82.9
72.0
93.3
77.8
92.8
65.5
80.5
8

---

Published as a conference paper at ICLR 2023
Table 6: Image retrieval performance
Table Video Object Segmentation (VOS)
RPar
ROx
Method
(I&F)m
Jm
Fm
Pretrain
M
M
H
ViT-Base/16
DINO
62.4
60.8
64.0
ViI-Base/16
DINO-vMF
63.4
61.6
65.2
DINO
63.7
35.8
34.3
10.8
iBOT
62.7
61.8
63.7
DINO-vMF
66.7
39.5
37.3
12.5
iBOT-vMF
63.1
61.9
64.2
iBOT
64.1
36.6
35.2
13.4
MSN
58.0
56.1
60.0
iBOT-vMF
65.4
38.1
38.1
13,8
MSN
56.3
29.3
31.8
11.8
ViT-Small/16
DINO
61.8
60.2
63.4
ViT-Small/16
DINO-vME
62.7
60.9
64.5
DINO
63.1
34.5
34.6
13.0
MSN
59.6
57.6
61.6
DINO-vMF
62.1
33.4
34.5
12.6
MSN
61.5
33.3
36.6
15.1
versions of the datasets. We observe improved performance with ViT-Base/ model when using the vMF versions compared to standard DINO and iBOT on both datasets. However, the ViT-Small/16 model pre-trained with DINO-vMF only on par or slightly worse than standard DINO pre-training. Further, find that the MSN pre-trained model for ViT-Small/16 also performs competitively.
Video Object Segmentation (VOS): We consider the DAVIS-2011 video instance segmentation benchmark (Pset-Tuset et al. 2017). We the frozen features from the ImageNet pre-traie me-trained model and use nearest neighbour based segmentation approach following the same experimental protocol (Caron al. 2021) and (Jabri et 22200). We compare our results with standard DINO, iBOT and MSN in Table We find that the vMF versions show marginal improvements on both ViT backbones. We also find that the MSN transfers poorly this task compared to DINO and DINO-vMF.
CONCLUSION
We reinterpreted the DINO (Caron et al. 2021) ) SSL method as a mixture of von Mises-Fisher distributions in latent space. Based on this interpretation, we proposed to use unnormalized prototypes with appropriate normalization of the component logit scores to enable a more flexible mixture model. Our proposed modifications enable stable training of DINO using larger VIT models and better cluster utilization. The modification requires log-normalizing constants during training. However, using the proposed approximation these are very fast compute and do not result in any noticeable computational overhead. At inference time the log-normalizer is not used at all. We empirically demonstrated consistent performance gains on range of transfer tasks. The larger ViT-Base/ model benefits the most from proposed modifications and shows significant improvements all the considered downstream tasks. We accredit this to the fact that, without the proposed modifications, the larger models require unfavourable 1.2 7.-normalization of the prototypes obtain stable training. The smaller models are already trained without I n-oomallization and for these models we see more marginal gains. Thus, our study has put a spotlight on seemingly important design choice that, in our opinion, has not been discussed enough in prior work. By improving iBOT using our vMF modification, show that our work extends other methods built on the DINO formulation.
DINO-vMF was motivated by a mixture model interpretation of DINO, but we nevertheless use quite similar training algorithm. For future work it would be interesting to investigate this interpretation can be used to motivate additional modifications. One could for instance relate the DINO training scheme stochastic versions of the EM algorithm (Delyon et al.) 1999 Celeux Diebolt 1992). The soft cluster assignments computed using the teacher network can be viewed as the E-step, where the EMA similar to the stochastic approximation of the auxiliary quantity used in SAEM (Delyon et al. 1999,. The stochasticity comes from mini-batch sampling and multi-crop training. The main difference is in terms of the objective that optimized in the M-step, where EM maximizes the ELBO whereas DINO(-vMF minimizes the KL divergence between cluster probabilities. Exploring this connection in more detail could possibly result in better understanding of clustering-based SSL. methods, as well additional performance boosts.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We intend to make a public release of the code repository and pre-trained models in order to aid the research community to reproduce our experiments. We do not introduce any new hyperparameters in our pre-training method and use the same hyperparameter setup as DINO (Caron et al. 2 and iBOT (Zhou et al. (22TI) which already publicly available. For other downstream tasks, closely follow the experimental protocol of other works and explicitly state any differences in the paper.
ACKNOWLEDGMENTS
This research is financially supported by the Swedish Research Council via the project Handling Uncertainty in Machine Learning Systems (contract number: 2020-04122), the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, and the Excellence Center at Linköping-Lum in Information Technology (ELLIIT). The computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In ICCV, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. ECCV, 2022.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von Mises-Fisher distributions. JMLR, 2005.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021.
Florian Barbaro and Fabrice Rossi. Sparse mixture of von Mises-Fisher distribution. ESANN, 2021.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative compo- nents with random forests. In ECCV, 2014.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. NeurlPS, 1993.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- pre-training of image features non-curated data. In ICCV, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in sulf-supervised vision transformers. In ICCV, 2021.
Gilles Celeux and Jean Diebolt. A stochastic approximation type em algorithm for the mixture problem. Stochastics: An International Journal of Probability and Stochastic Processes, 1992.
Ting Chen. Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI, 2020a.
10

---

Published as a conference paper at ICLR 2023
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422) 04299777 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Bernard Delyon, Mare Lavielle, and Eric Moulines. Convergence of stochastic approximation version of the em algorithm. Annals of statistics, 1999.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
DLMF. NIST Digital Library of Mathematical Functions. http://dlmf.nist..ow Release 1.1.6 of 2022-06-30, 2022. URL (tLLhhtp:///dmm mist gov/ F. W. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition scale. In ICLR, 2020.
Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How well self-supervised models transfer? In CVPR, 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In ICML, 2021.
Siddharth Gopal and Yiming Yang. von Mises-Fisher clustering models. In ICMI,, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own a new approach to self-supervised learning. NeurIPS, 2020.
Md Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, Liming Chen, et al. von Mises- Fisher mixture model-based deep learning: Application to face verification. arXiv preprint arXiv: NBKiv:770.042200 04264, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chie Chich Chen. Segsort: Segmentation by discriminative sorting of segments. ICCV, 2019.
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. NewrIPS, 2020.
Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. WACV, 2018.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In ICLR, 2018
11

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In ECCV, 2016.
Kuang-Huei Lee. Anurag Amab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. NeurlPS, 2021.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervvsed vision transformers for representation learning. In ICLR, 2021.
Fei-Fei Li, Marco Andreeto, Mare Aurelio Ranzato, and Pietro Perona. Caltech 101, 2022. URL
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In ICIR, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained
visual classification of aircraft. Technical report, 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of preeexttinvariiantteeeeese In CVPR, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Immgee Processing, 2008.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. CVPR. 2012.
Zhiliang Peng. Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. CVPR, 2007.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Lost in quantization: Improving particular object retrieval in large scale image databases. In CVPR, 2008
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine- Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. 1rXiv:1704.0065,, 2017. 2017,
Tim Salimans and Durk Kingma. Weight normalization: A simple reparameterization accelerate training of deep neural networks. NeurIPS, 2016.
Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What do we maximize in self-supervised learning? In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022.
Jalil Taghia, Zhanyu Ma, and Ame Leijon. Bayesian estimation of the von-Mises Fisher mixture model with variational inference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NewrIPS, 2017.
12

---

Published as a conference paper at ICLR 2023
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Hugo Touvron, Matthicu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In ECCV, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICML, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. CVPR, 2010.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arXiv preprint arXiv:2105.04553, 2021.
Boyu Yang. Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In ECCV, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In ECCV, 2016.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In ICIR, 2021.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins, Local aggregation for unsupervised learning of visual embeddings. In ICCV, 2019.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 INTUITIVE COMPARISON OF DINO AND DINO-VMF
In Section 3.3 we stated that in properly normalized formulation, the model cannot naively increase || to increase the probability of assigning data sample cluster or mixture component. We expand on this idea by providing an intuitive example. Consider 2-dimensional latent space on the unit circle. Let y denote vector in this latent space with |//| = 1. We consider simple setting with only two clusters, E (1, and let w and (2) be prototypes representing these clusters. The DINO formulation proposes the following logit score 'DINO for assigning vector y to cluster
315
1.5
0.5
270
90*
225
135
180
. (w01)) . (1 cos (())
Figure 3: von Mises-Fisher density the circle for a prototype vector pointing in the direction of 315 for different values of prototype magnitudes (larger magnitude: blue curve, smaller magni- tude: red curve).
For simplicity, let us ignore the temperature scaling that is applied in the sharpening step. Here, the model can simply increase [| to increase y(3) 'DINO as long as cos ((1) >> O. Instead, proposed DINO-vMF uses the following (()
DINO-MMF logit scores:
(1) DDNOOVMF (20(2),)) log Cp (1) (o 1l20(2)) cos ((2) + log Cp((1o(())
where log C,(+) O and log C₂ (1) monotonically decreases with [[wo ||. So, merely increas- ing []) (1) can increase the first term but this is counteracted by the second term. For given values of (1) and ((1) there exists a threshold ((1) below which increasing [1,] results an increased logit score. The logit score decreases when ((1) is larger than this threshold. So, the model needs to decrease ((1) by mapping the vector y closer to the prototype in order to benefit from increasing [1) We show an illustration of how the vMF density varies with the prototype magnitude in Figure This implies that, if the DINO-vMF model can consistently map a set of images close to a prototype, only then it can benefit from increasing the magnitude of that prototype.
A.2 VON MISES-FISHES NORMALIZATION CONSTANT
Given a von Fisher distribution in p-dimensions with parameters and k, its normalization constant (1s) is given by:
KP/ 22/2-11
Cp(k) (2/T)//2 1 (m:))(())
where I, denotes the modified Bessel function of the first kind and order v. We approximate the 1p/2-1 (x) term as explained in section 3.3 by using an asymptotic expansion. The softmax-logr scores for each mixture component is obtauned by adding the network output and the log- normaliaaiinnionn constant corresponding to that component. Because of the softmax formulation, it sufficient for the log-normalization constant approximation to be correct up to constant. We compare the values of log (,()) obtained using our approximation to those obtained by using the scipy implementation of I, instead. We consider E [20, 500] and in Figure 4 we show a comparison of the values of log (x) obtained using our approximation. We did produce the same figure with the same values computed using the scipy- based implementation. but the lines appear completely overlap with approximation, so this is not shown. We define an approximation error e(R) with respect to the scipy implementation follows:
14

---

Published as a conference paper at ICLR 2023
0.0015
100
x(x)
0.0010
log Ciei (i)
-200-
p=64
0.0005
p=128
-300-
p=256
0.0000
p=512
-400
250
500
250
X
K
Approximation error in computing log Cp()) compared to scipy-based implemen- tation. Note that the error is small compared to the approximated function values shown in Figure
Figure 5:
Figure 4:
Our approximation up constant,
log ( (R))
<(K) = log ((0) (k) log (K)
where cff) denotes our approximation and c/a) (k) denotes the scipy-based implementatior" The error values are shown in Figure for different bottleneck dimensions p in the DINO prediction head. For p 256, in the case of DINO, we can observe that the approximation error is small compared to the actual function values.
A.3 PROTOTYPE UTILIZATION AT DIFFERENT SIMILARITY TIRESHOLLDS
As explained in section 5.2 we identify unique prototypes based on cosine similarity threshold. Here, we present a further discussion about how the prototype utilization varies with the choice of similarity threshold. In FiguresB]and 7, the largest duplicate set of prototypes. An unusually large duplicate set indicates a void protolype set. We observe that the void prototype set in ViT-base/16 hardly changes in size even when the similarity threshold is increased up 0.99. We observe higher prototype utilization with DINO- vMF compared to DINO for both the ViT-Base and ViT-Small models. Only exception is when we set the similarity threshold to large value of 0.99 for ViT-Small/16 model trained with DINO. We expect these near-duplicates to further increase in similarity, if the model training is continued for more epochs.
we show the number of unique prototypes and the size of
A.4 RELATIONSHIP BETWEEN CENTERING AND CLUSTER PRIOR
The centering parameter (1)) computed according to Eq. S in DINO Eq. 9 in DINO-vME, is related to an estimate of the cluster prior ( in our mixture model interpretation. Indeed, for DINO, the centering parameter averages the logit scores of the components, scaled by the sharpening parameter Tt. so exp (c()) [c())/T]] can be viewed an estimate of ((k) up to normalization. For DINO- vMF we similarly average the cluster probability (i.e., responsibility) over a batch at each iteration, and update our estimate of c(() using an EMA of the corresponding logits; see (9). Note that the sharpening parameter 7; is included in the expression for the probabilities, which means that, in this formulation,
estimate (F) exp [c( up normalization, i.c. without the scaling т,, The
key difference between the two estimates is thus whether the averaging is done in logit space or in probability space. The latter is closer to how one would typically estimate 7 (Ak standard mixture model, e.g. using the EM algorithm,
average of the responsibility for cluster k.
Combining the aforementioned estimates with the expressions for the teacher probabilities (Eq. (4) for DINO or Eq. 7 for DINO-vMFS and comparing with the responsibilities computed for mixture model in Eq. 2 it is tempting simply relate the factors involving (k)) with the prior
15

---

Published as a conference paper at ICLR 2023
VIT-Base/16
ViT-Small/1
1200
Num ber of unique prototypes
1000
DINO
800
DINO-vME
600
0.50
0.60
0.70
0.80
0.90
0.99
0.50
0.60
0.70
0.80
0.90
0.99
Similarity threshold
Similarity threshold
Figure 6: Number of unique prototypes considering different cosine similarity threshold for identi- fying duplicate prototypes. Even relatively low similarity thresholds, DINO-vMF utilizes more unique and adequately separated prototypes.
VIT-Base/16
VIT-Small/16
duplicate prototype set
50000
Size of the largest
40000
30000
DINO
DINO-vME
20000
10000
0
0.50
0.60
0.70
0.80
0.90
0.99
0.50
0.60
0.70
0.80
0.90
0.99
Similarity threshold
Similarity threshold
Figure Size the largest set of duplicate prototypes. DINO pre-trained ViT-Base models contain large duplicate set, that we call void prototype set. This is avoided in DINO-vMF pre-training. Recall that ViT-Base models are trained with I,2-normalized prototypes whereas the ViT-Small models are trained with unnormalized prototypes.
probability (A) 2 However, this would be an incorrect interpretation since there a minus sign in the exponents in Eqs. and (7).
To understand this discrepancy, note first that in a mixture model we would estimate (+) from the available data, but also encourage data points to be assigned to components with higher ( - This can be thought of "rich gets richer" approach. If certain cluster is found to be dominant, then this information is fed back to the estimates of the responsibilities, increasing the probability of assigning data points this cluster.
Such an approach makes sense when the data is fixed, and we are simply trying to learn a clustering model that fits the data well as possible. However, it is also distinctly different from the role of the centering in DINO(- where the data are mapped to latent representations and the clustering is carried out in this latent space. For these models we fix as way to further encourage this uniformity. This is achieved by generating targets using the inverse of the estimated cluster probabilities, i.c. instead of multiplying with ा. (k) x exp [<(]] we multiply with 1 (A)( exp [+] This can be viewed by encouraging the model to assign more samples to less used components and less samples to more highly used components. We effectively obtain "rich gets poorer" and "poor gets richer" strategy.
be uniform. The centering can be viewed
way rebalance the component assignments,
16

---

Published as a conference paper at ICLR 2023
Table 8: ImageNet kNN classification accuracy ablating on the impact of I2-normalization of prototypes, vMF normalization and probability centering for 2 manually seeded trainings.
Centering space
Run 1 (seed=0)
Run (seed=42)
Normalization
logit
probability
logit
probability
None
70.43 f
70.74
69.93
70.24
L
69.72 $
70.18
69.30
70.17
vMF
70.51
71.12
69.91
70.62
f default setting for DINO/ViT-S,
default setting for DINO/ViT-B.
A.5 OUR MODIFICATION OF IBOT WITH vMF NORMALIZATION
We incorporate our vMF modification in iBOT Zhou et al. (2021) which uses DINO-derived loss formulation. The self-supervised pre-training in iBOT uses a self-distillation objective, & icis] and masked image modeling (MIM) objective, [[MIM]. Both objectives
formulated as cross-entropy terms similar to DINO based on the teacher and student probability distributions. We apply the vMF modification to the teacher and student probability distributions for the [CLS] and patch tokens as follows:
P. [CTS (x) (R) & exp
w (k)
[C15]
TT. + log Co (n!{*)]
plczs] (x) (z) & exp
(k)
-c(k) exp ( W y 10101) /Tx log Op (x4 (x!
ppatch (z)) & exp
(w(t), yattch) /T, log Cp (n(( (R) & exp [c(()) exp ( (w)( paach) /Te log Cp (ng (ki) ).
pnatch (z)
Consider image views and and N masked views à; and à; with masks my Then, the overall loss in iBOT is simply computed as sum of the objectives, (ecss) and LIMMM which are formulated as follows:
C [CLS]
= =-pIcLs! (v)² log p. p[CLs] (w)
N
C [[MM = (u, ( ppatch
A.6 ADDITIONAL EXPERIMENTS AND DETAILS
A.6.1 DETAILED ABLATION STUDIES
We run the ablation studies explained in 5.1.1 for 2 runs and all pre-trainings within a run are done with manually fixed seeds ensure that the results comparable and reproducible within each The results for each run are reported in Table g
A.6.2 IMAGENET CLASSIFICATION WITH FULL DATASET
We present additional baseline comparisons for HTT-Smal/116 and ViT-Base/16 architectures in Tables annd 10 respectively. In addition the results presented based DINO and DINO-vMF pre-trained ViT- GiTlSmal//16 and VÝT- BTT-Base/16 models, we consider the ViT-Small/8 backbone architecture that uses smaller patch size of 8. This model is more expensive train compared to the ViT-Small/16 model.
17

---

Published as a conference paper at ICLR 2023
DINO pre-trained this model for 800 epochs. We conduct smaller experiment for 40 epochs with the same hyperparameters as in DINO. The results are reported in Table The results are similar to observations regarding the ViT-Small/I architecture.
Table 9: ImageNet classification accuracy using linear and KNN classifiers for VIT- Small/16
Table 10: ImageNet classification aceuracy using linear and kNN classifiers for ViT- Base/16
Method
kNN
Lin. [C]
Lin. [D]
Method
KNN
Lin. [C]
Lin. [D]
DINO
74.44
76.09
76.98
iBOT
77.10
79.33
79.50
DINO-vMF
74.74
76.19
76.97
iBOT-vMF
78.66
80.20
80.27
MSN
74.86
76.20
76.62
DINO
76.11
77.88
78.17
SwAV
66.3
73.5
DINO-vMF
77.40
78.67
78.81
iBOT
75.2
77.9
BelT-v2
80.10
MoCo-v3
73.4
MSN
73.33
75.84
74.80
MoBY
72.8
MoCo-v3
76.7
MoCo-v2
64.4
72.7
MAE
68.0
BYOL
66.6
71.4
BelT
56.7
Table 11: ImageNet classification accuracy using linear and kNN classifiers for ViT-S/8 architecture
Method
kNN
Linear ([CLS))
Linear (DINO)
DINO
69.98
72.75
74.24
DINO-vMF
70.41
72.91
74.14
A.6.3 IMPLEMENTATION DETAILS FOR LINEAR CLASSIFICATION ON SMALL DATASETS
We consider the following small datasets: Aireraft Maji al. 2013). Caltech101 LLi al. 2022), CI-
FARIO, CIFAR100 100 (Krizhessky 2009). DTD (Cimpoi et al. 2014). Flowers (Nilsback & Zisserman 2008). Food (Bossard al. 2014), Pets (Parkhi al. 2012) and SUN397 (Xiao et al.) 2010). The linear classifier is trained with L ('-reullarization and the strength is chosen for each dataset by using 5-fold validaaniiotion among
45 values spaced linearly in the range [-6, 6,5, 5] log-space
following Ericsson et al. 22021.
A.6.4
IMAGENET FEW-SHOT EVALUATION
The few-shot evaluation involving 1, 2 and images per class were conducted using three different splits. Table and Table [3 show the mean and standard deviation of the top-1 validation classification accuracy over the three splits.
Table 12: ImageNet few-shot evaluation (as in MSN) with ViT-Base/16 model
ViT-Base/16
Dataset
DINO
MSN
DINO-vMF
iBOT
iBOT-vMF
img/cls
41.8 + 0.3
49.8 + 0.2
50.3 0.2
0.2
46.0
51.6 0.1
imgs/cls
51.9 + 0.6
58.9
+ 0.4
59.3 1 0.4
+6.0
61.1 + 0.7
imgs/cls
61.4 0.2
65.5
+ 0.3
66.1 0.2
6.
68.3 0.3
A.6.5 FINE-TUNING EVALUATION
We conduct fine- tne-tuning experiments on ImageNet- IK following the same training protocol as Bao al. (2021) which is found to consistently produce the best fine-tuning results at minimum training epochs Zhou et al. (202I). We train our VIT-S/16 and ViT-B/16 models for 200 and 100 epochs respectively. We report the best result obtained after conducting sweep the learning rates:
18

---

Published as a conference paper at ICLR 2023
Table 13: ImageNet few-shot evaluation (as in MSN) with ViT-Small/16 model
ViT-Small/16
Dataset
DINO
MSN
DINO-vMF
img/cls
38.9 0.4
47.1 + 0.1
39.2 0.6
imgs/cls
48.9 + 0.3
55.8 + 0.6
49.4 0.7
imgs/cls
58.5 + 0.1
62.8 + 0.3
59.1 = 0.2
18e-4, 9e-4, 1e-3, 2c-3). For fine-tuning on smaller datasets, we follow the fine-tuning recipe of Touvron et al. (2021) of training for 1000 epochs with small learning rate of 7.5c-6. We report the accuracy for CIFAR-10 10, CIFAR-100 and Flowers datasets and for the Aircraft dataset, we report the mean-per-class accuracy in Tables 14 and [3] We observe that the vMF variants are mostly on parpar or slightly better than their standard counterparts. While performance on CIFAR-10, 10, CIFAR-100 and Flowers are beginning to saturate, we observe greater performance gains on the Aircraft dataset.
Table 14: Fine-tuning results with ViT-Small/16 after pre-trate pre-training ImageNet- IK
Method
CIFAR10
CIFAR100
Flowers Aircraft
ImageNet-1K
Random (DelT)
99.0
89.5
98.2
79.9
Random (DeIT-III)
81.4
BeIT
98.6
87.4
96.4
DINO
99.0
90.5
98.5
84.2
82.0
DINO-vME
99.0
90.7
98.7
85.0
81.8
iBOT
99.1
90.7
98.6
85.7
82.3
Table 15: Fine-tuning results with ViT-Base/16 after pre-training ImageNet- IK
Method
CIFARIO
CIFARI00
Flowers Aircraft
ImageNet-IK
Random (DeIT)
99.0
90.8
98.4
81.8
Random (DeIT-IIII
83.8
BeIT
99.0
90.1
98.0
83.4
DINO
99.1
91.7
98.8
84.5
83.6
DINO-vMF
99.2
91.9
98.9
84.8
83.6
BOT
99.2
92.2
98.9
85.1
84.0
iBOT-MMF
99.2
92.6
98.8
85.8
84.1
MAE
83.6
19


Reviews:
[{"review_id": "v524V3gLdBX", "reviewer": "Reviewer_njCU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides an interpretation of DINO as a mixture model with components with a circular von Mises-Fisher distribution.\nUsing this interpretation, the authors propose a normalization when computing cluster assignments, which improves stability and flexibility of the mixture model.", "summary_of_the_review": "A useful and well-written paper.", "strength_and_weaknesses": "The paper proposes a very interesting interpretation of DINO and the proposed modification increases flexibility of the method by including per-cluster shape parameter. The experimental section includes a number of results to demonstrate the effectiveness of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XA4PMPg937", "reviewer": "Reviewer_SyZr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper demonstrates an interpretation of DINO as a von Mises-Fisher mixture model on the unit hypersphere. Using this interpretation, the authors alter the architecture of DINO in order to incorporate a normalization term directly in the distribution, and keep the embeddings unnormalized. Using this approach, the authors demonstrate an improvement over the performance of the version of DINO that uses normalized embeddings.", "summary_of_the_review": "Overall, I feel that this paper provides an interesting interpretation of DINO that leads to an interesting trick to improve its performance. I lean slightly towards acceptance due to the clean addition of this method to regular DINO training, but I believe that the paper can overall be improved due its somewhat limited novelty.", "strength_and_weaknesses": "Strengths:\n\n- The interpretation of DINO as a mixture model is interesting. The authors formalize an interpretation of DINO as a mixture model based on the probabilities that the student and the teacher assign to each cluster, for each given sample. This formalization provides a nice theoretical interpretation of DINO, leading to an alternative design choice for normalization, namely normalizing the distribution of the mixture model arising from DINO. This is more theoretically founded than simple $L_2$ normalization of the representations.\n\n- The authors perform an extensive set of experiments to examine whether their approach improves over the baseline DINO formulation. More specifically, they evaluate the performance of their approach on the downstream ImageNet classification task, the few-shot learning capabilities of their method, as well as the transferability of their learned representations on a given task. This suite of experiments provides a sufficient understanding of the capabilities of their proposed method, and can be used to compare it to regular DINO, as well as other baselines.\n\nWeaknesses:\n- The main issue I have with this work is that the novelty of the proposed method is limited, in that it consists of simply reparametrizing DINO by changing the normalization used. I believe that as it stands, the work focuses a lot on this change in normalization, without enough evidence on why it is important aside from the provided experiments. If possible, I believe that incorporating a simple theoretical example on why regular DINO may fail while the proposed DINO-vMF may succeed would greatly improve the paper and further justify the proposed method.\n\n- Regarding the experiments, the benefit over regular DINO is not clear, and seems to be very dependent on the architecture, as can be seen in the Tables in the paper (where DINO-vMF underperforms in the smaller ViT-Small model). Namely, the benefits seem to only be apparent for large models, as stated by the authors. Since there seems to be a major effect derived by the architecture of the backbone of DINO, I believe that the authors should also examine different architectures for this backbone (different ViT sizes, or ResNets).\n\n- As a more minor issue, the section in which the authors examine the void prototypes is somewhat unclear in my opinion. The way I understand it, the authors state that DINO has a tendency to assign a large number of samples to similar representation, while DINO-vMF tends to spread the samples further apart. I would be grateful if the authors could clarify whether this is correct.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H8_85Y35-P", "reviewer": "Reviewer_Xj5D", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the previous work DINO by interpreting it a new way, as a von Mises-Fisher mixture model, and in turn proposes DINO-vMF model and achieves very good performance on numerous tasks. \n", "summary_of_the_review": "\nThe work has an intuitive and well-motivated approach based on DINO, which shows impressive results on various tasks. The authors also conducted meaningful studies and analysis to back their intuition and rationale of their proposed method. \n", "strength_and_weaknesses": "Strength: The interpretation of vMF is appealing, especially leading to removing the L2 regularization to enable more flexibility. Apart from the representative strength, the results are also strong, even compared to the recent works such as MSN. \n\nWeaknesses: \n\n1/ An important question is how practical this solution is: many well-understood and interpretable frameworks seem to be less competitive to their counterpart deep models because of their speed. It seems even with necessary approximations such as in 3.3, it is not trivial for the case of other formulas, which might lead to further approximation to, e.g., trade for more speed. \nLikewise, the accuracy in the tasks given are appealing, yet how about the other performance aspects such as training time, inference time and else? In my humble opinion, even some discussions on the drawback of this solution would make the paper more–not less–convincing. \n\n2/ Since the normalization of prototypes is key to this solution, various related studies have been conducted esp. in Section 5. However, I suggest Table 1 should be extended to be more convincing in 2 aspects. First, in this small scale with this large dataset, running some rounds and get the variances as well would be more helpful. Second, do it with small datasets or any others, just to confirm the consistent practical reflection of the intuition and explanations given. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GgE8NVcXn9", "reviewer": "Reviewer_aACz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggests interpreting the recent self-supervised learning framework, DINO, as a mixture model of von Mises-Fisher components. Based on the interpretation, this paper proposes a modification to encourage DINO training to be more flexible and stable. This paper demonstrates the effectiveness of the modification under various experiments.\n", "summary_of_the_review": "Although the interpretation and the proposed modification is convincing, the empirical results are too marignal and not interesting. Hence I vote for weak rejection.\n", "strength_and_weaknesses": "Strengths\n- The interpretation is quite interesting, and the proposed modification makes sense.\n- This paper is easy to follow and well-written.\n\nWeaknesses\n- My major concern is about empirical results.\n  - This paper should compare the proposed method, DINO-vMF, with other SSL methods. The comparison with only DINO and MSN is not enough, and it makes some difficulty to check whether the obtained performance is good or not.\n  - The obtained gains seem marginal. For example, only 0.7\\% accuracy gain on ImageNet using ViT-Base, and no gain when using ViT-Small. Also, there is no meaningful gain in transfer learning (e.g., Table 5).\n  - There is no fine-tuning experiment.\n- Another concern is that this paper is too limited to DINO. This limited applicability would be critical in this field because there have been developed many SSL methods after DINO, e.g., MAE and iBOT. It would be better if this paper shows that the interpretation can be incorporated into other SSL frameworks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%