Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper extends the previous work DINO by interpreting it a new way, as a von Mises-Fisher mixture model, and in turn proposes DINO-vMF model and achieves very good performance on numerous tasks. 

strength_and_weaknesses: Strength: The interpretation of vMF is appealing, especially leading to removing the L2 regularization to enable more flexibility. Apart from the representative strength, the results are also strong, even compared to the recent works such as MSN. 

Weaknesses: 

1/ An important question is how practical this solution is: many well-understood and interpretable frameworks seem to be less competitive to their counterpart deep models because of their speed. It seems even with necessary approximations such as in 3.3, it is not trivial for the case of other formulas, which might lead to further approximation to, e.g., trade for more speed. 
Likewise, the accuracy in the tasks given are appealing, yet how about the other performance aspects such as training time, inference time and else? In my humble opinion, even some discussions on the drawback of this solution would make the paper more–not less–convincing. 

2/ Since the normalization of prototypes is key to this solution, various related studies have been conducted esp. in Section 5. However, I suggest Table 1 should be extended to be more convincing in 2 aspects. First, in this small scale with this large dataset, running some rounds and get the variances as well would be more helpful. Second, do it with small datasets or any others, just to confirm the consistent practical reflection of the intuition and explanations given. 

clarity,_quality,_novelty_and_reproducibility: The paper is well motivated and written, easy to follow the main logic. The vMF interpretation applied to vision is not entirely novel, such as also cited by the authors to Hasnat et al, yet applying to self-distillation is to my opinion new and helpful. The code is not provided, however, and thus is hard to evaluate reproducibility, given the implementation of probabilistic models is by no means without certain complication. 

summary_of_the_review: 
The work has an intuitive and well-motivated approach based on DINO, which shows impressive results on various tasks. The authors also conducted meaningful studies and analysis to back their intuition and rationale of their proposed method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper demonstrates an interpretation of DINO as a von Mises-Fisher mixture model on the unit hypersphere. Using this interpretation, the authors alter the architecture of DINO in order to incorporate a normalization term directly in the distribution, and keep the embeddings unnormalized. Using this approach, the authors demonstrate an improvement over the performance of the version of DINO that uses normalized embeddings.
strength_and_weaknesses: Strengths:

- The interpretation of DINO as a mixture model is interesting. The authors formalize an interpretation of DINO as a mixture model based on the probabilities that the student and the teacher assign to each cluster, for each given sample. This formalization provides a nice theoretical interpretation of DINO, leading to an alternative design choice for normalization, namely normalizing the distribution of the mixture model arising from DINO. This is more theoretically founded than simple $L_2$ normalization of the representations.

- The authors perform an extensive set of experiments to examine whether their approach improves over the baseline DINO formulation. More specifically, they evaluate the performance of their approach on the downstream ImageNet classification task, the few-shot learning capabilities of their method, as well as the transferability of their learned representations on a given task. This suite of experiments provides a sufficient understanding of the capabilities of their proposed method, and can be used to compare it to regular DINO, as well as other baselines.

Weaknesses:
- The main issue I have with this work is that the novelty of the proposed method is limited, in that it consists of simply reparametrizing DINO by changing the normalization used. I believe that as it stands, the work focuses a lot on this change in normalization, without enough evidence on why it is important aside from the provided experiments. If possible, I believe that incorporating a simple theoretical example on why regular DINO may fail while the proposed DINO-vMF may succeed would greatly improve the paper and further justify the proposed method.

- Regarding the experiments, the benefit over regular DINO is not clear, and seems to be very dependent on the architecture, as can be seen in the Tables in the paper (where DINO-vMF underperforms in the smaller ViT-Small model). Namely, the benefits seem to only be apparent for large models, as stated by the authors. Since there seems to be a major effect derived by the architecture of the backbone of DINO, I believe that the authors should also examine different architectures for this backbone (different ViT sizes, or ResNets).

- As a more minor issue, the section in which the authors examine the void prototypes is somewhat unclear in my opinion. The way I understand it, the authors state that DINO has a tendency to assign a large number of samples to similar representation, while DINO-vMF tends to spread the samples further apart. I would be grateful if the authors could clarify whether this is correct.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear and easy to understand (with the exception of Section 5.2.1, as stated above). I would also suggest the authors to put Equations 8 and 9 in a more prominent position, such right before Section 3.4 (so at the point where they first mention how they alter the logits).

As stated above, I believe the insight provided by this paper regarding the vMF formulation of DINO is original. However, I also believe that the algorithmic novelty derived by this insight in this paper is somewhat limited, since it consists of only a small alteration in the training procedure.

Regarding reproducibility, the authors use hyperparameters already publicly available for the training of their models.
summary_of_the_review: Overall, I feel that this paper provides an interesting interpretation of DINO that leads to an interesting trick to improve its performance. I lean slightly towards acceptance due to the clean addition of this method to regular DINO training, but I believe that the paper can overall be improved due its somewhat limited novelty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper suggests interpreting the recent self-supervised learning framework, DINO, as a mixture model of von Mises-Fisher components. Based on the interpretation, this paper proposes a modification to encourage DINO training to be more flexible and stable. This paper demonstrates the effectiveness of the modification under various experiments.

strength_and_weaknesses: Strengths
- The interpretation is quite interesting, and the proposed modification makes sense.
- This paper is easy to follow and well-written.

Weaknesses
- My major concern is about empirical results.
  - This paper should compare the proposed method, DINO-vMF, with other SSL methods. The comparison with only DINO and MSN is not enough, and it makes some difficulty to check whether the obtained performance is good or not.
  - The obtained gains seem marginal. For example, only 0.7\% accuracy gain on ImageNet using ViT-Base, and no gain when using ViT-Small. Also, there is no meaningful gain in transfer learning (e.g., Table 5).
  - There is no fine-tuning experiment.
- Another concern is that this paper is too limited to DINO. This limited applicability would be critical in this field because there have been developed many SSL methods after DINO, e.g., MAE and iBOT. It would be better if this paper shows that the interpretation can be incorporated into other SSL frameworks.

clarity,_quality,_novelty_and_reproducibility: - Clarity :: This paper is easy to follow.
- Quality :: Empricial results are not convincing.
- Novelty :: I think the proposed method is novel.
- Reproducibility :: Implementation details are provided.

summary_of_the_review: Although the interpretation and the proposed modification is convincing, the empirical results are too marignal and not interesting. Hence I vote for weak rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides an interpretation of DINO as a mixture model with components with a circular von Mises-Fisher distribution.
Using this interpretation, the authors propose a normalization when computing cluster assignments, which improves stability and flexibility of the mixture model.
strength_and_weaknesses: The paper proposes a very interesting interpretation of DINO and the proposed modification increases flexibility of the method by including per-cluster shape parameter. The experimental section includes a number of results to demonstrate the effectiveness of the proposed method.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and authors claim they will release code and pre-trained models.

Minor details:

Section 3.1
- Column notation is not consistent with vector notation in general. Column should be denoted as, for example, $\mathbf{w}^{(k)}$ instead of $W^{(k)}$. In general, notation for vectors and matrices should be improved.

Section 3.2
- $|\mu| = 1$ should be replaced with $\|\mu \|_2$
summary_of_the_review: A useful and well-written paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors focus on the last layer of DINO and show it can be interpreted as a mixture model. The authors multiply the probabilities for the teacher and student networks by a normalizing Von Mises-Fischer constant to improve the network capacity. The reviewers are very positive about the results obtained by the proposed modifications and, the results are well-detailed and reproducible. The discussion between the authors and the reviewers has been very positive and has led to an improved paper.
justification_for_why_not_higher_score: The paper proposed a modification to DINO (Self-distilled Visual Transformer) and the paper's novelty is not enough for an oral presentation.
justification_for_why_not_lower_score: The results are solid and well-explained and it presents novel results on one important topic that is being extensively researched. 

In any case, a presentation as a poster without a spotlight is also a fair outcome for this paper.
Decision: Accept: notable-top-25%