ICML 2025

Simplifying DINO via Coding Rate Regularization
Ziyang Wu 1 Jingyuan Zhang 2 Druv Pai 1 Xudong Wang 1
Chandan Singh 3 Jianwei Yang 3 Jianfeng Gao 3 Yi Ma 1 2 4
Abstract
DINO and DINOv2 are two model families be-
ing widely used to learn representations from
unlabeled imagery data at large scales. Their
learned representations often enable state-of-the-
art performance for downstream tasks, such as
image classification and segmentation. However,
they employ many empirically motivated design
choices and their training pipelines are highly
complex and unstable — many hyperparameters
need to be carefully tuned to ensure that the rep-
resentations do not collapse — which poses con-
siderable difficulty to improving them or adapting
them to new domains. In this work, we posit
that we can remove most such-motivated idiosyn-
crasies in the pre-training pipelines, and only need
to add an explicit coding rate term in the loss func-
tion to avoid collapse of the representations. As
a result, we obtain highly simplified variants of
the DINO and DINOv2 which we call SimDINO
and SimDINOv2, respectively. Remarkably, these
simplified models are more robust to different de-
sign choices, such as network architecture and
hyperparameters, and they learn even higher-
quality representations, measured by performance
on downstream tasks, offering a Pareto improve-
ment over the corresponding DINO and DINOv2
models. This work highlights the potential of
using simplifying design principles to improve
the empirical practice of deep learning. Code
and model checkpoints are available at https:
//github.com/RobinWu218/SimDINO.
1. Introduction
Self-supervised learning (SSL) is the toolkit of choice to
learn representations for large datasets of unlabeled images
1UC Berkeley 2TranscEngram 3Microsoft Research 4HKU.
Correspondence to: Ziyang Wu <zywu@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author...