Occam’s Razor for SSL: Memory-Efficient Parametric Instance Discrimination
Authors: Eric Gan
Patrik Reizinger
Alice Bizeul
Attila Juhos
Mark Ibrahim
Randall Balestriero
David Klindt
Wieland Brendel
Baharan Mirzasoleiman
Venue: Accepted by TMLR ()

Abstract: Self-supervised learning (SSL) is the prevalent paradigm for representation learning often relying on pairwise similarity between multiple augmented views of each example. Numerous learning methods with various complexities such as gradient stopping, negative sampling, projectors, additional regularization terms, were introduced in the past years. These methods can be effective, but they require careful hyperparameter tuning, have increased computational and memory requirements and struggle with latent dimensionality collapse. Furthermore, complexities such as gradient stopping make them hard to analyse theoretically and confound the essential components of SSL. We introduce a simple parametric instance discrimination method, called Datum IndEx as its Target (DIET). DIET has a single computational branch, without explicit negative sampling, gradient stopping or other hyperparameters. We empirically demonstrate that DIET (1) can be implemented in a memory-efficient way; (2) achieves competitive performance with state-of-the-art SSL methods on small-scale datasets; and (3) is robust to hyperparameters such as batch size. We uncover tight connections to Spectral Contrastive Learning in the lazy training regime, leading to practical insights about the role of feature normalization. Compared to SimCLR or VICReg, DIET also has higher-rank embeddings on CIFAR100 and TinyImageNet, suggesting that DIET captures more latent information.

Published in Transactions on Machine Learning Research (12/2025)
Occam's Razor for SSL: Memory-Eficiint Parametric Instance Discrimination
Eric Gan'
eyan&Bucke edu
Computer Science Department, University of Colifornia Los Angeles
Patrik Reizinger
putrik. wewiiienerrruuuuennnnnnn..
Mar Planck Institute for Intelligent Systems, Tübingen AI Center, ELLIS Institute, Tübingen, Germany
Alice Bizeul'
Dicc.bizeelliin ellz.ch
Department of Computer Science & ETH AT Center, ETH Zürich
Attila Juhos
Max Planck Institute for Intelligent Systems, Tübingen Al Center, ELLIS Institute, Tübingen, Germany
Mark Ibrahim
warksibruhimmmett... com
FATR, Meta
David Klindt
Mindttechhl.c edu
Cold Spring Harbor Laboratory
Randall Balestriero
rhalestr@broummeede edu
Computer Science Department, Brown University
Wieland Brendel
wieland. brendel@tuubiigen..yyy
Maz Planck Instátute for Intelligent Systems, Tübingen AI Center, ELLIS Institute, Tübingen, Germany
Baharan Mirzasoleiman
baharan ucla. edu
Computer Science Department, University of California Los Angeles
Reviewed on OpenReview: httpp:///pppeeereeerrree//eeeees
Abstract
Self-supervised learning (SSL) is the prevalent paradigm for representation learning often relying on pairwise similarity between multiple augmented views of cach example. Numerous learning methods with various complexities such as gradient stopping, negative sampling, projectors, additional regularization terms, were introduced in the past years. These methods can be effective, but they require careful hyperparameter tuning, have increased computational and memory requirements and struggle with latent dimensionality collapse. Furthermore, complexities such gradient stopping make them hard to analyse theoret and confound the essential components of SSL. We introduce a simple parametric instance discrimination method, called Datum IndEx as its Target (DIET). DIET has a single computational branch, without explicit negative sampling, gradient stopping or other hyperparameters, We empirically demonstrate that DIET (1) can be implemented a memory-efficien way; (2) achieves competitive performance with state-of-the-art SSL methods on small-scale datasets; and (3) is robust to hyperparameters such as batch size. We uncover tight connections to Spectral Contrastive Learning in the lazy training regime, leading to practical insights about the role of feature normalization. Compared to SimCLR or VICReg, DIET also has higher-rank embeddings on CIFAR100 and Tiny NmageNe,,, suggesting that DIET captures more latent information.
Equul contribution.

---

Published in Transactions on Machine Learning Research (12/2025)
xi
x2
Augmentation (xx)
x3
Encoder ((x)
22
Classification head
Loss C())
Z3
W3
104
Samples
Aug, samples
Representations
Logits
Figure 1: Overview of the Datum IndEx as its Target (DIET) method: The typical pipeline for SSL involves selecting a data augmentation strategy, a model architecture, and defining a loss function along with its corresponding hyperparameters. State-of-the-art [SSL methods often involve complex design choices across all three aspects. In contrast, DIET simplifies this process: the DIET pipeline has only one computational branch, does not require explicit negative sampling, regularization or elaborate techniques such as stop gradients or parameter averaging.
1 Introduction
Self-supervised representation learning (SSL) has become a powerful method for training neural networks without relying on labeled data (Chen et. al. 2020: Misra & Maaten 2020). What makes self-supervised
learning (SSL) possible is solving an auxiliary unsupervised task, enabling to pretrain models on large unlabelod datasots. The different principles include roconstruction-bass mothods such as MAEs (He et al.,
2022a), as well as contrastive (Chen et al., 20200 HaoChen et al. 2021b
Radford al. 2021 Khosla et al.
2020 and non-contrastive methods (Bardes et al. 2021 Chen & He 2021 Caron al. 2021a Oquab et al.
2024 Zbontar et al. 2021). Contrastive Learning (CL) relies on mapping similar samples (called positive
pairs) close to each other in latent space, while embedding dissimilar samples (called negative samples) far from each other. Non-contrastive methods do not have negative pairs, they avoid a collapsed representation via regularization terms. This principle led to many successful applications SSL in the tasks of semantic classification (Chen et al.] 2020). image segmentation (Caron et al. 2021a), and monocular depth estimation Fu et al.) 2018), as well as across diverse data domains, ranging from medical imaging (Eslami et al., 2021 to remote sensing (Tao et al.
2020).
The increasing interest in SSL has led to the emergence of a plethora of methods, each introducing its own variation of the core principles. [SSL. relies on asymmetric computational branches, predictor and projector networks, stop gradients, and many other techniques. Though these might address specific challenges, the field faces many problems. Problems include dimensionality collapse (Jing et al. 2021 von Kügelgen et al.] 2021), when some latent factor is not captured, high compute and memory requirements due to large batch sizes and storing augmented samples (Chen et al. 2020), not to mention the need to carcfully tune various hyperparameters. This additional complexity not only made it harder to navigate the [SSL] landscape in practice, it also confounds the truly essential components of [SST. Furthermore, highly complex methods, even though they might improve performance, are less amenable to theoretical analysis. We aim to build up the simplest [SLL pipeline to uncover the essential components of self-supervised representation learning. We propose DIET
parametric instance discrimination (PID) method that solves classificatio task based on the sample index, i.e., it learns to distinguish each pair of samples. DIET has single computational branch, does not require explicit negative sampling, is robust to perraaaaaeesrs,, and has advantageous learning dynamics with higher-rank embeddings compared to SimCLR (Chen et al. 2020) or VICReg (Bardes et al., 2021p. The inherent limitation of instance discrimination is that the classifier head linearly grows with the dataset size, limiting its scalability. Our insight from the DIET loss is that we can accurately approximate the gradients without keeping the whole classifier in memory, significantly improving efficiency. Additionally, exploit the structure of stateful optimizers such as Adam (Kingma & Ba 2014] for further gains we call this version Scaled DIET (s- [ We provide thcoretical insights on a parancterized feature model, pinpointing the positive ellect of feature normalization and demonstrate the feasibility of (scaled) DIET Empirically, [IET offers a simple yet state-of-the-
alternative to existing SSL baselines on small-scale datasets, providing practical value for practitioners working with specialized, limited data. The modifications introduced in (sDE enable DIET to scale more effectively to larger settings such as CIFAR-100 and Tiny TeageNt while preserving its advantages on small-scale datasets. Our contributions are:
2

---

Published in Transactions on Machine Learning Research (12/2025)
We propose Datum IndEx as its Target (DIET) simple parametric instance discrimination (PID)
method for self-supervieed representation learning $ 3):
We provide theoretical insights into the behavior of DIET in comparison to existing SSL methods and pinpoint the advantages of feature normalization
|S 4
We show how instance discrimination can be scaled up in form of [caled DIET (s-DIET) $ 50;
We provide extensive empirical evidence that DIET is competitive on downstream classification with SOTA on small datasets, and it has a higher-rank embedding ($ 6).
2 Background: Why Self Supervised Learning Needs Occam's Razor
The SSL model zoo. SimCLR (Chen et al., 2020) measures the distance between latent representations with cosine similarity, which is scaled by a tunable temperature parameter. DINO (Caron et al.] 2021b) utilizes a stucher-teachhe Vision Transformer (ViT) architecture (Dosovitskiy et al. 2020 to minimize the cross-entropy between the student and teacher probability distributions across K classes. SWaV (Caron 2020) incorporates clustering to assign labels representations, ensuring consistent cluster assignments between data points and their transformed counterparts. MAE (He et al. 2021 uses masking as data augmentation, encouraging the model to learn representations by reconstructing the masked-out information.
et al.
CLIP Radford et al.) 2021 relies on caption-image pairs as self- signal. Although the training pipeline for SSL is consistent overall (Morningstar et al. 2024 cf. Fig. 1), approaches differ regarding data augmentations, model architecture, and the loss function.
SSL is over-specialized. SSL development was mostly driven by industry, thus, focused on large-scale
natural images and sounds (Radford et al. 2021 Oquab et al.) 2024 Siméoni et al. 2025
This led to a
point where methods are archilecture- and dalasel-specific (He et al. 2022b Assran el al. 2023 Oquab et al,
2024.. This overspecialization imposes a high barrier of adaptation:
(i) Uninformative loss w.r.t. the DNN's quality (Reed et al. 2021: Garrido et al. 2022): : as the last few
layers (the projector) are discarded after training, the loss not necessarily indicative of performance;
(ii) Too many hyperparameters: for loss, projector, and augmentations, with hard-to-predict effect on performance (Grill et al. 2020a Tian et al. 2021 He & Ozay 2022):
(iii) Lack of hyperparameter transferability across dalasets and architectures Zhai el al. 2019
Cosentino et al.) 2022);
(iv) Heavy code refactoring, compared to supervised models, c.g., for generating positive pairs, handling
asymueetrcc computational branches, and parameter moving averages (Grill et al. 2020b Caron el al.,
2021bb.
This makes SSL implementation more costly than supervised learning, often requiring distributed training and long training schedules that reduce the accessibility and inclusivity of SST. research (Crowell 2023
3 Datum IndEx as its Target (DIET): a simple SSL method
Motivation: A Simple SSL Method. Recent SSL methods rely on various design choices and techniques to structure learned representations while avoiding representation collapse, including regularization terms, specialized architectures, and specific data transformations, leading to a multitude of sensitive hyperparameters that require careful tuning. This is a barrier to practical adaptation and theoretical study. In contrast, instance discrimination formulates SSL as cross-entropy maximization over instance labels, i.c., "learning by distinguishing individual data points within a dataset." " As we will show, this provides simpler SSL pipeline, has incentives to capture more information, and is more amenable to theoretical study.
Intuitition. SSL often requires large batch size to provide accurate entropy estimates in high dimensions to maximize the uniformity loss (aag & Isola 2020 Zhai al. 2023 The batchwise perspective is not only limited by GPU memory, but also by how it changes the underlying problem. Contrastive objectives such as InfoNCE/SSmCLR can be thought of as solving an underlying classification problem, discriminating the positive pair from all negative samples, given the anchor sample. This formulation only constrains relationships between data samples in the batch. This makes it possible that (negative) samples not in a batch have very similar representations to the anchor or posítive sample-uniee uunllss the batch size equals the dataset size. On the other hand, instance discrimination requires distinguishing each pair of data samples, eliminating the above failure case. Intuitively, this incentivizes capturing more information, as, e.g., two
3

---

Published in Transactions on Machine Learning Research (12/2025)
images of similar dogs need to be distinguished, which might be possible by picking up subtle variations such as in fur color.
3.1 The DIET method
Instance discrimination focuses on distinguishing individual samples within a dataset by treating each sample as its own class. Alexey et al. (2015) introduced parametric instance discrimination (PID) where they constructed surrogate classes via an elaborate gradient-norn basee strategy and designed class prototypes as
trainable parameters, whereas Wu et al. (2018) introduced a non-parametric alternative, where prototypes are selected from memory bank of previously observed samples. We adopt PID with a nonlinear backbone (encoder) fo with parameters 0 and linear projection WH. We use the sample (datum) index the classification label and call our method Datum IndEx as its Target (DIET) We optimize the cross-entropy between the probability distribution of a sample's predicted and ground truth indices:
(DIET(E,)) = XEnt i
yn n))
(1)
where H, E RDD denotes inpuls, Vn the corresponding label, which equals, n, i.e, the dataset index. The encoder so maps inputs to latents 2n fa(...) The learnable projection matrix WH then maps zn to logits corresponding to the dataset indices.
The simplicity of DIET boils down to (cf. Appx. C.1 for pseudocode):
(i) No explicit negative sampling: DIET relics on WH to ensure distinct reprosentations for cach
sample instead of negative sampling.
(ii) One computational branch: as the index contains the information that augmented views of the
same sample belong together, there is no need for two computational branches;
(iii) No specialized solutions: no stop gradients, asymmetric computational graphs, exponential moving averages and hyperparameters make [DIT simple.
The astute reader might notice one limiting factor in DIET WH E Nxd grows proportionally to the datasct
size N. However, this limitation can be overcome, as we show in 5
4 Theoretical analysis
Our main argument for DIET in 3 was its simple pipeline. However, these simplifications also make the connection to other popular SSL methods non-obvious. Thus, we present thcoretical analysis to show that under some assumptions, the instance discrimination loss of DIET has the same minimizer as the popular InfoNCE method (Chen et al. 2020 Zimmermann et al. 2021a). Our formulation also suggests that if we replace the cross entropy loss function in (1) with a Mean Squared Error (MSE) loss, then this MSE-DIET loss has the same minimizers as the Spectral Contrastive Learning (SCL) loss (HaoChen et al., 2021b). These results indicate that for the theoretical guarantees of these methods, not all bells and whistles are necessary, as the much simpler DIET algorithm can learn the same representations. Last, our formulation also enables to better understand (heoretically why feature normalization can yield better representations (S 4.2
4.1 A Framework Connecting Pairwise SSL Losses and Instance Discrimination
Given model f Rd + Rm and dataset D, many SSL losses in the literature are defined based the pairwise similarity between embeddings 2j = f(x,) of samples from the dataset (Chen et al. 2020 Chen
& He 2021 Grill et al.] (020a)- often instantiated as the cosine similarity, which for unit-norm vectors is equivalent to the inner product x z2. We call such losses pairwise similarily losses and denote them by (D; f) = 4ps ((z] 22)). This is in contrast to instancewise losses Cin () = b E(u.v) 4ix (f(x), y) defined as the average of a loss applied on each sample against labels Y, which are more common in the broader machine learning literature.
While pairwise losses avoid an explicit dependence on labels, their specialized construction has made SSL difficult to analyze theoretically, and prevents the direct application of tools from the broader literature aimed at addressing instancewise losses. We bridge this gap by developing a connection between models trained with pairwise loss and models with a single additional lincar projection trained with an instancewise loss and labels y that are simply the datum index. Relying on the invariance of the inner product in the SCL loss up to orthogonal transformations, empirical and theoretical results about the linearity of feature spaces learned
4

---

Published in Transactions on Machine Learning Research (12/2025)
by neural networks (Roeder al. 2020 Reizinger al. 2024 Park et al. 2023 and the invariance of linear
probe performance to invertible linear transformations (HaoChen et al. 2021a), we assume that the inner products are preserved by the projector, i.e. that the projector is -ottinonn.ttoona.......
Definition 1. Let H be a hypothesis class and D = { a dataset. For pairwise loss function Pes define the optimization progrom
min (P; JEH
(2)
We call instancewise loss Lin with instance labels y' Hykee instancewise equivalent of Lpe if, for model Wnf constructed by appending linear layer WH E mRx to the base model f, the optimization program
min JEN, WH
(D, WHf)
satisfies the following
Wn) is a minimizer 3 and WH is columm-orllhogoade then f is minimmzer of f is a minimizer 2 then there exists WH such that WH is column- column- orthogonal and (f. WH) is a minimizer of
The motivation behind the above definition is to reframe a pairwise loss as an instancewise loss by simply adding a linear projector, potentially opening new avenues of theoretical and empirical analysis. Our main theoretical result is to show that some commonly studied pairwise SSL losses have natural instancewise equivalents.
InfoNCE Loss. The InfoNCE objective is the most well-studied pairwise loss and the basis of the popular SimCLR method (Chen et al. 2020 Zimmermann et al. 2021a). Since SimCLR uses unit-normalized reprosentations, which have been shown to provide better performance, we consider the hypothesis class of functions that produce representations on the unit. hypersphere H = (ʃ: Rd + SID -1). Under this setting, we find that the instancewise equivalent the InfoNCE loss is the DIET loss from 3
Theorem 1. For the hypothesis class of unil-normalized embedding functions, equivalent of the InfoNCE loss.
DIET is an instancewise
Spectral Contrastive Learning One of the most well-understood pairwise losses in theoretical analysis is the spectral contrastive loss (SCL) (HaoChen et al. 2021a where ô is the Kronecker delta.
LScr. (m1.))) (m2.3))) [6 (x2)] + (. [[(()))
E
(4)
Just as SCL be viewed simplification of the InfoNCE objective by dropping the softmax objective, we call also define an MSE-DIET loss with one-hot encoded labels.
MSE
2 ER [[[,,.( - e;
(5)
We assume f is a parametric feature model f(x) = W (xx) constructed by composing a fixed, potentially high-dimensional and non-linear feature map Φ with a learnable linear operator W E m with m I N. This sctting captures the lazy training or ncural tangent kernel (NTK) regime of neural networks which is common in thcoretical analysis and accurate for neural networks in the infinite width limit (Jacot al. 2018). Tn this setting, again find simple instancewise equivalent.
Theorem 2. For the hypothesis class of parametric feature models, MSE-DIET is an instancewise equivalent of the SCL loss.
The proofs are presented in Appx. A.2. These results indicate that, at least in a simplified setting, simple instancewise losses can accomplish the same as the more complex pairwise losses. With DIET and MSE-DIET being inslancewise equivalents to the well known SSL losses, this justifies the exploration of parametric instance discrimination as an alternative approach to SSL. From a wider perspective, this creates a rigorous connection between the theory on SSL losses, which has been largely independent and self-contained, and the broader machine learning literature, which focuses primarily on instancewise losses. We provide empirical evidence of the given equivalences in S 6 with additional validation in Appx. D2.7
Equation differs from some previous definitions by few constant factors. This does not affect any of the analysis, cf.
Appx. A.I.2
5

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: DIET trained on small datasets achieves similar accuracy to Imagenet pre-trained SSL for numerous small-scale datasets. Benchmarks are taken from
Yang al. (2022).
Ericsson al. (0211)
Arch.
Pretrain
Method
Aireraft
DTD
Pets
Flower
CUB-200
Food101
Cars
IN100!
SimCLR
24.19
54.35
46.46
75.00
16.73
Resnel18
DIET
37.29
50.62
64.06
72.01
33.03
62.00
42.55
SimCLR
44.90
74.20
83.33
90.87
42.74
67.47
43.73
SimCLRv2
46.38
76.38
84.72
92.90
52.78
73.08
50.37
McCov2
41.79
73.88
84.00
90.07
43.81
71.63
39.87
BYOL
53.87
76.91
89.10
94.50
52.14
73.01
56.40
IN-1K
VICReg
53.41
76.12
89.45
93.72
62.37
75.59
61.51
Resnet50
SimSiam
5.97
53.03
62.17
57.93
15.34
35.45
0.85
DeepClusterv2
54.49
78.62
89.36
94.72
59.06
77.94
58.60
Swav
54.04
77.02
87.60
94.62
54.14
76.62
54.06
DIET
44.81
51.75
67.08
73.32
41.03
71.58
55,82
SwinTing
DIET
33.15
51.88
58.06
70.78
32.11
8,86
47.12
Commentt-8
DIET
43.13
9.52
61.72
67.72
31.44
69.84
40.63
4.2 Learning More Features via Normalization
Zimmermann al. 2021b). We seek understand how normalization affects the features learned by DIET. For this, analyze feature learning with content and style latents (von Kūgelgen et al. 2021) in a variant of sparse coding that is common in feature learning (Wen & Li, 2021 Zou et al., 2021] Chen et al. 2023 Xue et al. 2023 cf. Appx. A.1.4 for details.
Normalizing features to the hypersphere is common SSL
We model the interaction of content and style concepts in simple additive model. Let C (I...., C) label set of latent concepts. To each E C we assign content (low noise) feature ue and a style (high noise) feature Ve. For a cat, a content feature could be ear shape (almost always pointed one), while style feature could be fur color (often highly varying between breeds), with Ve, U E R We assume all u, and on are orthonormal, the feature noises Cu: fy are drawn from symmetric, zero-mean distributions p(su); p(c)) with variances ou og and bounded support such that for Va, 1/ < 1, |u| S Va and |al S ve, while the background noise & is drawn from Gaussian distribution scaled by some parameter y.
<(x) = (1 + uu)uu + (1 en e. + Ei
where e E C. We define data augmentation A to replace the noise components eu: Ev, with a different. realization from the same distribution. By studying the standard and normalized MSE DIET losses, we can prove that only normalized DIET captures both features (proof is in Appx. A.3):
Theorem If W minimizes
MSE and WN ménimizes NORM then
CSS
vell 02 W
1- Wxeell
1 + Vh
o2 + oo()
Wuell
W
1 1+
Wy
Vv
Thm. shows that the smaller variance (content) feature u implies a small alignment between the weight matrix W and the style feature Ve at the optimum of CMSE Thus, DIET may fail to learn style features if content feature is present in line with similar result for contrastive learning from |yon Kügelgen al.
(2021) Thm. 3 characterizes the alignment quantitatively, supplementing the qualitative non-identifiabiltty result of von Kügelgen et al. (2021). Informally, in the unnormalized model, the larger noise from the style feature introduces larger loss, so to minimize the loss the model ends up focusing primarily on the lower noise content feature. In contrast, normalization introduces an additional dependency between the directions, which has the effect of balancing the learning between the directions. We show that normalized DIET learns both fcatures approximately equally so long as the noise does not significantly corrupt the features. For example, if the noise ratio bounded by Va, Ve S 2, then the alignment with the style feature and the content. feature will differ by at most factor of 3. We validate these findings in

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: DIET achieves higher validation accuracy on medical datasets than SSL with standard hyper-parameteen The supervised model is pretrained on ImageNetlk and a linear probe is trained on top of fixed representations.
Architeclure
Pretraining
Method
DermaMNIST
BloodMNIST
PathMNIST
SimCLR
66.88
14.56
11.80
MoCov2
66.88
53.70
18.97
BYOL
65.89
80.56
65.68
Resnet18
VICReg
66.78
47.18
11.31
SimSiam
66.88
43.23
17.17
DIET
73.92
89.24
44.53
s-DIET
76.71
98.16
84.78
IN-1K
Supervised
74.06
88.13
59.37
5 Making DIET Memory Efficient
5.1 Batch Cross Entropy
DIET's classificr head WH scales with the number of examples, limiting scalability due to memory requirements. For our insight, consider the gradients w.r.l. We (the derivation is in Appx. A.4
B
V un .DITT
Σ 1=1
UCA p(k(zi))
(6)
B
where V is the ith component of the true, one-hot label of sample k, and p(k|z;) is the predicted class probability distribution, given the embedding of sample and only the logits corresponding to samples in the batch appear. Thus, for a batch Xr with elements and with indices I [N], let WH [Z] e Bxd collect. the jth row of WH, for all E I. We hypothesize that this subsampling provides reasonable approximation of the gradients of all parameters 0:
VeXEnty (WHfe(X))) & VoNEntg(Wn([TToC
(7)
Instead of calculating standard cross entropy on the N-dimensional outputs of WH, we select its B rows corresponding to the indices in the batch, reassign samples a distinct label from (O,..., B 1} and calculate a B-dimensional batch cross entropy. This can be interpreted as batchwise instance discrimination. For empirical validation, cf. 7.3 for an illustration, Fig. 2
Memory savings: an illustration on ImageNet. Batch cross entropy requires only that B rows of W are memory, which in the standard dim >> dim case provides a small memory cost compared to loading the data. For example, 256 ImageNet (Deng et al. 2009) images with 224 224 resolution require 150 MB per batch, opposed to only MB for the 2048 dimensional classifier.
5.2 Multi batch crossentropy
built around the batch cross entropy loss, there is generalization, where we decouple the batch size from the number of rows of W H loaded. Instead of only loading the rows of WT that correspond to index set I, we would actually load a nunber of heads corresponding to m B samples from an index Im- where m is chosen arbitrarily large. In this case, would make that the upcoming batches, each with B elements, cover exactly, without overlap hiatus, the training datapoints from Imn-
Although experiments
Wy
:
Wrpp
I= (f) Exi
:
(f(x))f1
WN
Figure 2: Batch cross entropy: The index set I collects all sample indices that are in the current batch. We then use I to select which rows of WN to load into memory, decreasing the memory footprint
6 Experiments

---

Published in Transactions on Machine Learning Research (12/2025)
1.6
Memory Usage Rel. to SimCLR
SimICR Beseliie Barlow Twins
SimCUR
Sctup. We perform experiments on toy, synthetic, and 4 real-world datasets: CIFAR-10, CIFAR-100 (Krizhevsky)
1.4
Simsiam
BYOL
DIET
et al. 2009,, ImageNet-100 (Tan et al. 2020). and Tinylm-
1.2
S- DIET
ageNet (Le & Yang 2015). For our models, we study the
1.0
ResNet. larily of architectures, specifically ResNet-18 and
ResNet-55 (He et al., 2016), and vision transformers (ViT)
0.8
Dosovitskiy et al., 2020), specifically ViT-B/16. ResNet-18,
0.6
ResNet-50, and V/t-B/16 have embedding dimensions 512, 2048, and 768, respectively. We use a three-layer ReLU MLP as a projection head during the training of s- S-DIT We fix the default label smoothing to 0.8 and the data augmentation pipeline to a combination of cropping, flipping, color jitter and gaussian blurring. Details on data augmentation are pre- sented in Alg. 3) We use an AdamW optimizer with a 10-3 learning rate and 0.05 weight decay with cosine learning rate decay. We fix the batch size to 256 for all experiments and train DIET and s-DIET for 5000 epochs. Our baselines were trained until convergence using the same data augmenta- tion as for DIET, All baseline syper-parameteed were kept to the default values proposed by the original works. After training, we evaluate our representations by training a linear classifier on top of frozen representations perform semantic classification on the validation set.
0.4
0.2
0.0
CIFAR-R18
CIFAR-R50 IN100-R50
TIN-R50
Figure 3: GPU Memory Comparison rel- alive to SimCLR with a batch size of 256. OOM indicates out-of-memory on Nvidía A40 GPU. DIET requires 2x, SimCLR up to
2.2x more memory than S-DTTT Absolute val- ues are deferred to Tab. 15
6.1 An edge on small datasets
Transfer learning. We investigate how a trained--from-scratch DIET performs on small datasets that are commonly handled by SSL through transfer learning: Aircraft MMaji et al. 2013), DTD (Cimpoi et al.) 2014), Pets (Parkhi et al. 2012),, Flowers (Nilsback & Zisserman 2008). CUB200 Wah et al. 2011), Food101 [Bossard et al. 20144), Cars (Krause et al. 2013). These datasets have much fewer samples than Imagenet but their image distribution is often much less diverse, e.g., focusing only on aircraft. The current. SOTA is to pretrain one's favorite SSL method on larger dataset such as Imagenet100 or Imagenet-Ik 1kk and to fine-tune on the target dataset. But pretraining over large, uncurated datasets can introduce risks such as data poisoning or bias amplification, which are critical to avoid in high-stakes scenarios (Zhang et al. 2024j. Porhaps surprisingly, DIET provides an alternative without pre-training i.c., by training directly on the small dataset this can be leveraged scenarios where light control over the data is required, e.g., to avoid data poisoning. [Tab. I] shows that DIET matches or surpasses SimCLR pre-trained on ImageNet-IK across three of the evaluated transfer datasets. When compared to SimCLR trained on IN100, DIET consistently outperforms it-often substantially. These findings suggest that DIET can serve simpler yet competitive alternative to more complex and less interpretable methods in small-scale set tingings.
DIET is SOTA beyond Natural Images. Medical datasets generally have very few samples as such data is notoriously hard to collect. Furthermore, pre-training ImageNet is less sensible as the data distributions dilfer significantly. Thus we compare SSL methods (DIET, SimCLR, MoCov2, VICReg) trained from scratch on three datasets from the MedMNISTV2 medical imaging benchmark (Yang et al. 2023) (i) 180 train/ best split); (ii) DermaMNIST (10, 015 2, 005 split); and (iii) BloodMNIST
PathMNIST (90, 000- 180 (17, 0920092222 3, 421 split). For SimCLR, MoCov2, VICReg, we use the default hyperparameters from Susmelj ct al. (2200) which yield good performance (> 80%) on CIFAR10, comparably small dataset of 60, 000 images. All algorithms achieve high training accuracy via a linear probe, but the baseline SSL methods do not generalize well to the test sets
(Tbb. 2p. By contrast, DIET achieves much higher performance.
For ablation for DIET with VIT, see Appx. C.S and training curves in Fig. 16 showing that DIET's hyperparameters transfer. DIET also has a speed advantage: for ResNet18, DIET is 1.75x faster than SimCLR (and 1.72x faster than VICReg). These findings provide strong practical guidance for SSL practitioners: On small-scale in-the-wild datasets often characterized by distribution shifts from standard image benchmarks- DIET serves as a simple yet effective alternative for achicving statc-of- the-art performance. BYOL is a strong baseline, and on the largest of the medical examples we consider, PathMNIST, it even outperforms supervised learning.
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: S-DIET
achieves higher accuracy than existing CL methods by up to 2.72%. Linear probe accuracy of DIET against DIET and various SSL baselines on CIFAR-10, CIFAR-100, ImageNct-100,, and TinylmageNet. DIET
obtains otate-of-the-art with limited GPU memory.
Method
CIFAR-10
CIFAR- 100
ImageNet-100 ResNet-50
TinylmageNet ResNet-50
Re
ResNet-50
ResNet-11
ResNet-50
SimCLR
90.00
91.64
63.56
67.90
79.68
46.32
MoCov2
81 .00
82.53
63.75
68.10
70.90
38.81
BYOL
90.76
92.32
65.26
68.10
(OOM)
40.72
VICReg
91.15
92.67
66.76
70.11
43.05
Simsiam
90.78
92.42
65.66
69.62
80.12
40.48
DIET
54.64
89.70
62.93
68.96
73.50
51.66
>DIET
91.48
93.08
66.88
72.34
80.16
52.52
6.2 Scaling DIET to Large-Scale Natural Datasets
While DIET achieves near state-of-the-ar performance on smaller datasets like CIFAR-10/-100 (50,000 samples), its original formulation begins to show limitations when scaled to more challenging datasets such as ImageNet-100 and TinylmageNett (Tab. 3). On ImageNet-100, DIET struggles to match SOTA, while on longer memory efficient due to the larger number of samples which directly impact the size of WH Fig. 3 In these scenarios, we use batch cross entropy (S 5 to improve the memory efficiency
Tin DIET
of DIET while adding representation normalization to improve the feature learning ability of DIET A thyee-layer MLP projection head is added during training and removed at evaluation, following prior findings that this improves the
$ 4.2
Table 4: Training time of 8- S-DIET versus SimCLR on CIFAR 10/100, in hours, on a single NVIDIA A5000 GPU. Although more memory efficient,
learning efficacy of self-supervised methods (Bordes et al., 2022 Xue
et al. 2024). The full S-DIET algorithm is summarized in Apdx. D.1
This scaled version DIET simplicity of DIET and practical performance: [ 3-DETT more than 2x more memory efficient than DIET TinyTmageNet and up to 2.2x more memory efficient than other SSLL methods, while outperforming SSL baselines by up to 2.72% points. The compromise is that 3-DITT
(sDTIT achieves balance betweeen the
DDET compromises training time.
Method
Model
Training Time
s- D-ET
18
16.2
is slower to converge |S 6.2). Improving the convergence rate of SDIET is an interesting direction for future research. For an ablation with a ViT backbone, cf. Tab. 21 Tab. 3] we find that -DDIET matches and outperforme popular SSL methods for ImageNNe-1000
SimCLR
ResNet-18
3.9
+- DDETT
ResNet-50
51.0
SimCR
ResNet-50 11.5
and TinylmageNet respectively. For even larger datasets such as ImageNet-1k, DIET is completely infeasible
due the size of WH, so S-DIET must be used to avoid memory inefficiencies. However, we find that the slow convergence rale of a DIET is limiting factor, especially in resource limited scenarios: (DDITT achieves 52.01% linear probe accuracy after 500 epochs, whereas SOTA SSL methods can achieve around 65% accuracy using the same number of epochs.
7 Ablations
In the previous ions, we evaluate DIET and its extended variant, -DIET on standard semantic classifica- tion benchmarks. We now shift focus to a more general unsupervised evaluation the learned representations.
7.1 Improved Feature Learning
Toy Sctting: clean and noisy features. We use the setup presented in S 4.2 with 4 latent classes and noise parameters on 0.01, a, 0.1 (for details, cf. [Appx. D.4j and compare the alignment between the weights and the clean and noisy feature of the first class. As illustrated in Fig. 4), normalization enables the model to learn both features, whereas, without it, only the clean feature is learned.
The effect of normal- ization on downstream classifi- cation accuracy in CIFAR-100.
Table
Normalization Accuracy
Yes 5-DIET
66.88
No DIET
62.60
Normalization Increases Embedding Rank. Interestingly, we show that the normalization applied in DIET further improves the singular value spectrum of DIET embeddings, resulting in representations with
9

---

Published in Transactions on Machine Learning Research (12/2025)
1.0
with norm
0.25
0.8
0.20
alignment
s[i] s[O]]
0.6
0.15
1.00
0.10
clean feature alignment
clean feature alignment noisy feature alignment
0.05
noisy feature alignment
0.00
1000
2000
5000
4000
5000
0.00
1000
2000
3000
4000
5000
100
200
300 400
500
step
index
Figure 4: Normalization leads to richer representations. (left & middle) Alignment of weight
matrix W with clean feature 4 and noisy feature V, (calculated as [uull and ||W respectively) when using DIET (left) and normalized DIET (right). DIET only learns the clean feature but normalized DIET learns both features almost equally. (right) Singular values of representations for CTFAR-100 are sorted decreasing order. Values are normalized by the largest singular value.
even higher rank Tab. 7j. We also confirm that these enhancements translate into richer representations, in line with prior findings (Garrido et al. 2022 Thilak et al. 2024), and they also translate into improved classification accuracy on CIFAR-100
Tab. 5).
Combined MNIST and CIFAR-10. We construct synthetic dataset modelling the data generation process from $ 4.2 where each input example consists of CIFAR-10 image and MNIST image of the same label index concatenated along the channel dimension akin to the design of Shah et al. (2020); Chen et al. (2021). We use weaker augmentations on the MNIST image, making the MNIST image the content and the CIFAR-10 image the style feature (for details, cf. Appx. D.5l. We train a ResNet-18 using DIET with and without normaliza- tion. During linear probe evaluation, we may mask the MINST digit to compare how well the models learned the CIFAR-10 image. We observe in the inset table that DIET quickly overfits the MNIST digit, even when it is masked, indicating that the CIFAR-10 features are not well learned. Normalization main- tains high performance regardless of whether the MNIST digit is present, showing that the CIFAR-10 features are learned.
Table 6: The effect of masking on down- stream classification accuracy on the com- bined MNIST and CIFAR-10 dataset.
Normalization
No Masking
Masking
Yes s-DIET
83.9
84.06
No DIET
13.76
43.56
Higher-rank inputs can improve downstream linear classifier performance (Cover 1965). which inspired recent.
works to propose the lank-based RankMe (Garrido et al. 2022) and LiDAR TThilkk et al. 2024] metrics
to evaluate the embedding rank in SSL. (Garrido et al. 2022 Thilak et al. 2024 find that these metrics strongly correlate with downstream accuracy.
DIET learns high-rank embeddings. As real-world datasets do not grant access to ground-truth features, we also adopt a proxy- based evaluation using rank-bace based metrics and track singular values of the learned representa- tions, as in (Xue al.] (0022). In Tab. 7 we compare the RankMe Garrido et al. 2022 and LiDAR (Thilak et al. 2024) scores CIFAR- 100 and TinyImageNet. We find that, despite its simplicity, DIET improves dimensional col- lapse, a challenge typically addressed with bells and whistles, resulting capturing a richer set of features. In addition, we show that the singular values of DIET|s embeddings converge faster to narrow range, whereas [SLL base- lines converge slower and span wider range, showing DIET's clear advantage (c.f., Fig. 11)
Table 7: DIET produces high-rank embeddings: DIET achieves substantially higher RankMe and LiDAR scores using ResNet18 architectures.
Dataset
Method
RankMe (1)
LIDAR (t)
DIET
499.58
479.21
SimCLR
355.05
326.58
VICReg
422.28
377.37
CIEAR100
MoCov2
313.49
309.17
SimSiam
441.52
308,88
BYOL
261.42
222.38
DIET
318.38
414.88
SimCL.
365.87
343.13
VICReg
408.02
391.60
VinylmageNe
MoCov2
335.73
394.33
SimSiam
419.42
294.21
BYOL
238.04
208.03
10

---

Published in Transactions on Machine Learning Research (12/2025)
7.2 Experimental Validation of Thm. 5
We train a model using SCL and MSE-DIET on the toy dataset described in 4.2 and Appx. D.4 We then convert the MSE-DIET model into an equivalent model where the head is column-orthogonal as described in [emma and then we compare the procrustes distance between the embeddings produced by the SCL and equivalent DIET model. Figure P shows that the procrustes distance vanishes, indicating that the models learn the same embeddings up to an orthogonal transformation. This finding is especially interesting in light of our loss ablation in Appx. C and [Tab. 8] where the MSE and cross entropy formulations the DIET objective show substantially different classification performance. This finding is in line with the known trade-off between the richness of the representation and downstream performance (Rusak et al. 2024).
procustes distance
-2-
3000
2002
3005
400
500
step
Figure 5: Procrustes distance ween embeddings learned by MSE-DIET and SCL vanishes.
7.3 Sensitivity analysis
Finally, we explore the sensitivity of DIET to its remaining
Loss Function. We compare the performance of using mean-squared versus cross entropy loss on CIFAR-100 (Tab. 8). Although our experi- mental validation of Thm. 5 shows that the embeddings learned by the MSE-DIET and SCL objectives have a vanishing procrustes distance, this similarity does not necessarily transfer to similar downstream per- formance. Namely, we find that the cross-entropy loss provides better performance in practice. This is consistent with standard practice in supervised learning.
Table 8: The effect of the loss on downstream classification accuracy.
Model MSE CE
RN18 58.21 66.88
CIFAR100 RN50 64.44 72.34
Batch size. We investigate the effect of batch size on TinyImagenet and report the accuracy in Tab. 9
Remarkably, the performance DIET is stable across a broad range of batch sizes, and even as low as 16 causes a relative performance drop of only 5%. Similar conclusions are drawn for s-DIET Fig. 8).
Batch Cross Entropy. To confirm that batch cross entropy closely approximates standard cross entropy, calculate the cosine similarity of base model gradients (i.e., excluding the projection head or classifier head) of randomly initialized models on C1FAR-100 for the base model. Tab. shosowws that the cosine similarity between the gradients batch cross entropy and full cross entropy is nearly across the board, with higher cosine similarity for larger models and batch sizes.
Table 9: The effect of batch size on down- stream classification accuracy on Tinylma- genet (3000 training epochs).
batch sizel 16 32 64 128 256 512 RNI8 37.994.7743
03.773.44 43. 43.74.37
Label smoothing. Finally, we investigate the effect of label smoothing (LS) on downstream performance and observe that apply- ing LS with values between 0.4 and 0.8 significantly accelerates the convergence rate of DIET As a result, label smoothing also enhances the performance of DIET as much as 5% points when training for a fixed number of epochs, as shown in Fig. 14
Table 10: Cosine similarity of gradi- ents for CE and batch CE with ResNet models on CIFAR-100. Batch CE ap- proximates CE.
Batch Size
RN18
RN50
0.9944
0.9960
Projector network. A projector network often used in SSL
128
0.9965
0.9980
256
0.9975
0.9990
to improve performance (Chen et al. 2020: He et al., 2022b). We observe that l-yayer ReLU MLP as projector also improves linear (Tb.. 11}.
512
0.9980
0.9995
probe accuracy for s- DIET on CIFAR-100
Table 11: The effect of the projector network on linear probe accuracy on CIFAR-100.
Data Augmentations. Previous works have emphasized the impor- tance of data augmentation (DA) for the success of SSL (Balestriero ct al. 2023 Morningstar et al., 2024 Ciernik et al.) 2024).. Thus, we consider three DA regimes: Low only includes random crops and hor-
Model Pre-project. Post project
RN18 RN50
66.88 72.34
63.46
izontal flips; Intermediale further adds color jittering and grayscaling:
67.60
11

---

Published in Transactions on Machine Learning Research (12/2025)
and High further adds Gaussian blur and random erasing [Zoog et al. 2020 for the exact setup, cf. Alg. 3 Tab. shows that on TinyImagenet (for ablations, cf., Fig. 15 and Tab. 16) DIET greatly benefits from intermediate DA, however, the high regime does not have a large further improvement.
8 Discussion
Our work focuses on understanding the simplest set of components that make SSL work, for which we introduce Datum IndEx as its Targel (DEET) a parametric instance discrimination (PID) method. [IET has only one computational branch and roquires no explicit negative sampling other specialized techniques such stop gradients. In a simplified linear model, provide theoretical insights about how feature normalization can help recover more features in the presence of content (lower variance) and style (high variance) features and investigate connections to the well known InfoNCE, and SCL, objectives from contrastive learning. To improve memory efficiency, we introduced a batched cross entropy strategy based on analyzing the gradients of DIET providing a scalable version of the algorithm.
Table 12: The effect of data augmentation strength on downstream classification ac- curacy in CIFAR-100. Refer to the text for details
DA
Low
Inter.
High
RN18
31.48 43.62
43.88
RN50 40.21
48.80
50.81
RN101
40.07
49.74
50.76
Through extensive evaluation, we show that DIET offers state-of-the-art results over other SSL methods on small-scale datasets. We also demonstrate DIET's memory efficiency on ImageNet-100 and Tiny ImageNet and find that DIET learns higher-rank embeddings, corroborating our insights about the role of feature normalization. As SST continues to be adopted across wider range of tasks and domains, DIET offers simple yet effective approach for real-world applications, requiring minimal tuning while learning rich representations in small-scale settings.
Acknowledgments
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Patrik Reizinger and Attila Juhos. Patrik Reizinger acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: D1IS18039A. Wieland Brendel acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. Wieland Brendel is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This rescarch utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. Alice Bizeul's work is supported by an ETH AI Center Doctoral fellowship. Baharan Mirzasoleiman was partially supported by the National Science Foundation CAREER Award 2146492, the NSF-Simons AT Institute for Cosmic Origins (CosmicAI), Institute for Foundations Machine Learning (IFML), and an Okawa Research Award.
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
Dosovitskiy Alexey, Philipp Fischer, Jost Tobias, Martin Riedmiller Springenberg, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Trans. Pattern Anolysis and Machine Intelligence, 99, 2015.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-Supervised Learning from Images with Soint-Embedding Predictive Architecture, April 2023.
URL http::/axxiv...gg//aaaaaaaaaa wttp://wrrggaab///..00222
arXiv:2301.0822 (cs, cess].
Pranjal Awasthi, Nishanth Dikkala, and Pritish Kamath. Do more negative samples necessarily hurt in contrastive learning?, 2022. URL
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A Cookbook of Sell-Supervised Learning, April 2023. URL http://arraboog//a//332..22
arXiv:2301.22110 [cs].
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-iisariannccccowaaaa regularzzaiinn for self-supervised learning. Xiv preprint arXiv:2103 1102220000.400 005004906, 2021.
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In International Conference on Machine Learning, pp. 517-526. PMLR, 2017.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Improving deep networks generalization by removing their head. arXi preprint arXiv:2206. 13378, 2022.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of
visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132- 112-149, 2018
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912 9924. Curran Associates, Inc., 2020. URL httpsssspppppeeccnnserrrr.ppoopp
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVP International Conference Computer Vision, pp. 9650-9660, 2021a.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar-
mand Joulin. Emerging Properties in Self-Supervised Vision Transformers. pp. 9650 9660, 2021b. URL
http://pp..ccee....ooooooooooooo wttp:tteennnccccc//eeeeeee httpimeeerr_cc__ee_ee___ ICCV 20t___222_1pppeeerraaaa
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Ting Chen, Calvin Luo, and Lala Li. Intriguing Properties of Contrastive Losses. orXiv:2011.0280 [cs, stat], June 2021. URL httt:://axxiv.org////// httpor/gg/////a...eeee
arviv: 2011.02803.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 15750 15758, 2021.
Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Understanding and improving feature learning for out-of-distribution generalization, 2023. URL
Laure Ciernik, Lorenz Linhardt, Marco Morik, Jonas Dippel, Simon Kornblith, and Lukas Muttenthaler. Training
objective drives the consistency of representational similarity across datasets, November 2024. URL http: / //arxiv.org/ abs/ /ata//2411.5555 arXiv:2411.05:5 [cs].
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Procerdings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Romain Cosentino, Sarath Shekkizhar, Mahdi Soltanolkotabi, Salman Avestimehr, and Antonio Orlega. The geomelry of sell-supervised learning models and its impact on transfer learning. arXiv preprinl arXiv:2209.08022 2022.
Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, (8):326-334, 1965.
Rachel Crowell. Why al's diversity crisis matters, and how to tackle it. Nature, 2023.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learm: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1 6, 2022.
URL
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXie preprint arXiv:2010. 100:2000.1199,, 2020.
Yann Dubois, Tatsunori Hashimoto, Stefano Ermon, and Percy Liang. Improving self-supervised learning by characterizing idealized representations. arXio preprint urXim:2209.0080 2022.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5414-5423, 2021.
Sedigheh Eslami, Gerard de Melo, and Christoph Meinel. Does clip benefit visual question answering in the medical domain as much it does in the general domain? arXiv preprint arXi::2112 13906, 2021.
Huan Fn, Mingming Gong, Chaohui Wang, Kayhan Ratmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2002-2011 2018
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. orXiv preprint orXiw:2110.028 028888, 2022.
Shanel Gauthier, Benjamin Thérien, Laurent
Muawiz Chaudhary, Irina Rish, Eugene Belilovsky, Michael Eickenberg, and Guy Wolf. Parametric scattering networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5749 5758, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to sule-super learning, Advances in neural information processing systems, 33:21271-21284, 2020a.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoghu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised Learning. or orXiv:2006.000 077333 fes, stat], September 2020b. URL hatp:://arrtvoorr//a//2006000
arXiv: 2006.07733.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Newral Information Processing Systems, 34:5000- 5011, 2021a.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss. arXiv:2106. 26110116600016666661 fes, stat], August 2021b. URL http://arxiv.org/bbbl
2106,04158 arXiv: 2106.04156.
Bobby He and Mete Ozay. Exploring the gap between collapsed & whitened features in self-supervised learning. In International Conference on Machine Learning, pp. 8613 8634. PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yaughao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable 2021.
vision learners. arXiv preprint
14

---

Published in Transactions on Machine Learning Research (12/2025)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 16000-16009, 2022a.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. pp. 16000-16009) 2022b. URL
Be Masked Autoencoeers Are Scalable Vision Learners _CVP 2022 paper
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in neural information processing systems, 30, 2017.
models. Advences
Sergey loffe and Christian Szegedy, Batch uormalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp- 116.456. PMLR, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXi preprint X80: 07.00780.. 2018.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiu preprint orXic:2110.099: 2021.
Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddison. Contrastive learning can find an optimal basis for approximately view-invariant functions, 2023. URL htcps://araavoor//aaa//220..0 883
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Tsola, Aaron Maschinot, Ce Lin, and Dilip Krishnan. Supervised Contrastive Learning. In Advances in Neural Information Processing Systems, volume 33, pp. 18661-18673. Curran Associates, Inc., 2020. URL https: http::::/tttcseeee http://://ppooooooooooooooo///op 2020/bash/dB
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint orXiv:1422.0000 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss, 2021. URL btttp:://arxlv.orr abs/ ab3/2012.08465
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Procendings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Warren Morningstar, Alex Bijamov, Chris Duvarney, Luke Friedman, Neha Kalibhat, Luyang Liu, Philip Mansfield, Renan Rojas-Gomez, Karan Singhal, Bradley Green, and Sushant Prakash. Augmentations vs Algorithms: What Works in Self-Supervised Learning, March 2024. URL http:://Aakkvoorr/////220..2.222 [cs).
arXiv:2403.05726
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729. IEEE, 2008.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, February 2024. URL http: //ap:://///iiii http:////wwww.oogg///////...../
arXiv:2304.07198 [cs].
15

---

Published in Transactions on Machine Learning Research (12/2025)
Edouard Oyallon, Sergey Zagoruyko, Gabriel Huang, Nikos Komodakis, Simon Lacoste-Julien, Matthew Blaschko, and Bugene Belilovsky, Scattering networks for hybrid representation learning. 1EEE transactions on pallern analysis and machine intelligence, (1()):2208 222212211, 2018.
Serdar Ozsoy, Shadi Hamdan, Sercan Ô Arik, Deniz Yuret, and Alper T Erdogan. Self-supervised learning with an information maximization criterion. orXiv preprint or orXiv2209900090 07999, 2022.
Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models, November 2023. URL
ar Stiv22311.03655 [es, stat].
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognitiin,, pp. 3498 3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 8748 8763. PMLR, July 2021. URL htt.://ss.pooccccuottttsssssss/s 2640-34498.
ISSN:
Colorado J Reed, Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer, Selfaugment: Automatic augmentation policies for self-supervised learning. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 2674-2683, 2021.
Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cruss-Eutropy Is All You Need To Invert the Data Generating Process. October 2024. URL https
Geoffrey Roeder, Luke Melz, and Diederik P. Kingma. On Linear Identifiability of Learned Representations. arXiv: 2007.00814
arXi::2007.008 fes, stat), July 2020. URL http:/ppp////wworr///////0000000
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland S. Zimmermann, and Wieland Brendel. In- foNCE: Identifying the Gap Between Theory and Practice, June 2024. URL/ttp:///axxxxor//////2200.
arXiv:2407.00143 jas, stat].
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573 9585, 2020.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3, arXiv preprint erXiv:2508. 10104, 2025.
James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pp. 31852-31876. PMLR, 2023.
Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, and Malte Ebner et Lightly. GitHuu.. Note:
https:// htttes://gttbb.... com hoom//ightly-aitiihhaeee 2020.
Chao Tao, Ji Qi, Weipeng Tai, Hao Wang, and Haifeng Li. Remote sensing image scene classification with self-supervised paradigm under limited labeled samples. IEEE Geoscience and Hemote Sensing Letters, 19:1-5, 2020.
Vimal Thilak, Chen Huang, Omid Saremi, Laurent Diuh, Hanlin Goh, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. LiDAR: Sensing linear probing performance in joint embedding SSL architectures. In The Twelfth International Conference on Learning Representations, 2024. URL htpps://ppprrrrieeneettt wttp:e/wwwww.eeeeoeeeeeieeea
F3g5XpL9K3
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishman, Cordelia Schmid, and Phillip Tsola. What makes for good views for contrastive learning? Advances in Neurol Information Processing Systems, 33:6827-6839, 2020.
Yuandong Tian, Xiulei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10277) PMLR, 2021.
Laurens van der Maaten and Geolfrey Hinton. Visualizing data using L-sue. Journal of Machine Learning Research, 9 (86):2579 2605, 2008. URT. http:////t...........o.eo./ooeen
16

---

Published in Transactions on Machine Learning Research (12/2025)
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Sel--Supervieed Learning with Data Augmentations Provably Isolates Content from Style, June 2021. URL
arXiv: 2106.04619.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Cub200 dataset. Technical Report CNS-TR California Institute of Technology, 2011.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Mochine Learning, pp. 11112 11122. PMLR, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742 2018.
Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sahato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 24851 24871. PMLR, 17-23 Jul 2022. URL v162/xue xt12//22222a.mm
Yihao Xue, Siddharth Joshi, Eric Gan, Pin-Yu Chen, and Baharan Mirzasoleiman. Which features are learnt by contrastive learning? on the role of simplicity bias in class collapse and feature suppression, 2023.
Yihao Xue, Eric Gan, Jiayi Ni, Siddharth Joshi, and Baharan Mirzasoleiman. Investigating the benefits of projection head for represenlation learning, 2024.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2- a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data 10(1), jan 2023.
doi: 11038//$1557-002201722211 URL /ttpps////oollg///000033/////222 022-0122---001--22-1
Kaiwen Yang, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Identity-diseentanleed adversarial augmentation for self-supervised learning. In International Conference on Machine Learning, pp. 25364-25381. PMLR, 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXim:2103 203.03303, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. 2019.
Xiaolma Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. pp- 11975-11986, 2023.
Jinghuai Zhang, Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. Corruptencoder: Data poisoning based backdoor atlacks to contrastive learning, 2024. URL httpp:://axxvoooo//////11..1
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001-13008, 2020.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp- 12979- 12990. PMLR, 2021a.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process. arXiv:2102.08850 fes], February 2021b. URL http://aaxxiv.og// abs abs/2102.08850 arXiv: 2102.08850.
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. the the gemerraiiaannnnnnnnnnn of adam in learning neural networks with proper regularization, 2021. URL http:///aard..cr///////0...11.11
17

---

Published in Transactions on Machine Learning Research (12/2025)
Supplementary Materials
The supplementary materials is providing the proofs of the main's paper formal results. We also provide as much background results and references as possible throughout to ensure that all the derivations are self-contained. Some of the below derivation do not belong to formal statements but are included help the curious readers get additional insights into current SSL methods.
siomese/ siamese/teacher seracher-studen//aeeee student DNN representation collapse
informatiiv training loss
N-way classifier
cul-of-the-box ucross architaclares/daleenn
Training dataset
sample, e
sample e2
sampl ex
XEnt(n, Ww
select
sample 1 e
Deep Network
Figure 6: DIET uses the datum index (n) as the class-target effectively turning unsupervised learning into a supervised learning problem. In case, employ the cross-entropy loss (X-Ent), extra care needed to handle different dataset or architectures. As opposed to current SOTA, we do not rely on a projector nor positive views i. change needs be done to any existing supervised pipeline obtain DIET. As highlighted in Fig. 7 DIET's training loss even informative of downstream test performances, and as ablated in Appx. C there is no degradation of performance with longer training, even for very small datasets (Tb.. 1h.
A Theoretical Analysis and Proofs
A.1 Technical Setup
A.1.1 Notation and Setup
We use regular font for scalars, bold lowercase font for vectors, bold uppercase font for matrices. We use || to represent the Euclidean norm for vectors and || || to represent the Frobenius norm for matrices. The vector e, represents the é-th standard basis vector. For a matrix M, we write Mt for the Moore- Penrose pseudoinverse M.
We say a matrix M E Rmxx is an isometry if M M = Im- Equivalently, (Me, Mv2) = (w), v2) for all U1, U2 E R" We say M a partial isometry if M acts as an isometry on the orthogonal complement of its kernel.
For a matrix M E m and a scalar function g Rmxx + R, 3M consists of the partial derivatives of with respect to the entries of M, namely
0g ūM
0g
JM
0g 9M
0g
-DMmi
abdmm
We use the Kronecker delta function orj, which is defined as 1 if í = otherwise 0.
A.1.2 Definition of Spectral Contrastive Loss
Recall the given definition of the spectral contrastive loss
Laet
E
[- [- f(x))) (mn. (...
E
[(f(=))T {((z)) f(x2))
[2) (2
In Xue et al. (2023 the positive pair term in the contrastive loss was instead defined as
E Peww) [ax))
18

---

Published in Transactions on Machine Learning Research (12/2025)
that
Lant
E (=:0)(a:s/)-Daaa""
2f(x)) f(x')]
E (na
[(((z)) f(xx))) [(a)))
This only differs from the current definition by some constant multiple a, where is the inverse of the probability that randomly chosen pair is positive pair. The reason for changing this normalization is that with the original formulation, the norm of the optimal weights and embeddings would grow with the number of classes. Quantitatively, it is not hard to check that
c Cocl(f
That is, the loss landscape of the two loss functions is the same up to rescaling. It turns out this is the correct scaling factor to keep the norm of the optimal weights and embeddings bounded, with scale matching those produced by DIET.
A.1.3 Isometric Classifier Head
Assumption 4. The embedding dimension is at most the number of labels. That is, m S n.
If Assumption 4 satisfied, then requiring that WH be an isometry does not restrict the expressivity of the model class since any model can be converted into an equivalent one where WH is an isometry:
Lemma 1. Suppose Assumption 4 holds and f is a linear model fw(x) We and WH is the projection head. For any model (W,., h. W), there exists another model (W H: W') such that the model outputs agree, i.e. WnW- = W,WW'' W', and W is an isometry.
Proof. Let WH = T be an SVD of WH, where U e Ruxa E Raxm V E mmm Since rank WH S m S. n, this decomposition can be truncated so that
WH = U
where U1 E Raxm En E mxxm and U,U1 = I Then taking W = =U and f V works.
A.1.4 Theoretical Setting for Normalization Theory
In this section, we formally define the theoretical setup used in 4.2
Let C E Z Mu: Ve. Ju. oe, & R+ be constants. Let C fl.....C C} label set of latent concepts. To each C we assign a content. (low noise) feature and style (high noise) feature For cat, content feature could be ear shape (almost always a pointed one), while a style feature could be fur color (often highly varying between breeds), with Ve, u, E Rdd. We assume all u, and v; are orthonormal, the fcature noises Eu Ey are drawn from symmetric, zero-mean distributions p(ew), p(ev) with variances o < of and a bounded support such that for Vu, Vo < 1, |u| S Vu and |o| S De; while the background noise drawn from Gaussian distribution scaled by some parameter d.
We also make the following technical assumptions:
1. Balanced classes: The number of examples from each latent class are equal.
2. Isometric classifier head: WH fixed isometry. As before, this allows us study the structure of the embodding space induced by the loss function without worrying about the effect of WH-
3. Alignment: For all i, h; = [WTe * O. TI h; = 0, then the model outputs would always be perpendicular to e,, so the normalized DIET loss æ; would be a constant. Requiring h; # 0 ensures that Ø; can contribute to the learning.
4. Initialization: We initialize W 0, and train using gradient descent on the population loss.
5. Sparse concepts: [C| = o(d).
While some of these theoretical assumptions are idcalized, we demonstrate that similar behavior occurs in general real-world settings in Section 7
19

---

Published in Transactions on Machine Learning Research (12/2025)
A.1.5 Normalization of Zero
Note that normalizing the zero vector is not well-defined. This can be an issue in the setup of Theorem 3 because we initialize =
= O. In PyTorch, this is handled by redefining norm(a)) t max[[ for negligible e. We will take similar approach, where we simply define norm(() This can be seen taking с - and rescaling the Jacobian at 0 so that it does not blow up. Note that in the standard formula for the Jacobian of the normalization function,
and the Jacobian as J (0) I.
J
+
the same formula holds when æ we drop the ||^||| terms. In the following proofs, this is how we will interpret such formulas in case we need to normalize vector.
A.2 Proof of Theorems
A.2.1 Proof of Thm. 1
Proof. A result from Lu & Steinerberger (2021 shows that the global minimizer of DIET is the simplex ETF configuration. Awasthi et al. (2022) showed that the global minimizer of the InfoNCE object is also the simplex ETF configuration. Since an orthogonal transformation preserves both norms and simplex ETF structure, the theorem follows.
A.2.2 Proof of Thm. 2
The statement of Thin. equivalent to the following:
Theorem 5. Suppose that Assumption 4 holds and is a parametric feature model f(z) W Wo(z). Then,
If (W, WH) is global minimizer of DIET and WH is column- orthoooot then W global minimizer of LSSCL-
MSE
If W is global ménimizer of LSCL, then there exists WH such that WH B columm-orthhgooal and (W, Wn) is a global minimizzr of LIETT-
Denote by N = [D] be the size of the angmented dataset. We represent this dataset in matrix form
D = [X, Y) E I2 x n
where every column of X is the representation of an augmented input in feature space and the corresponding column of Y is a one-hot encoding of the label. Define the following useful matrices to characterize the structure of the data:
M ((an)) xx 1 X
N
M pos
E (1 ( syr
N2 XYT
Here M is the expected outer product of all examples with themselves, and Mpos is the expected outer product between pairs of examples if they are in the same class (known as positive pairs).
We outline the proof as follows. First we leverage result from Xue et al. (2023) which characterizes the critical points and global minima of the spectral contrastive loss in the same setting. We then prove a relationship between the critical points of MSE diet and the sepctral contrastive loss. Finally, we prove a relationship between the global minima of the two loss functions.
For the rest of this section, we will just write Laiet in place of Cusee The following is a statement and slightly simplified proof of the key theorem from Xue et al. 2023
Theorem 6. A linear function f())= Wæ is critical point of Last ilf there basis such that
M Mpos ''M
diag A1 Ar, Art
W WM = ding ()1, Xp, 0, ........)) 0)
W WMM pos diag (2, ..... x 0, .....
20

---

Published in Transactions on Machine Learning Research (12/2025)
with A1...., da 2 and we have rank W I m. It is global minimum of Cast iff it satisfies
WWM-- [M Mposmm
Proof. The first order condition for Soe
dL
MMMp MMWWWM-
(8)
OW
Since M and Mas are positive semidefinite, M Mpos is diagonalizable. Therefore we can construct basis {Vi...., Fat of eigenvectors of M Myos with corresponding eigenvalues λ1,..., A
Now we have im Mpss C im M, which implies that Mpos MM MpoS. Then Equation S implies that
(W WM) W WM Mpos )o WMe;
Thus either WTWMe, or W WM, an eigenvector of W WM with eigenvalue d Since W W M is diagonalizable, the latter implies that v, is also an eigenvalue of W W M with W W WMu x M MMM U,
Thus, with possible reordering of the vs, we have basis U1 ...... Up Vy such that in this basis
M Mpos diag( A1 ..... by, ,r.., Aa)
W
W WMM = diag(As (A1,
X,, 0, .,.....)) 0)
W W pos
diag (27
Xxx 0, .,.,, 0)
with .i.... As 2 0 and we have and S rank W S m.
Note that if W admits the above form, then
WWW WWMMpos W WMW WM
which implics
W Mpps = WMW WM WM
hence all such W are critical points. Then for all such W,
L= TW WM pas + W WWWMWW WWWWWWW
-2
" x Σ x
i=1
X
It is clear from the above expression that the minimum among critical points is achieved when r is maximal and Ai...., Am are the largest eigenvalues. This happens if and only
WWMM= [M MposIm
It remains to check the behavior as || W |F grows large. Equivalently, W W has a large eigenvalue Let be corresponding eigenvector. If w E ker M, then Mu Mp 0, unchanged. Otherwise, w has some nonzero alignment with im) But then Tr[W WMW MM) grows quadratically in X, but Tr|-2W M grows at most linearly in X, hence the loss is large. We conclude that the previously found condition in fact specifies the global minimizers of L.
that the loss is
The following lemma establishes connection between the critical points of Laiet versus L
21

---

Published in Transactions on Machine Learning Research (12/2025)
Lemma 2. The following are true:
IJ (W, Wn) is crilical point of Laiet and Wn is isomelry, then W is critical point of Lest
I W is a critical point of Leet, then there exists a partial isometry WH such that (W, WH) is a crilical point of Ediet-
Proof. The first order condition for Lateet requires that
dL = W (WnWX - YIX =0
(9)
OW
d (WnWX- Y)X W =
(10)
OW
On the other hand, the first order condition for Liset
W WMpss WMW WM.
Indeed, if W is a critical point of Ediet, then Equation implies
wxx W YX
(11)
And Equation 10 gives
WNWXX =YX T
Taking transposes, we have
WXX W WT = WXY
(12)
Right multiplying by WH h and using the fact that W WH Im gives
Wxx W WXY WH
(13)
Combining Equations 1
and
WC get
W WXY WX
We claim that
WXY W WXY
Indeed, since WH is an isometry, W H is a partial isometry, so WH W has basis {ui...., U, such that
W W D; = vn or WH W = 0. If the former is true, then clearly XY WH W 0; = WXYT u, If the latter is true, then we know that V 0; = 0. But then by Equation 12 we have
WXY 0; = WXX W W
Since equality holds on basis, we conclude the two matrix products are equal, claimed. Thus now have
WXX W WXXT Tx
Substituting the values M = X and M XY TXX,
W Mps = W WWWT WM
desired.
22

---

Published in Transactions on Machine Learning Research (12/2025)
For the converse, suppose that W is critical point of Let,, namely
WMpos pos = WMW WM
Let V = ker(MMp pos MW W.). M)) Since Mpos MW WM M is symmetric, VL is spanned by eigenvectors
with nonzero eigenvalues. Let v such an eigenvector with eigenvalue À o Then
0 W ((M pos MW WM) XWO
It follows that We 0, so V- ker W. Set = (WXX [V), (V).. Since
(Y (XX) Mpos = MW WM (
when restricted to V. there exists isometry W U Z such that YX W on V and XY = Xx W W on Z. Extend W to partial isometry W R + R" such that W = W g and WHlU==0 0. Now using the fact that im(WT) ker(W)+ (-) c VV, we have
YX = WxXXX W
Also
XY WT = XX W W
because any vector in Im can be written as u + t1 where E U, un U and
XY W H (u +1 XY WHU
= =XX W W WHU
= -X W W WH u1)
These are the two conditions for being a critical point of Caies, completing the proof.
We now narrow our attention from critical points to global minima. The above Lemma means that we can restrict our study to the critical points of Leet- Using this fact, we now characterize the global minimizers of Lates as follows:
Theorem Assume that WH n is an isometry. Then (W, WH) global minimizer of Ediet iff the following hold
W WTMM= [ MposJm
N T WT) Tr[[MM] Mposlm]
Proof. Suppose (W, WH) is global minimizer of Caiet and W an isometry. By Lemma W is critical point of Las. By Theorem 6 there is basis such that
M M'M pos = diag d1
Ap .p.,, Aa)
W WM diag (11,
X,, ...
W T WWWWA Mpos = diag X2 ......0.
with AI...., , Aa 2 0 and we have < rank W m
23

---

Published in Transactions on Machine Learning Research (12/2025)
Now calculating the value of the loss
1
Ediet
EDl WH Wx; c2.11
2
WW W - YII
2N
TH((WEWX - Y) (W=WXXX))
2N
Tr(XTW W WTW -XW - W
2N
(TrWW WXX - 2T +
2N
Observe that
T wxx WXX") = Tr(W WM =
A
N
Also W WMpss
WXY YX W are diagonalizable and have the same N2 W WXY YX and N2
nonzero eigenvalues, namely X2 Using the fact that
MXY-WHWE WXYT
we have
WXY (XX(Tn)( Wx) 1 N
WXY
N2
conclude by the Spectral Theorem that
N TWWXYTWwh)
A
(14)
Finally, note that Tr(Y is constant. Therefore the minimum possible value of the loss is when W WM = M and equality holds in equation 14 with r = m and Am the m largest cigenvalues of M Mpss It only remains to show this value of the loss is achievable. Indeed, it is not hard to find W such that W WM = [M' Mposln (for example take global minimizer of Cxa).
Let W = -UV be a singular value decomposition of WXYT, Let WH R + R" map the ith eigenvector of U to the ith eigenvector of V for 1, ,....,p. Then
WXY = UEU In particular, WXY WH is a positive semidefinite matrix, and
WWY W N2 W
N2
has nonzero eigenvalues X2 so WXY WH has eigenvalues A,.... A Thus 1 T(WWY WH) ( = Σ d and (W, Wn) as constructed achieves the minimum value of Laiet This completes the proof.
With the above two results, we obtain the desired result:
Theorem 5. Suppose that Assumption holds and is parametric feature model f(x) Wo(x). Then,
If (W, WH) global minimizer of LIETT and WH h is column- orthnnoothhoonal,, then W global minimizer of LSCL-
If W is global ménimizer of LSCL, then there exists WH such that WH is columm--rthogonal and CMSE
(W, Wn) is a global minimizer of DIETT
The first claim is immediate from Theorems and For the second claim, we in fact constructed the necessary WH n in the proof of Theorem 7
Proof.
24

---

Published in Transactions on Machine Learning Research (12/2025)
A.3 Proof of Theorem 3
We will first prove the claim about ddiet. Then we will prove the claim about Lmse lemmas.
diet a sequence of
Lemma 3. If W is minimizer of diet as deftned in Equation
then
W vell Well
of
+o(1)
0
Proof. Since WH is fixed, minimizing Laiet in fact just standard linear regression. The closed form solution is well known:
-1
n Eale, Exlesd(a:))))
( Ea [A(A(c))A(x=)))
{ "
W W
(15)
Now we calculate
E. EA(A(z))A(E))T)
+ = + (1 + 20) + E) +)(+ + (1+ + 22) -)
= (1 + (1+of) + vcw "00 + 2010 + (1 + o2+0))ow
+ & d
(Ia uc0 v(()
Therefore
BA(A(x:)A(x))] Σg
(1 + of
uo + uew Wo (1+ oz)ec( + veto ue()
72
n
6²
+
II 40m - v(n)
d
Φ² (Ia Σ u -2z
Σ anu.u T + u.v u.u. + asaaoe +
d
where we set 0n = 1 + of +
(C-1)6²
02 = 1 +
(C-1)6 d
Taking the inverse,
d
C
Σ
u2 -u v.u. + 4ave
0102 -
c=
d
la
Σ
u T
v.ve
Also, we have
1
1 "
Σ
E (1 (1 e2) + T]
n
n
1
(uo(() + vC(o)
25

---

Published in Transactions on Machine Learning Research (12/2025)
Now using the previously calculated expressions,
1
m
Wu. Wi
ExlerA(z)))))
E (A(A(E))A(xi)))
Ue
71
1
Ca2
C
-W
Exle(A(zi)1)
Ue
- Ve
n
0102
0102
Co
C
-Wiil
Σ
n C(i)=e
0100
0102
Cla2 - 1) Σ
WHe
n(ai02 1) c(i)=c
C(o3 + (C-1)0?
Σ
W H
n(a11022 - 1)
C(s)m
Similarly, we have
We, C(aî + (C-1)0²
W WHe e
n(a1002 1)
c(i)me
It follows that
C(of+ C(oR+1C-110 [C-110
We c.
a(a-aa=1)
W u
Cla [C=1)
a(a-ag-1)
of + 1)
o + (C 1)
Using the fact that C²= o(d), this shows that
Weell + o(1,, as desired. [w..
For normalized diet, we prove the result via the following lemmas. First we define some notation.
Let C(t) E C represent the concept associated with K;, and set re = Eo Wi e, Also as shorthand we write
(6) diet
1 EA [|] (n (A(a)((( ( e:
2
MSE diet- Rorm
diet-norm
Lemma 4 (Useful facts). In the assumed setting, the following hold
If i + j, then W.Tee)T(WTTee If C(i) = c, then r. W e h2
Proof. Since WH is an isometry by assumption, WT is a partial isometry. Since e; L ej, the first claim follows.
For the second claim, we calculate that
W Σ (W
WTe
CUJ=c
=(Wiee:) WTe, -WWe,
h?
26

---

Published in Transactions on Machine Learning Research (12/2025)
Lemma 5 (Step 1). When training with diett- -diet-norm norms at every step in training Wu, and Wo, parallel to Tez and Wp = O for any orthogonal to all the Ue and Ve-
Proof. We proceed by induction on the itcration of SGD. The base case follows from the initialization W =0
For the inductive step, we calculate the change due the gradient descent update. We first note that the inductive hypothesis implies the following useful fact: if C(1) and q E Rd is orthogonal to ue and Vo: then Wq Spon(tre e' cc). In particular, by Lemma W q and W f e, are orthogonal.
Now denoting 2² A(x;), x W a the gradient is
diet norm
EA
1
|| 2 (=
W (W - (a;())
OW
||
EA
I
|| 21 (=())
=a/x1) - ||2
1 (I || (((1)))()))))) (hea(a;))
||
1
| zi (2/))T (H
-A
I
||
Thus
ac
(x uc
diet
u -EA
I (I- || 2 (=/+)T Wiee
OW
||
We consider two First assume C(i) Writing 2x (1 + 1) + (1 62) E, and W((1 ((1+G))u, (1 + (2) a.ec
acl0
diet- norm
11 ||
(acre + (arre + Wie (
u. = EA
OW
|
Now by the symmetry of the noise distribution, can replace & with -g. By induction, W is orthogonal to Te, this does not change || so the above is equal to
+
E1
1
(laor WE)(a +WE) (aste (E)(a ( Wiee,
=EA
I
2 ||
2 2|
1
=A
El
I
p (WE) WHee
||
Using the useful fact from above and Lemma this is equal to
+
W (1+ (1)ar)
EA
||
W He,
Te
+ G
(1 (1+c))a3
EA
|ل
W
W
|
re
Now suppose C(i) = d' CC C. A similar calculation shows that
0c6)
diet morm
OW
u
EA
&Tu.. Ucc ||
(I
2
12
1
(agTe + WE) + WE)T WHee
27

---

Published in Transactions on Machine Learning Research (12/2025)
Again using the symmetry of the noise and the useful fact, this is equal to
E Uc (I
2 E
(aore We) + We) W
||
||
E &Tu, Suee
(I
|| ( WE) - We) W
1
|
E T uuc
EA
2 (aeto + (WE)(aoTT)) WHe
Okor (5 uo) (r) ( We
-EA
12
ofe (E T uc)
EE
WE
Now isolating the component of along uee write & Ec E'. consider replacing g' with - so
Again by the symmetry of the noise,
a 12 (5 Ue)
a
Eoue 5')
EA
WE
= -
W (Ea +
2z, 3
2||2a|
12. c2
-AA
Wue
Ue
||, 3
Combining all these results, we have
MSES
morm
Uc
1
Σ
OC
diet norm
OW
OW
1=1
1+
(1+ 61 (a. )
EA
W H
|
C(i==c
(0) 2 Wue ue
EA
|
C(1)=c' #
(1+ + c (a (0)
)02
-EA
C1
Te +
Σ EA
|
re
C(i=ze
a
EA
Wu,
|
n C(1)=cffc
By the inductive hypothesis Wu, is parallel to Te, so the change from the gradient update is parallel to The same argument shows that Wo, is parallel to Te. Now consider any p orthogonal to all the v, and up- We calculate that
diet
P
1 n2
E
P
( |
((A))) W W e
1
OW
Decomposing æ Bp + 7, by the symmetry of the noise we can replace 3 with -3. Since Wp 0 by
induction does not change, so we have
MSE morm
EA (mn
p
|| ( Wites
OW
p
2n
||
I
1=1
0
28

---

Published in Transactions on Machine Learning Research (12/2025)
Thus the change from the gradient update is 0, This completes the induction.
Lemma 6 (Step 2). Assume that train to convergence using diiet-norm
Then re Wee To Wo, + 0.
Proof. Using the gradient calculation from the proof of step
MSE
diet diet-norm morm
OW
uh
-A
1
|
+
|r
+
Σ EA
(1
(1) (a(
|
||re||"
C(i)me
1
+
E
Te Wur
1
(1 + (a (3) 2m12
= -A
61
|ral| 2 +
Σ
EA
||ra|1
|
C(i)=e
Recall from Lemma that h2 S 1. Also note that by definition (a (a. S || 1|. Hence
OLMSE diet-norm -EA
1 EI
(1+ + Ej (0) (ae (1) 2
Σ EA C(i==c
<
OW
2
OCMSE
This implics that r
norm
Uh 0. This contradicts the fact that we have converged to a point where
aw
Oc OCMSC
Oldtet-porm porm
O.
OW
Lemma (Proof of Theorem for Normalized Diet). Assume that we Iruin to converyence using dier.. Then
1 V1
Weel
+ 1 < 1
1 + Wuel
Proof. From the previous steps, there exists d1,, ac, bi, bi,...,,le be + O such that Wu, a,r, and We, bare- Therefore, for given example æ; with C(i) c, the distribution (A(xx;) over choice of augmentation A takes the form
0²
(a. + 0 + be + b.e2)ro +
Σ d
a3 + V
(16)
etc
where ~ G1,62 ~ G2, and & ~ N(0, 1) for each c' To ease notation, set a. + 0g61 + b, + b,e2 and 2
de d Voz + b2fe. Note that since the re are orthogonal, (((x,))) follows the distribution
k 22 Σ X3l1rel12
c'
We can now treat the loss as multivariate function in aj...., ac, bj..... by. Suppose we vary a, and such that a, + badb, = O. It suffices to calculate the directional derivative induced by this variation and show that it cannot be zero if [& 1tm or 1 I-21
29

---

Published in Transactions on Machine Learning Research (12/2025)
The loss term due to example 2; is
C(t
RAll W (n(omm((((((( - -ell²]
dict-norny
2
27
Wi( + Ee(c(1) )c)
2 E
Coollre 11 + Eex0r1) /2 Terl?
WH(HowOTT1 + Ee( Ad(re)
E
Kecy |nowllll 12 + EE+4(()) 1g11rel1
e WHTC()
-1-B E
Kow |roml 112 + Σezc(s) 23 11
ko
=1 E
22 V (otolllo || + Eezc(s X 11
Observe that by construction d(aō (2) 0, which implies dAg = Thus if C(t) e, the change in the loss ((t) diet-norm is zero.
On the other hand, if C(i) = now calculate the derivatives
8 EA W W (W((m))))W((())) -
C
Da
dict-norm
Dan
h2 Eet #c X3 ||
(+22) |rll || + Ed 2 |r 12) (1+61)
0 (i)
22 Ee'c #c X3 ||r 12
C
-
(12ll + Ee n/ 2/ Ir 12) (1+
Ob.
diet-norm
Hence
Ocli)
Oc(i)
diet da +
diet dbc
dh norm
Da.
Obe
C(i)=c
1 Ee' X3, |rc 1
E
((1 e1)da + (1 + e2 dbbe)
(N3) |rcl2 + Ee X2
11))
C(i)=c
First consider the case that g > 0. Suppose for the sake contradiction
1t I-
Then
a
0>1+4
+ V2
b
b
a
> >1
(-
b
(1+
(1 + (2)
b
In the case that ff <0 0
a
a
0>1+
12
b
a
1
62
b
= 2 - (1+ (1 + (2)
(1 (1+6)) 1 (1+ + >2 2
a
30

---

Published in Transactions on Machine Learning Research (12/2025)
CIFARI00
IMAGENET100
TINYIMAGENET
12
12
10
11
DIET train loss
8
10
10
9
8
4
7
2
20
40
60
0
20
40
60
20 topl test acc. (%)
40
topl test acc. (%)
topl test acc. (%)
Figure 7: DIET's training loss is indicative downstream test performance. We depict DIET's training loss (y-axis) against the online test linear probe accuracy (x-axis) for all the models, hyper-parameters, and training epochs. Yellow to purple correspond to different label smoothing which plays a role DIET's convergence speed (Appx. Cj. For given label smoothing parameter, there exists strong relationship between DIET's training loss and the downstream test accuracy enabling label-free quantitative quality assessment one's model.
Table 13: DIET is competitive and works out-of-the-box across architectures, We keep the settings of Fig. Benchmarks from Dubois et al. (2022
Ozsoy al. 22022
Imagenet-100 (IN100)
Resnet18
SimMoCo
58. 58.20'
MocoV2
60.52*
SimCo
61.28
Resnet50
W-MSE2
69.06
MoCo++Hypen
75.60
ReSSL
74.02*
MoCo+DCC DCL
76.80
MoCoV2 Hyper.
77.70
DINO
74.16*
MoCoV2
76.48
BYOT.
78.76
BYOL
76.60*
MoCoV2 + DCL
80.50
SimCLR
77.042
SimCLR
80.70
SimCLR
78.722
SimSiam
81.602
SimCLR + DCL
83.10
MocoV2
79.282
VICReg
79.402
MarlowTTins
80.382
DIET
resmeL18
64.31
resnet50
73.50
wide resnet50 2
71.92
conanezt smmallllllll
71.06
resnext50 32x4d
73.07
MLPMirer
56.46
densenet121 conmmett. tiny
67.46
swin
67.02
69.77
vit b 16
62.63
Either way (1 (1) - (1 + (2) is strictly positive or strictly negative. Now write
(1 + (1) + e2)dbe = ((1+ a (1+ c2)) da
N Eeeoo 11
Combined with the fact that
Trel++++ vo 13 18)) always nonnegative and not always zero, follows (<3
that dictt-norm + O, contradicting the fact that we have converged local minima.
The same argument shows that b < I-m
1tz2 giving the lower bound.
31

---

Published in Transactions on Machine Learning Research (12/2025)
Table 14: DIET trained on small datasets competes with Imagenet pre-trained SSL. We also report performances for VIT based architecture (SwinTiny) demonstrate the ability of DIET to handle different models out-of-the-box following Fig. Benchmarks from
Yang et al. 2022 + Ericsson et al. 2021)
Aircraft
DTD
Pets
Flower CUB-200
Food101 Cars 6509 196
Arch.
Pretrain Frozen N
6667
1880
2940
1020
11788
68175
C=
100
47
37
102
200
101
Yes
SimCLR
24.19
54.35
46.46
75.00
16.73
NN100
+CLAE
25.87
52.12
43.55
76.82
17.58
Resnel18
+IDAA
26.02
54.97
46.76
77.99
18.15
None
No
DIET
37.29
50.62
66.00
72.01
33.03
62.00
42.55
InsDis
36.87
68.46
68.78
83.44
63.39
28.98
MoCo
35.55
68.83
69.84
82.10
62.10
27.99
PCL.
21.61
62.87
75.34
64.73
48.02
12.93
PIRL
37.08
68.99
71.36
83.60
64.65
28.72
PCLv2
37.03
70.59
82.79
85.34
64.88
30.51
SimCLR
44.90
74.20
83.33
90.87
67.47
43.73
IN-
Yes
MoCov2
41.79
73.88
83.30
90.07
68.95
39.31
SimCLRv2
46.38
76.38
84.72
92.90
73.08
50.37
Resnet50
SeLav2
37.29
74.15
83.22
90.22
71.08
36.86
InfoMin
38.58
74.73
86.24
87.18
69.53
41.01
BYOL
53.87
76.91
89.10
94.50
73.01
56.40
2
54.49
78.62
89.36
94.72
77.94
58.60
Swav
54.04
77.02
87.60
94.62
76.62
54.06
None
No
DIET
44.81
51.75
67.08
73.32
41.03
71.58
55.82
Tiny
None
No
DIET
33.15
51.88
58.06
70.78
32.11
68.86
47.12
Convnext-S
one
No
DIET
43.13
49.52
61.72
67.72
31.44
69.84
40.63
A.4 Gradient Analysis
Consider a batch of B representations { with class probabilities {w: for N classes and set of N class prototypes {wx) The cross-entropy loss for a batch is given by:
LCF
1 ΣΣ
M
Na. log pkkls))
B
where
exp(wx 2:)
pkk2z,) = E, exp(w)] 2;)
To compute the derivative with respect to 2,, we proceed as follows:
θLCE
N
Vikk 0z:
log p())))).
Ozi
B
Since
logp(kk/zz)
w. 2, log exp(w1 2i);
we have
0
N
log (kkp2))))) = we Σ p(il:))u0.
Oz:
Substituting back, we get:
θCCE 0z,
N
N Σ
Vi.k
W
p(jl 2z )
1
k=
32

---

Published in Transactions on Machine Learning Research (12/2025)
Table 15: GPU Memory Usage in MiB for s-DIET DIET, and other SSL methods with a batch size of 256. OOM indicates mut-ory on an Nvidia A40 GPU, which has 46068 MiB of memory. (-DIE- reduces the memory requirements of DIET by more than 2x on large datasets such as TinylmageNeet, and make it up to 2.2x more memory efficient than CL methods.
Method
CIFAR
Image.Net-100 ResNet-50
Tiny-Imagens ResNet-50
ResNet-18
ResNet-50
Barlow Twins
4026
17090
44698
4532
BYOL
4512
17296
(OOM)
4842
SimCLR
3896
16408
40322
4352
Simsiam
3964
16562
45264
4390
DIET
2556
9720
31164
6676
G-DIET
2312
7770
23634
2976
Duc to Ex- Vo.k = 1, we simplify to:
OCCF
N
N
(UE p(A|2:)) wkk =
Σ Di.k WK
B
B
To compute the derivative of LcE with respect to Wa, we start with:
LCB
1
U log (kgp(k)2z))
Vi
Vi log Σ exp(unf zi),
B
B
k=1
B
N
LCE
1
Σ
Σ
Min logg(k/zz)
1
o. 21 +
log
exp(w? 2:),
B
B
B
We then apply the partial derivation operator,
aLCe
1
B
1
Σ
21.k2
OwL
B
1=1
1=1
OCcE Ow
1
Σ (Vi,k - p(k(z;))
1
Σ
Σ p(k))z)
Ui
B
B
In summary, the gradients of the DIET objectives according to class protolypes and representations are defined by:
B N
exp(wr 2,)
LCE (o,,Y,,,W,0
S log
E, exp(w w zi)
i=1
N
V 2, LCE
1
p(k|zi kk/z;)w;
1
Σ
(gi,k p(klzz)) wk.
B
B
1
1
V w. LoE
+
pkk|zz)
(Vi. p(k(zz))
B
B
B
1=1
1=1
1=1
B
Extended Related Works
33

---

Published in Transactions on Machine Learning Research (12/2025)
Despite DIET's simplicity, could not find existing method that considered it perhaps due to the common belief that dealing with hundreds of thousands of classes (N in Fig. 6 the training set size) would not produce successful training. As such, the closest method lo ours is Ezemplar CNN Alexey el al. (015)) which extracts a few patches from a given image datasct, and treats cach of them as their own class; this way the number of classes is the number of extracted patches, which is made independent from N. A more recent method, Instance Discrimination Wu et al. (2018) extends this by introducing inter-sample discrimination. However, they do so using a non-parametric softmax, i.e., by defining a learnable bank of centroids to cluster training samples; for successful training those centroids must be regularized to prevent representation collapse. Lastly, methods such as Noise as Targets Bojanowski & Joulin (2017) and DeepCluster Caron et al. (2018) are quite far from DIET as (i) they perform clustering and use the datun'' cluster as ils class, i.e., greatly redueing the dependency on N; and (ii) they perform clustering in the output space of the model fo being learned which brings multiple collapsed solutions that force those methods to employ complicated mechanisms to ensure training to learn non-trivial representations. We note that while the added complexity enables those methods to scale to large datasets, it also greatly increases the performance sensitivity to the training hyper-parameters.
Accuracy
65
32
64
128
256
512
Batch Size
Figure Linear Probe Perfor- mance batch size of S-DIET on CIFAR-100 with ResNet-18. Perfor- mance is consistent, even for small batch sizes.
B.1 The Effect of Projection Head in DIET
This section discusses how the results from Xue et al. 2024 can be applied to analyze the benefits of using a projection head with DIET. Given an / layer linear model f(xx) = W, Wi. Xue et al. flow with any loss function and initialization ())))(
2024 all show that when training with gradient
W.+1(0) T Ws+1(0), then
((()) = (1) (+)
(17)
holds all training times We note that if weight decay used, then Equation 17 holds for all 1sis regardless of initialization when taking t - 80
Now Equation| 17 implies that the singular values of each layer are equal, and hence weighting of features by we go deeper into the model. As result, intermediate layers learn more
the model change exponentially balanced and less specialized representations.
The setup from [ 4 can casily be translated to two layer model I = 2 with W = W the weight matrix of the linear model and W₂ WH the classifier head. This can also be generalized to the case where we explicitly include a projection head along with the classifier head. Specifically, given j layer linear backbone model and a k layer projection head, this equates to a setup with / = + k + where the linear model is represented by Wj... W1, the projection head is represented by represented by WJ+++1-
W and the classifier head is
C Additional Experimental Details: DIET
C.1 DIET Pseudocode and setup
C.2 Training dynamics
To understand feature learning in DIET
compare its learning dynamics to other SSL methods, which exhibit step learning dynamics Zimmermann et al. 2021b Rusak et al. 2024 von Kügelgen et al. 2021 Reizinger et al. 2024 i.c., with small-scale initialization, the cigenvalues of the learned represcntations evolve in discrete steps rather than continuously (Simon et al. 2023). We observe the same for DIET In Fig. 11 and Fig. 12] we show that training a ResNet18 on CIFAR100 and TinylnageNet using DIET leads to a step-wise increase of the embedding's singular values, similarly to other SSL methods. Additionally, we observe that the range of the singular values drops substantially for DIET, much more than for SimCLR and VicReg methods. DIET representations are thus high-rank embeddings compared to other SSL methods.
34

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm DIET's algorithm and dataset loader.
any preferred DNN g-g- resnet50
see Alg. 2 for other examples
from torch. utils data import Dataset,
totchvision models. resmet50) () # fe
DataLoader
torchvision datasets import CIFARI00 from
cames with a classifier so we remove it f.fc..inffeaturee
class DataserWichlndices (Dataset);
f. = an. (dentityy
def
init initt_nsellf dataset):
self dataset dataset n):
define DIET's linear classifier and XEnt
def
__etittm_ # disregard the labels delf..dtaset//f return x, n
W = nn (Kn N. bias=False) Ww in Equation (1)
XEnt an.
U.3)
define dataset and train (Fig
def __en__(_el]]
train baain_dataser = DatasetWithlndiees (taainndatt___ train_loader DataLoader train_dataset. ...)
return len (self. seelf.ddtaseet.
example with CIFAR100 C100 CIFAR100 C100 ind
for x, n in train loader: train_loader:
lass f(x)), n) Equation (1)
DatasetWisslndicces((Co))))
# backprop aptimmzzer scheauler
DIET's experimental setup:
Official Torchvision architectures (no changes in init./arch.), only swapping the classification layer with DIET's one (right of Fig. 60, projector DNN
Same DA pipeline (T in Fig. 6) across matasets/arcciteecte with batch size of 256 to fit on GPU
Adamm optimizer with linear warmup (10 epochs) and cosine annealing learning rate schedule, XEnt loss
(right of Fig. 6) with label smoothing of 0.8
of 0.001/0..5 for non transformer architectures and 0.0002/0.01 for transformers Figure 9: In underlined are the design choices directly ported from standard supervised learning (not cross-validated for DIET), in ilalic are the design choices cross-validated for DIET but held constant across this study unless specified otherwise. Batch-size sensitivity analysis is reported in Tab. 16 and Fig. 15 showing that performances do not vary when taking values from 32 4096. XEnt 's label smoothing
Learning rate/weight-
parameter plays a role into DIET's convergence speed, and cross-validated in Fig. 14 and Tab. 16 we
also report DA ablation in Fig. 15 and Tab. 16
Table 16: Ablation studies indicate that DIET benefits from longer training and stronger data augmentation while being robust to architecture and batch-size changes, We report topl test accuracy CIFAR100 with varying training epochs (top left), TinyImagenet with varying DA pipelines Alg. 3,, and on Tinylmagenet with 3k training epochs and with varying batch-size (bottom) with learning rate 0.001 as idnalional comparisons MedMNIST Tab. 17.
256
Epochs
50
100
200
500
1000 5000
10000
DA strength
2
3
resnet 18
31.48
43.62
43.88
resnet18
33.46
42.94
48.24
54.54
58.81 62.63
63.29
resnet33
32.93
45.60
45.75
resnet50
37.71
47.86
54.04
60.23
64.24
69.51
69.91
resnet50
40.24
48.80
50.81
resnet101
34.03
46.59
54.3
60.8
64.71 70.56
71.39
resnet 101
40.07
49.74
50.76
batch-size resnet18
32
128 256 512 1024
32.9 37.9 42.7 43.4
43.3
43.7
43.7 42.6
C.3 Impact of Training Time and Label Smoothing
In Figure 14 we show the performance DIET on CIFAR100 across three label smoothing settings. We find higher values of label smoothing speed up convergence, although in this setting all cases greatly benefit from longer training schodules; final lincar probe performances reported in Tab. 16
C.4 Impact of Mini-Batch Size
We show in 15 ablations for TinyImagenet using DIET. In addition we show DIET's robustness to batch size by conducting an additional ablation by varying the batch size for the Derma MedMNIST dataset with batch sizes low 8. As shown in Table 17 we see DIET performs well even with very small batch sizes.
35

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 2 Get the output dimension and remove the linear classifier from given torchvision model (Pytorch
used for illustration).
model
torchvision madels _ictt ( architecture ]()
CIFAR procedure to adjust to the lower image resolution
is_cifar and "renntt" in architecture:
model. convl torch nn. (3, 64, kernel_size=3, stride=1, padding=2. bias=False] model. maxpool torch. nn. Identity ()
for each architecture remove the classifier and get the output dim. (K)
"alexnet" in architecture:
K model. classifier in_
model. classifier [6] torch nn. Identity ()
elif "convnext" in architecture:
model model. classifier i model. [2] -arch.
Identity ()
elif ccnvnnet" in architecture. model. classifier int
model. classifier torch torch nn. Identity
elif rrsnett architecture or "resnext" in architecture or "regnet" in architecture:
K model. fc = =tooch..n..deentity elif "densenet" architecture:
K model. m www.ccaaaiiieeeiifeettrre classifier torch Identity
elif "mobile" in architecture:
model. -lapealllllssieeei____eeeeaeae
model. classifier torch Identity
elif 'vi* in architecture:
model heads. head. ind.dinfee__eeueee
model. heads. head torch.nn. n.....nn..leeee Identity
Iddeniit..dentttyy ()
elif "swin" " architecture:
K model. head. inddiin__eatrr
model. head = torch. mn.ch.nn.... Identity
resnet18
resnet50
resnet101
convnext tiny
swin
CIFAR10
60
supervised
DIET
40
IMAGENETION TINYIMAGENET
40
60
Food101
60
40
20
300
14
2K
5K
10k
30k
500
IK
10K
30k
500
30k
500
5K
10K
30k
500
10k
30k
train size (N)
Figure 10: Reprise of Appx. C.2 on additional datasets depicting how DIET is able to compete with supervised learning for in-distribution generalization in very small dataset regime.
Batch-size does not impact DIET's performance. One important question when it comes to training a method with low resources is the ability to employ (very) small batch sizes. This is in fact reason hindering
36

---

Published in Transactions on Machine Learning Research (12/2025)
Singular values
Training steps
Training steps
Training steps
Training steps
Figure 11: Dynamics SSL: (left) Step dynannic similar (Smon al. 2023), we find that the embeddingss
singular values increase in sequential and step-wise fashion. (right) Top 200 singular values across the first 2000 training steps for DIET (left); SimCLR (middle), VicReg (right) on TinylmageNet. We find that for DIET the range of values taken by the singular values drops during training.
Singular values
Training steps
Training steps
Training sleps
Training steps
Figure 12: Dynamics of SSL: (left) Step dynamics similar to Simon al. 2023 find that the embeddings'
singular values increase in a sequential and step-wise fashion. (right) Top 200 singular values across the first 2000 training steps for DIET (left), SimCLR (middle), VicReg (right) CIFAR100. We find that for DIET the range of values taken by the singular values drops during training.
resnet18
resnet50
resnet101
convnext tiny
swin
30
IMAGENET
20
INATURALIST
4-
500
10k
30K
500
1K
10k
30K
10t
10k
30K
500
10k
30k
train set size (N)
Figure 13: DIET matches supervised learning datasets with only few samples per class. Depiction
of DIET's downstream performances (blue) against supervised learning (red) controlling training set size (x-axis);
evaluation is performed the original full evaluation DIET able to learn highly competitive representations
when the dataset is small with only a few samples per classes. See F Fig. 10 for additional datasets.
the deployment of SSL methods which require quite large batch sizes to work (256 is strict minimum in most cases). Therefore, we perform small sensitivity analysis in Tab. 16 where we vary the batch size from to 2048 without any hyper-parameter tuning other than the standard learning rate scaling used in supervised learning: Ir = 0.001 bs We observe small fluctuations of performances (due to a sub-optimal learning rate) but no significant drop in performance, even for batch size of 32. When going to 16 and 8, we observe slightly lower performances, likely due to behave erratically below batch size of 32 Toffe (017
256
loffe & Szegody (2055) which is known
37

---

Published in Transactions on Machine Learning Research (12/2025)
Resnet18
50
1500
Resnet.50
60
Tops Test 82C.
40
50
50
100
100
200
250
500
500
1000
2500
5000
5000
10000
Resnet
40
20
Figure 14: Depiction of the evolution of linear topl accuracy throughout epochs CIFAR100 with three Resnet variants and three label smoothing parameters represented by the different shades of blue going from light to dark shades with values of 0.1, 0.4, and 0.8 respectively.
Batch Size
8
32
64
128
512
DIET
71.87
72.52
73.07
74.36
71.02
MoCov2
66.88
64.64
66.73
66.88
61.40
SimCLR
63.14
66.43
66.83
66.88
66.83
VICReg 65.84
60.45
64.79
66.78
66.88
Table 17: Reprise of Tab. 16 DIET's performance across varying batch sizes on the Derma MedMNIST dataset with all other hyperparameter fixed demonstrating the stability of DIET do that hyper-parameter and across training iterations. All models are trained for 500 epochs.
C.5 Impact of Data-Augman Auggmentgnnttii
To further study the effect of data augmentation in DIET we study varying data augmentation strengths for TinylmageNet in Fig. 15 We also examine the effect of weaker data augmetnations for smaller medical images using PathMNIST in Tab. 18
Data-Augmentation sensitivity similar to SSL. When using DA, DIET able to perform on par with highly engineered state-of-the-art methods. Yet, knowing which DA to employ is not trivial, e.g., many data modalities have obvious DA. One natural question is, thus, concerning the sensitivity of DIET's performance to the employed DA. To that end, we propose three DA regimes, one only consistent of random crops and horizontal flips (strength:1), which could be considered minimal in computer vision, one which adds color jittering and random grayscale (strength:2), and one last which further adds Gaussian blur and random crasing Zhong et al. (2020) (strength:3); the exact parameters for those transformations are given Alg. 3 We observe on Tinylmagenet and with Resnet34 the following performances 32.93+ 0.6, 45.60 0.2, and 45.75+ 0.1 respectively over 5 independent runs, details and additional architectures provided in Fig. 15
38

---

Published in Transactions on Machine Learning Research (12/2025)
Tinylmagenet
resnet18
resnet34
TINYIMAGENET
50
topl test acc (%)
45
strengoh
40
strunigh
40
30
1r:0.00
(epoch 30)
(last epoch)
30
30
20
25
20
20
resnet50
resnet101
50
topl test (%)
45
200
40
16
35
8
16
64
128
256
512
2048
1022
batch size
25
20
1000
2000
3000
1000
2000
3000
epochs
epochs
Figure 15: Left:Tinylmagenet with fixed number of epochs and a single learning rate which is adjusted for each case using the LARS rule therefore per batch-size learning cross-validation can only improve performances, see Tab. 16 the per-epoch time includes training, testing, and checkpointing. Right: Tinylmagenet, see Tab. 16 for table of results, and the specific DAs can be found in Alg. 3
Algorithm 3 Custom dataset to obtain the indices (n) in addition to inputs and (optionally) the labels y4. to obtain train_loader used in Appx. C.1 (Pytorch used for illustration).
transforms =
Random Deeeeeeeeeeeeeeeelllllleeeeeeeee ((size size)).
Handoomoorizonnaallii(
if strength transforms... append
RandomApply
torch. nn. ModuleList ([T. (olorJitterr..4, 0.4, 0.4. 0.2) p=0.3
transforms append (T. RandomGrayscale((.22)
if strength 2:
transforms a
RandomApply
torch. nn. ModuleList ([T. GaussianBlur ((3, 3), (1.0, 2.0)))), p=0.2
transforms..o .tttoorms.apppeeeeo append (T. (awdodmErainnggO... (0.255)
and Tab. 16 in the Appendix. We thus observe that while DIET greatly benefit from richer DA (strength:1 + 2), it however does not require heavier transformation such as random crasing.
C.6 Impact of Label Smoothing
Label smoothing helps. One important difference in training behavior between supervised learning and SSL is in the number of epochs required to the quality of the representation plateau. Due to the different loss used in DIET, one might wonder about the differences in training behavior. We observe that DIET takes more epochs than SSL until the loss converges. However, by using large values of label smoothing, e.g., 0.8, it is possible to obtain faster convergence. We provide a sensitivity analysis in Fig. 14 and Tab. 16 in the Appendix. In fact, one should recall that within single epoch, only one of each datum/clas is observed, making the convergence speed of the classifier's WH matrix the main limitation; aim explore improved training strategies in the future discussed in g
39

---

Published in Transactions on Machine Learning Research (12/2025)
DIET Loss
DIET Training Accuracy
bloodanist
pathamist
deramnis
51 bloodenist
200
500
300
200
400
600
800
Figure 16: DIET McdMNIST training loss curves for the DIET criterion (left) and training accuracy (right)
with a ResNet18 backbone.
C.7 DIET compared supervised learning
DIET matches supervised learning on datasets with only a few samples per class. In Appx. C.2 we directly compare DIET with supervised learning on a variety of models and datasets but with controlled training size. We clearly observe that for small dataset, i.e., for which we only use a small part of the original training set (less than 30 images per class), DIET's learned representation is efficient as the supervised one for the in-distribution classification downstream lask.
DIET works with scattering network architectures As an additional test, scattering networks (Oyallon et al. 2018 Gauthier al., 2022) hard-code part of the model parameters be wavelet banks. That specification naturally makes such scattering networks very competitive for small data regimes since the number of degrees of freedom is reduced. We therefore performed two additional experiments: Training hybrid scaltering network in supervised setting Training hybrid scallering net with DIET and then learning linear probe on top (keeping the hybrid scattering frozen) We perform both cases above on the full CIFAR10 training set and on a reduced training set of 5000 (10% of the training data) samples. Supervised training of the scaltering network results 72.1% 2%) test set accuracy, whereas unsupervised DIET pretraining followed by a linear probe results in 77. 77.64% (62.8%) for the same architecture. From that experiment we obtain two novel insights. First, DIET works out-of-the-box on DNs such as the hybrid scattering network, with reduced number of parameters. Second, even in that regime, DIET provides strong performances.
C.8 Additional Results for MedMNIST
In Figure 16 show training curves for DIET with ResNet18 architecture. We perform ad- ditional experiments with DIET using a vision transformer architecture (ViT-Small with patch size 4) based on the architect from https: htttp:://gttub..co wtg.tttb..oo//ucidraariit.e. potoorrrrnn///ttttooooooo/////b pptotch/ooobbbooooopoo hollbb/main/////mmm
vit hat__pytorch/iit for smallm dataset.py We find DIET achieves good performance on the same MedMNIST datascts with this ViT architocture without additional hyperparameter tuning as shown in Table
19 and in comparison all three bascline SSL methods in Table I8
We find evidence of the default augmentations for PathMNIST being too aggressive and confirm DIET's performance improves with the use of weaker augmentations in Table 18 Surprisingly, we find DIET performs quite well with augmentations al all, setting in which most. standard SSL methods would be impossible to train.
Additional Experimental Details: S-DIET
D.1 S-DIET Pseudocode
Uppercase variables stored on disk
Lowercase variables stored in memory
X: train head
H: classifier classifier head
40

---

Published in Transactions on Machine Learning Research (12/2025)
bloodmnist
dermamnist train
pathmnist
train
test
test
train
test
DIET
77.65
81.85
71.03
68.88
56.37
21.27
SimCLR
82.48
79.45
69.13
32.37
69.45
21.80
VICReg
86.71
81.03
69.89
46.33
82.94
12.76
MoCov2
62.76
51.01
66.78
63.39
72.9
41.75
DIET
PathMNIST
Augmentation
train
test
Default
56.37
21.27
Weak
44.90
48.95
None
44.65
45.67
Table 18: Top:DIET performance across the three MedMNIST datasets using transformer (ViT-S) architecture with patch size 4 in comparison to standard SSL baselines with the same ViT architecture. Bottom:Comparigg DIET's performance across data augmentations for PathMNIST using transformer (ViT-S) architecture with patch size 4. Weak augmentation corresponds to only random resized cropping and horizontal flipping-
Table 19: DIET performance across the three MedMNIST datasets using a transformer (ViT-S) archilecture with patch size 4. In the first row show the performance of baseline SimCLR model with the default ResNet18 encoder for comparison.
dataset bloodmnist
dermamnist train
pathmnist train test
train
test
test
DIET 77.65 81.85 71.03 68.88
56.37 21.27
Table 20: DIET competitive and works out-of-the-bun box across architectures. We keep the settings of Fig. Benchmarks from Dubois et al. 2022). 2 Ozsoy et al. (022
Imagenet-100 (IN100)
Resnel18
TinyImagenet
SimMoCo
58.20
V2
60.52*
8esnet18
SimCo
61.28
Resnel50
SimSiam
44.54
W-MSE2
69.06
HoCooHHper
75.60
DIET
45.07
Resnet50
ReSSL
74.02*
MoCo+DCL
76.80
SimCLR
46.21
SimSiam
46.76
DINO
74.
MoCoV2 + Hyper.
77.70
BYOL
47.23
SimCLR
48.12 2
MoCoV2
76.48*
BYOL
78.76
MoCo
47.98
DIET
51.66
BYOL
76.60*
MoCoV2 + DCL
80.50
SimCLR
48.70
SimCLR
77.042
SimCLR
80.70
DINO
49.20
SimCLR
78.722
SimSiam
81.60
SimCLR + DCL
83.10
DIET
MocoV2
79.282
resnet18
45.07
resnet50
51.66
VICReg 79.402
resne134
47.04
conveert Liny
50.88
Barlow
80.382
resnet101
51.86
conmert small
50.05
DIET
wide__eeeeeee resnet50
50.03
MLPMixer
39.32
resnet18
64.31
resnet50
73.50
resnert50 32r/0
52.45
swin
50.80
wide resnet50 2
71.92
conventt small
71.06
densenet121
49.38
48.38
resnert50 32r/d
73.07
MT.PMizer
56.46
densenet121
67.46
swin
67.02
conmerl liny
69.77
wil. b 16
62.63
V: second moment for classifier head
10 indices: indices for the current batch
11 wan
def train_ step (X. H. M. V. indices model. criterion optimizer): head optimizer state into memory inputs. head. optimizer_m, optimizer_v X[indices], H[indices]. Mindices], V[indices]
13 # Load data, head weights,
15
labels [0, ..... len len (indices)
41

---

Published in Transactions on Machine Learning Research (12/2025)
17
Forward and backward pass
18
outputs = head (ideu(inputs)).
18
criterion (outputs, labels)
20
Setimizer.zer__gr-e0 zero
21
loss backward
aptimizer.stepi
23
head. m. v
(head, m, v)
24
25
# Save head weights and head optimizer state Done asynchronously
26
27
H[ indices] M[indices]. V[indices] head,
28
30
def perform_ perform_multistee multistep_adaw__hhddde (head, m. v):
31
& head.grad
32
38
# first step
head (1 Ir weight_decayy head
35
m betal m + (1 betal)
beta2
beta2)
37
head = head Ir * =m, (v) + eps)
39
# all other steps
40
mu betal / (beta2)
alphal (1 Ir reight_deeaay (t 1) + mu Ir (mi ** t)) / (1 - Ir weight_decay - mu)
42
alpha2 (alphal
44
head = alphal head alpha2 m/ (sqrt(v) + eps)
46
(betal (t 1])
46
(beta2 *x (t 1))
Listing Pseudocode for S-DIET training step
D.2 Handling Stateful Optimizers in s-DIET
AdamW. Stateful optimizers provide a further opportunity. Considering the AdamW update rules
Loshchilov
& Hutter 2019:
my + Bim-1 + - -11)9t. oi c 1-120-1- + (1 (1-32)9' 0, (1- - 7/ V +
m
1-p5
where 1/2
1-9 For simplicity, we replace di. 1. For default scttings B1 0.9, 3, = 0.999, this can be interpreted as learning rate warmup. As the optimizer may update the weights even if their gradient at the current step gr zoro. Thus would require loading the entire WH even with batch cross entropy. The i-th row of WH f used only with sample i in the batch; otherwise the corresponding batch cross entropy gradient is zero. Since not each batch contains sample i, when does, we perform f optimizer steps on the i-th row of WH. As we do not exactly know f, i.e. the number of steps until the sample is again in the batch, we approximate t t= N B. This yields what we call the multistep update formula for AdamW. Assuming have dropped the 1i term and is negligible, if = 0 for all 1, the above update formulas for AdamW become an inhomogeneous linear recurrence relation which has a closed form solution:
0. = (1-1 7 + (1 TP TH
(+
ma
m, 3/m.
Ut S.0.
(18)
1- -nnd-1 -
Uo
+
where fa denotes the ratio A. In summary, at each step we only update the weights and optimizer state
VB2
of rows of WH that were selected for batch cross entropy at that step. We perform the update by first taking one step with 9: as the calculated gradient, and then apply the multistep update given by Eq. 18 for 1= 1.
SGD with Momentum. Recall the update rule of SGD with learning rate 7, momentum μ,, dampening T, weight decay A:
m. - pm,-1 + TgL.
0, - (1 -
(19)
If 9: for all f, the above update formulas for SGD with momentum become inhomogeneous linear recurrence relation which has an exact solution:
(1 - 11)) hp np npt+1
m, = p'mo-
0, -1 (1- (1-1A)'00 0000++
mo
(20)
nd p
42

---

Published in Transactions on Machine Learning Research (12/2025)
D.3 SSL Methods
Pretrained models for SSL methods are obtained using the solo-learn library (da Costa al. 2022). We use the batch size and augmentations specified in the previous section, and change the precision 32-bit for consistency, All other hyperparameters are left unchanged.
D.4
Toy Dataset
We instantiate the scenario from Section 4.2 with more realistic training setup:
we make the classifier head WH trainable from random initialization.
instead of taking the expectation over all augmentations, we sample a single random augmentation of the imput each step.
We also choose G1, 92 to follow normal distributions (this eliminates the requirement of bounded feature noise).
We set C 4 16, m 4, 32, 0. 0.01, 0.1, 0.004. We train for 5000 steps using the Adam optimizer with learning rate 0.1 and cosine learning rate schedule. We reset the state of the Adam optimizer after the first step to eliminate the effect of gradient blowup from normalizing zero vectors, see Appx. A.1.5) for details.
D.5 Synthetic Dataset
For the synthetic dataset described in Section 7.1 we modify the first convolutional layer the ResNet model to take 4 imput channols instead of 3. For MNIST augmentations, we replace random horizontal flip and random grayscale with gaussian blur. We also modify the random cropping to keep at least 0.75 of the area of the original image. We train for 500 epochs. All other hyperparameters are set as described above.
D.6 Equivalence of DIET and Spectral Contrastive Learning for Ideal Encoders
A common line of study is to analyze the minima of loss functions assuming an ideal encoder, namely one that rcalize any output configuration, since such analysis depends only the loss function and not how exactly the encoder is parameterized. TThm. 5) directly covers this case, an ideal encoder can be parameterized as fixed feature map & which maps the inputs to linearly independent set and then applying a linear encoder. In this case, the global minima of the spectral contrastive loss is any encoder for which
All augmentations an example are collapsed to a single unit vector
2. Embeddings of different examples are orthogonal.
We note that this recovers a result from Johnson et al. 2023 which, after applying the rescaling discussed in Appx. A.1.2 states that the global minima of spectra is achieved when
Ko(01, = f(xz(((x))))))))))))) fx: f(x2) [ = Sy1.g2
Meanwhile, minimizing the MSE DIET loss requires that the outputs of the classifier be exactly equal to the specified targets. Assuming that the classifier head is an isometry, we exactly recover the previous two conditions on the learned embeddings.
D.7 Comparison Between DIET and CL
In Figure 17
we compare t-SNE visualizations Ivan der Maaten & Hinton 2008 of test embeddings produced
by sDET and SimCLR (Chn et al. 2020) on CIFAR-10 with ResNet-50. We observe that the high level structure of the embeddings remarkably similar for both methods.
D.8
D-DITT
experiments with ViT backbone
43

---

Published in Transactions on Machine Learning Research (12/2025)
80
20
23
seplone
Fag
-25
teg
automo0re
astomoble
trick
-00
Drg
-75
-80
-00
-10
-20
-40
-20
20
60
00
(a) DIET
(b) SimCLR
Figure 17: TSNE of embeddings produced by DIET and Sim on
CIFA R-10 using ResNet-50.
Table 21: Linear Probe Accuracy of VIT trained with DIET and s-ET Again s- DIET performance gains over DIET.
provides significant
Method
ImageNet-100
Tiny Eingemeeeeet
DIET
62.63
48.38
DIET
74.04
55.82
E
Acronyms
CL Contrastive Learning
PID parametric instance discrimination
DIET Datum IndEx as its Target
S-DIET Scaled DIET
SCL Spectral Contrastive Learning
MSE Mean Squared Error
SSL self-supervised learning
44


Reviews:
[{"review_id": "BuezmfAV2x", "reviewer": "Reviewer_fK7E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper under submission proposes a simple parametric instance discrimination (PID) method, called DIET, for self-supervised learning and analyzes the theoretical properties of DIET. Numerical experiments are also conducted to demonstrate the empirical advantage of DIET, compared to available benchmarks. In particular, from what I read, one important claimed advantage of DIET is that it bypasses negative sampling, which may lead to suboptimal results due to the biased comparison within batches. Instead, DIET compares every pair of samples, leading to exhaustive comparisons. Despite the seemingly large number of comparisons needed, DIET has a fast algorithm by using a batch-wise update of the gradient. The theoretical results are somewhat preliminary, but the overall methodology is interesting and may be of interest to certain areas of AI, particularly when data is difficult to collect.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. I think the theoretical results are generally sound, which are further corroborated by empirical evidence.", "audience_yn": "Yes", "audience_explanation": "Since self-supervised learning or contrastive learning is quite popular among practicing AI researchers, the proposed method should be of interest to those interested in applying self-supervised learning or understanding its theoretical underpinnings. The method proposed is also related to representation learning, which may be of interest to the general audience of TMLR.", "requested_changes": "Major comments:\n\n1. I recommend that more details be added in certain cases. For example, in equation (2), $\\mathcal{D} = (\\mathbf{x}, y)$. Is $y$ the index label $i$? At least it is not immediately clear to me why $y$ suddenly appears and what it should correspond to.\n\n2. I recommend that more intuition be given in the paper. For example, the advantage of the normalization part, to me, is simply coming from the possibility that $\\mathbf{u}_{c}$ and $\\mathbf{v}_{c}$ are at different scales, if I am not mistaken. Although they are hidden variables, if the data-generating process is really the additive model given by the first equation in Section 4.1, the benefit of normalization seems hardly surprising.\n\n3. I do not quite follow Theorem 3. Why $(\\mathbf{W}, \\mathbf{W}_{H})$ is a global minimizer of $\\mathcal{L}^{\\rm MSE}_{\\rm DIET}$? Is this a typo?\n\n4. Since the theoretical results are very preliminary, please consider adding some discussion on the limitations of the theoretical results.\n\n4. The essential idea of the paper is to maximize the prediction discrepancy between different samples in the training data, which makes perfect sense when the dataset contains diverse information. However, what if the sample is somewhat homogeneous? Suppose that in the dataset, the class \"dog\" contains many pictures of one single dog, with minor differences in its position or angle, but the images are largely similar otherwise. Will the proposed method still work well in this case? It will be helpful to discuss the potential caveat of using the proposed method in applications.\n\nMinor comments:\n\n1. Some of the references are mixed with the texts, making the paper less readable than it should have been. For example, in the first sentence of Section 2, it reads \"SimCLR Chen et al. (2020) ...\", which should be corrected to \"SimCLR (Chen et al. 2020) ...\".\n\n2. The first sentence on page 3: \"on performance(Grill et al., 2020a...\" misses a space.\n\n3. On page 4: \"By studying the standard (2) and normalized (2) MSE DIET losses\" to \"By studying the standard (2) and normalized (3) MSE DIET losses\".\n\n4. Page 7: \"differ more significantly\" to \"differ significantly\".", "broader_impact_concerns": "None.", "additional_comments": "NA.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RupmdV0zgA", "reviewer": "Reviewer_ADzt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a representation learning approach based on a self-supervised instance discrimination method, which replaces the need of labeled training data by an instance classification escenario. This approach also does not require having negative training instances. The proposed idea seems to be efficient, to obtain competitive performance on small datasets, and to be robust against changes in values of training hyperparameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The explanation of the model confirms that it does not require labels o negative instances.", "audience_yn": "Yes", "audience_explanation": "- The idea is novel and interesting.\n- The proposed model does not require labels nor instances from the negative class.\n- The document is well organized and written.\n- The introduction section clearly identifies the research gap.\n\nMainly, the proposed approach presents the opportunity for rich latent representations, that could be later refined and exploited for mainstream tasks.", "requested_changes": "- The authors present their analysis using a MSE function, instead of using categorical cross entropy (CCE), as it is common for multi-class classification problems. The authors justify this decision stating that MSE allows to perform a tractable analysis. However, CCE is also tractable. This claim must be justified better.\n- It is unclear why there is an assumption of orthonormality mentioned in section 4.1? Is it needed? What is its impact?\n- The setup in section 6 needs to indicate the number of neural units in the hidden layers.\n- The setup in section 6, talks about the same data augmentation approach. Please, indicate the number of total random samples that are generated.\n- In page 6, \"can introduce risks such data poisoning\" --> \"can introduce risks such as data poisoning\".\n- Although the idea is interesting, and elegant, the numeric results indicate that the method provides performance either below or comparable with previous methods. This makes it difficult to assess the actual contribution of this work. Is the model smaller in terms of parameters, or does it train much faster with respect to other models?\n- Indicate which metrics are reported in all tables. I guess it is accuracy. also indicate whether they correspond to the train, validation, or test set.\n- The purpose of the experiment that combines the MNIST and CIFAR-10 datasets is unclear.\n- Avoid inline tables. All inserted objects must be numbered and cited though the main document.", "broader_impact_concerns": "No concern about ethics.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KS4Rlj86xs", "reviewer": "Reviewer_WEha", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Introduce a simple parametric instance discrimination approach (DIET) that learns data representations without relying on complex techniques employed by state-of-the-art (SOTA) methods, such as gradient stopping or negative sampling, to prevent representation collapse.\n\nStrength:\\\n1-\tThe problem is explained well. \\\n2-\tThe paper’s motivation to reduce the complexity and resource consumption of self-supervised learning presents an interesting aspect.\n\nWeakness:\\\n1-The baseline comparisons presented in the paper lack consistency and clarity. Specifically, Table 1 includes only SimCLR as a baseline for ImageNet-100 with a ResNet18 backbone. In contrast, Table 2 adds MoCo-v2 and VICReg, while Table 3 further expands the comparison to include BYOL, SimSiam, Barlow Twins, and SimCLR. Table 4 then reverts to only SimCLR and VICReg. This inconsistency creates confusion regarding the chosen baseline.\\\n2- SOTA approaches are typically trained for 800 to 1000 epochs, whereas DIET is trained for 5000 epochs. This raises concerns regarding the time complexity of training DIET. \\\n3- SOTA approaches typically demonstrate their robustness across a variety of downstream tasks. In contrast, the authors evaluate their method solely through linear evaluation. \\\n4-The fundamental objective of self-supervised learning (SSL) is to eliminate the reliance on manual data annotations by enabling models to learn representations directly from large-scale unlabelled datasets. Consequently, many contemporary SSL frameworks, such as DINO-v2 and DINO-v3 [1,2], emphasise scalability to increasingly extensive datasets. However, as noted on page 8, paragraph 2, the author indicates that the DIET approach becomes infeasible when applied to large datasets. This limitation appears to contradict the central aim of SSL, thereby constraining the scalability and reducing the overall significance of the proposed method.\n\n[1] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023). Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.\n\n[2] Siméoni, O., Vo, H. V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al. (2025). Dinov3. arXiv preprint arXiv:2508.10104.", "claims_and_evidence_yn": "No", "claims_explanation": "The author claims that the proposed approach can compete with SOTA without the complexity of the SOTA approaches. The author needs to conduct variant experiments across different downstream tasks to demonstrate that the proposed approaches perform competitively with SOTA methods.", "audience_yn": "Yes", "audience_explanation": "The idea of reducing the complexity of SSL approaches is interesting and might inspire researchers to build on it.", "requested_changes": "1- It is recommended that the authors clearly define the baseline method and maintain consistency across all tables. Furthermore, top-performing approaches such as MoCo-v2 and SimSiam should be consistently included, as these methods demonstrate superior performance with the batch size of 256 used by the authors.\\\n2- It is recommended that the authors report the training time (in hours) or computational cost (e.g., FLOPs) to enable a fair comparison and to assess the efficiency of their proposed approach.\\\n3- It is recommended that the authors conduct additional downstream experiments, such as object detection and semi-supervised learning, to assess the quality and generalizability of the learned representations more comprehensively.\\\n4- It is recommended to explicitly provide practical scenarios in which the DIET approach is preferable over state-of-the-art SSL methods.\\\n5- Finally, I recommend that the author organise the ablation study into distinct sections or subsections, separate from the main experimental results, to enhance clarity and readability.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)