Preserving Angles Improves Feature Distillation
Authors: Evelyn Mannix
Liam Hodgkinson
Howard Bondell
Venue: Accepted by TMLR ()

Abstract: Knowledge distillation methods compress models by training a student network using the classification outputs of a high quality teacher model, but can fail to effectively transfer the properties of computer vision foundation models from the teacher to the student. While it has been recently shown that feature distillation—where a teacher model's output features are replicated instead—can reproduce performance for foundation models across numerous downstream tasks, they fall short in matching critical properties such as robustness and out-of-distribution (OOD) detection performance. This paper overcomes this shortcoming by introducing Cosine-similarity Preserving Compression (CosPress), a feature distillation technique that learns a mapping to compress the latent space of the teacher model into the smaller latent space of the student, by preserving the cosine similarities between image embeddings. This enables direct optimisation of the student network and produces a more faithful reproduction of the teacher's properties. It is shown that distillation with CosPress on a variety of datasets, including ImageNet, produces more accurate models with greater performance on generalisability, robustness and OOD detection benchmarks, and that this technique provides a competitive pathway for training highly performant lightweight models on small datasets. Code is available at github.com/emannix/cospress.

Published in Transactions on Machine Learning Research (11/2025)
Preserving Angles Improves Feature Distillation
Evelyn J. Mannix
enelyn..manniin@@nnnmccc.. wdw.eeewbbbee...ooo.oooooo.
School of Mathematics and Statistics Uniuersity of Melbourne
Liam Hodgkinson
thodgläinsonn@uniiilll...c
School Mathematics and Statistics University of Melbourne
Howard Bondell
howard hww..moeooddl@@@eeeebb..e.. wdw.ellbbedd..ooo.
School of Mathematics and Statistics University of Melbourne
Reviewed on OpenReview: https:// //tp::///penrevieuu. net/ forum? id= ZEhgODZWWW
Abstract
Knowledge distillation methods compress models by training a student network using the classification outputs of a high quality teacher model, but can fail to effectively transfer the properties of computer vision foundation models from the teacher to the student. While it has been recently shown that feature distillation wweiionnwweeeeeee a teacher model's output features replicated instead- reproduce performance for foundation models across numerous downstream tasks, they fall short in matching critical properties such robustness and out-of- ditro--dittributin (OOD) detection performance. This paper overcomes this shortcoming by introducing Cosine-similarity Preserving Compression (CosPress), a feature distillation technique that learns a mapping to compress the latent space of the teacher model into the smaller latent space of the student, by preserving the cosine similarities between image embeddings. This enables ciirect optimisation of the student network and produces more faithful reproduction of the teacher's propertics. It is shown that distillation with CosPress on variely of datasels, including ImageNet, produces more accurate models with greater performance on generalisability, robustness and OOD detection benchmarks, and that this technique provides a competitive pathway for training highly performant lightweight models on small datasets. Code available at githti. com/ wtt.//ooom//iiiieeeee prraanixxcosspeer
1 Introduction
Deep learning computer vision approaches have become the standard for automating vision problems across
range of fields, from medical imaging (Zhang & Metaxas 2024) to analysis of satellite imagery (Bastani
et al. 2023) and detecting weapons in luggage (Andriyanov 2024). However, models trained with commonly
used supervised learning approaches can have poor robustness (Bai et al. 2021 Hendryeks et al. 2021b
and struggle to detect out-of-distributioo (OOD) data (Yang et al. 2022a Nguyen el. al. 2015).
By leveraging large Vision Transformer (VTT) architectures and pretraining on large and diverse datasets, foundation models in computer vision comprise significant step forward toward addressing these challenges, providing significantly improved generalisation ability and robustness in comparison to purely supervised approaches (Oquab et al. 2024 Radford et al.) 2021). Large VIT models enjoy superior performance after pre-training (Zhai et al.] 2022). and can be distilled to produce smaller models that are more practical for deployment. For example, smaller DINOv2 foundation models were distilled from their largest variant with 1.1 billion parameters by optimising the self-supervised training objective with a frozen teacher (Oquab al., 2024). This approach not replicable, it was conducted on the proprictary LVD-142M dataset and the teacher head weights never publicly released.

---

Published in Transactions on Machine Learning Research (11/2025)
DINOv2 CosPress
DINOv2
CosPress
DINOv2
CosPress
Figure 1: Patch features. PCA visualisation of patch features for the DINOv2 ViT-S/14 model, and the distilled ViT-Ti/14 model produced using the CosPress feature distillation approach.
Nevertheless, knowledge distillation approaches (Hinton et al. 2015 that leverage the classification outputs of the DINOv2 models trained on a particular datasels can be used to Irain performant student models. However, it has been shown that fundamental properties of the foundation model need not transfer to the student models, impacting generalisation performance on downstream tasks (Zhang et al. 2025).. Feature distillation approaches that use Mean Squared Error (MSE) or L2 loss and student head map the activations from the latent space of the student to the teacher are more effective in this respect. The recent Proteus (Zhang et al. 2025) approach has shown that it is possible to distill the DINOv2 models on ImageNet--1K and obtain comparable performance on downstream classification and segmentation tasks. However, sub-optimal results are still obtained for key robustness metrics, as well as generalisability and performance dense tasks such as image segmentation. Most concerning, however, is that models distilled using Proteus do not failhfully reproduce the latent space of the teacher model, as shown by their severely reduced performance out-of-distribution (OOD) detection tasks (Table D
This paper presents Cosino-similarity Preserving Compression (CosPress), feature distillation approach that addresses such shortcomings by learning a teacher head mapping that compresses the latent space of the teacher model into the latent space the student. CosPress achieves this by preserving cosine similarities between points in the teacher's latent space and allows the student to be directly optimised. This significantly improves the faithfulness of the learned student on OOD detection (Table D and robustness benchmarks in comparison to Proteus, while achieving competitive performance across all of the considered challenges including classification accuracy, generalisation and semantic segmentation. CosPress reproduces the high- quality patch features of the foundation model (Fig. D. and can be used to produce specialised models, that have improved accuracy on particular tasks but retain these foundation model properties. In this work, we:
present CosPress, an approach for distilling VIT foundation models that learns a mapping from the latent space of the teacher to the student that preserves cosine similarities and allows direct optimisation of the student model;
demonstrate that CosPress produces more faithful student model, better replicating the perfor- mance of the teacher across a range of metrics including robustness, generalisability and out-of- distribution detection; and
Table 1: Out-of-distribution detection. Comparison of performance on the OpenOOD benchmark for the ImageNet-1K dataset. The t means larger values are better and the 4 means smaller values are better.
Method
Arch
Teacher
Near OOD
Far OOD
DINOe2
AUROCT
FPR]
AUROCT
FPRJ
Proteus
VIT Ti/14
VIT V/14
64.17
85.73
74.22
67.97
CosPress
VIT-1/114
VIT-S/14
70.49
77.29
91.03
37.21
VIT-S/14
74.12
92.67
29.55
Proteus
VFT-8/14
VFI-B/114
61.19
94.56
61.92
86.78
CoePress
VIT-S/14
VIT-B/14
73.5
73.84
92.93
28.98
2

---

Published in Transactions on Machine Learning Research (11/2025)
show that CosPress can be used to train specialised models with improved performance particular vision task, while retaining foundation model properties such as improved generalisability and out- of-distribution detection performance.
2
Related Work
Foundation models Foundation models in computer vision follow the success of ceasffomer-baacee foun- dation models in language, such as BERT (Devlin et al. 2019), and encode images as vectors in latent space, where the distance between vectors describes the semantic similarity of the images. Two approaches have emerged for training these models: self-supervised learning (Oquab et al., (024) and contrastive language- image pretraining (CLIP) (Radford al. 2021). The CL.IP models were among the first. to show that by using a large Vision Transformer (ViT) architecture (Dosovitskiy et al. 2021 ; and large, diverse and high quality training dalasel, generalist vision model could be produced that achieves high performance across range of applications (Radford et al. 2021). The DINOv2 foundation models followed, and using a com-
bined bootstrapping (Grill et al. 2020 Caron et al. 2021 and masked patch prediction (Zhou et al., 2022)
approach to train loundation models with strong performance on image classification and segmentation tasks (Qquab et al. 2024).
Knowledge distillation. Knowledge distillation is the process of transferring knowledge from a large model or model ensemble to single smaller model. The earliest approaches aligned the output probability vectors of the student and teacher classifications using a Kullback-Lcibler (KL) divergence loss (Hinton et al. 2015). There is a wide range of literature demonstrating how this approach can improve the performance of smaller Convolutional Neural Networks (CNNs) (Wei et al. 2020) and Vision Transformers (ViTs) (Touvron ett al. 2021 Yang et al. 2024). by leveraging strong teacher or one with different inductive biases to the student model. It has been shown that knowledge distillation is most effective when it is treated as function matching problem (Beyer et al. 2022 with the same inputs being provided to both the teacher and student model.
Feature distillation. Feature distillation where the output features of the teacher are used for training the student instcad of the classification outputs-is less well studied for models without class outputs. Generally speaking, leature distillation is used in combination with a knowledge distillation objective and often focuses on supervised models. However, the Proteus approach (Zhang et al., 2025) demonstrated that using pure feature distillation objectives is important for preserving foundation model properties. The components of Protens- - student head to align output dimensions, MSE loss on class and patch tokens, and an iBOT [Zhou et al., 2022) inspired masking objective- are a logical adaption of components in prior supervised fcature distillation methods such as Masked Generative Distillation (MGD) (Yang et al.) (022b). SRD (Miles & Mikolajezyk/ 2024) and VeD (Miles et al.] 2024) to a VIT architecture with the aim of preserving both the local and global features of the teacher.
are
Dimensionality reduction. Feature distillation and the challenge of compressing latent spaces to train performant student models are closely related to the broader ideas of dimensionality reduction and minimum distortion embeddings (Agrawal et al. 20211. Stochastic Neighbor Embedding (SNE) (Hinton & Roweis 2002 Van der Maaten & Hinton 2008) is dimensionality reduction technique that projects high-dimensional embeddings into low-dimensional space (typically two dimensions) while preserving local relationships. SNE acheives this by constructing probability distribution over pairs of points in the original space and then oplimising corresponding set of points in the low-dinensional space to match this higher dimensional distribution as closely as possible. However, SNE does not learn an explicit mapping between the original and reduced spaces, only lower dimensional representation. Other approaches, contrast, explicitly learn projection functions, often with the goal of preserving local geometric structures such as distances or angles
(Saul & Roweis 2000 He & Niyogi 2003 Gao et al. 2020 Fischer & Ma 2024).
3

---

Published in Transactions on Machine Learning Research (11/2025)
Proteus
CosPress
Compression
Frozen teacher
Proteus loss Cproteus
Teacher head he
loss
T
ddim-red
Image
x
Cosine-similarity
Student Sg
Student heads
Student loss
Preserving Compression Map
98: 9 Iv
Lstudent
Figure 2: Feature distillation frameworks. Proteus, student heads g are used to map the outputs of the student network Sg into the latent space of the teacher T, so that a MSE loss can be applied. In CosPress, a teacher head trained to compress the teacher T outputs into the student latent space, preserving the cosine similarity of image embeddings and allowing direct optimisation. The Proteus student head does not preserve cosine similarity, even when the projection matrices are forced to be right-orthogonal.
3 Methods
Notation. We consider a feature distillation setting, where there is a small student network So with output dimensionality Ds and larger frozen teacher network T with output dimensionality Dr- These networks use a ViT architecture, batch stochastic gradient descent setting, defined for batch of images E X. When only the output class tokens are considered STxr)), (T(;) is used. We write S(r,). to refer to a matrix of the concatenated patch and class token outputs.
assume Dr > Ds. The loss functions presented in this paper consider mini-
Motivation. We are interested in the problem of training a student network to mimic the behaviour of a large, high quality teacher model using a ViT architecture, such as the DINOv2 foundation models. A key property of these models is that the cosine similarity between images and patch embeddings captures their semantic similarity, as shown by the use of this measure for zero-shot classification and identifying duplicate imagery (Jose et al. 2024 Oquab et al., 2024; Radford et al. 2021 p. However, larger ViT architectures have larger output dimensionality, which prevents the embeddings produced by smaller student models being directly compared to a teacher model embedding.
Proteus (Zhang et al. 2025 addresses this problem by introducing a student head g RDs - RDr that maps the outputs of the student model into the latent space the teacher model, allowing a MSE (L2) loss to be applicd. This student head contains a projection matrix, W Dr that maps the from the latent space of the student to the teacher, and is commonly discarded. Two issues arise with this approach. First, the projection may encode information specific to replicating the teacher network, potentially distorting the outputs of the student model, which is only indirectly optimised (Miles et al. 2024). Second, there is no guarantee that the projection matrix W will be faithful in preserving cosine similarities between teacher embeddings of different images similarities that reflect semantic relationships (Jose et al. 2024 within the studont's 's latent space.
Prior work has found that requiring W to be right-orthogonal addresses the first problem (Miles et al., 2024
Right-orthoognaliiy means that WWT = IDs must be satisfied where Ipg DsxDs is the identity matrix of rank Ds. However, this implics that for any image i, we have
W R (x,)) - S5(1,) x (T)
(1)
Consoquently, the cosine distance between image embeddings the student network relates to the teacher, for any two images i, via
S5(2,) (5 S I| [,,,,,
WT)) (x)
(2)
[T<x,; W )

---

Published in Transactions on Machine Learning Research (11/2025)
This implics that W also needs to be left-orthogonal to address the second problem and ensure that cosine similarities are preserved. Asserting left-orthogonlityy with W W = aaalp for some scalar would yield the desired relationship
ST(2,) S5(e,) |S (1,))]
(3)
Unfortunately, the projection W can only be approzimately left-orthogonal, as W is only of rank Ds which means the product W can only ever be of rank Ds. As ID, of rank Dr > Ds, then W W o alD, Consider the following definition.
Definition 1 (Approximately Orthogonal Matrix). A matrix M Rmxd is said to be approzimately or- Uhogonal if
|MM - almllF < and |M M - Aalle <E,
for sufficiently small > 0 and real scalars a, B. If only one of these conditions are satisfied the matrix is said to be approximately right or left orthogonal, as appropriate. This notion expands upon the idea of orthogonality, where a matrix M can only be orthogonal (s = 0, = = 1) if it is square (m = d).
Lemma 1. Let M E Rmaxd with m < d and rank (M) = m. Then
|MM T
- m L S M M- - La
Moreover, the converse inequality does not generally hold.
This lemma shows that approximate rightt-orUhoonnality is sufficient for matrix to also be approximately left-orthogonal, and therefore to be approximately orthogonal overall. Consequently, both conditions Eq. ([) and Eq. (3) are satisfied, ensuring that the mapping does not encode information while preserving the relationships between image embeddings. The proof of Lemma Theorem stemss from M having more columns than rows, which can be downsampled to form a square matrix, and is provided in Section of the supporting information. While prior methods such as V&D introduce parametrisations that require W to be right-orthogonal (Miles et al., 2024), they do not also guarantee approximate left-orthogonality.
While Eq. 3 could be optimised directly using SNE (Hinton & Roweis 2002 Van der Maaten & Hinton 2008) inspired approach, our initial experiments found that this was less effective than Proteus due to this target having complex loss surface with many local minima. Instead, it is proposed to use teacher head, rather than a student head, to learn a function h& Pr + Ds that compresses the representation of the teacher into the latent space of the student while preserving cosine similaritics
T( (13) [T=(3:)] [T)
h (T(T)))))) (xT()( [T>a(T Ihh (Tx)(
(4)
which allows the student Sa;;)) to be directly optimised against the compressed teacher representation hs (x.))) Learning the he mapping is tractable problem as described by the Johnson Lindenstrauss (JL) Lemma, which states that such mapping can be constructed with a margin of error that depends the dimensionality of the larget space and the size of the dataset of interest (Freksen] 2021). Further details are provided in Section and Fig. 2 highlights the differences between the Proteus and CosPress frameworks.
Proteus. Zhang et al. 2025 propose to minimize the MSE (L2) loss between the outputs of the teacher and that of the student, when passed through a dimension-raisingg map called the student head g RR + R Pr. To achieve best performance, they use three student heads with different weights Φ, 1/, v and minimise the Lg loss separately on the class tokens, fcatures (class and patch tokens), and on randomly masked tokens x similar to the MGD Yang al. 2022b) approach. This leads to the following optimisation loss
Sproteus (x: (X,), (2G
(5)
+ (S(()
(So ( M)
5

---

Published in Transactions on Machine Learning Research (11/2025)
CosPress. Our approach, CosPress, separates the challenge of feature distillation into two parts, CoaPrress (X; Φ. 0) = Ldim-red (X: Φ) + Letudent (X;
(6)
Firstly, a leacher head hig : PR D2 is learnt. to map the teacher outputs the latent space of the student network Sş while preserving cosine similarities. This dimensionality reduction loss term Ldim-red is independent of fitting the student network. It only requires the target dimension in order to fit the teacher head ha-
Secondly, the student network So is trained to match the image of the teacher under the teacher head ha T in the student loss term Lstudent- The most effective way to fit this term is to freeze the teacher head he gradients and train on both losses concurrently as shown in Fig. Using a weighting scheme was observed to produce similar results (Section
Dimensionality reduction objective. To build a loss function that will ensure the mapping he satisfies Eq. an SNE (Hinton & Roweis 2002 Van der Maaten & Hinton 2008) inspired approach is used. This involves defining kernel build distributions describing the similarity between vectors, allowing for embeddings in the high dimensional imput space to be aligned with the low dimensional target space by minimising the KL divergence between these distributions.
We define kernel using the von-Mises Fisher distribution, where for input vectors and we have
2y; exp ( y y.z
|b [1]]| (T)
(7)
with temperature hyperparameter т. As result, Eq. (4 becomes
k (T (x:): ky (ho(T (T(T
(8)
for each i,j. Then, for a set of vectors in the input space E and target space E of size N, we construct the matrices P", Q that define the input and target distributions by
P Pale + Pay
hi kr
2.V
Paja
Σner k ky (p:) pk))
(9)
Q 4z + Dil
hi (yr 4i)
4if Σeer ky (q6 4k)
(10)
2N
where the first equation builds symmetric PT, Q" matrices, allowing for greater flexibility in the solution. If these Pr, Q" matrices are equal, the cosine similarity between pairs of points in p.q will be equal and Eq. 4 will be satisfied. This can be achieved approximately by minimising the KT, divergence (DKI.) over T, a vector of temperature values via
TRL q)
Σ TET
DKI((TT)
(11)
T
An ablation study on the best values of is described in Table S21 in the supporting information.
Putting this all together, we propose dimensionality reduction loss that conserves cosine similarity at two levels between the image class tokens in batch, and between the features (patch and class tokens) within
an image
Edim-rod (X: x) (TT)
(12)
+ X
Σ
LKL (Txx)))
The calculation of LKI. (ha(T (X)) the only term in the CosPress loss that is calculated between examples in batch, and that will scale non-linearly with increasing batch size. All other terms in both CosPress and Proteus are computed within individual examples and scale linearly with batch size.

---

Published in Transactions on Machine Learning Research (11/2025)
Teacher head architecture. As done for the Proteus Zhang et al. 2025 student heads, the teacher head architecture in CosPress uses LayerNorm as
Ba 2016 followed by linear layer. This can be written
(11))
h2(2)=
|||| 7 $1
W -2,
(13)
where & is the average of the vector elements, and the initialisation scheme sets the biases B1. 32 to zero and the scaling y to at the start of training. The linear map W E I Ds is initialised using random normal distribution as is standard, which is consistent with the mapping constructed in the JL Lemma (Section Aj. For the Proteus student heads W is replaced by W and the dimension of the bias vectors are adjusted accordingly.
Student objective.
The student objective minimises cosine distance
Leusiue (z.y)
Σ
(14)
2 13lll
where y sets of input vectors of the same length Considering the teacher head he learns to conserve cosine similarity, this is a natural choice for the student network and is found to result in improved performance with CosPress than the L2 loss (Section C
Similarly to the dimonsionality reduction objective, the final student loss employs both a class token loss and a feature loss term
Estudent (X;0) = Locstue (S)(X), (T)())))
(15)
+ Lossine (X, (T(X))
4 Experiments: Feature distillation
In this section, the CosPress feature distillation approach is compared to Proteus and distilled variants of the DINOv2 models. While we do not have access to the proprictary LVD-142M dataset used distill
the DINOv2 models, it has been shown that ImageNet-1KK (Russakovsky et al. 2015) is sufficient to distill models with comparable accuracy across range of measures (Zhang et al. 2025
4.1 Experimental setup
Vision Transformer (Dosovitskiy el al., 2021 models are distilled using larger DINOv2 teachers on the ImageNet-Ikk (Russakovsky et al. 20155) training dataset, comprising 1000 categories across more than 1.2 million training images. To enable fair comparison, we reproduce the results of the Proteus paper and train CosPress models using unified codebase. This ensures consistency in the optimizers, samplers, augmentations and other hyperparameters. Following Proteus (Zhang et al. 2025), student networks are distilled for 300 epochs using a batch size of 1024, cosine learning rate decay with five warmup epochs [oshccillv & Hutter, 2017a), an AdamW optimizer [Loshchilov & Hutter 2017b), a repeated augmentation sampler with three views per image (Fort et al. 2021), and Rand (Cubuk et al., 2020) image augmentations (Wightman) 2019 An ablation study on the hyporparameters introduced by CosPress is provided in Section C of the supporting information.
Following the DINOv2 kNN evaluation (Wu et al. 2018 and linear probing approach (Oquab et al. 2024 with an additional batchnorm layer (Lee et al. 2023), evaluations are undertaken on the ImageNet validation set, as well as nine fine-grained classification benchmaaks (Oxford Pets (Parkhi et al. 2012). FGVC Aircraft (Maji et al. 2013), Describable Textures (Cimpoi et al.) 2014). Stanford Cars (Krause et al. 2013). CUB200
(Wah et al. 2011), CIFAR-10/100 (Krizhevsky et al. 2009), Flowers- (Nilsback & Zisserman 2008 ) and
Food-101 [Bossard et al. 20144) and the Pascal VOC 2012 segmentation task (Everingham et al. 2012). Performance is also tested several robustness and generalisation benchmarks including ImageNet-V V2

---

Published in Transactions on Machine Learning Research (11/2025)
Recht et al. 2019 Sketch Waan et al. 2019 ImageNet-R Hendrycks et al. 2021a] and ImageNet-A Heeddyykks et al. 2021b
We additionally consider the OpenOOD benchmarks (Yang et al. 2022a). Foundation models are trained on a diverse dataset and excel in this task, and whether distilled students can reproduce this performance has not. been previously considered. This section focuses on the ImageNet-1K OpenOOD) benchmark, which uses SSB-hard (Bitterwolf et al.) 2023) and NINCO (Vaze al. 2022) as near OOD data, and iNaturalist [Van Horn et al, 2018). OpenImage-O (Wang et al. 2022 and Describable Textures (Cimpoi et al. 2014) far OOD data.
4.2 Results
Table 2: ImageNet classification. Comparison of performance on ImageNet-IK under kNN and linear probing evaluation approaches. We report the mean and standard deviation over four runs with different random seeds for the Proteus and CosPress ViT-Ti/14 models.
Method
Arch
Teacher
kNN
Linear
Proteus
ViT-Ti/14
DINOv2 VIT-8/14
73.0 0.1
76.1 + 0.2
Proteus V,D
VIT T1/14
DINOv2 VIT 8/14
73.0
75.9
CosPress
VII-Ti/14
DINO22 VIT-S/14
743
76.6
DINOv2 VIT-S/14
79.0
81.1
Protens
VIT-S/14
DINOv2 VFT-B/14
79.8
82.0
CosPress
VIT-S/14
DINOv2 VIT-B/14
80.4
82.3
Table 3: Distillation components. Results for kNN evaluations different components of the distillation process for models distilled on ImageNet-1K. For Proteus, results are shown for the class token sludent head.
Method
Arch
Teacher
kNN
DINOc2
Backbone
Stu. head
Tea. head
Teacher
Proteus
VIT-Ti/14
VFT-S/14
73.1
73.5
79.0
Proteus-V.DD
VIT-Ti/14
VIT-S/14
73.0
73.3
79.0
CosPress
VIT-Ti/14
VIT-S/14
74.3
78.8
79.0
Proteus
VIT-S/14
VIT-B/14
79.8
80.0
82.1
CosPress
VIT-S/14
VFT-B/14
82.1
CosPress trains more competitive students. Table 2 shows that CosPress trains students with better performance in comparison to Protous (Zhang et al. 2025), for both the lincar probing and kNN evaluation methods. These improvements are statistically signilicant, taking into account the low variability observed across different random seeds. It is also found that the teacher head can project the embeddings from the teacher network into the latent space of the student with minimal loss of kNN accuracy, and that the token student head from Proteus has higher kNN accuracy than the model backbone (Table 3 These observations confirm the motivations for CosPress the Proteus student heads are not. an uninformative mapping into higher dimensional space, but are contaminated with information relevant for reproducing the teacher model. Further, a high-quality projection that compresses the teacher emebeddings into the latent space of the student that preserves cosine similarity- can be learnt, and this provides more effective supervision.
In Table 2 we also consider Proteus-V,,D where the projection matrices W in the Proteus student head are constrained to be right-orthogonal using the V&D approach (Miles et al. 2024). This method builds penaamettristtoonn map using skew symmetry and matrix exponential approximation to construct W such that it is approximately right-orthogonal. Table 2 shows that in this context, this pe-aaammttrisaoonn does not significantly impact performance, and does not. completely prevent. contamination of the student. head Io-
It is also observed in Table that the CosPress and Proteus approaches can outperform the distilled DINOv2 models of the same size. However, it is challenging to determine if these distillation approaches
8

---

Published in Transactions on Machine Learning Research (11/2025)
effective, as the DINOv2 models were trained on larger proprictary dataset, of which the ImageNet- 1K dataset was only a small subset.
Scaled Non-diagonal Elements
Scaled Diagonal Elements
30
CosPress
30
Proteus
Proteus-V,D
20
Density
Density
20
10
0
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
0.6 0.8
1.0
1.2
1.4
Figure 3: Visualising orthogonality. Kernel density estimate plots of the diagonal and non-diagonal elements for the scaled Gram matrices of the lincar maps W in the teacher and student heads, drawn from CrsPress and Proteus respectively. The dashed coloured lines represent W W/a and the solid lines represent W3T/3, where a are defined as in Table 4 A perfectly orthogonal matrix W will have a Gram matrix with density on the black dashed vertical lincs.
Table 4: Measuring orthogonality. Distance measures of the scaled Gram matrices A WT and B = WWT/B for the projection matrix in the Proteus student and CosPress teacher head, and their respective identily matrices ID,, D of the same dimensions. For each matrix o, B is set to the mean of the diagonal elements of A, B, which minimises error under the Frobenius norm.
Method
Distance the Identity Measures
Jpy
|B Tpelle
'Ir[[A
Tr Ioel)
Proteus
27.3
17.0
Proteus: D,D
152.7
CosPress
20.3
2.6
3.4
CosPress learns an approximately left and right orthogonal projection. As theorised in Lemma found that CosPress learns a lincar map W the teacher head (Eq. (13 that is approximately left and right-lrthogonal, up to a scaling factor. More concisely, we find that
W W
(16)
where a, 3 are positive real numbers. Qualitatively, this can be seen in Fig. where kernel density plots are shown of the elements from Gram matrices formed using the linear maps W, W These projections are taken from the CosPress teacher head and the Proteus class token student head obtained while training the ViT-Ti/14 student network. The scaled CosPress Gram matrices are much closer to the identity matrix, and this measured quantitatively using the Frobenius norm and trace in Table
While the Proteus- DgD approach in Fig. 3 Band Table 4 does learn projection W that is approximately right- orthogonal, there is a large degree of error. This is due to the approximation of the matrix exponential that is used in the V&D method (Miles et al. 2024 and CosPress is able to learn a right-orthogonal projection matrix with less error.
CosPress improves performance on classification tasks. Table 5 shows that the models distilled by CosPress have improved or similar performance over Proteus for all downstream fine-grained classification tasks. Competitive accuracy is also achieved with the distilled DINOv2 models the same size, which CosPress outperforms on six of the nine datascts. Poorest performance is achieved on the FGVC-Aireraft dataset, which likely reflects differences in the training data used for distillation. The LVD-142M. dataset used to train and distill the DINOv2 models contains million images with high similarity to the FGVC-Aireras
9

---

Published in Transactions on Machine Learning Research (11/2025)
Table 5: Fine-grained classification. Comparison performance on fine-grained classification tasks using linear probe evaluation.
Method
Arch
Teacher
Dataset
C10
C100
Food
CUB
DTD
Pets
Cars
Airer
Flowers
Average
Proteus
VIT-T1/14
DINO,2 VIT-S/14
95.1
$1.4
83.5
84.1
72.9
94.2
72.8
51.1
96.0
81.6
CosPress
VIT-T//14
DINO.2 VIT-S/14
94.9
81.9
84.6
85.1
73.8
94.1
75.3
55.7
96.8
82.5
DINOy2 VIT-S/14
97.7
$7.5
89.1
88.1
80.6
95.1
81.6
71.0
99.6
88.1
Protens
VIT-8/14
DINOv2 VTT-B/14
97.8
87.7
89.7
88.4
78.0
95.9
82.8
62.9
97.6
86.8
CoePress
VIT S/14
DINOv2 VIT B/14
97.8
87.6
90.3
88.9
78.0
95.9
84.0
63.4
98.8
87.2
dataset (Oquab et al. 2024), whereas ImageNet only contains a single airliner class with approximately 1300 images.
Table 6: Semantic segmentation. Comparison of performance on the Pascal VOC 2012 semantic segmen- tation task using a linear probe.
Method
Arch
Teacher
mloU
Proteus
ViT-Ti/14
DINOv2 VIT-S/14
70.5
Proteus w/o patch loss
VII-Ti/14
DINO<2 VIT-S/14
09.7
CosPress
ViT-Ti/14
DINOv2 VIT-S/14
71.1
DINOv2 VIT-S/14
81.2
Proteus
VIT-S/14
DINOv2 VIT-D/14
77.3
Proteus w/o patch loss
VIT-8/11
DINOv2 VIT-B/14
77.1
CosPrees
ViT-S/14
DINOv2 VFT-B/14
77.9
CosPress improves segmentation performance. Table shows that CosPress also improves accuracy downstream segmentation tasks comparison Proteus. CosPress does not include the masked patch loss objective, that confirm improves the performance of Proteus dense tasks, and incorporating it into CosPress may improve performance further. The DINOv2 distilled model outperforms CosPress in this case. Further pretraining with an increased image resolution was found to be key to improving the performance of the DINOv2 models on dense lasks (Oquab et al. 2024), but is not undertaken in training the CosPress and Proteus student models.
Table 7: Robustness and generalisation. Comparison of performance on ImageNet-1K robustness and generalisation benchmarks.
Method
Arch
Teacher
Test Dataset
DINO.2
IN. V2
Sketch
IN-R
IN
Proteus
VrT-17/14
VIT-S/14
64.3
23.5
37.8
11.4
CusPress
Ti/14
VIT S/14
64.9
27.9
40.7
13.2
VIT-S/14
(Oquab et al. 2024
70.9
41.2
53.7
33.5
Proteus VIT-S/11
VIT-B/14 VFT-R/14
72.2
38.1
50.0
29.6
CoePress
V/1-
72.5
40.4
52.3
31.5
CosPress distills a more robust student model. Table 7 shows that CosPress results in improved performance over Proteus across a range of ImageNet-1K robustness and generalisation benchmarks. The DINOv2 distilled model obtains better performance in this instance for all benchmarks except ImageNet- but CosPress closes the gap between the ImageNet-II and LVD-142M distilled models significantly.
CosPress reproduces the OOD detection performance of the teacher. CosPress is faithful to the teacher networks when comes to OOD detection performance, as shown in Table Proteus performs very poorly on this benchmark, with worse performance observed for larger student models. In contrast, CosPress is able to distill models that have strong OOD performance, even outperforming their DINOv2 counterparts.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Table 8: Feature distillation with different teachers. Comparison of performance on ImageNet- under kNN and linear probing evaluation methods with other kinds of teacher backbones, using different architectures and training approaches.
Method
Arch
Teacher
kNN
Linear
Proteus
VIT-T/14
DINOv2 VIT-B/14 w/breg
71.1
76.1
CosPress
VIT-T/14
DINOv2 VIT-B/14 w/reg
74.0
76.4
Proteus V/T-
CLIP ViT-B/16
63.6
71.4
CosPress
VIT-T/16
CLIP VIT-B/16
64.0
72.0
CosPress improves performance across other teacher networks. Table 7 demonstrates that Cos- Press also trains higher performing student networks than Proteus when using CL.IP (Radford al. 2021) and DINOv2 w/ w/reg DDarett et al. 2024 teacher networks. These experiments employ the same hyperpa- rameters as those in Table 2 Additional results exploring feature distillation with ViT-T students and larger teacher works are provided in Section B of the supporting information.
Table 9: Training time. Comparison of training time on ImageNet for 300 epochs with batch size of 1024 using Nvidia A100 GPUs.
Method
Arch
Teacher
GPUs
GPU hours
GPU memory
Proteus
VIT Ti/14
DINOv2 V/T-
55GB
CosPress
WII-T//11
DINOv2 VIT-S/11
95
17GB
Proteus
VIT-S/14
DINOv2 VIT-B/14
182
111GB
CosPress
WII-S/11
DINOv2 VIT-B/14
154
SIGB
CosPress does not require additional computational resources. Table g provides timings and GPU memory usage for fitting the Proteus and CosPress models described in this section. Training time is similar for the ViT-Ti/14 and DINOv2 ViT-S/14 student-teacher pair, but CosPress is more efficient for larger models. This is due the masked patch loss in Proteus, which requires that the student network is evaluated once unmasked inputs, and second time on masked inputs. As result, training is faster for CosPress with larger students. CosPress is also slightly more memory efficient compared to Proteus, and further computational savings could be made by freezing the teacher head h& once sufficiently high quality map has been learned.
5 Experiments: Specialist models
This section explores the potential for CosPress feature distillation to improve the performance of specialised models that solve one particular task (e.g. classifying images food). We refer to this process, where an additional feature distillation training step is undertaken on a larget dataset, as CosPress finetuning, This approach can train highly performant small networks, that also have improved results on generalisability and OOD detection benchmarks.
5.1 Experimental setup
The CosPress models distilled in the previous section are compared with models that have been further finetuned with CosPress additional pretraining step where distillation is undertaken on smaller target dataset of interest. The strong DeiT (Touvron et al.) 2021) pretrained weights are also considered, which were distilled from ImageNet-1K with a larger CNN network using class-based knowledge distillation (Hinton et al. 2015). The same hyperparameters and training methodology is used as in Section with the exception of the number of training and warmup epochs.
This section focuses on set of small-scale tasks, including CIFAR- 100 (Krizhevsky et al. 2009 Food- 101 (Bossard et al.] 2014) and Oxford Pets (Parkhi al. 2012). We employ 300 training epochs and 10 warmup epochs for CIFAR-10/ and 3000 training epochs and 100 warmup epochs for Oxford Pets. The
11

---

Published in Transactions on Machine Learning Research (11/2025)
DINOv2 linear probe evaluation method is employed (Qquab et al., 2024), as well as finetuning using the DeiT recipe (Touvron et al. 2021p. When training models with this latter approach, the linear prediction head is trained before finetuning the backbone, to avoid distorting the pretrained features (Kumar et al. 2022
5.2 Results
Table 10: Specialist models - accuracy. Comparison of performance on fine-grained image classification tasks.
Method
Arch
Teacher
Pretraining dataset
Linear
DefT
C10
C1000
Food
Pets
C10
C100
Food
Pets
Deil Touvron al 2021
VIT-Ti/16
RegNeeY-16GF
ImageNet
98.1
17.7
93.3
98.3
89.9
CosPress
VIT-T//14
DINOv2 VIT-S/14
ImagoNet
94.9
81.9
84.6
94.1
96.7
80.0
91.7
92.7
CoePresss
VIT-T1/14
DINO22 VIT-S/14
ImageNet -> Target dataset
97.6
86.3
89.8
94.9
98.8
89.6
92.7
93.0
CosPress finetuning improves the performance of specialist models. Table[[0]showe that CosPress finctuning improves downstream performance, even when the training datasets are quite small. For every dataset considered, this additional pretraining step improves linear probe evaluations with a frozen backbone by significant margin (1-5%). These benefits remain under the strong DeiT training recipe, which further finetunes the model backbone. While CIFAR-10/100 and Food-101 have much stronger results under DeiT finetuning, we find that Oxford Pets has best performance with a linear probe evaluation after CosPress finetuning. This reflects the small size of the Oxford pets dataset, which makes training ViT networks challenging.
Table 11: State-of-the-art lightweight models. Comparison of best CosPress models to other approaches for training state-of-the-art lightweight models for specialised tasks.
Method
Architecture
Parameters
Dataset
C10
C100
Food
Pets
NAT
2021
MobileNetV2 Sandler
2018
1.5-9.0M
98.4
88.3
89.4
91.3
Cei'l
Y Tua
2021
CeiT-T
6.4M
98.5
88.4
93.8
CoePress
VIT-Ti/14
5.5M
98.8
89.6
92.7
94.9
A ViT-Tiny network finetuned with CosPress have competitive accuracy compared to other approaches in the literature that have been highly optimised to perform well specialist tasks with small and efficient model. Table I shows that CosPress finetuning trains competitive networks in comparison to Neural Architecture Transfer (NAT) (Lu et al. 2021) and Convolution-enhancel image Transformers (CeiT) (Yuan et al. 2021). Feature distillation methods like CosPress are an additional approach, that could used conjunction with these techniques to build highly performant lightweight vision models.
Table 12: Specialist. models - generalisability. Comparison of generalisability of specialist. models on the cartoon subsets of the CIFAR-10-W benchmark (Sun et al. 2024). We report mean per-class accuracy due datasct imbalances. In-distribution training images (top) and cartoon images (bottom) are included for reference.
Method
Arch
Teacher
Pretraining dataset
Linear
DefT
Dif
Bin
Bai
Diff
Bin
Bai
DeiT Touvron al.
2121
VIT-TE/16
HegNet
ImnageNet
65.9
51.0
47.6
48.8
86.9
62.6
56.0
60.1
CosPress
VIT-T//14
DINOv2 VIT-S/14
ImageNet
70.8
19.5
48.7
50.3
88.5
63.2
56.3
60.7
VIT-T1/14
DINO22 VIT-S/14
TmaggNet
Tarpot datasot
73.4
52.5
48.6
49.9
89.1
64.1
67.8
12

---

Published in Transactions on Machine Learning Research (11/2025)
CosPress finetuning improves the generalisability of specialist models. The challenging cartoon subsets of the CTFAR-1O-W benchmark (Sun et al. 2024) are used to test generalisation performance on CIFAR-10. Table 12 shows that CosPress finetuning leads to improved generalisability for specialist models on CIEAR-10. Under a linear probing evaluation, CrsPr finctuning strongly improves generalisability on two of the four datasets, and improves generalisability for all datasets even after DeiT finetuning.
Table 13: Specialist models - OOD detection. Comparison of performance on the OpenOOD bench- mark Yang ct al. 2022a). The AUC is reported for detecting OOD images.
Mathod
Anch
Touchee
Pretraining
Frusees lackhonn
DoiT fineumned
CI
CIFAR
CIFAR
CIFAR:
Nec-OOD
Par-OOD
Nea--ODD
Far-OOD
Near-000
FarOODD
NeanOOD
Far-O00
DelT Touveon
2021
VIE-T/1G
BegSoY-166GP
ImageNNa ImageNet
67.01
47.01
68.14
$6.19
96.75
95.00
87.45
86.73
VII-1/114
DENOv2 Vrn-5/14
03.04
05.97
85.81
76-37
06,00
08.00
87.00
CooPrea
TV/4
DENOv2 S/14
ImageNNt
Tatget datasct
CosPress finetuning improves the OOD detection performance of specialist models. The
CIFAR-10/100 10/000 OpenOOD benchmarks (Yang et al., 2022a) are used to test OOD detection performance. Table 13 shows that CosPress finetuning leads to improved performance on the OpenOOD (Yang et al,,
2022al) benchmark for specalist models CIFAR-10/ 10/00.. Without DeiT finetuning, strong improvements
in OOD detection are observed under CosPress finetuning over the ImageNet pretrained baselines. Some improvements remain after DeiT finetuning, but are smaller.
6 Discussion
Significance of angles in deep learning. Language models have been shown to exhibit a principle of su- perposition, wherein concepts are encoded along nearly ort hogongl directions in representation space (Bricken) it al. 2023 While only d vectors can be exactly orthogonal in a d-dimensional space, high-dimensional geometry allows for the construction of up cxp(d) approximately orthogonal vectors (with pairwise cosine similarity less than 0) enabling the representation of vastly larger sel of concepts in practice (Elhage et al. 2022 This phenomenon is closely related to the Lindesta-s lemma (Freksen 2021). and
similar properties have been observed in foundation models for computer vision (Bhalla et al. 2024]. By
preserving angular relationships between image embeddings, CosPress maintains the semantic structure of the foundation model feature space in the student networks it. trains.
Limitations. Even with CosPress, a small generalisation gap remains. Models trained with CosPress do not generalise quite as well as the original DINOv2 distilled variants (Table 7). Without access to the proprietary LVD-142M dataset, it is difficult to determine whether this gap arises from the limitation of performing feature distillation solely on ImageNet-1K, or from a shortcoming in the methodology itself.
7 Conclusion
This paper introduces CosPress, a feature distillation approach designed to train highly performant student networks from foundation model teacher with Vision Transformer (Dosovilskiy et al. 2021) architecture, that reproduces their properties in regards to generalisation, robustness and OOD detection. This is achieved by introducing teacher head, that maps from the higher dimensional latent space of the teacher network into the smaller dimensional space of the student, and training this mapping to preserve the cosine similarity of images within these embedding spaces. CosPress trains a faithful student, that more closely replicates the behaviour of the teacher network in comparison to the Proteus approach (Zhang et al. 2025 where a
student head is used to align the student outputs with the teacher.
Acknowledgements
We acknowledge the Traditional Custodians of the unceded land on which the research detailed in this paper undertaken, the Wurundjeri Woi Wurrung and Bunurong peoples of the Kulin nation, and pay
13

---

Published in Transactions on Machine Learning Research (11/2025)
respects to their Elders past and present. This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE1 70100200. This research was also undertaken with the assistance of resources and services from the National Computational Infrastructure (NCT), which is supported by the Australian Government. Evelyn J. Mannix was supported by an Australian Government Research Training Program Scholarship to complete this work.
References
Akshay Agrawal, Alnur Ali, Stephen Boyd, et al. Minimum-distortion embodding. Foundations and Trends@ in Machine Learning, 14(3):211 378, 2021.
Nikita Andriyanov. Intelligent computer vision systems in the processing of baggage and hand luggage X-ray images. In Advances in Artificial Professor John Psarras, pp. 283 324. Springer, 2024.
Decision Support Systems: Papers in Honour of
Jimmy Lei Ba. Layer normalization. arXiv preprint arXio:1607.0644 2016.
Yutong Bai, Jieru Mei, Alan I. Yuille, and Cihang Xie. Are transformers more robust than CNNs? Advances in neural information processing systems, 34:26831 26843, 2021.
Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. SatlasPretrain: A large-scale dataset for remote sensing image understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognátion, pp. 10925-10934, 2022.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Interpreting CLIP with sparse linear concept embeddings (spliCE). In The Tighth- Annual Conference on Neural Information Processing Systems, 2024.
Julian Bitterwolf, Maximilian Müller, and Matthias Hein. In or out? fixing imagenet out-of-listribution detection evaluation, 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, Seplember 6-1 2014, Proceedings, Parl VI 13, pp. 661.441.666 Springer, 2014.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
URL lttp:::/trranafomeerccccrra htt.:/ruttt..ppp22222ooooaaaaaaa features.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised Vision Transformers. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 6650-9660, 2021.
Mircea. Cimpoi, Subhransu Maji, Tasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Lc. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702 703, 2020.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers need registers. In The Twelith International Conference on Learning Representations, 2024.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 4171 4171-4186, Minneapolis, Minnesola, June 2019. Association for Computational Linguistics.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDoodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. URL
https://arxiv: org/ abtp//ba../2..///////
2209. 10652
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pasce/ network.org/challengge. //C////////////// /tt////0////wwwooo/////ii/ workshopp/nndex..ue 2012.
Jonas Fischer and Rong Ma. Sailing in high-dimensional spaces: Low-dimensional embeddings through angle preservation. arXiv preprint arXiv:2406.0995 2024.
Stanislav Fort, Andrew Brock, Razvan Pascanu, Soham De, and Samuel Smith. Drawing multiple augmen- tation samples per image during training efficiently decreases test error. ar preprint arXiv:2105. 13343, 2021.
Casper Benjamin Freksen. An introduction to Johnson-Linndenstraus transforms. arXiv:2103.0055 2021.
or preprinl
Yunlong Gao, Shuxin Zhong, Kangli Hu, and Jinyan Pan. Robust locality preserving projections using angle-basod adaptive weight method. IET Computer Vision, 14(8):605-613, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Xiaofei He and Partha Niyogi. Locality preserving projections. Advances in neural information processing systems, 16, 2003.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out- of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial ex- amples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp- 15262 15271, 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 0rX2511508.0251,, 2015.
Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advonces in neural information processing systems, 15, 2002.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szalraniec, Michaäl Ramamonjisoa, Maxime Oquab, et al. DINOv2 Meets Text: A Unified Framework for Image-and Pixel-Level Vision-Language Alignment. arXiv preprint arXiv:2412.16334, 2024.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3D object representations for fine-gres theegrained catego- rization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554 561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. URL https://wwa cs. toronto. toronto.edu//kkiiz wewria//eenninn-eeaaaeee
pdf) Technical report.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can dis- tort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022.
Jae Lee, Doyoung Yoon, ByeongMoon Ji, Kyungyul Kim, and Sangheum Hwang. Rethinking evaluation protocols of visual representations Iearned via self-supervised learning. arXiv preprint arXiv:2304.03. 03456, 2023.
lyya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017a.
Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017b.
Zhichao Lu, Gautam Sreckumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, and Vishmu Naresh Boddeti. Neural architecture transfer. IEEE transactions pallern analysis and mochine intelligence, 23(9) 2989 2021.
Avner Magen. Dimensionality reductions in that preserve volumes and distance to affine spaces. Discrete Computational Geometry, 38:139 153, 2007.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1806.5151, 2013.
Roy Miles and Krystian Mikolajezyk. Understanding the role of the projector in knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4233 4241, 2024.
Roy Miles, Ismail Elezi, and Jiankang Deng. VkD: Improving knowledge distillation using orthogonal pro- jections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15720-15730 2024.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 427 436, 2015.
Maria-Elewe Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. Tn 2008 Sirth Indion conference on compuler vision, graphies & image processing, pp. 722-729. TEFF, 2008.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pallern recognition, pp. 3498 3505. TEEE, 2012.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt,, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International conference on machine learning, pp. 5389-5440. PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211-253) 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. MobileNetV2: Inverted rosiduals and linear bottlenecks. In Proccedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 4520, 2018.
Lawrence Saul and Sam T Rowcis. An introduction to locally lincar embodding. unpublished. Available at: htlp://www. cs. toronto. edu/" roweis/lle momeis//le/uubliccc pullicibiiassssns hlml, 2000.
Xiaoxiao Sun, Xingjian Leng, Zijian Wang, Yang Yang, Zi Huang, and Liang Zheng. CIEAR-11-wwarcouss:: Broad and more realistic lestbeds in model generalization analysis. In The Twelfth International Confer- ence on Learning Representations, 2024.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neigh- bors. In International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347 10357. PMLR, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pottern recognition, pp. 8769 3769-8778, 2018.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022. URL https //openreview
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD 2irds-200-2011 dataset. Jul 2011. URL https://www http:://wwwvviiion.ommm vision. caltech. edu /tt///aatttttuuee0aaaaaaa 2011/
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. ViM: Out-of-distribate with virtual-logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4921-4930, June 2022.
Longhui Wei, An Xiao, Lingxi Xie, Xiaopeng Zhang, Xin Chen, and Qi Tian. Cireumventing outliers of AutoAngment with knowledge distillation. In European Conference on Computer Vision, pp. 608-625. Springer, 2020.
Ross Wightman. Pytorch image models. http:////tttttu........aaaaaaaap
2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733 3742, 2018.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Lı, Yiyou Sun, et al. OpenOOD: Benchmarking generalized out-of-distribution de- tection. Advances in Newral Information Processing Systems, 35:32598 32611, 2022a.
Zhendong Yang, The Ti, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In Europeon Conference Computer Vision pp. 53 69. Springer, 2022b.
Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. ViTKD: Feature- based knowledge distillation for Vision Transformers In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 1379 1388, 2024.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In Proceedings of the LEEE/CVF international conference on computer vision, pp. 579-588, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 12104 12113, 2022.
Shaoting Zhang and Dimitris Melaxas. On the challenges and perspectives of foundation models for medical image analysis. Medical image analysis, 91:102996, 2024.
Yitian Zhang, Xu Ma, Yue Bai, Huan Wang, and Yun Fu. Accessing vision foundation models via ImageNet- 1K. In The Thirteenth International Conference on Learning Representations, 2025.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Supplementary Material
A The Johnson-Lindenstrauss Lemma
The Johnson-Lindenstrrauss (JL) Lemma FFreksen 2021
states that for set of points X in high dimen- sional space, there exists function that can map these points into lower dimensional space, within error e, where this error depends on the dimension of the target space m and the size of the set of points [X]. In its standard form, the JL. lemma states that Euclidean distances are preserved.
Lemma 2 Boonsson-Lindenstransss Freksen
For every d E N1, CE (0, 1) and X C RR, there exists a function f Rdd + R. where m = O log X such that for every 2, y E X,
2021 ).
[1](()) - r())l2 || y222
<||0r
v12
(17)
Morever, the map f can be constructed using simple approach. Given a matrix M gmndd with every element drawn from standard normal distribution N(0, 1), then
(()
Vm Ma -M
(18)
is a linear map that satisfies Lemma 2 with a probability given by the norm preservation lemma. This function also referred to as a JL transform.
Lemma 3 (Norm preservation; (Freksen 2021)). Let e E (0,1). If is constructed as above with m O(c (lg-2log--1) and x E TRd is unit vector, then
P ( c) 21 5
(19)
Again, for target spaces with larger dimension m this Lemma 3 states that it is more likely that a high quality map will be sampled. A similar result also holds for angles Magen 2007), which gives
Lemma 4 (Angles; (Magen 2007)). Let e < and let n, t be integers for which > 60c 20g-2logn, logn. Then for any n-poinl subset X of the Euclideam space TR there is a linear contracting embedding s(X) - R', under which angles are preserved within (double-sided) factor of 1 1f188/
The proof of Lemma 4 also relies on f being generated as random projection as above [Magen 2007) The JL transform matrix M has further interesting property, as observed in this work, that we refer to approximate left and right orthogonality.
Lemma 5 (Approximate left and right orthogonality for JL transforms). Given matriz M pmxd with elements drown from (0,, 1), there is
MM
OI
m
E M M
d
MM
d+ d'm
E M
M
d
Proof. Using the proportics of the Wishart distribution, which given the construction of M, we have
MM W...m.d
(20)
M T M Wa(Ta,m),
(21)
from which we have the following results for the mean and variance
J|MM dI,, Var(MMM3 d
(22)
F M mLa, Var (M m.
(23)
These expectations imply Lemma 5
19

---

Published in Transactions on Machine Learning Research (11/2025)
Morever, this property of approximate left and right orthogonality does not just apply to JL transformations, but any linear map M E IRmad with d >> m such that M is approximately left-orthogonal with M M R la- M cannot be exactly left-orthogonal, as the rank of M and M are both at most m, while Ig is of rank d which is greater than m.
For any matrix M E R with | M Ialle. The converse inequality m < with rank m, we have the inequality |MM m I S
Lemma 1. (Approzimate left-orthoggnaality implies right-orthogonn
does not hold.
Proof. We can write M H La E, which provides that
| M I = E
(24)
where E is an error matrix. Let us sample m columns of M produce square matrix K E Rmxm This provides for K, that we have K-K Im Ex where Ex the same columns sampled from the matrix E. This means that || S F
We then introduce the singular value decomposition of K = UEVT, where U and V are orthonormal matrices and Σ is a square diagonal matrix containing the singular values of K, to give
K VE ΣV- I, Ex
(25)
Σ-= V [m + EK)
(26)
- + Ex
(27)
which we can use, as ET ΣΣ for a square diagonal matrix, to find that
KK UEE U
(28)
= U(m + V EK VRVUJUT
(29)
= In + UV U
(30)
Let us consider a sample of sets of columns, such that each of the d columns selected m times
produce set of K2 matrices where 1 E {1, d) without repeated columns. Then we observe that,
d
MM J
M
(31)
MRj
(32)
2
Ki (Kik
(33)
km
have sampled our set of K2 matrices from the columns of M such that each M element in the sum of their products occurs m times. This gives
MM ΣΣ K.
(34)
m
and introducing Eq. 30 from above obtains
MM
In + UV EK, U VUT)
(35)
m
d
Im
UV Ex VUT
(36)
711
1=1
20

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Algorithin for feature distillation with CosPress.
Input: Training set X, teacher model T, student model Sp, teacher head has NØ number of epochs, Aug(.) augmentation strategy
Randomly initialise student model So and teacher head hø, or initialise using previous weights if fine- tuning;
while i < Ng do
Randomly split X into B mini-batches;
for Fa (Xi,,.., Xn.....n)) do
Generate augmented views: X = Aug(zs);
Compute dimensionality reduction objective (Eq. 12
Edim-red (X; o) =
+ [ Σ. LKL (h T(x:))
Compute student objective, while freezing Φ (Eq. 151):
Estudent (X; 0) = Leosine (Si(X), hetroomn (X)) + Lcosine (Sa(X) Morooonn (T())))
Combine losses: L. Latuudnt (X;0) + Ldim-red (X; Φ);
Minimise loss C by updating parameters of and Φ;
end for
i+1; end while
which provides that
d
1 m
|MM
Im
UV EK, VU
(37)
m
E
(38)
m
which proves the first statement. It is casy to the converse incquality is not true, simply by constructing
matrix M E Rdxm with the first m rows equal to the identily. This matrix satisfies that MM T = In, but
M M will have - m rows with only zeros in them. This proves the second statement.
21

---

Published in Transactions on Machine Learning Research (11/2025)
B Further Results
Table S14 ImageNet classification
larger teachers. Comparison of performance on ImageNet-1
under kNN and linear probing evaluation approaches.
Method
Arch
Teacher
kNN
Linear
Backbone
Student head
Teacher head
Teacher
Proteus
VIT-Ti/14 DINOv2 VIT-S/14
73.1
73.5
79.0
76.1
CnsPress
VIT-41/14
DINOv2 VFT-8/14
74.3
78.8
79.0
76.8
Proteus VIT-Ti/14
DINOv2 VIT-B/14
73.4
73.8
82.1
76.9
CusPress
WT-Ti/14
DINOv2 VIT VTT-B/14 B/14
75.6
81.9
82.1
77.9
Larger teachers result in better accuracy but have poorer quality dense features. We observe that training with larger teachers in comparison to the student requires longer training runs. To achieve best results efficiently, the pretrained models from a smaller teacher are used as a starting point. Then, the student and teacher heads are first trained with a frozen pretrained student model (allowing the CosPress teacher heads to minimize the student loss) for 30 epochs. Finally, the models are trained for 300 epochs using the same distillation approach as previously. Table shows that this improves the performance of the student models, with CosPress seeing larger improvements in accuracy ImageNet-1K in comparison to Proteus.
However, this results in poorer results in dense image lasks, like semantic segmentation. Table $15 shows that the students trained with a larger teacher network have poorer mloU for a linear probe on the Pascal VOC 2012 dataset. Undertaking longer distillation runs, or continuing training using a larger image resolution potentially might be helpful these cases.
Table S15: Semantic segmentation - larger teachers. Comparison of performance on the Pascal VOC 2012 semantic segmentation task using a linear probe.
Method
Arch
Teacher
aloU
Proteus
VFT-15/14
DINO22 ViT-8/14
70.5
CoePress
VIT-Ti/14
DINOv2 VIT-S/14
71.1
Proteus
VIT-1i/14
DINO.2 VIT-B/14
68.1
CoePress
VIT-T1/14
DINOv2 WT-B/11
69.7
Further out-of-distribution detection results. The OOD detection results for all of the OpenOOD
datasets (Yang al. (222aa) are presented in Table S16 Table S17 and Table S18
The tables report the threshold for including
ROC-AUČ for detecting OOD images and the False Positive Rate (FPR) using
95%
all in-distribution images. To produce these results, the KNN+ (Sun et al. 2022)
OOD metric is used to mcasure the performance of the model backbones. For ImageNet- we sample 1% of the dataset (12,812 images) and set. k = 10 to measure distance to OOD samples. For CIFAR-10/100, 1000, we sample 100% of the dataset (50,000 images) and set ==
Table S16: Out-of-distributtion detection. Comparison of performance on the OpenOOD benchmark for the ImageNet- 1K dataset. The t means larger values are better and the | means smaller values are better.
Method
Arch
Teatber
Near DOD Datascts
Far OOD Datasets
SSD hard
NINCO
Average
INaturalist
Openlmage
Textures
Averoge
AUROCt
FPR
AUROC!
FPR.
AUROCT
FPR4
AUROC-
FFR.
AUBOC
FPR
AUROC1
FPR
AUROC;
FPR.
Proteus
VIT-15114
DINO VIT-8/14
55.39
92:24
72.76
70.22
64.17
85.73
60.08
91.47
73.11
75.34
89.46
37.1
74.22
67.97
CuePress
VIT-T1114
DINO VInS/M 14
63.59
84.98
77.58
69.59
70-19
77.29
95.81
22.57
90.72
40.02
80.57
05.15
91.03
$7.21
DINO VI--5/1
05.76
81.8
79:39
00.45
72.58
76.12
98.74
4.76
90.23
35:41
8704
48.15
90.07
20.55
Protens
VIT-8/14
DINOWI VIT-B/14
53.78
97.4
00.61
91.71
61.18
94.30
30.80
98.5
61.7
91.11
86.10
09-73
61.02
96.75
CiePross
VIT-8/14
DINO VIT-R111
65.75
62.98
61.34
64.71
73.5
73.84
97.51
12.27
92.77
32.05
88.72
41.71
92.93
28.98
22

---

Published in Transactions on Machine Learning Research (11/2025)
Table S17: Specialist models near OOD detection. Comparison of performance the OpenOOD
benchmark Yang et al. 2022a. The t mcans larger values are better and the + mcans smaller values are belter.
IDD
Mothod
Arch
Teacher
Pretraining dataset
Near OOD Datasets
Average
CIFAR-10
CIPAR-1OU
Thy ImageNet
CIPAR-100
AUROCH
FPRJ
AUROC1 FPR.
AUROCT
FPR
AUROC1
EPR.
Proze
DeiT
VIT-T//16
RegNeeY-18GFF
IneageNet
63.53
94.96
02.75
85.44
00.2
CosPress
VTTT//14
DINOv2 VIT 9/14
LenageNes
60.30
71.45
90.07
34.94
85.23
53.2
CosPaess
VIT-TV114
DINOv2 VIT-S/14
ImageNet
dataset
DefT
RegNetY-18CIF
IouggeNet
83.91
CosPress
VIT-T/14
DINOv2 VIT-8/14
Nee
65.45
56, 76.76
00.34
47.36
87.3
52.06
CosPreess
VTT TY/14
DINOv2 VIT S/14
ImageeNes
Targes dataset
86.7
$6.45
92.35
30.37
$9.52
47.91
CIFAR-10
8/14
Proze
DelT
VTT T2/16
RegNerY 19GF
LmageeNes
50.58
94.08
63.15
84.75
57.01
89.42
CosPress
VIT-T/114
DINOv2 VIT-S/14
LmngeNet
90.22
41.5
966,07
13.63
93.44
27.57
CosPress
DINO22 VIT-S/14
Imn
Tarpet
95.76
DoiT finetuned
DeTT
VIN-T5/16
RopNesY-11G3F
Int
86.5
16.34
07.08
12.52
96.70
CosPreess
VIT T2/14
DINOv2 VIT 9/14
ImnageNes
96.15
16.57
97.24
10.62
95.00
13.6
CosPress
VIT-T//14
DINOv2 VIT-S/14
ImageNes
Target dataset
15.94
VIT-S/14
94.08
20:22
97.50
10.28
Table S18: Specialist models far OOD detection. Comparison performance on the OpenOOD benchmark Yang al. 2022a). The t means larger values are better and the + means smaller values are better.
IDD
Mathod
Arch
Touchur
Protraining datinnt
Fur OOD Datmuls
Average
DTD
MNIT
SVILX
Placos363
C-P0R-100
AURDC!
FPR]
AUROCY
TPR.
AUROCE
FPR]
AUROCT
PPR]
AUROCT
FPRE
Frezen
De'T
VIT-1//16
RegNe:Y-J6GF
ImageNet
14.97
100.0
71.86
75.77
43.55
98.15
34.07
$3.06
46.10
87.6
CoePress
WTT-11114
DENOVE VIT-S/14
Iet
33.00
14..5)
97.21
11.00
75.94)
67.21
96.22
7.18
76.37
51.15
CoePress
VTT: 11/14
DENOv2 VIT-S:14
ImageNet
Target dotaset
44.05
58.26
96.75
15.04
96.17
67.34
07.39
12.91
M1.55
40.41
DH'T Snetone
De'T
VT--1/16
RoyNeeYY11GPP
Net
71.07
67.50
65.15
07.00
05.51
21.57
91.87
(3.50
56.75
53-15
CosPiess
VIT T1/14
DENOv2 VIT-S:14
ImageNNtt
83.41
03:51
$7.49
56.64
36.5
21.11
92.05
37.52
50.57
47.02
CoePros
VIT-1/114
DENO<2 VIT-S/14
InsgeeNt Turgs distuert
70.51
67.44
90.24
$5.05
06.12
22.54
92.27
$7.01
50.58
45.95
CIPAR-10
DENOv2 VIT-S:14
42.05
59.8
96.25
15.08
77.75
88.13
07.89
5.45
78.08
50.02
Freacn
DeT
VTT I1/16
BegNe:
ImageeNt
17.01
100.0
71.26
73.15
42.00
92.25
57.74
50.08
47.04
86.49
CoePress
VIT T1/14
DENOv2 VIT-S/14
ImageNett
05.24
31,77
99.17
2.54
89.13
60.24
09.07
0.15
56.57
23.69
CeePeos
VTT T1/14
DENO<2 VIT-S/14
InsggeNet Targst distzeet
98.2
7.44
98.92
6.38
05.08
38.70
90.85
0.35
56.02
L1.52
ThiT mnstmee
DeTT
VTT T1/16
Nnt
97.88
9:53
97.55
9.69
09.67
0.71
96.7
0.95
58.69
5.17
CoePress
VIT 11/14
DINOvz VIT-S/14
ImageNet
99.35
1.51
CoePreess
VTT- 11/14
DINOvZ VIT-S/14
ImageNet
Dargee datawet
99-37
DINO<2 VIT-S/14
97.0
19.08
98.95
6.25
30.6
$6.45
90.96
0.18
06.62
10.74
23

---

Published in Transactions on Machine Learning Research (11/2025)
C
Ablation Studies
In this section we modify number of hyperparameters and component choices for CosPress to investigate how these impact performance. In the tables below the bold parameter sets are the default ones used throughout the rest of the paper.
Table S19: Ablation study: weighting. Comparison of KNN performance on ImageNet-IK IK for CosPress models trained with different weightings y for the dimensionality reduction component. The first row uses the frozen gradient approach described in the paper.
Method
Arch
Teacher
kNN
CoePress
VTT-15114
DINOv2 VIT-8/14
74.3
CosPress
VIT-T1/14
DINO,2 VIT-S/14
10
74.3
CosPress
VII-T//11
DINOV2 VIT-S/11
100
74.2
CosPress dimensionality reduction loss weighting. We consider the performance of weighting the dimensionality reduction loss in Eq. G rather than freezing the gradients of Φ in the student loss. This gives alternative loss function
CossPeess X; 0,000) 7 Ledd (x:) Φ) + Lstudent (X; 6, Φ)
(39)
where 7 is weighting factor prioritises the dimensionality reduction loss when is set to be greater than one. Table S19 shows that the approach of weighting or freezing gradients leads to similar results.
Table S20: Ablation study: metric. Comparison of kNN performance on ImageNet-1K for CosPress models trained with different metries for the student loss.
Method
Arch
Teacher
Loss Metric
kNN
CosPress
VIT-Ti/14
DINOv2 VIT-S/14
Cosine distance
74.3
CosPress
VIT-T//14
DINOv2 VIT-S/14
MSR
74.2
CosPress student loss metric. Table 520 for the student loss Letudent for Eq. (15). It is found that using a cosine distance loss leads to slightly better performance in comparison to a mean squared error loss.
considers the impact performance of using different metrics
Table $21: Ablation study: temperature. Comparison of kNN performance on ImageNet-IK for Cos- Press models trained with different sets of temperatures + for the dimensionality reduction loss.
Method
Arch
Teacher
KNN
CuePress
VIT-Ti/14
DINOv2 VIT-S/14
0.01, 0.010,.02...03,000 0.05, 0.06, 0.07, 0.08,
74.3
CoePress
VIT-T//14
DINOv2 VIT-S/11
[0.01]
71.1
CoePress
VFT:15/14
DINOv2 VIT-S/14
[0.10]
74.5
CuePeess
VIT-T//14
DINO,2 VIT-S/14
(0.01, 0.10
74.3
CosPress dimensionality reduction temperature parameters. Table S20 shows the performance
impact of different sets of temperatures in the dimensionality reduction loss for Eq. (11). is found that these values have small impact performance, with the best set being [0.10], which obtains slightly better performance that the set of parameters chosen for the paper.
24


Reviews:
[{"review_id": "nf6V22vLdK", "reviewer": "Reviewer_g1HL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary\n\nThe paper presents Cospress,an adaptation of distilling foundation models to smaller student models that considers the role of angle preservation explicitly. While previous work (the paper compares mainly with Proteus [https://arxiv.org/abs/2407.10366](https://arxiv.org/abs/2407.10366) ) uses a final up-projection to match the teacher model in it’s latent space (following Fitnets [https://arxiv.org/abs/1412.6550](https://arxiv.org/abs/1412.6550) ) approach, Cospress instead projects the teacher down to the students latent space, while trying maintain cosine similarities between different images (via a SNE inspired regularization loss comprised of the $D_{KL}$ between the distributions of von-mises fisher kernel distances between pairs, at different temperatures). The angle perserving dimensionality reduction is trained 1) the pairs of images within a batch 2) the features (patch and class tokens) within a single image. A small analysis is done to justify this approach, drawing links to JL random projection theory and the recent studies of feature superposition is drawn, since the dimensionality reduction can only be achieved using *approximately* orthogonal projection matrices, while fully orthognal (left and right) would be required to not distort the outputs of the student model AND preserve cosine similarity w.rt. to the teacher.\n\nThe student is then trained to minimize cosine distance to the approximately angle preserving downprojection, on both feature and class tokens.\n\nThe method is evaluated following the approach of Proteus (which it compares to, together with Proteus V_k) , with ViT-Tiny models distilled from DINOv2 ViT-S/14 and DINOv2 ViT-B/14 teachers on ImageNet-1k, evaluated using kNN and linear probing. Ablations are performed using only Backbone, student head , a teacher head, or directly probing the teacher.\n\nAn ablation is also performed enforcing strict right-orthogonality instead of regularized approximate orthogonality, yieldig worse performance, while it is confirmed that the regularization doe syield more orthogonal matrices than unconstrained proteus.  \nFurther fine grained clasisfication linear probin is performed on various datasets (only against proteus), consistently showing small gains. A semantic segmentation linear probing on Pascal VOC 2012 is also eprfomred, shoing improvement again.\n\nThe method also shows strong results on robustness, generalization and OOD tasks, compared with the vanilla proteus, while improving training efficiency.\n\nFinally, the method is evaluated again (against pretrained De-IT) after being fine-tuned (2nd round f distillation after distilling with ImageNet 1k, if I understood correctly) on specialist atks (C10,C100,Food, pets), again showing strong results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The method is explained decently well, although additional details/pseudocode would be appreciated especially for the cospress finetuning setup.\nI have some nits on the evaluation, but it is clear that it is effective, and while I think some related work might be missing, this is overall a good paper.", "audience_yn": "Yes", "audience_explanation": "As models become bigger, highlighting efficiency gains that can be unlocked at all stages as in this paper is always interesting, and I think the community should appreciate the nudge towards moving beyond L2 loss towards other geometric properties .", "requested_changes": "- multiple seeds should be run and  CIs should be reported, with suitable statistical signficicance tests performed https://arxiv.org/abs/1811.12808\n- please add an algorithmic listing or pseudocode to the appendix to clarify both distillation and finetuning for readers\n- I think there is some related work missing around the ideas of isomap, minimum distortion embeddings, locally linear embeddings etc. a quick search yielded the following\n\n[1] X. He and P. Niyogi, “Locality Preserving Projections”.\n[2] Y. Gao, S. Zhong, K. Hu, and J. Pan, “Robust locality preserving projections using angle-based adaptive weight method,” IET Computer Vision, vol. 14, no. 8, pp. 605–613, Dec. 2020, doi: 10.1049/iet-cvi.2019.0403.\n[3] L. K. Saul, T. Labs, P. Ave, F. Park, and S. T. Roweis, “An Introduction to Locally Linear Embedding”.\n[4] M. Kim and V. Pavlovic, “Covariance Operator Based Dimensionality Reduction with Extension to Semi-Supervised Settings”.\n[5] J. Fischer and R. Ma, “Sailing in high-dimensional spaces: Low-dimensional embeddings through angle preservation,” Jun. 14, 2024, arXiv: arXiv:2406.09876. doi: 10.48550/arXiv.2406.09876.\n[6] F. Tessari, K. Yao, and N. Hogan, “Surpassing Cosine Similarity for Multidimensional Comparisons: Dimension Insensitive Euclidean Metric,” Mar. 10, 2025, arXiv: arXiv:2407.08623. doi: 10.48550/arXiv.2407.08623.\n[7] A. Agrawal, A. Ali, and S. Boyd, “Minimum-Distortion Embedding,” FNT in Machine Learning, vol. 14, no. 3, pp. 211–378, 2021, doi: 10.1561/2200000090.\n\n\nI could also _swear_ that I saw specifically an inner product/angle preserving dimensionality reduction method from 2013 during the beginning of my phd, but this was in 2017 and I have not been able to find the paper (it too was trying to match the gram matrices of a kernel, similar to the approach  taken in the paper).\n\nI think adding this context doesn't detract from the methods novelty but helps readers to put it into context", "broader_impact_concerns": "I think broader impact statement would be approriate (at least for the appendix) weighing of the benefits of more efficient inference vs. jevons paradox as we see more and more energy and emissions flowing towards ML model deployment, as well as the dual use nature of segmentation and classification for surveillance via more efficient edge devices (inherent to the research of this nature, but still)", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XX1p6x9nQo", "reviewer": "Reviewer_pBfZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes CosPress, a knowledge distillation method that aims to preserve the angular geometry of teacher embeddings when compressing large vision transformers into smaller students. The main point is that the authors introduce a teacher head that projects teacher features into the student’s latent space while explicitly preserving cosine similarities between embeddings. The student is then trained to align with these compressed teacher features using a cosine distance loss.\n\nStrengths:\n- This paper is well-motivated. The idea of enforcing cosine similarity is interesting and sound.\n- The authors proposed many useful losses and techniques: the teacher head and similarity distribution loss are simple but effective.\n- The authors performed extensive experiments including classification, robustness, segmentation, and OOD detection shows effectiveness over baselines.\n\nWeaknesses:\n- The scope of the paper is limited to ViTs and ImageNet; unclear generalization to other architectures or domains.\n- Some metrics has limited improvement over baselines, such as in Table 10, Table 7.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental results are convincing and systematically support the paper’s claims. The authors compare against strong baselines across multiple tasks, such as classification, segmentation, robustness benchmarks, and OOD detection.", "audience_yn": "Yes", "audience_explanation": "Knowledge distillation is a well-studied area in machine learning. This paper presents a new method for knowledge distillation.", "requested_changes": "- Broaden evaluation scope, such as non-ViT or non-self-supervised teacher (e.g., CNN or supervised transformer) to demonstrate generalization of the method.\n\n- Provide more training details and analysis of the cost of similarity distribution matching (pairwise computations).\n\n- Discuss the reason why some of the evaluation only has limited improvement.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0eNZZ9PtsN", "reviewer": "Reviewer_KnZq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a novel feature distillation method, CosPress, which improves the Proteus method. First, the authors notice that a projection from a student feature space to a teacher feature space in Proteus might not preserve token similarities well. They suggest an improvement that changes the direction of the projection from teacher to student and justify this choice by the Johnson-Lindenstrauss lemma. Second, the authors demonstrate that their modification enhances Proteus in various vision tasks (with a focus on ViT-Ti distilled from DINOv2 ViT-S on ImageNet). Additionally, the authors demonstrate that their method indeed provides the expected conceptual benefit in terms of orthogonality of the projection matrices. Finally, the authors show that CosPress is well-suited for a specialized setting that includes an additional distillation step.\n\n**Strengths**\n1. The motivation is clear.\n2. The proposed method is theoretically sound.\n3. The evaluation was performed for a variety of vision tasks.\n\n**Weaknesses**\n1. As I understand, the authors use only one training run to compare CosPress with Proteus. Thus, the statistical significance of the findings is unclear.\n2. Text lacks clarity in several places.\n3. The experiments explore a limited number of architectures for the student and the teacher.\n4. Similarly, the experiments primarily focus on distillation using the ImageNet dataset, without ablations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I believe that the bulk of the evidence is sufficient since the authors utilize a wide range of vision tasks in their evaluation. However, I still think that the scope could be improved with experiments on different architectures and datasets. Additionally, I believe that some improvements are rather marginal; therefore, repeating training with different seeds is necessary to evaluate statistical significance.", "audience_yn": "Yes", "audience_explanation": "The problem is well-motivated, and the method seems to improve the previous baseline.", "requested_changes": "1. In Section 3, please clarify which losses are calculated between examples in a batch and which are calculated between patches in an image, both for the CosPress and Proteus methods.\n2. Similarly, in Section 3, please define what an \"approximately orthogonal matrix\" means. Why is this notion related to orthogonality? What is different? Why is Lemma 1 important?\n3. In Section 4, please provide training details (augmentations, etc.) and **the number of training runs**.\n4. In Section 5, please clarify the details for an additional distillation step. Do the hyperparameters of the initial distillation change? Are the hyperparameters of the additional distillation step different?", "broader_impact_concerns": "I don't see major broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)