Feature Representation Transferring to Lightweight Models via Perception Coherence
Authors: Hai-Vy Nguyen
Fabrice Gamboa
Sixin Zhang
Reda CHHAIBI
Serge Gratton
Thierry Giaccone
Venue: Accepted by TMLR ()

Abstract: In this paper, we propose a method for transferring feature representation to lightweight student models from larger teacher models. We mathematically define a new notion called perception coherence. Based on this notion, we propose a loss function, which takes into account the dissimilarities between data points in feature space through their ranking. At a high level, by minimizing this loss function, the student model learns to mimic how the teacher model perceives inputs. More precisely, our method is motivated by the fact that the representational capacity of the student model is weaker than the teacher model. Hence, we aim to develop a conceptually new method allowing for a better relaxation. This means that, the student model does not need to preserve the absolute geometry of the teacher one, while preserving global coherence through dissimilarity ranking. Importantly, while rankings are defined only on finite sets, our notion of perception coherence extends them into a probabilistic form. This formulation depends on the input distribution and applies to general dissimilarity metrics. Our theoretical insights provide a probabilistic perspective on the process of feature representation transfer. Our experimental results show that our method outperforms or achieves on-par performance with strong baseline methods for representation transfer, particularly class-unaware ones.

Published in Transactions on Machine Learning Research (02/2026)
Feature Representation Transferring to Lightweight Models via Perception Coherence
Hai-Vy Nguyen
Ampere Softwore Technology Institut de mathématiques de Toulouse Institul de Recherche en Informatique de Toulouse
Fabrice Gamboa
fabrice, gwwbooommmaaaoooo wwwv-toulouse
Instifut de mathématiques de Toulouse.
Sixin Zhang
sizin. Strin.zhang@Prri...
Institut de Recherche en Informatique de Toulouse
Reda Chhaibi
rela.chhaabii@umiiiicttoeei fr
Laboraloire Jean Alezandre Dicudonné, Université Côte d'Azur Serge Gratton
sery..grulon@@oodlosss--...nn
Instifut de Recherche en Informatique de Toulouse
Thierry Giaccone
thiery.jdocconeeaappeeee.rrr
Ampere Software Technology
Reviewed on OpenReview: https: // openreview. net/ normm? id- id-9QbibeSEUs
Abstract
In this paper, we propose a method for transferring feature representation to lightweight student models from larger teacher modcls. We mathematically define a now notion called perception coherence. Based on this notion, we propose a loss function, which takes into account the dissimilarities between data points in feature space through their ranking, At high level, by minimizing this loss function, the student model learns to mimic how the teacher model perceives inputs. More precisely, our method is motivated by the fact that the representational capacity of the student model is weaker than the teacher model. Hence, we aim to develop a conceptually new method allowing for better relaxation. This means that, the student model does not need to preserve the absolute geometry of the leacher one, while preserving global coherence through dissimilarity ranking. Importantly, while rankings are defined only on finite sets, our notion of perception coherence extends them into probabilistic form. This formulation depends on the input distribution and applies to general dissimilarity metrics. Our theoretical insights provide probabilistic perspective on the process of feature representation transfer. Our experimental results show that our method outperforms or achieves on-par performance with strong baseline methods for representation transfer, particularly class-unaware ones.
1
Introduction
The need of lightweight models and Knowledge distilling (KD). Deep learning models are at the forefront of performance in different domains and tasks, such as classification (He et al. 2016 Tan Le 2019
2021 He et al., 2020) or object detection (Ren et al.) 2015) Lin et al., 2017a by He et al. 2020). However, the performance of neural networks increases,
does the cost of increasingly large models. In many applications where resources are limited (c.g. mobile devicos) or one needs a fast exccution, it is preferable to use lightweight models. Among the many approaches to obtain lightweight. models, KD (Buciluă et al. KD
2006 Hinton et al.] 2015 Wang & Yoon, 2021) stands out as a particularly interesting direction. setup essentially consists of teacher model and student model. The student model (here the targetod lightweight model) is generally smaller than the teacher model, and therefore more efficient for the task in question. At high level, the KD process consists in teaching the student. model that it mimics the way

---

Published in Transactions on Machine Learning Research (02/2026)
Teacher feature space
Teacher
model
Transfer
Representation Constraints
dataset
Student feature spaca
Student model
Figure 1: Illustration of feature reprosentation transfer. non-labeled transfer dataset is passed through both teacher and student models to obtain their respective feature representations. The transfer process consists in training the student model to somehow capture the input-input. relationships encoded in the teacher model's feature space.
the Leacher model understands the input. This is usually achieved by using a so-called transfer dalasel. The student model is trained to behave like the teacher model on this set in some way Many previous works
have shown that KD improves the genoralization capacity of the student model (Hinton et al., 2015: Chen
et al. 2017: Zhu et al., 2018 Beyer et al. 2022 Huang et al. 2022). Different up-to-date KD approaches
will be discussed in Section 2
Feature representation transferring. Classical KD techniques generally focus on matching the outputs of student and teacher models, typically for classification tasks (Hinton et al. 2015). However, this approach has limitations, such as requiring the same mumber of classes between the teacher and student models. To
address these issues, previous works (Passalis & Tefas 2018a Passalis et al. 2020b Tian et al. 2020 Park let al., 2019) propose to directly transfer the feature space representations through learning. This enables the student model to better capture the input-input relations represented in the feature space of the teacher model, by preserving its geometry and distribution (illustrated in Fig. [ This leads to more effective knowledge transfer. Furthermore, note that this approach is unsupervised transferring method, it only uses transfer set without labels. As consequence, the learned features can be used for various downstream tasks. Because of these appealing properties, our method aligns with this last approach.
Motivation and an overview of our method. Generally, the student model much smaller than the teacher model. Thus, in terms of representational power, the student model cannot. learn to produce features with the exact same geometry as the teacher model. Hence, we aim develop method such that (1) the student model does not need to copy the feature representation of the teacher model; (2) the learned representat ion retains certain overall consistency with the teacher model's 's feature represental ion and (3) the method works for representation transferring between spaces having different dimensions. For these purposes, we develop a novel fairly simple notion called perception coherence, that is explained subsequently. To begin with, let us consider a reference input x and a set of n point {x,}² (called compared set). These points are fod into the student and teacher models and are mapped into the feature space of each model. The feature space reflects how the models perceive the inputs. Placing ourselves in the two feature spaces, let di and de be dissimilarity measures for the teacher and student models, respectively. Then, if d (2.,2;) xx) S di 2;) for some j.j E 21, n) (i # j), we expect that dz(x, x.) I d2 æ5) also holds. Less formally, if the teacher model perceives x to be more similar to than to Zj, we expect the student to have the same perception. We refer to this as the property of perceplion coherence. By adopting this behavior for different reference points and compared sets, the student model learns to have a coherent perception with respect to the teacher model. Morcover, since such rankings are only defined over finite scts, we further generalize this concept into a probabilistic form-oo our formal definition of perception coherence (Definition 3.2 which depends on the input distribution and applies to general dissimilarity metrics.
2

---

Published in Transactions on Machine Learning Research (02/2026)
f,(0)
dr(x,xg)
dj(xxx
Teacher feature extractor f
Training f22 such that for m, E (1, if d,(xx,m S sd then, dy(x,x,,)= S dy(xx). This is achieved based on differentiable soft ranking function.
Reference point x Compared set (2
(x,),, 11()))
fx(x)
d2(xx1)
dxlxxal
Student feature extractor f,
(())), (x(xg)
12(
Figure 2: General scheme of our method. In each mini-batch, use each input the reference point to compute the dissimilarity to the others (compared set). Then, using loss function in Eq. 4 the student model is trained to respect the perception coherence. Notice that fi and f2 can be the whole model (when using the output layer) or a part of the model (when using an intermediate layer).
The general idea of our method is illustrated in Figure 2 (further details will be illustrated in Figure 3 Note that we do not quantify how much more similar is to r than Ty. Instead, we only preserve the relative ranking of dissimilarities. This relaxation provides flexibility, as the student model is not required to replicate the exact geometry or distribution the teacher model's feature space. However, the ranking operation is non-differentiable. To address this, we introduce a simple differentiable approximation based on soft ranking function (Eq. 3 Our contribution can be summarized follows:
We propose new method for distilling knowledge to lightweight models, based on deep representation transferring. Central to our approach is a novel probabilistic concept, perception coherence, which natu- rally leads to a simple and musy-to-implmmet loss function (Eq. 44] based on a soft ranking operation.
We provide tcalretical insights through a probabilistic framework that connects representation transfer to local and global dissimilarity rankings in the feature spaces of teacher and student models. In particular, we show in Section 4.1 that the perception coherence level can be approximated via ranking operations mini-batches with convergence rate O(1/BB), where B is the mini-batch size. Purthermore, we shall also demonstrate that larger global expected coherence level leads to higher chance that the learned representation respects the dissimilarity ranking of the teacher one (Sections 4.2 and 4.3
We conduct experiments on retrieval and classification tasks to compare our method with strong bascline methods. The results show that our method outperforms or achieves competitive performance with strong baseline methods on lightweight representation transferring settings. Our code is available at pprceptionn coherence-code
2 Related Work
KD based on soft label produced by teacher model. This approach has been studied since the early days of neural networks (Buciluă et al.; 2006). Therein, the authors show that knowledge can be transferred from a large ensemble of models to single small model. A standard KD method consists using output of classification models (Hinton et al.] 2015 Tang al., 2016 Huang et al., 2022), with temperature, leading to smoother soft labels. Based the same spirit, (Chan et al. 2015)) proposes to match the output distributions using Kullback-Leibler divergence. Soft label was also used in the context of domain adaption Tzeng et al.) 2015). While all these methods show the benefits of soft labels, the main limit of this approach is the need of having the same number of classes bet models.
KD based on distance matching. In Romero al. (2014), the authors propose to minimize the distance between student and teacher's features to guide the student model during training. However, in the case of dimension mismatch between the models, one need pass through a linear transformation, which leads to
3

---

Published in Transactions on Machine Learning Research (02/2026)
substantial information loss. In Yim et al. 2017 the authors propose to minimize distance of the so-call FSP matrir between teacher and student models (capturing inter-layer relations). More advanced methods consist in minimizing the distance between the derivatives w.r.t. to the features (Czarnecki et al. 2017) or between the attention map of student and teacher (Zagoruyko & Komodakis 2016). All these approaches require the dimension match, and need some form of interpolation in the case of dimension mismatch. Overcoming the drawback of dimension mismatch, Passalis Tefas (2018b) proposes to match the the pair-wise similarity matrix of the training points (in the transfer set) between the student and teacher models. The method proposed in Yu et al. (2019) is to match the distance between pairs of samples. However, this brute force approach is not likely to work as the student model is much smaller than the teacher one (as discussed in Passalis et al. 2020bb).
Feature representation transfer. Some previous works treat the representation transfer as the problem of distribution matching, based on Maximum Mean Discrepancy (Huang & Wang 2017) or adversarial train- ing (Belagiannis et al. 2018). The latter method is inspired by Generative Adversarial Networks (GANs) Goodfellow et. al. (2014). 4). One major drawback of distribution matching is the need of having the same dimension between the two feature spaces. Probabilistic frameworks based mutual information are pro-
2019 Tian ct al. (2020). However, the method of Ahn et al. (2019) requires Gaussian assumption (this is not necessarily satisfied in practice) while the method of Tian et al. (2020) requires the input features to have the same dimension (hence requiring a linear transformation, which may reduce the useful information). Overcoming the requirement of dimension match, Park et al. (2019) proposes to transfer the input relations using wistance-wise and angle-wise distillation loss. However, the structural information is quite limited, as it only considers a single triple of points at time. Overcoming this drawback, Passalis & Tefas (2088a) proposes a method of probabilistic knowledge transfer (PKT). The high level idea of this last method is to transfer the local geometry learned by the teacher model based on kernel techniques. Based on this work, Passalis et al. (2020a b) basically share the same idea, but they apply the technique for multiples intermediate layers with an auxiliary model. The benefits of using auxiliary models are discussed in Mirzadeh : et al. 2020). As in Passalis & Tefas (2018a); Passalis et al. (2020a bb, our method takes into account the relation between the inputs and allows representation transfer between space having different dimensions. However, in Passalis & Tefas (2018ab; Passalis al. (2020a b), a common kernel is used for all points (e.g. Gaussian one with fixed bandwidth), which cannot adapt to local density variation. This contrasts with our method, where we only rank the dissimilarities but. do not use the dissimilarity magnilude. Thus, by using the dissimilarity rankings (or equivalently cumulative function of dissimilarities). our method captures local probabilistic closeness a property that raw distance magnitudes fail to reflect in the presence of density variations.
posed in Ahn et al.
Class-awareness. Knowledge distillation is a broad research domain, known for leveraging the "knowledge" of teacher model to improve a student model. However, most slate-of-the-art methods adopt class-aware approach. This means that these methods rely on class information in some form- -mst notably, the number of classes. The standard approach involves matching the output logits (Hinton et al.] 2015 Huang ct al. 2022). More recent class-aware methods reuse the teacher's classifier (Chen et al. 2022) and then retrain the student encoder. Other hots employ the leacher's classificat ion head to implicitly align the student's intermediate representations with those of the teacher (Wang et al.) 2024]. While such methods demonstrate strong performance on certain benehmarks, they are often tailored to specific tasks. In contrast, our method is conceptually designed for general-purpose representation transfer, applicable to any feature space equipped with pairwise dissimilarity measure. This class- formulation conceptually enables broader applicability, e.g., regression models even hand-crafted features.
3
Our method
3.1
Perception coherence and other notions
We consider a task where the input space is denoted by X' associated with the underlying distribution Dx- Let us consider two feature extractors fi X + Fi and fz : X + F2, endowed with symmetric dissimilarity metrics d,- FFiFi xFi R and d : F22xF2 + R, respectively. For ease of following, we can think of fi and f2
4

---

Published in Transactions on Machine Learning Research (02/2026)
teacher and student models, respectively, For brevity, for any x, E X, let (xx,z'))= df)()))))(((((((( and da(a, x') = d (22())
At a high level, if fi perceives x to be more similar to 2 than to 22, then it should be also the case for f2 after the representation transferring. This is represented through the feature space of each model, endowed with their corresponding dissimilarity metric. We present this intuition via a more formal concept named absolute perception coherence as follows.
Definition 3.1 (Absolute perception coherence) We say that f2 absolutely perception coherent with Ji x E X if di (z, æ2) 2 di (x, e x2) 2 (1), for any 21, 22222 E X.
However, the absolute perception coherence too strict. Thus, we shall introduce a more relaxed notion subsequently, based on probabilistic framework by using the distribution Dx. To begin with, given x, e X, let us define the cumulative functions as follows
(z == Px (d1 (2, X) di(xx'))), : X) S dz(x,
(1)
Remark 3.1 We have the following remarks on the relation between the previous function and the ordering of dissimälarities to given reference point
21( S 22) & di(2, S d2(1,22), for i {1,2}.
A small value of Fi(x,,))) x') means that x' is probabitistieally close to x. In other words, if we choose randomly another point T according to Dx, there is high chance that dx(x, 2 (z,,²)).
Similarly, large value of F'(x,x² means that x' is probobilistically far from æ.
Thus, for each model Jú, x')x,,") represents "probabilistic dislance" belween x and x'.
Remark 3.2 (Relation between cumulative function and ranking) Let consider discrete distri- bution uniformly distributed over N examples { and the extractor J endowed with dissimilarity metric d. We consider x, as the reference point and we compute dissimilarity between and other points zj 's to obtain ;
= {d(Ii, } Assuming there is no ties, i.e., d(I,, & d(æ,, I2) for all k # j, then, the rank of d(z,,2z) in the set can be computed as r(dij) = Σx (duSdy)) Moreover, the cumulative function F F(,,,1,) is the normalized rank of 2(;; in the set D. That is, F(z,,I;) = r(di))/N = x E. [(da
This remark allows us to intuitively understand the dissimilarity cumulative function through the lens of ranking. Hence, for a given reference point, we can know its "probabilistic distance" the other points by using dissimilarity rankings to compute the cumulative function. This is illustrated in Fig.
Using the cumulative function of Eq. we define the perception coherence level as follows.
Definition 3.2 (Perception coherence level) The perception coherence level of fa with respect to (w.r.t.) fi at X is defined
- X) - F2(x, X)I] (x) = Ex
(2)
Remark 3.3 The coherence level always valued in [0, 1]. Indeed, 0 s [Fi(x, F2(0, s 1 for all u, X. Hence, Ex (x) Fa(x, X)] 1.
Proposition 3.1 If fa is absolutely perception coherent with fr at x, then (1..f(()) = 1.
Proof 3.1 By definition, we have that Fi = Px (di I dı and F2(x, Xx (dz X') S dz(x, X)). As f2 absolutely perception coherent with fi x, we obtain that d X') )' d1 X) de(a,X)) S (z, Thus, Fx (d, X') S (x, X)) Px (dz X') s dz(z,))), which lends to the result.
5

---

Published in Transactions on Machine Learning Research (02/2026)
X4
X1
X3
X
X, reference point
X₂ as reference point
11
X4
d
XA
X1
X
d 13
(Similarly, and X3 as reference point)
d A2
X3
X
X
X2
Ranking: ,= sd 52 s3
Ranking: 0=pير 5d43 d2 i [E(1,,,3,4), F x)) 1 Σ sd
iE (1,2,3344) x)) Ei= (d, sd
Figure 3: Illustration of dissimilarity ranking (in feature space) in the case of 4 points. Each of all the data points considered as the reference point, and the dissimilarities to other data points are ranked to capture the cumulative distribution of dissimilarity (in feature space).
Proposition 3.1 tells that if we have an absolute perception coherence between the two models at a point x, the coherence level is equal to 1 æ. Now, natural question ask whether the reverse statement holds. In short, our theoretical analysis yields positive answer to this question. This will be presented in Section
The coherence level naturally gives rise to the following definitions for the local and global perception coherence.
Definition 3.3 (Local a-perception coherence) We say that f2 is a-perception coherent with fi at x E X if (z) 2 a.
Definition 3.4 (Global a-perception coherence) We say that is globally a-perception coherent with fi Ex (X) 2 That is, the global expected coherence level is at least
3.2 Practical implementation with differentiable ranking operation
simple Monte-Carlo sampling, processing by mini-batch, like any standard machine learning algorithm. Consider training batch B = {a:) For each f E [2] endowed with the metrics d E ddz, we first compute the pair-wise dissimilarity d(z;,I;) for all i,j (1,2,.... B}. Let dy denote d(r,, 2y) for brevity. Let us denote (B(()) (de) Let R(( be the ranking of D (B). That is, the jth component of R/ (B) is the rank of di in D(( Notice that
To implement our method,
R (B) is permutation of (1,2.... B) (so, R/ (B) E BR.. Based on Remark 3.2 intuitively, with sufficiently large E for each f {f1, f2), we have
R. (B) (R,,,))))) R/²(B) a (Pa(2,,2;))) B
To maximize the perception coherence level, we need somehow minimize the discrepancy between F1 and Fz. Thus, empirically, we can minimize
B
R RB((B)
R (B)
B
C(i, =
Σ
(B)
B
B
B
i=1
Thhrough our qualitative observation, for B from 10, results stabilize. Section 5.5 for results and discussions.

---

Published in Transactions on Machine Learning Research (02/2026)
We use here the squared Euclidean norm for its simplicity, but other metrics be used to compute the difference between R. (B) and (B). Besides, notice that each i E 21,2.... B} corresponds to being the reference point. Hence, averaging over in the batch allows for maximizing the global expected coherence level (the expectation is taken over different reference points, see Definition 3.4 for recalling). However, the ranking operation is not differentiable. Hence, we propose a simple soft ranking operation based on sigmoid function. Assuming that there no tics, i.c., dij # da, for all j * k that generally holds practice. In this case, we notice that. the rank of dy in (B can be computed as r(de)) = Ex=1 !l (duSd,)- Hence, we can soften the ranking operation as follows
B F(db)) = Σ
di d dik
A
where A is the sigmoid function and T < 1 is the temperature playing the role of scaling. The above function approaches the step function as T tends to 0. Hence, this allows us to conveniently approximate the ranking operation. Let (B) be the corresponding ranking oblained with this softened version. So, our loss function can be written as,
B
R! R.. RB)(
Cours(fi,
(4)
B
4 Theoretical aspects
In this section, we first investigate in Section 4.1 the convergence behavior of the estimator of perception coherence level using mini-batches. Then, we derive theoretical insights about the local perception coherence at given point in Section 1.2] Next, we also investigate about the global coherence in Section 4.3 Finally,
study the stability of the perception coherence around given local region in Section 4.4 Throughout our thcoretical analysis, make the following assumptions.
Assumption 4.1 (Measurability of Distribution Function) For 1,2, the dissimilarily function d X' X + TR measurable, and the marginal distribution D is such that the function F1(x,x') == PX-D (di(a, X) d')x,x²)) is jointly measurable in (x,x²) X X.
Assumption 4.2 Let X and X' be the iidd random variables following the law Dx- We assume that for X')] [2(x, - Fi (i, X)]) = O. We remark that this assumption is
any x E Fx,x' (1F2(z, X') reasonable in the contert of continuous variable modeling.
4.1 Expected Convergence of the Mini-Batch Estimator
Recall from Eq. 2 that the perception coherence level fz with respect to (w.r..)) fi at x E X' is defined
as
$5. == Ex-D
[ (2 F2,xx
Averaging over the data distribution yields the global perception coherence,
Ex Exlo).122 = - Ex.xx
(I(X). ( - -x)X
Maximizing the global perception coherence is equivalent to minimizing the following complementary quan- tity, which we call the Difference Coefficient:
DC
Ex Ex(on. Ex,xx, [x
[ X2) F2(X,, X2)]]
Intuitively, DC measures the discrepancy between the rankings induced by fa and fa over the data distribu- tion (i.e., discrepancy between F, and F2).

---

Published in Transactions on Machine Learning Research (02/2026)
Mini-batch approximation. In practice, computing DC exactly is infeasible, as it requires access to the full distribution (or the entire dataset). Instead, approximate it using i.i.d. samples drawn in mini-batches. The corresponding empirical estimator defined as follows.
Definition 4.1 (Empirical Estimator) Let B = (Xi...., Xn) be mini-batch of B sumples drown i.i.d. from D. For each E {1,2}, define the empirical distribution function
F,, = 1 E B k=1
(d, S d(xxx'))
Then the empirical difference coofficient is
B
B
DCB = B
Xx,((,,))) - (x,
This estimator depends only on the samples in the mini-batch and provides a tractable approximation to DC. Before analyzing its convergence properties, we first verify that the inner terms are unbiased estimators of the population functions.
Proposition 4.1 (Unbiasedness) For each fized i E {1,2} and any pair x') E X2, we have
E8 ( Fi(x,x)).
porticular, P.. is on unbiased eslimator of PŞ.
Proof 4.1 By linearity of expectation,
B Σ = =Pxwo(d(t, d(a,,')) -((,,")))((((
E [FB(,,,'')
dd(a,) =x) d(<,2')))
Convergence of the mini-batch estimator. We now establish that DCp converges to the true DC the standard Monte Carlo rate.
Theorem 4.1 (Expected Convergence Rate) Let DCB be the empirical estimator of DC based batch of size B. Then there erists constant C > 0, independent of B, such that
Eg DCB [DCCB-DC]] < VB
C
Consequently, DCB
DC in expectation the rate O(1//B) as B oc.
Proof 4.2 The detailed proof is provided Appendir
Discussion. Theorem 4.1 ensures that the estimation error decays proportionally to I/VB. In other words, enlarging the mini-batch size systematically reduces the variance the estimator, thereby improving the accuracy the approximation to DC. This theoretical guarantee strongly supported by our experiments, that will be shown in Section 5.5 (Fig. 7 and Table 5) More precisely, we shall show that the empirical estimation error exhibits the predicted scaling behavior with respect to B.
4.2 Local perception coherence
Theorems and 4.3 provide insight into how well the student model f2 mimics the teacher model fi, given the local perception coherence level.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Theorem 4.2 (Local rank preservation) Assume that f2 is a-perception coherent with f1 at & (t.e. Ph (a:) 2 0). Then, for any E > 0, we have
Ca (1 E2 E1
Px,x' (F2(z,X') - F2(x 02
SE)
(5)
where Cs is positive constant depending on E1.
The proof is in Appendix B.1 Consider the case where approaches then the right-hand side (RHS) of Eq. 5 approaches O. Theorem 5 shows that, if we pick up two random points and I2 such that the teacher model fi perceives them to have close dissimilarities w.r.t. a given point x (i.e., their dissimilarities differ by at most E1); then there high chance that it also the case for the student model f2. That is, f2 also peroeives that their dissimilaritics differ by at most 51. This is because if 1, the RHS equal 0 for any &2 > 81. While this theorem tells us only about the difference between the dissimilarities (absolute value), Theorem 4.3 also considers the ordering of the dissimilarities.
Theorem 4.3 (Local relative order preservation) Assume that f2 is a-perceplion coherent with S1 al x EEX (1) 2 a). Then, for any ε > O, we have
Fx,x/ (F)( F2 2 Fi(x,))) S Fi(x,))) s C(1 a)/E.
(6)
where C is a positive constant.
The proof of this theorem can be found in Appendix B.2 Notice that the RHS of Eq. 6 approaches 0 approaches 1 for any > 0. Hence, in this case, Theorem [.. shows that, if we pick up two random points 21 and 22 such that the teacher model fi perceives 21 to be more similar to than ø2 (i.e., F (z < (x,x)))) then probability that fa perceives 21 to be less similar to than 22 is upper bounded. In the case where = 1, this probability is O, that can be formally stated as follows.
Corollary 4.1 Assume that fo is 1-perception coherent with fi at x X (z) = Then, for ony >>0 0, have
Px,x' (F2(x, F2(x; X) 2 (1xxxX) (i X)) =0.
Note that this holds for any O. Thus, by letting -> 0, the above corollary tells that, the probability that fa perceives 21 to be less similar to than 22 is mull. That is, the student model fz learns to have the same perception as the teacher model fi-
4.3 Global perception coherence
In the above section, we analyzed the perception coherence at reference point x, knowing that the local coherence level at x is at least a. Now, natural question concerning the global coherence follows: What conclusions can be drawn from the global expected coherence level when the reference point x is not fized? Theorem formally answers this question.
Theorem 4.4 Assume that fa is globally a-perception coherent with S1 (i.e., Ex(enss(X)) a). Let X1, X2 and X3 be i..dd. random variables following the law Dx. Then,
1. Global rank preservation. For any 82 > 81 > O, we have
C1
TP ( F X3)1 2 62 [F X2) ,(( sa)
Eg E3
where Ce, a positive constant depending only on E
2. Global relative order preservation. For any & > 0, we have
(F((X, Fx 2E F1(1, x (x), C(1 C(1-a) a)
where C is positive constand.

---

Published in Transactions on Machine Learning Research (02/2026)
The proof of this theorem can be found in Appendix C.1 Theorem reference point, knowing the global expected coherence guarantees bound. This result. can be interpreted similarly to the local perception case in Theorems 4.2 and 4.3
4.4 shows that, even without fixing the
4.4 Stability of perception coherence around local region
We now study the fluctuation around local region A S X. We seek to answer the following question: If the local perception coherence level is at least in the region A, how stable is the coherence level under perturbations around A? In short, our theoretical result in Theorem will will show that the fluctuation of the coherence is bounded by the perturbation level around A. For this, make the following assumptions.
Assumption 4.3 We make the following assumptions.
The dissimilarities of both models are symmetric and satisfy the triangular inequality.
For 8 2 0, let p(0) m1.4 SUPZEX, Px (d, X) [d, d o]). We assume that p(5) O - 0.
For E {1,2], x, x E ex d((xx²') ((F((z,z)) as Fi(x,2') -
Let F(x',x) = max(Fi (4r, ,), =2)) Intuitively, a small value of F(a', x)) signifies that x and x' are probabilistically close to each other in the feature spaces of both models fi and f2. Then, the relation belween the fluctuation the perception coherence and the perturbation around A is stated as follows.
Theorem 4.5 Consider non-empty set A X (with Px (X E A) > 0) such that $ 2 a, Vr A. For 0 < 8 < 1, let As = {x E X min F(x,,z) S 5}. Then, for any 0 < <ES a, we have
0
Fx (o) aa-5 (xEA)) <
(7)
ECA
where CA is a positive constant depending on A.
The proof can be found in Appendix D Notice that As is the perturbed set around A at perturbation level (A S As). Theorem 4.5 states that if the coherence level in A is at least a, then the probability that the coherence level of point around drops below a - E is bounded above by O(8). This holds for any >>> 0, indicating that the coherence level does not decrease significantly increasing 8). Thus, the perturbation + the upper bound also tends to zero, characterized by O()). This ensures the stability of coherence level within each local region.
we expand the region As (by
5 Experiments
In this section, we first conduct experiments on 2D and 3D data for proof-of-concept (Section 5.1 Next, in Section 5.2 we present a stylized empirical study showing the correlation between the perception coherence level during transfer and the performance downstream classification task. Then, in Section 5.3 we conduct experiments on lightweight settings to evaluate the quality of the learned representation through retrieval (asks. Besides, we demonstrate how our method boost the performance of student models in classification task in Section 5.4 Note that in all experiments conducted in Sections 5.3 and 5.4 the student models are
smaller than the teacher models?] [ to meet practical requirements. Finally, we shall also conduct ablation
study in Sections 5.5 and 5.6 Without further mention, we use cosine dissimilarity as the dissimilarity metric on neural nerksrks (defined in Appendix For brevity, the details for all experiments can also be found in Appendix
28e Appendix F detailed computational cost comparison of different, pairs teacher-student models.
10

---

Published in Transactions on Machine Learning Research (02/2026)
5.1 Proof-of-concept with experiments on 2D and 3D toy examples
In this section, we conduct experiments on 2D and 3D data, without neural networks, to qualitatively demonstrate how our method works. For this, we use reference data configuration { (can be seen the teacher configuration, N is the number of data points). Then, we randomly initialize another configura- tion {aā |= (student configuration), where each point 2 is associated with a fixed point z/. More formally, for each (1,2... N), (i) x; (fixed) and f2(i) = x²- We then apply our loss function from Eq. (4) where the dissimilarity metric the Euclidean distance. All experiment details are in Appendix E. The evolution of the student configurations in different setups are shown in Fig. [and 5 We intentionall lly set the teacher and student configurations in different scales of magnitude to show the effectiveness of the method (teacher configuration has data points in scale ~ whereas student configuration is in scale ~ 10). We see that our method results in a very smooth configuration transfer, despite the scale difference. From Fig. E we remark that, while the exact geometry not completely preserved, the learned configuration preserves a global structural coherence. The same phenomenon is also observed the case where teacher and student configurations have different dimensions (3D + 2D) Fig. 5 Another experiment with more clusters is in A Appendix E.1 Thus, this qualitatively proves the effectiveness of our method.
Loured Gorogrtto
Figure 4: Transferring process on 2D datascts. Top row represents the teacher configuration (fixed), bottom row represents the evolution of the student configuration along the transfer process (from left to right). Each gray line associates of with z! (for the same i). We observe that the learned configuration preserves global structural coherence without preserving completely the geometry.
(a) 3D reference (b) Evolution along the transferring process of learned configuration configuration. in 2D.
Figure 5: Transferring process from 3D to 2D data.
5.2 A Stylized Empirical Study of Perception Coherence and Downstream Classification
To show the correlation between perception coherence and downstream task performance, we study its rela- tionship with classification performance in controlled setting. We consider a two-moon binary classification dataset with 800 samples, randomly split into 400 training and 400 test points. For teacher mode neural network with two hidden layers followed by softmax classifier is trained in fully supervised manner, using the labeled training set. After convergence, the teacher achieves perfect accuracy on both training and test sots.
Student model and unsupervised transfer. For simplicity, the student network shares the same archi- tecture as the teacher model. Using our proposed unsupervised transfer method, we distill the representation from the teacher's penultimate layer into the student, relying only on unlabeled training data. Importantly, the final softmax layer of the student is excluded from this transfer process. During training, save student checkpoints at ten different epochs, denoted by [S
Forr brevity, please refer Appendix E.2 model details, including training details.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Perception coherence. For each checkpoint Sk, we compute the perception coherence coefficient the training set, yielding values {PC+)
Downstream classification. To evaluate the usefulness of the transferred representations. we freeze all feature layers of each student model in {S+]== and train only the final softmax layer in a supervised fashion on the labeled training set. This protocol isolates the quality of the learned representation: higher accuracy directly reflects task-relevant features. After training, by testing the trained models on the test set, we obtain the test accuracies {Accs ] corresponding to different checkpoints {
Table annd Fig. show the relation between the perception coherence values and corresponding downstream classification accuracies (i.e., {PC& v.s. [Acc)Jkmm). We observe a strong positive relationship between perception coherence and downstream performance: student models with higher PC consistently achieve better classification accuracy. Quantitatively, the Pearson correlation coefficient between training PC and test accuracy is 0.920. These results establish a direct task-level connection between perception coherence and classification error. Even in this stylized setting, improved coherence of the learned representation leads to higher linear separability and better downstream accuracy. This supports the interpretation of perception coherence as meaningful proxy for representation transferring quality (without the need for label).
Table 1: Training perception coherence versus downstream test accuracy for student models at different training epochs.
Training perception coherence level
0.841
0.877
0.889
0.901
0.911
0.921
0.931
0.940
0.949
0.956
Test accuracy (%)
82.00
83.50
83.25
85.75
86.75
88.00
86.75
89.25
87.00
90.00
Test accuracy for downstream task (%)
0.90
Fitted Ling
Pearson correlation: 0.920
0.86
0.84
0.84
0.06
0.00
0.90
0.92
0.94 0.96
Training perception coherence level
Figure Training perception coherence versus downstream test accuracy for student models different
training epochs.
5.3 Learning Metric Evaluation on Neural Networks
In this section, we perform experiments on CIFAR10 (Krizhevsky et al. 2009) and the subset the first 30 classes of CUB-200 (Wah et al. 2011) with very small student models. Following the previous works (e.g. (Passalis & Tefas 2018a Passalis al. 2020bla ), we evaluate the quality of the learned representation using the content-basee image retrieval setup. We used the official code released with the work of Passalis et al. 2020aa) to evaluate different metrics, including the interpolated mean Average Precision (mAP) (at the standard 11-rocall points) and the top-k precision (Christopher et al. 2008). For a fair comparison, we use the same leacher and student model archilecture as in Passalis el al. (020a).. The student models are very small, composed of only convolutional layers and a fully connected layer. All details are in Appendix [E.] To conduct the experiments, for cach dataset CIFAR10 and CUB-200, we first train the teacher model on the training set. Then, the transfer only the training set without label. Then, the retrieval setup is used evaluate the quality of the learned representation. The database contains the training set and the test set is
12

---

Published in Transactions on Machine Learning Research (02/2026)
used for evaluating the performance of each model. If the student is trained to learn useful features, then the test examples should be close to the training examples (database) of the same class. Hence, higher retrieval score indicates better quality of the learned representation. The results for CIFARIO and CUB-200 are shown in Tables 2 and 3 respectively. Our method is also compared with classical methods, including the standard kee distilling (KD) (Hinton el al. 2015 FitNet. (Romero el al. 2014), metric knowledge transfer (MKT) YYu al., 2019), probabilistic knowledge transfer (PKT) (Passalis & Tefas) 2018a) and Heterogeneous Knowledge Distillation (HKD) (Passalis et al.] 2020ap. The results displayed for all these methods in Tables 2 and 3 are extracted from Passalis et al. (2020a). Therein, one used the same teacher and student models. For FitNet, MKT, PKT, there are two versions: the first one applies only single
penultimate layer, while the second one applies on multiple layers (denoted by -H).
Table 2: Learning Metric Evaluation on CTFAR10.
Multi-layer single layer distilling Method Teacher Model (ResNet-18)
Auxiliary Model
mAP
top-100
91.94
93.50
Ours
No
54.25
65.00
KD (Hinton et al. 2015
No
40.53
58.56 36.56
Single Layer Distilling
Fit.Nt Romero ct al.
2014
No
48.99
62.42
MKT Yu al. 2019
No
38.20
52.72
PKT Passalis & Tetas
2018a
No
51.56
62.50
FitNet-H
No
46.46
60.59
MKT-II
No
43.99
57.63
Multi-layer Distilling
PKT-II
No
51.73
63.01
HKD Passalis et al. 2020a
Yes
53.06
64.24
Table 3: Learning Metric Evaluation subset of the first 30 classes of CUB200.
Multi-layer or single layer distilling
Method
Auxiliary Model
mAP
top-10
Teacher Model (ResNet-18)
67.48
74.38
Ours
No
28.42
36.55
KD (Hinton et al. 2015
No
18.55
26.57
Single Layer Distilling
FitNet Romeeo et. al.
2014
No
15.98
23.41
MKT YYu et al. 2019
No
13.39
20.59
PKT (Passalis Tefas
2018a
No
18.57
26. 26.70
FitNet-H
No
15.37
22.61
MKT-H
No
15.39
22.76
Multi-layer Distilling
PKT-H
No
17.77
25.39
HKD Passalis et al. 2020a
Yes
19.01
27.67
From Tables 2 and we observe that MKT method provides the worst results. This method is composed of main ingredients: (1) minimizing the distance between the student and teacher features and (2) enforcing the student model to result in similar distances between the data points the teacher model (in the feature space). Hence, in our setting of very small student models, this brute force approach provides the worst results. Our method also outperforms by large gap the standard approaches of KD (using the soft label) FitNet (using the distance between the student and teacher fcatures). More advanced methods such as PKT (based on a kernel method) performs better, but is still outperformed by our method. Besides, HKD is a more advanced version PKT, where one uses a particular mechanism to transfer the features. However, our method still provides better results. Notice that our method applied only on the penultimate layer, but outperforms the other methods using multiple layers. Especially on CUB-200, our method outperforms the others by wide margin. This proves the effectiveness of our method for feature representation transferring.
13

---

Published in Transactions on Machine Learning Research (02/2026)
5.4 Our method guide for knowledge distilling: comparing with other methods of feature representation transfer in the context of classification
In the above section, we show how our method helps the student model to produce useful features. Tn this section, we demonstrate that by transferring the feature representation, our method boosts siguificantly the performance of the student model in the context of the classification task on CIFAR100 (Krizhevsky) 2009). While different methods have been proposed and tailored specifically for the classification, we focus mainly on state-of- the-arrtthh-ttttoo feature representation transferring methods. Notice that different methods (such as FitNet [Romero et. a 2014h) apply the transferring on different intermediate layers. However, we propose to apply our method only on the penultimate layer and on the logit one (before applying the softmax function). We also fix all the hyperparameters for training. without further tuning to show the simplicity of our method. The most recent VRM method (Zhang et al. 2025) introduces virtual relation matching and applies RandAug (Random Augmentation). For a fair comparison, we add RandAug to the inputs using the official code associated with this last work. Then, we apply our method under the same setting. All details can be found in Appendix E.4 While different methods have been proposed to effectively transfer knowledge between model of the same archilecture lype (e.g. ResNet50 + ResNet118) we intentionally choose pairs of models having different architecture types. More precisely, we use 3 pairs of teacher and student models: ResNet-50 MobileNett2 V2 and ResNet-32x4 + ShuffleNet / Vet We follow the same setup and use the same pretrained teacher models officially released with the work of Tian et al. 2020) for fair comparisons.
Table 4: Test classification accuracy (%) on CIFAR100 using different. knowledge distillation methods.
Results of ReviewKD (Chen et al., 2021), TTM (Zheng & YANG, 2024) and VRM (Zhang et al.] 2025 are extracted from the corresponding original papers, other methods are extracted from Huang et al. 2002 where one used the same setups.
Method
ResNet-50 MobileNetV2
Resdet-32xd
ShuffleNetVi ResNet-32x4 ShuffleNetV2
Teacher Student
79.31
79.42
79.42
64.6
70.5
71.82
KD
Hinton et al.
2015
67.35++0.32
74.07+0.19
74.45+0.27
PitNet
[Romero of aL
2014
63. 16=0.447
73. 73.59+0.11
73.54+0.22
VID
Abn et gl
2019
67.37=0.28
73.38=0.00
73. 13.10:0.111
RKD
Park et al
61,1320.12
72.28:0.33 0.39
73.21:0.28
PKT
Tassulia
2018a
06.3220.33
74.10:10.25
74.6910.34
CRD
Iin ml.
2020
10.16
75.11:10.32
75.6510.10
Review KD CChen ct al.
2021
69.89
77.45
77.78
DIST
Hnang ct al,
2022
68. 66.66.223
76.34+0011
77.35+0.22
TTM
Zheng & YANG
69.59
74.37
76.55
VRM
with Aand)ngg
Zhang ot al. 2025
72.30
78.28
79.34
Ours
[with RandAng)
01.10=00.1
78.59=0.13
79.14++0.10
From Table we see that our method boosts significantly the performance of the student models (when not using any KD technique). It also provides much better results than standard techniques such as KD or FitNet. This proves its effectiveness for extracting knowledge from the teacher model. Besides, our method systematically outperforms the PKT method based on kernel techniques (see Section 2). This reinforces our intuition that our method is more effective than applying the same kernel for all points, as discussed in Section 2 Other methods based on mutual information such as Variational Information Distillation (VID) is also outperformed by ours. The method of Contrastive Representation Distillation (CRD) is also based on an informationntheoreiii framework. Our method outperforms this last one. Moreover, it outperforms all the other methods on ResNet-32x4 1aaf-322x///SfffNNeWWWW benchmark. We also observe that our method achieves on-par performance compared to VRM, even though VRM includes additional components such as inter-class relation modeling and edge pruning strategies. Our approach intentionally uses minimal setting, yet still matches VRM performance. All these remarks prove the effectiveness of our method for KD.
Finally, notice that some different. knowledge distilling methods are tailored specifically for classification (e.g. Huang et al.] (2022)). However, our main objective is to study methods for feature representation transferring, as this class of mothods is more generic and applicable on wider range of tasks (e.g. retricval). Through our experiments, we aim to show that a good feature representation transfer allows us to effectively distill the knowledge from neural network.
14

---

Published in Transactions on Machine Learning Research (02/2026)
5.5 Ablation study: effect of mini- size
In Section 4.1 we established a theoretical result showing that the global perception coherence level (GPCL), when estimated using mini-batches of size B, converges in expectation to its true value at a rate of O(1/v (Theorem 4.1 We now turn to an empirical validation of this result. To this end, we employ the estimator defined in Definition CIFAR10 and CUB200 experiments. The outcomes summarized in Figure and Table 5
4.1 with varying batch sizes, and evaluate it on trained student models from the
From Table 5 we observe that for very small batch sizes (e.g., B = 4), the estimated GPCL, deviates significantly from the reference value, reflecting the limited information available in such small batches. As the batch size increases (c.g., B 16 or B = 64), the estimated GPCL rapidly converges toward the true value. In fact, with 32 or B 64, the estimator already provides a stable and reliable approximation. For instance, in the CIFARIO case with 10,000 test examples, using B 64 yields an accurate estimate with low variance, thereby confirming the predicted convergence rate with respect to batch size.
Table 5: Estimation of global perception coherence level, using different mini-batch sizes.
Mini-basch
256
CIFAR10
0.803 + 0.00888
0.377 040066
0.8131 0XRN332
0.8021 + 0.00022
0.7960 + 0.00028
0.7944 + 0.00077
0. PM51 10851+000 0.000114
0,7919
CUB200 sabwet
0,8919 4 0.0027
0,8312 2 0.0111
08073
0.00855
0,7059
0.001.6
0.7808 3 0.0088
0.7878 a. 0.0009
0.7874 4 0.00067
0.7859
0.50
0.90
True value (full test set)
True value (full test set)
Global Perception Coherence Level
Estimated values by mini-batch, run
Global Perception Coherence Level
Estimated values by mini-batch,
0.88
mini-batch, Estimated values by mini batch, run
Estimated values by mini-batch, run
Estimated values by mini-batch, run
Estimated values by mini-batch, run
Estimated values mini-batch,
Estimated values by mini-batch, run
0.86
Estimated values mini-batch,
0.86
Estimated values by mini-batch, run 5
Estimated values (averaged over runs)
Estimated values (averaged over 5 runs)
0.84
0.82
0.80
48
32
64
96
128
192
256
4826
64
95
128
192
256
Batch Size
Batch Size
(a) CIFAR-10.
(b) CUB200 subset.
Figure 7: Estimated global perception coherence level at different batch sizes.
To further investigate the qualitative impact of batch size, conduct experiments toy datasets where representations are transferred from R + R² (Figure B and from IR2 + R22 (Figure gh. Here, we explicitly vary B and analyze the resulting student configurations. When B is too small (e.g., B = 3), the learned configuration fails to preserve the structural coherence of the reference configuration. This is expected, as such small batches provide only very limited relational information between points. Starting from = 7, however, the lcarned configurations begin recover the global structure, and from B 15 onward, the results become stable
These findings suggest that excessively large batch sizes are not necessary for capturing global coherence. Indeed, during training, each epoch involves drawing random mini-batches of size B, and within cach batch, every reference point is compared to B - others. Due to random sampling, across successive epochs each point is paired with different comparison sets. As a result, the model gradually aggregates sufficient global structural information, even when B is relatively small compared the dataset size. This explains why reliable approxinations of GPCL and structurally coherent student configurations can be obtained without resorting to very large batch sizes.
15

---

Published in Transactions on Machine Learning Research (02/2026)
(a) Reference configura-
tion.
(b) Batch size equal to 3.
(c) Batch size equal to 7.
(d) Batch size equal to 15.
Batch size equal to 32.
(f) Batch size equal to 64.
gg) Batch size equal 128.
Figure 8: Ablation study of the effect of batch size on 3D + 2D Gaussian dataset, where we train with different batch sizes. Plots from left to right in each row represent evolution of the learned configuration during training.
5.6 Ablation study: Student model size
In this section, we investigate the influence of student model size on transfer performance. Recall that in the CIFARIO and CUB200 experiments, the baseline student architecture consists of three convolutional layers (cony1/conv2 //o/vo3), each followed by a ReLU non-linearity, and a final fully connected layer. To explicitly examine the effect of model size, we construct progressively smaller student variants by truncating the network different depths and applying Global Average Pooling (GAP over the two spatial dimensions to obtain the final representation:
Formally, GAP: XXXWXC RC, such that for XXWWXC for (1,2,-.. ,C, GAP(2) FW E" En Fikk-
16

---

Published in Transactions on Machine Learning Research (02/2026)
(a) Batch size equal to 3.
(b) Batch size equal to 7.
(c) Batch size equal to 15.
(d) Batch size equal to 32.
(e) Batch size equal to 64.
(l Batch size equal to 128.
Figure 9: Ablation study of the effect of batch size on two-moon dataset, where we train with different batch sizes.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Model 1 (one layer): conv1 + GAP.
Model 2 (two layers): convl + conv2 + GAP.
Model 3 (three layers): convl conv2 conv3 GAP.
Model 4: the full student model (original architecture).
The results are reported in Table For both datasets, we observe a consistent trend: as the student model becomes larger, the global perception coherence level (GPCL) increases. This indicates that deeper models are more capable of preserving the ranking structure induced by the teacher's feature space.
Moreover, we note strong correlation between GPCT, and downstream performance metrics such as mean Average Precision (mAP) and top-k accuracy. In particular, models with higher GPCL systematically achieve better classification performance. This empirical evidence reinforces the role of GPCL as a meaningful and effective objective for guiding student learning. also highlights a practical limitation: when the student is excessively small, limited represcntational capacity hampers the preservation of structural coherence, leading to degraded performance.
Table 6: Performance of different models, with different sizes, with the same teacher model.
Data set
Metric
One layer
Two layers
Three Layers
Full model
Global PC Level (on test set.)
0.7015
0.7491
0.7700
0.7919
CIFAR10
mAP
17.08
26.52
37.69
54.25
top-100
20.43
35.36
50.29
65.00
Global PC Level (on test set)
0. 7015
0.7266
0.7602
0.7859
CUB200 subset
mAP
6.63
8.66
18.88
28.42
top-10
10.35
13.38
27.28
36.55
6 Scope and Practical Considerations
Scope. Our method is intentionally designed for heterogencous teacher-student scttings, where the teacher and student may differ substantially in architecture and capacity. This scenario reflects practical deploy- model to a smaller and ment needs, in which knowledge is often transferred from a large, contrast, we are convinced that ho- structurally different model for efficiency or hardware constraints. mogeneous settings (e.g., similar architectures) are typically well served by standard knowledge distillation (KD) approaches. Our goal therefore not replace or improve conventional KD homogeneous cases, but rather to focus on practically relevant heterogeneous regime.
Moreover, our focus is to introduce new measure, termed perception coherence, for feature representation transfer between models, ideally to enhance representation alignment specifically within the KD) setting, Accordingly, our empirical comparisons are restricted to distillation- baselines. Broader general-purpose representation learning frameworks, such as self-supervised representation learning, are beyond the scope of this study. Nevertheless, these methods are conceptually complementary to our framework, and their integration with the proposod approach represents promising avenue for future investigation.
Computational Complexity. The theoretical per-batch complexity of the proposed loss is O(B3) due to the computation and comparison of pairwise dissimilarities under soft ranking. However, the implementa- tion relics entirely on batched tensor operations (distance matrix computation, sigmoid-based soft ranking, and squared loss), which are highly optimized on modern GPUs. Moreover, soft-rauking computations for different reference (anchor) points are independent and can be parallelized. With N GPUs, the batch can be partitioned along the reference dimension, allowing cach device to process approximately B/N anchors. The final loss is aggregated via standard reduction operations (e.g., all-reduce), enabling near -inear scaling in practice and substantially reducing wall-clock training time.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Besides, it is worth emphasizing that, in many practical and industrial scenarios, lightweight models are trained on servers or high-performance hardware and subsequently deployed to edge devices for inference. Therefore, the training cost does not impact the deployment of the final student model on the edge devices.
Practical Guidance for the Temperature Parameter т. The temperature parameter T in the soft- ranking formulation controls the smoothness of relative distance comparisons. Empiricallly we find that values in the range {0.1, 0.3} work well for both teacher and student models. From a training-dynamies perspective, using slightly larger for the student can facilitate optimization, values may weaken gradient signals. Conversely, smaller т for the teacher encourages sharper relative ranking, since large values render the sigmoid function more linear and reduce ranking meaning In our experiments, we intentionally use the same across different teacher student pairs for simplicity and fairness. Nevertheless, task-specific tuning of t may further improve performance in practical applications.
excessively small т
7 Discussion and Conclusion
Preserving dissimilarity ranks constitutes a flexible requirement, as it does not enforce a complete replication of the teacher's 's geometry. Nevertheless, an important theoretical question remains open: to what eatent can a given studenl model preserve these rankings as a function of ils represenlational expacily? Addressing this question lies beyond the scope of the present work but represents a promising direction for future theoretical investigations.
Why rankings? Generally, in the teacher model's feature space, smaller distances (or dissimilarities) often reflect higher semantic similarity. By teaching the student model to preserve the relative rankings, it learns to order semantic dissimilarities in a way that aligns with how the teacher model perceives the input somantic dissimilarity. order to replicate this ordering, the student model has to undersland features that capture similar semantic content-thus encouraging the learning of meaningful and transferable representations.
By considering dissimilarity rankings, method implicitly captures structural information about the un- derlying data manifold. This perspective can be viewed as form of topology-award aware representation transfer. Topology is concerned with invariant properties of configuration under continuous deformations. Analo- gously, in our framework, as long as the relative distance orderings are preserved, the learned representation remains invariant under local or global distortions. This provides additional flexibility and stability, distin- guishing our approach from peometry-preserving methods. notable difference from classical topological approaches lics in the probabilistic framework that we introduce. While topology typically characterizes data configurations deterministically, our method relaxes this view by embedding ranking preservation in a probabilistic setting. Furthermore, by integrating neural networks into this framework, open the path toward topology-aware knowledge transfer in deep learning.
In summary, this paper introduced an original method for representation transfer between models with arbitrary feature dimensions, based on the novel concept of perception coherence. The probabilistic formu- lation provides new theoretical insights into the knowledge distillation process, showing that in the ideal case the student perceives the input space in the same way as the teacher. Experiments confirm that the method enables efficient lightweight transfer while maintaining competitive performance. Importantly, the proposed framework is generic and can be applied with different dissimilarity metrics for the teacher and stu- dent models. This flexibility suggests promising applications in task-specific contexts, such as multi-domain or multi-modal transfer. We believe that this work paves the way for future advancements in knowledge distillation, especially those grounded in topological perspectives for deep learning.
Sgee Appendix F.5 for a brief analysis about the temperature.
19

---

Published in Transactions on Machine Learning Research (02/2026)
References
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVP conference on computer vision and pollern recognilion, pp. 9163 9171, 2019.
Vasileios Belagiannis, Azade Farshad, and Fabio Galasso. Adversarial network compression. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 10925-10934, 2022.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 535-541, 2006.
William Chan, Nan Rosemary Ke, and Tan Lane. Transferring knowledge from a rmn to a dnn. arXiv preprint WrXiv:1504.01455 2015.
Defang Chen, Jian-Ping Mci, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 11933 11942, 2022.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in newral information processing systems, 30, 2017.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp- 4008-5017 2021.
D Manning Christopher, Raghavan Prabhakar, and Schutze Hinrich. Introduction to information retrieval, 2008.
Wojeiech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. Advances in neural information processing systems, 30, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXie preprint arXiv:1503.0 025511 2015.
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. Advances in Newral Information Processing Systems, 04:37766-33722 2022.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXio:1707.01211 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of
Toronto, 2009. Technical Report
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). Technical report, University of Toronto, 2009. URL http::/www. www.::/w...roooo.ed..... woronto.edu/-krii//eae html
20

---

Published in Transactions on Machine Learning Research (02/2026)
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2117 2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017b.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 34, pp. 5191-5198 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3967 3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-28-4 2018a.
Nikolaos Passalis and Anastasios Tefas. Unsupervised knowledge transfer using similarity embeddings. IEEE transactions on neurol networks and learning systems, (0)9): 950, 2018b.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogemeous knowledge distillation using in- formation flow modcling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recoguition, pp. 2339 2348, 2020a.
Nikolaos Passalis, Maria Tzclepi, and Anastasios Tefas. Probabilistic knowledge transfer for lightweight deep representation learning. IEFE Transactions Neural Networks and learning systems, 32(5):2030 2039, 2020b.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 141. 2014.
Vladislav Sovrasov. ptflops: flops counting tool for neural networks in pytorch framework, 2018-2088 URL
https://giin http:/ggttttttt..t/ wttp:///////ooo////ooo/oo--/-o-e www.teo..ttoorrreeeee
Mingxing Tan and Quoc Lo.
Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Mingxing Tan and Quoc Lc. Efficientnetv2: Smaller models and faster training. In International conference on machine learning, pp. 10096-10106 PMLR, 2021.
Zhiyuan Tang, Dong Wang, and Zhiyong Zhang- Recurrent neural network training with dark knowledge transfer. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5900-5904. IEEE, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision, pp. 4068 1488-407,, 2015.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds- 200-2011 dataset. California Institute of Technology, 2011. Technical Report CNS-TR-2011-001.
Jiabao Wang, Yuming Chen, Zhaohui Zheng, Xiang Li, Ming-Ming Cheng, and Qibin Hou. Crosskd: Cross- head knowledge distillation for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16520-100000 16530, 2024.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and crudent-teacher learning for visual intelligence: A review and new outlooks. IEEE Imnsactions on pollern analysis and machine intelligence, 44(6):3048- 3068, 2021.
Junho Yim, Donggyu Joo, Jihoon Bae, and Jummo Kim. A gift from knowledge distillation: Fast optimiza- tion, network minimization and transfer learning. In Proceedings of the IEEE conference computer vision and pallern recognilion, pp. 4133 4141, 2017.
Lu Yu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, and Arnau Ramisa. Learn- ing metries from teachers: Compact networks for image embedding. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pp. 2907-2916, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.000 0.0928, 2016.
Weijia Zhang, Fei Xie, Weidong Cai, and Chao Ma. Vrm: Knowledge distillation via virtual relation match- ing. In Proceedings of the IEEE/CV International Conference on Computer Vision (ICCV), pp. 2707 2717, October 2025.
Kaixiang Zheng and EN-HUI YANG. Knowledge distillation based on transformed teacher matching. In The Twelfth International Conference on Learning Representations, 2024.
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. Advances in neural information processing systems, 31, 2018.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix A Convergence of the global perception coherence level estimated by mini-batch
We provide here a full proof of Theorem 4.1
Proof A.1 Step 1: Decompose error
We note that
DCB- DC Σ (f),)(X),))) - -x,,(X,, - -I,
xx,1
1
+ - XX) 11)
Thas,
DCB DC s
Σ (Axx(X,,,)) - X2, - (F,(X, xX,X,,,,,))
B²
+
Σ (F((,,)))))) -XI
B
We now lake expectation both sides:
E
DCB DC < Ei + Ez.
where
(x,)(( - Fx,((X,, - |F,(,, X2) Bxx6 x3xxx11)
E =
B 1.j
Eg =
X FXXX,,,)) DC
B
ij
Step 2: Bound Ei
We use the reverse triangle inequality: for any real numbers
||a|
|b|| - |a b).
23

---

Published in Transactions on Machine Learning Research (02/2026)
Applying this, obtain:
B
(X,, - Xx, - (1, -18(X,,) -,
E1 =
B
- E
-((( -
B
W
(X) - PxXi,
(Ri,(X,,)) X,, -F((XXX))))
$
E
B
i-=1
Σ
-
-i + - F x))))
B +
Control empirical CDF deviations. For fixed i E {1,2} and fixed x, E N, recall that the empirical CDF is defined as:
B Σ
x,)( =
(di(a,, l
Xx) S d(xx,')))
k=1
where XI,..., Xn ~ i.i.d.
Each summand is a Bernoulli random variable with mean:
E (d, Xx) S (=),,,"'))== s de(x,xx)) = (1),
Hence, (x,)(x the average of i.i.d. Bernoulli random variables with parameter Fe(2, 2'), and has
variance:
Var (Fie(z,2') = 1 Fi(e, (1F(((())))) 4B
B
By Jensen's inequality with function t + t convex, we have:
F,,,(1, - Fo(x, x'), s Var(Fr. x()) - E
2 B
As this hold for any x, x', for each fixed E {1,2), we therefore obtain
1
B 1
1
E
2
2/B
B
VB
VB
Step 4: Bound E2
Using Lemma A.1 there ezists a constant C' (independent of B) such that
C"
Eg < BB
Final bound:
B Σ FX(,,))) F2(X, X,)1
1+C
DC
SE+
B
/B
/B
Remark A.1 No assumptions about lies or continuily of d; needed for this expectation convergence.
24

---

Published in Transactions on Machine Learning Research (02/2026)
A.1 Technical lemma
Lemma A.1 Let XI....,Xp XB be i.i.d drows from distribution D measurable space X. Let h X x X + R be a measurable kernel (not assumed symmetric). Define 0 == E[, X2)]- Assume h. is uniformly bounded: |/blloo == supz,y |h(a, y)1 =: M < xc. We define
B
B
BB =
M(X, Xj).
B
Then,
B B - 01] - 2(v2 + 6/3)
VB
In particular, V 01 O(B-123)
Proof A.2 The proof follows the standard Hoeffding (projection) decomposition and a counting argument for the variance of the degenerate component; no symmetry of h required because we sum over all ordered pairs (i.
1. Hoeffding decomposition (ordcred pairs). Let X' denote an independent copy of X1. Define the first-order projection
h1(x) = E[h(x, x']] - 0,
and the second-order (degenerale) remainder
h2(2, =- ((, h1(2) - -v) 0,
where for notational symmetry we also set
:= = F[M('' :)] - O.
By construction
[X)] = (X)) (X)) = 0, E[he(X, v)1 E[h2(1, X)] 0 for all x. y.
The kernel decomposes as
h(o, v) = 0 + h1(2) + hı(v) + (2(
2. Decomposition of the V-statistic. Plugging this into VB gives
VB - 8
(X,, + hx(X,) + h2((,,,)))
B2
i.j
h1 (X,) + B
Σ (x,))) + B
Σ h2((,;;))
B
j=1
B
1
hi (X.) + B3
Σ
hx(X)) + Ra:
B
i=1
where we denote
RB B
1
Σ
ha(X Xj).
i,==1
Define the linear part LaB 1/2 hn
1/2 E, (X;),,
VB - .. RB.
25

---

Published in Transactions on Machine Learning Research (02/2026)
3. Bound the linear part LB.
The two sample sums are independent and mean zero; hence
1 (X)( + B² Var(hi(X)) - //8 Var(h
Var(Ln) B
By Jensen,
B|l & /Var(LB) = 1
vB Var(hı(X)) + Var(ha(X)).
It is easy to see that |hi| S 2M. Hence, Var(hi (X)) - 4M2. Similarly, Var(hi S 4M2, Therefore
2/2M
E B|LB v
(1)
VB
4. Bound the degenerate part Rg. We bound B[[ via its second moment:
B|| S v E Va)(R
Compute
Var(RB) 1 B1 K.J.k.Eml
Σ (x,, X,), ( Xe)).
Independence implies that if the inder sets {i,j} and {ki, l are disjoint (i.e. all four indices are distinct), then the covariance is zero. Thus only quadruples with at least one shared index contribute. We count these quadruples.
The number of ordered quadruples with all four indices distinct equals
- 3)) B -1
so the number of quadruples with at least one common indez
B (B* - 6B + 11B= -BB = - + 6B < 12B".
Notice that E [2222(X,,))) x,)] = O. Therefore,
Cov (Xi, X2 (Xx, Xx)) = | [h2(,, h2(2,, Xx)l S sup 1/2
Moreover, hhel S | lh1 |h1 + 10). It easy to see that |0| s M, |h1 $ 2M and |h1l S 2M. That is, hal I 6M. Therefore,
Cov Xx,,
(Xx, Xz))
36M2
Consequently,
12B3
(12 M?
Var(Rn) < B1 36M2
B
Therefore
12v3 M
B[BBl] S / (a)(BB)) <
(2)
VB
5. Combine the two bounds.
Using the triangle inequality,
E [[ - 01] B||L R[]
Plugging (1) and (2) yields
E IVB 01] <
M
22 + 12V3) (2V2+1233)
2(v2 + 6 6/3)M
VB
VB
26

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix B Proof of Theorems 4.2 and 4.3
B.1 Proof of Theorem 4.2
Theorem B.1 (Theorem 4.2 in the main text) Assume that f2 is a-perception coherent with fi at æ E X (0)))2(z) 2 a). Then, for all E2 > Ei > 0, we have
-) S (1-aa a)
xx,x (F2(2, (2 02
(8)
E2 E1
where Co is a posilive constant depending on E1
Proof B.1 For the sake of brevity, let A(X,X, X') be the event (x, - F1 & E1- We have
Fx.x/ (EE(E, - (2(2,
A(X,X')
= -xxxx' (12(1, X') - Fi X') + (1, (x, X') - Fi (2. X) Ri X) F2 X)| 0 A(X,X')
< xx
(12(2, X') - Fz FR (I, x') Fi x X) - F2(x, X) 2 A(x,X))
< x
(1a(xx, X') - (x), |8 (z, X) P2(x, 2 22--88 61
|A(XX'))
62 -2
| A(X,X))
< max - 1F P2
X1))
2
Using Lemma
B.1. we have
Xx,x
x))
82 -1
A(X,X')
4(1-a) (82 -8--811 Ex) (A(X,X'))
C1
E2 -1
where Cz, Fx,x(( Fi(x,))) -((..))53))) Therefore,
Px,x (IF((s, P2(x, F2(x,
A(X,X') <
Cs (1-(aa))
82 E1
This completes the proof.
B.2 Proof of Theorem
Theorem B.2 (Theorem 4.3 in the main text) Assume that f2 is a-perception coherent with f1 at x e X (1))a(()) 2 a). Then, for all e > 0, we have
Fx,x
5 s C(11a)/E
(9)
where C is a positine constant.
Proof B.2 For the sake of brevity, let A(X,X) be the event Fi X') S X), We have that
Px,x' (Fa(e, X')
A(X,X))
= =FX.X' F2(x,X)) - - - Fi X)) 2 A(X,X))
=PX,X' (Ba(x,X))
(xx
+((,,))+ - F1 X) 24 A(X,X'))
S PXxx' (F2xx,X) - X1) - Fx(x,)) 2 A(X,X'))
< Pxxxx (max (F2xx,') (x,, (x, X) F2(x, X)) 2 2//2 A(X,X')
< Px,xx (max (1,2(1, X') (xx XF,, X) F2(x, X)) 2 8/2 |((XXx))
27

---

Published in Transactions on Machine Learning Research (02/2026)
Using Lemma B.1
we have
Fx,x (max (1F2(2, (2, (F),,))))))))))))))))))))))))))) (2)x 2/22 A(X,X')
4(1 10-0 a)
/E/
xxxxxx'' (A)X
This leads to the result of Theorem which completes the proof.
B.3 Lemma B.1
Lemma B.1 Let X and X' be the i.i.d random variables following the law Dx and A(X, be event depending on these two variables, A is symmetric in X and X'. Assume that fz is a-perception coherent with J1 at x E X. Then, for any & > 0, we have that
Xx.x' (max (2, X') - (11, - F2(2, x))) 2 A(X,X))
2(1 - a))
$ PPx (A(XX")))
(10)
Proof B.3 Let B'X,X') be the event ((X',)) is the event (2, X) (i,
-
-2(x)) We note that Parthermore, let C(X, be the
- -
event max (x) - (1,x,)),, (x,,,) - F2(x,X)) 2E.
Px,x' (C(X,X') A(X,X'))
PXx.x' (C(X,X') n B(X,x') A(X,X)) + Px.xx ((X.X')B((X.X) A(x.x))
Using Assumption 4.
we have
Fx,x' XX
A(X.X') Pxxxx (C(X.)))((XX. A(X,X))
Therefore,
Px,x' (C(X,X') | x(X,x))
Pxxx' (CIX,X) B(X,
A(X,x)) + Pxxx (C(X,X)) B(X',x) A(X,X))
2Fx.x'
(C(X,X))
B(X,x)
A(X.X') (A(X,X') and C(X,X') symmetric in X and X')
xFx,x''' (c(X,X)
A(X,X') 0 B(X,x') Px,x (B(X,X') A(X,X))
We note that (B)X,X') A(x,x)) B(X, A(X,X')= =1. Using Assumption 4.2 we have that
(B(X, x()xx)) (B(X,x') 1(X,x)) =1 As A(X,') is symmetrie X and X', we obtain that 2 (B(X,'') A(X,x') 1. Hence, (B(X,x') A(Xxx')) = 1/2. Therefore,
Pxx (C(X,X') A(X,X')
x (C(X,X)
| (
=Px,x'
(C(X,X') A(X,X)) n X(X,x))
= -xx,x'
(Fi - x2(x,X) 2 |(Xx,')) (B(, (x, X) 12 2 B(X,X')
Px,x/
Px.x' (A(X, X
28

---

Published in Transactions on Machine Learning Research (02/2026)
S Px, (Fx,
Px,x' (A(X,x)))(
Pxx
(numerator independent of X'
Px,X' ((XxX))]B(XXX))
Ex (I, -F,
S EX (A(X, (XX, (XX, xx)) (Markou's inequality)
(1-a)
xx (A(X,X)) B(X, x'))
Note that by using Assumplion
once again, we have
Xx,x (A(X, = -xx,x' (A(X.X') (ACX.)) B(X,x)) PX,x ((X,X")B(XX,.
= =Xx,X' (A(X,X') U B(X, (B(XX))
+ +Px.X' B(X,
2Pxxx (A(X.X') nB(x,)))
Hence, we obtain
Px.x' (c(X,X)
| A(X,X3) S sxxxxx' (A(X,X1)
2(1 aa) a)
This completes the proof.
Appendix C Proof of the theorem on global coherence
C.1 Proof of Theorem 4.4
Theorem C.1 (Theorem 4.4 in the main text) Assume that f2 is globally a-perception coherent with Si (iee, (X)) 2 a). Let X1, X2 and Xa be it.dd random voriables following the law Dx. Then,
1. Global rank preservation. For all 82 81 0, we have
Px,.XexXs (132 - - 52 (F( S SEI)
Ce, <
82 -1
where Ce positive constant depending on E1
2. Global relative order preseroation. For all > 0, we have
C(1 a(1
Xx..X Fx) 2E ( <
where C is positive constant.
Proof C.1 1. Global rank preservation.
Let A(X1, X3) be the event Fi(XI, Fi (X1, X3)l S1. Let sa in the proof of Theorem 4.2 we can obtain that
Using the some technique
Px Px,.X (122XX,, X2) - Fx(X,,
X2, Xs)
P maxx (IF2(X1, - (((XI, (X), Xa) - F2(X1, 28 A(X1,X2, X2)
29

---

Published in Transactions on Machine Learning Research (02/2026)
Applying Lemma
C.1
have that
Px, Xs (Fa((1))22) - F2 Xx)] 2 02 X2,
2(1 (1)
4(1 - a))
s Xa ( (A(X1, X2, (2,
(82-61)
- EI X2, ( (A(X1, X3))
4(1 -1)
Ce (1 a
(E2 El X2 (X1, Fi(X1, Xz) Eii)
E2 -1
where Ce, = Xx (1 (X1. X3) This completes the proof.
2. Global relative order preservation. Let A(11, Xz) be the event F X2) S FX Xz). Using the same technique as in the proof of Theorem 4.3 can obtuin that
X2,, (F2(X1, - (2(X1, 2E X2,
< <a max (11, FI [1 Xs) Xx112 2 A(X1,X2,
Applying Lemma
C.1
leads to
Px,,Xx.Xs (1F2(X1:X)) F2(X1,X8)) 2 X2,
4(1 -x)
sPP X2 (A(X1, X3))
4(1 (x)
C(1 a)
EP X XXX,XXX,,AX,,,, (A(X1,
Xz))
where C Px..N.,,,((AX.
This completes the proof.
C.2
Lemma C
Lemma C.1 Let X1, X2 and Xa be the i.i.d vandom variables sollowing the law Dx and A(X,, X2, Xa) be event depending these three variables, A is symmetric in X2 and X3. Assume that f2 is a-perception coherent with fi (Exløn,s2 (X)] 2 a). Then, for any > O, we have that
P Xx X3 (max (1 X2) Fi X2),
F2(X), X(X1,
2(1 -a)
(11)
EXxx.Xxxx, (A(X1,X2,XX) Xz))
Proof C.2 For brevily, we shall use some compact nolations. Let B(X1,X2,XX X2, be the event X2) Xi()
R (X,, Xx), We note that B(X,,X3,,3 is the event Furthermore, let C(X1, X2, X2, Xz) be the event
F2 X2) Xx)) I (x) Xz)l
max (122(X1, X2),,11),,
F2(X1, X3))) 2.. We have
Px..X (2, |
A(XI, A(X1;x2,
= =PxixX
(C(X), Xz) / ((x1.
X(x,, X2,
+ P (C(X1, X²) n B(X,, X2, Xz)
A(X1, X2, X2, Xz)
Using Assumplion 4.2
we have
Px (C(X1,X2,Xa) X2,
A(X1,X2,Xa))
30

---

Published in Transactions on Machine Learning Research (02/2026)
= Xx X2, X X2,
Therefore,
Xx
A(XI;X2,XX) (2, X3)
= -X
(2, X2)
B(XX, X2,
X(,
+ P
(((X,, X2, X1) 0 B(X1, X3, X2)
1(X1, X2)
= XPxxxx..x
(C(X), X3) n B(X,, X2,
1(X,, 12, X3)
(C(XI, X2, X2)
X2,
PX,
(B(X, X2, Xz))
A(x1,xx,
XP
(O(X), N2, X²)
A(X,, X2, X3) n 1 Xa) x x12
-Px
A(X,, (C(X),X2, Xz)
0 B(X), (x,
-Xx (Fi(X,, FXG Xz)) 28 1(X,, X2,xxx 1 B(x,, X2,
Xx (FF, X2) F2 X2) e A(X1,
A(x X2, X2) n B(1,, Xz) X²) B X2,
Px.xx
Xx,,X,,X, (X,, X2) X2)
S Xx,,X X2, X3) B(X,
Xx (IF X2) - (X,, X2) 26)
(numerator independent of X3)
Xx (A(X1,X2, Xa)
X2) B( X2, Xa))
Ex,,x, [111((
= Xx,,X,,Xx Xx, X2 Xz)
Ex. [xx, [1
Xx,, (A(Xi, X2, X X2, X3))
Ex, [xx, (1F) X2) F2 X2) 2 e)]
PxI,X2.X, (A(X,, X2, X3) B(x,, X2, Xz))
Ex,
< Xx
8
(Markoe's 's inequalily)
(A(X1, X2, X2)N B(X,, 1, X2, Xz)) Ex, [1 -bnisa (X1)]
XP ((,1,X2,XX, n B(X,, X2, Xs))
1- 1-Ex, T (Xi)]
eP XXx,, (A(X,, X2, Xz) (BX,, X2, X1))
< XP. (A(X,,X,, B X2,
B(X,,
2(1 a)
el X (A(X1, Xa))
This completes the proof.
31

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix D Proof of Theorem 4.5
Theorem D.1 (Theorem 4.5 in the main text) Consider non-empty set A S X such that fo.fa(a) 2 0, Vx E A. For 0 0<8sl, let As = fI LrEX mingred F(x',x) x) S 53. Then, for any (0 < & S a.), we have
(() sa xx 0(0)
cCA.
where CA. is posilive conslant depending As.
Proof D.1 Let us consider an x As and let a(x) be an arg min EAA F('',)) x)) (if not unique). We have that Fi(a(z), S and Fe(a(x), x) S
which implies (x, a(x))= = (() and (z, a(x)) = 0(5). Notice that (()) is uniform on E As and x' E X. Moreover, for any x' E X, we have
d1(a(z),xx) d1(z,((x)) < (I,xx') di(a(x). + d1 (z)a(x))
Hence,
Fxxxx') S Px +
(x =xd (1, S di(a(x),x)) +d((),((x)))
0 +(de)))))++(+(
On the other hand,
xx, 2 Px (((d)))) Px (d) (x, a(x)) S (i x) s di(a(x),
Si =((a(x), - -((o))))))))((((((( Fi(a(t), =( 0())
-xx
Therefore,
z')l 0()), for any a' X. Proceeding in the same way, we also have F2(x, ==((=))=))))) 0(8). Thus,
2 (x(,=='))
= =|F((z,z') Fi(a(x), F2(a(z), - F2(x, P((()))) + (a(x)), - F2(a(x), $|Fi(d,2') [(( F2(x,x') (Fi((()), F2(a(x),
= = =
Therefore, there erists positive constant C O()) such that (F((x,'') - F2(2,z))
Fi(a(z), - F3(a(z), + Cs, for any (x,x') E As X. Consequently,
Px ( (X) S Ax
Px [1 11 Ex: [X,((XX))) - FX]( < x
Px
- 21- a
< Px
Ex -X) -
1-a+s X (xE
Px (Ex +
nX E As)
Fx (X As)
Px (x (a(x) X') - (2(()), X'I] + 2
Px (X As)
Px (1 (a(x) + C's 2 - a +
Px (x E As)
32

---

Published in Transactions on Machine Learning Research (02/2026)
a (asl--at(22---r- ((n,ss(((X)) 2 a))
Px (X E EAs)
Px (Cs 26)
[(G26))
Px (X E As)
PPx E As)
Co
C3.
(1(C622) S
xx (X E As)
O(6)
<Ixx x (X E As)
0
0(6)
sx (X E A)
sCA
This completes the proof of Theorem
Appendix E Experiment details
For toy datasets on 2D and 3D, we simply use Euclidean distance as the dissimilarity metric. For all the experiments neural networks, use the cosine dissimilarity defined
1 (1 {u, v) ||0||]
d(u, 1) 2
(12)
|
All the experiments are conducted using a single GPU NVIDIA GeForce RTX 4090. The running time for each experiment is reported in each sub-section.
E.1 Toy dataset on 2D and 3D
For these experiments, we use scikit-lean package in Python to simulate the datasets. The two-moon dataset has 700 points, and the 2D Gaussian clusters have 1000 points. The 3D clusters also have 1000 points. The student configuration is randomly initialized using Gaussian distribution. For training, we use batch size equal to 64, and learning rate equal to 0.1. We use Adam optimizer of the Pytorch package for training during 800 epochs. We set the temperature equal to 0.1 in the soft ranking function.
E.2 Stylized Empirical Study of Perception Coherence and Downstream Classification
For simplicity, the teacher and student models share the same architecture, consisting of a feature extrac- tor followed by classification head (softmax layer). The feature extractor has the following structure Linear(2, 20) + ReLU + Linear(20, 20). The classification head (softmax layer) consists of a linear layer Linear(20,2) 2), which maps the extracted features to two output classes, followed by softmax function.
The teacher model is first trained with the training set (with binary labels) for 200 epochs using the Adam optimizer with a learning rate of 10-3 Knowledge transfer then performed by optimizing the perception coherence of the student model to match the feature structure produced by the teacher's feature extractor. This is done by using the same training set (no label). Note that during this transfer stage, only the student feature extractor is optimized, while the classification head is excluded. The student model is trained using the Adam optimizer with a learning rate of 10-3 for 40 epochs. We save the student model every 4 epochs, resulting in a sequence of checkpoints {5
For each checkpoint {S we freeze the feature extractor and train a new classification head in a super- vised manner using the training set (with binary labels). This supervised training conducted for 20 epochs with the Adam optimizer and a learning rate of 10-3 Finally, using the test dataset, the test accuracy after this supervised training is recorded for each checkpoint, yielding {Accs] 110
33

---

Published in Transactions on Machine Learning Research (02/2020)
E.3 Retrieval experiments on CIFAR10 and CUB-200
Following Passalis al. (2020a). we ResNet 18 the teacher model for both datasets. We also the same student models for each datasets, shown in Fig. I0 We apply our method only the penultimate layer (dim 64 in Fig. 10 without using the final classification layer.
Training teacher model on CIFARIO. We use batch size equal to 64. We apply during training the data augmentation techniques, inchuding random cropping (32x32, padding=4) and random horizontal flip- ping. We use the optimizer with following hyper-parameeers:: SGD(Ir=0.1, momentum=00f, nesterov weight decay=5e-4). We train the model from scratch for 100 epochs, where we decay the learning rate by factor of 0.2 at epochs 40 and 80. The training time is less than 30 minutes.
Training teacher model on the subset of CUB-200. We use batch size equal to 64. All images are resized to 256x256. We apply during training the data augmentation techniques, including random cropping (224x224), random horizontal flipping and random rotation (30 degrocs). We use the optimizer with following hyper-parameters SGD(1=00.001, momentum=0.9, nesterov=True. weight_ decay=5e-4).. We train the model from the default. pretrained model of Pytorch for 100 epochs, where we decay the learning rate by factor of 0.1 at epoch 50. The training time less than 5 minutes.
Transferring process. We use the training set the transfer set, without using label. For both datasets, we train for 150 epochs. The technique is applied on the penultimate layer of teacher model (dimension of 512) and the penultimate layer of student models (dimension of 64). For temperature, we fix T 0.1 for leacher model and - = 0.3 for student models. Notice that we use only the training set without label. We use the optimizer as follows: SGD(Ir=0.05, momentum=0.9, nesterov=Trre. weight weight_deeay= decay=5e-4). We decay the learning rate by factor 0.1 at epochs and 100. For each dataset, during transferring, we apply the same data augmentation techniques for training the teacher model. The training time for CIFAR10 is about 20 minutes. The training time for the subset of CUB200 is less than 15 minutes.
CNN-1
CNN-2
Convolutional 2D (3 3, filters)
Convolutional (9 x 9, 8 filters, stride
Convolutional 20 x,
Convolutional (5 5, 16 filters)
Convolutional (3 S, 32 filters)
Convolutional 2D x5 5, 32 filters)
Fully Connected (64 Neurons)
Fully Connected (64 Neurons)
(a) Student model for CIFAR1O.
(b) Student model for CUB200.
Figure 10: Student models for CIFAR10 and CUB200. Images are extracted from Passalis et al. 2020a
Notice that we perform transferring on the features of penultimate layer (dimension of 64), and do not use the final classification layer.
E.4 Classification on CIFAR-100
For teacher models, we use pretrained models released in the official code of the work Tian et al. (2020) without any modification. We also use the same student models from the same code. We follow the same training scheme as the work Tian et al. (2020). More preciscly, for optimizer, we use SGD(1=00001, momentum=0.9, weight_ weight_docayae decay=5e-4). We use the training set for transferring, during 240 epochs, where we decay the learning by factor of 0.1 at epochs 150, 180 and 210. The batch size is 64. Data augmentation
34

---

Published in Transactions on Machine Learning Research (02/2026)
techniques include random cropping (32x32, padding=4) and random horizontal flipping. The most recent VRM method (Zhang et al.) 2025) introduces virtual relation matching and applies RandAug (Random Augmentation). For a fair comparison, in addition to random cropping and random horizontal flipping, we also add RandAug to the inputs using the official code associated with this last work. We fix temperature T= 0.2 for teacher model and T = 0.3 for student models. Our lechnique is applied on the penultimate layer and the logit layer (before softmax function). The total loss for training the classification student model is Liotal -eross-entrooy +ourss, where we fix À = We perform independent runs to report results in Table The rumning time for one run (240 epochs) is follows: ShuffleNetV1: about 45 minutes. Shuffeeet V2: about 52 minutes. Vob:leNetV22: about hour.
E.5 Ablation study: effect of scaling temperature in the soft ranking
In this section, we investigate the effect of scaling temperature in the soft ranking. We fix the temperature of the teach models as precised in Section E.3. and use different temperatures for the student model. Results are shown in Table 7
Table 7: Ablation study: performance model with different values of T.
(temperature)
0.001
0.01
0.05
0.1
0.3
0.5
5.0
10.0
mAP
15.61
16.93
14.30
17.24
28.42
30.38
14.16
13.17
CUB200 subset
top-10
22.67
24.29
20.11
23.86
36.55
38.49
18.87
18.04
mAP
50.22
49.36
49.50
50.86
54.25
53.72
27.38
25.14
CIFAR1O
lop- 1op-100
100
61.02
60.27
60.22
61.36
65.00
63.89
32.43
28.51
From Table 7 we have the following remarks:
Small Temperature. With very small T, the sigmoid approaches step function, causing vanishing gradients almost everywhere. This was confirmed in our ablation study on CUB2200, where using T E {0.001, 0.01, 0.05} consistently led to degraded performance.
Large Temperature. Conversely, large T (e.g., T 2 5) causes the sigmoid linearly, with: (x) z 2 + 17 for small |x/T|/ This weakens the ranking effect and makes the method rely more on raw dissimilarity magnitudes, which is suboptimal in lightweight settings (e.g.. when fi & f2 in capacity). We observed performance drops in both CIFAR100 and CUB200 in this case.
behave nearly
Summary. In practice, intermediate values {0.1, 0.5) yield the best results. This suggests that further temperature ure tuning might be useful to have optimal results.
Appendix F
Computational Complexity Comparison
In this section, we compare the computational complexity of different teacher-student pairs in the experi- ments. As required in practical deployment scenarios, all student models are strictly smaller and computa- tionally cheaper than their corresponding teacher models, with details reports in Table g. These results are obtained using the python package ptflops (Sovrasov 2-22-4)
Table 8: Computational complexity comparison (in MACs) of teacher-student pairs. MAC denotes multiply- accunulate operations, where MAC x 2 FLOPs (one multiplication and one addition).
CIFARLO
SVIN
CIFARIO0 (RoNNer 50 - MolcleNesV2) CIFARIO0 ResNet-32e - StrfleNer CIFARLOO (ReeNet 32xl StrafleNNet3)
Toscher sudle
557.2 22 MEIAC
GMAC
1.31 GMAC
LOGGMAC
LIOGMAC
Student model
623.1 KAMA
47.26 MMAC
7.35 MSAAC
MMAC!
46.06 MMAAC
35


Reviews:
[{"review_id": "RKfaqpUgTK", "reviewer": "Reviewer_ST3p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new _perception-coherent_ feature transfer objective for knowledge distillation, based on aligning the rank order of pairwise similarities between teacher and student embeddings. The key idea is that while absolute feature magnitudes and layer geometries may differ across architectures, the teacher’s internal perception structure can still be transferred through a rank-based loss. The method is label-free (the student receives no ground-truth supervision) and is applicable even when teacher and student differ significantly in capacity, architecture, or embedding dimensionality.\n\nThe authors demonstrate two main empirical results: (1) strong feature transfer performance in representation retrieval tasks on CIFAR-10 and a subset of CUB-200, where the proposed loss outperforms KD, FitNet, PKT, and other distillation baselines; and (2) competitive or better classification results compared to standard distillation methods (KD, FitNet, VID, PKT, and CRD) under heterogeneous teacher-student settings on CIFAR-100, such as ResNet-50 → MobileNetV2 and ResNet-32×4 → ShuffleNet.\n\n**Key Strengths:**\n\n-   Clear and well-motivated formulation based on rank consistency.\n    \n-   Unsupervised student training setup (no labels) while still leveraging teacher knowledge.\n    \n-   Demonstrates flexibility under cross-architecture and dimension-mismatched distillation settings.\n    \n-   Strong gains over several established KD baselines in both feature retrieval and classification.\n    \n-   Ablations on batch size, temperature, and student capacity improve transparency.\n    \n\n**Key Weaknesses (high-level):**\n\n-   Representation experiments use a very small and non-standard student model, limiting generality.\n    \n-   The retrieval setup aligns closely with the proposed loss, making it unclear how broadly the representational gains extend.\n    \n-   No controlled same-architecture KD setup (e.g., ResNet-50 → ResNet-18).\n    \n-   No compression ratio reporting, despite KD’s strong connection to model compression.\n    \n-   All experiments are on small datasets (CIFAR, CUB subset), raising scalability questions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper claims that its rank-based loss can transfer the teacher’s internal perception structure to a student without using labels, that this works even when the teacher and student differ in architecture and feature dimensionality, and that this leads to improved representation transfer and competitive knowledge distillation performance. These claims are supported by the experiments. The retrieval results on CIFAR-10 and a CUB-200 subset show clear improvements over KD, FitNet, PKT, and HKD under identical teacher–student setups. The CIFAR-100 classification results demonstrate that the method matches or exceeds established KD methods, including CRD, across heterogeneous teacher–student pairs such as ResNet-50 to MobileNetV2 and ResNet-32×4 to ShuffleNet. The ablations on batch size, temperature, and student capacity further support the proposed design. While the evidence is convincing within this scope, several aspects of the evaluation could be strengthened, and I outline these in the requested changes section.", "audience_yn": "Yes", "audience_explanation": "Researchers working on knowledge distillation, model compression, and feature transfer would find this paper relevant, especially given its cross-architecture applicability, and improved results over established KD baselines.", "requested_changes": "The following experiments are requested.\n\n1.  **Controlled KD setup:**  \n    While the heterogeneous teacher–student pairs are interesting, a standard homogeneous setting (e.g., ResNet-50 → ResNet-18) would serve as a clean baseline to isolate the effect of the proposed loss from architectural mismatch. These should also be compared to standard distillation methods (KD, and CRD).   \n    \n2.  **Representation claim clarity:**  \n    The retrieval experiments show strong improvements over KD-style baselines, but the setup is non-standard and tightly aligned with the proposed ranking loss. If the authors intend to make claims about representation learning more broadly, a comparison to at least one SSL baseline (e.g., DINO) or a cross-dataset transfer probe would strengthen the argument. Otherwise the claim should be framed explicitly within the KD context.\n    \n3.  **Compression ratio reporting:**  \n    Given that KD is fundamentally a model compression technique, it would be helpful to report teacher vs. student parameter counts or FLOP ratios, so readers can better interpret the practical efficiency gains. Since CRD is one of the main KD baselines and is known to report such compression metrics, including similar comparisons here would make the results easier to contextualize.", "broader_impact_concerns": "I did not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MEUwn7LI2H", "reviewer": "Reviewer_Uj7N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces “perception coherence,” a probabilistic notion of agreement between a teacher’s and a student’s feature spaces. Instead of matching feature magnitudes or full geometries, the student is trained to preserve the teacher’s **relative dissimilarity rankings** around each reference point. The authors formalise this via cumulative functions and define a pointwise and global coherence level that measures how close these distributions are across inputs. They propose a differentiable, mini-batch loss that approximates ranking with a soft pairwise sigmoid and minimises the squared distance between teacher and student rank vectors (Eq. (3)–(4)). Theoretical results include an (O(B^{-1/2})) convergence rate for the mini-batch estimator of their discrepancy measure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The theory supports the estimator properties and gives qualitative guarantees about rank preservation at both local and global levels. The proofs and derivations match the claims. The experiments cover both retrieval and classification, compare with strong baselines, and include ablations that align with the theoretical signals.", "audience_yn": "Yes", "audience_explanation": "Representation transfer and knowledge distillation for small models are active topics. A probabilistic, rank-based objective that handles different feature dimensions is of clear interest to readers who work on compression, retrieval, and cross-architecture transfer. The paper is relevant to practical settings where labels are not available for the transfer set.", "requested_changes": "First, add a task-level connection between coherence and error. Even a stylised result (for example, a 1-NN retrieval or margin-based classification bound under separability) would make the metric operational for downstream evaluation.\n\nSecond, discuss time and memory complexity. The loss touches many pairwise comparisons per reference.\n\nThird, give practical guidance for the soft-ranking temperature. The ablation shows a narrow good range; a simple data-driven rule (for example, set temperature to a constant multiple of the median absolute deviation of within-batch dissimilarities) would help adoption.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pKraY67K5C", "reviewer": "Reviewer_N8ii", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new knowledge distillation (KD) method for transferring feature representations from a large teacher model to a smaller student model. The main contributions are as follows:\n\n1. Introduction of “Perception Coherence”: The authors present a novel framework emphasizing consistency in how teacher and student models perceive data. Rather than forcing the student to replicate the teacher’s exact geometry, the method aligns their relative rankings of pairwise dissimilarities across data points, offering a more flexible and realistic goal for low-capacity students.\n\n2. Differentiable Loss Design: The paper introduces a differentiable ranking loss based on soft sigmoid approximations, enabling gradient-based optimization over inherently non-differentiable ranking operations.\n\n3. Theoretical and Empirical Validation: The authors provide theoretical guarantees for convergence and demonstrate the relationship between perception coherence and order-preserving probability. Experiments on toy data, image retrieval, and classification tasks show consistent improvements over prior KD methods such as PKT and CRD.\n\n\n__Strengths__\n\n- Novel and Intuitive Concept: Shifting from absolute distance alignment to relative ranking preservation is both practical and well-motivated, especially for lightweight models with limited representational capacity.\n\n- Strong Theoretical Foundation: The paper rigorously analyzes the mathematical properties of the proposed framework, linking theory to empirical behavior and increasing the credibility of the approach.\n\n\n__Weaknesses__\n\n- Computational Cost for Lightweight Models:\nThe loss computation involves all pairwise dissimilarities within a mini-batch, resulting in \n $O(B^2)$ complexity. For lightweight student models, this can dominate training cost, potentially becoming the main bottleneck. Unlike contrastive learning—where large models and distributed setups justify this cost—here the mismatch between method complexity and target application (efficient model training) warrants deeper discussion.\n\n- Missing Comparisons with Recent Work:\nThe experimental section lacks comparisons with post-2020 KD methods such as ReviewKD (CVPR 2021) and SimKD (CVPR 2022). These methods introduce high-order or classifier-reused relational structures, conceptually close to the proposed idea. Including at least ReviewKD or SimKD in classification benchmarks would better situate this work in the current literature.\n\n__Reference__\n- [ReviewKD] https://arxiv.org/pdf/2104.09044\n- [SimKD] https://arxiv.org/abs/2203.14001", "claims_and_evidence_yn": "No", "claims_explanation": "The main claim that preserving relative perceptual order enables effective representation transfer is theoretically sound, but the empirical support could be stronger.\n\n- Theoretical Evidence: Section 4 provides clear guarantees, e.g., the estimation error of perception coherence decreases with the square root of batch size (Theorem 4.1), and higher coherence correlates with better order preservation (Theorems 4.2–4.4).\n- Empirical Evidence: Figures 4–5 effectively visualize structural transfer on toy data. Tables 1–2 show clear mAP improvements over PKT and HKD, confirming stronger feature learning. The only gap is the absence of comparisons with newer baselines in classification tasks.\n\n__Main Concern__\nOne concern is the lack of comparison with methods developed after 2020. The authors’ empirical evidence is limited to comparisons with studies published before 2020 and does not sufficiently capture the progress made in the field over the past five years.", "audience_yn": "Yes", "audience_explanation": "The paper’s theoretical framework and practical method for knowledge distillation would interest readers in TMLR who focus on representation learning, model compression, and efficient deep learning.", "requested_changes": "While the contribution of this paper theoretically sound, the empirical validation and comparison with recent related work are somewhat limited. The following revisions would substantially strengthen the paper.\n\n__Clarify the role of the hyperparameter τ (temperature) in soft ranking:__\nThe appendix currently includes a discussion on τ, but a concise summary should also appear in the main text. In particular, please explain why different τ values are used for the teacher and student models—e.g., whether this relates to differences in feature variance or prediction confidence. This clarification would help readers understand the method more deeply and facilitate reproducibility.\n\n__Expand comparative experiments with recent related work:__\nAs noted in the Weaknesses section, please consider adding experimental comparisons in the classification task with more recent KD methods, at least ReviewKD (CVPR 2021) and SimKD (CVPR 2022). This would allow the proposed method’s advantages to be validated in a contemporary context and further enhance the impact of the paper.\nFor reference, VRM: Knowledge Distillation via Virtual Relation Matching (ICCV 2025, the first version (v1) of the preprint was uploaded at the end of February 2025, and the third version (v3) was uploaded at the end of July.)  provides a comprehensive overview of comparable relational KD approaches and may serve as a useful benchmark.\n\n__Minor Comment:__\nSome experimental plots are not properly embedded and are difficult to read (e.g., Fig. 7).\n\n__Reference__\n- [ReviewKD] https://arxiv.org/abs/2104.09044\n- [SimKD] https://arxiv.org/abs/2203.14001\n- [VRM] https://arxiv.org/abs/2502.20760", "broader_impact_concerns": "The paper addresses model compression and efficiency rather than application-level AI behavior. No ethical concerns are apparent. On the contrary, improving lightweight model training can reduce computational demands and broaden AI accessibility on edge devices.", "additional_comments": "This is a well-written and theoretically solid paper introducing a creative and meaningful idea. With stronger empirical grounding through up-to-date comparisons and clearer discussion of computational trade-offs, the paper could make a significant contribution to the KD literature.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)