Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Authors: Jieting Wang
ZhangZelong
Feijiang Li
Yuhua Qian
Xinyan Liang
Venue: ICML 2025 poster (2025)
Keywords: Loss function
Random consistency
Generalization bound
Similarty matrix
Primary area: general_machine_learning->supervised_learning
TL;DR: This paper proposes a novel loss function using Euclidean distance between similarity and adjacency matrices, proven unbiased with strong generalization. Boosts accuracy, $F_1$-score, and class separation.

Abstract: Deep learning excels at capturing complex data representations, yet quantifying the discriminative quality of these representations remains challenging. While unsupervised metrics often assess pairwise sample similarity, classification tasks fundamentally require class-level discrimination. To bridge this gap, we propose a novel loss function that evaluates representation discriminability via the Euclidean distance between the learned similarity matrix and the true class adjacency matrix.
We identify random consistency—an inherent bias in Euclidean distance metrics—as a key obstacle to reliable evaluation, 
affecting 
both fairness and discrimination. To address this, we derive the expected Euclidean distance under uniformly distributed label permutations and introduce its closed-form solution, the Pure Square Euclidean Distance (PSED), which provably eliminates random consistency. Theoretically, we demonstrate that PSED satisfies heterogeneity and unbiasedness guarantees, and establish its generalization bound via the exponential Orlicz norm, confirming its statistical learnability.
Empirically, our method surpasses conventional loss functions across multiple benchmarks, achieving significant improvements in accuracy, $F_1$ score, and class-structure differentiation. (Code is published in https://github.com/FeijiangLi/ICML2025-PSED)

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Jieting Wang' Zelong Zhang'
Feijiang Li'
Yuhua Qian
Xinyan Liang'
Abstract
1. Introduction
The representation power (Goodfellow al., 2016; Ghande- harioun et al., 2024) of deep learning refers to its ability automatically learn and extract features from data through multi-layer neural networks, eliminating the need for man- ually designed features. Owing to this powerful capability, deep learning has been widely applied across various fields, including machine vision (Chen et al., 2020; Kondratyuk et al., 2024; Dosovitskiy et al., 2021), time-series signal analysis (Xu et al., 2022; Bian al., 2024; Crabbé et al., 2024), and many others.
Deep learning excels capturing complex data representations, yet quantifying the discrimi- native quality of these representations remains challenging. While unsupervised metrics often assess pairwise sample similarity, classifica- tion tasks fundamentally require class-level discrimination. To bridge this gap, we propose a novel loss function that evaluates representation discriminability via the Euclidean distance between the learned similarity matrix and the true class udjacency matrix. We identify random consistency-an inherent bias in Euclidean distance metrics-as key obstacle to reliable evaluation, affecting both fairness and discrimi- nation. To address this, we derive the expected Euclidean distance under uniformly distributed label permutations and introduce its closed-form solution, the Pure Square Euclidean Distance (PSED). which provably eliminates random consistency. Theoretically, we demonstrate that PSED satisfies heterogeneity and unbiasedness guarantees, and establish its generalization bound via the exponential Orlicz norm, confirming its statistical learnailliy.. Empirically, method surpasses conventional loss functions across multiple benchmarks, achieving significant improvements in accuracy, Fi score, and class-structure diffcrentiation. (Code is published in htts://giinbbccmm//jjaaaaiiiCCLt PSED)
The strategies for enhancing network representation ability can be grouped into four categories. First, structural opti- mization improves feature extraction through multi-scale learning (He et al., 2016; Lin al., 2017) or self-attention mechanisms (Vaswani et al., 2017). Second, data enhance- ment boosts model robustness through data augmentation and generative modeling, improving generalization (Good- fellow et al., 2014). Third, training process optimization prevents overfitting and enhances stability via multi-task learning and regularization (Srivastava et al., 2014; loffe Szegedy, 2015; Ng. 2004). Finally, loss function optimiza- tion aims to guide effective learning by designing suitable loss functions (Rangapuram al., 2018). Intuitively, evalu- ating the quality of sample similarity at the representation layer in a loss function can be an effective approach.
Recently, an unsupervised measure, din (K), was pro- posed to quantify the informativeness of similarity matri- ces (Brockmeier et al., 2017). computes the distance between a similarity matrix K and set of non-informative matrices. Let N. be the of non informative matrices N₂ {(1 a)l + aJ, 0 S S 1}. is the identity matrix, J is the full one matrix. The similarity matrix described by N₂ represent scenarios where different samples exhibit uniform similarity. The measure dinfor (K) is defined as:
'Institute of Big Data Science and Industry, Key Laboratory of Evolutionary Science Intelligence of Shanxi Province, Shanxi University, Taiyuan, China. Correspondence to: Yuhua Qian <jinchengqyh@e @ 126.com>.
din,for (K)
min K -Nall USa51
(I)
1
K
n 2K +1),
n
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
where K = 11 n is the number of sample, and
1|3 the Frobenius norm (the square root of the sum of

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
squared elements of the matrix). This measure effectively captures the information embedded in K and can guide K to assign different similarity values to each pair of samples. However, the underlying assumption in classification tasks is that samples from the same class should exhibit higher similarity compared to those from different classes, The pairwise-based evaluation orr (K) overlooks the broader class-level distinctions necessary for effective classification.
similarity matrix selection, ensuring more reliable represen- tations of hidden layers. Furthermore, we provide learning bound for PSED based on statistical norm, offering theoret- ical guarantees on the method's s generalization performance. In summary, the main contributions are as follows:
A loss function for measuring the ability of the repre- sentation layer is proposed, and explicit solution for the loss function in the version of eliminating random consistency is given.
To evaluate the discriminative ability of the classification model, an intuitive measure and novel loss function is the Square Euclidean Distance (SED), which compares K to the true adjacency matrix YYT,
Through theoretical analysis, the advantages of this metric in heterogeneity and unbiasedness have been demonstrated, and a generalization bound has been provided for the generalization performance of the loss function in fully connected layer network structures.
dSED ( JK- I
where Y is the one-hot encoding of the true label Y, Y E {0, 1 is the one-hot encoding of the true label vector, k is the number of classes. However, SED is biased toward certain -on-informative matrices, restricting its capacity to establish meaningful similarity relationships.
A fully connected network classification model based on this loss function was proposed, and the effective- ness of the algorithm was verified through extensive experiments.
Consistency metrics measure the agreement between two random variables, while random consistency(RC) refers to spurious agreement arising purely from randomness ( (Wang et al., 2023). A canonical manifestation of RC occurs when examinees achieve measurable test scores solely via random response patterns. The mechanisms by which RC harms the learning process include evaluation distortion, optimization misguidance, and generalization barriers. Failure to deduct the RC baseline may lead to overestimating the model's actual consistency performance (e.g., an original consis- tency score of 0.6 vs. a random baseline of 0.2 means the true effective consistency should be 0.4). When loss functions include RC without proper correction, they can induce optimization bias, causing algorithms spuriously improve consistency metries by overfitting to noise (Wang et al., 2020a) or data bias (Li et al., 2024; Vinh et al., 2010) instead of learning genuine data patterns. These would con- sequently impair the model's generalization ability. The Pure Consistency Measure (PCM) framework (Wang et al., 2020a;b) addresses RC in metrics like accuracy (Wang et al., 2023) and the Gini index (Wang et al., 2024), mitigating decision and multi-value bias. In clustering, mitigating RC reduces cluster number bias (Vinh et al., 2010), and in causal learning. PHSIC (Li et al., 2024) reduces bias related to dimensionality and sample size.
The proofs and some experiment results are in Appendix.
2. Related Work
The main contents involved are loss function and general- ization bound, and we will review these two aspects.
2.1. Loss Function in Deep Learning
Loss functions in deep learning measure the discrepancy between model predictions and actual values, guiding the optimization of model parameters. Common loss functions include Mean Squared Error (MSE) for regression tasks (Le- Cun et al., 2015), Cross-Entropy for classification (Hin- ton et al., 2012), Hinge loss for binary classification with SVMs (Cortes & Vapnik, 1995), Huber loss combining MSE and absolute error for robust regression (Huber, 1964), Bullback-Leibler Divergence for comparing probability dis- tributions in generative models (Kingma & Welling, 2014), and Contrastive loss for evaluating sample similarity in met- ric learning tasks like face verification (Chopra et al., 2005). Selecting the appropriate loss function is crucial, and cus- tom ones may be necessary for specific tasks. In this paper, we propose a metric to measure the quality of similarity matrices as a loss function to guide deep learning.
To address RC in SED, we propose a novel Pure Square Eu- clidean Distance (PSED) under the Pure Consistency Mea- sure framework. PSED refines SED by incorporating the expected distances of adjacency matrices generated through label permutations as baseline. This measure can address the shortcomings of for (K.) and dsED (K).
2.2. Learn ability
The generalization error represents the gap between the training error and the test error, with this bound captur- ing the factors influencing the test error. Existing tradi- tional theories based on VC dimension and Rademacher complexity are insufficient to explain the performance of
Theoretical analysis of our approach highlights two main advantages: improved heterogeneity and unbiasedness in
2

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
deep learning (Vapnik & Chervonenkis, 1971; Bartlett & Mendelson, 2002). While numerous norm-based bounds have been proposed (Neyshabur et al., 2015; 2018; Bartlett et al., 2017; Golowich et al., 2018; Arora et al., 2018), we choose an exponential Orlicz norm-based concentration in- equality (Vershynin, 2018). This choice is motivated by the fact that this norm characterizes the concentration be- havior of the network parameters, rather than merely the range of parameter values considered by traditional norms. Furthermore, exponential Orlicz norm-based inequalities encompass traditional norm-based inequalities, as for vari- ables with bounded values, their exponential Orlicz norms must also be bounded.
According to the definition of PSED, Ey (dSED (K, F") requires the computation of all possible cases that follow the same distribution as the true label Y, involving to- n. tal of my mmy.mmml-mmml terms. As a result, its computational complexity is relatively high. To improve computational efficiency, an analytical solution for (dSED Y")) has been proposed:
Theorem 3.2. Ler be the set ofll r-tuples drawn without
replacement from the set (1.... ,.). The analytic solution of the expectation of Ex (dsED Y')) is:
Ey (dSED (K,Y")
(6)
k
m
= |
+
m,
2
2
K
Kij
3. Definition and Analytic Solution
Given a hypothesis function space F, the task of classifi- cation is to learn a function h(X) E F that maps from the feature space X E X' S Rd to the discrete label space Y E Vr To measure the representation ability of the fune- tion, we seek the Euclidean distance between the adjacency matrix of the true labels and the similarity matrix calculated by l()). We provide an analytical solution as follows:
where
|. denotes the size set.
set.
Based on Formula 29, Formula 5 and Theorem 38, the analytic solution of PSED is:
dPSED (K)
a
k Σ r=1
Σ
K
Σ (.Jiig}
K3j
1 mm
Kr [+]1)] 1m.)
dSED (K, Y)
||K --Y-
1
k
& K
where Kij is the value of the i-th row and j-th column of matrix K. From the analytical expression, it is evident that the smaller the value of the expression, the closer the matrix K is to K Krjr This shows that in this case, the structure of K is closer to the class structure.
= K
m
2
T m4
where m, is single column all vectors of length m. m. is the number of objects of i class and Kj]j is the sub kernel matrix of objects in class i.
Next, we provide an analytical solution for PSED with computational complexity of O(kn² + (1 k)yn + Σ m2). Compared to the computational complexity of nf my Img! m-. x (2kn? + 2n²) in Formula 5, the analytical solution significantly accelerates the computation speed of the expectation of PSED, ensuring that PSED can serve as efficient computational objective function.
Since dsED is consistency measure, previous work (Wang et al., 2020a;b) has shown that random consistency exists in consistency measures. To mitigate random consistency in Formula 28, we adopt the pure consistency framework.
For two random variables 71, Zz, the framework of pure consistency measure (PCM) refers to eliminate random con- sistency from consistency measure (Wang al., 2020a;b):
4. Properties Analysis
Z2) CM(Z,, Z2) RCM Z2), (4)
In this section, we mainly analyze the advantages of PSED compared to dinfor and SED.
where CMM 72) represents the degree of consistency between random variables Z1 and Ze and RCM (Z1, Z2)
Property 1. (Homogeneity of dinfor) Suppose there are n elements, let K and K' be two square matrices that are generated by arranging the n² elements in different ways. Then we have for (K) donfor
represents the degree of consistency generated by chance.
Then we provide the definition of Pure Square Fuclidean Distance (PSED) in the framework of random consistency: Definition 3.1. The PSED is defined as:
Property 2. (Heterogeneity of dpSED) Suppose there are n² - n elements, let the diagonal positions of K and K' he I and their other positions are assigned by the n² n elements in different ways. Then if dSED S (SED(K'), we have that drsED (K) < dPsED (K').
dPSED K)- = dsED K. Y Ey (dsED Y))
(5)
= |K- Y I Ey [| Y'Y 12))
where Y' denotes the one-hot encoded label matrix gener- ated by the permutation of the true label vector Y and Ey the expectation over the uniform distribution of Y".
Properties and 2 can be easily derived from the definition of dinnoor and Theorem 38, respectively. These properties
3

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
camm 0.45
larity matrices.
0:38
0:38
0.45
0.45
0.67
We also verify the above properties in Figure 2 with provid- ing some Gaussian kernel matrices with different parame- ters. Figure 2 depicts SED and PSED values between the target matrix (the first one) and the kernel matrices. From Figure 2(a), we observe that SED bias to the identity ma- trix. From Figure 2(b), both the second diagonal and the last full one matrix obtain the highest score. This signifies that PSED is unbias to any informative matrix. The above advantages ensure that PSED is more appropriate to measure the quality of similarity matrix.
(a) The same for value for different adjacency matrices
PRED
-1.04
0.40
025
(b) The different PSED value for different adjacency matrices
5. Learn ability of dESED Loss
Figure 1. Comparison with fin
In machine learning, the underlying probability distribution of X y is usually unknown. Only a collection of empirical data S, (1), ..., (I }n)} } is available. Based on these empirical data, the dESED is estimated by:
10
0.64
0.36
O 600011623
0.031823
0.316603
3.1823
31 1298
Grussion borddyicch
(a) The bias of SED in the imbalanced scene
åPsED(K) =
Σ i=1
n
k
K,j
ij
(8)
i.jج j
PSED
00 0.01
0.20
07
00.1
k
1m,
K
[ri[i Im,
80001
0.331623
0.116P3
3.1673
Gaussian bandwicth
(b) The unbias of PSED in the imbalanced scene
where K.y = Ker(h(æ;;) h(x,)) is a kernel function, h e H. is the hypothesis function that outputs the embedding representation vector, K Kriri the sub kernel matrix of class and m, is the number of objects of class.
Figure Comparison with SED.
demonstrate that, compared ferfor, PSED can effectively distinguish matrices with different internal structures.
By minimizing the empirical dYSED, a currently optimal classifier can be obtained. The generalization ability of this classifier can be characterized by the quality of the convergence of empirical loss to the true one. Due to the randomness of samples, the convergence is analyzed in terms of probability. Formally, let e >> 0, the convergence analysis aims to find upper bound d(e) on the probability of deviation inequalities:
We also verify the above properties in Figure 1 by provid- ing three sets of adjacency matrices. Each set consists of matrices with identical proportions of zeros and ones but differing their internal structural arrangements, as shown by ((3, 4, 5), {6, 7,8). {9, 10, 11)). The identical forfor value for each set indicates that dorfor cannot distinguish matrices with different internal structures. In contrast, from Figure 1(b), the 4, 6, 8, 10 matrices are closer to the first true adjacency matrix and exhibit lower PSED values. This phe- nomenon demonstrates the discrimination ability of PSED. Property 3. (Bias of dsED) For the matrices I and J, when n²+n, we have d[) dJED(J); other- wise, (I) < dJ)
P(IdrSED(K) - -ee00-drsseeie (K) 2) $ (()),
(9)
where K,; = Ex,.x, (or()((,))) is the expecta- tion of the kernel function.
m I
2
The probability upper bound quantifies how quickly and accurately empirical measure approaches the true mea- sure the sample size increases. Additionally, it provides insights into how the model structure and complexity affect the convergence performance. To establish the general- ization ability bound, we employ the exponential Orlicz norm-based concentration inequality.
Property 4. (Unbias of dpSED) For any matrix in No = {(1 - a)I + aJ, 0 S a a 1]. where I the identity matrix and the full one matrix. We have (ASED(A) O.
From the above two properties, we conclude that SED is bi- ased towards the non- -onioformative matrices and J, whereas PSED assigns the same score to the non-informative simi-

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
5.1. Exponential Orlicz Norm-based Concentration Inequality
Then with probability least 2(0 + 8), where 8 > 0. we have,
Concentration inequalities (Boucheron et al., 2013) provide bounds the probability that random variable deviates from its mean or median. These inequalities are powerful for understanding the behavior of random processes, particu- larly in machine learning, where they help analyze a model's generalization ability and stability. Traditional inequalities, such as Markov's 's inequality, often yield loose bounds. By utilizing the bounded difference property, tighter inequali- ties, such as McDiarmid's Hoeffding's 's inequalities, can be derived. Recently, an inequality based on the exponential Orlicz norm has been proposed (Escande, 2024).
If(X xn) - [f (XI,..,, E B8
2 nlog ())
4n|lblle. vD + elbbler
+ log
Theorem 5.1 states that if one sample among the objects is modified, the change in the function value over all objects is bounded by the magnitude of the change in the individual sample. Consequently, the deviation between the function value and its expectation can be characterized by the exponential Orlicz norm of the change magnitude.
5.1.1. EXPONENTIAL ORLICZ NORM
5.2. Network Structure
For q 2 1, the g-exponential Orlicz norm of a random variable X on the probability space (X, p)) is defined as:
This paper considers fully connected layer networks to ob- tain representation vectors. Given I. weight matrices W (W1, 1,.., W&) v) and L activation functions 10,,..,, where o : R - Rdd and d; is the output dimension of i-th layer. The fully connected network hw,c is:
| v e>0
(E (exp ([XX = 2).
(10)
When g = 1 and q 2. the norm are corresponds to sub-exponential and exponential Orlicz norms, respectively. When X E IR a random vector, its bhy norm is defined by [X||,,,, = suprese-: [|(X,v) lly, where gd-1 is the unit ball in Rdd space. Next, we list three properties:
-. hw.r o r. 1. (W 1-11- -M o (W W)))).,
(14)
where o is the nonlinear activation function. The common used activation functions, coordinate-wel wise ReLU and sig- moid function, are p:-Lipschitz continuous (Bartlett et al., 2017). The p. p--Lipschitz continuous requires that for all z' in its domain, the following inequality holds:
Property 5. Let X and Y be random variables, we have,
||X + 1l I 1l + l1
(11)
|[o,(z) a3((')ll I p: 2'Ilp-
(15)
Property 6. Let X,, = 1,.., L be random variables, we have,
where || ||p is the p-norm. This ensures that the function does not change too rapidly, with p serving as the Lipschitz constant that bounds the growth.
II X;
<
(12)
11
With the Lipschitz continuity, the output variation of a fully connected network can be bounded by the sample perturba- tion. For two observations X and x their output satisfies:
Property Let X E Rd be random vector, we have,
|||| ||lle,, S Vddjxllv.
(13)
[Iw,,(x))) - hw: & |Ix - xxllp II Ps W
5.1.2. CONCENTRATION INEQUALITY
(16)
The inequality based on |X offers sharper bounds and provides generalization performance bound of order O(1/n), where n is the sample size.
The reason that based on the Lipschitz continuity, for two observations xi and x'c, we have:
Theorem 5.1. (Escande, 2024) Let f: X + R and B E x such that p = P(X" $ $) S 3/4. For any two samples with only one different observa- tion: Sn
(l/,,(xx)) hw. (xx))xxx)))l
(17)
= (l........ (W
(18)
S P: (Wchhw:-- - W,lww. (x(I)l
(19)
(.0,,-.,, 1k-1, Fk; ..., a..) and Sn, {ø1,... .... 1k-1, 1, D'y.... Dn)) assume there exist a pseudo met-
S P: [allWllel//www.-((
(x)))
(20)
ric b: X x X + Rt with [|blly, < 30 such that:
where the last inequality is based on the Cauchy Schwartz in- cquality. Further, by successive application of this property, we can obtain Eq. (16).
(s(S.) f(S,,k) < b(x,,x()-
5

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
5.3. Generalization Bound for deSED Loss
Prejentation Coper
Clowssfoottie loyer
Based Theorem 5.1, our bound is:
Spaerr
Theorem 5.2. When the hypothesis function is the fully con- nected layer networks and K. is the RBF kernel K (Z), exp - (xx }² ). Let B E X such that p IP & 2) < 4, for 5 > 0, Then with probability at least 1 1- 2(p + 6). we have,
Figure 3. The framework of the proposed method fully con- nected networks.
(PSSE(()) - desen(K)) E
EBI
measures. This generalized formulation maintains theo- retical rigor while enabling practical applications across multiple deep learning architectures.
< 11bblea 4/D+ + (2/) (2 (v/7 log
1
+
log
8)
Presentation layer (PL)
where,
The model consists of three layers, each containing a fully connected layer, a ReLU activation layer, and a Dropout regularization layer. The fully connected layer performs linear transformation of the previous layer's activations through the weight matrix W, producing new feature rep- resentations. The ReLU activation introduces nonlinearity, enhancing the model's ability to capture complex patterns. The Dropout layer randomly drops neurons to prevent over- fitting. Subsequently, the similarity matrix is computed by taking the inner product of the hidden layer activations. Fi- nally, the processed features are optimized using the PSED loss function, with backpropagation applied to minimize the prediction error. The relevant formula is as follows:
[1blle, =(2+ + 40 - 1) + 4 max
(21)
MyM (.M(Vlldiam))))) P2 Il Pival Will
C -1=
and M = max |x||-
From Theorem 5.2, we can conclude that, for the fully con- nected network, the generalization bound is related to the following terms: [diom(x)] bb, is the exponential Orlicz norms of the input domain diameter; dj is the number of nodes in each layer of the network, and d is the input di- mensional; Pt is the Lipschitz constant of 1-th activation function; WWll,,, Wellez is the exponential Orliez norms of 1-th weight vector; is the number of training instances. From Theorem 5.2, the smaller the exponential Orlicz norms of the input domain diameter and the network parameter vector, the fewer the network nodes and the more the samples, the smaller the model's generalization error.
Z' WI-1Z'-1
(22)
z ReLU(Z)) = max(0, O)
(23)
Z' Dropout(Z'),
(24)
K (Z²)²
(25)
where 1-1 represent the parameters of the 1)th fully connected layer.
6. Methodology
Classification layer (CL)
This paper presents a deep learning framework designed to learn discriminative feature representations through novel loss function that serves as a universal similarity matrix qual- ity measure, applicable across diverse learning paradigms including deep network training, metric learning, and ker- nel methods. To demonstrate its versatility, we implement the approach on three fundamental architectures: fully con- nected networks (whose schematic diagram shown in Figure 3), Vision Transformers (ViT), and Convolutional Neural Networks (CNN), with experimental details for ViT and CNN provided in the Appendix.
The prediction labels Ý are generated through additional fully connected layers:
ÿ W Y = softmar(( (
(26)
To quantify the discrepancy between the predicted probabil- ity distribution and the true distribution, the cross- (CE) loss function is used. The CE is defined as:
k Yis log Y.g.
CE -
(27)
The framework operates by first transforming input data into latent embeddings, computing pairwise similarity matrices in the feature space, then optimizing network parameters through backpropagation using our proposed debiased dis- tance metric that effectively evaluates representation qual- ity while overcoming limitations of conventional similarity
where n is the number of samples, k the number of classes, Yi is the predicted probability that sample i belongs to class j and Y.j is the true label. The loss value is minimized using the back propagation algorithm, which optimizes the parameters of the final fully connected layer.
6

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
7. Experiment
In this section, we compare our proposed method with three common loss functions on 20 benchmark datasets and 5 image datasets. And we compare the methods based on CE, SED, 1 (-dinfor(K), and PSED loss functions the presen- tation layer. Additionally, we conduct analysis experiments to further demonstrate the advantages of our method.
(a) Accuracy (Benchmark)
(b) F-measure (Benchmark)
7.1. Experimental on Benchmark Dataset
(c) Accuracy (Image)
(d) F-measure (Image)
7.1.1. PERFORMANCE ANALYSIS
Figure 4. CD diagrams w.r.t Accuracy and F- F-measure.
In this section, report the average accuracy and F- measure of four methods on benchmark datasets with 10 partitions and model layers. Table 1 shows the results, with each row representing a dataset and columns divided by evaluation metric. The highest value in each section is bolded. If our method significantly outperforms the others, black dot will be placed next the method (for significance testing, please refer to the Appendix). As indicated by the table, the PSED-bard based loss function demonstrates superior performance in terms of average convergence accuracy and F-measure, with values surpassing those of other methods on most datasets.
(a) model layers:3
(b) model layers:5
(c) model layers:8
Figure 5. Significance test w.r.t. Accuracy and F-measure.
7.1.2. SIGNIFICANCE TEST
To demonstrate the superiority of CE-PSED, we first duct Friedman test to confirm significant differences among the methods, followed by a Nemenyi post-hoc test to identify specific pairs with differences (details for layers 5 and 8 are in the Appendix). Figure shows the CD diagram for 3-layer models, where the x-axis represents the average rank and the CD line indicates the critical difference from the Nemenyi test. Methods marked with red star indicate the best performance, while those not connected by red line show significant performance differences. As shown in Figure 4, the CE-PSED- P-a----DDe-dddd method has a significantly lower average rank. Not only does CE-PSED achieve the best performance, indicated by the red star, but is also not connected by a red line to any other method. This indicates that its accuracy and F-measures are superior to those of other algorithms across multiple datasets.
7.1.3, CONVERGENCE ANALYSIS
The Figure 6 (a) and (b) shows the performance of the four methods over the training epoch in the benchmark datasets with the highest and lowest degree of imbalance (see ap- pendix for other dataset results), where the points on each line represent the average accuracy of the corresponding period. The results show that the CE-PSED method ex- hibits significant performance advantages in most datasets and training epochs, and can quickly converge, which fully demonstrates its robustness and effectiveness on datasets.
7.1.4. NETWORK LAYER ANALYSIS
To verify the effectiveness of the CE-PSED method at dif- ferent network layers, this study set the model layers and 8, respectively, and selected one dataset for presentation (see Appendix for other datasets). Tables 3 presents the F- F-mmasure values of four methods at different levels. The results show that as the number of network layers increased, the F-measure values of other methods significantly de- creased, while the F-measure value of the CE-PSED method decrease less and remain the highest. This indicates that other models experience feature representation collapse as the number of layers increases, that is, the model tends to classify samples into the same category, while the CE-PSED method performs well at different layers, effectively avoid- ing feature collapse and ensuring that the model maintains good feature learning and classification capabilities.
Additionally, further significance test was conducted to val- idate the enhanced performance of CE-PSED, with method- ological details available in (Wang et al., 2023; Li et al., 2019). Figure presents the significance test results for all datasets, where each bar chart illustrates the difference between the number of times the algorithm's significance wins and losses. As shown in Figure 5, the PSED-basee method significantly outperforms the other methods.
7

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Table 1. Accuracy and F-measure based on different loss functions on the henchmark datasets when model layers is 3.
Data
Accuracy
F-measure
CE-SED
CE-inform
CE-PSED
CE-SED
CE-inform
CE-PSED
1
0.6796.10.109
0.8900110.0104
0.7869110.0888
0.9118:1000000
0.6940=0.09955
0.8727:10.0034
0.7703-10:716
0.8965110.0001
2
0.0062:1..0000
0.7920:10.0001 0
0.8049::000002
0.6758=0.0001
0.0067
0.7490110.0006
0.7806:10.0000
3
0.705
0.7667:10000055
0.7662-+0.0006
0.7723::0.0007
0.7486:10.0002
0.7602-+000005
0.7623:+0.0006
0.7686:100007
0.9219:::.00066
0.9219:1000003
0.0035
0.9278:10.0002
0.9200:10.0007
0.9204:10.0004
0.520411000044
0.9264:10.0000
5
0.8313+000003
0.8362:4000003
0.6707=000000
0.8423-+:000005
0.8306+0.0000
0.865420.0003
0.5393=0000000
0.8424++000005
0.862=000000 6
0.8700=t000002
0.6767=0.0015
0.8007:=000003
0.8661#0.0005
0.8698:0000002
0.5760=#00002
0.8706=+000003
7
0.0002
0.0046
0.568611.00000
0.5093
0.000
0.0056_
0.0000
0.5903 0.00004
8
0.9157
0.8841
0.000
0.0000
0.0057
0.8840=0.0002
0.3738-10.0027
0.9 0.9166-10.0000
9
0.6864=1000002
0.6175=0000011
0.4775
0.7083
0.6759=0.00003
0.0011
0.35981000034
0.7382
10
0.9002:=000001
000000
0.8490=000000
0.9347=+0.0001
0.9200==000001
0.9184=20.0001
0.7795+000000
0.9034-10.0001
11
0.0000
0.0000
0.9711==0.0000
0,9607=000000
0.9568==0.0000
0,9568 0.0000
0.9568=000000
0,9603-+0.0000
12
0.0083
0.7986 0.0033
0,6535
0.00207
0.8469 1.00015
0.7420.10.0482
0.7762 1.0026
0.5789
0.8430 0.84301.00000
13
0.9763110.0000
0.9760.1..0000
0,9474 0.0000
0,9838.1 0.0000
0.9753_000000
0,9754 0,0000
0.921811000000
0,9835 0.0000
14
0.9831=0.00000
0.000=
0,9484 0,00000
0.0000
0.9825=0.0000
0.9825=
0.023
0.900011000000
15
0.000
0.6833:1000000
0.6113=0
0.6000
0.6010=0.0002
0.0003
0.6927 0.600
16
0.9403=000001
0.9397110.0001
0.9628:10.0000
0,9495510.0003
0.9130:1000004
0.9118::000003
0.0552-10.0003
0.9299
17
0.9266:20.00000
0.0005
0.9217
0.9732 0.0000
0.9266=0.00000
19185=000000
0.0207
0.9732.1000000
18
0.000010.0000
0.0000:1.00000
0,983211000000
1.0000 0.0000
1.000010.0000
1. 0.000000.00000
0,9832 0.0020 0.00000
1.00001 0.0000
19
0.9532=#0.0000 0.9510=0.0000+
0.9033=0000000
0.9950=t0.0000
0.9526=0.0000
0.9505=0.0001
0.8924+0.0000
0.9050=0000000
20
0.8453==000000
0.8445::000000
0.8436=000000
0.8404=000000
0.839900.0009
*
0.8390=t0.0000
0.8000
0.8625=1000000
Table 2. Accuracy and F-measure based on different loss functions on the image datasets when model layers is 3.
Data
Accuracy
F-measure
CE
CE-SED
CE-inform
CE-PSED
CE
CE-SED
CE-inform
CE-PSED
Mpeg
0.6438=0.000l+
0.6538=00.0044
0.2098+0.0000
0.7338:20.0002
0.6342=00000
0.6426+000004
0.0549=0.0004
0.7244-000003
Mnist
0,9039110.0000
0.9046.10.0000
0..42310.0001
0,9062.1 0.0000
0.9037.10.0000
0.9044.10.000
0.8409 10.0001
0.90611100000
Pendigits
0,9611 10.0099
10.0000
0.7889110.00100
0. 0.9000100.0000
0.768410.0001
0.9610110.0000+
0.7808110:0013+
0.9908-10.0000
Caltech-101
0.0011=0.0001
0.2078:2000002
0.2758:10.0001
0.5005:10.0001
0.2814:000001
0.1786:20.0003
*
0.1754:2000002
0.4708:10.0001
ImageNer
0.9701 #000000
0.0090
0.9753-1000000
0.9762+10.000
0.9703=10.0000
000000
0.9754=0.0000
0.9762:100000
837 837.9047, 837.8022, and 837.8456 (see Appendix for additional results). These findings further validate the superior performance of CE-PSED in feature representation and class discrimination.
Table Comparison F-measure at different depths on Yeast
Method
Layer 3
Layer 5
Layer 8
CE
0,5475
0.1539
0.1662
CE-SED
0,5556
0.1734
0.1740
CE-inform
0,5470
0.1620
0.1646
7.2. Experimental on the Image Dataset
PSED
0.5903
0..951
0.4116
We also evaluate the proposed method on five additional image datasets to further validate its effectiveness. These experiments adhere to the same settings as the baseline dataset, differing only in the feature extraction methods (see Appendix for details), ensuring consistency and comparabil- ity of the results. As shown in Table and Figures 6(c) and (d), the results clearly demonstrate that the CE-PSEDD-bsse method outperforms other methods in both performance and efficiency in recognizing category structures. Specifically, the CE-PSED method excels in multiple key performance metrics, including classification accuracy and F-measure. Additionally, it exhibits a particularly strong capability in re- vealing structural differences between categories, a critical aspect of image recognition and classification tasks.
7.1.5. ANALYSIS OF DISCERNMENT ABILITY
To evaluate the discriminative ability of the four methods, we analyze the feature representations of the last hidden layer of the model. Figure 16 illustrates the t-SNE visu- alization of these feature representations for each method on the Pendigits dataset (see Appendix for results on other datasets). The figure demonstrates that the PSED-based method effectively separates classes. Additionally, we com- pute the Euclidean distances and information entropies be- tween the similarity matrices and yy" across all benchmark and image datasets. For the Pendigits dataset, the Euclidean distances for CE, CE-SED, CE-inform, and CE-PSED are 0055.0093, 1055.0093, 1042.0280, and 159, respec- tively, while the corresponding information entropies are
8

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Acknowledgements
This work was supported by the National Natural Sci- ence Foundation of China (Nos. 62306170, 621 62136005, U24A20253, 62476160, 62441239), the Major Project of National Natural Science Foundation of China (No. T2495251), the Science and Technology Major Project of Shanxi (No. 20220 1022000021000)) the Special Fund for Sci- ence and Technology Innovation Teams of Shanxi Province (No. 202304051001001),
04
(a) Ycast
(b) Ringnorm
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
(c) Caltech-100 101
(d) ImageNet
Figure 6. Accuracy curves when model layers is
References
Arora, S., Ge, R., Neyshabur, B., and Zhang, Stronger generalization bounds for deep nets via a compression approach. In Dy, J. and Krause, (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 254-263, 2018
(a) CE
(b) CE-SED
Bai, X., Liu, W., and Tu, Z. Integrating contour and skeleton for shape classification. In 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Work- shops, pp. 360-367, 2009.
Bansal, M., Kumar, M., Sachdeva, M., and Mittal, A. Trans- fer learning for image classification using vgg19: Caltech- 101 image data Journal of Ambient Intelligence and Humanized Computing. 14:3609 3620, 2021.
(c) CF-inform
(d) CE-PSED
Figure? 7. The t-SNE of Pendigits.
Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463 482, 2002.
8. Conclusion
Bartlett, P. L.. Foster, D. J., and Telgarsky, M. J. Spectrally- normalized margin bounds for neural networks. Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Cur- Associates, Inc., 2017.
This paper introduces novel Pure Square Euclidean Dis- tance (PSED) metric within the framework of pure random consistency and provides a corresponding analytical solu- tion. The unbiasedness and heterogeneity of PSED are rigorously validated through both theoretical analysis and simulation experiments. Additionally, the study investigates the learnability of PSED in fully connected neural network structures and establishes its performance. Furthermore, we propose deep network model that utilizes PSED as the loss function, demonstrating superior performance and effectively mitigating collapse. the future, we plan to analyze the optimization convergence properties of PSED and develop further learning models that optimize PSED.
Bian, Y., Ju, X., Li, J., Xu, Z., Cheng, D., and Xu, Q. Multi-pater prediction: Adapting language models for time series representation learning. In Proceedings of the International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 3889-39122 21-27 Jul 2024.
Boucheron, S., Lugosi, G., and Massart, Concentration

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 02 2013.
words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Brockmeier, A. J., Mu, T., Ananiadou, S., and Goulermas, Y. Quantifying the informativeness of similarity mea- surements. Journal of Machine Learning Research, 18 (1):2592-2652,222 2017.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Byerly, A., Kalganova, T., and Dear, I. No routing needed between capsules. Neurocomputing, 463:545-553, 553, 2021,
Escande, P.
On the concentration of the minimizers of empirical risks. Journal of Machine Learning Research, 25(251))1-53, 2024.
Cai, D. and Chen, X. Large scale spectral clustering via landmark-based sparse representation. IEEE Transactions on Cybernetics, 45(8):1669- 1680, 2015.
Fernandez- Delgadd,, M., Cernadas, E., Barro, S., and Amorim, D. Do we need hundreds of classifiers to solve real world classification problems? Journal of Machine Learning Research, 15:3133-3181 2014.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, 2016.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual repre- sentations. Proceedings of the International Conference on Machine Learning, pp. 1597-1607, 2020.
Ghandeharioun, A., Caciularu, A., Pearce, A., Dixon, L., and Geva, M. Patchscopes: A unifying framework for inspecting hidden representations of language models. Proceedings of the International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15466-15490, 2024.
Chen, X., Xie, S., and He, K. An empirical study of train- ing self-supervised vision transformers. International Conference on Computer Vision, pp. 9620-9629, 2021.
Golowich, N., Rakhlin, A., and Shamir, O. Size- Size-independent sample complexity of neural networks. In Bubeck, S., Perchet, V., and Rigollet, P. (eds.), Proceedings of the 31 Conference On Learning Theory, volume 75, pp. 297-299, PMLR, 2018.
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L.., Thapliyal, A., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B. K., Riquelme, C., Steiner, A., Angelova, A., Zhai, X., Houlsby, N., and Soricut, R. Pali: A jointly- scaled multilingual language image model, 2023.
Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016.
Goodfellow, J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. Maxout networks, 2013.
Chopra, S., Hadsell, R., and LeCun, Y. Learning a sim- ilarity metric discriminatively. with application to face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 539-5-9944 546, 2005.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Proceedings of the Inter- national Conference Neural Information Processing Systems, pp. 2672-2680, 2014.
Cortes, and Vapnik, V. Support-vector networks. Ma- chine Learning. 20(3):273-29) 297. 1995.
Greenfeld, D. and Shalit, U. Robust learning with the hilbert- schmidt independence criterion. In Proceedings of the 37th International Conference on Machine Learning, pp. 10, 2020.
Crabbé, J., Huynh, N.. Stanezuk, J. P., and Van Der Schaar, M. Time series diffusion in the frequency domain. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (. Pro- ceedings of the International Conference on Machine Learning, volume 235 of Proceedings of Machine Learn- ing Research, pp. 9407-9438. 21-27 Jul 2024.
Grigorescu, C. and Petkov, N. Distance sets for shape filters and shape recognition. IEEE Transactions Image Processing, 12(10):1274-1226, 2003.
He, K. and Sun, J. Convolutional neural networks at con- strained time cost. In 2015 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 5353 5360, 2015.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x16
10

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Lin, T.-Y,, Dollár, P., Girshick, R., He. K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. In 2017 IEEE Conference Computer Vision and Pattern Recognition, pp. 936-9996900999 2017.
Hinton, G. E., Ghahramani, Z., and Teh, Y. W. Learning to parse images. In Proceedings of the 13th International Conference on Neural Information Processing Systems, pp. 463-469, 1999.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022.
Hinton, G. E., Osindero, S., and Teh, Y.-W. A fast learning algorithm for deep belief nets. Neural Computation, 14 (8):1771-1800, 2012.
McConville, R., Santos-Rodriget R., Piechocki, R. J., and Craddock, I. N2d: (not too) deep clustering via clustering the local manifold of an autoencoded embedding. In 2020 25th International Conference on Pattern Recognition, pp. 5145-5152, 2021.
Huber, P. J. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73-101, 1964.
loffe, S. and Szegedy, C. Batch normalization; accelerat- ing deep network training by reducing internal covariate shift. In Proceedings of the International Conference on International Conference on Machine Learning Volume 37, pp. 448 456, 2015.
Neyshabur, B., Tomioka, R., and Srebro, N. Norm-based ca- pacity control in neural networks. In Grünwald, P., Hazan, E., and Kale, S. (eds.), Proceedings of the Conference on Leurning Theory, volume 40, pp. 1376-1401. PMLR, 2015.
Irle, A. and Kauschke, J. On kleinberg's stochastic discrimi- nation procedure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(7):1482-1486, 2011.
Neyshabur, B., Bhojanapalli, S., and Srebro, N. A PAC-bayesian approach to bounds for neural networks. In International Conference on Learning Representations, 2018.
margin
Kingma, D. and Welling, M. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations, 2014.
Ng, A. Y. Feature selection, L1 vs. L2 regularization, and rotational invariance. In Proceedings of the Twenty-Firs International Conference on Machine Learning, pp. 78, 2004.
Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Schindler, G., Hornung, R., Birodkar, V., Yan, J., Chiu, M.-C., Somandepalli, K., Akbari, H., Alon, Y., Cheng, Y., Dillon, J. V., Gupta, A., Hahn, M., Hauth, A., Hen- don, D., Martinez, A., Minnen, D., Sirotenko, M., Sohn, K. Yang, X., Adam, H., Yang. M.-H., Essa, I., Wang, H., Ross, D. A., Seybold, B., and Jiang, L. VideoPoet: A large language model for zero-shot video generation. In Proceedings of the International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 25105-25124, 2024.
Pham, H., Dai, Z., Xie, Q., and Le, Q. V. Meta pseudo labels. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11552-11563, 2021.
Rangapuram, S. S., Seldin, Y., and Bubeck, S. On the effectiveness of the softmax loss for deep learning. In Proceedings of the 35th International Conference Ma- chine Learning, pp. 4599 4608, 2018
LeCun, Y., Bengio, Y., and Hinton, Deep learning. Na- ture, 521(7553):436-444 2015.
Ren, S., He, K., Girshick, R., Zhang, X., and Sun, J. Object detection networks on convolutional feature maps, 2016.
Li, F. Qian, Y., Wang, J., Dang, C., and Jing, L. Clus- tering ensemble based on sample's stability. Artificial Intelligence, 273:37-55, 2019.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958 2014.
Li, H., Ye, X., Imakura, A., and Sakurai, T. Divide-and- conquer based large-scale spectral clustering. Neurocom- puting, 501 :6146664666 678, 2021.
Toth, C. and Oberhauser, H. Bayesian learning from se- quential data using gaussian processes with signature covariances, 2020.
Li, J., Qian, Y., Wang, J., and Liu, S. Phsic against ran- dom consistency and its application in causal inference. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 2108-2116, 8 2024.
van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of Machine Learning Research, 9(6)): 2579-2605, 2008

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Vapnik, V. and Chervonenkis, A. Estimation of dependences based empirical data. Theory of Probability & Its Applications, 16(1):264-280, 1971.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, u., and Polosukhin, I. At- tention is all you need. In Proceedings of the 3t Inter- national Conference on Neural Information Processing Systems, pp. 5-9855999999 00086, 2017.
Vershynin, High-Dimensional Probability: An Introduc- fion with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.
Vinh, N. X., Epps. J., and Bailey, J. Information theoretic measures for clusterings comparison: Variants, proper- ties, normalization and correction for chance. Journal of Machine Learning Research, 11:283-28544, 2010.
Wang. J., Qian, Y., and Li, F. Learning with mitigating random consistency from the accuracy measure. Machine Learning, 109(12):2247-228) 2020a.
Wang, J., Qian, Y., Li, F., and Liu, G. Support vector ma- chine with eliminating the random consistency. Journal of Computer Research and Development. 57:1581, 2020b.
Wang, J., Qian, Y., Li, F,, Liang, J., and Zhang, Q. General- ization performance of pure accuracy and its application in selective ensemble learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1798- 1816, 2023.
Wang, J., Li, F., J., Qian, Y., and Liang. Gini index and decision tree method with mitigating random consistency. Science China Information Sciences, 54:159-190, 2024.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-opes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, I.. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 23 Jul 2022.
17-
Xu, J., Wu, H., Wang, J., and Long, M. Anomaly trans- former: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations, 2022.
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners image- text foundation models, 2022.
12

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Appendix
9. Proof
9.1. Proof of the analytical solution SED
The Square Euclidean Distance (SED) is defined as:
dSEn(K) - || YY"
(28)
where Y is the real label vector, {0, 1)=8 the one-hot encoding of the true label vector, is the number instances, k is the number of classes and || is the Frobenius norm, which represents the square of the sum of squared elements of the matrix.
And we provide an analytical solution for SED:
dsEn (K, Y) = |K - YYT 1|
(29)
k |
m
1 m.
K M1 -
where m, single column all vectors of length m,, K Kij the sub kernel matrix of class and m, is the number of objects of class.
Proof For multi-class classification tasks, let be the total number of samples, &o be the number of categories, my m2, my be the number of samples in each category, such that mi + m2 + ... + n. The true label
matrix Y can be represented as:
T
1mxx1= O
Om, xix
0 Om x1 0mgx1
I I
(30)
m
Ompx1
1 1 nxc
where 1m,xi and 0mlx1 are single column all vectors and all 0 vectors of length my, respectively. The adjacency matrix generated by is:
Jmn Xm Ome xi
Ong Xing Jmzxma
Omg Xms Omg xms
YYT
(31)
Om Xmi
Ony Xma
Jun Xmay Xx
where / and 0 are the full one matrix and the full zero matrix, respectively. According to the category of samples, we block
the similarity matrix as follows:
K Kipp
K [1ja)
K [I[M]]
K [
K []
K
[2[[
(32)
[=|[2]
[M]]]]]
xxn
where K K|][a] represents the similarity matrix between class my and class my-
For si= < k.
K [|
YYT
m. xm, 12
(33)
=| 1aa 11 2 (1]))) xm,) + || Mm xm, 112
B||K 11 221 Kaaja 1m; + m
13

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Similarly, for isiijsh
YY [|b 0m,xm,
| M|
my xm, ]F
(34)
= M|
(35)
Then by performing some simple elementary operations, we obtain:
dSED(K, Y) = K YY" 1B
(36)
k
|
2
T
K
[10
m, m
-=1
1=1
9.2. Proof of Theorem 3.2
To provide proof for Theorem 3.2, first give a lemma.
Lemma 9.1. Let A fa1, a2..0 an) be of n elements. Select elements from A, calculate the sum of the products of any elements selected from these elements as S1 and compare it with the sum of the products of any elements selected from A as Sz. Since they are only related to the number of items, the relationship between S. S2 is:
S S2
(i
11
x 17
(37)
LiE(g ai
i
where (1%) represent the 1-th set of m elements taken from and
denotes the size of set.
Proof
Theorem 3.2 i7 the set of all r-tuples drawn without replacement from the set [n....n)). The analytic solution of the expectation of E (dsED(K, Y')) is:
Ey (dsED Y'))
(38)
K
m
K
Kg
Step 1: Convert the expectation about all permutation Y' into the mean about the mn-tuples, ma-tuples, mm- tuples.
According to the definition of PSED:
dpSED(K) dsED (K, Y) (dsED P')) = K T Ey K| -' (1?))
(39)
and the analytic solution of dsED:
dSED (K V) = K YYT 1
(40)
| 2
Σ
m - 2 >
1 T K K(
i=1
1=1
we have,
(yddEED Y'))
(41)
= || + Σ
m
2
Ey (1T
K'
K3jje m,)
m.
14

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
where K' []]
is the sub-block matrices of K'.
The matrix K' is the permutation similarity matrix after switching the positions of the samples according to Y'. Since Y' consists of all possible labels that maintain the distribution ratio my : m2 :.. mg, with the labels being uniformly distributed, the expectation can be expressed as:
4 T
By ( my
(r]jr]mmd
(42)
Bmy 1=1
Eise(im ) Kg
m 1=1
Eije(sm mi), Kij
1=
Fmn
n- my
mk
Σ
(i m-
Kij
1mn mi1
im
Imm
mi mk-1
where 1 E {1,2, 11 1b, (Im, )r U (im
) U ... U (Img
me ") = {I,... ,,), (1 )r O (i%,")) .... 0
(Ime (in--) ----m------------ = 0, n m1 )r} and Kij is the value of the i-th row and j-th column of matrix K.
mi-1 (Imy
(in - )r represents the set (1, 2. ,n) M R )r (17-m))
U (1m--1
Step 2: Convert the mean about tuples into the mean of elements in K.
Based on the observation, Formula 42 can be further computed using Lemma 9.1. In the lemma, when r = 1:
1 v= (IT)) aa
Si
fi x 117 if
(43)
S2
(=EE))))))) 0;a;
and when 2,
Σ F
S
a:a3 (i 0,ag
[i7"
(44)
52
[i Σ
i
Therefore, compute Formula 42 by separately considering the diagonal and off-diagonal elements.
When jj:
By (1Th. K (iller
( mim
X
Mi
K;; +
1m X
c m3
mg
X m2
K +
(45)
[m
X
i=
lin
x
Imz
X n1
1=1
im
m1
x
Iine
x mi
K
mm
x
[ime
m1
X
x
m-4-1
x n
mi
M2
mi
K
K
Küi
n1
15

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
When # j:
2 Ey (1 K (rlls] mm )eys
X
1
2
Kij zj +
X
X
2
1722
Σ
Kij
(46)
in
x
1781
x 1
im
p2-m1
X
1ink
17-m1
m.
2
mag
Kij
x
1mz
my
X
1mn
1
12 K
K i5
+
Kij
Kij
Combining Formulas 45 and 46, we obtain:
k
k
Ey (1 K
Kinl 1m.)
Σ
Kü
+
Kij
(47)
1=1
So, obtain analytical solution of Ey (dsED Y')):
Ey (dsEEDDKK (KYY) Y')
(48)
n
T
m
K
12
Σ
Kis
Based on Formula 39, 40, 41 and 48, the analytic solution of PSED is:
(PSED(K)
(49)
2
K
K 1
Σ 1=1
1mc
K Kaplsl 1mp
0
9.3. Proof of the computational efficiency of PSED
We provide the definition of PSED and the computational efficiency of analytical solutions.
For the definition of PSED, that is Formula 39, computational efficiency is divided into two parts, the first part is divided into three sub parts :(1) Y Y" (2kr²); (2) K YY O(z)); (3) KK yy" [| :On²). (dsED (K, Y')) requires the calculation of all cases that follow the same distribution as the true label Y. involving total of mi terms, so my.ing my (O(2kn?) + O(n²) + O(n)). Therefore, the overall
the computational efficiency of the second term is computational efficiency defined by PSED is
n!
1) x O2kn 2 + 2n²)
m. Imm...mm
For the analytic solution of PSED, computational efficiency is divided into three parts: (1) ET K, O(n); (2) Σx_ Eij K,y Ok(n22 n)): (3) Σ 1m, K Im, O m2). So, the overall computational efficiency of the analytical solution of PSED is O(kn² + (1 k)n =k m²).
Due to the existence of inequalities:
(a + b + +c² 20² 4 12+ +
(50)
n! Therefore, Okn + (1 (1-k) k)n E 1 m,) < 0(2m2 2n and ( (m, lmm......m + 1) is large, the computational efficiency of the analytical solution of PSED is significantly higher than that of the PSED definition. other words, the analytical solution of PSED has more effective computational efficiency.
16

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
9.4. Proof of properties 3 and 4
Property (Bias of dSED) For the matrices and J. when Σi= m dIED(I) < dsED(J). Proof According to the analytic solution dgED:
> n +m - 2
we have dsæn(1) > dJ)n(J); otherwise,
dSED (K, Y)
K YY l
(51)
k
K||
Σ
m 2 Σ
2 m.
K km
When K = J. we have:
(J)
(52)
k
= 1|
T
m. J mm Xm, 1m, +
Σ
m
1=1
n²
m
1=1
When K I, we have:
dsED(1)
(53)
k
||
Σ
1 T
I m. x m. 1m;
Σ m
m n.
Then, we have:
dSED(!) - dJED(J)
(54)
k
m n - (n²
Σ m+)
i=1
i=1
2
Σ m
n.
1=1
Thus, obtain the conclusion.
Property 4 (Unbiased of dPSEn) For any matrix A in N₂ = ((1-a)) + 1}, where / is the identity matrix and J is the full one matrix. We have (AsED(A) = 0.
Proof In fact, the same, that is: [M = ([ll ((1 a)l + aJ)mnxxm. Combining with the analytic solution dsED, we have that (A, Y)= = dsED (A, Y") = Ey (ASED(A, Y')). Thus, we have (pSED(A) =
the matrix with the diagonal is 1 and the other elements are a. For and Y', their sub block matrices are
9.5. Proof of Theorem 5.2
Proof. To use the concentration bound in Theorem 5.2, we firstly need to investigate the change in the loss when single object is modified. Secondly, we need give the exponential Orlicz norm bound of the loss change. The definitions and properties the sub Gaussian norm used in the proof are provided at the end of this section.
For the first step, without loss of generality, we assume that the changed sample belongs the first category. In this case,
17

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
we have,
[dpsED (S() - (K(()()()
(55)
= =12(Ket - kwx)+
Σ
Kkj
E
Kwi)) 4[KK[1]]m, (w(l)=mi)
(56)
just
jiith
< |2(KEs - REN) +40|
Σ
Kaj
RN's)
| + + 4k(Ks - Kww
(57)
1:1Z
where C' = Σド= 1/13/1
From Lemma 13 in (Greenfeld & Shalit, 2020), we know that assume K(2, y) = exp ( 1-2(2 =)))), as the case with RBF kernels, and suppose ||a||| V M for all x E y. Then k(--)) is MM-Lipschizz for all x E X. Therefore, we have,
2
RRR= - Kew!
(58)
= (h(xx)) h(x))) - (h(xx), + K ((b(x)), ((x)) - K(H((())
(59)
- 2 M(xx)) - ((M())))
(60)
s 2yy (M(x))) h(xx))122 S 2yM]IM(xX) M(xx))l
(61)
L-1
S M |x - x
II P Will
(62)
By a combination of Eq. (58) and the triangle inequality, for the second term and third term of Eq. (55), respectively, we
have:
( Σ Kas E Kes) ) s 2y -1)l(x xll1i II p Will,
(63)
jif
8:5t
(Katt)am) - 1 S 2y
x((ll p l| Will-
(64)
For the second step, sequentially by Property and 7, we have,
L-1 ||| x II
Pt
W Illall
(65)
L-1 1
I || |||
x vr. II
pi [ wllle
(66)
S vdllx - x'x llle
Pi / de Wi Willez
(67)
1=1
L-1
S <d||diam(x)) pzz
II
/iv de Welloz
(68)
where the last inequality is according to the definition of 1y norm.
Above all, we have,
dpsED((((.)) (K(TSE
(69)
L-1 II pt W 1=1
$ (2 + 4C 1) + 4m1)2y | - xxella
(70)
1-1 II pr / de Willow 1=1
< 32+40 1) + k2 (il
(71)
18

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Thus, obtain the final result.
9.5.1. PROOFS OF PROPERTIES
A random variable with finite | admits tail satisfying (Vershynin, 2018),
19
P (12) 2 t) S 2 exp
(72)
|
From this tail, we can observe that the smaller the ||XX| 112, norm, the more concentrated the distribution of variables.
Theorem 9.2. (Young's Inequality) Let 01, at 2 O, P1, PL > 1, E 11=1
1 1, there are Young's 's Inequality,
24
Pi
2m
2m
the equality holds when ap = a!h
PL
Proof of Property
Proof. Suppose |1 = c1/2 and || = c2/2, then by definition,
2X
2Y
E
exp
<
2,
E
exp
2
(73)
C1
C2
We have,
X Y
Exx
(74)
C1 02
X
Y
E Eexp
(75)
1 +
02
C1 +
02
X
Y
E Exxp
+
(76)
C2
Y
E
exp
exp
(77)
02
2X
2Y
E
exp
exp
2
(78)
2
C2
where the first inequality is based on the triangle inequality and the last inequality based on the Young's inequality.
Proof of Property 6
Proof. We assume that ||Xill,, C;s then,
X;
exp
2.
C3
19

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
There exists that,
X
E eexp
s E exp
C2
X
E
exp
L
X
L
E
$
2,
where the first and the second inequalities are based on Young's incquality.
Proof of Property? 7
Proof. There exists,
|
9
| 19
E
exp
E
exp
(79)
c
lax1) >
197
Vi
E
exp
(80)
Vd
where 1ax1 a column vector of which elements are all 1. We assume that ||I|
X X11le,, c. Then by definition, we obtain,
x
(x1)
c
(81)
Vd
44
Because J/2 1dx1 E Sd-1, we have that
sup wE8/-1
(82)
10. The algorithm process diagram of the method
The specific algorithm process of the method used in this paper is shown in Algorithm 1, where RL is the representation layer and CL is the classification layer.
Algorithm Model Construction
INPUT:
The training sample features and labels of features: Xtrain Ytrain-
The maximum number of iterations epo.
OUTPUT: The model parameters RRL. BCL-
1: Initialize model parameters and learning rate: WIL. WCL,
2: for epoch=1:epo do
3: (Ztrain) <- RL(Xtrain) WRL).
4: Calculate the loss of PSED and perform back propagation and update parameters W Wn..
5: (Y) t Cl(rrraini Wet).
6: Calculate the loss of CE and perform back propagation and update parameters Wet.
7: end for
20

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
11. Experiment
11.0.1. THE DATASETS AND MODEL STRUCTURE PARAMETERS
We provide detailed description of the dataset and download links. For more detailed information, please refer to Tables and 5. Among them, the imbalance ratio refers the ratio between the most common and rare categories in the dataset. The description of the image dataset is follows:
The Moving Picture Experts Group (MPEG) dataset includes video sequences designed for testing video encoding and transmission algorithms. The dataset consists of videos captured under various scenarios and conditions, making it ideal for evaluating the performance of video encoding techniques. We show some images the dataset as shown Figure (a).
MPEG Dataset:
The Modified National Institute of Standards and Technology (MNIST) dataset widely used for digit recognition tasks. It contains 60,000 training samples and 10,000 testing samples, each represented by 28x28 grayscale image of digits from 0 to 9.
MNIST Dataset:
Pendigits Dataset: The Pendigits dataset is dedicated to handwritten digit recognition. It includes over 10,000 32x32 grayscale images of handwritten digits (0-9), with separate training and testing sets.
Caltech-109 Dataset: A widely used object recognition dataset, Caltech-101 contains approximately 9,000 images across 101 object categories, including animals, vehicles, food, and furniture. The dataset features images taken in diverse real-world settings, offering a robust benchmark for image classification tasks. We show some images of the dataset as shown Figure (b).
ImageNet Dataset: ImageNet is comprchensive image recognition database with over 14 million labeled images spanning more than 20,000 distinct classes. It is extensively used for evaluating the performance of image classification models. We show some images of the dataset shown in Figure 8 (c).
Apple
Accordion
Kin
ponguin
Bal
Bonoai
Whit Sentir
Beeds
cougar face
Del
Emu
Ainhip
Snoopy
Alriner
(a) MPEG Dataset
(b) Caltech-100
Dataset
Image Net Dataset
Figure 8. Example pictures of image datasets.
To ensure consistency in the evaluation, each dataset is randomly divided into training, validation, and testing sets in 5:2:3 ratio. This hierarchical approach ensures that the performance evaluation of the model at different stages training is not affected by randomness. To investigate the impact of model architecture on performance, we evaluated configurations with 3, 5, and layers. This change allows us compare detail how model complexity affects performance under different loss functions. The training process uses the Adam optimizer, which widely favored for its efficiency in adjusting learning rates. The learning rate of each dataset has been fine tuned to achieve optimal convergence performance. Training for up 60 epochs provides ample time for model learning and reduces the risk of overfitting. We also customized hidden layers and regularization parameters for each dataset. This customization takes into account the unique characteristics and complexity of each dataset, ensuring that the model architecture is best suited for optimal performance. The batch size set to 16. For feature extraction, ImageNet employs the sulf-super learning model MOCO V3 (Chen et al., 2021), 1), while all other datasets utilize VGG Beermandez-Delgadd al., 2014).
21

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Table Description of the Datasets
Name
Object
Dimension
Class
Imbalanced Ratio
Wine Quality
11
12.85:1
Blood Transfusion Service Center
748
4
2
3,20:1
Energy Efficiency
768
9
2
1,87:1
Tic- Tic-TacTTee Endgame
957
9
2
1.88:1
Meeytte-Meruuccuu--ee
1022
41
2.03:1
QSAR Biodegradation
1055
41
2
1.96:1
Yeast
1484
8
10
92.60:1
8
Semeion Handwritten Digit
1593
156
10
1.05:1
Steel Plates Faults
1941
27
7
12.24:1
Cardiotocography
1950
5.61:1
11
Ozone Level Detection
2536
72
2
33.74:1
12
SkillCraft1 Master Table
3343
21
2
1.03:1
13
Gender Gap in Spanish WP
3355
21
2
17.95:1
Waveform Datahase Generator
3679
21
2
18.26:1
Abalone
4177
8
2.16:1
16
Page Blocks Classification
5242
10
2
14.93:1
17
Ringnorm
7400
20
2
1.02:1
18
Mushroom
8124
21
2
1.07:1
Nursery
12960
8
13.09:1
20
Adult
32561
14
2
3.15:1
21
Mpeg
1400
6000
70
1.00:1
22
Mnist
6996
784
10
1,25:1
23
Pendigits
7494
16
10
1.08:1
Caltech-101
8641
256
19.46:1
25
ImageNet
13000
256
10
1.00:1
Table 5. Addresses of Datasets
ID
Data Address
6
8
13
14
15
16
20 http://wccchiieeicccacccclllllea
21 http://ttaarrtttooogggeee/////er
22
23 http://////////////////////////e
25
22

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
11.0.2. THE EVALUATING MEASURE
In this analysis, we evaluate the model using two key performance metrics: accuracy and F-measure, which are defined as follows:
Accuracy Σ
I(Y Y.), 1=1
(83)
F-measure = 2Precision x Recall
(84)
Precision + Recall
where
E I(Y,) = Y, 1) EE= Ej=1 T(Y,) =1)
Precision
(85)
Recall E Σラ- [(Y = 1, 1)
(86)
Ei T(y,) 1)
Vig is the predicted label for sample and class j. Ye is the true label for sample i and class j. and the number of class. Additionally, IT is the indicator function, which takes a value of when the condition inside the parentheses is true, and otherwise.
11.1. Experimental
11.1.1. COMPARISON OF CLASSIFICATION PERFORMANCE
To assess whether there are statistically significant differences between the proposed method and other approaches, we first conduct one-sided t-test. The null hypothesis Hg assumes the proposed method is inferior to other methods, while the alternative hypothesis H1 posits that the proposed method outperforms the others. The significance level for the test is 0.05. If the p-value is below this threshold, Ho is rejected, indicating that the proposed method demonstrates statistically sigmificant superiority. As shown in Tables 6 and 7, if our method outperforms others significantly, black dot will be added next to the method. As shown in the tables, the PSED-based loss function demonstrates superior performance in terms of average convergence accuracy and F-measure, with values surpassing other methods on most datasets.
11.1.2. SIGNIFICANCE TEST
To investigate whether the proposed algorithm exhibits significant performance differences compared to baseline methods, apply the Friedman test. This non- -orrmmemetr test evaluates the rankings of multiple related samples across multiple datasets, enabling to determine whether significant differences exist among the four methods being compared. The null hypothesis (Hu) assumes no significant differences among the methods, while the alternative hypothesis (H₂) suggests that at least two of the methods differ significantly. A p-value threshold of 0.05 is used in this analysis. If the p-value is below this threshold, Ho is rejected, indicating that sigmificant differences exist among at least two of the algorithms. Upon obtaining a significant result from the Friedman test, we perform the Nemenyi post-hoe test identify which specific pairs of algorithms differ significantly. The Nemenyi test calculates the critical difference (CD) value, which is subsequently visualized in CD diagram. This diagram offers clear representation of the average ranks of each algorithm, with horizontal bars denoting significant differences between algorithm pairs.
m
150
(2483
(a) AccuracyBenccharkk))
(b) F-measuue(Benchnarkk)
(c) Accuracy(Image)
(d) F-measure(Image)
Figure 9, CD diagrams w.r.t. Accuracy and F-measure when model layer is 5.
23

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Table Accuracy and E-measure based on different loss functions when model layers is 5.
Data
Accuracy
F-measure
CE-SED
CE-inform
CE-PSED
CE-SED
CE-inform
CE-PSED
0.922211000099
0.9267:10.0000
0.8421110.031
0.9276:10.00000
0.8907110.002
0.8923:10.0000
0.8045110.0799
0.8928.1000000
2
0.6979
0.7076
0.000
0.7769110.0003
0.5950=0.03166
0.6027110.0318
0.66055.0.0000
0.7124110.0021
3
0.6506:20.0000
0.6089:0000000
0.6182:0000994
0.7667::000005
0.0068=0:0003
0.5126:20.0000
0.1552
0.7631:10.0005
0.6038:10.0122
0.0030
0.00112
0.7920:10.0001
0.4830=0.0132
*
0.0164=0.0000+
0.5208
0.7820:10.0000
5
0.6616=0.0002
0.6694::0.0000
0.6713+50.0000
0.7837::000004
0,5416+0.0000
0,5447=:00005
0.5455=000000
0.7733--000003
6
0.0002=0.0002
0.6354=0.010*
0.6625=0.0000
0.8770==0.0003
0.5611=0.0029
0.5013=:00144
+
0.5291=0.0001
0.8769++000003
7
0.0000
0.2910.10.0020
0.204410.0008
0.5231.10.00
0.153000
0.001
0.1620110.0008
0.4951 1.00001
8
0. 0.004
0.2001
0.0000
0.8749-10.0003
0. 0.0000
0.001
0.2100
0.8747-10.0000
9
0.41533=00002
0.4348:0000046
0.454410000028-
0.0002
0.2891=000051
0.3163=0.0066
000054
0.6185:10.000
10
0.8489 0000
0.8489#0.0000
0.0000
0.9246::0.0002
0.709
0.700
0.7796::0.0000
0.9237=-000002
11
0.9711=00000
0,971 0.000
0000000000000
0.9699=0000000
0,9568==0.000
0.9008=000000
0.9568=+00000
0 0.9000=t0.0000
12
0.9480-100000
0.9512.10.0001
0.9484110.000
0,9513.10 0.0000
0,9480110.000
0.9512.10.0001
0.95131.0.0000 0.908410000000
13
0.0000
0.0000
0.9474110.0000
0,9809 0.0000
0.9218_000000)
0,92181 19000810.0000+
0,9218 10.0000
0,9807 11007.000 0.0000
14
0.948
0.000
0.000
0.9002
0.9232=0.0000
0.9232==0.0000
0.9032#0.0000
0.9862:+000000
15
0.5003==00000
0.006
0.6000::000001 0.000
0.5537=
0.5815=0000155
0.6931=000000 0.0000
0.00000.00003+
0.0479=0000002 0.000
0.9800-10.0000
0.9334=0.0010
0.9000
0.9266-10.0000
0.9000-10.0000
17 0.007
0.9456:20.0001
0.9459
0.9726 0.0000
0.047
0.0456=0.0001
0.9459
0.9726:10.0000
18
0.986410.00000
0,9885 1 0.0000
0.9872.1 0.0000
1.000011 0.0000
0.9684_00.0000
0.9885
0.0000
0.9872.1.00000
1.00001 0.0000
19
0.9049=0..000 +000
0,9061 +
0,9064
0.000+
0.9000=:000000
0.8942=0.0000+
0.000
0.8956=000000 a
0.9000
20
0.8442 0.0000
0.8445=0000000
0.8451#000000
0.8007
0.839=00.0000
0.3001=0.0000
0.8000
0.8417=10.0000
21
0.0721 000.000
0.0002
1.622410000066
0.6283-10.0000
0.0044=0.0001
0.006
0.6148.10.0000 0.0094
22
0.7036=0000031
0..55=0000021
0.7222+00000.00
0.8963-1000000
0.7099=000046
0.7094
0,7033-:0.0043
0.8062--0.0000
23
46600300000057
0.6281 0.059
0.637440000022
0.9777
0.0441=0.0067+
0.0930
0.612001.00078
0.9777
24
0.2413 0.000
0.2359 0000
0,2421 +0000
0.4453::0.0001
O. 0.000 0.000
0.0000
0.4107:1000001
25
0.9703==00000
0.000
0.9712==000000
0,9770::0.0000
0.9014=0.0000+
0.0028=0.0000+
0.9713+0.0000
0,9770=10.0000
C0
89
0-500
Gutten
(a) Accuracy(Benchmrkk)
(b) F-meesure(Benchharkk)
(c) Accuracy(Image)
(d) F-measure((mage)
Figure 10. CD diagrams w.r.t. Accuracy and F-measure when model layer is 8.
11.1.3. CONVERGENCE ANALYSIS
The Figure 11 to 13 show the performance of the four methods over the training epoch in all benchmark datasets and image datasets, where the points on each line represent the average accuracy of the corresponding period. The results show that the CE-PSED method exhibits significant performance advantages in most datasets and training epochs, and can quickly converge, which fully demonstrates its robustness and effectiveness different datasets.
1..1.4. NETWORK LAYER ANALYSIS
To verify the effectiveness of the CE-PSED method at different network layers, this study set the model layers to 5 and 8, respectively. Tables and 7 show the accuracy and F-measure values of four methods at different levels. The results show that as the number of network layers increased, the F-measure values of other methods significantly decreased, while the F-measure value of the CE-PSED method decrease less and remain the highest. This indicates that other models experience feature representation collapse as the number of layers increases, that is, the features tend to be the same, while the CE-PSED method performs well at different layers, effectively avoiding feature collapse and ensuring that the model maintains good feature learning and classification capabilities in deep structures.
11.1.5. ANALYSIS OF DISCERNMENT ABILITY
To insight the discermment ability of the four methods, we analyze the feature representations of the last hidden layer of the model. Table 8 to Table 10 present the Euclidean distances and information entropies between the similarity matrices and YYT matrix for all dataset when model layers are 3, 5,and 8, where each row is a dataset, and the columns are divided
24

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Table Accuracy and E-measure based on different loss functions when model layers is 8.
Data
Accuracy
F-measure
CE
CE-SED
CE-inform
CE-PSED
CE
CE-SED
CEsinform
CE-PSED
0.9249-10.0012
0.80211.0.0731
0.9267.10.0001
0.9276.1
0.0021110.0003
0.80451:0.0780
0.893011..0000
0.8928110.000
0.6871=0.0345
0.7538:10.0004
0.6
0.7689-:00000
0.5951:20.0322
0.65571...000
0.58861.0.4439
0.6798:1000011
0.6455=0000000
0.5152::0001.9 0.6147::00088%
0.7580=:0.0007
0.5119:5000005
0.4804::000114
0.4452::0.0194
0.7575::0.0006
0.61841:00002.
0.6528:10.0000
0.7743::0.002
0.4937==00009
0.4867::0.01200
0.5156=0.0000
0.7597:10.0022
0.6655=0.0022
0.6704=0.0000
0.6707
0.7704=000005
0.0000
0.5393 0.5402=0.0000
0.7587:=000006
0.6025==00000=
0.6621 0.0000
0.6606=0.0000
0.8611=000044
0.5280=0.0000
0.5295 0.5300=0.0000+
0.86824000004
0.2971 1:0.0005
0.0007
0.2048110.0037
1.4303.100000
0.1662.1..0006
0.400 =
0. 0.0061
0.1410-1..0030
0. 0.06500003
0.05
1.749880000655
0.002
0.0679-10.0024
0.0718-00.0077
0.7478
9
0.3672=#0.0018
0.0042
0.3719
0.5736:000000
0.2213=0.0035
0.2396--000030 0.2245=0.0020
0.5302:10.0010
10
0.8489-10.00000
0.8080
O. 0.899 0.0000
0.9224=000000 0.0002
0.7795 00000
0.7795-0.0000
0.7795=0.0000
0.9007
11
0.9700=0.0000
0.0000
0.9011=-0.0000
0,9707 0.0000
0,0000
0.9568=00.000
0,9568 0,0000
0,9576-t00000 0000
0.0000
0.9507.1..0000
0,9482 0,0000
0,9534
O. 0.0000
1.949610.0000
0.9517.10.000
0,9481 0,00000
0.95341.0.000
0.404110.0000
0.9474.1
+ 0000
0.947411.00000
0,9808
0.0000
0.9218.10.00000
0.9218 0.0000
0.218110.0000
0. 0,9808 0.0000
0.9484=000000 0.000
0000
0.9484==00000
0.9868=000000
0.900
0.9232==0.0000
0.9232+=000000
0.9007
0.5381 0000066
0.5527=0000077
0.5409
0.6880=0.0000
0.4252
0.4785=0000075
0.6075:1000001 0.455000.0059
0.971=0000000 +.000
0.0000
0.9371 0.0000
0. 0.000
0.006610.0000
0.9103000000
0.90660000000
0.986410000000
17 0.9470=0000000
0.9018=
0.9466
0.9708:0000M0
0.9470=0000002
0.000
0.9466=000001
18
0.9895110.00000 0.980010.0000
0. 0.9909 10099100 0.0000
1.0000 1 0.0000
00.0000
0.9800 0.0001
0.0000
1.0000 0.0000
19
0.8203=t0.00390 0.8102
08896=00000700
0.0003
0.8075=#000022
0.7964
0.8165
0.0000
20
0.8442::000000+ 0.8421 0.000
0.84320000000
0.000
0.8389=#000000
0.8368
08379=+0000000
0.0020
21
0.0319110.0001 0.0271 0.0001
0.0276:10.0001
-.20731000033
0.0000010.0000
0.0062
0.0081 0.0000
0..9999-0.00039
22
0.4637-10.0111
0.4853=1.00888
0.4446
0.8485 0.0004
0.4112::000266
0.4318 14
0.3897
1.8484-100000 0.0000
23
10.0100
0.0075
0.0164
0.9519=000000
0.4194=00.0144
0.4706=00.0103
0.364
0.9518:10.0005
24
11803-+0.0023- 0.1522 0026
0.1014
0.3499
0. 0.0000=0.0007
0.0947:0.0012+
0.0901
0.31881100009
25
0.9205
0.1166=0.0048
0.9500=000004
0.9717=00000
0.916720.0043+
0.9108=0.003
0,9509
0.9717+1000000
into two parts based on different evaluation measures. The lowest value of each part in each row is underlined. Euclidean distances (dED) and information entropies (d1E) are defined:
dED(A,B)
(ai
bi))
(87)
i=1 1=1
my diE(A) Σ p)ylog(Pu)),
(88)
where aij and big are the elements in the i-th row and j-th column of matrices A and respecctvvel and Piş is the probability value of the i-th row and j-th column element matrix A [r][r]. Figure 14 17 presents the t-SNE of the similarity matrices of the feature representations for each method. As shown in the tables and figures, the CE-PSED method demonstrates superior performance in feature representation and class discrimination.
11.1.6. COMPARISON WITH BASELINE METHODS
In this section, we present a comparative analysis of the CE-PSED-basdd method against several existing baseline approaches across five image datasets, as summarized in Table 11. The results demonstrate that the CE-PSED-bers based method consistently achieves superior accuracy compared to the baseline methods, highlighting its effectiveness in diverse scenarios.
11.1.7. COMPARISON OF DIFFERENT NETWORK STRUCTURES
In addition, we use loss functions train more complex networks. The experimental results and detailed parameter configuration are as follows. In this study, we conducted experiments using hardware configurations including Intel (R) Core (TM) 17-14700F CPU, 16GB RAM, and NVIDIA GeForce RTX 4060 GPU. The experiment was conducted on the Windows operating system, with Python 3.10aa as the programming language and Py Torch 2.4 library for model development and training.
For training the Visual Transformer (ViT) (Dosovitskiy et al., 2020) model, we use a stochastic gradient descent (SGD)
25

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
020
2015
13
(a) Wine Quality
(b) BTSC
(c) Energy Efficiency
(d) TTTE
(e) OMN
(I) QB
0.-5 313
Pyass
Ppash
Ppash
(g) SHD
(h) Steel Plates (iults(i) Cardiotocography
(j) OLD
(k) SMT
(I) GGSWP
2
2
PCCS
AVD
Eaoch
Spech
Epech
(m) WDG
(n) Abalone
(o) PBC
(p) Mushroom
(q) Nursery
(T) Adult
Ranck
Opean
Panch
(s) Mpeg
(1) Mnist
(u) Pendigits
Figure 11. Accuracy curves hased on different loss functions when model layers is 3.
optimizer with learning rate set to 0.001. The batch size set to 64, and the model trained for total of 100 iteration cycles. The dataset is divided into training and testing subsets in ratio of 7:3. Similarly, the ConvNeXt (Liu et al., 2022) model was trained using the AdamW optimizer with a learning rate set to 0.004. The model also uses a batch size of 64 and is trained over 100 itcration cycles. Consistent with the VIT model, the dataset divided into training and testing sets, maintaining the same 7:3 ratio.
The performance evaluation of both models is conducted using accuracy and F-measure as the main measures. As shown in the table 12, the analysis of the results indicates that our proposed method has made significant improvements compared to existing methods. This demonstrates the effectiveness of our method.
11.1.8. COMPARISON OF RUNTIME FOR DIFFERENT LOSS FUNCTIONS
We conduct a runtime comparison, as shown in the table 13. For the first 20 benchmark datasets, the table presents the total time consumption (in seconds) for both training and testing. For image datasets, the recorded values represent the training and prediction time (in seconds) on the fully connected network shown in Figure 3, after feature extraction using either MoCo v3 VGG. From the table 13, it can be observed that method does significantly increase computational time.
26

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Table Euclidean distance and information entropy between similarity matrix and YY based on different loss functions when model
layers is 3.
Data
dED
dIz
CE
CE-SED
C'-inform
CE-PSED
CE
CE:SED
CE-inform
CE-PSED
76.1735
76.1735
77,5304
76,5485
37,0368
37,0368
37.0385
37,0500
2
118.0135
I 118.0135
1663866
106.0565
35,9276
35.9276
35.9308
35.8186
3
123.2038
123,2038
129.5480
110. 1027
35,4940
35,4940
35,5365
35,2933
4
162.3454
162.3454
159,9709
149.7737
37.2954
37.2954
37.3005
37,2043
173.1156
173.1156
175 2452245
154.3241
37,9086
37,9086
37,8876
37,7090
166-4008
166,4008
169, 8654
135
38 0959
38, 0959
38, 1075
37,9415
329.6840
329. 0840
332 4945
260.9751
874.8749
608.8749
8462
17
257.0381
257.0381
248. 1448
00715
5068
542.3536
9
399.3402
399,3402
390.1124
313.9013
359.1309
359.1309
359.0130
358.5208
10
254.0633
254.0633
247 581.
209 8767
44,2149
44.2149
44.2136
44.0434
UI
189.9269 9269
189, 9269
187.7281
180.3647
47,2859
47,2859
47,2887
47.2785
12 279.1663
279.1663
286.4833
276.5521
46,8595
46. 8595
46.8014
46 46.8092
311.5175 13
311.5175
309.4672
205.7426
49, 49,3400
49,3400
49,3379
49. 2880
14
336.9726
336.9726
335. 666699
219.6362
50,0835
50,0835
50.0865
50,0487
15
466.5055
466.5055
465.5321
444.39933
45.1351
45.1351
45,0850
45 2805
16
2421
338, 2421
381.2186
295 4883
52.8144
52.8144
52.8225
$2.8 146
895.9040
895.9040
845.0391
442.9792
53 23.2286
53,2286
53,2484
53,2705
517.6101
517.6101
$11.4658
223. 7286
54.0381
$4.0381
54,0401
54.0677
19
1734,1428
1734,1428
1674.9242
1809,1698
201. 6555
201 6555
201.6630
201.5141
20
4301.6353
4301.6353
4233.9531
4129.2095
66.0202
66.0202
65,9704
65.9258
21
233.7108
233,7108
239.6036
222.7377 7377
820.5886
820.5886
823.8793
825. 425.4946
22
1055.0093
1055.0093
1042.0280
919.8 1159
837 9047
837.9047
837.8022
837 8476
23
1187,5291
1187,5291
1180.5438
1043,7703
851 129
851.2291
2600
850.7767
24
1381.6508
1,508
1406.4387
1391,6364
29363. 1641
29363, 1641
29390,8047
29438 09438.4980
25
1755.2109
1755,2109
1602, 1602.9089
1572
962.3068
962.3068
162.1885
962, 1676
Table Euclidean distance and information entropy between similarity matrix and YY based on different loss functions when model layers is 5.
Data
dgD
CE
CE-SED
CE-inform
CE-PSED
CE
CE-SED
CE-inform
CE-PSED
79. 79.6850
78.5153
78 1867
78.6162
37,0328
37.0405
37.0459
37.0448
21.9233
124,8685
125.6604
113.0838
35.9199
35.9311
35,9453
35.9171
139.6086
139.7598
139.5939
112.8470
35,5393
35,5488
35,5446
35,4087
176.6676
177.8770
173.6018
142.6622
37,3264
37,2951
37.3265
37,2480
184,9006
180.9771
184.6760
1.2.6335
37.9203
37.9225
37,9254
37.7662
181. 1
186.9172
183.1280
141 9453
33.2
38, 1379
33.1374
37,9251
0222
350.8264
353 8266
270.3388
608.8852
609.0336
608.88
8669
1877
311.6756
321 5478
264.3753
541.0257
541 8883
542.0345
542 2803
451.0792
447.5596
452.1057 1057
332, 7613
359,3900
399.3643
359,3124
358 7696
10 272.9623
273.0852
270.0412
203
44,2175
44.2134
44,2204
1212
II
184.7525
186.9483
185.5223
201 9711
47. 47,2881
47.2837
47,2859
47.2416
12 269 4701
295.2358
286.0780
272.2967
46,8527
46.8941
46.8962
46.8753
13
314.
308.2535
310.5731
216.3729
49.3310
49,3399
49,3396
49,2533
14 338.29 138.2912
337.2201
335
214.1152
50.0805
50.0874
50,0830
50.0418
15
484,8264
475,2573
476.4166
452. 1942
45,3697
45,3445
45,2689
1519
16
370.6945
354.5944
945.9410
297.0087
52.8071
$2.8160
$2,8222
52.8105
829 2169
910.4088
802,9050
461.3801
53,2784
$3,2987
53,2917
$3,2539
486.7239
527.8088
569.7032
245.3424
54.0134
$4.0535
53.9989
$4,0640
1907. 5442
1875.5889
1901,7069
2104, 1467
201. 201.6993
201 1781
201.6916
201 6814
4519.4912
4370.4189
4509.1572
4183,5938
66.0612
66.0230
66.0429
65.9723
21
284.0164
279.8763
263.6331
238. 7444
826.9400
827.99 927.9311
825.7845
825 6633
22
1183.8347
1212.5712
1279.2507
1003.7283
838.1042
838.2512
837,7303
837.8041 8041
23
1321,9459 141.9949
1382.4373
1364.1901
1143.9182
851.9315
852,3873
852,1819
852.2212
24
1596.7644
16188 1927
1563.8016
1454.6886
39456.8047
29419.6973
29415.5566
29453, 1133
25
2040, 1495
2058. 1721
2063.2100
1792.5488
962.3511
962. 1644
962.4006
952.1599
27

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
0.75
0.6
0.7
Accuraay
Axcuuaacy
6:66
0.0
Epoch
Spoch
Spach
(a) Wine Quality
(b) BTSC
(c) Energy Efficiency
(d) TTTE
(e) OMN
0.5
0.8
0.0
0.7
Accunay
Asturmay
Bocumery
Acturny
5
SE.1
CC
200
CB
2000 2A60
0.2
3ED
05
CC- 230
PARO
93
90
60
60
10
20
30 Epoch
10
23
10
50
10
25
Epoch
Epoch
(f) QB
Yeast
(h) SHD
11 Steel Plates Faults
(i Cardiotocography
0.5
0.53
Accurscy
Accurmay
Rccuretyy
0.6
28--5EE SED
OLL APP
F. 200
AED
PS20
E8
738.
06-
PS20
0.45
12
20
50
02
10
20
20
42
50
10
23
62
10
20
52
22
40
Eposh
Epoch
Epoch
Epoch
Eposh
(k) OLD
(1) SMT
(m) GGSWP
WDG
Abalone
0.23
ACCUNNC
Accunery
Accurnry
0.25
2.0
0.03
AED PS30
SED
NA8D
CD
70ED
1522
<2 7500
0.4
30
0.6
22
0.2
10
50
0.53
Epoch
Epech
Epoch
Epech
Epoch
(p) PBC
(q) Ringnorm
(r) Mushroom
(s) Nursery
(t) Adult
0.6
Acaurccy
0.2
CD
0.1
reform
0.2
0.1
02
10
50
80
10
20
52
Epach
Epoch
Epoch
Epach
Epoch
(u) Mpeg
(v) Mnist
(w) Pendigits
(x) Caltoch
(y) ImageNet
Figure 12. Accuracy curves based on different loss functions when model layers is 5.
28

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
S
0.
Epoch
Spach
(a) Wine Quality
(b) BTSC
(c) Energy Efficiency
(d) TTTE
(c) OMN
0.5
83
115
0.8
0.23
2.6
0.5
0.7
Asturery
Accurery
SED
0.15
380 P000
CB
200 2460
300
06
CC-
230
PARO
93
90
60
60
10
23
10
50
10
28
Epoch
Epoch
(f) QB
Yeast
(h) SHD
11) Steel Plates Faults
(i Cardiotocography
0.0
0.53
0.6
Alcurraty
Accurmay
Accurney
0.6
OLL APP
0.5
FX 900
AED
Dr- AND
PS20
E8
738.
C2- ---2-7500
0.5
0.45
12
20
00
50
02
10
20
25
42
50
10
23
10
20
52
25
Epooh
Epoch
Epoch
Epooh
Eposh
(k) OLD
(1) SMT
(m) GGSWP
WDG
Abalone
88
Acturtoy
Accuenry
0.8
0.4
AEEO PSD0
SED
0.4
70ED
1520
21. 7580
0.6
12
30
22
10
50
Epoch
Epech
Epoch
Epech
Epoch
(p) PBC
(q) Ringnorm
(r) Mushroom
Nursery
(t) Adult
Accurety
Acaurecy
CT
0.1
0.2
reform
02
3.00
02
PA80
40
50
32
52
Epach
Epoch
Epoch
Epoch
Epoch
(u) Mpeg
(v) Mnist
(w) Pendigits
(x) Caltech-101 101
(y) ImageNet
Figure 13. Accuracy curves based on different loss functions when model layers is 8.
29

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Table 10. Euclidean distance and information entropy between similarity matrix and YY based different loss functions when model
layers is 8.
Data
deD
diz
CE
CE-SED
CF-inform
CE-PSED
CF
CE-SED
CF-inform
CE-PSED
78.5572
77.6711
77,5050
76.6356
37.0452
37.0449
37,0509
123,3103
124.2150
124.3540
119.5258
35,9527
35.9524
35,9383
35,9547
143,4134
142,9109
142.7595
125.0313
35,5476
35.5408
35.5183
35.5133
177.0358
175.9595
177.6065
142.1926
37.3276
37.3231
37.2873
37.1813
184.6225
186.0866
187.0768
158. 6200
37,9243
37,9152
37.9159
37.8397
192.7056
191.0630
194,8601
147 8844
38.1393
38. 1397
38, 1391
37.6 37.63944
351 8983
354,9803
352.5111
315
609.0766
609 0687
609.1129
608 3528
397.7234
347.8330
399 9164
256. 9694
542.7438
542 2402
542.5559
541 868]
460.3625
464.0630
466.8538
0200
359.4289
359,3731
359,4217
358 4846
273 0928
212.3560
276.0269
44,2209
44.2218
44,2104
184.6531
188.8407
187.7007
27.1
47.2871
47.2887
2832
312
312.4441
311.5305
185 2752
19.3347
49,3400
49,3412
49.3 19.3174
335 2605
333.6607
335.5419
203 2962
50,0845
50.0865
50.0830
50.0666
15
510.7383
517.9683
500 500.6602
469 6039
45 4402
45 45.4203
45,4393
45. 3898
16
516.2045
515.2233
500.0887
297.2825
52,8287
52.8268
$2,8293
52.7993
17
941.3668
688.6004
876.5587
484.4414
53,2947
$3.2633
$3.2934
53.2362
655.9843 18
51 1,2377
190.1942
416.2412
54.0622
54.0526
$4,0665
54.0659
2047.2732
2357,8250
1974.6055
2180.1816
201.6154
201.7770
201,6311
201 6664
4545. 1484
4479.099 0952
4472.4590
4337,5742
66.0634
66.0526
66.0532
65.9778
313.1884
294,9319
302.1780
243.5456
829.1549
820.5233
825.6597
815.4856
1269.7419
1149,5310
836.5143
1477.7767
2053.9851
1847.4176
1824.1750
1530.5894
29493.08200
29490.4590
29429 0820
29470. 7949
2384.8987
2204.3345
2293.9170
1886.0833
961.9738
962.0734
962.1173
961 9856
(a) CE
(b) CE-SED
(c) CE-inform
(d) CE-PSED
Figure 14. The 1-SNE of Mpeg.
CE
(b) CE-SED
(c) CE-inform
(d) CE-PSED
Figure 15. The 1-SNE of Mnist.
30

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
(a) CE
(b) CE-SED
(c) CE-inform
(d) CE-PSED
Figure 16. The t-SNE of Caltech-101.
CE
(b) CE-SED
(c) CE-inform
(d) CE-PSED
Figure 17. The t-SNE of ImageNet.
Table 11. Comparison with baseline methods in Accuracy
Data
Baseline Citation
Accuracy
CE-PSED
Mpeg
(Bai al., 2009)
0.5200
0.7338
(Grigorescu & Petkov, 2003)
0.5000
Mnist(6996 pes)
(Ren et al., 2016)
0.7946
(He Sun, 2015)
0.8073
(Goodfellow al., 2013)
0.8257
0.9062
(Hinton al., 1999)
0.8768
(7000(70000 pcs)
(Byerly al., 2021)
0.9987
Pendigits
(McConville et al., 2021)
0,8850
(Li et al., 2021)
0.8227
(Toth & Oberhauser, 2020)
0.9550
0.9908
(Cai & Chen, 2015)
0.8155
(van der Maaten & Hinton, 2008)
0.8930
Caltech- 101
(Bansal al., 2021)
0,4500
(Bansal al., 2021)
0.4400
0.5005
(Chen & Guestrin, 2016)
0.5000
(Irlc & Kauschke, 2011)
0.5000
ImageNet
(Chen et al., 2023)
0.9094
(Yu et al., 2022)
0.9100
0.9762
(Wortsman al., 2022)
0.9098
(Pham et al., 2021)
0.9020
31

---

Stabilizing Sample Similarity in Representation via Mitigating Random Consistency
Table Performance Comparison hetween Network Structures
Model
Dataset
Accuracy
F-measure
Original
PSED
Original
PSED
Mpeg
0.8238
0.8310
0.8170
0.8280
Mnist
0.9904
0.9925
0.9900
0.9930
ConyNeXt
Pendigits
0,9708
0.9832
0.9699
0.9826
Caltech-101
0.6429
0.6536
0.6100
0.6230
ImageNet
0.9651
0.9687
0.9650
0.9695
Mpeg
0.6820
0.7095
0.5515
0.6811
Mnist
0.8590
0.9290
0.8550
0.9277
VIT
Pendigits
0,9568
0.9711
0,9512
0.9700
Caltech-101
0.7393
0.8500
0.6034
0.7866
ImageNet
0.9964
0.9968
0.9952
0.9973
Table 13. Comparison of runtime (seconds) for different loss functions
Dataset
CE
CE-SED
CE-inform
CE-PSED
2.34
3.07
3.49
6.03
2
2.35
3.07
3.44
8.94
3
2.44
3,93
3.57
9.14
4
2.88
8.70
4,47
11.13
5
3.23
9.41
4.83
12.02
6
3.33
9.65
5.03
6.46
6.44
13.33
12.58
8.96
10.06
27.73
12.09
9
13.75
7.62
18.54
11.58
10
13.72
7.62
8.96
9,31
11
10.30
10.25
11.59
11.81
12
9.99
12,97
15.32
15.93
13
9.98
12.94
14.90
15.39
10.93
14.20
16.35
27.15
8.29
12.74
13.26
16
15.52
30.26
33.69
24.15
17
22.06
28.64
33.89
35.15
18
34.30
31.46
37.09
38.56
19
38.57
49,95
63.52
68.38
20
96.47
126.55
149.74
155.72
Mpeg
8.18
9,94
16.36
18.43
Mnist
27.39
35.20
51.52
55.51
Pendigits
22.40
29.61
45.99
50.55
Caltech-100
32.00
42.59
70.91
80.32
ImageNet
46.86
58.76
86.41
96.38
32


Reviews:
[{"review_id": "WQREHOfKgp", "reviewer": "Reviewer_72cG", "summary": "This paper proposes a loss function for image classification. It follows the idea of promoting better representation learning and proposes an improvement on mitigating the random consistency of existing methods. Properties such as unbiasedness and generalization bounds are theoretically investigated. Empirical evaluation of both accuracy and f-measures are conducted on multiple image datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 72cG,\n\nWe are very grateful for your valuable comments and questions. The responses are as follows:\n\n**Response to Claims And Evidence:** The proposed loss function serves as a universal quality measure for similarity matrices, which are foundational elements across various learning paradigms. Its generalized formulation enables applications in training deep network, metric learning, kernel learning and so on. To demonstrate the broad applicability, we applied it to train advanced network architectures (ViT and CNN). The experimental results and detailed settings are listed in our response to Reviewer WSCE Weakness 1. \n\n**Response to Experimental Designs Or Analyses:** By incorporating the random consistency mitigation mechanism (Eq. 5), we have enhanced the homogeneity and unbiasedness of the loss function, thereby improving the model's generalization capability. Significance tests demonstrate that this improvement maintains stable performance advantages across different data partitions, confirming that the enhancement is not attributable to random factors. \n\n**Response to Weaknesses 1:** This is an excellent suggestion. To enhance completeness and readability, we will add both a conceptual introduction to RC and an expanded literature review in the appendix of the next edition.  For further details, please see Question 3.\n\n**Response to Weaknesses 2:** This is a constructive suggestion. Indeed, the opening paragraph of Section 6 has some deficiencies in clarifying the motivation and scope behind our choice of network architecture. In fact, to validate the effectiveness of the loss function, we deliberately opted the proposed loss function for the most basic fully connected network structure. As mentioned earlier, our proposed loss function has relatively broad applicability. In the revised version, we will explicitly elaborate on the scope and adaptability of the method in section 6.\n\n**Response to Question 1:** The dependency of existing methods on uninformative matrices can be derived from Eq. 1, since  they measure the information content by computing the distance from uninformative matrices (e.g., identity matrix or full one matrix).  \n\n**Response to Question 2:** The motivation of this paper is to propose a loss function ($d_{PSED}$) for measuring the information content of similarity matrices when ground-truth labels are available. The shortcoming of $d_{infor}(\\mathbf{K})$ lies in its fundamental inability to properly assess the block-diagonal structure of similarity matrices, as evidenced by Property 1. To address this issue, $d_{PSED}$ introduces the ground-truth adjacency matrix as the reference standard, thereby enabling quantitative assessment of how closely the similarity matrix approximates the ideal block structure. The limitation of $d_{SED}(\\mathbf{K})$ manifests as differential biases toward identity matrix $I$ versus all-ones matrix $J$ across varying class distributions. Through mitigating random consistency, we obtain $d_{PSED}$, a weighted element-wise dissimilarity metric for similarity matrices that achieves unbiased evaluation of both $I$ and $J$. The $d_{PSED}$ mitigates the random consistency issue by subtracting the expectation of $d_{SED}(\\mathbf{K})$ under random label permutations (Eq.5). \n\n**Response to Question 3:** Consistency metrics measure the agreement between two random variables, while random consistency(RC) refers to spurious agreement arising purely from randomness[1]. A canonical manifestation of RC occurs when examinees achieve measurable test scores solely via random response patterns. The mechanisms by which RC harms the learning process include evaluation distortion, optimization misguidance, and generalization barriers.  Failure to deduct the RC baseline may lead to overestimating the model’s actual consistency performance (e.g., an original consistency score of 0.6 vs. a random baseline of 0.2 means the true effective consistency should be 0.4). When loss functions include RC without proper correction, they can induce optimization bias, causing algorithms to spuriously improve consistency metrics by overfitting to noise [2] or data bias[3,4] instead of learning genuine data patterns.  These would consequently impair the model's generalization ability.\n\n[1]Wang J, Qian Y, Li F, et al. Generalization performance of pure accuracy and its application in selective ensemble learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(2): 1798-1816.\n\n[2]Wang J, Qian Y, Li F. Learning with mitigating random consistency from the accuracy measure. Machine Learning, 2020, 109: 2247-2281.\n\n[3]Li J, Qian Y, Wang J, et al. PHSIC against random consistency and its application in causal inference. IJCAI, 2024, 5(10): 15-20.\n\n[4]Vinh N X, Epps J, Bailey J. Information theoretic measures for clusterings comparison: Variants, Properties, Normalization and Correction for Chance. Journal of Machine Learning Research, 2010, 11: 2837–2854.", "claims_and_evidence": "The first claim of the loss function proposal is well supported by the following contents.\nThe second claim of theoretical investigation is supported by section 4 and 5.\nThe last claim of a model proposal is also supported, but may narrow application scenes of the proposed method.", "methods_and_evaluation_criteria": "The proposed methods aim to solve multi-class classification problems. The evaluation criteria of accuracy and f-measures are suitable, theoretical investgations also show indirect evidence for the proposed method.", "theoretical_claims": "I checked the section 4 and 5 for theoretical claims.", "experimental_designs_or_analyses": "Significance tests seems to evaluate the motivation of the proposed method, that is to mitigate the random consistency of existing methods. It would be nice to elaborate more on the connection between the significance tests and the motivation of the proposed method.", "relation_to_broader_scientific_literature": "This paper relates to the broad literature of classification and loss function design. It also draws attention on representation learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- It is good to also discuss computation complexity of the proposed loss term.\n\nWeakness\n- This paper is not clear for some concepts and shows some difficulty for readers with less literature knowledge to follow.\n- Section 6 seems to heavily restrict the selection of network architecture, hindering further applications.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Existing methods seem to heavily rely on the design and selection of non-informative matrices. Is there any general summary on this point?\n2. Please elaborate more on the motivation of the proposed loss function. How it \"can address the shortcomings of dinfor(K) and dSED(K)\" and how it can mitigate random consistency?\n3. It says random consistency has been shown to exists in consistency measures. What is the detailed definition for random consistency, how is harms learning and how it exists in consistency measures?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WCojAQf6e3", "reviewer": "Reviewer_GtCV", "summary": "The manuscript introduced a new sample similarity measure. The main difference with the existing sample similarity measure is that the new measure mitigates random consistency. The measure forces class-level discrimination. Several theoretical results regarding the measure have been introduced (quality of stohastic estimation, heterogeneity, unbiasedness). Experiments with real-world datasets show that the measure improves classification quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GtCV,\n\n&nbsp;&nbsp;&nbsp;&nbsp; We are very grateful for your valuable comments and questions. The responses are as follows:\n\n**Response to Methods And Evaluation Criteria:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; For the image dataset, we firstly employed feature extractors such as MoCo v3 or VGG for processing, with downstream classification processing handled by fully connected layers. The specific methodology is detailed in Section 7.2 of the main text and Appendix 3.0.1. We note that there may be a typesetting oversight in Figure 3, where images were mistakenly labeled as direct inputs. This issue will be corrected in the revised version. Besides, in this modification, we implement our loss function on a more sophisticated network architecture(ViT and CNN). The experimental results and detailed settings are listed in our response to Reviewer WSCE Weakness 1.\n\n**Response to Experimental Designs Or Analyses:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; We adopted random splitting to generate the training and test sets, as described in Appendix 3.0.1. \n\n**Response to Other Comments Or Suggestion 1:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; The consistency metric quantifies the degree of agreement or alignment between different variables. Random Consistency (RC) refers to agreement or similarity that occurs purely by chance, not because of any true underlying pattern or relationship. When the sample size is limited or the noise level is high, the random consistency (RC) between the classifier's predicted labels and the true labels increases [1].  For more details on RC, please refer to Question 3 of Reviewer GtCV.\n\n[1] Jieting Wang, Yuhua Qian, Feijiang Li. Learning with Mitigating Random Consistency from the Accuracy Measure, Machine Learning, 2020, 109: 2247-2281 \n\n**Response to Other Comments Or Suggestion 2-4:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; $K$, $I$, and $J$ are defined in Equation (1), though we will include their definitions after each subsequent formula. Thank you for your suggestion. Thank you for pointing these out and we will modify them accordingly.\n\n**Response to Question 1:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; CE-PSED refers to  the summation of CE and PSED. Thank you for pointing this and we will add this in the final version. \n\n**Response to Question 2:** \n\n&nbsp;&nbsp;&nbsp;&nbsp;  This is indeed a profound question. As illustrated in Figures 7 and 9 in the appendix, the t-SNE visualization of cross-entropy (CE) for the MPEG and Caltech-101 datasets fails to reveal clear inter-class structures. By introducing the block-diagonal constraint on the similarity matrix—that is, incorporating PSED as part of the loss function—we can enhance inter-class separation in data representation, thereby improving model performance, shown as Table 7 in Appendix. We will add this explanation in the final version.\n\n**Response to Question 3:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; For the image datasets, we firstly employed MoCo v3 or VGG for feature extracting, with downstream classification processing handled by fully connected layers. The specific methodology is detailed in Section 7.2 of the main text and Appendix 3.0.1.", "claims_and_evidence": "Main claims of the paper are valid (see below).\n\n```\n• A loss function for measuring the ability of the representation layer is proposed, and an explicit solution for the loss function in the version of eliminating random consistency is given.\n• Through theoretical analysis, the advantages of this metric in heterogeneity and unbiasedness have been demonstrated, and a generalization bound has been provided for the generalization performance of the loss function in fully connected layer network structures.\n• A fully connected network classification model based on this loss function was proposed, and the effectiveness of the algorithm was verified through extensive experiments.\n```", "methods_and_evaluation_criteria": "If I understood correctly, for image datasets like ImageNet, Caltech-101 you used fully-connected network.\nThis is problematic because convolutional architectures are typically applied for images.", "theoretical_claims": "No", "experimental_designs_or_analyses": "Experiment design is mostly correct. Which train/test splits you have used?", "relation_to_broader_scientific_literature": "The manuscript is related to a broader literature of representational similarity measures.", "essential_references_not_discussed": "--", "other_strengths_and_weaknesses": "It is interesting that you have improved standard cross-entropy loss.", "other_comments_or_suggestions": "1. > \"Random consistency (RC), resulting from randomness rather than true similarity, often arises in consistency evaluation, particularly with limited samples or noise\" \n\nthis sentence is not clear. Can you please clarify it?\n\n2. K in eq.3 is not defined\n3. line 183, Property 3. Matrices I, J are not defined.\n4. lin 276, typo: \"symmilaritym atrix\"", "questions_for_authors": "1. CE-PSED metric is never defined explicitly, only CE and PSED losses separately.\n2. Why the performance gap between CE and CE-PSED is large for some datasets like Mpeg of Caltech-101 ?\n3. Which neural architecture you have used for the ImageNet dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "--"}}, {"review_id": "hRmXVlNhnq", "reviewer": "Reviewer_WSCE", "summary": "This paper addresses the challenge of measuring and improving representation quality in deep learning models. The authors propose a novel loss function called Pure Square Euclidean Distance (PSED) that measures the discriminative ability of representations by computing the Euclidean distance between a similarity matrix and the true adjacency matrix.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WSCE,\n\n&nbsp;&nbsp;&nbsp;&nbsp; We are very grateful for your valuable comments and questions. The responses are as follows:\n\n**Response to Weaknesses 1:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; For the image dataset, we solely employ VGG and MoCo v3 for feature extraction. The specific methodology is detailed in Section 7.2 of the main text and Appendix 3.0.1. In this modification, we trained more sophisticated networks (ViT and CNN) using our loss function. The experimental results and detailed parameter configurations are as follows.\n\n&nbsp;&nbsp;&nbsp;&nbsp; In this study, we conduct experiments utilizing a hardware configuration that includes an Intel (R) Core (TM) i7-14700F CPU, 16GB of RAM, and an NVIDIA GeForce RTX 4060 GPU. The experiments are carried out on a Windows operating system, with Python 3.10 serving as the programming language and the PyTorch 2.4 library employed for model development and training.\n\n&nbsp;&nbsp;&nbsp;&nbsp; For the Vision Transformer (ViT) [1] model training, we use a stochastic gradient descent (SGD) optimizer with a learning rate set to 0.001. The batch size is set to 64, and the model is trained for a total of 100 epochs. The dataset is partitioned into training and testing subsets with a ratio of 7:3.\n\n&nbsp;&nbsp;&nbsp;&nbsp; Similarly, the ConvNet [2] model employs the AdamW optimizer, with the learning rate set to 0.004. This model also uses a batch size of 64 and undergoes training for 100 epochs. In alignment with the ViT model, the dataset is divided into training and testing sets, preserving the same 7:3 ratio.\n\n&nbsp;&nbsp;&nbsp;&nbsp; Performance evaluation of both models is conducted using accuracy and F-measure as the primary metrics. Analysis of the results, as presented in the table, our method shows better performance over existing approaches. This indicates the effectiveness of our method.\n\n| Method| ConvNet| ConvNet-PSED | ViT| ViT-PSED |\n|- |- |- |- |- |\n|**Accuracy**\t|\t \t|\t \t|\t \t|\t \t|\n|Mpeg \t|\t0.8238\t|\t**0.8310**\t|\t0.6820\t|\t**0.7095**\t|\n|Mnist |\t0.9904\t|\t**0.9925**\t|\t0.8590\t|\t**0.9290**\t|\n|Pendigits |\t0.9708\t|\t**0.9832**\t|\t0.9568\t|\t**0.9711**\t|\n|Caltech-101   \t|\t0.6429\t|\t**0.6536**\t|\t0.7393\t|\t**0.8500**\t|\n|ImageNet       \t|\t0.9651\t|\t**0.9687**\t|\t0.9964\t|\t**0.9968**\t|\n|**F-measure**\t|\t \t|\t \t|\t \t|\t \t|\n|Mpeg\t|\t0.8170\t|\t**0.8280**\t|\t0.5515\t|\t**0.6811**\t|\n|Mnist |\t0.9900\t|\t**0.9930**\t|\t0.8550\t|\t**0.9277**\t|\n|Pendigits  |\t0.9699\t|\t**0.9826**\t|\t0.9512\t|\t**0.9700**\t|\n|Caltech-101   \t|\t0.6100\t|\t**0.6230**\t|\t0.6034\t|\t**0.7866**\t|\n|ImageNet       \t|\t0.9650\t|\t**0.9695**\t|\t0.9952\t|\t**0.9973**\t|\n\n[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.\n\n[2] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, et al. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11966–11976, 2022.\n\n**Response to Weaknesses 2:** \n\n&nbsp;&nbsp;&nbsp;&nbsp; Thank you for your suggestions. We have added a runtime comparison, as shown in the table below. In this study, we conduct experiments utilizing a hardware configuration that includes an Intel (R) Core (TM) i7-12700F CPU, 32GB of RAM, and an NVIDIA GeForce RTX 3060 Ti GPU. For the first 20 benchmark datasets, the table below presents the total time consumption (in seconds) for both training and testing. For image datasets, the recorded values represent the training and prediction time (in seconds) on the fully connected network shown in Figure 3, after feature extraction using either MoCo v3 or VGG. From the table, it can be observed that our method does not significantly increase computational time.\n\n|\tDataset\t|\tCE\t|\tCE-SED\t|\tCE-inform\t|\tCE-PSED\t|\n|--|--|--|--|--|\n|\t1\t|\t2.34\t|\t3.07\t|\t3.49\t|\t6.03\t|\n|\t2\t|\t2.35\t|\t3.07\t|\t3.44\t|\t8.94\t|\n|\t3\t|\t2.44\t|\t3.93\t|\t3.57\t|\t9.14\t|\n|\t4\t|\t2.88\t|\t8.7\t|\t4.47\t|\t11.13\t|\n|\t5\t|\t3.23\t|\t9.41\t|\t4.83\t|\t12.02\t|\n|\t6\t|\t3.33\t|\t9.65\t|\t5.03\t|\t6.46\t|\n|\t7\t|\t6.44\t|\t13.33\t|\t12.58\t|\t8.96\t|\n|\t8\t|\t12.81\t|\t10.06\t|\t27.73\t|\t12.09\t|\n|\t9\t|\t13.75\t|\t7.62\t|\t18.54\t|\t11.58\t|\n|\t10\t|\t13.72\t|\t7.62\t|\t8.96\t|\t9.31\t|\n|\t11\t|\t10.3\t|\t10.25\t|\t11.59\t|\t11.81\t|\n|\t12\t|\t9.99\t|\t12.97\t|\t15.32\t|\t15.93\t|\n|\t13\t|\t9.98\t|\t12.94\t|\t14.9\t|\t15.39\t|\n|\t14\t|\t10.93\t|\t14.2\t|\t16.35\t|\t27.15\t|\n|\t15\t|\t8.29\t|\t10.76\t|\t12.74\t|\t13.26\t|\n|\t16\t|\t15.52\t|\t30.26\t|\t33.69\t|\t24.15\t|\n|\t17\t|\t22.06\t|\t28.64\t|\t33.89\t|\t35.15\t|\n|\t18\t|\t34.3\t|\t31.46\t|\t37.09\t|\t38.56\t|\n|\t19\t|\t38.57\t|\t49.95\t|\t63.52\t|\t68.38\t|\n|\t20\t|\t96.47\t|\t126.55\t|\t149.74\t|\t155.72\t|\n|\tMpeg\t|\t8.18\t|\t9.94\t|\t16.36\t|\t18.43\t|\n|\tMnist\t|\t27.39\t|\t35.2\t|\t51.52\t|\t55.51\t|\n|\tPendigits\t|\t22.4\t|\t29.61\t|\t45.99\t|\t50.55\t|\n|\tCaltech-101\t|\t32\t|\t42.59\t|\t70.91\t|\t80.32\t|\n|\tImageNet\t|\t46.86\t|\t58.76\t|\t86.41\t|\t96.38\t|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, the experimental evaluation is extensive, covering both benchmark and image datasets, and it demonstrates the method's effectiveness.", "theoretical_claims": "Yes, The paper provides solid mathematical analysis of PSED's properties, including formal proofs of heterogeneity and unbiasedness, and establishes generalization performance bounds using exponential Orlicz norm-based concentration inequalities.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weakness:\n\n1. The experiments focus primarily on fully connected networks instead of exploring more complex architectures, such as CNNs or Transformers, which are common in modern deep learning.\n2. PSED still introduces additional computational complexity compared to the standard CE loss, especially for large batch size training. Additional computation cost analysis should be conducted.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All three reviewers provided positive ratings after the rebuttal stage, with one reviewer upgrading their initial score. The initial concerns primarily centered on the completeness of ablation studies, implementation details, and the need for additional experimental results. During the post-rebuttal discussion phase, the reviewers expressed satisfaction with the authors’ responses and the revised manuscript. Upon careful examination of the paper, rebuttal, and reviewer discussions, the AC concurred with the reviewers’ assessment, acknowledging the soundness and effectiveness of the proposed method and subsequently recommending acceptance.

However, the manuscript still requires substantial revision—particularly in terms of writing quality and clarity and more comprehensive experimental comparisons.
Decision: Accept (poster)