Stray Intrusive Outliers-Based Feature Selection on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Authors: Lixin Yuan
Yirui Wu
Wenxiao Zhang
Minglei Yuan
Jun Liu
Venue: ICML 2025 poster (2025)
Keywords: Feature selection
stray intrusive outliers
refined density-mean center
intra-class asymmetric instance distribution or multiple high-density clusters
data classification
Primary area: general_machine_learning->supervised_learning
TL;DR: This paper proposes the stray intrusive outliers-based feature selection method for high-dimensional data classification with intra-class asymmetric instance distribution or multiple high-density clusters.

Abstract: For data with intra-class Asymmetric instance Distribution or Multiple High-density Clusters (ADMHC), outliers are real and have specific patterns for data classification, where the class body is necessary and difficult to identify. Previous Feature Selection (FS) methods score features based on all training instances or rarely target intra-class ADMHC. In this paper, we propose a supervised FS method, Stray Intrusive Outliers-based FS (SIOFS), for data classification with intra-class ADMHC. By focusing on Stray Intrusive Outliers (SIOs), SIOFS modifies the skewness coefficient and fuses the threshold in the 3$\sigma$ principle to identify the class body, scoring features based on the intrusion degree of SIOs. In addition, the refined density-mean center is proposed to represent the general characteristics of the class body reasonably. Mathematical formulations, proofs, and logical exposition ensure the rationality and universality of the settings in the proposed SIOFS method. Extensive experiments on 16 diverse benchmark datasets demonstrate the superiority of SIOFS over 12 state-of-the-art FS methods in terms of classification accuracy, normalized mutual information, and confusion matrix. SIOFS source codes is available at https://github.com/XXXly/2025-ICML-SIOFS

Stray Intrusive Outliers-Based Feature Selection on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Lixin Yuan 12
Yirui Wu
Wenxiao Zhang'
Minglei Yuan"
Jun Liu
Abstract
500
For data with intra-class Asymmetric instance Distribution Multiple High-density Clusters (ADMHC), outliers are real and have specific pat- terns for data classification, where the class body is necessary and difficult to identify. Previous Fea- ture Selection (FS) methods score features based all training instances or rarely target intra-class ADMHC. In this paper, we propose supervised FS method, Stray Intrusive Outliers-based FS (SIOFS), for data classification with intra-class ADMHC. By focusing on Stray Intrusive Out- liers (SIOs), SIOFS modifies the skewness coeffi- cient and fuses the threshold the 3a principle to identify the class body, scoring features based the intrusion degree of SIOs. In addition, the refined density-mean center is proposed to repre- sent the general characteristics of the class body reasonably. Mathematical formulations, proofs, and logical exposition ensure the rationality and universality of the settings in the proposed SIOFS method. Extensive experiments on 16 diverse benchmark datasets demonstrate the superiority of SIOFS over 12 state-of-the-art FS methods in terms of classification accuracy, normalized mu- tual information, and confusion matrix. SIOFS source codes is available at https: //gi thub. com/xxx1y//// 220xxxyyy222552 2025-0022222222 ICML- SIOFS
School Resort
200
-300
(a)
(b)
4 4 4
6
4 4
9 4 4
9 9 9
(c)
(d)
(e)
Figure 1. (a) Similar scene images "Resort" and "School" on
AID. (b) 2D visualization of classes "I" and "3" from CLL via
T-SNE (2012). (c) Similar digits "4" and "9" on GISETTE. (d) Distance histogram of instances to center for class "3" CLL. (e) Distance histogram of instances to center for class "I" on TOX.
generated (Li et al., 2024; Wang et al., 2024). Feature Se- lection (FS), the selection discriminative feature subset, is an essential part of data processing (Cohen et al., 2023). FS makes the representations of instances interpretable and enables fast and compact models to be learned.
High-dimensional data classification with intra-class Asym- metric instance Distribution or Multiple High-density Clus- ters (ADMHC) is challenging in machine learning, espe- cially for small-sized datasets. such tasks, misclassified outliers inevitably exist. Moreover, some outliers intrude into other class bodies. As shown in Fig. Ia, some instances of "Resort" share very similar struetural and textural char- acteristics with instances of "School". Similarly, Fig. Ic shows that some instances of "4" and "9" are very similar.
1. Introduction
With the rapid advancement of data acquisition technolo- gies, increasing amount of high- high-dimensional data is being
"College of Computer Science and Software Engineering, Ho- hai University, Jiangsu, China "State Key Lab. for Novel Software Technology, Nanjing University, P.R. China 'Hefei Institute for Advanced Research, Anhui University of Finance and Economics, Anhui, China 'School of Computing and Communications. Lan- caster University, Lancaster, United Kingdom. Correspondence to: Yirui Wu wuyiiui hhu.edu.cn.>.
In Fig. 1b, x(") X2 (1) are three different types of outliers from class "1", where x(" intrudes into the body of class "3". and x, (1) is near the body of class "3". Compared to x2 the class label of x is easily mispredicted as "3" by the predictive model. Focusing on x can reasonably iden-
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Stray Intrusive Outliers-Basee FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
tify low-ranked features for classification tasks. We define such instances like x[" as Stray Intrusive Outliers (SIOs), which are far away from the class body they belong to and intrude into other class bodies. Most current F$ methods score features based on the characteristics of all training instances (Roffo al., 2021).
rationality and universality of the SIOFS method.
1.1. Brief Related Work
FS methods select subset of features from high- dimensional data to improve data compactness and reduce noisy features, in particular, to alleviate the overfitting, high computational cost, and low-performance issues (Nie et al., 2022; Cohen et al., 2023). There are three main FS methods: filter, wrapper and embedded (Tang et al., 2014). Filter methods evaluate each feature according to the intrinsic characteristics of the data. Wrapper and embedded meth- ods are prone to overfitting because their selection is part of training. Most FS methods treat each instance equally during selection, rarely focusing on the outliers that lead to misclassification, and score features according to the degree of this misclassification. The FSDOC (Yuan et al., 2022) method focuses on instances within one class that are out- liers in the direction of another class. The IOFS (Yuan et al., 2024) method further explores the outliers close to the other class. But importantly, both FSDOC and IOFS fail to ad- dress the FS problem in the context of ADMHC. Additional related work is shown in Appendix A.
Another important characteristic of high-dimensional data classification is intra-class multiple high-density clusters. As shown in Fig. Ib, class "3" has two high-density clusters. In such a class, necessarily, multiple distinct peaks exist in the distance distribution of instances to the class center (see Fig. Id and le), making it difficult to determine the class center and body. Existing FS methods rarely aim to identify the class body in the context of intra-class multiple high-density clusters.
With few outliers, the mean center can reflect the overall characteristics of the class body well (Lim & Kim, 2021). Clustering by Fast search and find of Density Peaks (CFDP) (Rodriguez & Laio, 2014) acquires density-based clus- tering center. However, the CFDP center is sensitive to local high density formed by a small number of instances in class. The threshold in the 30 principle of the normal distribution can identify the class body, but it requires the in- stance distribution to be centrally symmetric. The Skewness Coefficient (SC) measures the degree of asymmetry in data distribution (Vapnik, 1995), but it cannot be directly used to identify the class body. Intuitively, two classes intersect if an instance of one class intrudes into the body of another class. A general conclusion is that the sum of the radii of two hyperspheres is greater than the distance between the class centers when they intersect, which motivates us to quantify the intrusion degree of SIOs.
2. Refined Density-Mean (RDM) Center
Notations. Throughout this paper, the boldface capital let- ters (e.g., X) denote matrices, the boldface lowercase letters denote vectors (e.g., x), and the italic letters are scalars (c.g., X, a). n, d, c denote the total numbers of training instances, feature dimensions, and classes in a dataset, re- spectively. Given a dataset, for class k = 1, 1..,.... C, the instance set is (²()) =
(k) (k) x(1) ), where x; (k: -xx(x) X2
(- 22 ,... (K)) E Rdd denotes the ith instance, i = 1, ng. and nk is the number of training instances. To assess each feature equally, we use the & norm-based distance (Yuan et al., 2022). For example, the distance be- tween two instances x! obtained by
(R) F2d
In this paper, our main contributions are follows. (i) We propose Stray Intrusive bassed Feature Selection (SIOFS) method for the high-dimensional data classification with ADMHC. (ii) We present a new definition of instance density. A Refined Density-Mean (RDM) center is proposed to characterize the class body of different types of instance distributions. For the distance of intra-class instances to the center, the modified SC is fused into the threshold in the 3o principle, reasonably identifying the class body. The for- mulations about the intrusion degree of SIOs are provided. Mathematical proof's or logical expositions of the settings in SIOFS are complete. (iii) Extensive comparisons on 16 various benchmark datasets show that, for selecting dis- criminative features, SIOFS outperforms 12 state-of-tte of-t-----arrrr art methods with higher classification accuracies, normalized mutual information, and confusion matrices.
(8) and (8) (j (;=1,2..... n2) can be
||x (K)
X
111
111
d
(k) |oni
x 10
(1)
RDM Center. Outliers in a class are defined relative to the class body, and thus, an appropriate class center to de- scribe the general characteristies of the class body is crucial. Commonly, the class body is characterized by high-density instances and contains more than half of the total instances. Considering an imaginary hypersphere centered at instance (k) which contains at least half of the instances in class k. x
Compared with previous FS methods, SIOFS targets intra- class ADMHC for data classification, focuses on the SIOs, and scores features based on the intrusion degree of SIOs caused by features. Theorems and experiments ensure the
we define the instance density of x; as the reciprocal of the radius of this hypersphere. Combining the advantages of density- based and mean centers, we average the instances with relatively high density as the proposed RDM center.
2

---

Stray Intrusive
FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
The details of obtaining the RDM center are follows.
First, for class & and ny 23, we compute the fn-nrrm dis- tance (/)) = ||x x
s =
|| between x; and x(4) (i,j=
200
(d!))))) Dawin, is the The ith row of the matrix (di)
100
100
1,2, :......ma... mz).
distance from x; to each instance of class
(a)
(b)
(c)
Second, for i0,, Meg let e,10) = median (d)) ((h)) where median(X') acquires the median of ele- ments in X. Consequently, the hypersphere centered at x; with radius E(4) contains no less than d instances (includ- (k) the ing x (K) itself) of class k. The smaller the value of Eq
(k) d22
Figure 2. Histograms of distances between instances the center. The average is denoted by the solid black line and the mode is de- noted by the red dashed line. (a) Data from Class "2" of GISETTE. (b) Data from Class "3" of Carcinom. (c) Synthetic data.
higher the density of (k)
RDM(X a) and the distance between x!" and u via 8 1 un 11, i= 1,2, ........ ny.
Third, due to the fact that the ratio of higher density to all instances in a class is obviously different for different classes, we introduce ratio E (0, 1] to construct the higher-density instance threshold (T(k) as
To mine the intrusive relationship from class k to class L. it is necessary to identify the body of class Es- sentially, a segmentation threshold ( is required for
d , U2 ..... dul satisfying the conditions that (i) the body (!))
(*))_ max(mint(((])) (R) (R)
.22 .......!!!
(k)), Ja (a-n)))) (2)
of class can be extracted, and (ii) when x! E x2())
In (2), mint(S, f) finds the t smallest elements of set S, and [-] is a round up operation. Let ((a) denote the subset of higher density instances of class k. The instance x(()) can
meets ||x -1 < O(1, x;
(k) intersects the instances in class I. Condition (i) is equivalent to more than half of df) de) du? being less than ((), and thus ( should be larger. However, shown in Fig. 1b, if ((3) is large enough to make ||x[] -n) < @(3), but x does not intrude into the body of class "3". That is, condition (ii) requires a smaller (
be selected into ((() if it obeys E;
(k) <T()).
|11
Finally, the RDM center u (1+) ERR of class k is
PIEER Σ x( cHeck)
Exprr
x
(k)
(3)
The 30 principle of normal variable that the probability (r(5>x+300 <0.002 0.002 when N~ o²) (Vapnik, 1995) provides a probability- threshold 3--3o. However, this threshold is more suitable when the distribution of d() dg "......... ...., d!) coefficient "3" causes + fail in satisfying the con- dition (ii) of O In high-dimensional data classification with intra-class ADMHC, consequentially, the distribution of d dh) ....,, d!l is asymmetry or multi-peak.
where (+) is the size of ((()) In (3), the mean operation appropriately eliminates random errors and captures the general characteristics of the class body.
is symmetric. Additionally, the larger
Based on the sparsity of outliers and the definition of in- stance density, outliers have low instance density and are not allowed to enter (k)) unless a is large in (0, 1]. Two special cases are as follows: The RDM center is (i) the instance itself when ng = and (ii) the average of two instances when ną 2. In particular, the RDM center is dimension-indeeeedeen and can be calculated for a series of 1-dimensional numbers. We formulate the procedure for obtaining the RDM center as RDM(X, a), where X represents the input dataset, and is derived from (2).
The common Skewness Coefficient (SC) embodies the asymmetry degree of data w.r.t its mean, and the SC of d() (()) ...... del) (denoted as g()), usually (!!) E (-3,3) (Linton, 2017). The larger the value of |s()), the greater the asymmetry degree of the data distribution. Based on the statistical theory, we have the conclusion that "mode' <average" if s!!) > O, and "average < mode" if s(() < 0 (the prototype and properties of SC are shown in Appendix B).
3. Pattern of Stray Intrusive Outliers and Feature Selection Method
3.1. Identifying Stray Intrusive Outliers
For the representative ability of the body of d (1) d(l the RDM center is superior to the mean. Let n RDM((d() d(l) _...................... d(l))
.....
As shown in Fig. 1b, the SIO intuitively intrudes other class body and intersects the instances in other class.
a), the corresponding standard
Recognizing potential SIO of class l towards class I: For class 1 = 1,2, .,....C. C, we calculate the RDM center u!()
Mode is the value of the highest frequency in the statistical distribution and also the highest central tendency.
3

---

Stray Intrusive Outliers-Basse FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
To avoid many parameters, we have the same a value for acquiring u(() and the class center u!O, because both a values reflect the dispersion degree of instances within the same class I. Thus,
deviation x(0) = V nu (d,"
under s( = = 0 and Fig. le under 8()) = 1133 < 0). -
In these cases, u +2 ) is more appropriate than u + +2 capturing the values around the highest peak whether s(1) > or $(1) <0.
for dd), dg ..... d() the modified SC is formulated as
Here, three statistical indexes (i.e., (0,),00,,,, 80)) that we introduce into (5) can reflect more statistical properties of data and ensure the general applicability to different types of instance distributions.
3
Σ (d - 10)
(4)
n((60))
Based on the procedure of obtaining the RDM center, the RDM center u of ai)) _t())
About recognizing potential SIO of class k towards class 4, for x (k) -xx((, if there exists class (1 # k) that meets
du) is equal to the mean when 1=1, namely that 5() is special case of ((). Thus, g(D) has the property of g(t), The mathematical supports about the final ())) are in Theorem I and 2. The proofs of Theorem and are deferred to Appendix C.
111 < u ( + (2
50) - 3 )a"
(6)
(k) is identified as the potential SIO of class & towards then x; class l. Note that if more than one (denoted as 41, .....) satisfies (6), intuitively, x(*) is easily misclassified as the class whose center is closest to x; Therefore, we identify (k) x as the potential SIO of class k towards the unique
Theorem 1. Les & be the distance between the instance and the center in class I, & is a continuous random variable. d() d .....du! are given in (4). When σ(0) > O, we have the probability Pr(5- < ((() + 26(1)== > 3.
of u!") and d(0) dni is a random sample of E. The meanings
class to arg min (f1.2.... (llx(())__(1)1 -u0l)))
and all these potential SIOs form the set (klo)) In particular, (6) is independent of the data structures of classes k and to-
u g(t) are the same as in (4), and mode!!) the same as the foomote of Section 3.1. When (0,1) 1). mode < <n(9= S average holds
Theorem 2. For d()) d d.y ...., d(),,
Examining whether (A) k towards class lo if (Alaa f 0: When instances intrude from one class into another, it indicates that the two classes intersect. Considering the non-spherical data structures of the classes, we follow the general conclusion in Sec. 1 with the oriented radius from one class to another.
E ( is the SIO from class
with probability 1 if g()) > 0, and average"!) < (()= < mode(!) holds with probability I if g <0.
In Theorem 1, d0) >0 0 holds when class I contains at least two different instances. Based on Theorem 1, ( - (1) is greater than most of ,d',d) ...., de) However, it is improper to ignore different asymmetric distributions under
Let (11))) (x({ (6)) (x)( n un))) <0("), (i,,,,,=1,,,........ noo
g(h > 0 or g()) < 0 and employ the same coefficient "2". Based on Theorem 2, u0)++0)) (2 is relatively large for ob- taining the highest density value in d(", ab ,..,, d!l when 30)) >0 Similarly, if g(1) <0, +2)++2 relatively small for obtaining the highest density value in d(l) d!l) ...,...... when (0)) < (moode")u())
denote the instance subset where x; (1o) E Extlaa) but enters the body of class k. Note that, for same lu but different k (loo) may contain the same instances. This makes (o)) different from (look), Thus, if x(6)) 40, the radius ((6)) of class lo towards class k is calculated by
The two contradictory conditions of (0) demand that, we should employ the high density values in d() d2 d(0). Consequently, we need to reduce u( +2 () if g() 0 and increase (2)
D(lo)
Σ
Exgorr 20
(xx(")) - (()))(()))))))) (o0 (7)
The averaging operation in (7) can also properly eliminate the random errors in the data. If x((1) Ex(()))) meets
(0 (1) if g(l) <0. Usually, ( 3-),3), 3 E 1-,,1), and s()) is a special case of g((). For simplicity, whether g()) > 0 or g(i) < 0, we normalize g()) with the constant "3", i.e., g()
||x (k) ( + DOook - [|u((t) u(())) 0, (8)
3. to slightly regulate the coefficient "2" and assign the 2 - g500) to d That is, uniformly formulate (1) as
we identify (*a) as the SIO of class k towards class to (k: + 40). All these SIOs form the SIOs set x 2e(k If x (klo) + 0. the classes k, la are recognized as a S10 class pair. In addition, there is no intrusive relation between class k and to if either ((aa)* =0 or x x((kaa)=()
e0!) (1) + (2- (0)) W
(5)
More importantly, when the distribution of d d ..... de is multi-peak, it is particularly valuable to replace the coefficient "2" with 2 $000. The highest peak is on the left when g(1) > 0, while on the right when g()) < 0 (see Fig. ld
3.2. Feature Selection on Stray Intrusive Outliers
We further assess SIOs in xx(kfo) and the SIO class pair k. lo
on feature level. Let x (x==(((())) .a ,. id Tx ex(((lo)
(k) (k)
(k)
(klo)
4

---

Stray Intrusive Outliers- Bassd FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
and class center ( (uu]
(k)
(R)
y Considering
the corresponding vector (5((la) row of the matrix P p= (P.)) Jo.NN, Finally, let sy 1f= ..... d) represent the score of the fth feature. We can obtain the feature scores by
s(hlo) forms
u2
E Σ3 ois d
Bis and (1), (7) (8) are rewritten as
Duoak_
1
E
(( Ex
(1a) (m(10) U
(9)
E d
(1)())
u 1 DDD -luy (K)
(K)
= (bo) 1)00 (10)
[0.5.
1y
5-
($1....,,3.)
52) = T0.5
Pai...ry
Σ
Paa). (11)
We denote D4aAk= 1 E Extloo 1 -u and Sif = of u + DDlak
(1)
(o00)>111 -m() Mo0d They are the
(8)
Note that the fth feature has no discriminability between the two classes if its values are the same across all instances from both classes. In this case, we assign a score of +00 to such a feature. The scores $1, $2..... Sd are then ranked in ascending order, and the top m features are selected.
fth terms of (9) and (10) from feature/ f. respectively.
At the level of feature f= 1,,,...... d, Fii - " is the distance between x(*)) and ((t),, Dilak is the average dis-
x(oo) is in class 4o but enter the body of class k. |ug -ug (fo)] is the distance between the two class centers. Inspired by the expectation of small intra-class diversity and large inter-class scatter in classification tasks (Xu et al., 2022), |x:f u and should be large if feature f is sufficiently diseriminative. Consequently, for each x E x we expect s(Hlo) <0 with | being large. Thus, a reasonable FS strategy is to assign a higher rank to feature f whose s(ple) appears at the top of the ascending order.
tance from x (14) E x(ook to (o),,, where x
A Algorithms and time complexity. Obtaining the RDM cen- ter, the SIOs set, and the final FS cost +(n²+n), O(ne+=²) and (de) time, respectively. The corresponding algorithms and time complexity analysis for each step are deferred to Appendix E.
D(lak should be small, and [u] -u -
4. Experiments and Analyses
(6)
4.1. Experimental Settings
We use 12 state-of-the-ar supervised FS meth- ods as baselines, along with three popular unsupervised FS methods for extensions. The names, types, and references of all FS methods are given in Appendix F. For fair com- parisons, parameter settings and codes for all methods are directly taken from their original publications.
Baselines:
FS on one SIO class pair k. lo: To reflect the overall per- formance of the feature f on all SIOs in (ooo)))) we average
(blo) On all st values as 3(hlo) - 1x Ex (t) Ex Ex (t s(hlo))
the other hand, for the SIO class pair 4. E s/10))
Sixteen multi-type datasets are considered due to their classification challenges, including intra-class ADMHC and variations in the number of instances, features, and classes. Detailed information about the 16 datasets is given in Appendix F. For Dataset #1~#11 12 in Table 6 (see Appendix F), instances are represented by low-level features. For Dataset #1 #15 in Table 6, deep features are extracted using configurations similar to (Yuan et al., 2024). For #16 Caltech101 dataset, high-level visual feature, i.e., Fisher vector (FV), is extracted (Zhang et al., 2014; Yuan et al., 2022).
Datasets:
and 5(hte) reflect the average intrusion degree on all fea- tures and one feature, respectively. The larger the values of 5 5(Ala) ,5 ((kia) 5 g((kaa the greater their contribution to S 5(kkb)) Therefore, the appropriate evaluation eri- terion should rank the features from highest to lowest by sorting 5(klo) 5 5(hlo) 5 5(klo) in ascending order. The theoretical foundation for the rationality of this evaluation criterion is shown in Appendix D.
ES on all SIO class pairs: Real-world classification tasks seek to achieve higher overall classification accuracy while minimizing the intrusion degree between all class pairs. For the class pair with maximum intrusion degree, although many features can be discarded to reduce the degree, this will result in an inadequate representation of the instances from other classes. Taken together, we consider the half of all SIO class pairs with the lowest intrusion degree and score feature f by averaging the 5(kla) values on this subset of class pairs. For a dataset, let Nip denote the number of all SIO class pairs identified in Sec. 3.1.
Experimental protocol: We use 3 widely accepted metrics: accuracy (ACC), normalized mutual information (NMI) (Cai et al., 2011), and confusion matrix (CM) to evaluate the performance of the same features selected by the FS methods on a dataset. Higher values of ACC, NMI, and the sum of the principal diagonal of CM indicate better classification performance of selected features.
For datasets represented by low-level features, the classifi- cation tasks aim to achieve better performance with fewer selected features. Thus, for each FS method, the ACC and NMI results are computed under the top 50, 300 features, respectively. The mean and standard deviation
First, we compute Σ/=1 s(htoo for each SIO class pair k, lg- Second, the [0.5N,p ipl smallest sums are selected, and
300
5

---

Stray Intrusive Outliers- Based FS on Intra-Class Asymmetric Instance Distribution or Multiple High- Clusters
Lung,
20 200
300 Namber uddecie ParUI
236
350
303
159
Narber ARRICiE
Number www.deee
Nambbrr onderin PNTS
Nkibber
Fisher
QMI
RellefF
TRC
-0- ILFS O FSDOC -4- FSTU -
FSNS -Q- ReOLSR
MRMSR
S'DFS -
IDFS
SIOFS
Datnset: TOX
Dotaset: Caroioom
Oataset Lung
Dateset: ORL
Dataset: GIBETTE
041
Numbor delicted lockure
Numbor selected
Number vellccted
Namber of selected
Numbbrr Veleectedd fostures
Fisher
FSDOC
FSTU
FSNS
ReOLSR
MRMSR
S'DFS
SIOFS
AlFeat
Figure 3. ACC and NMI results of FS methods on some datasets w.r.t the top 50, 300 features. AllFeat: All features are selected.
Pedeatden
SIOFS (UGM)
S'DFS (UCM)
ReCLER (LCM)
(UGM)
True -0000-000----00
clans
True sloce
Predicted class
Predicted
Predicted dasss
Predicted closs
Figure 4. CMs on some datasets. The selected methods correspond to the largest four acd results in Table for each datasct. SIOFS has the same as in Table I for all datasets. 100 or 300 features are employed.
of six ACC results are denoted as aco For datasets represented by deep features, the classification task focuses on achieving the highest classification accuracy regardless of the number of selected features. Hence, for each FS method, both ACC and NMI results are calculated with the top 5%, 10%,... 95% of all features, and the best global ACC and corresponding NMI are reported. When the RDM center is involved, its parameter turned over 0.05, 0.1. 0.95.
fivefold cross- rros-vaaidation, except for ModelNet, where we use its partitioned training and test sets directly. In the fol- lowing tables, the best results are marked in bold, and the second-bes best results are underlined.
4.2. Results and Discussions
Challenge I: small-sized high-dimensional. The first 11 datasets (see Table 6 in Appendix F) are typical small-sized high-dimensional scenarios. Due to intra-class ADMHC and the small sample size, the main difficulty in classify- ing such data is identifying the true characteristics of class instances. Figure and show the results of ACC, NMI, and CM on some datasets, while others deferred to Ap- pendix G, Fig. 7. The following conclusions can be drawn:
Similar to most related work, we use the linear SVM (Guyon et al., 2002) as the classifier for ACC and CM. To be re- peatable, we adopt the same strategy as in the literature ( (Yuan et al., 2024) to replace the random functions in NMI algorithms with deterministic ones. All experiments employ
6

---

Stray Intrusive Outliers-Baseee FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Table Mean and standard deviation (td) of ACC (%) w.r.t the top 50, 100, ................................ 300 features on the 12 datasets. "N/A": ReOLSR
does not converge on GLIOMA and MRMSR code runs very slowly on GISETTE. (a) shows the parameter of SIOFS.
Baseline
CLL
TOX
Carcinon
Lung
Lung.dis
Lympboma
Net9
GLIOMA
colon
ORL
Yale
GISETTE
Fisher
57.66:12999
77.58:1-167
89.37:2.92
93,43:1.41
83.79122.7
83.8613354
51.671601
72.00:6.00
77.69.3.01
93.04:12.45
67.981241
56,5727742
QMI
64.7241198
80.60:14.114
92.05+42.92
94.66++0.77
85.62
85,59442.44
47.22445.66
70.33+4292
79.30-22.66
91.83-42.39
68.8944.115
64.534-8.71
ReliefF
69.22:44.04
79.63:16.6
91,4822.99
93.762:0.37
84.47:3311
86,4641233
51.6746311
68.67444277
81.18:6.15
92,33433311
63,641987
67.35+11.00
TRC
59.31:42.40
82.16:16.05
93.49.42.22
94.17:10.96
82.88-144399
86.46:1:225
54.17.43..0
73.67:10.75
81.72
92.13-1:.97
64.65
63.63
ILFS
55.40:13.00
79.24:1610
84:29::7:6
90.40:11.25
84:70:2:000
85,2511369
32.50:1::51
71.33:1994
81.18
91.83:13.22
59.901101
00.97:10877
FSDOC
65.01:10.96
80,51:18.28
90.13:2-.11
86.07:100.51
84.20:1.5.08
53.33-1.5.61
69.67:15.47
70.97
93.29::..80
70.91:1822
66.42.15.66
PSTU
61.71+-2.79
77.97-44855
80.65-+8.71
91.63++4477
84.47-4-4.99
89,24441.67
36.67+46.00
68.3345582
82.80-44.02
90.08444.92
63.3347
61.47--8.3
FSNS
64.41444.6
79.24=6601
80.37=4%%
90.72++4.31
85.39::2.33
89.58:
36.67=+6001
68.13:6626
82.53=4.311
90.58:44.99
50.00+5.20
60.84+770
ReOLSK
61.71443.89
77.39946.03
79.69-44.64
93.76-10.84
84,02-444.08
85.76:1-5.33
39.17-+6.85
N/A
80.38+22.54
87.63-48.95
59.09:55.50
62.06-+16.29
MRMSR
65.3242.9
74.27:444.3
84.87=5.59
90.64==0.81
86.07+1:884
82.124:7544
42.22
75.67 80
74.46=4.31
12.77
53.64:6635
NIA
DED
63.81:4489
78.85::3.78
84.77=5542
93.19=00.34
1..47=1..29
28.20+
41.95
72.33=+4007
76.08=4431
94.33==1.99
70.81
56.40=
1OPS
63.2146600
80,21++3.84
93.39++1..8 1.78
94.66
86.30++00.00
88.37-43300
54.72-+3.39
74.00++2.00
79.84+44.34
94.08++2.50
71.31++2266
68.15++7.06
SIOFS
71.3244384
83,24::-4.99
2.77
9..32::1002
1.999
89.76-42.20
54.1742.10
26.000
1..33=1.20
9..6
71.2114.70
73.08::6.0
(a)
(0.30)
(0.30)
(0.55)
(0.50)
(0.30)
(0.30)
{0.10)
(0.15)
(0.60)
(0.10)
(0.55)
(0.10)
(i) Across varying numbers of selected features, SIOFS almost always provides comparable or superior results to other FS methods. In particular, when the top 50 features are selected, SIOFS outperforms the second-best by 5.48% on Lung-dis. (ii) SIOFS achieves larger NMI results compared to the baselines, ranking higher on the line chart. (iii) The ACC and NMI results of SIOFS fluctuate less than these of the baselines, indicating that SIOFS is less affected by the number of selected features and data distributions. (iv) Based on the CMs in Fig. 4, SIOFS can correctly predict more instances that are challenging for the baselines, partic- ularly for classes with notable ADMHC, such as class "3" on CLLL and class "2" on GISETTE (see Fig. 1b and 2a).
Table 2. acc of SIOFS with RDM, CFDP and MEAN centers. And the ace difference of SIOFS and w/o SC. "N/A": the SIOs can not be captured by MEAN center.
Dataset
RDM
CFDP
MEAN
w/o SC
(a)
SIOPS
0=0.1
-Ri/ SC
CLL
66.37:311
65.0223.87
58.71
68.47=1.73
(0.20)
2.85
TOX
79.8244254
74.56
82.07
(0.15)
1.17
Carcinom
91,48+1.00
90.23+330z
92.05-2.088
93.01
(0.75)
0.76
Lung
94.5010.34
93.68
11.
95
(0.40)
-0.08
Lung.dis
1.799
85.84++1299
N/A
87.21=1.29
(0.15)
-0.22
Lymphoma
86.81:3.06
86.28
86.80-12.18
89:41
(0.45)
0.35
Ne19
54.17+2110
55.28
49.17++3.37
53.89-0.99
(0.10)
0.28
GLIOMA
75.00:11.53
78.00:442
69.33:100.94
76.33=269
(0.20)
-0.33
colon
82.53
80.38++..54
81.99+1.45
83,96
(0.70)
0.27
ORL
1.66
94, 143
91.42:3.06
94,42=1.54
(0.10)
0.21
Yale
69.29
70.10::468
66.
16.11.333776
3.441:345
(0.55)
-0.20
GISETTE
73:08::6.7
61.67.14066
69 69.40:26.75
22.9 (0.25)
0.10
To further demonstrate the superiority of SIOFS, Table summarizes the corresponding acc std results for each FS method. It can be seen that SIOFS achieves the best or second-best performance on all datasets.
ace std, SIOFS outperforms the second-best by 4.93% on the severe inter-class intrusion dataset GISETTE.
Note that, due to the imbalanced classes in Lymphoma and
4.3. Ablation Study
Nci9 datasets, at one of the fivefold vross-validation, we
have only one training instance of one class. When ni m,=1, 1,
Influence of different class centers for SIOFS. We only replace the proposed RDM center in SIOFS with CFDP and mean centers, respectively. Since CFDP and mean centers are parameter-freet frre, for a fair discussion, we experimentally assign a = 0.1 to the RDM center.
for (5), u =x1" di = ||x] - -u ||1 = 0, thus u =
RDM(fdd) db
20)) J, a) =0, ((0) = 0. and ( O.
At this situation, let (16)) =0 and ( ,...., ( (co <c)
be the ones larger than 0 according to (5), we employ the coefficien coefficient (denoted as v\) of (
variable
((ro), and
For Dataset #1~#12 represented by low-level features, the corresponding aco results for SIOFS with RDM (a = 0.1), CFDP, and mean centers are presented in the left 4 columns of Table 2. It can be seen that when using the RDM center with a 0.1, the performance of the features selected by SIOFS achieves the highest acc on eight datasets, and the second -hiondt ace on the rest. Notably, on TOX, the ace result for RDM center (79.82%) is 3.31% higher than the second-best CFDP center. On GISETTE, the acc for RDM center (73.08%) exceeds the second-best mean center by 3.68%. These results demonstrate the importance and effectiveness of the idea that averaging high-density instances reflects the overall characteristics of class body.
@(ca)), ), where the variable coefficient v of data is the ratio of the standard deviation to the mean, i.e., -= and reflects the degree of data dispersion (Vapnik, 1995)."
reset ((o)) = v(c- - 1)- min(@(1),
Challenge 2: severe inter-class intrusion. Recently, min- ing valuable information in the context of severe inter-class intrusion is necessary in some Al application domains (Deng et al., 2024; Kumar& & Kumar, 2024). Here, we consider a handwritten digits classification dataset from the NIPS FS challenge (GISETTE). The corresponding results of ACC, NMI, and CM on GISETTE are reported in Fig. 3 and Fig. 4, and the ace stds are shown in Table 1. SIOFS achieves superior performance in terms of ACC, NMI, and CM. For
Effectiveness of skewness coefficient. We further evaluate
7

---

Stray Intrusive Outliers- Basce FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Detemet: CLL
Carcinom
Figure 5. Performance of a for SIOFS on the 5 representative datasets. m: Number of selected features.
the effectiveness of SC by comparing the ald results. Based on (5), the coefficient 2- g0) is only simply replaced by "2" (denoted as "w/o SC"). 3 RDM center is still turned over 0.05, 0.1,..., 0.95.
Table 3. Best global ACC (%) and NMI with corresponding num-
ber of selected features (in (.) obtained by FS methods on UCM,
The parameter a for the
AID and ModelNet, under the top 5%. 10%........... 95% of all fea-
tures. AllFeat: All features are selected.
ACC
NMI
Baseline
The highest ace results and corresponding values on Dataset #1-##12 are reported in Table 2, under "w/o SC" and (a). The ace differences between SIOFS and w/o SC are shown in Table 2, under "SIOFS -w/o SC We observe that SIOFS achieves better classification performance on CLL, TOX, and Carcinom. For the other datasets, the ace differences are within (-0.5%, +0.5%). In addition, the standard deviation of the 12 values for SIOFS in row "(a)" of Table 1 is 0.18, while 0.22 for "w/o SC" in column "(a)" of Table 2. As analyzed in Sec. 1, both CLL and TOX contain classes with notable multiple high-density clusters (see Fig. 1d and le). To sum up, for datasets with intra- class ADMHC, SIOFS yields more robust results than w/o SC, and the SC is beneficial in regulating the threshold for obtaining the class body.
UCM
AID
ModelNer
LCM
AID
ModelNer
AllFeat
94.48(244)
84.36/2088)
(2.71(2043)
2.720 0.522(204)) 0.797(2048)
Fisher
94,38(1946)
84.92(1434)
92.79(1638)
0,745(1126) 0.552(1434)
0,809(1946)
QMI
94.33(1741)
84.42(17413
92.91(922)
0.742(1536) 0.550 (.824(1843)
RelielT
94.43(1843)
84.8011434)
92,95(819)
0.729(1743)
0.548(1422) 0.836/614)
TRC
94.48(1741) 85.05(1434)
92.83(1638)
0.755(1658)
0.559(1638)
(.811(1946)
ILFS
94.48(1741)
83.93(1434)
93.03:100
0,731(1638)
0.560:410)
0.8136614;
ESDOC
94.62(819)
84,51(1434)
92.91(1946)
(1737(1126)
0.552(1126)
0.828/614)
ESTU
94.76(1946) 84.51:1946)
92.67(18433
0.748/717)
0.557(153))
0.808(1434)
FSNS
94.67(1946) 84.52(1946)
92.6341946)
0.749(717)
0.558(1536)
0,506(1434)
ReOLSK
94.81(1556)
84.48:1946)
92,83(512)
0.723(1843)
0.541(1946)
0,808(1946)
MRMSR
94.67(1024)
84.64(410)
93.03:122)
0.730(1454)
0.550(1843)
0.811(1843)
S'DFS
94.81(1741) 84.35(18433)
92.59(1843)
(1735(1136)
0.545(1536)
0,824(717)
IOFS
94.33(1741)
85.83(922)
92.71(1946)
0.7781333)
0.594(614)
0,807(1966)
STOPS
94.95(1946) 85.83(1024)
(1.03(143))
0,775(1638)
0.5891512)
0.815(1434)
(a)
(0.60)
(0.05)
(0.80)
(0.60)
(0.05)
(0.80)
wlo SC
94.81(1566)
85.80(1126)
(2.95(1946)
0,769(1434)
0.583(614)
0.812/1946)
(e)
(0.30)
(0.15)
(0.80)
(0.30)
(0.15)
(0.80)
Table 4. Best global ACC (%) on Caltech 101 dataset under the top 60%, 70%, 80%, and 90% of all features. MRMSR fails to obtain the results within a limited time.
comparing parameter-free SIOFS with IOFS. We freeze a = 0.1 as the parameter-free SIOFS. Comparing the row "IOFS" of Table 1 and the column "RDM a = 0.1" of Table 2, we observe that aces increase on 6 out of 12 datasets and decrease on the rest. The max- imum increment is 4.93% on GISETTE, while the maxi- mum loss is 2.02% on Yale. The 6 datasets with reduced acc share a common characteristic that the distributions of class instances are relatively scattered. Consequently, as o changes, the RDM center has a larger drift, causing inappro- priate representations of intra-class instance characteristics. On Dataset #1~#12, the sum of all aces is 954.67 (%) for SIOFS with a = =0.1 and 948.24 (%) for IOFS, indicating that the proposed SIOFS generally outperforms IOFS. Im- portantly, as analyzed in Sec. 1, the CLL dataset has distinct intra-class multiple high-density clusters, and GISETTE has severe intra-class asymmetric instance distribution. The parameter-free SIOFS significantly outperforms IOFS on both datasets. These imply that SIOFS is more applicable for high-dimensional data classification with strong intra- class ADMHC.
An extension:
Fuseer QMI ReliefF TRC ILFS FSDOC ESTU PSNS ReOLSR
D'D tors STOES
n=0.1
60% 40.79 39.7 38.58
37.76 42.38 13.56
10.53
2063
40.99
42.38
45.45
45.74
70% 41.88
40,92 40.36
39.11
42,38
43.45
43.47
45.47
41.45
43.43
48.84
48.98
80% 4261
12,57 41888
40.63
42488
43,93
42,77
42.77
42.67
43.56
51.52 51.58
90%
43443
41.19
43.60
43,20
42.44
42.77
43.66
et al., 2022). The challenges in aerial image classification (UCM, AID) and 3D object recognition (ModelNet) tasks arise from high inter-class intrusion (Xia et al., 2017).
Following the experimental protocol, Table 3 shows the best global ACC and NMI results of the same features selected by FS methods on UCM, AID, and ModelNet datasets. Some of the CMS are shown in Fig. 4. It can be seen that SIOFS achieves the best ACC results on these three datasets. Al- though the best NMI results for all datasets are not achieved by SIOFS, they are only slightly lower than the best. By comparing "SIOFS" and "w/o SC" Table 3, we conclude that SC is also beneficial for FS on deep features.
Challenge 4: FS large-class-numiee dataset. To vali- date the performance of SIOFS on large-class-uumber data, we use the 101tech101 dataset (Fei-Fei et al., 2004). This dataset is chosen also for its asymmetric intra-class instance distribution. Following the iterature (Chatfield et al., 2011; Yuan et al., 2022), Fisher Vector (FV) is employed to de-
4.4. More Experimental Results and Analyses
Challenge 3: FS on deep features. Applying FS on deep based cues is a trend in data classification (Lee
8

---

Stray Intrusive Outliers-Band Based FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
of feature on SIOs. The RDM center is proposed charac- terize the class body. By measuring the distances from the class instances to the center, the modified SC is regulated and fused into the 30 principle to define the class body. The intrusion degree is modeled based on the widely adopted conclusion of two intersecting spheres. Mathematical proofs or logical explanations for the key components of SIOFS are provided. Extensive experiments with 15 state-of-the-an FS methods are conducted on 16 multi-type datasets. Theoreti- cal basis and experimental results demonstrate that, for data classification with intra-class ADMHC, evaluating features based on the intrusion degrees of features on SIOs is promis- ing, and defining the class body by fusing the modified SC is appropriate. Theory and experiments support the superior performance and broad upplicability of our method. Future work will focus on designing more powerful class centers and better mining the patterns of SIOs.
Table 5. Time complexity and runtime (seconds). 7 is the number of iterations of algorithms. m is the number of selected features. "N/A": time complexity is not provided by original authors.
Runtime (Seconds)
Baseline
Time Complexity
CLL
GISETTE
Yale
AID
Fisher
(ddn)
0.22
0.74
0.08
0.65
QMI
NIA ((dx)
1.29
Relieff
1.50
1323.96
0.38
1589.22
TRC
(ddn" +d
4.63
271.22
0.07
329.09
ILFS
O(n 2:37 + ++)
67.54
61.66
0.40
93.33
ESDOC
O(dn+
5.04
226.21
0.42
207.80
ESTU
NIA
3.25
13.07
0.81
100.91
FSNS
1.93
57.36
ReOLSR
NiA
10422.36
7960.63
67.13
4383,46
s D'D
(md2d+nd")
8129:74
1785.38
11.72
973.72
JOFS
On +no+ +dc)
0.53
982.17
0.10
46.61
SIOPS
O(n +n + de)
0.92
865.58
0.10
259.60
scribe the images from this dataset. As FV is a high-level visual feature, and too much runtime should also be avoided, we conduct systematic sampling (Harris & Stöcker, 1998) of the original FV features and select the top 60%, 70%, 80%, 90% of all features, and report the best global ACC result. a = 0.1 is also frozen for SIOFS. As shown in Ta- ble 4. SIOFS outperforms the comparative methods in all cases. The corresponding information of Caltech101 dataset is given in Appendix F.
A Acknowledgements
This work was supported by the Fundamental Research Funds for the Central Universities under Grant B250201042, B250201046, and by Open Research Project of State Key Laboratory for Novel Software Technology, Nanjing Univer- sity, under Grant KFKT2025B04, This work was also sup- ported in part by the National Key R&D Program of China under Grant 2023 by the Natural Science Foundation of Jiangsu Province of China under Grant BK20242050, and in part by the Major Science and Technology Program of the Ministry of Water Resources of China under Grant SKS2022072. This work was supported in part by the High Performance Computing Platform, Hohai University.
Comparison with unsupervised FS methods. The com- parative methods and results are given in Appendix G, Table 7. SIOFS achieves the best second-best performance 15 datasets.
2021 YFB3900601, in part
Parameter sensitivity and runtime comparison. Figure 5 illustrates the influence of a for SIOFS w.r.t. different numbers of selected features on 5 representative datasets, with the others in Appendix G. For some datasets, SIOFS is slightly sensitive to a when few features are selected, but it is relatively robust in other cases. According to the definition of RDM center, a is the ratio of higher density to all instances in a class, thus a is affected by the aggregation behaviour of class instances. For some datasets, such as CLL (see Fig. 1b), due to the scattered distribution of class instances and multiple local high-density clusters, a has a distinct impact on the selected features.
Impact Statement
The primary goal of this work is to advance the field of ma- chine learning. SIOFS has the potential to improve analysis results in sectors that contain the classes with intra-class asymmetric instance distribution or multiple high-density clusters, from healthcare to finance. For instance, in health- care, SIOFS can help identify critical health factors from longitudinal patient data, ensuring more accurate diagnoses and personalized treatment plans. Our work provides the theoretical foundation for further development of effective and interpretable models.
Table 5 shows the time complexity and average runtime of 5 repeats for the FS methods on 4 representative datasets. The runtime of MRMSR is not reported due to its exces- sively long computation time. For larger datasets, such as GISETTE and AID, SIOFS takes more time. Objectively, most of the runtime for SIOFS is attributed the On?)) calculation of the RDM center.
References
Alelyani, S., Tang, J., and Liu, H. Feature selection for clustering: A review. In Aggarwal, C. C. and Reddy, C. K. (eds.), Data Clustering: Algorithms and Applications, pp- 29-60. CRC Press, 2013.
5. Conclusion
In this work, we propose the SIOFS method for high- dimensional data classification with intra-class ADMHC, where feature ranking determined by the intrusion degrees
9

---

Stray Intrusive Outliers-Bassee FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Cai, D., He, X., Han, J., and Huang, T. S. Graph regularized nonnegative matrix factorization for data representation. IEEE Transactions Pattern Analysis and Machine In- telligence, 1550::11188160, 2011.
Kononenko, I. Estimating attributes: Analysis and exten- sions of RELIEF. In ECML,, volume 784, pp. 171-182. Springer, 1994.
Kumar, N. and Kumar, U. Artificial intelligence for classifi- cation and regression tree based feature selection method for network intrusion detection system in various telecom- munication technologies. Computional Intelligence, 40 (I), 2024.
Cai, J., Luo, J., Wang, S., and Yang, S. Feature selection in machine learning: A new perspective. Neurocomputing, 300:70-79,, 2018.
Chatfield, K., Lempitsky, S., Vedaldi, A., and Zisserman, A. The devil is in the details: an evaluation of recent feature encoding methods. In British Machine Vision Conference, BMVC, pp. 1-12. BMVA Press, 2011.
LeCun, Y., Bottou, L.., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1 1):2278-222 00127822324- 1998.
Lee, C., Imrie, F., and van der Schaar, M. Self-supervision enhanced feature selection with correlated gates. In ICLR. OpenReview.net, 2022.
Chen, X., Zhang, W., Xu, X., and Cao, W. A public and large-scale expert information fusion method and its ap- plication: Mining public opinion via sentiment analysis and measuring public dynamic reliability. Information Fusion, 78:71-85, 2022.
Li, C., Huang, L., Shao, Y., Guo, T., and Mao, Y. Feature selection by universum embedding. Pattern Recognit., 153: 10514, 14, 2024.
Cohen, D.. Shnitzer, T., Kluger, Y., and Talmon, R. Few- sample feature selection via feature manifold learning. In International Conference on Machine Learning, ICML, volume 202, pp. 6296-6319. PMLR, 2023.
Lim, and Kim, Pairwise dependence based unsuper- vised feature selection. Paltern Recognition, 11 111:107663, 2021.
Deng, J., Wang, Q.. and Liu, N. Masked second-order pooling for few-shot remote-sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 21:1-5, 2024.
Linton, O. Chapter 1 probability theory. In Linton, O. (ed.), Probability Statistics and Econometrics, pp. 3-9, Academic Press, 2017. ISBN 978-0-12-810495--
Lohrmann, C. and Luukka, P. Nonspecificity, strife and total uncertainty in supervised feature selection. Engineering Applications of Artificial Intelligence, 109: 104628, 2022.
Duda, R. O., Hart, P. E., and Stork, D. G. Pattern classifica- tion, 2nd Edition. Wiley, 2001. ISBN 9780471056690.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 178. IEEE Computer Soci- ety, 2004.
Nie, F., Xiang, S., Jia, Y., Zhang, C., and Yan, S. Trace ratio criterion for feature selection. In AAAI Conference on Artificial Intelligence, pp. 671-676. AAAI Press, 2008
Nie, F., Yang, S., Zhang, R., and Li, X. A general framework for auto- weighwe feature selection via global redundancy minimization. IEEE Transactions on Image Processing. 28(5):2428-2433 2019.
Guyon, I., Weston, J., Barhhll, S., and Vapnik, V. Gene selection for cancer classification using support vector machines. Machince Learning, 46(1-3):389 422, 2002.
Nic, F., Wang, Z., Tian, L., Wang. R., and Li, X. Subspace sparse discriminative feature selection. IEEE Transac- tions on Cybernetics, 52(6))4221 1221--4233,210 2022.
Harris, J. and Stöcker, H. Handbook of mathematics and computational science. Springer, 1998.
Nie, F., Ma, Z., Wang, J., and Li, X. Fast sparse discrimi- native k-means for unsupervised feature selection. IEEE Trans. Neural Networks Learn. Syst., 35(7))9943-999) 2024.
Hc, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 770-778. IEEE Computer Society, 2016.
Robnik- Jobnik-Sikonja, M. and Kononenko, I. Theoretical and em- pirical analysis relieff and rrelieff. Machine Learning, 53(1-2):23 69, 2003.
Jovic, A., Brkic, K., and Bogunovic, N. A review of feature selection methods with applications. In International Convention on Information and Communication Technol- ogy. Electronics and Microelectronics, pp. 1200-1205. IEFE, 2015.
Rodriguez, A. and Laio, A. Clustering by fast search and find of density peaks. Science, 344(6191):1492-114 2014.
10

---

Stray Intrusive Outliers-Basee FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Roffo, G., Melzi, S., Castellani, U., and Vinciarelli, A. Infi- nite latent feature selection: A probabilistic latent graph- based ranking approach. In International Conference Computer Vision, ICCV, pp. 1407-1415, IEEE Computer Society, 2017.
Xia, G., Hu, J., Hu, F., Shi, B., Bai, X., Zhong, Y., Zhang, L., and Lu, X. Aid: A benchmark data set for perfor- mance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7): 3965-3981, 2017.
Roffo, G., Melzi, S., Castellani, U., Vinciarelli, A., and Cristani, M. Infinite feature selection: A graph-based feature filtering approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12):4396-4410, 2021.
Xiang. T., Zhang, C.,, Song, Y., Yu, J., and Cai, W. Walk in the cloud: Learning curves for point clouds shape analysis. In International Conference on Computer Vision, ICCV, pp. 895-904. IEEE Computer Society, 2021.
Xu, X., Wu, X., Wei, F., Zhong, W., and Nie, F. A gen- eral framework for feature selection under orthogonal regression with global redundancy minimization. IEEE Transactions on Knowledge and Data Engineering, 34 (11):5056-509) 2022.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.. Karpathy, A., Khosla, A., Bernstein, M. S.,, Berg, C., and Li, F. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)) 2015.
Yang, Y. and Newsam, S. D. Bag-of-visual- tial extensions for land-use classification. In International Symposium on Advances in Geographic Information Sys- tems, ACM-GIS, pp. 270-270 ACM, 2010.
and spa-
Tang, J. Alelyani, S., and Liu, H. Feature selection for clas- sification: A review. In Data Classification: Algorithms and Applications, pp. 37-64. CRC Press, 2014.
Yuan, L., Yang. G., Xu, Q., and Lu, T. Discriminative feature selection with directional outliers correcting for data classification. Pattern Recognition, 126:108541, 2022.
Theng, D. and Bhoyar, K. K. Feature selection techniques for machine learning: survey of more than two decades of research. Knowl. Inf. Syst., 66(3):1575-1637, 2024.
van der Maaten, L. and Hinton, G. E. Visualizing non non-metric similarities in multiple maps. Machine Learning, 87(1): 33-55, 2012.
Yuan, L., Mei, C.,, Wang, W., and Lu, T. Feature selection based on intrusive outliers rather than all instances. IEEE Transactions on Image Processing. 33:809-825 824, 2024.
Vapnik, N. The Nature of Statistical Learning Theory. Springer, 1995.
Zhang, R., Zhang, Y., and Li, X. Unsupervised feature selection via adaptive graph learning and constraint. IEEE Transactions on Neural Networks and Learning Systems, 33(3):1355-136) 2022.
Wang, W., Xie, E., Li, X., Liu, X., Liang, D., Yang, Z., Lu, T., and Shen, C. PAN++: towards efficient and accu- rate end-to-end spotting of arbirarill--shapd text. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 44(9):5349-5377, 2022..
Zhang, Y., Wu, J., and Cai, J. Compact representation for image classification: To choose or compress? In IEFE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 907-914, 2014.
Wang, Y., Li, X., and Ruiz, R. Feature selection with maxi- mal relevance and minimal supervised redundancy. IEEE Transactions Cybernetics, 53(2):707-717, 2023.
Zhang. Y., Wu, J., and Cai, J. Compact representation of high-dimensional feature vectors for large-scale image recognition and retrieval. IEEE Transactions on Image Processing, 25(5):2407-241, 20116
Wang, Y., Wang, J., and Pal, N. R. Supervised feature selection via collaborative neurodynamic optimization. IEEE Trans. Neural Networks Learn. Syst., 35(5):6878- 6892, 2024.
Zhao, H., Wang. S., and Wang, Z. Multiclass classification and feature selection based on least squares regression with large margin. Neural Computation, 30(10), 2018.
Wang, Z., Nie, F., Tian, L., Wang, R., and Li, X. Discrimi- native feature selection via A structured sparse subspace learning module. In International Joint Conference Artificial Intelligence, pp. 3009-3015, 2020.
Zhao, W., Li, G., Yang, H., Gao, Q., and Wang, Q. Embed- ded feature selection on graph-based multi-view cluster- ing. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), AAAI, pp. 17023.1702..... AAAI Press, 2024.
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. 3d shapenets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912-1920, 2015.

---

Stray Intrusive Outliers-Basse Based FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
A. Related Work
FS for classification. Most real-world classification problems require supervised learning, where the underling class probabilities and class-conditional probabilities are unknown, and each instance is associated with class label (Tang al., 2014; Theng & Bhoyar, 2024). In real-world cases, little knowledge about relevant features is available for learning, Therefore, more candidate features are introduced that are expected to better represent the instances, resulting in redundant features (Cai et al., 2018). For many classification tasks, it is challenging to learn discriminative classifiers before eliminating the redundant features. Mcanwhile, this learning process is time-consuming. FS for classification aims to select a minimally sized subset of discriminative features according to certain criteria, improving feature interpretability and reducing the running time of learning models (Nie et al., 2019; Li et al., 2024).
Review of FS. FS methods can be categorized into three types: filter, wrapper and embedded methods (Wang et al., 2024; 2022). Filter methods rank feature importance based on the intrinsic characteristics of the data, using predefined criterion, and are independent of the classifiers. As result, they are computationally efficient and scalable for high- sigh-dimensional data. A typical filter algorithm consists of two steps (Tang et al., 2014): In the first step, features are ranked according to certain criteria; And in the second step, the highest-ranked features are selected for use in downstream learning tasks, such as classifiers. In wrapper methods, classifier performance is critical to FS. The performance of a specific classifier is optimized by searching for the best subset of features. Examining all possible subsets is an NP-hard problem, so a suboptimal solution used, which remains costly for complex data classification tasks (Alelyani et al., 2013). Embedded methods bridge the gap between filter and wrapper methods by performing selection and classifier learning simultaneously (Jovic al., 2015; Zhao et al., 2024). However, both wrapper and embedded method are prone to over- ever-fitting because selection is part of the training process (Chen al., 2022). The proposed SIOFS method is supervised filter method.
B. The Skewness Coefficient (SC)
According to the literature (Linton, 2017), the prototype of SC is
1 T2 (1) - 8)3
800)
1 T1
(d.)) x)2)
where X is the average of d ()) d0 n RDM((d{) ( . .d" ..... d. 1.1) = X when = 1, thus, we replace the
X with ( That is, the term (d - n 3 in Equation (4) is directly derived from the term (a.)) X)3. The SC is a statistical measure that quantifies the asymmetry of a data distribution. It indicates the degree to which data deviate from a symmetric, bell-shaped normal distribution.
One property of g(9) is that when ny is large enough and > the frequency of mode( is less than the frequency of mode + æ if g()) > 0 (see Fig 6a and 6b), and the frequency of mode!!) is greater than the frequency of mode(! E if (!)) O.
C. Proofs
We begin by proving Theorem from Section 3.1. We reiterate the theorems for completeness.
Theorem 1. Let & be the distance between the instance and the center in class I, & is continuous random variable.
"....... ....d))) is a random sample of E. The meanings of (!)) and (I)) are given (4). When u()) > we have the probability Pr(g (1)) ()) -
dh dhe .....
Proof. Suppose the probability density function of & is f(x), based on the property of probability density function (Harris & Stöcker, 1998), we have / ((x) 2 0.
When x 2 ( + (2) i.e., x - ((()= 2 20(1) > 0, we have
f(x) < (x (0-1
4(011132 f(x).
12

---

Stray Intrusive Outliers-Baseee FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
1500
s
Probabiitty Density
0.6
0.4
1000
0.2
500
mode
3
0
4, 4.
4
(b)
Figure 6. (a) A probability density function of skewed distribution. (b) Histograms of 100,000 data sampled from the distrihution in (a).
Then,
Pr(g < +22l1) = =1-
-
+20
f(x)dr
+200)
+00
(a-u(0y
2 1 -
4(610)2 f(x)dx
+2(1)
+00
> 4(6(0)2
(x-u f
Here, 1 (x: (x- (x)))((())))))))))))))())))))( dr is the variance w.r.t u!!) and (a(0)3 is precise estimation of f (æ (x- uu(0) (xx)dx. Thus, have Pr(E- < (1) 20(1))
Onis n 50) are the same as in (4), and mode ( is the same the footnote in Section 3.1. When E (0, 1}, mode < ( < average!!) holds with probability I i())))) 0 and averagell) < u(0) S mode! holds with probability 1 g(1) O.
Theorem 2. For a dd
11)
Proof. Assume g(0) > O. According to the definition of RDM center, ( is the average of the top fa n4) high density values in d d(l) ..... d(l) When a is smaller in (0, 1] but Ja m4] 1, we have u(!) (1de(1) For d()) dh" dags let di mode ( and d2, dg denote the 2nd and 3rd highest density values, respectively. g(() has the same property of (()) When is smaller in (0, 1] but [a n = 2. based on the property of s!)) (see Appendix B), and considering that ny is often not large enough, at least, d2 > d1 holds with probability 1. Then ( d d1 with probability 1. When is smaller
in (0, 1] but [a- 3, d3 d1, then ( 1 (di d2 + da) > di if dg < di (see Fig. 6b), dg only be very close to dj, namely, d1 - d3 I d2 - d1, ( {dd + d2 + da) 2 dj holds with probability at least. When [a ni] = 4, 5.... the same conclusion can be obtained.
When (1)) > 0, the smaller values in d/)) dh
du denser. Based the difinition of RDM center, if 8()) 0
and a < 1, after the largest [[1 - a.) nı] values in d (1) (!) are removed, the remainder is averaged as the u As we know, the average of set of numbers with large values removed is less than the average of all. Mcanwhile, ( 1. Consequently, if g()) > 0, as decreases from 1, u!)) gradually Combined with the conclusion that mave<ave if (()) >
is the average of all d (1) ,..,,,,,
()) when
gets smaller from the average of all (i)) d9) ...., d(l).
we have !ode!! I u!!) I average when is the larger value in (0, 1]. In conclusion, when a (0, 1], we have (!de( < (1)) (verage!)) holds with probability if g0)) 0.
Similarly, when E (0, 1], we have (verage S a < mode( i() < 0.
D. Theoretical Foundation for Rationality of Evaluation Criterion
g(klo) due (10), and thus E 3(hlo) > 0. However, for feature 1,,,..... For every x (k) (((k))))))))) we have S/=
5(hifo) can positive or negative. There must be least one f 21,2, .....,................ d} that meets 5(Maa) >0. O. Based on our evaluation
13

---

Stray Intrusive
FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Densit Clusters
Algorithm I Obtaining the RDM center of class (ri 2 3).
Input: Data {x(= X2 Xnx J. parameter o;;
(k)
(k)
Output: The RDM center (*)) R4:
1: Initialize the high density instance subset H(*) 0:
2: Compute d( ||x x
11 for all pairwise x((*)
and x) .i. = 1,2, ...., Ag5
3: for = 1, ...,.... ....... ny do
4: Compute E;
modian ( dl ...., Ging
5: end for
6: Compute (()) via (2);
7: for 1,2, ...... . B1 do
8: Assign x; (k) (h) if (M) - T(k);
9: end for
10: Compute ( via (3).
Algorithm 2 Obtaining the SIOs Xo (bll) for class & towards to (lo + k)
Input: {x] (k) (k)
x (k), ( k
(k)
X2
Output: Xo (kol))
1: for k = ..........
2: Acquire (Moo) for the class pair k, ko
via (6) and lu arg me 12... y (11xx 110)11);
3: Acquire X(16)x = {xx(1)) |xx("0) 1l1 (()), i = 1,2, ........ nig );
(1o)
4: if (Kilo)) + 0 and 2(6))k + 0 then
5: Compute D(oa) via (7);
6: Acquire ((koo) where x E (kllo) meets (8).
7: end if
8: end for
Algorithm 3 Final Feature selection of SIOFS method
Input: x(x) (tf k ,,,, the number m of selected features;
Output: The selected features;
1: while Nip do
2: for f= 1,,,.... do 3: Compute D.
[ Ex x Ex [s. u and S. 1iif uu
(1o)
(1o)
(kfa)
(k)
(1)
1u
(k)
(10)
4: Compute 5 ((blo)
Ex ex!! Sig (kla).
5: end for
6: end while
7: Acquire P P= -Pts (o.N whose row is the vector (5(1)) S s(lb)) s(tto));:
8: Compute (81, .....,, 88) via (1I);
9: Sort $1, . Sd in ascending order and select the features corresponding to the first m values.
criterion, the first unselected feature fo must satisfy 5(10)) > 0. Thus, _) ((hto)
E/= s 5(kfo), That is, the intrusion degree between class ls and descending in the feature space {1, 2, .,.,....d)) {.fo} than (1.2.....d d).
E. Algorithms and Time Complexity Analysis
Obtaining the RDM center. Corresponding procedure is shown in Algorithm 1. For class K (k = 1, ,,,...,........ c). the time n22) complexity of Step 2 (r) 2) due to the symmetry matrix (dey my xnai and computing Step 4 and Step 6 cost ((n)) and
ny time, respectively. Thus, the total time of Algorithm Oi 1222 + 2nx)). Due to Ei=1 n 2 < (Eiz1 n4)? the time complexity of Algorithm limits On" + nn.
14

---

Stray Intrusive Outliers-Bascee FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Procedure of obtaining the SIOs set and the final feature selection. The procedure of obtaining the SIOs set x! and the final feature selection are presented in Algorithm 2 and Algorithm respectively, Let m denote the number of selected features. The notations n, d, have the same meanings as them in Section 2.
Algorithm 2: For k 1, ...... C, computing Step 2~7 cost (nc) time for [|x{{)] ull) and O(c²) time for ||u - (16) 1I. Thus, the total time of Algorithm 2 is (nc + c²).
Algorithm 3: We assume that [ < ny and all classes contain the SIOs, i... Nip = c. Computing Steps 1~6 costs (dd) time. Computing P costs (dd)). Thus, the total time of Algorithm 3 is ((d)).
Combining the time of Algorithm 1, the total time of SIOFS is On + n d) for small-sized high-dimensional datasets due to c < d, and the upper bound of the time complexity of SIOFS is Onn² + n + + c² + de) for large-scale datasets. Objectively, the total time of SIOFS is mainly due to Algorithm
F. A Aditionnl Experimental Settings
In this section, we describe the experimental details, including baseline methods, datasets and the extraction for deep features. All the experiments are performed on windows-7 operating system (Intel Xcon Gold 6128 CPU @ 3.40GHz 16.0GB RAM).
Table 6. Information of the baseline methods and datasets.
The types of baselines are f=fileer, e=embedded and s=supervised,
unnsupervseed..
Baseline
Ref.
Type f
Dataset
Type
#Instances
#d
#c
Fisher
(Duda et al., 2001)
#1
CLL
Biology
111
11340
QMI
(Zhang et al., 2016)
f
#2
TOX
Biology
171
5748
ReliefF
Soobnik-Sikonp & Kononenko, 2003)
f
#3
Carcinom
Biology
174
9182
11
TRC
(Nie et al., 2008)
8 f
#4
Lung
Biology
203
3312
ILFS
(Roffo et al., 2017)
f
#5
Jung_dis
Biology
73
325
7
FSDOC
(Yuan al., 2022)
f
#6
Lymphoma
Biology
96
4026
FSTU
(Lohrmann & Luukka, 2022)
f
#7
Nci9
Biology
60
9712
FSNS
(Lohrmann Luukka, 2022)
f #8
GLIOMA
Biology
4434
ReOLSR
(Zhao et al., 2018)
f #9
colon
Biology
62
2000
2
MRMSR
(Wang et al., 2023)
f
#10
ORL
Face Images
400
1024
40
D'
(Nic et al., 2022)
#11
Yale
Face Images
165
1024
15
IOFS
(Yuan et al., 2024)
s f
#12
GISETTE
NIPS FS
7000
5000
InfFS
(Roffo et al., 2021)
f
#13
UCM (2010)
Acrial Images
2100
2048
21
EGCFS
(Zhang et al., 2022)
#14
AID (2017)
Aerial Images
10000
2048
30
FSDK
(Nie al., 2024)
#1.5
ModelNet (2015)
3D CAD Models
12308
2048
40
#16
Caltech101 (2004)
Objects
3030
5243
101
Descriptions of baselines. As shown in Table 6, twelve supervised methods and three unsupervised methods are considered for comparison. As a classical filter FS method, Fisher (Duda et al., 2001) method scores features as the ratio inter-class variance and intra-class separation of data. QMI (Zhang et al., 2016) method uses the quantized discrete variables of data entropy for FS. ReliefF (Robnik-Sikonja & Kononenko, 2003; Kononenko, 1994) relies on instances and their neighbors to estimate feature quality, which is greedy to minimize the redundancy among the selected features. TRC (Nie et al., 2008) method proposes a trace ratio criterion to evaluate features. ILFS (Roffo et al., 2017) is an infinite latent FS method, which performs the ranking step by considering all the possible feature subsets. In FSDOC (Yuan et al., 2022) method, the characteristics of directional outliers are mined and employed for feature scoring. Supervised FSTU and ESNS (Lohrmann & Luukka, 2022) methods score features from a total uncertainty and non-specificity perspective of possibility theory, respectively, in the context of classification. ReOLSR (Zhao et al., 2018) method adopts the orthogonal constraint on the transformation matrix in least squares regression model preserve more data structure information, and uses an iterative algorithm to transform the unbalanced orthogonal Procrustes problem into balanced one. In MRMSR (Wang al., 2023) method, criterion of max-relevance and miin-supervise--rednnee metric and supervised similarity measure. D2S (Wang et al., 2020; Nie et al., 2022) method uses the trace ratio formulated objective functions to ensure the discriminability of selected features. IOFS (Yuan et al., 2024) scores features based on the characteristics of a few outliers that are within the body of the class to which they do not belong. InFFS, (Roffo et al., 2021) is an unsupervised filter method, where features are identified as nodes in a graph and the selection is a path through
is introduced using the normalized feature relevance
15

---

Stray Intrusive Outliers-Basee Based FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
these features. EGCES (Zhang et al., 2022) method directly embeds graph learning into the optimization process. FSDK (Nie et al., 2024) is fast sparse discriminative K-means method, where the weighted pseudo-label matrix with discrete trait is introduced to avoid trivial solution from unsupervised least-squares regression.
Descriptions about datasets and difficulties for data classification. Table 6 shows the dataset types, total numbers of all instances, feature dimensions and classes, respectively. Descriptions and difficulties for data classification are as follows.
The first 11 datasets are in small-sized high-dimensional scenario, where the instances difficult collect and the number of measurements made each instance can easily reach the order of thousands (c.g., set of DNA sequences). They bring challenges such as the curse of dimensionality for FS. These datasets are chosen for their variability in terms of the number of features (from 325 to 1 11340), characterizing 50 to 400 instances. Datasets are downloaded from https: http////nnddngnn// gigggituuii thub. 1 a scikit- deeatureedataseeeeeeeeee
html.
GISETTE dataset from the NIPS 2003 FS challenge, which has two severely confusable handwritten digits 4 and 9 extracted from MNIST data (LeCun et al., 1998) (see Fig. Ic). It is also downloaded from ptctps://jundons html.
github.
UCM (Yang & Newsam, 2010) dataset is publicly available on http: /ttp:://vssie //vssiov..oio// wcmer edu/ het//daaasetttt landuse. html. It consists of 21 classes of land-use images selected from aerial orthoimagery with the pixel resolution of 1 ft. The original images were downloaded from the United States Geological Survey National Map of the following U.S. regions.
AID (Xia et al., 2017) is large-scale dataset for aerial scene classification (see Fig. Ia). It has number of 10,000 images within 30 classes. The numbers of sample images vary a lot with different aerial scene types from 230 up to 420. A main difference between AID and UCM datasets is that AID has multiresolutions that the pixel resolution changes from about 8 m to about half a meter, and thus, the size of each acrial image is fixed to be 600 600 pixels to cover a scene with various resolutions. AID has higher intra-class variations, smaller inter-class dissimilarity and a relatively large scale. The AID downloaded from https: / captain- -wh- github. ic/AID/.
ModelNet (Wu al., 2015) consists of 13,211 3D synthetic models for general objects, with 9843 training samples and 2468 testing samples ranged within 40 classes. We only use its training and test sets, which are downloaded from https:/ /me/eeeoeeeeeeeeemmeemmmm cs .es.pprin wwwppciioonnn......... edu/.
Caltech101 (Fei-Fei et al., 2004) is designed to advance research in multi-class object recognition and image classifica- tion. This dataset contains 101 object classes, including animals, vehicles, everyday objects. It also has high inter-class similarity, intra-class variability in pose, lighting, and scale, and the cluttered backgrounds in some classes. Caltech101 dataset is downloaded from https: /data.aaaaaa.o.aaa caltech. wwwechheeee redord/eerd////eeeoooooeo //z///z/e// jq- 6wc02.
Deep feature extraction. To extract the deep features, we use residual network with depth of 152 (He et al., 2016) pre-trained on ImageNet (Russakovsky et al., 2015) the backbone for UCM and AID datasets. For ModelNet dataset, we use CurveNet (Xiang et al., 2021) as the backbone and use the input of the classification layer as the feature values of the instances. The dimensions of all these deep features are 2048. All implementation details are same as in literature (Yuan et al., 2024).
Systematic sampling of FV features. Following the settings in (Chatfield al., 2011; Yuan et al., 2022), the dimension of the original FV feature is 262144. We conduct the systematic sampling (Harris & Stöcker, 1998) of FV feature, where the 1st, 51st, 101st, .... features are sampled. The final dimension of the sampled FV is 5243.
G. Additional Results and Analyses
Additional ACC and NMI results. The ACC and NMI results of the remaining datasets (CLL, Lung, Lymphoma, Nci9, GLIOMA, colon and Yale) are shown in Fig.
Comparisons with unsupervised FS methods. Unsupervised FS methods select features by seeking the intrinsic structures of data without label information. We follow the same implementations as in Challenge I and 3 for 15 dataset. Table 7 shows the results of SIOFS in comparison with three unsupervised FS methods (InfFS,, EGCFS and FSDK). As shown in Table 7 that SIOFS achieves the best classification results on 13 out of 15 datasets and the second best on the rest. This also shows the superiority of SIOFS.
16

---

Stray Intrusive FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Datsset: Lung
Lymphoma
NCC.09)
ADC
ACC.1%)
ACC/19)
500
Dataset: GLIOMA
Dataset: colon
Dataset: Yale
ADC
ACC
50
100
192
200
251
200
50
100
152
200
253
200
90
100
130
200
220
302
Nimber sedecded Iesturees
Number sndected leatures
Namber selected Resturios
FSDOC
FSTU
ReOLSR
S'DFS
SIOFS
Lung
Lymphoma
0.39
050
0.6
0:60
0.5
MM
0.5
0.40
0.
0.05
0.5
0:25
Number selected
Dataset: GLIOMA
Datsset: colon
Dataset: Yale
0.59
0.35
0.35
0:35
0.25
0.1
0:5
50
602
150
200
200
200
50
100
1.00
200
200
00
100
150
200
200
200
Number of selected lecturee
Numbbrr of selected festures
Number of selected festures
S
Figure 7. ACC and NMI results of FS methods the remaining datasets w.r.t the top 50, 100, .... 300 features. AllFeat: All features selected.
17

---

Stray Intrusive Outliers-Barce FS on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters
Table Comparison of ACC (%) for unsupervised FS methods and SIOFS. selected features.
The values enclosed in (+) show the corresponding number of
Dataset
InfFSt
EGCES
FSDK
STOES
CLL
62.76a5.12
65.32.15.09
1:246
71.32::3.4
TOX
90.45 4.13
78,85
85.28 6.668
83,24-14.99
Carcinom
93.49 a.228
83.34
90.13
93.77 =272
Lung
93.7621009
92.29
94.75
95.32
Lung dis
1..39
84.93+1.12
84.70
86.99=11.0
Lymphoma
27.68
88.37
87.85
89.76=22.20
Nci9
54.45 +:.42
47. 47.78++5.75
43.06++:5.22
54.17 22110
GLIOMA
62.33
72.00+:3.88
74.67
76,00 =.000
colon
79.30
80.83++6.99
77.42+5.51
83.33
ORL
80,88
95.13:11.18
93.92.1262
94, 1.66
Yale
10.28
70,91
64.00 64.04:18.1
71.21 4.770
GISETTE
67.89
61.56:19.89
68. 14:16.11
73.08 =6670
UCM
94.62/1946)
94.71
94,48(1741)
(4.95
AID
83.94(1965)
84.40(1710)
84.27(196)
(1283
ModelNet
92.67(1986)
92.75(512)
92.71
(3.03
Dataset: Lung
Dataset: GLIOMA
Dataset: colon
90
70
ACC (%)
70
ACC
60
ACG
70
70
50
0.
60
50
60
55
300
50
60
50
0
Dataset: ORL
Dataset: GISETTE
95
80
70
ACC (%6)
90
60
80
55
70
30
300
300
200
150
2200
Figure 8. Performance of for SIOFS the rest 5 datasets. Since the S1Os be obtained with larger a, results Lung and Lymphoma datasets are not reported.
18


Reviews:
[{"review_id": "e1qHiEjzaY", "reviewer": "Reviewer_52hZ", "summary": "For the problem of high-dimensional data classification with intra-class asymmetric instance distribution or multiple high-density clusters (ADMHC), a novel supervised feature selection (FS) method named Stray Intrusive Outliers-based FS (SIOFS) is proposed. The proposed method uses the RDM center to characterize the class body, and the modified skewness coefficient (SC) is adjusted and fused into the $3 \\sigma$ principle to define the class body. Then, intrusion degree is modeled based on the conclusion of intersecting spheres. Finally, the feature ranking is determined by the intrusion degrees of SIOs. Experimental results demonstrate the effectiveness of the proposed method over other state-of-the-art FS methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 52hZ for the recognition of our work and for providing constructive comments.\n ## Q1: Explain the inconsistency between Theorem 2 and results about $\\alpha$.\nSorry for the incomplete statement about Theorem 2. We clarify that the condition for $\\alpha$ in Theorem 2 does not contradict the experimental results. **When $\\alpha$ is small, the conclusion in lines 183-187 (left column) still holds**. For easy understanding, we add the following statement.\n\nAccording to the definition of RDM center (see lines 96-98, right column), $\\mathbf{u}^{(l)}$ is the average of the top $\\lceil\\alpha\\cdot n_l\\rceil$ high density values in $d_1^{(l)},\\dots,d_{n_l}^{(l)}$. When $\\alpha$ is smaller but $\\lceil \\alpha\\cdot n_l\\rceil=1$, we have u$^{(l)}=mode^{(l)}$. Following Theorem 1, let $\\xi$ be the distance between the instance and the center of class $l$ and f(x) be the probability density function of $\\xi$. Assume that the SC of $f(x)$ is greater than 0. There are two properties of statistical probability (Harris & Stocker, 1998): (i) $f(mode+\\varepsilon)>f(mode-\\varepsilon)$, for any $\\varepsilon>0$. (ii) f(x) is monotonically increasing near the left side of the $mode$ and decreasing near the right side. In Theorem 2, $d_1^{(l)},\\dots,d_{n_l}^{(l)}$ is a random sample of $\\xi$, let $d_2,d_3$ denote the second and third largest density in $d_1^{(l)},\\dots,d_{n_l}^{(l)}$, respectively, and $d_1=mode^{(l)}$. Obviously there is $f(d_1)>f(d_2)>f(d_3)$. When $\\alpha$ is small but $\\lceil\\alpha\\cdot n_l\\rceil=2$, $\\hat{\\sigma}^{(l)}>0$ and $\\hat{s}^{(l)}>0$. Given the chance of errors in the sampling process, the probability that $d_2>d_1$ holds according to condition (i) is 1, then u$^{(l)}=\\frac{d_1+d_2}{2}>d_1$, i.e., at least u$^{(l)}+2\\hat{\\sigma}^{(l)}>mode^{(l)}$ holds with probability 1. When $\\alpha$ is small but $\\lceil\\alpha\\cdot n_l\\rceil=3$, $\\hat{\\sigma}^{(l)}>0$ and $\\hat{s}^{(l)}>0$. If $d_3>d_1$, then u$^{(l)}=\\frac{1}{3}(d_1+d_2+d_3)>d_1$; If $d_3<d_1$ (see figure https://i.postimg.cc/GmzqF1fV/PDF.png), since $f(d_3)<f(d_2)$, combining the property (i), $d_3$ can only be very close to $d_1$ in this skewed distribution, i.e., u$^{(l)}=\\frac{1}{3}(d_1+d_2+d_3)$ is very close to $d_1$. Combined with $\\hat{\\sigma}^{(l)}>0$, it holds with probability 1 that u$^{(l)}+2\\hat{\\sigma}^{(l)}>mode^{(l)}$ for $\\lceil \\alpha \\cdot n_l\\rceil=3$. When $\\lceil\\alpha\\cdot n_l\\rceil=4,5,\\dots$, the same conclusion can be obtained. Similarly, when $\\hat{s}^{(l)}<0$ and $\\alpha$ is small, but $\\lceil\\alpha\\cdot n_l\\rceil=1,2,\\dots$, u$^{(l)}+2\\hat{\\sigma}^{(l)}<mode^{(l)}$ holds with probability 1.\n\nIn addition, we rewrite \"Based on Theorem 2...\" (line 181) as \"Combining Theorem 2...\" to make the discussion and expression more rigorous. Based on the above explanations and Theorem 2, the value of $\\alpha$ is reasonable in (0,1), and thus we set $\\alpha=0.1,\\dots,0.9$ in Figures 5 and 7. Since $\\alpha$ is the ratio of higher density to all instances in a class (see lines 113-115), it is affected by different dataset characteristics. For example, due to the scattered distribution of clusters and multiple local high-density subclusters in CLL (see Fig. 1b), $\\alpha$ has a significant impact on ACC (see Fig. 7). This problem can be solved by selecting more features. Above statements will be added in the final version. \n## Q2: Explanations of the adjustment of the coefficient \"2\" mentioned in the left column at line 189 and the normalization operation on line 193.\nPlease see our response to **Reviewer aEFk, Q5**.\n## Q3: Typos: conditions i-->(i), ii-->(ii) and iii-->(iii).\nAs suggested, we go over the paper and will revise them in the final version.\n## Q4: Whether SIOFS can still have a significant advantage over other methods when the number of classes is large?\nAs suggested, we add comparisons with the baselines on Caltech101 dataset, which has 101 classes and 3030 images. Like (Yuan et al., 2022), we also use the Fisher Vectors (262144 dimensions). Additionally, in order to reduce the computation time and preserve the main properties of the original representation, we uniformly sample these very high dimensional feature vectors with 50 components, i.e. $1,51,\\dots,262144$, and obtain the final feature vector (5243 dimensions) for each image. Following the setting in (Yuan et al., 2022), we select 60%,70%,80%,90% of features. Same with the original paper, ACC and NMI are calculated over all baselines. And SIOFS outperforms all comparative methods in all cases. Some results are shown in **Rebuttal Table A**.\n\n**Rebuttal Table A.** ACC (%) on the large-scale high-dimensional Caltech101 dataset. Details will be added in the final version. \n|ACC (%) $\\uparrow$|60%|70%|80%|90%| \n|-|-|-|-|-|\n|FSDOC|43.56|43.43|43.93|43.20|\n|S2DFS|42.38|43.43|43.56|43.66|\n|IOFS|45.45|48.84|51.52 |53.83|\n|EGCFS|43.96|43.89|43.86|44.26|\n|FSDK|39.14|40.46|42.15|42.97|\n|SIOFS|**45.74**|**48.98**|**51.58**|**54.13**|", "claims_and_evidence": "$\\bullet$ This paper mainly proposes the claim: In high-dimensional data classification with intra-class ADMHC, the distribution of distances is asymmetry or multi-peak. Existing FS methods rarely aim to identify the class body in the context of intra-class ADMHC. The proposed SIOFS method targets intra-class ADMHC for data classification.\n\n$\\bullet$ The theoretical results in this paper and the related experimental verification provide strong and clear evidence for the claim.", "methods_and_evaluation_criteria": "Yes. The proposed method indeed demonstrates advantages over other state-of-the-art FS methods in experiments.", "theoretical_claims": "Yes. I have reviewed the theoretical proofs and the correctness of the proofs supports the claim.", "experimental_designs_or_analyses": "Yes. The experimental designs to verify the effectiveness of the proposed method are complete, and the comparison of other state-of-the-art FS methods and the comparison and analysis of experimental results on several diverse benchmark datasets well demonstrate the effectiveness of SIOFS.", "relation_to_broader_scientific_literature": "Prior FS methods rarely aim to identify the class body in the context of intra-class ADMHC. This paper explains the motivation for quantifying the intrusion degree of SIOs and proposes the SIOFS method to deal with intra-class ADMHC for data classification.", "essential_references_not_discussed": "No. All the essential references have been adequately discussed.", "other_strengths_and_weaknesses": "$\\bullet$ Strengths:\n\nThe proposed method effectively handles the data classification problem with intra-class ADMHC and achieves better performance than existing methods.\n\n$\\bullet$ Weaknesses:\n\n1. Theorem 2 assumes that $\\alpha$ is a larger value, but experimental results show that the value of $\\alpha$ is often relatively small, with the largest being $0.6$ that occurs by chance. What is the exact range of values ​​of $\\alpha$ in Theorem 2? How to explain the inconsistency between theoretical results and experiments?\n\n2. In Equation (6), the coefficient $2$ is adjusted to $2-\\frac{1}{3} \\hat{s}^{(l)}$, and the motivation and basis for this adjustment are not explained in detail. Can coefficient $\\frac{1}{3}$ of $\\hat{s}^{(l)}$ be adjusted to other values ​​between 0 and 1?", "other_comments_or_suggestions": "Line 161, \"Condition i\" --> \"Condition (i)\",\n\nLine 123, right column, \"condition ii\" --> \"condition (ii)\",\n\nLine 130, right column, \"condition ii\" --> \"condition (ii)\".", "questions_for_authors": "For classification problems, the increase in the number of classes will have a significant impact on the performance of the method, that is, it will increase the difficulty of classification. I noticed that the number of classes in the datasets involved in the experiments is actually relatively small, with the largest number of only $40$ classes. A natural confusion is whether the proposed method can still have a significant advantage over existing methods when the number of classes is large? Therefore, as the number of classes increases, the effectiveness of the proposed method needs further experimental verification.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fwnovldy7J", "reviewer": "Reviewer_aEFk", "summary": "This paper proposes a supervised FS method, Stray Intrusive Outliers based FS (SIOFS), for data classification with intra-class ADMHC. By focusing on Stray Intrusive Outliers (SIOs), SIOFS modifies the skewness coefficient and fuses the threshold in the 3σ principle to identify the class body, scoring features based on the intrusion degree of SIOs.\n\nExtensive experiments on 15 diverse benchmark datasets demonstrate the superiority of SIOFS over 12 state-of-the-art FS methods in terms of classification accuracy, normalized mutual information, and confusion matrix.\n\n SIOFS has the potential to improve analysis results in sectors that contain the classes with intra-class asymmetric instance distribution or multiple high-density clusters, from healthcare to finance. In all, SIOFS provides the theoretical foundation for further development of effective and interpretable models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s feedback. Below, we address the concerns in detail.\n## Q1: Some ambiguous descriptions about the theorems. Inconsistency between theory and experiment about α.\nAs addressed in our response to **Reviewer 52hZ, Q1**, we clarify that the condition for in Theorem 2 does not contradict the experimental results. When $\\alpha$ is small, the conclusion in lines 184-187 still holds. $\\alpha$ has a significant impact on ACC because of the scattered distribution of clusters and multiple local high-density subclusters (see Fig. 1b). This problem can be solved by selecting more features.\n\nIn addition, we regret the unclear descriptions. We delete \"larger\" and \"the most of\", and **revise Theorem 2** as follows: $d_1^{(l)},d_2^{(l)},\\dots,d_{n_l}^{(l)}$, $\\dots$. When $\\alpha\\in(0,1)$ and $\\sigma^{(l)}>0$, $u^{(l)}+2\\sigma^{(l)}>mode^{(l)}$ holds with probability 1 if $\\hat{s}^{(l)}>0$, and $u^{(l)}+2\\sigma^{(l)}<mode^{(l)}$ holds with probability 1 if $\\hat{s}^{(l)}<0$. We will include the above details in the final version.\n## Q2: Lack evidence or practical examples to validate the advantage on deep learning-based FS.\nWe appreciate the constructive feedback provided. As mentioned in lines 417-418, deep FS has extensive applications, such as Remote Sensing (RS) scene classification and 3D object recognition. We add the ACC results by selecting all features (denoted as \"AllFeat\") in **Rebuttal Table B** and the CMs with the top 5% of features (see https://i.postimg.cc/HLw5K8z6/CMs-AID.png) on AID dataset. These results demonstrate that the SIOFS is superior to AllFeat while some comparative methods (such as ILFS, S^2DFS) is inferior. As shown in CMs, our SIOFS can further improve the classification performance on deep features against other methods. Combined with Fig.1a, for the \"School\" with intra-class ADMHC, the number of instances correctly classified by SIOFS is 28, while it is 24 by TRC and 27 by Fisher and ReliefF. SIOFS gains better performance in predicting the confusing classes, being able to identify key factors influencing RS monitoring. On UCM, SIOFS (94.95) also performs better than AllFeat (94.48); On ModelNet, SIOFS (93.03) has higher ACC than AllFeat (92.71). We will include these details in the final version.\n\n**Rebuttal Table B.** ACC (%) on AID. Six FS methods are randomly selected for comparison.\n||ACC (%) $\\uparrow$| \n|-|-|\n|*AllFeat*|*84.36*|\n|Fisher|84.92|\n|ReliefF|84.80|\n|TRC|85.05|\n|ILFS|83.93|\n|FSDOC|84.51|\n|S^2DFS|84.35|\n|SIOFS|**85.83**|\n## Q3: Maintain the spirit of open source.\nWe appreciate the valuable comment. The code will be released once accepted.\n## Q4: Add an essential reference.\nAs suggested, we will include this paper in the Related Work section in the final version.\n## Q5: Explain the transition from 3σ to Eq.(5), the adjustment of the coefficient \"2\" in line 189, and the normalization in line 193.\nAs mentioned in lines 129-131 (right column) and lines 179-187, and combined with the revised Theorem 2, the transition to Eq.(5) is a heuristic reasoning that follows the conclusion in lines 183-187 (see \"That is, ...\"), and it is necessary to introduce SC for normalization. The original formula of $SC$ (see our response to Reviewer 5k2J, Q5) and the literature [1] (see the response to Reviewer 5k2j, Q2) clarify that $s^{(l)}\\in(-3,3)$ is an empirical guideline, providing intuitive thresholds to assess skewness severity and guide data adjustments, and reflecting realistic bounds for most practical datasets. That is, we have $\\frac{1}{3}s^{(l)}\\in(-1,1)$, then $2-\\frac{1}{3}s^{(l)}\\in(1,3)$. As mentioned in lines 198-205, the $2-\\frac{1}{3}s^{(l)}$ with its range of (1,3) is more rational than 2 for $\\sigma^{(l)}$ to address the highest-density subcluster in the context of ADMHC. As demonstrated in **Rebuttal Table C**, the proposed $2-\\frac{1}{3}s^{(l)}$ outperforms other formulas in most cases, and the average ACCs over the 11 datasets further quantitatively prove the superiority of our method. We will incorporate the above discussions in the final version.\n\n**Rebuttal Table C.** Results of different adjustments and normalizations on some datasets. Our SIOFS is equipped with \"$2-\\frac{1}{3}s^{(l)}$\". Additional settings are included in the table.\n|ACC (%) $\\uparrow$|$2-\\frac{1}{3}s^{(l)}$|$1-\\frac{1}{3}s^{(l)}$|$3-\\frac{1}{3}s^{(l)}$|$2-\\frac{1}{2}s^{(l)}$|$2-\\frac{1}{4}s^{(l)}$|\n|-|-|-|-|-|-|\n|CLL|**71.32**$\\pm$3.84|69.52$\\pm$1.42|66.37$\\pm$1.24|70.57$\\pm$2.53|69.52$\\pm$4.58|\n|TOX|83.24$\\pm$4.99|**83.63**$\\pm$4.64|82.26$\\pm$2.84|83.43$\\pm$3.72|82.65$\\pm$3.98|\n|Carcinom|**93.77**$\\pm$2.72|93.10$\\pm$1.05|92.82$\\pm$2.14|92.72$\\pm$2.36|93.10$\\pm$1.96|\n|Lung|95.32$\\pm$1.02|**95.63**$\\pm$0.91|95.07$\\pm$0.94|95.22$\\pm$1.73|95.07$\\pm$1.21|\n|Lymphoma|**89.76**$\\pm$2.20|88.54$\\pm$2.95|87.50$\\pm$2.48|**89.76**$\\pm$3.58|88.19$\\pm$1.87|\n|Over 11 datasets|**81.80**|81.35|80.98|81.68|81.48|81.64|\n## Q6: Minor errors.\nWe will revise them in the final version.", "claims_and_evidence": "In Theorem 2, ‘the larger value in …’ is not rigorous, a better description is ‘if α1>α2∈（0，1）, then …’ It’s confusing to define 'larger'. Especially from experiments table 1, The α varies widely across datasets, ranging from 0.1 to 0.6. In the ablation experiment and the appendix, it is also shown that the change of α has a significant impact on the performance of some datasets. Is the final selection of a determined by the test results?", "methods_and_evaluation_criteria": "The scheme designed in the experiment is meaningful and fair, and can explain the superiority of the algorithm to a certain extent. The designed evaluation methods are consistent with most related work, which reflects the rationality of the validation framework.", "theoretical_claims": "Theorem 2 indicates that a larger α is necessary. However, the significant variation in α across different datasets presented in Table 1 raises concerns about the alignment between theoretical derivation and experimental validation. From the results in Figures 5 and 7, why does α differ so significantly for different data sets?", "experimental_designs_or_analyses": "The elements of this experiment have been fully addressed; however, there are additional considerations regarding the deep learning component. While feature selection does not appear to offer significant advantages over deep feature extraction in terms of performance, as noted at the end of the article, its primary advantage over deep learning lies in interpretability—specifically, the ability to identify key factors influencing health outcomes rather than focusing solely on accuracy.", "relation_to_broader_scientific_literature": "From the recent review, the challenge of high-dimensional data classification has always existed, among which feature selection is a highly concerned scheme. In particular, asymmetric instance distribution and multi-density cluster, which are concerned in this paper, belong to strong prior knowledge, which can be used for model design with strong pertinently and predictable performance improvement. Recently, the use of outliers for feature selection [Yuan et al., 2022; Yuan et al., 2024] is a novel and niche research topic, which is beneficial to the development of machine learning. However, we hope to maintain the spirit of open source, so that more people can find the highlights of work and apply it to practice instead of doing closed development. Open-source code will not only increase your citations, it will also allow peers other than reviewers to review your work and provide more professional opinions.", "essential_references_not_discussed": "‘Feature selection techniques for machine learning: a survey of more than two decades of research.’ This article's descriptive classification of feature selection is very similar to the related work Review of FS in the appendix, but it is not cited. Of course this is not crucial, as the two most important and relevant articles are cited: FSDOC (Yuan et al., 2022) and IOFS (Yuan et al., 2024) methods.", "other_strengths_and_weaknesses": "The article demonstrates a high degree of originality and clarity in its narrative. However, the descriptions of theorems and key indicators are somewhat ambiguous, which poses challenges for code implementation. For instance, at line 161, \"the most of\" should be specified with a precise percentage (e.g., 0.8, 0.75, or 0.6). Additionally, the transition from 3σ to Formula 5, as well as the adjustment of the coefficient \"2\" mentioned in the left column at line 189, the normalization operation on line 193, all appear to be based on heuristic reasoning rather than rigorous derivation. Should such parameters be supported and fed back by experiments?", "other_comments_or_suggestions": "1. The discussion and expression need to be more rigorous\n2. The experiment needs to be considered carefully", "questions_for_authors": "I am particularly concerned about the selected value of α, specifically whether the experimentally determined optimal α aligns with the expectations derived from theoretical considerations. My primary concern is the potential discrepancy between theory and experimental results: the experimentally selected optimal α may not satisfy the criteria of \"larger\" and \"the most of\" as anticipated in the theoretical derivation. This concern has been exacerbated by some minor errors identified in this paper, which have raised doubts about the consistency between theory and practice. \n\nLine 50 right column，class “2”, There is no class 2 in the figure 1 (b); line 71 left column: radii; line 445 left column: multipple.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p5XiBuapmz", "reviewer": "Reviewer_5k2J", "summary": "This paper proposes a supervised FS method, Stray Intrusive Outliers-based FS (SIOFS), for data classification with intra-class ADMHC. By focusing on Stray Intrusive Outliers (SIOs), SIOFS modifies the skewness coefficient and fuses the threshold in the 3σ principle to identify the class body, scoring features based on the intrusion degree of SIOs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 5k2j for the constructive and valuable comments. The concerns are addressed as follows.\n## Q1: In Fig. 1b, class \"2\" has two high-density clusters, but there is only class \"1\" and \"3\" in the figure.\nMany thanks for the comment. We revise this typo from class \"2\" to class \"3\".\n## Q2: What is the definition of 3σ principle? Can authors explain its insight?\nSorry for the unclear description. We add the classical literature (Harris & Stocker, 1998) and **[1]** to support the 3$\\sigma$ principle. For $\\xi\\sim N(\\mu,\\sigma^2)$, the probability (denoted as $Pr$) that $Pr(\\xi\\in(\\mu-3\\sigma,\\mu+3\\sigma))>0.997$. This provides a robust framework for analysing variability in normally distributed data and guides thresholds for outlier detection in the context of ADMHC.\n\n**[1]** Groeneveld, R. A., & Meeden, G. (1984). Measuring Skewness and Kurtosis. Journal of the Royal Statistical Society: Series D (The Statistician), 33(4), 391-399.\n## Q3: There are different types of outliers and low density assumption can not guarantee that all outliers are not included.\nWe clarify that due to random errors in the data, especially in synthetic datasets, it is impossible to correctly identify all of them. In this paper, only low-density cases are identified and used as outliers. We will include the above details of the outliers we have identified in the final version.\n## Q4: Why using l1 distance in Eq.(1)?\nAs presented in the original paper (see line 79, right column), the $\\ell_1$ norm can treat each component of the feature vector equally. $\\ell_1$ treats all $\\vert x_{if}^{(k)}-x_{jf}^{(k)} \\vert$ linearly while $\\ell_2$ penalizes large $\\vert x_{if}^{(k)}-x_{jf}^{(k)}\\vert$ quadratically. This makes it easier to identify outliers in $\\ell_1$ than in $\\ell_2$.\n\nAs suggested, we add comparisons with $\\ell_1$ distance and the typical $\\ell_2$ distance. Note that we only replace $\\ell_1$ distance with $\\ell_2$ distance. As shown in **Rebuttal Table D**, SIOFS with $\\ell_1$ distance outperforms that with $\\ell_2$ distance on all datasets. We will include these details in the final version.\n\n**Rebuttal Table D.** Comparative results of SIOFS with $\\ell_1$ distance (denoted as \"SIOFS\") and $\\ell_2$ distances (denoted as \"w/ $\\ell_2$\") on some datasets. \"w/ $(\\cdot)^1$\" means that the $(\\cdot)^3$ terms are directly replaced by $(\\cdot)^1$ in Eq.(4). The average ACC over all 11 datasets is also reported for a comprehensive comparison.\n|ACC (%) $\\uparrow$|SIOFS|w/ $\\ell_2$|w/ $(\\cdot)^1$|\n|-|-|-|-|\n|CLL|**71.32**$\\pm$3.84|65.47$\\pm$2.58|70.57$\\pm$1.85|\n|TOX|**83.24**$\\pm$4.99|81.19$\\pm$2.93|**83.24**$\\pm$3.09|\n|Carcinom|**93.77**$\\pm$2.72|**93.77**$\\pm$2.33|93.30$\\pm$2.19|\n|Lung|**95.32**$\\pm$1.02|94.09$\\pm$1.80|95.24$\\pm$0.61|\n|Lymphoma|**89.76**$\\pm$2.20|**89.76**$\\pm$2.44|**89.76**$\\pm$1.52|\n|Over 11 datasets|**81.80**|81.33|81.64|\n## Q5: Explain the modified SC in Eq.(4). Why $(d_i^{(l)}-\\mathrm{u}^{(l)} )^3$ is used? How about other terms like $(d_i^{(l)}-\\mathrm{u}^{(l)} )^1$.\nAccording to the literature (Linton, 2017), the **original formula of the SC** is $SC=\\frac{\\frac{1}{n}\\sum_{i=1}^n(X_i-\\bar{X})^3}{\\left(\\frac{1}{n}\\sum_{i=1}^n(X_i-\\bar{X})^2\\right)^{3/2}}$, where $X_i$ are the individual data points, $\\bar{X}$ is the mean and $n$ is the number of instances. As mentioned in lines 142-150 (right column), the modified SC can be obtained by substituting $\\mathrm{u}^{(l)}$ and $\\hat{\\sigma}^{(l)}$ into the formula $SC$. That is, the term $(d_i^{(l)}-\\mathrm{u}^{(l)})^3$ is directly derived from the term $(X_i-\\bar{X})^3$ and has the same statistical meaning. The SC is a statistical measure that quantifies the asymmetry of a probability distribution. It indicates the degree to which data deviate from a symmetric, bell-shaped normal distribution **[1]**. Since we address the highest-density subclusters in the context of ADMHC, it is reasonable to use and modify the SC via Eq.(4).\n\nIn addition, we appreciate the kind suggestion to discuss other terms such as $(d_i^{(l)}-\\mathrm{u}^{(l)})^1$. We add the comparison between $(d_i^{(l)}-\\mathrm{u}^{(l)})^3$ and $(d_i^{(l)}-\\mathrm{u}^{(l)})^1$. To make the comparison reasonable, two terms with $(\\cdot)^3$ in Eq.(4) are directly replaced by $(\\cdot)^1$ (denoted as \"w/ $(\\cdot)^1$\"). As demonstrated in **Rebuttal Table D**, our SIOFS using the term $(\\cdot)^3$ achieves higher ACCs compared to $(\\cdot)^1$. We will incorporate the above discussions into the final version.\n## Q6: In Line 192, why normalize $\\hat{s}^{(l)}$ by $\\hat{s}^{(l)}/3$?\nPlease see our response to **Reviewer aEFk, Q5**.", "claims_and_evidence": "Please see Weaknesses.", "methods_and_evaluation_criteria": "Please see Weaknesses,", "theoretical_claims": "Please see Weaknesses.", "experimental_designs_or_analyses": "Checked.", "relation_to_broader_scientific_literature": "Most current FS methods score features based on the characteristics of all training instances. Existing FS methods rarely aim to identify the class body in the context of intra-class multiple high-density clusters. This paper addresses them.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths  \n1. The paper structure is clear and is easy to follow.  \n2. The proposed method is novel.  \n3. The method is supported by theoretical evidence and empirical evidence.  \n\nWeaknesses  \n1. At the third paragraph in Introduction, the authors mention that as shown in Fig. 1b, class “2” has two high-density clusters, but there is only class \"1\" and \"3\" in the figure.  \n2. What is definition of $3\\sigma$ principle? Can authors explain its insight?  \n3. The explanation in Line 132 for equation 3 assumes that outliers have low instance density and thus will not be included. However, there are different types of outliers and low density assumption can not guarantee that all outliers are not included.  \n4. Why using l1 distance in equation 1?  \n5. The authors need to explain why the modified SC is formulated as the type of Eq.4. Why $(d_i^{(l)} - u^{(l)})^3$ is used? Is this mean that other terms like $(d_i^{(l)} - u^{(l)})^1$ can not be used?  \n6. In Line 192, why normalize $\\hat{s}^{(l)}$ by $\\hat{s}^{(l)}/3$?", "other_comments_or_suggestions": "Please see Weaknesses.", "questions_for_authors": "Please see Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel Stray Intrusive Outliers-based Feature Selection (SIOFS) method for high-dimensional data classification with intra-class asymmetric instance distribution or multiple high-density clusters (ADMHC). The key innovation lies in focusing on stray intrusive outliers that intrude into other class bodies, using a refined density-mean center to characterize class bodies and modifying the skewness coefficient fused with the 3σ principle to identify class boundaries. Theoretical guarantees are provided for the proposed method, and extensive experiments on 15 benchmark datasets demonstrate its effectiveness.

The paper received generally positive reviews, with all three reviewers acknowledging its ​​novelty​​ in addressing ADMHC scenarios and its ​​strong empirical results​​. Reviewer aEFk highlighted the method's potential applications but initially expressed concerns about theoretical-experimental alignment. While minor weaknesses were noted, the reviewers unanimously leaned toward acceptance after the rebuttal.
Decision: Accept (poster)