A Self-Representation Learning Method for Unsupervised Feature Selection using Feature Space Basis
Authors: Prayag Tiwari
Farid Saberi Movahed
Saeed Karami
Farshad Saberi-Movahed
Jens Lehmann
Sahar Vahdati
Venue: Accepted by TMLR ()

Abstract: Current methods of feature selection based on a self-representation framework use all the features of the original data in their representation framework. This issue carries over redundant and noisy features into the representation space, thereby diminishing the quality and effectiveness of the results. This work proposes a novel representation learning method, dubbed GRSSLFS (Graph Regularized Self-Representation and Sparse Subspace Learning), that mitigates the drawbacks of using all features. GRSSLFS employs an approach for constructing a basis for the feature space, which includes those features with the highest variance. The objective function of GRSSLFS is then developed based on a self-representation framework that combines subspace learning and matrix factorization of the basis matrix. Moreover, these basis features are incorporated into a manifold learning term to preserve the geometrical structure of the underlying data.
We provide an effectiveness and performance evaluation on several widely-used benchmark datasets. The results show that GRSSLFS achieves a high level of performance compared to several classic and state-of-the-art  feature selection methods.

Published in Transactions on Machine Learning Research (07/2024)
A Self- Seff-Representation Learning Method for Unsupervised Feature Selection using Feature Space Basis
Prayag Tiwari
preyog.twwarlkheee.oy
School of Information Technology, Halmstad University, Sweden
Farid Saberi-Movaher
f. s.aberrimorchhetBkkpu..cccc
Department of Mathematics, Faculty of Sciences and Modern Technologies,
Graduate University Advanced Technology, Kerman, Iran
Saeed Karami
karami@iashs.aa.
Department of Mothematics, Institute for Advanced Studies in Basic Sciences (IASBS), Zanjon, 45137-66791, Iran
Farshad Saberi-Movahed
Tmooahed@@widdi com
NVIDIA, Santa Clara, CA 95051, USA
Jens Lehmann
jens.eehmamm@tt-drreeeee de
Technische Universität Dresden, Dresden, Germany Amazon (work done outside of Amazon), Dresden, Germany
Sahar Vahdati
sahar, wwwdati@tu- dreeiittttdddeeenn.mm
Noture-Inspired Machine Intelligence Group, At center, Technische Universität Dresden, Dresden, Germany Institute for Applied Computer Science (InfAI), Dresden, Germany
Reviewed on OpenReview: https: optpr/wweee.e.e.emmme///ee net/ forun? id= LNobbgBPPP
Abstract
Current methods of feature selection based self-representation framework use all the features of the original data in their representation framework. This issue carries over redundant and noisy features into the representation space, thereby diminishing the quality and effectiveness of the results. This work proposes a novel reprosentation lcarning method, dubbed GRSSLFS (Graph Regularized Relf-Representation and Sparse Subspace Learning), that mitigates the drawbacks of using all features. GRSSLFS employs an approach for constructing a basis for the feature space, which includes those features with the highest variance. The objective function of GRSSLFS is then developed based repfereerrsnnnttiion framework that combines subspace learning and matrix factorization of the basis matrix. Morcover, these basis fcatures are incorporated into manifold learning term to preserve the geometrical structure of the underlying data. We provide an evaluation of effectiveness and performance GRSSLFS on several widely used benchmark datasets. The results show that GRSSLFS achieves high level of performance compared to several classic and state-of-the-arl feature selection methods.
1 Introduction
Feature selection is a prominent technique for managing high-dinennsional data, aiming identify the most representative and effective features within the original feature set Dinh & Ho (2020); Majumdar & Chatterjee 2022). Feature selection is applied across various tasks, including image processing Shi et al.
Corresponding author

---

Published in Transactions on Machine Learning Research (07/2024)
bioinformatics Moslemi (2023 and genomies Sun et al. (2022). Moreover, recent advances in this ield involves different perspectives such as subspace learning Li et al. (2021). graph learning Roffo et al.
2023
(2021) and self-representation learning Chen et al. (2022).
Subspace learning is a concept that aims to find the best lower- divenssimenion subspace within the original high-dimensional space Ren et al. (2021). These methods can be categorized into two primary types: Linear Subspace Learning, and Nonlinear Subspace Learning. The first category, which comprises well-established techniques such as Local Preserving Projection (LPP) He & Niyogi (2003) and Principal Component Analysis (PCA) Marukatat 2023 focuses recognizing linear combinations of the initial features that define lower- dimensional subspace. Nonctheless, in situations where the inherent data structure departs from lincarity, nonlinear subspace learning methods become inevitable. Methods like Kernel PCA Ghojogh et al. (2023) and Locally Linear Embedding (LLE) Roweis & Saul (2000) serve as examples of approaches for nonlinear subspace learning. While subspace learning and feature selection can be employed separately, combining them can result in even more potent approaches to dimensionality reduction in data and modcling strategics. For instance, MFFS Wang al. (2015a) stands oul a prominent example of unsupervised feature selection methods developed within the context of subspace learning and matrix factorization techniques. Subsequently, mumcrous approaches that integrate subspace learning and feature selection have been introduced, such as
those presented in SLSDR Shang et al. (2020), DGSLFS Sheng et al. (2021) CNAFS Yuan et al. (2022) and
SPLR Li et al. (2022).
In addition to subspace learning, representation learning has been widely used in various domains of dimensionality reduction methods [Shang et al. (2021); |: Wang et al. 2002)). When it comes to deal with high-dimensional data with large amount of redundant features, it will be complex task to analyze and mine such data, which will generally lead to imprecise results. One preliminary goal of representation learning is to discover basic information, such as the inherent structure and characteristic, from the data Wu et al. (2023); Shao et al. (2023). As a popular category of representation learning, self-representation learning achieves high performance in combination with other methods such as subspace lcarning Chen et al. (2022) Self-representatiion learning indeed originales from one of the earliest mathematical concepts called a basis for vector spaces and is rested on the assumption that every sample or feature vector of the original data can be described in terms of other sample foature vectors. According to this merit of self-representation, some effective feature selection methods have been recently developed to focus on utilizing the correlations among features Tang et al. (2019); Zhang et al. 2022): Lin et al. (002))
Feature selection techniques that fall under either the subspace learning or self-representation learning framework are a prevalent category within the realm of dimensionality reduction. However, these techniques encompass all the features from the initial dataset, including redundant ones, in their representations, which can potentially harm the effectiveness of feature selection. In order to get the root of such problem, the philosophy behind the basis for vector spaces, which forms the foundation for the development of self- representation- based feature selection methods, can be revisited. In linear algebra, it is known that basis for vector space can be defined as set of lincarly independent vectors that can uniquely produce the whole space. In fact, because of the linear independency among the members of a basis, it is highly probable that redundant vectors are excluded from the basis. Hence, with a basis for the space formed by the features of the original data, redundant features will have a diminished impact on the self-representation process, thereby allowing their exclusion from the set of selected features. The comprehensive propertics of the basis for the ura space, namely the linear independence of its members and the unique representation of the feature space, serve as motivation for to introduce more efficient form of self- represspprnnnntiioonn and subspace learning frameworks for feature selection.
The aim of this paper is to establish a new type of feature selection method which is called Graph Regularized Self- Relleeppesenntinnn and Sparse Subspace Learning (GRSSLFS). The primary contributions of the paper are outlined as follows:
Expressing both the subspace learning and self-ropresentation problems by employing a basis for the feature space. Moreover, GRSSLFS merges the subspace learning and repeesepprnnnnnoonnn problems, leveraging the basis of the feature space, to simultaneously eliminate redundant features and perform feature selection.
2

---

Published in Transactions on Machine Learning Research (07/2024)
To the best of our knowledge, GRSSLFS is the first feature selection method that integrates basis of linearly independent features into a unified framework of subspace learning and self-representation.
Introducing the Variance Basis Extension (VBE) framework, which utilizes the variance information of features to establish a basis for the feature space. The objective is to identify basis features with the highest variance scores, ensuring that the basis for the feature space consists of elements with the greatest dispersion.
Through conducting comprehensive experiments on various real-world datasets, the results indicate that GRSSLF'S outperforms several existing related and state-of-the-art feature selection methods. Additionally, the application of GRSSLFS the PneumoniaMNIST dataset Yang et al. (2021) is assessed to preciscly detect and analyze the cardiac silhouette in chest X-ray images.
Finally, the key charactoristics of some unsupervised feature selection methods based on subspace learning or self-representation, logether with our proposed GRSSLFS are outlined in Table IT The detailed descriptions of these methods are provided in Appendix A.T
Table 1: Comparison of some related unsupervised feature selection methods.
Sullepac Learning SelFFoprrennttinn
Graph Negularizston
Sparnn Regubritation Orthegunailly
MFFS
Wing
MPMR Vrngg oi 20130
RXDFS Qr
DSRMK
SCF5 Phane al
RSE
1370
8
DGSLPS
CNATS Your
SPLE at Ji2
RDMRS22 sjc
2023
(Omma)
2
Methodology
In this section, we first introduce our approach to constructing a basis for the feature space and then explain how we use this basis in the frameworks of selfreepeecntattin and subspace lcarning to propose the GRSSLFS method. In what follows, E R indicates the data matrix with m samples and n features. Here, Rmx denotes the set of non-negative matrices of size m x n. For an m dimension vector space P, the set B S P is called a basis if the span of elements of B can produce W and B is linearly independent. Here, the span of some vectors refers to all the possible linear combinations of them. The Frobenius norm, the norm, the trace, and the transpose of matrix A are denoted by ||r,, ||A 1l2,1, Tr(A), and AT, respectively. Finally, (a, b) = b is the Euclidean inner product of the vectors a and b.
2.1 Constructing a Basis for the Feature Space
A basis for a vector space can be defined as a set of linearly independent vectors that can produce the whole space. This section introduces a novel framework, called Variance Basis Extension (VBE), to select a basis for the space generated by the original features. Let us consider the data matrix X = [f1, fe..... F,] so that the rank of X r. The Basis Extension (BE) method Mcyer (2000) starts by choosing an arbitrary feature like fú from among all the features and sets B ) It is clear that B is linearly independent. Ifr 1, the basis building process by the BE method terminates at this point, and B will be basis. Otherwise, we have r I n, and we can choose a new feature like f,, from (f,, ff,,f,...,ff, f - (f., } so that B U (f,r (f., f. is a linearly independent set. The process described in these two steps contimues in the same way, and it can be easily proved that during the r steps, the BE method is able to construct the set {fe, ......, f.. which is a basis for the space generated by the whole features.
As seen in the first step of the BE method, we are allowed to choose any arbitrary vector from (f. ....... f.). Therefore, there is ideal situation to choose features that contain useful in information from the data and
3

---

Published in Transactions on Machine Learning Research (07 (07/2024)
be effective in the feature selection process. As one of the most popular tools in data mining, the variance information of the data. has gained widespread acceptance in some well-known dimensionality reduction methods such as the PCA and variance score methods. The primary motivation for the use of the variance information stems from its simple implementation and its power to display the amount of data dispersion. According to this advantage, the variance score method simply but effectively maintains a number of features with the highest variance score and removes the remaining features.
In the following, introduce the BE method with the aim of constructing basis of features with the highest variance score. The VBE method involves the following three steps:
VBE framework which integrates the variance information of features into
.....,,,,, n, its variance score is calculated. Then, the set of features {fr f2..... fn For each feature f,, i= -, 1.,.... is sorted in descending order based on their variance score. Next, we will have C., ,.. where f refers to the case that f has the ith-ranked among all features based on the variance scores. Now, set X' = [f', ,f,
Set B [ so that f/ has the highest. variance score.
Let rank(X) members of B is equal to then the VBE method terminates at this point, and B will is a basis for the space generated by the features {fi, fa,..
For k = 2,... ...... n, if f. & span set = B BU {fz). B
In this case, if the number of
The main difference between the proposed VBE method and its original BE method is that the features selected by VBE have the highest variance score. This method gives basis with the members that have the most dispersion in space and avoid the accumulation of the features as much as possible.
2.2
Self-representation Problem and the Basis
In the field of linear algebra, a direct connection exists between redundancy and the concept of linear independency. Each basis for the feature space exhibits two essential properties. Firstly, basis represent the complete feature space through its elements, achieved with a significantly reduced number of elements compared to the entire feature space. The second characteristic of basis is that its elements are linearly independent which leads to a significant. reduction in data redundancy which is a crucial aim in feature selection. Continuing on, is discussed how the concept of a basis for the feature space can lead to the definition of an effective form for the selfrrepresentation problem.
The rell-represnnttionn problem of features is considered as linear representation of all features in dataset, including redundant ones which potentially harm the effectiveness of feature representation. Formally, the self-representation problem of features is defined as, mingeR*.a XX - -XCI which can also be expressed as:
fi = enf + F2 ... Cm
fa c12f czff ca2fn;
x XC
f. ~ C.a f + ozn + Canfar
where X [f, fg...., f,] and C [cyy] the coefficient matrix. Without loss of generality, let us assume that rank(X) m, where m n. In the case, where m n, similar discussion can be also applied. Under this assumption, it can be shown that there exists a basis B with linearly independent features (f...... fr such that the set (fi, f2.... based on the basis can be obtained for the features:
......,f f,} can be represented by the basis B. Therefore, the following representation
fi guff, + g12fis + ... + 9ml fi
fa gi2fi, 02afi1 + Im2fimm
f. ~ ginh, + gmzlii + +... + gonlems
4

---

Published in Transactions on Machine Learning Research (07/2024)
which implies that X ~ BG, where B [f...... ................................ f.m. R= is basis matrix, and G 1o.yl R=x" is
the basis coefficient matrix. Now, compared to the linear system of equations 1 the sell-representation problem based on the basis can be defined as:
min X BG
(3)
Problem 3 clarifies that rather than utiliziig all features, many of which may be redundant, it is feasible to introduce new formulation for the sell-reppreenntaioon problem, which utilizes basis containing linearly independent features.
2.3
Subspace Learning Problem and the Basis
A popular technique in machine learning is subspace learning, which used to discover low-dimensional representations of high-dimensional data through lincar or non-linear mappings. Several subspace learning models using Euclidean distance have been introduced in the past few years Wang et al. (2015a); Qi el al.
2018
Shang et al. (2020). Among the efficient subspace learning models is the following one:
distsr. (X, XU) = min | - | XUV
(4)
where U E Rank and V E Rxx are called the feature weight and the representation matrices, respectively. We shall note that the subspace learning model (4) indeed calculates the distance between the space of features generated by X and the space of the selected subset of fcatures generated by XU. As result, if U is chosen such that XU B, then it can be seen that the distance between X and B is zero, i.e., distsI (X, B) O, which means that the space of X and the space of B can align or be identical. More information regarding this issue is provided in the following theorem.
Theorem 2.1. Let X [fr. fi..... fu] E IRmx be dataset with features such that rank(X) m. Let also assume that B = [f....., [.,.] is basis for the space generaled by X. Then, according to the dislance-based subspace learning problem introduced in 4 the dislance between X and B is zero, that is distsr. (X, B) = 0.
......,f]]
Proof. If the matrix U is defined as U [u......, U.m where u, (for = 1,..., m) is vector whose iy-element is 1 and other elements are O, then it can be casily seen that B = XU. With this assumption, the
distance-basee based subspace learning problem 44 will be of the following form:
distsI. X, XU) = distst. (X, B) min ||X - BV
V
It turns out from the above problem that (X, < BVI , for each V. On the other hand, since
B is a basis for the space generated by X, there is basis coefficient matrix G such that X BG. Taking this
observation into account and assuming V G, it becomes evident that 0 S distsI (X, B) < | BGG| = 0, which completes the proof.
Remark. It should be noted that in the case where rank(X) < basic matrix for the feature space, and this will not cause any issues in designing our feature selection framework.
n, it is still possible to create
Now, we can state the subspace learning model (4 in terms of the basis matrix B. In order to accomplish this goal, let us consider the representation relation Therefore, the expression X XUV can be shown as X BGUV. On the other hand, according the vector space generation property of the basis, since the distance between X and B is zero, it straight.orward to see that X can be substituted using the basis matrix B. As result of the preceding discussion, basis- subspace learning problem for feature selection can be ostablished as follows:
B BGUVI G.U.V
min
(5)
where G R m U Baak and V E R&xmm such that & S is the dimension of the selected feature space.
5

---

Published in Transactions on Machine Learning Research (07/2024)
2.4 The Importance of the Roles of U, V. and G
The framework defined in (5) can be employed as bedrock for the feature selection process, in which the feature weight matrix U, the representation matrix V, and the basis coefficient matrix G play major role in selecting the underlying features in our proposed method.
The Role of U: First, U is used as the weight matrix in the feature selection process, that is xelected = BGU = EB=) where (BG); is the i-th column of BG, and u, is the i-th row of U. In fact, the sparser the construction of rows in U, the higher the likelihood of selecting effective features. This description allows the utilization of the -nrmm to promote row sparsity within the matrix U, which can be demonstrated as || The importance of row sparsity in the L2,r-norm lies in its effectiveness in managing structured sparsity, where the objective is to choose discriminative features.
The Role of V: Second, let us consider that B = [f.......ff.. f.nml and V= [Vi...... Vm). It can be concluded from Problem 55] that B = BGUV, which means that [f.,...,,,,, [...] ~ [BGUV,.... BGUvm]. This implies, each feature f, of the basis can be represented by BGUv, for I= -....,,.. m. To be more precise, we can see that f. = Σ E (BGU), Vrl. Hence, the more sparse the columns of V are, the fewer members of BGU are used in the representation of f; of the basis matrix B. As a result, a reduction in redundancy is more likely to occur. In order to reflect the importance of the representation matrix V in our proposed feature selection process, the idea of inner product regularization can be used. Let us consider the representation matrix V. The Gram matrix of V can be written as VTV [(vi, [(vi, where vi is the I-th column of V, for 1= 1...., m. Now, it can be easily seen that the expression Trm1mxm VT vTr(V"V) readily corresponds to the summation of the off-diagonal elements within the Gran matrix VTV, that is to say VTV)- Tr(V"V) Eip
- 11pp (V1, Vp)) Thercfore, by the subsequent problem given as:
min (Tr(Imxm V"V) - TVVVV)))
(6)
and under the assumption that V, it can be deduced that the optimal solution to Problem (6) is attained when the inner product terms (V1, vp) tend toward zero for all pairs of I and p except when T=p. On the other hand, since (p) = (vi)² vp and considering the non-negativity assumption for Vi and Vpr the value Vp become zero extremely sparse vectors. Consequently, the optimization Problem 6 can result in a significant degree of sparsity within the coefficient matrix V.
of (V1, vp) lends towards zero when either
Considering the importance of the role of U and V, we integrate two efficient sparsity regularizations into the objective function (5) in order to make both U and V sparse. Two of the most useful and simple tools are the -nomm and the inner product regularizations, which benefit from the robustness in row and column sparsity, respectively. We can now state the sparse form of the subspace learning problem based on the basis as follows:
min BBB-BUUIIB+ G.U.V
+ +1U + (Tr(1mxm V"v)- - V)(V
The Role of G: Graph regularization is essential in feature selection methods as it integrates the inherent. relationships between features into the selection process Khoshraftar & An (2024). In doing so, graph regularization aids in choosing features not solely based on their individual predictive capabilitics but also on their relationships with other features. Typically, graph regularization is expressed through an additional term in the objective function of feature selection algorithms. This term imposes penaltics on solutions that deviate from the smoothness enforced by the structure of graph. Common graph regularization terms include the graph Laplacian, which encourages solutions where neighboring nodes in the graph have similar function values Hamilton 2020).
To establish a framework for the graph regularization in our proposed method, let us consider the relation X ~ BG and assume that G = .g......... gn]. It is concluded that fe.. f,] / As a result, according to the basic principle of preserving the goometric structure the feature manifold, if the two features f, and f, have similar structure in the feature space, then it can be expected that their corresponding representations Bg and Bg, will also have the similar structure. This observation can be
6

---

Published in Transactions on Machine Learning Research (07 /024)
expressed follows:
1 "
min Tr (BGLG'B") 2
Σ 2.r=1
agr Bg,
Bg- 12.
(7)
where A = [prr] Rux= is the similarity matrix for the features, and the Laplacian matrix L is defined as L P such that P diag(Ps) is a diagonal matrix with the diagonal entries P44 = Drm Agr: for = n. Here, we consider agr = exp (- if f E (f.),) or f, E Nx( where Nf, refers the set of & feature vectors that are closest to fr, and denotes the scale parameter of the Gaussian distribution, and Our otherwisc. Now, according 7 the graph regularized self-representation problem based the basis can be formulated follows:
min (| BGI Tr BBGLG"")
(8)
2.5 The GRSSLFS Method
Based on the the previous sections, we establish the novel GRSSLFS method according the combination of "self-represontatioon based on the basis" and "sparse subspace learning based on the basis" as follows:
min [|X-BG] BGI + |BB BGUVII a (BGLG"B") + 1ll21,2.1 + 2 (Ir(1mxm VTV Tr Ir(VT G.U.V
where 3 and y are the regularization parameters. Morcover, we assume that G E RT V E R such that k < is the dimension of the selected feature space. To solve Problem 9 let consider the function J(G, U, V) as follows:
UURRI and
JGG = X BG
| BGUV T (BGGG") + ST((UEEU) + (Ir(1mxm V Tr(v" v)) + TH(MG") TT(NU)) Ir(V"))
where E E Raxa is a diagonal matrix with the diagonal elements E, = U. for i 1,..., n. Moreover, M Rxxx N gax= and W Æxx are the Lagrange multipliers. The problem mentioned above can be effectively solved by applying an alternative iterative procedure involving G, U, and V. To achieve this, variable needs to be fixed, whereas the other variables need to be determined. That is to say
Update G with fixed U and V. Taking the partial derivative J in terms of G shows
- + BB BG 2aB BGL -V" + BGUVV" o + aJ aG
Update U with fixed G and V. Taking the partial derivative of J in terms of U shows
aj おu B"T""" +2c" 2 B B"BGUVV" 23EU N.
Update V with fixed G and U. Finally, taking the partial derivative of J in terms of shows
aJ .v
B"BBB" 2 G"BBBGGBU +2x(Vnmxm V) + W.
Now, putting the Kreuush-KhunTueeee conditions aJ/oV= = 0 results in the following updating rule:
Lee & Seung 2001 together with 0J/0G = -JJOU--
(B + BBTBAA BT
Gi + Gij BG oB BGP B BGUVV"
(10)
7

---

Published in Transactions on Machine Learning Research (07 (022))
(GTBBVV)), k5
Ug - D BG"B"TGUVT + JEUN
(11)
( + VV) 1)
Vy - Voy
(12)
(U"GTB" + +Vimxx))
Remark. Regarding the square root operation applied in the updating rules (10) and (12 it is essential to highlight that the objective function incorporates the socond-order matrix polynomials concerning the variables G, U, and V. To derive efficient update procedures for solving these variables, prior 20055) Ding & et al. (2006) suggest the utilization of update rules (like the ones given in 10), 11 and 12) that are founded on the concept of the square root. Algorithm 1 is a summary of the procedure developed above to solve the objective function of GRSSLFS. Morcover, the framework of the proposed GRSSLFF method is displayed in Figure
studies Ding et al.
Algorithm 1 The proposed GRSSLES
Regulatzaaion
Feature Graph:
Part Gaph Rasedonn the Basis
method.
1: Input: Data matrix X; the number of selected features k.
Input
Baaiic Representatin
Matris
2: Construct the basis matrix B using the
Sperse Subepace
VBE method discussed in Section 2.1
3: Initialize the matrices G, U, and V
4: while not converged do steps 4 and Update G according to the rule I0
Regslattttoon Regalrriasiin
6: Update U according to the rule Update V according the rule 8: Output: Put the rows of U in descend- ing order of value according the 2-norm. Nexl, sort the features of X so that the k features have the highest 2-norm score in U.
on
Radi Conticcint
Representation
Matris
Matrle
Figure Framework the proposed GRSSLFS method which integrates the seffreepresentatinn based the basis with the sparse subspace learning based on the basis.
2.6 Convergence Analysis
In the following, detailed analysis of the monotone property of the objective function for the GRSSLFS method is conducted.
Theorem 2.2.
The objective function of the GRSSLFS method monotonously decreases according to the
update rules
10
11
The proof starts by assuming that the matrices U and V 2 are fixed, and is shown that the objective function of GRSSLFS monotonously decreases in terms of the other variable G 2 With this assumption, define the function Z(G) |X BGI
Proof.
- + oTr (BGLG"B") + |B BGUV Using
simple calculations, the following can be deduced:
Z(G) Tx(X"X) 2Tr(X B'GGB +TT (BG(P- A) TTBBBB
- -BTr(B BGUV)
Additionally, suppose that G' 2 0 is given. It can be easily shown that
(MG'N).
G2
THGTTMGN Σ
(13)
G
for any matrices M e mm and N E paxn, Using the inequality >>> log(c), for any c >> O, we have
13 in conjunction with the inequality
BG) < Σ ((B"X)),,G. (1 log
Gi G
(14)
8

---

Published in Transactions on Machine Learning Research (07 2024)
G Gi
BGG &
Σ (B' )r
(15)
4r
T(BGPG"B")
B BG' lr
(16)
G
1,
G Gir G¹ ql G/
T+(BGAG"B)) S Σ
(B aa Gir
[1 log
(17)
1,rg
BGUV s E
(B"BV" ) G2+
log Gi G
(18)
br
G
B"V'' "B''BG'VV''
(19)
G
Let us now define the following function:
Q(G. G') = (x
>
(B"x)
log Gir G
) + Σ
[BB"BG') Jr
Gi Gi
1r
1,
B B²'''')
(B' qH (1+ log G' G/₁,
G Gir
Ir
Lina
(1+1og Gir
G/ )) + "BB'C'UU''"") (r
G2
+ TrBBBB - -
( BB" Gi
G
1.r
lr
Now, in view of the relations 14
it turns out that (1) G) G) Z(G) and (2) Z(G) S S(G G'), for any G 2 0. Taking these observations into account, it be inferred that QGG G) represents an auxiliary function for Z(G) Lee & Seung 20001. Consequently, using the following relation
G = min QGG, G'),
(20)
the objective function of the GRSSLFS method monotonously decreases in terms of the variable G. In order to compute the minimization problem 20 the derivative of Q(G G') in terms of Gú leads us to the following relation:
an(G, G') äGij
G
G G
+ 2o G - A)
G'
-2
+
BB"
y
G.j
G
Gúj
G C 2 BG'UVV"
2BB
Gij
Thus, assuming that @ G')/OG,j 0, it follows that
OB' B"BV"U"), (G)))²
G
which leads to the following relation:
(B B''A Gij 5 G BT + BBTB B"BG'UVVTU",
(21)
In conclusion, solving the minimization problem 20 leads to a solution in the form shown in 21 which is exactly consistent with the relation (10) introduced as the update rule for variable G. Here, the proof for the monotone property of the objective function of the GRSSLES method in terms of is completed. Similar what has been explained above for the case of G, is possible to prove the monotone property of the objective function of GRSSLFS in terms of U and V, which are omitted due the similarity in the process of existing relations.
9

---

Published in Transactions on Machine Learning Research (07/ (2044)
2.7 Complexity Analysis
Let m, n and & be the number of samples, the number of features and the number of selected features, respectively. In order to update the matrices G, U, and V, several matrix multiplications must be done in GRSSLFS. From the updating rules (0) 11 and [2 it can be found that among all the operations, the most time-consuming parts to update of G, U and V are B"BGUVV' G BGUVV?, and U"G" 3" BGUV, respectively, so that the time complexity of each of them is almost equal to O(k2m²n²) per iteration. Besides, the required time complexity is O(n²) for constructing the feature graph. In summary, assuming that < (m, the computational complexity of GRSSLFS is almost equal to Om.?n²)
3 Experimental Studies
In this section, we conduct empirical comparisons between our proposed approach, GRSSLFS, and a range of other supervised and unsupervised fcature selection techniques. The unsupervised methods consist of LS He
let al. (2005), CNAFS Yuan et al. (2022), OCLSP Lin et al. (2022), RNE Liu et al. (020). VCSDFS Karami
let al. (2023). and CAF Balmn et. al. (2019)- Furthermore, the supervised methods examined include Laaso Tibshirani (1996), SLAP Chen et al. (2018a), and CD-LSR Xu et al. (2023). We evaluate the performance of
the comparison algorithms in terms of how the selected features impact the downstream task of clustering in
various datasets. We refer to Appendix A.2 for the detailed information on benchmark datasets, Appendix
for the compared feature selection algorithms, Appendix A.4 for the evaluation metrics, and Appendix
A.5 for the experimental settings details.
Table 2: Comparing the best values of the clustering ACC and NMT of ten algorithms on all datasets. The values in parentheses denote the selected number of features at which the ACC and NM1 values are reported.
Dataseis
Lineu
1.5
CNAPS
OCESP
RNE
SLAF
VCSDPS
CAE
CDEIR
GRSSLFS
00:33
(.43000)
55.05
1.00 (80)
03.54
1.02
(50)
65.31
1.51 (70)
16.35 +
0.00
(10)
<3 56
+ 1.17
(70)
66 00 2.11
170
GC 6L0100 (0(00)
61,67:11
53090)
75.33 = 0.01 (20)
GLIDNIA
02.20 = (.03)400
41.00
0000
2305
11.11 2
2283
(4))
14.77
1 (40)
47.31
2
19
120
01.71 =
2.01
(9]
64 25
0.77
(30)
54 6212223
131 40
45.77233.00 363
54,30 = 2.71/3
TOCIITI
63.71
00:33
(10)
1:30
(4))
67
0.59
51.40-
(10)
0.24
(90)
59 m
(0)
15.37141330 18
57.38
(40)
SEIECI
09.58
1.80 (40)
00.021
kw (19)
1T
1:55
(90)
3.07 (91)
13:32
3.27
(98)
16
(90)
10:09 8:0
130
52.53-1 1130
53.18
(60)
SOMEC
07:52
0.991101
63.11=
2.30
00.12
1.72 (40)
68.31
53.51
1100
02.01
8.44
2.00
02.90 2.02
401
64:71=2:33
05.72
ARGAARICP
2.18/70
450
36
73110)
31.54:2:330 76)30)
46.06
5.51(10)
N15I
RNE
VOSDES
CRSSLES
USS
(910)
2:21
(50)
2.12
0.00 (10)
CLIOMA
68:46= 2.10(50)
(40)
28.92
1000
12,00 2.34 (40)
TOOTTI
17:00 1.00
11:68
1.58
25.75
25.25
093)
0.27 (300)
01.12:4
31.25
36,00 0:00 (100)
STLECIT
00.85 2.307.20
06.212 8.0 (90)
4.21 4.38 1869
42.44 2.50
25.46 6 3.52 (96)
41.311 2.36 (90)
08.00 4.00 (14)
45.28:15 00
48.35.5 96.500
38.87 4.15 (M6)
R3EK
5,35 2: 0.09
2,372 0.05
(9))
3.75 = 0.00 443
4.72 0.22
(30)
5.17 0.06
8.632 0.2
5.65 1.55 (00)
2.0120000000
1.1221
8.20 0.53 (10)
ATT
71 77.21:40.00 80, (0))
08-40
1.07 (0)
77.06+ 1 41(90)
78 -9
0.53 100)
27:17 + (01
71 36-33-33 5556 (00)
75.02 +0 (100)
75.27-52 94(140
77. 172---21880 (8()
81.71 1.31((on)
DRI
04.73
12 13) 100)
53.45
81 100000
T1.11
41 10 100)
72
18 22.34
110)
07.59-1 1,33% (00)
72.41 5 1101000
73:00 +1 -1.19(100))
73.61--11110 (1.00
79.55 02.07-7.770 (00)
71.56 1.72(50)
SHITATTIPP
12:45 1.56 400
19:39
(0))
45.00 (1000)
15:16 13300190)
4T:7211 (8123)
63 11 11.22 90)
43.35 12.35-12:2211
31.322 05.577 (01
08.83 10)
3.1 Results and Analysis
The best value of ACC and NMI metrics at specific number of selected features is listed in Table 2 It is evident from these results (please see Appendix A.6 for more illustration) that our proposed method, GRSSLFS, outperforms all other nine methods in terms of selecting features that result. in better clustering performance. There are only two exceptions (on the GLIOMA and ORL datasets) where the Lasso, CAE and CD-LSR methods led to higher ACC or NMI compared to our method. For example, the ACC corresponding to Lasso is 62.2 versus 54.10 in our method. Despite the superiority of our method over other methods, its behavior across all datasets is not the same. For example, we can refer to the CNS datasct, where the NMI values are considerably lower compared to the ACC values. This trend is also observed for the other nine feature selection methods. It is mainly due to
Average results ACC and
Raseiig
Lawn
DHAS
Deur
VCSONS
NMI ACC
Figure 2: Average of the best ACC and
NMI results over all datasets for each feature selection method.
10

---

Published in Transactions on Machine Learning Research (07/2024)
the nature of datasets, some of which pose difficulties in selecting distinguishable features and hence have higher clustering accuracy.
On average, the results of the VOSDPS, CAE and CD-LSR methods are closer to those of the proposed method, compared to other methods. Nevertheless, the proposed method is still superior to such methods for two main reasons. First, the basis vectors in the GRSSLFS method are constructed as an independent linear combination of the selected features. Second, the GRSSLF'S method uses rellesepresnnntooonn learning along with subspace learning in which the basis of features is used to build both the self-representation and subspace feature spaces. This combination of self-representation and subspace learning with the underlying basis of features makes the proposed method efficient compared to other methods. The effectiveness is observable in Table [ where the consistency of our method compared to the other methods across all the different datasets is evident. For instance, the ACC values in Table 2 show that while Lasso has a superior performance with respect to GRSSLFS on the GLIOMA dataset, its performance degrades on the other datasets. To summarize the final clustering performance, the average value of ACC and NMI across all the datasets for cach feature selection algorithm is also remonstrated as bar plot in Figure 2 On this account, Figure confirm the superiority of the proposed feature selection model in almost all cases (please refer to the Appendix section for Clustering Performance Results in Figurcs 4 and 5 Running Time Comparison in A.8 Convergence Test in A.9. and Non-parametris Statistical Test in A.10 In particular, comparing our proposed GRSSLES method (an unsupervised approach) with other supervised methods, such as Lasso, SLAP, and CD-LSR, reveals that our unsupervised feature selection technique is fully capable of competing with these supervised alternatives.
3.2 The Role of Hyperparameters
In this part, the effect of hyperparameters on the performance of our proposed method is investigated. To achieve this, we conduct an ablation study and sensitivity analysis. The objective function of GRSSLFS in (9) has three adjustable parameters; a is for the feature graph regularization term the selfrrepreeentation framework and and y control the sparsity of the feature weight matrix U and representation matrix V, respectively, in the subspace learning process.
Table 3: The outputs related to the ablation study.
For the ablation study, we isolate the im- pact of the hyperparameters by setting their values to zero, as shown in the con- figurations in the first column of Table 3 We then choose one biological (GLIOMA) and one image (warpAR10P) dataset to run through our algorithm for feature se- lection and subsequent clustering. The clustering metrics for the seven experiments are reported in Table 3 and are compared against the best
Distasess
GLIOMA
warpAR1oP
Fouiuation metries
ACC+
NMI+ and
ACC-
NME+ aid
Main experiment
$4.10 = (.7)
32.09=2.144 24/22 (0)
46.68 +: (16)
48.63 1 (10)
44.51-2.96(50)
21.56+-467(50)
32.07+2.75(50)
36:01+3-16(00
o
50.214429(30)
27.0613.08:70
30.65_13.88100)
(2.50.4.00(30)
48:30:3. 15(10)
27.52:-4.0:00
38.15:33:4(30)
(1.55:3378(80)
46.6012.16 16)1))
(6(15.29(0))
40.8113.05:10
(5.0113336630
30.0==3.4((0)
30.33 30.57=1.25
43.34=4.11/30)
45.68=3336(10)
A 0
49 80-1 89(55))
20 80+-2.26(3))
42. 42.10-+4.972 (7(2-)97((22)
46.0++3.56/00
51 20.2012..00 (6(20)
28.971411877)
40.2611.55130)
45.99.12.19110
results we obtained with non-zero hyperparameter values of our proposed method. The results indicate that, all parts of the objective function in (9] have a critical role in the feature selection process and can not be removed from this process.
The results of the sensitivity analysis on two datasets, ORL (image) and TOX171 (biological), indicate that while model's performance depends the values of the parameters and datascts, it demonstrates relatively consistent ACC and NMI values. Notably, our model operates well with moderate values of the hyperparameters. For detailed information on the sensitivity analysis, refer to A.7 in the Appendix.
3.3 Application to the PneumoniaMNIST Dataset
In patients diagnosed with pneumonia, chest X-rays reveal various diagnostic features, particularly in the cardiac scction. Notable among these characteristics are an augmented cardiac shadow, alterations in cardiac positioning, the presence of elfusion, the pericardial effusion, and the instances of cardiomegaly Reed (2017). In this section, the application of our proposed feature selection method to the PreumoniaMMNIST dataset Yang et al. (2021) is asscssed with the goal of precisely identifying and analyzing the cardiac silhouctte in chest X-ray images. The Pneumonia.NNIST dataset, designed for Chest X-Ray radiographs and forming part of the MedMNIST collection Yang et al. (2023), comprises 5,856 Chest X-Ray radiograph images, each measuring
11

---

Published in Transactions on Machine Learning Research (07 (07/2024)
(a) PueumoniaMNNIST: Original Images
(b) PneumoniaMNNSST- Selected Features 10
(c) PneumoniaMNIST Selected Features 50
(d) PneumoniiaMNIST.. Selected Features 100
Figure The visualization of selected features on PneamoniaMNNST images.
28x28 pixels in grayscale. To this end, we test the representativeness of the selected fcatures using GRSSLFS by runming it on the PneumoniaMNIST image database. Here, the first ten Chest X-Ray radiograph images are chosen from this dataset. Moreover, select {10, 50, 100} pixels by rumning GRSSLFS. Figure g shows the outputs, where the original and the feature-selected images for each Chest X-Ray radiograph image are plotted in the corresponding row from left to right. In Figure g the selected features are depicted by red dots and the un-selected feat remain at their original pixel colour. As can be seen, when the number of selected features increases from to 100, subsequently GRSSLFS able to capture the most representative parts of the images, like the central part the chest, which contains the most discriminative features of the Chest X-Ray radiograph images and can detect the cardiac silhonette. The performance the obtained feature-selected images has been validated through assessment by radiologists, which demonstrates the proficiency of our proposed method in capturing the fundamental features of the PreumoniaMNNST dataset. This capability shows the interpretability and explainability power of our model. The reason for this selection may lie in the cooperation of the basis matrix in self-representation and subspace learning terms of our objective function. Finally, Section A.11 reports the results pertaining to the selection of 100 features by other comparative methods. Examining this figure, proposed method, SLAP, and VCSDFS, exhibit a tendency to identifythe central area of chest X-ray images. Conversely, some other methods, while successfully identifying portions of the central areas, may overlook additional regions that could provide valuable information. These methods fail to capture such information in radiological analysis.
evident that some methods, including our
4 Conclusion
In this paper, we addressed the self-representation learning problem by establishing basis for the feature space, with the primary aim of eliminating unnecessary and redundant features for the representation of the initial data. The motivation behind this work stems from fundamental concept in lincar algebra known the basis for a given vector space. Easentially, this concept implies that every feature vector can be expressed as a linear combination of a basis set features, which are linearly independent. Leveraging this idea, we also designed subspace learning framework select features with minimal redundancy. Through the introduction of these novel variations in self-represnntation and subspace learning, we proposed an unsupervised feature selection method called Graph Regularized Self-eeprrsnntation and Sparse Subspace Learning (GRSSLFS).
12

---

Published in Transactions on Machine Learning Research (07/2024)
Comparative experiments conducted in this study demonstrated that the proposed GRSSLFS method exhibits high level of effectiveness in feature selection.
The performance of our proposed GRSSLFS method is closely tied to the choice of the feature space basis. As discussed, GRSSLFS employs the variance information of features to establish a basis for the feature space. However, raises the question: what alternative mathematical or statistical tools could be used instead of variance information for constructing such basis? This prompts us to recognize the challenge of devising a meaningful basis for the feature space through potentially more optimized process. As such, the pursuit of constructing an optimal basis for the feature space remains open problem within the domain of unsupervised feature selection, warranting further research in future studies.
References
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 444 453. PMLR, 2019.
Verónica Bolón-Canedo, Noelia Sánchez-Marono, Amparo Alonso-Betanee Benso-Betanzos, José Manuel Benítez, and Francisco Herrera. A review of microarray datasots and applicd feature selection methods. Information Sciences, 282:111- 135, 2014.
Hao Chen, Hongmei Chen, Weiyi Li, Tianrui Li, Chuan Luo, and Jihong Wan. Robust dual-graph regularized and minimum redundancy based relffrepresnntatoon for seme-supervised feature selection. Neurocomputing, 490:104-123, 2022.
Xiaojun Chen, Guowen Yuan, Wenting Wang, Feiping Nic, Xiaojun Chang, and Joshua Zhexue Huang. Local adaptive projection framework for feature selection of labeled and unlabeled data. IEEE Transactions on Neural Networks and Learning Systems, (9(1))63362 6373, 2018a.
Xiaojun Chen, Guowen Yuan, Wenting Wang, Feiping Nie, Xiaojun Chang, and Joshua Zhexue Huang. Local adaptive projection framework for fcature selection of labeled and unlabeled data. IEEE Transactions on Newral Networks and Learning Systems, 29(12):6362 6373, 2018b.
Chris Ding and et al. Orthogonal nonnegative matrix (-factorizations for clustering, In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006.
Chris Ding, Xiaofeng He, and Horst D. Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the 2005 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2005.
Vu C Dinh and Lam Ho. Consistent feature selection for analytic deep neural networks. In Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2420 2431. Curran Associates, Inc., 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://aarchivre.cc..uu.
1
Benyamin Ghojogh, Mark Crowley, Fakhri Karray, and Ali Ghodsi. Principal component analysis. In Elements of Dimensionality Reduction and Manifold Learning, pp. 123-154. Springer International Publishing, Cham, 2023.
William L Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020.
Xiaofei He and Partha Niyogi. Locality preserving projections. In Advances in Neural Information Processing Systems, volume 16, pp. 153-160, 2003.
Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. In Advances in Neural Information Processing Systems, volume 18, pp. 507-514, 2005.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Saced Karami, Farid Saberi-Movahed, Prayag Tiwari, Pekka Marttinen, and Sahar Vahdat. Unsupervised feature selection based on criaacce-ovvrrian subspace distance. Newral Networks, 166(1):188-20)) 203, 2023.
Shima Khoshraftar and Aijun An. A survey on graph representation learning methods. ACM Transactions on Intelligent Systems and Technology, 15(1):1 55, 2024.
Daniel Lee and H. Sebastian Seung. Algorithms for non-negative matrix factorization. In Advances in Neural Information Processing Systems, volume 13, pp. 536-562, 2001.
Changsheng Li, Kaihang Mao, Lingyan Liang, Dongchun Ren, Wei Zhang, Ye Yuan, and Guoren Wang. Unsupervised active learning via subspace learning. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 35, pp. 8332 8339, 2021.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys (CSUR), 50(6):94, 2018.
Weiyi Li, Hongmei Chen, Tianrui Li, Jihong Wan, and Binbin Sang. Unsupervised feature selection via self-paced learning and low-redundant. regularization. Knowleege--aate Systems, 240:108150, 2022.
Xiaochang Lin, Jiewen Guan, Bilian Chen, and Yifeng Zeng- Unsupervised feature selection via orthogonal basis clustering and local structure preserving. IEEE Tronsactions on Neural Networks and Learning Systems, 33(1)):6881 6892, 2022.
Yanfang Liu, Dongyi Ye, Wenbin Li, Huihui Wang, and Yang Gao. Robust neighborhood embedding for unsupervised feature selection. Knowledge-Bond Basedd Systems, 193:105462, 2020.
Subhabrata Majundar and Snigdhansu Chatterjee. Feature selection using e-values. In International Conference on Machine Learning, pp. 14753-14773 14773. PMLR, 2022.
Sanparith Marukatat. Tutorial on PCA and approximate kernel PCA. Artificial Intelligence Review, 56(6): 5445 5477, 2023.
Carl D Meyer. Matris analysis and applied linear algebra, volume 71. SIAM, 2000.
Amir Moslemi. tutorial-based survey on feature selection: Recent advancements on feature selection. Engineering Applications of Artificial Intelligence, 126:107136, 2023.
Mohsen Ghassemi Parsa, Hadi Zare, and Mehdi Ghatee Unsupervised feature selection based on adaptive similarity learning and subspace clustering. Engineering Applications of Artificial Intelligence, 95:103855, 2020.
Miao Qi, Ting Wang, Fucong Liu, Baoxue Zhang, Jianzhong Wang, and Yugen Yi. Unsupervised feature selection by regularized matrix factorization. Neurocomputing, 273:593 610, 2018.
James C Reed. Chest Radiology: Patterns and Differential Diagnoses. Elsevier Health Sciences, 2017.
Dongchun Ren, Wei Zhang, Ye Yuan, and Guoren Wang. Unsupervised active learning via subspace learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8332-8334 2021.
Giorgio Roffo, Simone Melzi, Umberto Castellani, Alessandro Vinciarelli, and Marco Cristani. Infinite feature selection: A graph-based feature filtering approach. IEEE Transactions on Pollern Analysis and Machine Intelligence, (3(11):4396-444 44100 2021.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323 2326, 2000.
Ronghua Shang, Kaiming Xu, Fanhma Shang, and Licheng Jiao. Sparse and low-rodundant subspace carning-based dual-graph regularized robust feature selection. Knowledge-Bart Based Systems, 187:104830, 2020.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Ronghua Shang, Lujuan Wang. Fanhua Shang, Licheng Jiao, and Yangyang Li. Dual space latent representation learning for unsupervised feature selection. Poltern Recognilion, 114:107873, 2021.
Chenyu Shao, Mulin Chen, Yuan Yuan, and Qi Wang. Projection concept factorization with self-representation for data clustering. Neurocompuling, 517:62-1 100622-----0 2023.
Chao Sheng, Peng Song, Weijian Zhang, and Dongliang Chen. Dual-graph regularized subspace learning based feature selection. Digital Signal Processing, 117:103175, 2021.
Dan Shi, Lei Zhu, Jingjing Li, Zheng Zhang, and Xiaojun Chang. Unsupervised adaptive feature selection with binary hashing. IEFE Transactions Image Processing, 32:838 853, 2023.
Saúl Foraaio- J Ariel Cchoaso-OOhoa, and José Fco Martinez- Triniddd.. A review of unsupervised feature selection methods. Artificial Intelligence. Review, 53 -98, 2020.
Yu Sun, Haicheng Li, Lei Zheng, Jinzhao Li, Yan Hong, Penglei Liang, Lai-Yu Kwok, Yougchun Zuo, Wenyi Zhang, and Heping Zhang. iProbiotics: a machine learning platform for rapid identification of probiotic propertics from whole-genome primary soquences. Briefings in Bioinformatics, 23(1):bbab477, 2022.
Chang Tang, Xinwang Liu, Miaomiao Li, Pichao Wang, Jiajia Chen, Lizhe Wang, and Wanqing Li. Robust unsupervised feature selection via dual self-representation and manifold regularization. Knowledge- Based Systems, 145:109-13 120, 2018.
Chang Tang, Mciru Bian, Xinwang Liu, Miaomiao Li, Hua Zhou, Pichao Wang, and Hailin Yin. Unsupervisod feature selection via latent representation learning and manifold regularization. Newral Nelworks, 117: 163- 118-178,44 2019.
Robert Tibshirani. Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society: Series B (Methadological), 58()))267-228) 1996.
Shiping Wang, Witold Pedrycz, Qingxin Zhu, and William Zhu, Subspace learning for unsupervised feature selection via matrix factorization. Pattern Recognition, 48(1):10019, 2015a.
Shiping Wang, Witold Pedryez, Qingxin Zhu, and William Zhu. Unsupervised feature selection via maximum projection and minimum redundancy. Showledge-Basee Systems, 75:19 29, 2015b.
Yu Wang, Liang Hu, Wanfu Gao, Xiaofeng Cao, and Yi Chang. Adans: Adaptive negative sampling for unsupervised graph representation learning. Pattern Recognation, 136: 106:109266, 2023.
Wenhui Wu, Yujie Chen, Ran Wang, and Le Ou- Yang. Selfrrepresentative kernel concept factorization. Knowledge-Basse Based Systems, 259:110051, 2023.
Lei Xu, Rong Wang, Feiping Nie, and Xuelong Li. Efficient top-k feature selection using coordinate descent, method. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10594-10601, 2023.
Jianchong Yang, Rui Shi, and Binghing Ni. MedMNIST classification decathlon: lightweight. AutoML benchmark for medical image analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 191-195, 2021.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Lin, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data, 10(1):41, 2023.
Aihong Yuan, Mengbo You, Dongjian He, and Xuelong Li. Convex nog-negative matrix factorization with adaptive graph for unsupervised feature selection. IEEE Transactions on Cybernetics, 525))55522,5534, 2022.
Yongshan Zhang, Xinxin Wang, Xinwei Jiang, and Yicong Zhou. Robust dual graph self-representation for unsupervised hyperspectral band selection. IEEE Transactions on Geoscience and Remote Sensing, 60: 1-13, 2022.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Wei Zheng, Hui Yan, and Jian Yang. Robust unsupervised feature selection by nonnegative sparse subspace learning. Neurocompaling, 334:156-17 171111 2019.
A
Appendix
A.1 Related Work
Table I presents a detailed summary of various unsupervised feature selection methods, systematically classified into the two main categories: Subspace Learning and Self-represenattion. Both categories are crucial for dimensionality reduction and offer ways to enhance the efficiency and clarity of feature selection models without the need for labeled data. Subspace learning methods aim to identify a lower-dimensional representation of data that maintains its inherent structure, thereby enabling more efficient feature selection. Conversely, self-representation methods strive to represent each data point or feature as a combination of other points or features within the dataset, revealing the underlying relationships and supporting effective feature selection. To further enhance these methodologies, various regularization and constraint techniques can be applied. Graph regularization utilizes the geometric structure of the data, ensuring that the selected features preserve the inherent relationships among data points. Sparse regularization encourages the selection of minimal subset of features, thereby improving the efficiency and interpretability of model. Meanwhile, the orthogonality constraint ensures that the selected features are uncorrelated, promoting a more robust and diverse feature set. By incorporating these regularization and constraint techniques with subspace learning- based and reresseprttnttnn based methods, researchers can attain more robust, efficient, and interpretable feature selection outcomes.
Table 1 lists some of these methods and their potential combinations, providing a valuable resource for advancing the field of unsupervised feature selection. To begin with, Wang et al. (2015a) introduced the MFFS method, which is notable example of unsupervised feature selection developed using subspace learning and matrix factorization techniques. Subsequently, Wang et al. (2015b) proposed the MPMR method, a subspace learning-based approach that utilizes maximum projection and minimum redundancy. This method aims to capture the most significant variance while ensuring that the selected features provide complementary information, thus enhancing clustering and classification performance. Building on subspace learning and matrix factorization, Qi et al. (2018) introduced the RMFFS method. This technique employs regularized matrix factorization for feature selection, effectively decomposing the data into lower-dimensional matrices and facilitating the identification of informative features while imposing an inner product regularization constraint to ensure orthogonality. Tang et al. (2018) introduced robust unsupervised feature selection approach, called DSRMR, which leverages dual self-representation and manifold regularization. By capturing both local and global data structures, this method ensures that the selected features preserve the intrinsic geometry of the data. Inspired by MFFS, Zheeng el. al.] (2019) suggested the NSSLFS method, which utilizes the nonnegative subspace learning combined with the L1 sparsity to select highly discriminative features. Parsa al. (2020) introduced SCFS, non-linear self-representation-beee based method that incorporates the adaptive similarity learning and subspace clustering techniques. SCFS dynamically adjusts similarity measures to enhance the capture of feature relationships. Recent advancements have continued to advance the unsupervised feature selection methods. For example, Liu et al.] (2020) introduced the RNE method, a robust neighborhood subspace learning technique designed to preserve local neighborhood structures and enhance feature selection quality, particularly in noisy environments. (Shang et al. (2020) and [hhnn et al. (202T) introduced the SLSDR and DGSLFS methods, respectively. These fcature selection techniques utilize dual-graph regularization within the subspace learning framework to capture data structures effectively.
Further extending these advancements, Yuan et al. (2022) proposed the CNAFS method, which utilizes convex non-negative matrix factorization alongside an adaptive graph. CNAFS ensures that feature selection maintains data manifold structures through a resilient and adaptive subspace learning strategy. Li et al. (2022) developed SPLR, a subspace learning method that is based on the principles of self-paced learning. SPLR socks to preserve the local manifold structure and utilizes the 12.1/2-norm maintain discriminative features while reducing the impact of noise the data. Chen al. (2022) introduced RDMRS2FS, feature selection technique rooted in self-representation. RDMRS2FS focuses semi-supervised feature selection
16

---

Published in Transactions on Machine Learning Research (07/2024)
with the goal of reducing redundancy among features. This method utilizes dual-graph regularization to precisely capture data characteristios and uncover relationships between uret.res.
A.2 Datasets
Table 4 demonstrates the eight benchmark datasets, which are used to perform our experiments. We selected a mix of five different gene expression datasets and three facial image datasets to diversify our data samples and hence better evaluate the performance our proposed feature selection method. Tt should be noted that all of these datasets are accessible through the ecikit-featuro feature selection repository Li al. (2018) and the UCI Machine Learning Repository Dua & Grall (0177).
Table 4: The information of eight datasets used in this study.
ID
Dataset
# of Samples
of Features
# of Classes
Type
1
CNS
60
7129
2
Biological
2
GLIOMA
4434
Biological
3
TOX 171
171
5748
4
Biological
4
SRBCT
2308
Biological
5
SMK
187
19993
2
Biological
6
ATT
400
10304
Image
7
ORL
400
1024
40
Image
8
warpAR10P
130
2400
10
Image
A.3 Comparison Methods
In this study, the following feature selection methods are selected with their track records of performance in the literature to be compared with GRSSLES. These methods are briefly explained below.
Baseline: All features of data are considered.
The least absolute shrinkage and selection operator (Lasso) Tibshirani [1966]: supervised feature selection method that efficiently selects features by assuming a lincar relationship between input features and output values.
Laplacian Score (LS) He et al. (2005): The features selected by this method can best preserve the local manifold structure the original data.
4. Convex non-negative matrix Factorization with an Adaptive graph constraint Feature Selection (CNAFS) Yuan et all (022]: Through convex matrix factorization with adaptive graph constraint, selects the best feature subset by considering the correlation between the data while preserving the local manifold structure.
Orthogonal basis clustering and Local Structure Preserving (OCLSP) Lin et al. 2022 : An orthogonal basis clustering and an adaptive graph regularization are used to gain cluster separation while keeping the local information of data.
Supervised Feature Selection with Local Adaptive Projection (SLAP) Chen et al. (018b): a supervised feature selection method that simultaneously learns an adaptive similarity matrix and a projection matrix within an itcrative framework.
Robust Neighborhood Embedding (RNE) Liu et al. 2020): Tn this method, features are selected using the neighborhood embedding and L norm as the loss function.
17

---

Published in Transactions on Machine Learning Research (07/2024)
8. Variance-Covariance Subspace Distance based Feature Selection (VCSDFS) Karami et al (002): Variance- CavaanccCovaiinccee subspace distance is used to select the subset which contains the most. representative features.
9. Concrete Autoencoders (CAE) Bahn et al. (019 CAE has an encoder-decoder architecture with concrete selector layer as the encoder and standard neural network as the decoder. It aims at selecting the most informative subset of global features, which are used to simultancously train a neural network to reconstruct the input data.
10. Coordinate Descent based Least Square Regression (CD-LSR) Xu et al. 2023 This method leverages coordinate descent-basee optimization framework to solve the general 42 norm constrained feature selection problem.
A.4 Evaluation Metrics
In the experiments, the clustering performance of feature selection algorithms is evaluated using Accuracy (ACC) and Normalized Mutual Information (NMI) metrics Golorio-Fernánder et al. (2020). The higher ACC and NMT values are, the better the clustering performance is, which in turn results from better performance of the feature selection algorithm to select informative features. The ACC is defined as:
ACC = Ei= 8(g,, map(p:))
where and are, respectively, the clustering and the ground-truth labels of the sample Ii, and is the Kronecker delta fonction. In addition, map(.) demotes the best permutation mapping function that uses the Kuhn-Munkee Munkres algorithm to map each clustering label to the corresponding ground-truth label. The NMI is defined as
NMI MI(P.Q) Q) H(())
where P and Q are the of centers of the predicted and the ground-truth clusters, and MT(.) is the mutual information. Furthermore, H(P) and H(Q) denote the entropy of P and Q, respectively.
A.5 Experimental Settings
Our experiments are performed for various numbers of selected features in the range of {10, 20........ 100), for all datasets. Furthermore, hyperparameter tuning is done find the most optimum configuration of various feature selection models in our study. We fix 5 for the k-ncarest neighbor algorithm and for all the datasets in LS, OCLSP and RNE methods. For CNAFS, the hyperparameters a, 1, 7, and are tuned by searching from {10 3 1 ..... 10²} as mentioned in author's literature in |Yuan et al. (202)). To implement OCLSP, Lin et al. (2022), the parameter a is fixed 10a and 7,, 3 are tuned by selecting from {10-3, 1,0-3,10-2..0 10-2........ 10y) For SLAP, we set the projected dimension m equal to the number of classes, and the regularization parameter is selected from the set {10-3 10-2,0...... ,....., 10 In VCSDES, the values of the parameter p is selected from {10-6 10-5 ..... 10%), Karami al. (2023). We adopted the values for the parameters of RNE from Liu et al. 2020 and lct the penalty cocfficient be 10 For the CAE method, we used the values in the original paper Balm el al. (2019) The performance of CD-LSR was assessed using the provided code based on the author's implementation as described in the original paper of the method Xu] et al. (002).. We tuned the parameters a, 3 and of GRSSLFS method on {10-5 ,..,,,,. 103}. Finally, due to the random initialization in the k-means clustering algorithm, we repeated the clustering runs 20 times with different random starting points over which we reported the average and standard deviations of the ACC and NMI metries. Finally, it should be noted that our tests were conducted utilizing Matlab 2018a on personal computer equipped with 16GB of RAM and Intel Core i5-4690 processor.
A.6 Experimental Results
For more illustration of the reported clustering performance of all comparison methods, in Figures and depict the values of ACC and NMT metries for different. number of selected features.
18

---

Published in Transactions on Machine Learning Research (07 (022)
Baseline
Lesso
L8
CNAFS
OCL3P
RNE
SLAP
VCSDFS
CAE
CD-L3R
GRSSLFS
(a)
(b)
(h)
Figure Clustering ACC of 11 algorithms across eight datasets as function of the number of selected features.
Lesso
VCSDFS
(a)
TOK171
SHACT
(d)
(e)
ONK
ATT
ORL
(h)
Figure 5: Clustering NMI of 11 algorithms across eight datasets as a function of the number of selected features.
19

---

Published in Transactions on Machine Learning Research (07 2204)
A.7 Sensitivity Analysis
We conducted a sensitivity experiment on one image dataset, ORL, and one biological dataset, TOX17I. We fixed the number of selected features at k = 100. Since we have three hyperparameters for evaluating sensitivity, we fixed one parameter and calculated the values of ACC and NMI with respect to the other two varying hyperparameters. We set the value of the fixed parameter to 10-1 for this experiment. The results are illustrated in Figure [ It is evident that our model has relatively steady performance in terms of the hyperparamoters, and the results don't show high fluctuations. As an observation, we can say that GRSSLFS produces belter outpuls for moderale values of the hyperparameters in the range of 10-2 to 10.
1007
TOKTH
TOWT
TOWTH
Figure 6: Sensitivity experiment: impact of the hyperparameters a and 3, and y on the values of ACC and NMI for the datasets ORL and TOX171.
A.8 Running Time Comparison
Figure 7 also demonstrates a comparison between the proposed method and all other unsupervised feature selection algorithms in terms of the runming times (in seconds) for selecting 100 features from the TOX171 dataset. It is evident that CAE is the slowest method, followed by RNE and GRSSLFS methods, as the second and third slowest algorithms. The high running time of CAE is ascribed to the training neural networks in this deep, encoder-decoder feature selector Balmn et al. (2019). The relative sluggishness of the RNE method can be attributed to its non-smooth nature, although it is convex Liu et al. (2020). To solve this issue, RNE employs the alternation direction hod of multipliers (ADMM) to minimize its loss function. The proposed method is the next slow method due to the fact that the GIRSLFES method uses both subspace
Running Time
DMF
Figure 7: Running time (in seconds) of all algorithms to select 100 features for the TOX171 datasct.
20

---

Published in Transactions on Machine Learning Research (07/2024)
learning and self-representationn, wherein multiple matrix multiplications are carried out. Consequently, as demonstrated in Section 2.7 the time complexity of the GRSSL.F method is quadratic in terms of both the number of original features and the number of input data points. Furthermore, as we discussed in Section 2.5 an alternative approach is used to solve the loss function minimization problem of the proposed method.
A.9
Convergence Test
We set out to empirically evaluate Theorem 2.2 wherein we presented a theoretical proof for the monotonic decreasing behavior of the objective function of the GRSSLFS method. Figure S depicts the evolution of the objective function over the number of iterations for all datascts. One can clearly see that the values of the objective function across all datasets plummet quickly and the objective function converges quickly. This observation demonstrates the effectiveness of the alternative iterative algorithm that is proposed solve the objective function of the GRSSLF'S method.
CNS
GLIOMA
Tox171
Oljective Function Value
Objectivv Functton Valuue
Objective Fanction Value
objjettive Function Value
Number iterations
Number of Iterations
Number Rerations
Number Iterations
SMK
23
ATT
ORL
warpAR10P
Objectiv Fumetion Valee
Objecttive Sunction Value
Objective Fundion Value
Mumber Terafons
Number Iterations
Figure 8: The convergence curves of the objective function of GRSSLFS on the eight datasets.
A.10 Non- Non-parametric Statistical Test
We demonstrated in previous sub-sections that the proposed method, GRSSLFS, has superior performance over the state-of-the-art unsupervised feature selection methods. Here, employ the Friedman test as non-parametric statistical test to evaluate the statistical significance of GRSSLFS's performance compared to other methods. The null hypothesis is then expressed as follows: there is no significant difference between any of the methods, and the selected features by these methods lead to the same clustering performance. The alternative hypothesis is that there are at least two methods whose selected features result in significant differences in the clustering performance.
We use the data presented in Table 2 to run the test. In this setting, the datasets, the feature selection methods, and the ACC or NMI values are considered as, respectively, the subjects, the treatments, and the measurements in the Friedman test terminology. The feature selection methods are then ranked to calculate the Friedman statistics. The average rankings over all datasets for each feature selection algorithm and two different measurements (i.e., ACC and NMT) are presented in Figure g The lower ranking means the corresponding measurement is higher.
We obtained 38.022727 and 36.090909 for the Friedman statistics based on ACC and NMI measurements, respectively. Accordingly, the p-values corresponding to these Friedman statistics are 0.000038 and 0.000081,
21

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
Ranking
6.5
73325
5.5
1,75
8MM
8.5
73355
5.25
125 7335
525
4.825
5125 1.25
Figure 9: Average ranks obtained by the Friedman test for cach method with respect to the ACC and NMI evaluation metrics on the datasets. Note that the lower rank of the evaluation metrics, the better the performance of the methods.
respectively. With the significance level of 5%, these two p-values lead us to reject the null hypothesis in the lavor of the alternative hypothesis. With this description, it can be seen that there are al least two methods, VCSDES, CAE and GRSSLES, that select features with statistically significant differences the clustering performance.
Following the Friedman Lest, we also carry out post-hoc statistical test using the Holm's method (aka., Holm Bonferroni method) Bolón- Canedo et al (2014) to identify which pairs of methods result in significant difference in the clustering performance. For this purpose, we select the proposed method, GRSSLFS, the control method and compare it with all other 10 methods including the Baseline which are listed in the columns of Table 2 Tt means that we end up having 10 mull hypotheses for each pair of GRSSLFFS-X where "X' refers to those 10 other methods. Each null hypothesis states that there is no significant difference between GRSSLFS and the method "X' in terms of their feature selection effectiveness with performant downstream clustering. The significance level a is set to 0.05 for each null individual null hypothesis. The p-values of the nine null hypotheses presented in Table 5 based ACC and NMI measurements, respectively. The corresponding Holm's p-values are also included in these tables.
ACC
NMI
Method
p-value
Holm's p-value
Method LS
p-value
Holm's p-value
LS
0
0.005
0.000001
0.005
Baseline
0.000009
0.005556
Baseline
0.000006
0.005556
RNE
0.000526
0.00625
RNE
0.000221
0.00625
CNAFS
0.000911
0.007143
Lasso
0.000396
0.007143
Lasso
0.001998
0.008333
CNAFS
0.000526
0.008333
SLAP
0.003285
0.01
OCLSP
0.015861
0.01
CD-LSR
0.019454
0.0125
SLAP
0.015861
0.0125
VCSDFS
0.034808
0.016667
CD-LSR 0.019454
0.016667
OCLSP
0.050016
0.025
CAE
0.023739
CAE
0.059505
0.05
VCSDFS
0.041831
0.05
Table 5: Post-hoc test using Holm's method to compare the effect of GRSSLES and the 10 other feature selection methods on the clustering ACC and NMI metrics. The significance level of 0.05 set for each individual pairwise mull hypothesis. Holm's procedure rejects mull hypothesis when the p-value of the individual null hypothesis is less than equal to Holm's p-value.
Holm's 's method rejects those hypothescs that their initial p-values are less than or equal to the calculated p-values by Holm's 's method. Thus, we infer from the results in Table 5 that we have enough evidence to reject the null hypothesis for the pairwise comparison between GRSSLES and all methods except for SLAP, OCLSP, CAE, VCSDFS, and CD-LSR. To be more precise, our proposed GRSSLFS method exhibits significant differences from all other methods, except for the aforementioned methods, in terms of the ACC and NMI measures. Nevertheless,
observed in the results of Figures 4 and 5 GRSSLFS displayed
22

---

Published in Transactions on Machine Learning Research (07/2024)
consistent trend over wide range of number of selected features compared to all comparison methods. In addition, although there is no significant difference between proposed GRSSLFS method (an unsupervised approach) and the two supervised approaches, SLAP and CD-LSR, the numerical results demonstrate that the proposed unsupervised feature selection method in this article is fully capable of competing with these supervised methods.
A.11 Application to the PneumoniaMNIST dataset
This section presents the results the selection of 100 features from the PneumoniaMNNST dataset, conducted by our proposed GRSSLFS method and other comparative methods (see Figure 10
A.12
Code Availability
The source code is available at https///sthhab.oom//arrlbbbe1113
23

---

Published in Transactions on Machine Learning Research (07 (07/2024)
(a) Original Images
(b) Lasso
(c) LS
(d) CNAFS
OCLSP
(f) RNE
(g) SLAP
(h) VCSDES
(i) CAE
CD-LSR
Our proposed GRSSLES method
24
Figure 10: The visualization of 100 selected features obtained by different feature selection methods PneUnoniaMNIST images.


Reviews:
[{"review_id": "FX8alDxsiL", "reviewer": "Reviewer_zrhA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "N.A.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I apologize that I didn't review the paper in time and won't be able to do so in the next few days.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N.A.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pTj9XT6aaH", "reviewer": "Reviewer_X5pW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The authors propose a new feature selection method called GRSSLFS that merges the subspace learning and self-representation problems, leveraging the basis of the feature space, to simultaneously eliminate redundant data features and feature selection.\n- The proposed method is compared with a variety of comparison baselines on multiple datasets. The results demonstrate the effectiveness of the method. The ablation study also shows the necessity of each component.\n- Theoretical analysis of the convergence of the method increases the solidness of the article.\n\nWeaknesses:\n\n- The proposed method is called \"Graph Regularized Self-Representation and Sparse Subspace Learning\" (GRSSLFS). However, the authors do not provide a clear explanation of the term \"Graph Regularized,\" which may lead to confusion among readers who are not familiar with this area.\n- The related unsupervised feature selection methods and their key characteristics, as presented in Table 1, can be thoroughly explained. This will provide readers with a comprehensive review of the current state of research in this area.\n- The parameters $\\alpha$, $\\beta$, and $\\gamma$ need to be tuned in a large search space $\\{10^{-5},...,10^5\\}$. This extensive range indicates that the method requires careful adjustment of these hyper-parameters to achieve optimal performance. Consequently, there is a concern regarding the method's sensitivity to these hyper-parameters when applied to various datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To solve the problem of redundant features in the representation space, the authors propose a new unsupervised feature selection method called GRSSLFS, which integrates a basis of linearly independent features with the highest variance scores into a unified framework of subspace learning and self-representation. Both theoretical analysis and experimental results demonstrate the effectiveness of the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors could provide a detailed explanation of the term \"Graph Regularized.\" Specifically, it is important to describe how graph regularization is integrated into the method, what role it plays in enhancing the feature selection process, and how it helps to preserve the geometric structure of the data, which would strengthen the work.\n- The article can include a section dedicated to related work. In this section, the authors can discuss the various unsupervised feature selection methods and their key characteristics as outlined in Table 1, which would provide a comprehensive overview of existing approaches. This addition would not only strengthen the article but also help readers better understand the significance and novelty of the work.\n- The article can include a detailed sensitivity analysis of the hyper-parameters $\\alpha$, $\\beta$, and $\\gamma$. This analysis would examine how variations in these parameters affect the performance of the proposed method across different datasets. By providing this information, the authors can demonstrate the robustness and stability of their method, as well as offer practical guidance on how to tune these hyper-parameters for optimal results, which would strengthen the work.", "broader_impact_concerns": "The work does not raise any concerns related to broader societal or ethical impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vg8wFQIxiW", "reviewer": "Reviewer_3sXe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength is that that the method is evaluated against many other baseline methods across many datasets. To me, this part of the paper seems convincing.\n\nThere are two weaknesses:\n1. The method is only learning in the linear features, but I guess this is not a fixable point\n2. The motivation feels unclear. Why do unsupervised feature selection when there is already quite good supervised feature selection method? I think this point is related to the fact that the paper do not compare with any supervised feature selection method (such as straightforward lasso). In the tables, I think the authors should also show the performance of supervised feature selection methods and convince the readers that using unsupervised learning methods give competitive results (if not better)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a linear feature selection method based on the feature space basis", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness 2.", "broader_impact_concerns": "Fine", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for unsupervised feature selection, which has been verified as effective across multiple datasets. Two reviewers have provided positive comments, highlighting the effectiveness of the method. However, one reviewer has raised concerns about the presentation, such as the use of the term "Graph Regularized," the discussion of related work, and the method's sensitivity. The authors are required to revise the paper carefully according to the reviewer's suggestions. Overall, this paper fits the TMLR acceptance criteria, and I recommend accepting it with minor revisions.
Decision: Accept (poster)