Second-Order Unsupervised Feature Selection via Knowledge Contrastive Distillation
Authors: Han Yue
Jundong Li
Hongfu Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Machine Learning
Unsupervised Feature Selection
Knowledge Distillation

Abstract: Unsupervised feature selection aims to select a subset from the original features that are most useful for the downstream tasks without external guidance information. While most unsupervised feature selection methods focus on ranking features based on the intrinsic properties of data, they do not pay much attention to the relationships between features, which often leads to redundancy among the selected features. In this paper, we propose a two-stage Second-Order unsupervised Feature selection via knowledge contrastive disTillation (SOFT) model that incorporates the second-order covariance matrix with the first-order data matrix for unsupervised feature selection. In the first stage, we learn a sparse attention matrix that can represent second-order relations between features. In the second stage, we build a relational graph based on the learned attention matrix and perform graph segmentation for feature selection. Experimental results on 12 public datasets show that SOFT outperforms classical and recent state-of-the-art methods, which demonstrates the effectiveness of our proposed method.

Under review as a conference paper at ICLR 2022
SECOND-ORDER UNSUPERVISED FEATURE SELEC- TION VIA KNOWLEDGE CONTRASTIVE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised feature selection aims to select a subset from the original features that are most useful for the downstream tasks without external guidance informa- tion. While most unsupervised feature selection methods focus on ranking features based on the intrinsic properties of data, they do not pay much attention the relationships between features, which often leads to redundancy among the selected features. In this paper, we propose a two-stage Second- -rderr unsupervised Feature selection via knowledge contrastive disTillation (SOFT) model that incorporates the second-order covariance matrix with the first-order data matrix for unsuper- vised feature selection. In the first stage, learn sparse attention matrix that can represent second-order relations between features. In the second stage, we build relational graph based on the learned attention matrix and perform graph segmentation for feature selection. Experimental results on 12 public datasets show that SOFT outperforms classical and recent state-of-the-arr methods, which demonstrates the effectiveness of our proposed method.
INTRODUCTION
In the digital world, huge amounts of high-dimensional data (Guyon et al., 2004 Bajwa et al. 2016 Yang et al., 20199] Maoying et al. 2020) are captured every day. Due to the existence of irrelevant or redundant features, data high dimensions may significantly increase the computational costs and bring challenges for efficient and effective data management. Dimensionality reduction is one of the most well-known techniques to address the above issue, which can be categorized into feature transformation and feature selection. Feature transformation, also known representation learning, aims to project the original high-dimensional features into a new www-dimnsioonl feature space. The new feature space is usually linear nonlinear combination of the original features and does not have semantic meanings, so it is hard to be interpreted. On the other hand, feature selection methods try to select a subset of relevant features from all available features based on a predefined criterion, maintaining physical meanings of the original features for better interpretability.
Without label information, unsupervised feature selection methods aim to select feature subset that can preserve the intrinsic structure of the whole feature set accurately. There are many algorithms Kononenko
designed to solve the unsupervised feature selection problem. ReliefF (Robnik-Sikonja
2003), HSIC (Gretton al., (005),, Laplacian Score (He et al., 2005). SPEC (Zhao & Liu| (007)),
SPFS (Zhao et al. 2011) evaluate features by their capability in preserving the pairwise sample similarity. UDFS (Yang al. 201T) FSASL (Du & Shen 2013), and TSFS (Mirzaei et al., 2020)
employ pseudo labels as the supervision to guide the feature selection along with sparse constraint.
Most of these methods apply the linear feature selection matrices and select the representative features by ranking their feature weight vector. Such operations treat the feature set independently and fail to tackle the complex high-order relationship (Zhang et al.) 2017 Zhu al., 2020) among original features which inevitably brings in the redundancy among the selected features.
We propose a two-stage Second-Order unsupervised Feature selection via knowledge contrastive disTillation (SOFT) model that incorporates the second-order covariance matrix with the first-order data. In the first stage, SOFT learns sparse attention matrix to explore the second-on -odder feature relationships. In the second stage, perform graph segmentation the learned attention matrix for feature selection. In summary, we highlight our contributions as follows: (1) We consider the second-order feature relationship in the unsupervised feature selection problem and propose
Contributions:

---

Under review as a conference paper at ICLR 2022
the SOFT algorithm to distill knowledge from both original features and their second-order feature covariance matrix; (2) Our SOFT learns a mask matrix on or off the covariance matrix and obtains the attention matrix and masked matrix. Throughout distilling the structural knowledge, the sparse attention matrix contains such knowledge as much as possible while excluding that from the masked matrix well; (3) Different from selecting features according to weights, we propose a graph segmentation-base feature selection method on the attention matrix, where only one representative feature is selected from each segment to avoid redundancy; and (4) Experimental results validate that our SOFT model outperforms classical and recent state-of-the-art art models on 12 public datasets. We also provide in- -e-epthh exploration for both stages to demonstrate the effectiveness of SOFT.
METHODOLOGY
Motivation. Unsupervised feature selection aims to select small portion of the original features that are most useful for the downstream tasks without external guidance. Most previous methods focus ranking features based on the values of individual fea-
08
0.
04
tures (He et al. 2005 /Zhao& & Liu, 2007 Yang al.) 2011 while neglect the high-order relationships be- tween features. Unfortunately, this might lead to the redundancy of the selected features and further deteri- orate downstream tasks. Figure provides an illustra- tive example of selection results on Sonar (Rossi & Ahmed (2015) and Waveform (Breiman et al. 1984) by Laplacian Score (LapScore) (He et al. 2005). The heatmap shows the initial relationships between fea- tures based on the covariance matrix of features. We remove the diagonal values and calculate the abso- lute values of covariance, which are normalized for visualization. In this example, we select the top four features on Sonar and Waveform by LapScore, which are highlighted with red lines in Figure [] Obviously, features a and b, features and d on Sonar are from two groups within of high similarity, indicating that one feature might be denoted by the other. Similarly, On Waveform, features e, f. and h contain highly relevant information lead to redundancy, which disobeys the purpose of feature selection. This drives us to explore the complex relationships derived from the second-order feature covariance matrix in the unsupervised feature selection problem. Motivated by the above situation, we design a method that can avoid such redundancy by considering both first-order data and second-order data.
02
000
1.0
088
04
00
Sonar
Haweform
Figure 1: Visualization of feature relations of Sonar and Waveform datasets. Selected features by LapScore are marked by red lines.
well. However, features from the same feature group might
Framework Overview. To incorporate the correlation feature information, we propose a two-stage model SOFT, which takes the first-order data matrix and second-order feature correlation matrix as inputs. In the first stage, SOFT learns a mask on the feature correlation matrix via knowledge contrastive distillation, which is beneficial preserve the data structure. In the second stage, select the features on the masked correlation matrix via graph segmentation.
Figure 2 shows the overview of our SOFT framework. With the first-order data matrix and second- order correlation feature matrix as inputs, we aim to learn a mask matrix applying to the feature correlation matrix for feature selection. By this means, we obtain the attention matrix and masked matrix by the learnable mask on or off the feature matrix, respectively. The key idea of SOFT is distill the structural knowledge via making the attention matrix contain such knowledge as much as possible while excluding that from the masked matrix as well. The green and red lines in Figure 2 demonstrate the knowledge contrastive distilling process. To achieve this, we adopt shared Graph Convolution Network (GCN) (Kipf & Welling 2016) to generate antention//oiginal/maaa representations from the dncention/featuremmaa
matrices, respectively. Then we use pseudo labels generated by attention representation as positive guidance for original representation and negative guidance for masked representation so that attention representation and original representation are close to each other and far away from masked representation. Throughout the above knowledge contrastive distilling process, we can get a sparse and effective feature relation matrix that represent the second-order correlation, where each node denotes feature and weights of edges are the
2

---

XXXX Under review as a conference paper at ICLR 2022
Sample
Attention
Shared GCN
Attention Representation C₁
Feature Matrix
Learnable Mask 0N
Original Representation G7
Pscudo Label
Masked Representation Gy
Masked Matrix My
Shared GCN
Figure 2: SOFT model framework. The inputs of our SOFT model consists of the first-order data matrix and second-order correlation feature matrix. The green line shows that attention representation is expected to preserve the intrinsic information from the pseudo labels and gets close to original representation, while the red line presents that the masked representation contains little structural information and gets away from the attention representation. Finally, we could get sparse and effective feature relation matrix that can represent the second-order correlation.
corresponding values the learned attention matrix. In the second stage, different from the existing work, which calculates the weight of each feature and suffers from the redundant feature issue (shown in Figure TT. our attention matrix delivers the weight for a pair of features. To proceed with the feature selection,
graph segmentation cut the attention matrix into partitions and select one feature that has the highest relationship to others from each partition as the final selection result. By this means, the redundancy among the selected features can be mitigated.
Attention Matrix Learning. Given n data in- stances with d features, we have the first-order n x data matrix X and the d x d second-order feature matrix Mp X T X. Our SOFT model calculates a learnable mask for feature selection. the first stage, SOFT consists of four components, attention layer, shared GCN, pseudo label generation, and contrastive learn- ing. We use θ {0M, 0g, fc) to denote the trainable parameters set in the SOFT model, where fM, fc. and Pc denote the parameters of the attention layer, shared GCN and con- trastive learning, respectively. Note that the pseudo labels are generated from the attention matrix and shared GCN, which are controlled by On and 0g. Therefore, no learnable parameters needed for pscudo label generation. Table I shows the notations used in our SOFT model and their descriptions. Each part is detailed as follows.
Table 1: Notations and description
Notation
Dimension
Description
scalar
Number of input samples
scalar
Number of festures
scalar
Number of clusters
X
12 d
Input data matrix Inpui
Mr
matrix
MA
d x d
Attention matrix
Ma
matrix
GF
Representations generated X and My
GA
T2 d
Representations generated by and M Representations generated X and M
GM
PF
Predictions of samples by GF labels generated G. Predictions of samples by Ga
P'A
PAr
X
Attention Layer. The initialized feature matrix Mp may not be good enough to represent the relationship between features, so we add attention layer to better capture second-order feature interactions by highlighting the important relations and reducing others through learnable mask fxr- The attention matrix MA, A, which represents the important part of Mr, is calculated as follows:
MA M O 0m.
(1)
where @ is the element-wise product. Due the symmetry of Mp, 0w is forced to be symmetric by adding a x d parameter matrix with its transpose. With the learnable mask 0M and the attention matrix MA. We assume that the input feature matrix My can be decomposed summation
3

---

Under review as a conference paper at ICLR 2022
formulation of attention matrix MA and masked matrix MM. Then we define the masked matrix Mas by the following function:
MM = Mp- M
To make the learned attention matrix MA sparse enough to identify crucial and principle relations. we apply 1nomm to learnable mask MM on both row and column level, which is:
221 = 1|0a1112,1 + 110
(3)
Shared Graph Convolutional Nenwork. GCN (Kipf & Welling 2016) helps generate embeddings that contain both data information and feature relationship information. We apply a shared 2-layer GCN representations from the attentinnfeaturrrmassee matrices, respectively. In Kipf & Welling (2016), nodes of graphs denote samples, while in case, nodes of graphs are features. Therefore, the computation of the GCN layer is little different from Kipf & Welling (2016) and is described by the following equation:
to extract the attention/original/maan
(1+1) ReLU D ADD 20()
(4) where M is the input relationship matrix, M M + Ia. Ia Raxdd is the identity matrix, D is the degree matrix of M, and 0 is a layer-specific trainable weight matrix. GO denotes the embeddings in the (-th layer, and ( = X. In our case, we use a shared 2-layer GCN to process samples with different input relationship matrices MA. Mr, and MM. Thus we have the attention representation GA, original representation Gr, and masked representation GM., respectively.
Gea
Pseudo Label Generation. In the unsupervised feature selection, pseudo labels are usually used as the criterion to guide feature selection. In our experiments, we use Principal Component Analysis (PCA) (Wold al., 1987) and K-means to generate pseudo labels (Caron et al., 2018). The embed- dings processed by the clustering part is generated by attention matrix MA. which is also the matrix we expect to learn in the SOFT model. PCA helps get principal features, and K-means clustering generates pseudo labels PA for the input samples X.
Contrastive Learning. With the above representations, we expect that attention representation and original representation are close to each other but far away from masked representation. To achieve such contrastive learning, we design two losses to measure the predictive abilities of different representations and compare them with the pseudo labels. Specially, we employ a 2-layer Multi-Layer Perceptron (MLP) to get predictions of the samples with original representation GF and masked representation G about which clusters they belong to,
1((+1) = aPPl p!l) 8c x )
(5)
where 0 is layer-specific trainable weight matrix, p()) denotes the embeddings in the 1-th layer, and p(0) GF or p()) G for the original representation and masked representation. respectively. We use (-) ReL.U()) as the activation function for layers before the last layer, and a()) = Softmax(-) for the last layer. In our experiments, we adopt a shared 2-layer MI.P and obtain PF and PM for original representation GF and masked representation GM.
Pseudo labels PA generated by the clustering part are used for the following positive and negative training. By positive guidance for Pr, we get cross-entropy loss LE for PF, which is defined as:
LF = Σ ==
ΣIeR (PAi log PFij + (-PA - -PFy)),
(6)
where Pry denotes the probability that the i-th sample belongs cluster j based on Mp. PAi 1 if the i-th sample belongs to cluster j based on the clustering result, otherwise PAij = 0. Eq. ( is designed to make Pe similar to pseudo labels PA. While by negative guidance for predictions of the masked part Par, we apply attention loss described in [L et al. (2018b) to reduce the weights of unimportant relations, which stated as follows based on our notations:
CM
PAcyPMai
4

---

Under review as a conference paper at ICLR 2022
where PMry denotes the probability that the i-th instance belongs cluster based MM. Through minimizing Eq. 77. the model tends to put instances to cluster that they do not belong to, thus driving predictions Pw generated by masked representation different from pseudo labels PA. which would finally lead to the difference of masked representation Gas and attention representation A-
Objective Function. Combining Eq. can be written as follows:
Eq. 66. and Eq. 7 Our overall objective function of SOFT
mine LF + aLM + 3L2,1,
where a and 3 are hyperparameters for LM and L2.1, respectively. We adopt Adam opti- mizer (Kingma & Ba 2014) to minimize the objective function. The detailed algorithm can be found in Appendix A
Feature Selection on Attention Matrix. Different from the traditional feature selection methods that return a weight vector to choose the top-ranked features, here, our attention matrix provides the weight of pair of features. To proceed with the feature selection, we build a feature graph based on the attention matrix. Nodes of the graph denote features, and edges are the relationship between features in the attention matrix. Then we perform graph segmentation for feature selection.
Graph Construction. Instead of transforming the attention matrix MA to a graph immediately, we add two additional processes so that the generated graph is more suitable for graph segmentation and feature selection. The first one is to remove some irrelevant features. We calculate score for each feature by Ei= MAi. where matrix, which measures the importance of each feature. We delete the last 10% features based on the sorted The second step is to set all values smaller than the median value M, A to zero so that the number of unimportant edges is largely reduced. We also set all negative values in MA to zero to make sure no negative edges would exist in the constructed graph.
the summation of each row or column of the attention
Graph Segmentation. With the processed graph, we apply the graph segmentation method provided by Karypis & Kumar [1998) and cut the graph into parts, where & is the number of features we aim to select. For each partition, we choose the feature that has the largest value in S, since we consider features in the same partition are highly related.
Through learning attention matrix in the first stage of our method, we incorporate both first-order data and second-order data and get a refined feature relation matrix that can better reflect feature relations. Then in the second stage, build graph based on the learned attention matrix and perform graph segmentation, which groups nigh-correlated features together. Therefore, by selecting features from each partition, our method reduces redundancies among the selected features. Our proposed method focuses on learning pair-wise relationships of features and uses graph segmentation to select features. The time complexity of our method is (nd?). If the number of samples larger than the number of features (n > d), the space complexity of our method is O(nd), otherwise (d?).
3 EXPERIMENTAL RESULTS
Experimental Settings. We introduce the experimental settings below.
Datasets. We select 12 public feature selection benchmark datasets of different types for evaluation:
COIL20 Nene et al., 1996). Colon (Alon al. 1999). Gisette (Guyon et al. 2004). Lung-Cancer (L. Cancer) (Hong & Yangl 1991), Madelon (Guyon et al. (007), MovementLibree (M. Libras) (Dias)
(et al. NC79 (Ross al. 2000), ORL (Cai | (006)) Sonar (Rossi & Ahmed (015). CAVI
and UAV2 (Zhao et al.) 2018)) and Waveform (Wave.) (Breiman et al., 1984). The instance numbers of
these dataset range from 32 to 19937, the feature numbers range from 40 9712, and the true cluster
numbers are from 2 to 40. The sample/feature ratios of these datasets range from 0.01 to more than 350, indicating the diversity of the datasets. Detailed dataset statisties can be found Appendix B
Comparative Methods and Implementation. We choose 10 classical and recent state-of-the-an unsupervised feature selection methods for comparison. Laplacian Score (LapScore) (He et al. 2005) selects features by scoring features with Gaussian Laplacian matrix. SPEC (Zhao & Liu 2007) is a more general framework for feature selection based on spectral graph theory, where LapScore is special case of it. MCFS (Cai et al. 2010) uses spectral analysis and sparse regression to select features and capture the multi-cluster data structure. UDFS (Yang et al.) 2011) selects features by discriminative analysis and lg11 minimization. NDFS (Li al. 2012 selects the most
5

---

Under review as a conference paper at ICLR 2022
Table 2: Results of different UFS methods on 10% selected features in terms of accuracy
Datasct
LapSoore
SPEC
MCFS
UDFS
NDFS
LRPFS
NSSLFS
TSFS
CAE
InfFS
SOFT
CO1120
00.56:0003
0.59::002
0.63=0022
0.55
0.61=6002
0.57=002
0.642204
0.60
0.654am2
0.58=0.05
0.67::01
Colon
0.34:10.0
0.55::0.0
0,53-0,00
0,52+0000
0.55=0.00
0.551:001
0.55:0000
0.54-00.3
0,53:1011
0.55
0.56:10000
Gisette
0.67::0000
0.70:10000
0.80=0.00
0.58::000
0.74=0.01
N/A
NA*
0.57::0.00
0.62
0.61:10.c
0.811:0.00
L.ancer
0.75+0000
0.59:0000
0.69=0.00
0.65+0011
0.89=0.00
0,78+0000
0.78++000
0.60=0.05
0,56+0002
0.69-0.10
0,78+000
Modelon
0.51
0.51:0000
0.53-0:00
0.51
0.60
0.50
0.53+0000
0.57=0..5
0.56::000
0.52=0.05
0.60::000
M.Librus
0.274-0.01
0.294+001
0.38-0.01
0.35+002
0,43=0:01
0.39:0001
0,37++0.11
0.36=0003
0.4340022
0.44--0.05
0,46++0.00
NC19
0.44+0002
0.43++002
0.38=0:02
0,44+003
0.43=0:02
0.42#000
0.37=0003
0.44=0.02
0.46::004
0.40=0005
0.44+008
ORT
0.49:0002
0.56::002
0.56=0.02
0.47:002
0.57=6:02
0.43:00.2
0.562003
0.57=0.01
0.51:001
0.52=0.02
0.58:0001
Sonar
0.57::000
0,54::0.00
0.58=0.00
0.5410000
0,52=00.0
0.52::000
0.64::000
0.56220.04
0,56:10.00
0.58:0.00
0.641:000
UAVI
0.56::0:00
0.67:40.00
0.54=0.00
0.55
0.65=00.0
NIA
0.60:40:00
0.65
0.56:10.00
0.78 0.55
UAV2
0.80+0000
0.60+:000
0.76=0:00
0.814000
0.58=0:00
0.55+0000
0.56+0000
0.64+0.01
0.81+000
0.58=0.02
0.82++0.00
Wave.
0.00
0.34
0.51 =0:00
0.52+000
0.49=0.00
0.48
0.37=0000
0.52=0.00
0.51 0.51
0.54
Average
0.55
0.53
0.57
0.54
0.57
0.52
0.54
0.55
0.56
0.54
0.64
N/A:
indicates the algorithmic running crror, and
means our-of-64CB-mmmmoyy ervor.
Accuracy
0.70
Accuracy
0,59
Accuracy
0.80
0.66
0.57
0.75
0.62
0.55
0.70
0
0.58
0.53
0.53 10%
0.65
100
15%
20%
25%
30%
15%
20%
25% 30%
10%
15%
20%
25%
30%
Selected Features
Selected Features
Selected Features
(a) COIL20
(b) Colon
(c) Gisette
Agcuracy
0.85
0.62
0.48
0.75
Accuracy
0.58
Accurecy
0.44
0.65
0.54
0.40
0.55
0.50
0.36
10%
15%
20%
25%
30%
10%
25% Selected Features
15%
20%
30%
10%
15%
20% 25%
30%
Selected Features
Selected Features
(d) Lung- Cancer
(c) Madelon
(f) MovementLibras
0.47
0.60
0.64
Accurscy
0.45
Accuracy
0.59
Accuracy
0.60
0.58
0.43
0.57
0.56
0.4110%
0.41 10%
0.56
0.52.
15%
20% 25%
30%
10%
20% 25% 30% Selected Features
15%
10%
25% Selected Features
15%
20%
30%
Selected Features
(g) NC19
(h) ORI.
(i) Sonar
0.85
0.55
Accuracy
0.85
0.75
Accuracy
0.75
Accurncy
0.53
0.65
0.65
0.51
0.55
0.55
0.49 10%
10%
15%
20% 25%
30%
10%
15%
20% 25% 30%
15%
20%
25% 30%
Selected Features
Selected l'eatures
Selected Features
(j) UAVI
(k) UAV2
(I) Waveform
LapScore
MCFS UDFS
NDFS LRPFS
NSSLFS TSFS
CAE
SOFT
SPEC
Inf
Figure 3: Performance of 12 UFS methods on different percents of sclected features. On each dataset,
only top 5 methods on average performance are displayed for better visualization.
discriminative features with a nonnegative constraint and L2,1 regularization. LRPFS (Zheng et al. 2018) adopts low-rank constraint to preserve the subspace structure information. NSSLFS (Zheng al. 2019) learns the feature weight matrix with the f2orm and the neg-neeative constraint based on the low-dimensional sparse subspace learning. TSFS (Mirzaei et al. 2020) employs a teacher-student scheme for deep feature selection. CAE (Balin et al., 2019) uses the concrete distribution and the reparametrization trick to differentiate through an reconstruction loss and select input features. InfFS (Roffo et al.) 2020) is a fast graph approach which ranks and selects features by considering the possible subsets of features as paths a graph.
For LapScore, SPEC, MCFS, UDFS, and NDFS, we adopt implementations and default settings provided by seikit-feature (Li et al. 2018a). For LRPFS and NSSLFS, set the values of hyperpa- rameters in their objective functions to I.0. For TSFS, CAE (non-linear version), and InfFS, we use default settings provided in their open-source codes. The settings of our SOFT model are follows.

---

Under review as a conference paper at ICLR 2022
Attention Matris
08
0.6
94
92
Learnable Mask
04
02
-00
Epoch 0
Epoch- 50
Epoch 100
Epoch 150
Epoch spoch-200 200
Figure 4: Visualization of attention matrix and learnable mask during iterations on MovementLibras.
In the stage of Attention Matrix learning, we implement the networks by PyTorch. The Learnable Mask is initialized by the normal distribution. The weights a and 3 in the objective function are and 0.001, respectively. We adopt adam optimizer (Kingma & Ba 2014) to minimize the objective function and set the learning rate to lc-4. We run a total of 300 epochs to learn the attention matrix. In the stage of graph segmentation, attention matrix and set the least 50% values of the remaining 90% features in attention matrix to zero before graph construction. Then we perform graph segmentation and select feature from each partition. For the re-implementation, the link of code and datasets can be found in Appendix C
first remove 10% features irrelevances based on the learned
Evaluation Metric. We employ k-means++ (Arthur & Vassilvitskii, 2006) on the selected features, and compare the obtained partition and ground truth by clustering accuracy, settings of which follows scikit-feature (Li et al. 2018a).
Algorithmic Performance. Table 2 shows the experimental results of different unsupervised feature selection methods on 10% selected features in terms of accuracy. The best results are highlighted in bold. "N/A" means the corresponding method cannot process the dataset successfully due to running error or out of memory error. We can see that our SOFT model achieves the best on 11 of the 12 datasets. One of the possible reasons is that SOFT explores the second-order relationships among features, while other competitive methods only used the first-order data, while we incorporated second-order data in the feature selection process. By grouping features based on the learned attention matrix, our method can avoid the redundancy described in Section [ Moreover, the average accuracy of SOFT is significantly better than other methods with large margins of 5% to 10%, which demonstrates the positive effects of second-order feature exploration on the unsupervised feature selection problem. Beyond the algorithmic effectiveness, also provide the algorithmic execution time of different algorithms in Appendix D
Next, show the performance of all the methods on selecting different percents of features in Figure LapScore achieves the best performance with 20% selected features on Madelon, LRPFS is always the best among these methods on I. Cancer, NSSLFS outperforms other methods with 15% and 20% selected features on Colon, and TSFS with 15% selected features is ranked the first on NC19. It is worthy to note that sometimes the performance of SOFT is not so stable as other methods, such as on L. Cancer when the number of selected features increased. This is because the comparison methods generate a ranking list for features and select features based on the feature ranking scores, while SOFT select features based on the graph segmentation result. With the number of selected features increasing, the feature partitions might change notably, which results a different selection. In general, SOFT delivers the compromising features with different percents compared with others, especially when the percent of selected features is relatively small. For results with all percentages and datasets, SOFA outperforms all other methods with average accuracy of 61 %, which is significantly better than two recent methods (CAE achieves 54%, and InfFS gets 51%).
In-depth Exploration of SOFT. We provide the in-depth exploration of SOFT from the following aspects. Additional experimental results can be found in Appendix E-H.
Parameter Analysis. There two parameters in the objective function of SOFT, and 3, denoting the weight of attention loss and 4,1 norm loss, respectively. We vary a and 3 from le - 3 to 1e + 3 to explore the impact of these two parameters the final performance. Figure Figure[[(a)ssa shows the results
7

---

Under review as a conference paper at ICLR 2022
0.8
Maselax
L.Cuer
Ghatt
Skm
0.5
0.6
Millvws
Largod
SOFT
0.15
04
A3
Fist
NC19
CDIL.0
Covnrisnce
1000
SOFT
GR
Worc:
0.001
6/ I0on
on
coll coth
JN
UND
Hom
UXP2
DAYI
(a) Parameter analysis
(b) Attention matrices
(c) Strategies
Figure 5: In-depth exploration of SOFT. (a) Parameter analysis of a and B Colon, (b-c) perfor-
mance of feature selection with different attention matrices and different feature selection strategies.
on Colon with 10% selected features. We can see that despite the large range of parameter values, the final performance does not change much, indicating that the Learnable Mask is well learned and SOFT might not be sensitive to the value of and 3 in large range.
Visualization of Artention Matrix and Learnable Mask. To further analyze the performance of SOFT, we visualize the Attention Matrix and the corresponding Learnable Mask different epochs on Movementl ibras, which is shown in Figure 4 The darker color indicates the stronger correlation of the corresponding pair of features. To better recognize feature relations, we remove the diagonal values for visualization. In the beginning. there is no identified pattern in the Learnable Mask because the learnable mask is randomly initialized. As the training epochs increased, the Icarnable mask becomes sparser and seeks the dataset-dependent patterns. While the attention matrix is generated by an element-we -wise product of the original feature matrix and the learnable mask, the attention matrix became sparser well. Therefore, the most important part of the attention matrix for representing samples was highlighted through network training.
Then we do further step explore the effectiveness of the learned attention matrix by comparing our results with two designated baseline methods. The first one is the First Order method, which only uses intrinsic properties of the data. We remove the GCN part of SOFT and utilizes vectorial learnable mask for the first order network. The First Order learns the ranking scores for features, which is denoted by the Learnable Mask. The second one is the Covariance method, which uses the original feature matrix directly for the second stage of SOFT. Figure 5(b) shows the experimental result in terms of accuracy. On almost all the datasets, both SOFT and Covariance performed better than First Order, which demonstrates the effectiveness of incorporating feature relation in feature selection. While sometimes Covariance achieves better result than SOFT, SOFT performed the best in most cases and achieved the best average, proving that SOFT could learn sparse and effective feature relation matrix to represent the second-order correlation.
Feature Selection Strategy. We adopt two baseline methods for feature selection from Attention Matrix as the comparison with the graph segmentation method in SOFT. The first baseline method, Weight Sum, utilizes row-sum method to get the total relation value of each feature all other features as the ranking scores for features. Features with the highest scores are selected. The second baseline method, Largest Weight, each time finds the largest value in Attention Matrix and selects the corresponding feature pair until features are selected. Comparison results among Weight Sum, Largest Weight, and the graph segmentation in SOFT are shown in Figure 5(c) Overall, the graph segmentation method in SOFT has significant advantage over the baseline methods. This results from that SOFT avoids selecting highly correlated features by graph segmentation, thus reducing redundancies and bringing in more complementary features for performance boosting.
RELATED WORK
We briefly review related work on unsupervised feature selection and deep feature selection below.
Unsupervised Feature Selection. In the past decades, a large amount of unlabeled data are generated. To solve the feature selection problem for the unlabeled data, researchers have proposed many unsupervised feature selection methods (Liu et al. 2016 Li et al. 20) which can be divided into three main categories: Filter, Wrapper, and Hybrid. Filter methods evaluate features based on the data itself. Dash et al. (1997) propose one of the earliest filter unsupervised feature selection method, Sequential backward selection method for Unsupervised Data (SUD). SUD introduces a similarity matrix representing the pair-wise similarity between objects. By measuring the entropy
8

---

Under review as a conference paper at ICLR 2022
of the data, the relevance of each feature is quantified as ranking score. Features with the highest scores are selected. Another example is Laplacian Score (He et al. 2005), which weights features according their abiliity to preserve predefined manifold structure represented by the Laplacian matrix. Similarly, SPECtrum decomposition (SPEC) (Zhao & Liu 2007) also introduces an object similarity matrix. SPEC measures the consistencies between features and nontrivial eigenvectors of the Laplacian matrix and ranks features based on the consistencies. Wrapper methods select the most relevant features by using a clustering algorithm. (Dy & Brodley (2004) introduce a method
to select feature subsets using Expectation Maximization (EM) (Dempster et al. 1977) clustering
and evaluate them with maximum likelihood and the scatter separability criterion. Kim et al. (2002) present an evolutionary multi-objective local selection algorithm to search feature subsets with K-means (Hartigan & Wong 1979) and EM (Dempster et al., 1977) clustering. Instead of selecting feature subsets, Law et al. (2004) propose to estimate set of real-valued quantities carried out by an EM algorithm through adopting a minimum message length (Wallace & Dowe 2000) penalty. Hybrid methods try to take advantages of both approaches by adopting a two-stage process: filter stage and wrapper stage. In the filter stage, the features are scored based on the intrinsic properties of the data. And in the wrapper stage, feature sets generated by specific clustering algorithm. For instance, Dash & Liu (2000) adopt the method of Dash et al. (1997) for the filter stage to sort the features and the method of Dy & Brodley (2004) for the wrapper stage to build clusters. [Solorio-Feemånder it al. (016)) combine the spectral feature selection framework using the Laplacian Score (He et al. 2005) ranking and modified Calinski-Harabase index (Calinski & Harabasz 1974). Different from the above-mentioned methods which are based on ranking, Kim & Gao (2006) propose a method that starts with the wrapper stage by Seast-Square-Fstimnee Estimation based evaluation and then selects feature set through a Bayesian network in the filter stage. InfFS (Roffo et al., (020)) is graph-based filtering approach, which evaluates the values of paths in a graph and selects discrete input features by exploiting properties of power series of matrices and the concept of absorbing Markov chains.
Deep Feature Selection. Recently, deep learning techniques have gained much attention and brought in some studies deep feature selection (Chang et al.) 2017 Taherkhani et al. 20188). DFS (Li et al., 2016b) adds a weight layer to Multi-layer Perceptron (MLP) together with a sparse regularization term as take advantage of deep structures to model nonlinearity. /Zhao al. (2015) propose to combine deep neural networks with sparse representation for grouped heterogeneous feature selection. The model first converts the multi-modal data into unified representation, then selects features through solving a sparse group lasso (Friedman et al. 2010) problem. In recent years, some studies also involve data reconstruction error deep unsupervised feature selection. AEFS (Han al. 2018) jointly learns a self- representation autoencoder model and the importance weights of each feature for feature selection. Furthermore, GAFS (Feng & Duarte, 2018) not only adopts a single layer autoencoder but also incorporates the spectral graph analysis for learning. UDSFS (Cong et al., 2016) selects the most discriminative features and meanwhile designates appropriate weights to the feature dimensions by utilizing the group sparsity of the features. TSFS (Mirzaci et al.) 2020) presents a teacher-student scheme for deep feature selection, in which a teacher network is used to learn low-dimensional representations, and student network is employed for feature selection by minimizing the reconstruction error. CAE (Balin et al. 2019) uses a concrete selector layer as the encoder and standard neural network as the decoder, stochastically selects discrete input features by concrete random variables and the reparametrization trick to get a subset of input features.
CONCLUSION
In this paper, we proposed a two-stage framework named SOFT for unsupervised feature selection, which incorporated second-order data with first-order data. Specifically, in the first stage, we first obtained the Attention Matrix and Masked Matrix by applying a learnable mask on and off the input Feature Matrix. Then we generated Attention/OriiinllMMasae Antention/FeatureMMaskkee Matrices by a shared Graph Convolutional Network. To train the learnable mask, we used pseudo labels generated by Attention Representation as positive guidance for Original Representation and negative guidance for Masked Representation, such that Attention Representation and Original Representation were similar to each other and different from Masked Representation. In the second stage, we constructed graph based on the well-learned Attention Matrix and utilized graph segmentation to separate the graph into several parts. We chose one feature from each partition the feature selection result. Experiments on public datasets demonstrated that our method outperformed classical and recent of-e-oofthhear methods tackling the unsupervised feature selection problem.
Representations from

---

Under review as a conference paper at ICLR 2022
REFERENCES
Uri Alon, Naama Barkai, Daniel A. Notterman, Kurt Gish, Suzanne Ybarra, Daniel Mack, and Arood J. Levine. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. National Academy of Sciences, 96: 6745 6750, 1999.
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
Gagan Bajwa, Ralph DeBerardinis, Baomei Shao, Brian Hall, David Farrar, and Michelle A. Gill. Cutting edge: Critical role of glycolysis in human plasmacytoid dendritic cell antiviral responses. The Journal of Immunology, 196:2004-209,, 2016.
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference machine learning, pp. 444 453. PMLR, 2019.
Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. Classification and regression trees. CRC press, 1984.
Deng Cai, Xiaofei He, Jiawei Han, and Hong-Jiang Zhang. Orthogonal laplacianfaces for face recognition. IEEE Transactions on Image Processing. 15:3608-3614, 2006.
Deng Cai, Chiyuan Zhang, and Xiaofei He. Unsupervised feature selection for multi-cluster data. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, 2010.
Tadeusz Caliński and Jerzy Harabasz. A dendrite method for cluster analysis. Communications in Statistics-theory and Methods, 3:1-27, 1974.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference on Computer Vision, pp. 132-149, 2018.
Chun-Hao Chang, Ladislav Rampasek, and Anna Goldenberg, Dropout feature ranking for deep learning models. arXiv preprint 1rXiv:1712.08645: 2017.
Yang Cong, Shuai Wang, Baojie Fan, Yunsheng Yang, and Haibin Yu. Udsfs: Unsupervised deep sparse feature selection. Neurocomputing, 196:150-158, 2016.
Manoranjan Dash and Huan Liu. Feature selection for clustering. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2000.
Manoranjan Dash, Huan Liu, and Jun Yao. Dimensionality reduction of unsupervised data. In IEEE International Conference Tools with Artificial Intelligence, 1997.
Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, 39:1 1977.
Daniel B. Dias, Renata C.B. Madeo, Thiago Rocha, Helton H. Bíscaro, and Sarajane M. Peres. Hand movement recognition for brazilian sign language: a study using distance baased neural networks. In International Joint Conference Neural Networks, 2009.
Liang Du and Yi-Dong Shen. Unsupervised feature selection with adaptive structure learning. In ACM SIGKDD International Conference Knowledge Discovery & Data Mining, 2015.
Jennifer Dy and Carla E. Brodley. Feature selection for unsupervised learning- Journal of Machine Learning Research, 3:845-889, 2004.
Siwei Feng and Marco F. Duarte. Graph autoencoder-based unsupervised feature selection with broad and local data structure preservation. Neurocomputing, 312:310-323, 2018.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A note on the group lasso and a sparse group lasso. arXiv preprint 1001 1001,0336, 2010.
10

---

Under review as a conference paper at ICLR 2022
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, 2005.
Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the nips 2003 feature selection challenge. In Advances in Neural Information Processing Systems, 2004.
Isabelle Guyon, Jiwen Li, Theodor Mader, Patrick A Pletscher, Georg Schneider, and Markus Uhr. Competitive baseline methods set new standards for the nips 2003 feature selection benchmark. Pattem Recognit Letter, 28: 1438 1444, 2007.
Kai Han, Yunhe Wang, Chao Zhang, Chao Li, and Chao Xu. Autoencoder inspired unsupervised feature selection. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2018.
John A. Hartigan and Manchek A. Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the Royal Statistical Society, 28: 100-108, 1979.
Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. In Advances in Neural Information Processing Systems, 2005.
Zi- Zi-Quan Hong and Jing- Yu Yang. Optimal discriminant plane for small number of samples and design method of classifier on the plane. Pattern Recognition, 24:317-324, 1991.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International conference on machine learning, pp. 1558-1551 1567. PMLR, 2017.
George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on Scientific Computing. 20:359-399 -392, 1998.
YongSeog Kim, W. Nick Street, and Filippo Menczer. Evolutionary model selection in unsupervised learning. Intelligent Data Analysis, 6:531-556, 2002.
Young Bun Kim and Jean Gao. Unsupervised gene selection for high dimensional data. In IEEE Symposium Biolnformatics and BioEngineering, 2006.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks arXiv preprint 1609. 02907,, 2016.
Martin H.C. Law, Mario A.T. Figueiredo, and Anil K. Jain. Simultaneous feature selection and clustering using mixture models. IEEF Transactions Patlern Analysis and Machine Intelligence, 26:1 154-1166, 2004.
Jundong Li, Xia Hu, Liang Wu, and Huan Liu. Robust unsupervised feature selection networked data. In SIAM International Conference on Data Mining, 2016a.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys, 50:94, 2018a.
Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In Computer Vision and Pattern Recognition, 2018b.
Yifeng Li, Chih-Ya Yu Chen, and Wyeth W. Wasserman. Deep feature selection: theory and application to identify enhancers and promoters. Journal of Computational Biology, 23:322-336, 2016b.
Zechao Li, Yi Yang, Jing Liu, Xiaofang Zhou, and Hanqing Lu. Unsupervised feature selection using nonnegative spectral analysis. AAAI Conference on Artificial Intelligence, 2012.
Hongfu Liu, Ming Shao, and Yun Fu. Consensus guided unsupervised feature selection. In AAAI Conference Artificial Intelligence, 2016.
11

---

Under review as a conference paper at ICLR 2022
Qiao Maoying, Yu Jun, Liu Tongliang, Wang Xinchao, and Tao Dacheng. Diversified bayesian nonnegative matrix factorization. In AAAI Conference on Artificial Intelligence, volume 34, pp. 5420-5427, 2020.
Ali Mirzaei, Vahid Pourahmadi, Mehran Soltani, and Hamid Sheikhzadeh. Deep feature selection using a teacher-student network. Neurocomputingg 383:396-408, 2020.
Sameer A. Nene, Shree K. Nayar, and Hiroshi Murase. Columbia object image library: Coil-20. Technical report, Columbia University, 1996.
Sarah Nogueira, Konstantinos Sechidis, and Gavin Brown. On the stability of feature selection algorithms. J. Mach. Learn. Res., 18(1):6345-6398 2017.
Marko Sabnik-Sikknnj and Igor Kononenko. Theoretical and empirical analysis of relieff and rrelieff. Machine Learning, 53:23-69, 2003.
Giorgio Roffo, Umberto Castellani, Alessandro Vinciarelli, Marco Cristani, et al. Infinite feature selection: a graph-based feature filtering approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Douglas T. Ross, Uwe Scherf, Michael B. Eisen, Charles M. Perou, Christian Rees, Paul Spellman, et al. Systematic variation in gene expression patterns in human cancer cell lines. Nature Genetics, 24:227-235, 2000.
Ryan Rossi and Nesreen Ahmed. The network data repository with interactive graph analytics and visualization. In AAAI Conference on Artificial Intelligence, 2015.
Saúl Calorio-Fernández.. J. Ariel Carrasco-Ochoa and José Fco Martínez-Triniddd. A new hybrid filter-wrapper feature selection method for clustering based on ranking. Neurocomputing, 214: 866-880, 2016.
Aboozar Taherkhani, Georgina Cosma, and T. Martin McGinnity. Deep-f': A feature selection algorithm for deep boltzmann machines. Neurocomputting. 322:22-37, 2018.
Chris S. Wallace and David L. Dowe. Mml clustering of mtlti-state, poisson, von mises circular and gaussian distributions. Statistics and Computing, 10:73-83, 2000.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2:37-52, 1987.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-4888888 487. PMLR, 2016.
Erkun Yang, Tongliang Liu, Cheng Deng, Wei Liu, and Dacheng Tao. Distillhash: Unsupervised deep hashing by distilling data pairs. Computer Vision and Pattern Recognition, pp. 2946-2955, 2019.
Yi Yang, Heng Tao Shen, Zhigang Ma, 7i Huang, and Xiaofang Zhou. (2, 1-norm regularized discriminative feature selection for unsupervised learning. In International Joint Conferences on Artificial Intelligence, 2011.
Zhihong Zhang, Yiyang Tian, Lu Bai, Jianbing Xiahou, and Edwin Hancock. High-order covariate interacted lasso for feature selection. Pattern Recognition Letters, 87:139-146, 2017.
Lei Zhao, Qinghua Hu, and Wenwu Wang. Heterogeneous feature selection with multi-modal deep neural networks and sparse group lasso. IEEE Transactions on Multimedia, 771196--148, 2015.
Liang Zhao, Amir Alipour-Fanid, Martin Slawski, and Kai Zeng. Prediction-time efficient classifica- tion using feature computational dependencies. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018.
Zheng Zhao and Huan Liu. Spectral feature selection for supervised and unsupervised learning. In International Conference Machine learning, 2007.
12

---

Under review as a conference paper at ICLR 2022
Zheng Zhao, Lei Wang, Huan Liu, and Jieping Ye. On similarity preserving feature selection. IEEE Transactions on Knowledge and Data Engineering. 016619632,, 2011.
Wei Zheng, Chunyan Xu, Jian Yang, Junbin Gao, and Fa Zhu. Low-rank structure preserving for unsupervised feature selection. Neurocomputing, 3 314:360-370, 2018.
Wei Zheng, Hui Yan, and Jian Yang. Robust unsupervised feature selection by nonnegative sparse subspace learning. Neurocomputingg 15:4116-17, 2019.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. arXiv preprint arXiv:2012.11854, 2020.
13

---

Under review as a conference paper at ICLR 2022
A ALGORITHM OF SOFT
Algorithm Second-Ordes unsupervised Feature selection via knowledge contrastive dis Tillation
Input: Input Data Matrix X; Feature Matrix Mr
X x X;
Output: & selected features;
1: Initialize 6 {fas, Gg |c);
2: repeat
3: Generate Attention Matrix MA and Masked Matrix Mas by applying Learnable Mask or
off Mp:
4: Use GCN process with MA, MM and MM generate Attention Representation GA. Original Representation GF and Masked Representation G
5: Adopt clustering G to get pseudo labels PA:
6: Employ MLP on Original Representation Gr and Masked Representation Gar to get predic- tions PF and Par:
7: Use PA as positive guidance for PF, and negative guidance for PM to train the objective
function;
8: until the model is converged.
9: Build graph based on MA via removing noisy features;
10: Apply graph segmentation the graph get partitions and select feature from each partition.
Algorithm I describes the whole process of SOFT. Through learning Attention Matrix in the first stage of our method, we incorporate both first-order data and second-order data and get refined feature relation matrix that can better reflect feature relations. Then in the second stage, we build graph based on the learned Attention Matrix and adopt the graph segmentation method, which groups high-correlated features together.
B DATASET STATISTICS
We select 12 public feature selection benchmark datasets different types for evaluation:
COIL20 (Nene et al., 1996), Colon (Alon et al.) 1999), Gisefte (Guyon et al. 2004), Lung-Cancel (L. Cancer) (Hong& Yangl 1991. Madelon (Guyon et al. 2007). MovementLibras (M.raasas) (Dias ct al. 2009). NC19 (Ross et al., 2000), ORL (Cai et al. 2006). Sonar (Rossi & Ahmed 2015). UAVI and UAV2 (Zhao et al. 2018). and Waveform (Wave.) (Breiman et al.) 1984). Tabel shows the statisties of these datasets, where the sample/feature ratios range from 0.01 more than 350, indicating the diversity of the datasets.
Table 3: Statistics of datasets
Datasct
Type
#Sample
#Fcature
#Cluster
Ratio
Deasity
COIL.0
Face Image
1440
1024
20
1.41
0.656
Colon
Biological
62
2000
2
0.03
0.584
Gisette
Handwritten
13500
L. Cancer
Biological
32
56
2
0.57
0.940
Madelon
Artificial
Mov Libras
Gesture
360
90
15
4,00
1,000
NCI
Biological
9712
9
0.01
0.503
ORT.
Face Image
400
1024
0.39
1.000
Sonar
Sonar Signal
208
60
2
3.47
0.999
Trailic
358.5
DAV2
Traffic
17256
54
2
319.56
0.983
Wave
Artificial
0.997
C RE-IMPLEMENTATION
Codes and datasets of work can be found http::/ghhhaaacmmm////2222//ssss
14

---

Under review as a conference paper at ICLR 2022
D
TIME COST
We record the running time for each method on 10% selected features, which is detailed in Table 4 When the number of features is much larger than the number of instances, our method runs relatively slow but faster than methods employing the matrix inverse calculation like UDFS. But when there are much more samples than features, our method faster than others except for LapScore, NSSLFS, and InIFS. This is because SOFT focuses relations between features, where the number of features contributes more to the time cost than the number of instances. general, the speed of SOFT acceptable even for large-scale datasets.
Table 4: Time cost of experiments in seconds
Dutaset
LapScore
SPBC
MCFS
UDFS
NDFS
LRPES
NSSLFS
TSFS
CAB
InfFS
SOFT
COI120
11.71
23.98
41.81
27.92
18.35
78.21
126.69
62.83
102.55
8.35
180.18
Colon
6,88
2.19
Gisette
$2.65
1070,95
101.09
8021.12
2011.81
NIA
NIA
687.24
1599.15
529.53
15129.18
L.Cancer
0.57
0.87
0.69
0.85
0.63
0.03
0.17
2.79
138.78
0.10
1.86
Madelon
M.Libras
3.61
4.38
4.28
4.23
3.29
1,82
1.05
16.19
256.30
0.23
3,80
NCW
3.61
3,46
105 29
3664.82
527.26
41.98
1833.71
11:42
333.74
42.55
1034,50
Sanar
0.98
1.25
1.15
1.11
1.02
0.56
0.39
7.93
414.20
0.05
2.17
DAVE
51.38
871.47
5176.43
27726.59
13020.19
NIA
11.65
508.86
605.86
57,40
3770.58
17091.52
Wove
7.07
31.46
12% 46
315.26
657.80
2168.22
8.28
136.66
748.83
1.10
15:26
Note: All experiments ran physical machine with Ubuntu 18.04, total memory of 64GB, AMD Ryzen Threadripper 29 29X0X 12. Processor,
NVIDIA GP102 GPU..
1.0
COIL20
I.Cancer
0.8
Stability
0.6
0.4
0.2
0.0
LapScore SPEC MCFS UDES NDFS LRPFS NSSLPS TSFS
CAE
InfFS SOFT
Figure 6: Stability of different methods with 10% selected features COIL20 and L. Cancer.
E STABILITY TEST
The "stability" of feature selection algorithm refers to the robustness its feature preferences, with respect to data sampling and to its stochastic nature. An algorithm is "unstable" if small change in data leads to large changes in the chosen feature subset. typical approach to measure stability is to first take M bootstrap samples of the provided data set, apply feature selection to each one of them, and then measure the variability the A/ feature sets obtained. Here we conduct the stability tests on COIL20 and L. Cancer. Specifically, we generate M = 50 bootstrap folds and run different unsupervised selection methods on these bootstrap folds. And then, we use the stability measurement proposed by Nogueira et al. 2017) ranging from -1 to 1, which returns the stability score with confidence intervals. The large value means more stable. The collection of the M feature sets therefore be modeled features. A row in Z represents a feature set, and a column represents the selection of given feature over the M repeats as follows:
binary matrix Z of size M d, where d is the dimension of
21,1
Z1,2
21,d
Ø2,1
2g,2
22,d
Z
2M.1
2M2
2M.
15

---

Under review as a conference paper at ICLR 2022
Based on 2,
Nogueira et al.
2017
define the stability score as follows:
Σ/=1 s
((2) 1
with
k(d- k)
(10)
M
d
M
M 2
8f = FM 1M-1111111 (1(1-111) 1z), P3= = M
2.
M
1=1==1
where & is the average number of features selected over the M feature set, sÿ is the unbiased sample variance of the selection of the fth feature, and here Py denotes the mean of the fth column of 2.
Figure G shows the stability test of different feature selection methods on COIL20 and L. Cancer. Our SOFT algorithm performs very stable with high scores and small intervals and excels others by large margin in terms of stability scores.
F DIFFERENT WAYS TO GENERATE PSEUDO LABELS
SOFT is flexible with different pseudo label generation methods. So far, we use PCA + K-means to generate pseudo labels that are further used guide feature selection. Here we test the performance of SOFT with other ways to generate pseudo labels in Table 5 Specifically, two popular deep clustering methods DEC Xie et 2016 and IMSAT (Hu et al. 2017 with their default parameters are used here. We can see some imprvvments by the deep clustering methods. For example, SOFT with DEC and IMSAT outperforms our default setting (PCA + K-means) on COIL20 and NC19, which demonstrates the power of deep methods. But in general, the default setting is slightly better than the deep methods. We conjecture that some dedicated network architecture design is needed based on the data property to pursue better performance. If we compare the results in Table 5 and Table 2 SOFT with DEC and IMSAT to generate pseudo label still excels other baselines by a large margin on the average level. Moreover, we also report the execution time of SOFT with different ways generate pseudo labels. Previously, SOFT might leave an impression of high computational cost just because of the long execution time Gisette dataset. Actually, the majority of the running time is occupied by PCA. The speed of SOFT on Gisette dataset can be dramatically accelerated from 15129.18 18 to 739.55 1068.37 by second with DEC or IMSAT,
Table 5: Performance and execution time of SOFT with different ways to generate pseudo label
10% selected features hy accuracy
Execution time by second
Dataset
PCA+K
DEC
IMSAT
PCA+K
DEC
IMSAT
COIL20
0.67=#0.11
0.75-40000
0.73-1-0.03
180.18
22.23
36,83
Colon
0.56=0.00
0,55++000
0.55+:0.00
46.38
40.75
55.17
Gisette
0.8120.00
0.7140000
0.761:0.00
15129.18
739.55
1068.37
L. Cancer
0.78=0001
0,721:000
0.691:0002
1.86
2.75
6.19
Madelon
0.60:10.00
0.56:1:000
0.561:0.00
75.84
18.25
35.79
M.Librur
0.46=00.0
0.46-4-0.02
0.49::0.02
3,80
4.17
8.03
NC19
0.44
0.50+::04
0.501:0.4
1034.50
6186.08
4550.16 16
URL
0
0.63-4-0.0
0.58+-0.02
Sonar
0.64=0.00
0.52+:0000
0.53-:0000
2.17
3.42
6.61
UAV7
0.78=0.01
0.67:1:000
0.661:0000
57.40
94.37
166.51
UAV2
0.82:0000
0.83::0.00
0.811:0.00
51.52
84.69
149,89
Ware.
0.54:50.00
0,55++-0.00
0.51
15.26
25.21
44.79
Average
0.64
0.62
0.61
1387,80
603.04
512.68
G DIFFERENT GRAPH SEGMENTATION METHODS
Graph segmentation on the attention matrix is a crucial stage of SOFT. Here we change the graph segmentation algorithms with multilevel recursive bisection on different percents of selected features. Figure 7 shows our SOFT performance and standard deviations features without and with multilevel recursive bisection. We can see that more smoothness over different percents is brought by multilevel recursive bisection on L. Cancer and UAV2, compared with the default one. The standard deviations on different percents of selected features are narrowed by multilevel recursive bisection well.
different percents of selected
16

---

Under review as a conference paper at ICLR 2022
0.90
0.30
Defrolt
0.08
18Sec
Accrecyy
0.75
0,06
L.nnner Delant
0.04
0.60
L.Canerr
UAV2 Defiolt
UAV2 BiSec
0.4
10%
15%
207s
25%
30%
co1t.0 Colon
Goets
Comoor
Madelon
since
Car
eav2
Wert
Selected Ecatunas
(a) Accuracy
(b) Standard deviation
Figure 7: SOFT performance and its standard deviation different percents of selected features without and with multilevel recursive bisection. Default means without multilevel recursive bisection, and BiSec means with multilevel recursive bisection.
H ABLATION STUDY ON NOISE REMOVAL FROM THE ATTENTION MATRIX
In the graph construction of SOFT, we have two ways to remove noises, including setting the values that are smaller than the median be zero and deleting 10% features according to their importance. More details can be referred to in the paragraph of Graph Construction. Here we provide the ablation Our default setting brings improvements on 7 out of 12 datasets compared with the single noise removal way and non-noise removal. On the average level, our default setting is also slightly better than others.
study on noise removal from the attention matrix in Table f
Table 6: Ablation study on noise removal from the attention matrix
Dataset
Both (Our defaulo)
Remove Values < Median
Delete 10%
No Noise Removal
COI120
0.67=0.01
0.71
0.73:20.03
0.74=0.03
Colon
0.56:00.0
0.55
0.56:20.01
0.55=0000
0.7240000
0.74::0.00
0.73=0.00
1. Cannee
0.78:00.01
0.72:1:004
0.69::0000
0.72=0.04
Madelon
0.60:20.00
0.58:10.00
0.58:10.00
0.58-0.00
M.Librax
0.46-600000
0.49
0.46-10001
0.49=0.02
NCI9
0.44=00.5
0
0.48:20.04
0.52
ORT.
0.58g0001
0.61
0.58:
0.64=0003
Sonar
0.64:20.00
0.53:10.00
0.50:10.00
0.52=0.00
0.78-0001
0.78:0000
0.78:10.00
0.78=0.00
UAV2
0.82::000
0.81
0.81:10.00
0.81=0.00
Wave
0.54::0.00
0.50
0.50:100000
0.50=0.00
Average
0.64
0.62
0.62
0.63
I ACKNOWLEDGEMENT
We would like thank the anonymous reviewers for their suggestions and interactions during the review period.
17


Reviews:
[{"review_id": "g23WTmOUjyr", "reviewer": "Reviewer_wcmh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think it is an interesting approach to unsupervised feature selection. Although the computational cost, both spatial and temporal, is very high (a correlation matrix is used), the idea of searching for strong feature correlations is interesting. However, I have some concerns regarding the contribution:\n\n1.- Some of the decisions are not clearly stated. For instance: In the graph construction, why the authors delete the las 10% features? Why they set to zero all values smaller than the median? In my opinion, the authors should include an ablation study that would shed some light over all those decisions.\n\n2.- The same idea should be applied to the pseudo label generation and the evaluation metrics. I expect to see how this algorithm behaves when using a more complex approach than kmeans, like DEC [1] or IMSAT [2], for instance.\n\n3.- Regarding to the problem of the accuracy drop whenever they increase the selected features (20% or higher), I wonder if there could be a different graph segmentation algorithm that could prevent it.\n\n\n[1] Xie, J., Girshick, R., & Farhadi, A. (2016, June). Unsupervised deep embedding for clustering analysis. In International conference on machine learning (pp. 478-487). PMLR.\n\n[2] Hu, W., Miyato, T., Tokui, S., Matsumoto, E., & Sugiyama, M. (2017, July). Learning discrete representations via information maximizing self-augmented training. In International conference on machine learning (pp. 1558-1567). PMLR.", "summary_of_the_paper": "The authors propose a two-step unsupervised feature selection algorithm. Contraru to classic FS approaches, the authors try to infer important relations between pairs of features. In the final step, a graph segmentation approach is used to select the most important features. Experimental results show promising results.", "summary_of_the_review": "Overall, I think it is an interesting idea with promising results, but more experiments have to be done to clearly state the performance of the algorithm, as well as a clear reasoning behing all the decisions the authors made in it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y8jctUOMXcA", "reviewer": "Reviewer_xFNs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The motivation is well introduced with illustrative examples in Figure 1.\n2. The graph segmentation-based framework is novel and interesting to me, which is different from the mainstream weight-based feature selection.\n3. The philosophy of knowledge contrastive distillation is logically sound. I have a minor question that will be posted in the Cons section.\n4. The experimental results are extensive. The authors compared with 10 methods including several recent deep methods on 12 datasets and demonstrated the significant improvements. \n5. The in-depth exploration is a plus, which helps understand the proposed algorithm.\nCons:\n1. The authors employed the knowledge contrastive distillation to learn the feature correlation matrix. The traditional unsupervised feature selection methods based on the first-order feature matrix usually employs a clustering method to purse pseudo labels for feature selection. It is suggested that the authors would like to do another ablation study without knowledge contrastive distillation.\n2. The motivation of this paper is to address the redundancy issue of selected features. Although the authors provided the illustrative examples in Figure 1, it would be better to demonstrate the proposed method does not suffer from this issue on the same datasets in the experimental part. \n", "summary_of_the_paper": "This paper considers the well-defined feature selection problem. The authors pointed out that the redundancy issue of the weight-based feature selection methods. To tackle this, the authors explored the second-order feature covariance matrix and proposed a two-stage framework including feature correlation matrix via knowledge contrastive distillation and feature selection on the masked correlation matrix via graph segmentation. Extensive experiments demonstrated the effectiveness of the proposed methods. ", "summary_of_the_review": "This paper considers the well-defined feature selection problem. The authors pointed out that the redundancy issue of the weight-based feature selection methods. To tackle this, the authors explored the second-order feature covariance matrix and proposed a two-stage framework including feature correlation matrix via knowledge contrastive distillation and feature selection on the masked correlation matrix via graph segmentation. Extensive experiments demonstrated the effectiveness of the proposed methods.  The motivation is well introduced with illustrative examples. The graph segmentation-based framework is novel and interesting to me, which is different from the mainstream weight-based feature selection. The philosophy of knowledge contrastive distillation is logically sound.\n It is suggested that the authors would like to do another ablation study without knowledge contrastive distillation. Although the authors provided the illustrative examples in Figure 1, it would be better to demonstrate the proposed method does not suffer from this issue on the same datasets in the experimental part. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PEQmjiB7FXc", "reviewer": "Reviewer_ELaK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n1. The proposed method is interesting;\n2. Experimental results are good;\nweaknesses\n1. Some presentations are not clear enough.\n1.1 The first line of page 4, why $\\Theta_M$ is forced to be symmetric?\n1.2 Why the masked matrix can be defined by Eq.(2)?\n2. Since the GCN training process relies on pseudo labels, how to ensure the reliability of the clutering results obtained by PCA and K-means?", "summary_of_the_paper": "This paper proposes a two-stage second-order unsupervised feature selection via knowledge contrastive distillation model that incorporates the second-order covariance matrix with the first-order data matrix for unsupervised feature selection. A sparse attention matrix that\ncan represent second-order relations between features is learned in the first stage, a relational graph based on the learned attention matrix is learned to perform graph segmentation for feature selection.", "summary_of_the_review": "Interesting idea, but some unclear presentations and motivations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8JPCf2sYXv", "reviewer": "Reviewer_kEpS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For unsupervised feature selection, this paper proposes a two-stage second-order method. Knowledge contrastive distillation is also incorporated for feature learning. Experiments on various datasets validate the effectiveness.\n\nPros:\n\n(1) The overall presentation and organization are good.\n\n(2) The presented results are good on these related datasets.\n\nCons:\n\n(1) My main concern lies in the experiments. NDFS achieves the best results among the compared baselines. However, NDFS is published in AAAI 2012, which is 9 years ago. The improvement of the proposed method over NDFS is also marginal on several datasets. Besides, the difference among these baselines is also marginal. Though the authors also compare with two recent methods, including CAE and InfFS, their performance is even lower than NDFS since they adopted different datasets in their original papers.  Therefore, the experimental results are not convincing. \n\n(2) The novelty is not satisfying. All these modules, including the second-order information, have been well studied in the area of unsupervised learning. The proposed method is a combination of these methods. The perspective for unsupervised feature selection by feature relationship learning and graph segmentation is not new from my point of view.\n\n(3) For the clustering task, many methods have already achieved much better results on COIL20, ORL, and even larger datasets, including CIFAR10 and CIFAR100.\n\n(4) The computational complexity is very expensive, which is much higher than many compared methods.\n\n(5) This paper is rejected by NeurIPS 2021, and I happened to review this paper several months ago. The authors do not incorporate any suggestions in this resubmitted manuscript. The main body is totally the same, and only one dataset for evaluation is changed.\n\n", "summary_of_the_paper": "For unsupervised feature selection, this paper proposes a two-stage second-order method. Knowledge contrastive distillation is also incorporated for feature learning. Experiments on various datasets validate the effectiveness.\n\n", "summary_of_the_review": "This paper proposes a rather complex and expensive new method for unsupervised feature selection that performs marginally better than the baselines (many quite old) on the set of datasets they have chosen to present in the paper. The exposition could also be clearer, but the idea seems more complex than it is interesting. In all, the paper does not rise to the standards of significance and novelty of ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6iAc39YEB2f", "reviewer": "Reviewer_3WQu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\nThis paper proposed an interesting feature selection algorithm, which leveraged the information from the second-order covariance matrix with the first-order data matrix. On empirical experiments, the proposed method showed superior performance over several baselines, to some extend.\n\n# Weaknesses\n\nHere are some of my comments/questions. I would appreciate it if the author(s) could give a response. If I am wrong, please correct me. Thanks.\n\n- The author(s) stated that \"Most of these methods apply the linear feature selection matrices and select the representative features by ranking their feature weight vector. Such operations treat the feature set independently and fail to tackle the complex high-order relationship...\". While, as far as I know, the model in [3] actually can globally exploit the complex feature relationships, and the models in [1] and [3] are both flexible to nonlinear feature selection. So what are the limitations of existing methods that the paper was trying to address?\n\n- The authors demonstrated the effectiveness of the proposed method through empirical experiments. Currently, there are many new developments in feature selection, for example, [1] and [2]. I noticed the author(s) cited [1], but why not compare SOFT to more non-linear feature selection models such as AEFS? If so, the effectiveness of the proposed method will be relatively convincing.\n\n- Comparing Table 2 with Figure 3, it is observed that the performance of SOFT is not so stable as other methods, not only on L.Cancer as the author(s) pointed out. So I encourage the author(s) might compare the stability of selected features from SOFT with baseline comparisons (for example, do stability analysis for selected features from different methods) before this paper is published in influential conferences or journals.\n\n- From Table 4 in supplementary material,  it seems that SOFT has no obvious advantage over other baselines in terms of efficiency, doesn't it? If so, efficiency has no advantage and performance is not too stable, then what is the purpose of SOFT? could the author(s) clarify or explain the motivation more? I look forward to hearing that. Thanks.\n\n- Additionally, this paper is generally well written, but some places (some issues) in this paper should be further clarified/fixed.\n\n   - The font for the legend in Figure 3 is too small;\n\n   - The same full name and abbreviation appeared several times in the paper;\n\n   - About the architecture of used baselines, the author(s) stated that \"For TSFS, CAE, and InfFS, we use default settings provided in their open-source codes.\"; however, it is not so clear. For example, for CAE, both linear and nonlinear structures are discussed in [3]. Which one did the author(s) of this paper use?\n\n\n\n\n\n\n\n   - [1] Han et al., Autoencoder inspired unsupervised feature selection. In International Conference on Acoustics, Speech and Signal Processing, 2018.\n\n   - [2] Doquet et al., Agnostic feature selection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2019. \n\n\n   - [3] Muhammed Fatih Balın, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference on machine learning, 2019.", "summary_of_the_paper": "Feature selection reduces high-dimensional data by identifying comprehensible informative features. This paper proposed an unsupervised feature selection method, SOFT, by combining the information from the second-order covariance matrix with the first-order data matrix. By empirical experiments, the author(s) demonstrated the effectiveness of the proposed method.", "summary_of_the_review": "In general, I think this paper is relatively interesting, and I was initially interested by the title of this paper. I would like to increase my score if the author(s) could give convincing responses to the previous comments/questions in the Weaknesses part. Thanks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new two stage second-order unsupervised feature selection method via knowledge contrastive distillation. In the first stage, a sparse attention matrix that represents second order statistics is learned. In the second stage, a relational graph based on the learned attention matrix is constructed to perform graph segmentation for feature selection. 

This proposed method contains some new and interesting ideas and is novel in the unsupervised feature selection setting, though some components such as the second order affinity matrix are not totally new. The proposed method is technically sound. The authors compared their method with 10 methods including several recent deep methods on 12 datasets and demonstrated consistent improvements.
 
However, there are some concerns from the reviewers, even after the discussion phase. 1) The computational efficiency of the proposed method seems to be low. Since one goal of feature selection is to speed up downstream tasks, the efficiency of feature selection itself should also be considered. I suggest the authors analyze the computational bottleneck of the proposed method and improve the efficiency. 2) More ablation studies can be added to illustrate how the proposed method removes the redundancy issues of the selected features. 3) Some metrics like supervised classification accuracy can be potentially used as a metric. Though supervised classification is impossible in the unsupervised learning setting, running the experiments on some datasets that have labels by pretending having on label is one way to evaluate the method.

Overall, the paper provides some new and interesting ideas. However, given the above concerns, the novelty and significance of the paper will degenerate. Although we think the paper is not ready for ICLR in this round, we believe that the paper would be a strong one if the concerns can be well addressed.
Decision: Reject