Where to Pay Attention in Sparse Training for Feature Selection?
Authors: Ghada Sokar
Zahra Atashgahi
Mykola Pechenizkiy
Decebal Constantin Mocanu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Feature Selection
Sparse Training
Dynamic Sparsity
High dimentional data
Big data

Abstract: A new line of research for feature selection based on neural networks has recently emerged. Despite its superiority to classical methods, it requires many training iterations to converge and detect the informative features. For datasets with a large number of samples or a very high dimensional feature space, the computational time becomes prohibitively long. In this paper, we present a new efficient unsupervised method for feature selection based on sparse autoencoders. In particular, we propose a new sparse training algorithm that optimizes a model's sparse topology during training to quickly pay attention to informative features. The attention-based adaptation of the sparse topology enables fast detection of informative features after a few training iterations. We performed extensive experiments on 10 datasets of different types, including image, speech, text, artificial, and biological. They cover a wide range of characteristics, such as low and high-dimensional feature spaces, as well as few and large training samples. Our proposed approach outperforms the state-of-the-art methods in terms of the selection of informative features while reducing training iterations and computational costs substantially. Moreover, the experiments show the robustness of our method in extremely noisy environments.

Where to Pay Attention in Sparse Training for Feature Selection?
Ghada Sokar
Zahra Atashgahi
Eindhoven University of Technology g a. a.a.2.n. s...artuo nl
University of Twente 2. at atashgahi Sutwente .nl
Mykola Pechenizkiy Eindhoven University of Technology m. weceenizkkkie nl
Decebal Constantin Mocanu
University of Twente Eindhoven University of Technology
d. wwc.mmamuutteeee.e....mmmm .nl
Abstract
A new line of research for feature selection based neural networks has recently emerged. Despite its superiority to classical methods, it requires many training iterations converge and detect informative features. The computational time becomes prohibitively long for datasets with a large number of samples or a very high dimensional feature space. In this paper, present new efficient unsuper- vised method for feature selection based on sparse autoencoders. In particular, we propose new sparse training algorithm that optimizes model's sparse topology during training to pay attention to informative features quickly. The attention adaptation of the sparse topology enables fast detection of informative features after a few training iterations. We performed extensive experiments on 10 datasets different types, including image, speech, text, artificial, and biological. They cover a wide range of characteristics, such as low and high-dimensional feature spaces, and few and large training samples. Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while re- ducing training iterations and computational costs substantially. Moreover, the experiments show the robustness of our method in extremely noisy environments
Introduction
Feature selection plays a crucial role in data mining and machine learning tasks with the explosion in the size and dimensionality of real-world data 68 30 44 23 aims efficiently select a subset of the features that are most informative and remove the irrelevant or redundant ones. It is useful for alleviating the curse of dimensionality, interpretability of model-driven decisions, generalization of downstream tasks, reducing the prohibitive memory and computational costs, and avoiding the expensive costs of collecting the full set of features as in biological studies [1]
52
Many methods have been proposed for feature selection in supervised, semi-supervised, and un- supervised settings (75) 10 60
51 19 I. Recently, using neural networks for feature selection has received increasing attention due to its power to learn non-linear dependencies among input features 39 One limitation of current neural network-based methods is that they are computationally expensive. The computational costs result from training dense models for many training iterations until informative features are recognized. The costs increase for datasets with very large number of training samples or high dimensional feature space. This limitation is recently addressed in [4] by utilizing sparse networks with dynamic sparsity for feature selection. An autoencoder is trained with
Coode is available at http:://gttuu..com/// http:///gthbb.coo///caaaa//as/s
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the sparse training algorithm SET [54], which explores different sparse topologies during training by dropping portion of the connections and growing the same portion randomly. The importance of an input feature is estimated after convergence based on its outgoing sparse connections. Training sparse networks from scratch decreases memory and computational costs compared to previous dense-based methods. Yet, the random exploration of sparse topologies requires many training iterations to identify informative features.
In this paper, we propose an efficient unsupervised method for feature selection. We introduce a new sparse training algorithm for autoencoders that quickly detects the informative input features. Specifically, we optimize the sparse topology to learn Where to pay Attention during Sparse Training We named our method WAST. WAST exploits the information from the reconstruction loss and the weights of sparse connections to guide the topological search. It redistributes the sparse connections to spot informative features quickly. We evaluate our proposed approach on ten datasets from various domains, including image, speech, text, artificial, and biological. Through extensive experiments, we find that WAST identifies the informative features after a few training iterations and outperforms state-of-the-art unsupervised feature selection methods. Our main contributions are:
We propose a new efficient unsupervised method for feature selection, named WAST. WAST optimizes the sparse topology of an autoencoder to detect the informative features quickly.
We perform extensive experiments on 10 benchmarks that cover various types and character- isties. Experimental results show the effectiveness of our method over the state-of-the-ar methods in terms of selecting the informative features.
WAST reduces the computational time compared to neural network- methods substan- tially by reducing training iterations and employing highly sparse neural networks.
We illustrate the robustness of our method in extremely noisy environments and its effective- ness for datasets with very high dimensional feature space and a few training samples.
2 Related Work
2.1 Feature Selection
Many feature selection methods were introduced for the supervised, semi-supervised, and unsuper- vised settings depending whether the data is labeled, partially labeled, unlabeled, respectively
75 10 60, 1. Typically, feature selection methods are divided into three categories: filter, wrapper, and embedded methods. Filter methods are independent of the learning algorithms. They use ranking techniques for providing scores to the features, such as Laplacian score [26]. Despite being fast, they do not consider the relationship between the features, which may result in the selection of redundant features [ In the supervised setting, filter methods consider the relation between the feature and the class label, such fisher score (22], CIEF [1], and ICAP [33]. Wrapper methods exploit the performance of predictive model to evaluate the quality of a subset of the features 770: 12, (74.. They are more effective than filter methods, yet they are computationally expensive (39]. Embedded methods incorporate feature selection into the learning phase of another algorithm. Feature Selection (MCFS) 99] uses regularization select the best features that keep the multi-cluster structure of the data. Unsupervised Discriminative Feature Selection (UDFS) [72) uses l2,1 regu- larization and discriminative analysis to select the most discriminative features. Similarly, in the supervised setting, RFS (56] and L1_L21 (5 exploit (2, nnorm to introduce feature sparsity.
ClusteCluster
Another direction under the embedded category has recently emerged. It uses neural networks to perform feature selection by learning the non-linear dependencies among input features [40). The success of autoencoders as tool for feature extraction encourages unsupervised methods to explore their power for feature selection 77 6 67 (5 [8]. AutoEncoder Feature Selector (AEFS)
25 combines autoencoder regression and group lasso tasks. Concrete Autoencoder (CAE) S learns concrete selector layer (encoder) that selects stochastic linear combinations of input features during training. After training, converges to the target number of features. Despite the high performance of these methods, the large number of iterations required to train dense models increases the computational costs significantly. The conceptually closest work to ours is the recent work, Quick Selection (QS) 4 It trains a sparse autoencoder from scratch. During training, different sparse topologies are explored using the SET method [54]. After each training epoch (i.e., pass on the full data), a fraction of the connections with the least magnitude is dropped, and the same fraction
2

---

Initialization
Iteration
Sparse Training
Iteration 1+1
After training
Topology update
Weight update
Weight
update
Topology
update
Drop
Drop Loast important att
Grow
OMest imporiant
Grow
Selected Features
Figure I: An overview of our proposed method WAST for unsupervised feature selection. sparse autoencoder is initialized with uniformly distributed sparse connections. During sparse training, the connections are redistributed in the most important neurons at iteration / during the "drop-and-grov" cycle. After convergence, neurons with the highest importance are selected as the most informative features.
randomly regrown. Few works based on multi-layer perceptron networks are proposed for the supervised setting (37 61 [7]. In section we denote non Neural Network-based (NN-based) methods classical methods.
2.2 Sparse Training
Sparse training with dynamic sparsity, also known as Dynamic Sparse Training (DST), is a recent research direction that aims accelerating the training of neural networks without sacrificing performance [27 54 47]. A neural network is initialized with a random sparse topology from scratch. The sparse topology (connectivity) and the weights are jointly optimized during training. During training. the sparse topology is changed periodically through drop-and trop-and-gron cycle, where a fraction of the parameters are dropped, and the same portion is regrown among different neurons. An update schedule determines the frequency of topology updates. Many works have been proposed, focusing on improving the performance of sparse training for supervised image classification tasks by introducing different criteria for connection regrowth [20 34 59) 49 73 50]. DST demonstrated its success in
other fields as well, such as continual learning 622]. feature selection 4 ensembling [46), federated
learning 77 g adversarial training (57]. and deep reinforcement learning [63]
3 Where to pay Attention in Sparse Training (WAST)?
Problem Formulation. Let D be dataset with samples {x0)) where x(i) R and and are the number of features and samples, respectively. The goal is to select a subset of the features of a size K in an unsupervised manner, which is most representative of the whole feature space m.
WAST is new sparse training method for neural seasor-bbased unsupervised feature selection. The basic idea of WAST is to pay attention to input features based their estimated importance during training to detect the most informative features quickly. An overview of WAST is shown in Figure We use sparse autoencoder with single hidden layer of f neurons. Let fw be an autoencoder model parameterized sparse weights W- =(W(1) w(2)). J. The network is initialized with certain |w
sparsity level = 1- mxc where I|-Ilp is the Lo norm. The sparsity level is kept fixed during training. Initially, the sparse weights are uniformly distributed on the neurons. During training, besides weights optimization, the sparse topology is optimized such that the sparse connections are gradually redistributed on the most informative features. To achieve this goal, we propose new criteria to update the sparse topology and new update schedule,
will explain next.
Training, An autoencoder model is trained minimize the reconstruction loss. We use mean squared етrоr to measure the loss between an input sample and the reconstructed one x as follows:
L
x x
where n = /x(x) = (x ((2)) and a is non-linear activation function. After each weight update using batch of the data, we adapt the sparse topology. The effect of the update schedule on performance is shown in Appendix The sparse topology is updated through drop-and-grow cycle.
3

---

Algorithm 1 WAST
1: Input: Autoencoder network fw with one hidden layer of size h, Dataset 'D with m features
2: Input: Learning rate v), target number of features K, sparsity level 8, X, a
3: Output: Indices of selected features from the m features (R)
4: Initialize W = (W(1), w with sparsity level
5: Iı 60 To. < Vi E [m]
6: for each training step t do
7: Sample a Batch B, ~ DD with size
x00) + (w(((()),
vx ()) E Br
8:
9:
L+ ;
((6)
x(1] ?
1
10:
W + W -//WL
11:
Io. Io. + Xl
Xx
ØL
+ (1 (1- (1-)) Ek= W
Vi (1,.., m} Vie {l...., m}
12:
II IT A
A1 (
WIll) Drop (W Io. a) 6 (1,2)
13:
W + Grow (Io.I,, a)) V (1.2)
15: end for
16: R k-Lop(II,
A fraction of the connections with the least importance is dropped from each layer (W(22) w(2)), J, and the same portion of new connections are regrown. New connections are initialized to zero.
The main contribution of this work lies in where the new connections are regrown. Different from the previous method (QS) where the connections are randomly regrown, we optimize the topology to pay fast attention to informative features during training. To this end, we measure the importance of each neuron the input and output layers by leveraging the information available during training. The importance of neuron is estimated by its impact on the reconstructed loss and the magnitude of its connected weights.
Specifically, we measure how sensitive the reconstruction loss L is to changes in the reconstructed output x. The first-order approximation for the change in loss with respect to a small perturbation ö = } in the output x = [ can be written as a sum of its individual components as follows:
m
aL
L(X + [(X)
8,.
0x
where ãx, is the gradient of the loss with respect to the output neuron x,, We define the importance of neuron in the output layer at training iteration as:
I00
(1-1) I0, -
aL + (1
2 H
where [W(2) is the magnitude of the incoming connection from neuron to neuron and X is a hyperparameter coefficient to balance the two components (See Appendix A.. The same criterion applied for the input neurons (Z(!), except that the magnitude of input weights ((W(2)) used instead of the output weights. This criterion considers two cases: the information from large gradients, especially at the beginning of the training, and the large weights resulting from the optimization at later stages. The new connections in the input and output layers are regrown on the neurons with the highest importance. For instance, the new connections in the input layer are given by top an (I, x IH, r), where top( Q, k) gives the indices of the top-k elements in a matrix Q, Ih
is the importance of the hidden neurons, (()) is the set of remaining weights after the drop phase, and the number of regrown connections based on the fraction a. We consider the neurons the hidden layer be equally important.

---

Table 1: The characteristics of datasets.
Type
Dataset
#FFatures
#Classes
#Samples
#Train
#Test
Artificial
Madelon 24
500
2600
2000
600
UPSP 32
9298
7438
1860
COIL 20
55
1024
20
1440
1152
288
Image
MNIST 36
784
10
70000
60000
10000
Fashion MNIST 69
784
10
70000
60000
10000
Speech
Isolet [21
617
26
7737
6237
1560
Time Series
HAR
516
10299
7352
2947
Text
PCMAC 35
3289
1943
1544
MM--CAN-187 66
19993
187
149
38
Biological
GLA-BRA-ISH
65
49151
180
144
36
During the drop phase, the importance of connection w in the input (output) layer is estimated by its magnitude and the importance of its connected input (output) neuron (I(()) as follows:
I w
where I takes the value of - and T.0 for the input and output layers, respectively. The effect of each component in the neuron and connection importance criteria is studied in Section [.. The periodic update of the sparse topology redistributes the connections on the effective features.
Feature Selection. After training, select the top- K neurons with the highest importance from the input layer as the most informative features. The details of WAST are provided in Algorithm [
Experiments
4.1 Baselines
We compared method with several autoencoder-based (QS 4. CAE (5. AEFS
25 and classical methods (lap_ score (26), MCFS 19, DUFS (42)) for unsupervised feature selection. On top of that, we include comparisons with seven ootte--f-hh-arrr supervised feature selection methods. Although the latter is not the focus of this paper and not a typical comparison in the literature, we prefer to show that WAST performs competitively also with the supervised methods while being computationally efficient and applicable for cases where labels are not available or expensive to collect.
(lap_score
4.2 Datasets
We evaluate method 10 publicly available datasets, including image, speech, text, time series, biological, and artificial data. They have a variety characteristics, such as low and high-dimensional features and small and large number of training samples. Details in Table I
4.3 Experimental Settings
Evaluation Metrics. An efficient feature selection method should have high learning accuracy with less memory and computational costs [10]. Hence, we evaluate multiple metrics. (1) Classification Accuracy: It typically evaluated by machine learning model [10. We trained classifier using the K selected features by the studied methods. Here, we use one of the most popular classifiers, support vector machines (SVM) [1]. Other classifiers could be used. Yet, we choose use non-NN-based classifier for evaluation to avoid any biased advantages towards the NN-based baselines over the others. We studied the performance of 6 different values of the number of selected features (K). We report the average accuracy of the test data over 5 seeds. (2) Memory cost: We calculate the number of network parameters (#params) used by each NN-based method. (3) Computational cost: We calculate the number of Floating-poii -point operations (FLOPs) consumed to train neural network. See Appendix A.2 for details.
Implementation. We implemented WAST and QS 4 with PoTorc 58
(see Appendix H
For CAE
and AEFS 25 we used the code provided by the authors of CAE with MIT license For DUFS
5

---

Table 2: Classification accuracy (%) (f) using unsupervised and supervised feature selection methods. 50 features are selected for all datasets except Madelon, where 20 features are used. The best performer from the unsupervised methods indicated in bold font, while the best performer from the supervised methods is in blue.
Method
Madelon
USPS
COIL-20
MNIST
MaSII
lap_score score
25
49.50-100000
70.54=0.00
78.12-+00.00
23.94:00.00
27.07-10.00
Classical
MCFS
9
51.83++0.00
93.33-0000
97.22:+0.00
DUFS
52.57+1.335
95.62=0.54
97.43:+1.22
62.091 0.00
74.691 86
Unsupervised
AEFS
60.16:4.61
95.86=0.488
99,481:0441
93.22 1.32
80,8810.71
NN-based
CAE
80.90++2.86
95.04-0.599
94.54-+2.972
96.20+0.14
84.66 0.16
4
82.07:+1.10
95.88=0.31
99.17410.22
94.07=0.044
82.65
WAST (ours)
83.27-40.3
9.269=0.27
99.58-10.14
95.27=00.26
82. 12::0775
Fisher Score
75.6710:00
91:02:0:00
88.89:10:00
86.11=0.00
67.8510.00
L2] 45
49.33:10.00
86.99=0.00
92.01+:0.00
62.26::0.00
69,57:40.00
Classical
CIHE 41
54.50++0.00
61.29-00.00
09.38+
89.30+0.00
66. 06.00
Supervised
ICAP 38
78.00:10.00
95.22=00.00
99.31110000
89.03
59.5241.000
RFS
83.00-:0.00
95.32=0000
66.32:10.00
NN-based
Nes
79.20-+1.22
95.80-0.12
9583411188
94. F0.12
826340.25
STG
19.53:+
9..78=0.60
97
92.53
83.3221.:5
Method
Isolet
HAR
PCMAC
SMK
GLA
lap_score score 26
7..71=0.00
82.80:10.00
49.87:10.00
81.58:0.00
66.67:10.00
Classical
MCFS
81.41=0.00
80.29:+0.00
53.47:00.00
78.95=00.00
75.00=+0.00
DUFS 42
85.62+ 2,53
86. 1,06
57.79+3.118
81.05+3.07
70.83+11.39
Unsupervised
AEFS
25
80.94=2.02
89.54:20.44
60,40:2..7
19.48:3.07
67.76:6.22
CAE
78.90+1.224
86.26+22.41
55.08++0.00
78.94+22.77
7..56++4.50
NN-based
4
74.62+22.12
+.338
3,.78+3.25
81.58+3.72
68.89 68.89++4.77
WAST (ours)
13.33=1..39
91.20:10.16
20.51+2.53
84.74#
75.56++4.08
Fisher_scure scure
28
75.64100.0
83.68110.00
0..38110.00
73.68:10.00
63.89 03.89.0000
Ll L21 45
55.90=0.000
81.30:1.000
53.98:10.00
84.21
69.4411.000
Classical
59.81110.00
84.15:10.00
0.00000000
81.58:10.00
58.33-10.00
Supervised
ICAP 33
75.0620.00
88.70110.00
87.66.10.00
73.6811..00
72.22-10.00
RFS 56
77.31
28.2000000
67.61+0000
76.32:10.00
LassoNel
37
85.70110.88
9393110.1
86.53.11.25
71.371 3.57
76.67:1222
NN-based
STG
89.38.1119
91.75:10.59
56.04:11.00
81.05:1.29
71.1112.83
42 LassoNet 337, and STG 71 we used the official codes with MIT license
ii
For classical NN-based and
baselines, used the Scikit-Feature library with GNU General Public license (39 classical methods are trained on Nvidia GPUs and CPUs, respectively. We consider a 12 hours limit on the running time of each experiment. Experiments that exceed this limit are considered.
Implementation Details. For all NN-based methods except CAE [5], we use a single hidden layer of 200 neurons. The architecture of CAE consists of two layers. The size of the hidden layers is dependent on the chosen K" [K, 3 K]. For WAST and QS, we use a sparsity level 0.8. Following [4. we report the accuracy of NÑ- -ased baselines after 100 epochs unless stated otherwise. Note that some baselines reported higher number of epochs (e.g.. CAE (S uses 200 epochs for some datasets). Yet, we keep 100 epochs for all cases for fair comparison. For WAST, train the model for 10 epochs. Following (4) we add a Gaussian noise with a factor of 0.2 to the input in WAST and QS Details of the hyperparameters in Appendix A.I
4.4 Results
Accuracy. Table 2 shows the classification accuracy of the studied datasets. Here, we report the challenging case where a few best informative features have to be selected. We use a K of 50 for all datasets except for Madelon, where K of 20 is used as the remaining 480 features are pure noise. Experiments with various values for K E {25, 50, 75, 100, 150, 200) can be found in Appendix with summary provided next.
Unsupervised NN-based methods outperform the classical ones on all datasets except one, where the performance difference is marginal. WAST outperforms unsupervised NN-based methods on
htmps//gittuh.cmm/ffiiiinnDD
https://ghtub..comm/sssoe//assss "httss//junddongl.ginn ittp:////...onn.ii...iiooio.iiie

---

Table Memory and computational costs (4) estimated by the #params and FLOPs (10*2), respec- tively, for NN- methods. Unlike other methods, the architecture of CAE [5] is dependent on the
value K. Here, we report the costs for K = 50.
TrashionJMNIS
USPS
COIL-20
Isolet
Method
Type
#params
FLOPs
#params
FLOPs
#params
FLOPs
#params
FLOPs
25FS
Dense
313600
11.28
102400
0.45
409600
0.28
246800
0.92
CAE(S
Dense
101750
3.66
35750
0.15
131750
0.09
80875
0.30
QS(4]
Sparse
62720
2.25
20480
0.09
81920
0.05
49360
0.18
WAST
Sparse
62720
0.22
20480
0.009
81920
0.005
49360
0.01
HAR
PCMAC
SMK
GLA
25
Dense
206400
0.91
1315600
1.22
7997200
0.71
19660400
1.69
CAE
Dense
68250
0.30
414875
0.38
2502875
0.22
6147625
0.53
QS
Sparse
41280
0.18
263120
0.24
1599440
0.14
3932080
0.33
WAST
Sparse
41280
0.01
263120
0.02
1599440
0.01
3932080
0.03
Score (Accuracy)
FLOPs (10
40
AEFS
Matetan
30
QS
CAE
AEFS
15
COIL
USPS
20
WAST (ours)
QS
MINST
WAST (ours)
10
FastionMNIST
20
25
50
75
100
150
200
Merocyy Computation Effciency
SMK
K
Effciency
ABFS
CAE
Q5 WAST (surs)
CLA
Figure 2: FLOPs (4) re-
Comparison us- ing various evaluation met- rics computed on all datasets and 6 different values for K.
Figure 3:
Figure 4: Score (1) (how many times method is the best performer) of differ- ent methods on each dataset for 6 differ- ent values of K.
quired for all datasets ex- cept Madelon different values of K.
all datasets except MNIST and Fashion MNIST, where CAE is the best performer. is worth emphasizing that the superior performance of WAST is achieved with a 90% reduction in the number of training iterations. Supervised baselines outperform the classical supervised ones in 6 cases. Interestingly, WAST achieves competitive performance the supervised baselines. It outperforms the best-supervised performer in 5 cases.
Memory and Computational Costs. Table 3 shows the memory and computational costs consumed by each unsupervised NN-based method. The high memory and computational costs are caused by the high number of training samples, in MNIST, or the high-dimensional feature space, as in GLA. Training sparse model from scratch, as WAST and QS, reduces these costs substantially. QS reduces the computational cost by 80%. Interestingly, besides the reduction resulting from using sparse neural networks, WAST reduces the computational cost further to 98% due to the fast identification of the informative features. Both WAST and QS reduce the network size by 80%.
The network size of CAE dependent on the target value of K. This has two limitations: training the model for every target K and the increase of the memory and computational costs with higher values for K. On the other hand, WAST derives the importance of each feature with single training run independently of the target K. This is illustrated in Figure 2 The figure shows the accumulated FLOPs for datasets except Madelon each value of the studied K.
Overall Performance. To give a holistic view of the performance of each method across all dimensions (accuracy, memory cost, and computational cost) on all datasets and studied values of K. we calculate the performance in each case. Figure 3 the normalized performance using min-max scaling for all cases. The "score" metric represents the number of times a method is the best performer in terms of accuracy. Details are provided in Appendix A.2 CAE has higher score than AEFS, while the former is the best only for image datasets with large number of samples (i.e., MNIST and Fashion MNIST), as shown in Figure AEFS has the highest memory and computational costs. QS improves the memory and computational costs but has lower score. WAST has the highest score and spans more dataset types with different churacteristics (e.g.. high dimensional data) than the baselines 4Fi The performance gain is accompanied by a significant improvement memory and computational efficiency. Detailed results provided in Appendix B
7

---

97
100
96-
80
Accuracy 1
96
98-
Accurasy (58)
94
Accuracy
60
95
96
92
90
40
(a) Madclon.
(b) USPS.
COIL-20.
(d) MNIST.
90
Accuracy 1%)
80-
Accursy 13
85
Accuracy
855
80
75
75
80
55
70
5
9
70
5
75
8
50
5
8
Epoch
Epoch
Epoch
Epoch
(c) MNSS.
(f) Isolct.
(g) HAR.
(h) PCMAC.
80
Accurscy (96)
75
80
70
AEFS
CAE
WAST (ours)
75
65
70
2
5
10
60
5
16666 7
Epoch
Lpoch
ii) SMK.
(i GLA.
Figure 5: The performance of different methods at the early stages of the training. The test accuracy reported for at first 10 training epochs using K of 50 for all datasets except Madelon, where of 20 is used.
5 Analysis
5.1 Effect of Fast Attention During Training
To analyze the impact of attention the informative features in WAST during training, study the learning behavior of all unsupervised NN-based methods on the test data for the first 10 epochs. Since CAE has parameter that is dependent the max number of epochs, separate runs and varied the max number of epochs each time. FigureB]showe the accuracy using K of 50 for all datasets except Madelon, where K of 20 used. The analysis reveals the following findings: (1) Robustness to noisy environments: The results on Madelon illustrate the robustness of WAST against the noisy features (480 out of 500). The figure shows the performance gap between WAST and the baseline methods starting from epoch 1. After 10 epochs, WAST outperforms the second-best performer by 22%. More experiments on noisy enviroments can be found in Appendix G (2) Performance on datasets with a large number of samples: WAST outperforms CAE by 3.6% and 1.6% on MNIST and FashionMNISTT, respectively, although CAE was the best performer on these datasets after training it for larger number of epochs (Table - (3) Consistency across different dataset types and characteristics: We observe that some baselines achieve high performance in some domains while being worse in others. WAST has the highest consistency on different dataset types and characteristics, outperforming all baselines after 10 epochs (Appendix(C) (4) Stability: WAST is more stable at the early stage of the training. This is represented by the standard deviation across different training runs (shaded region).
performed ten
5.2 Visualization
EngurreGilluseer how the sparse topology changes during training in WAST and QS. performed this analysis on MNIST, where handwritten digits are centered in 28 x 28 grayscale images (Figure 6aa. Initially, the sparse connections are uniformly distributed on the input neurons. With the guided adaptation of sparse topology via attention, WAST detects the informative features quickly after one

---

Initialization
54
200
160
42
120
120
16
80
80
24
30
24
(a) MNIST
(b) WAST (ours).
Initialization
Epach
Epoch
54
48
48
42
24 16
36
40
36
30
30
24
20
16
24
16
24
16 24
(c) QS
4
Figure 6: Visualization. (a) Samples of the MNIST dataset. Each sample is 28 x 28, and the informative pixels centered in the middle. The distribution of the sparse connections during training (i.e., the number of outgoing connections from each input neuron) on MNIST using WAST (b) and the QS method
(c). WAST has faster attention to the informative features after one epoch.
Table 4: Ablation study. Effect of each component the criteria of neuron importance and connection importance. The accuracy (%) is reported using K of 20 and 50 Madelon and others, respectively.
Criteria #
Method
Madelon
USPS
HAR
PCMAC
SMK
Neuron
"wlo 12
55.50:+2.45
95.91
87.88+1.79
55,58:42.78
84.21:3.16
Neuron
2
"w'o IW""
82.47.10.75
95.5611.118 18
92.20-10.50
58.66114.18
75.26:12.11
Neuron
3
"wlo momentum"
81 1,000
96.60:0.40
90.7410.94
58.30:1.880
78.95-3.33
Connection
4
"wo Z,"
83.27=0.63
96.5420.17
90.11:0.522
53.53:13.07
84.74:21.05
WAST (ours)
83.27-10.63
96.6910.27
91.20-10.20
60.5112.53
84.74.1..05
epoch. On the other hand, the random growth of the connections in QS requires more training epochs. The effect of increasing the frequency of topological exploration in QS is studied in Appendix F
5.3 Ablation Study
We performed an ablation study to assess the contribution of each component in the proposed criteria of neuron and connection importance in the performance of WAST.
Neuron Importance. (#1) "w/o aL I" Using only the magnitude of connected weights to determine 0), (#2) "wwo IWI: Using only the sensitivity of the neuron to
the importance of neuron (i.e., HK the change in the loss (i.e., À = 1), and (#3) "w/o momentum": The importance of a neuron is based only the current estimate of its sensitivity to the loss and currently connected weights.
Connection Importance. (#4) "w/o I,": The importance a connection estimated by its magnitude without considering the importance of its connected neuron I,-
We performed this analysis on datasets of different types. The results are summarized in Table We observe that removing the sensitivity of a neuron to the loss (#1) has the biggest effect on performance. has the highest impact on noisy environments, where it reduces the performance by 27.77% on Madelon. On the other datasets, it results in a 0.53 - 4.93% reduction in the performance. Excluding the magnitude of the weights (#2) affects mainly the datasets with few training samples (e.g., SMK). Neglecting the neuron importance in the previous training iterations (#3) reduces the performance by 0.09 5.79%. Aligned with many prior works 220 54 27 we find that the weight magnitude an effective metric for estimating the connection importance (#4). Moreover, our results show that enhancing with the importance of connected neuron improves the performance even further on some datascts without diminishing it on the others.

---

94
Accuracy [%)
84
Accuracy [%)
98
Accuracy [%)
96
82
90
80
92
20%
80% Sparsity level
40%
60%
90%
20%
40%
60%
80%
90%
20%
40%
60%
80%
90%
Sparsity level
Sparsity level
(a) Madelon.
(b) USPS.
(c) HAR.
Figure 7: The performance WAST using different sparsity levels for the autoencoder model. The test accuracy is reported using K of 50 except on Madelon, where K = 20.
5.4 Effect of the Sparsity Level
We further study the effect of the sparsity level of the auto-encoder model the performance of WAST. We evaluated 5 different sparsity levels (20%, 40%, 60%, 80%, 90% All other settings are the same as the ones stated in Appendix A
Figure iillustrates the classification accuracy using K = 20 for Madelon and K=50 for other datasets. We observe the performance of WAST is robust to the sparsity level. Yet, mostly care about the performance at high sparsity levels, as the goal is to achieve high performance at low computational cost.
Discussion
Conclusion. In this paper, we propose WAST, new efficient neural network-based method for unsupervised feature selection. We train a sparse autoencoder from scratch and optimize the sparse topology during training detect the informative features quickly. We performed extensive exper- iments in which we evaluated 55 cases selected features. WAST achieves the best performance on 19 cases, while the second- best unsuper- vised performer has a score of 11 cases. More interestingly, the superior performance is achieved with a few training iterations. WAST reduces the memory and computational costs by 80% and 98%, respectively. Moreover, we show the robustness of WAST towards very noisy environments, outperforming the state-of-ther oftthe--rt methods by 22% with limited training iterations. Finally, we show that WAST performs competitively with supervised-base datasets. This demonstrates the promise adapting WAST the supervised setting and motivates new sparse training algorithms for supervised and unsupervised feature selection.
various datasets and different values of the number of
methods, outperforming them image
Limitations. This work a step toward exploiting the power of neural networks for feature selection in computationally efficient manner. Besides the improvement in the accuracy of selecting the informative feature, fast attention the important features during training reduces the number of training iterations substantially. Yet, like most sparse training methods in the literature, this proof-of- concept does not fully utilize the memory and computational advantages sparse neural networks. This is due to the lack of hardware support for sparsity and the higher focus of the community on the algorithmic side [27]. Nevertheless, there is recent growing attention to hardware and software support for sparsity (88 76 66) 48 15 3 (See Appendix for discussion). This enables the pure sparse implementation of the method in the future.
Societal Impacts. With the emergence of big data, systems that can quickly select informative features and remove redundant ones become crucial. Besides the performance gain that could be achieved in the downstream tasks by removing irrelevant features, reducing the number of features improves memory and computational efficiency significantly. This enables providing energy-efficient systems. Moreover, it is useful for improving the interpretability of model-driven decisions. The unsupervised selection of the informative features effective cases where labeled data limited very expensive to collect. We do not expect that there is negative societal impact.
10

---

References
[1] Ang, J.C., Mirzal, A., Haron, H., Hamed, H.N.A.: Supervised, unsupervised, and semi- supervised feature selection: review on gene selection. IEEE/ACM transactions putational biology and bioinformatics 13(5), 971-989 (2015)
[2] Anguita, D., Ghio, A., Oneto, L., Parra Perez, X., Reyes Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In: Proceedings of the 21th international European symposium artificial neural networks, computational intelligence and machine learning. pp. 437-442 (2013)
[3] Ashby, M., Baaij, C., Baldwin, P., Bastiaan, M., Bunting, O., Caimcross,, A., Chalmers, C., Corrigan, L., Davis, S., van Doom, N., al. Exploiting unstructured sparsity on next-generation datacenter hardware (2019)
[4] Atashgahi, Z., Sokar, G., van der Lee, T., Mocanu, E., Mocanu, D.C., Veldhuis, R., Pechenizkiy, M.: Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. Machine Learning pp. 1-38 (2021)
[5] Balın M.F., Abid, A., Zou, J.: Concrete autoencoders: Differentiable feature selection and reconstruction. In: International conference on machine learning. pp. 444-453, PMLR (2019)
[6] Bengio, Y: Learning deep architectures for AL. Now Publishers Inc (2009)
[7] Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35(8), 1798-1828 (2013)
[8] Bibikar, S., Vikalo, H., Wang, Z., Chen, X.: Federated dynamic sparse training: Computing less, communicating less, yet learning better. Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI) (2022)
[9] Cai, D., Zhang, C.,, He, X.: Unsupervised feature selection for multi-cluster data. In: Proceedings of the 16th ACM SIGKDD international conference Knowledge discovery and data mining. pp. 933-342 (2010)
[10] Cai, J., Luo, J., Wang, S., Yang, S.: Feature selection in machine learning: A new perspective. Neurocomputing 300, 70-79 (2018)
[11] Chandrashekar, G., Sahin, F: A survey feature selection methods. Computers & Electrical Engineering 40(1), 16-28 (2014)
[12] Chen, J., Stern, M., Wainwright, M.J., Jordan, M.I.: Kernel feature selection via conditional covariance minimization. Advances in Neural Information Processing Systems 30 (2017)
[13] Chen, Y.H., Yang. T.J., Emer, J., Sze, V: Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9(2), 292-308 (2019)
[14] Cortes, C., Vapnik, V.: Support-vector networks. Machine learning 20(3), 273-297 (1995)
[15] Curci, S., Mocanu, D.C., Pechenizkiyi, M.: Truly sparse neural networks at scale. arXiv preprint arXiv:2102.01332 (2021)
[16] Defferrard, M., Benzi, K., Vandergheynst, P., Bresson, X.: FMA: A dataset for music analysis. In: 18th International Society for Music Information Retrieval Conference (ISMIR) (2017),
ftttps://ariv..omm
abss/612..11..
[17] Defferrard, M., Mohanty, S.P.. Carroll, S.F., Salathé, M.: Learning recognize musi- cal genre from audio. In: The 2018 Web Conference Companion. ACM Press (2018). https://doi.org/00.1 0115://185558311211.roorr..
org/ /tabs////...../ 05337
[18] Doquet, G., Sebag, M.: Agnostic feature selection. In: Joint european conference on machine learning and knowledge discovery in databases, pp. 343-358. Springer (2019)

---

[19] J.G., Brodley, C.E.; Feature selection for unsupervised learning. Journal of machine learning research ((uu), 845-889 (2004)
[20] Evci, U., Gale, T., Menick, J., Castro, P.S., Elsen, E.: Rigging the lottery: Making all tickets winners. In: International Conference on Machine Learning. pp. 2943-2952. PMLR (2020)
[21] Fanty, M., Cole, R.: Spoken letter recognition. Advances in neural information processing systems 3 (1990)
[22] Gu, Q., Li, Z.., Han, J.: Generalized fisher score for feature selection. In: 27th Conference Uncertainty in Artificial Intelligence, UAI 2011. pp. 266-273 (2011)
[23] Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of machine learning research 3(Mr)), 1157-1182 (2003)
[24] Guyon, I., Gunn, S., Nikravesh, M., Zadeh, L.A.: Feature extraction: foundations and applica- tions, vol. 207. Springer (2008)
[25] Han, K., Wang. Y., Zhang, C., Li, C., Xu, C.: Autoencoder inspired unsupervised feature selection. In: 2018 IEEE international conference acoustics, speech and signal processing (ICASSP). pp. 2941-2945. IEEE (2018)
[26] He, X., Cai, D., Niyogi, P.: Laplacian score for feature selection. Advances in neural information processing systems 18 (2005)
[27] Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N.,, Peste, A.: Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research 22(241), 1-124 (2021)
[28] Hong, C., Sukumaran- Rajam, A., Nisa, I., Singh, K., Sadayappan, P.: Adaptive sparse tiling for sparse matrix multiplication. In: Proceedings the 24th Symposium on Principles and Practice of Parallel Programming. pp. 300-314 (2019)
[29] Hooker, S.: The hardware lottery. Communications of the ACM 64(12), 58-65 (2021)
[30] Huang, D.S., Yu, HJJ.: Normalized feature vectors: novel alignment-free sequence com- parison method based on the numbers of adjacent amino acids. IEEE/ACM Transactions Computational Biology and Bioinformatics 10(2), 457-467 (2013)
[31] Huang, S., Chaudhary, K., Garmire, L.X.: More is better: recent progress in multi-omics data integration methods. Frontiers in geneties 8, 84 (2017)
[32] Hull, J.J.: A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence 16(5), 550-554 (1994)
[33] Jakulin, A.: Machine learning based on attribute interactions. Ph.D. thesis, Univerza v Ljubljani (2005)
[34] Jayakumar, S., Pascanu, R., Rae, J.. Osindero, S., Elsen, E.: Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems 33, 20744-20754 (2020)
[35] Lang, K.: Newsweeder: Learning to filter netnews. In: Machine Learning Proceedings 1995, pp. 331-339, Elsevier (1995)
[36] LeCun, Y.: The mnist database of handwritten digits. http://yann. lecun, com/exdb/mnist/ (1998)
[37] Lemhadri, I., Ruan, F., Tibshirani, R.: Lassonet: Neural networks with feature sparsity. In:
International Conference on Artificial Intelligence and Statistics. pp. 10-18. PMLR (2021)
[38] Ji Cheng, K., Wang, S., Morstatter, F., Trevino, R.P., Tang. J., Liu, H.: Feature selection: A data perspective. ACM computing surveys (CSUR) 50(6), 1-45 (2017)
[39] Li, J., Cheng, K., Wang, S., Morstatter, F,, Trevino, R.P., Tang, J., Liu, H.: Feature selection: data perspective. ACM Computing Surveys (CSUR) 50(6), 94 (2018)
12

---

[40] Li. Y., Chen, C.Y., Wasserman, W.W.: Deep feature selection: theory and application to identify enhancers and promoters. Journal of Computational Biology 23(5), 322-336 (2016)
[41] Lin, D., Tang, X.: Conditional infomax learning: An integrated framework for feature extraction and fusion. In: European conference on computer vision. pp. 68-82. Springer (2006)
[42] Lindenbaum, O., Shaham, U., Peterfreund, E., Svirsky, J., Casey, N., Kluger, Y.: Differentiable unsupervised feature selection based on a gated laplacian. Advances Neural Information Processing Systems 34, 1530-1542 (2021)
[43] Liu, C., Bellec, G., Vogginger, B., Kappel, D., Partzsch, J., Neumärker, F., Höppner, S., Maass, w.. Furber, S.B., Legenstein, R., et al:: Memory-efficien deep learning a spinnaker prototype. Frontiers in neuroscience p. 840 (2018)
[44] Liu, H., Motoda, H.: Feature selection for knowledge discovery and data mining, vol. 454. Springer Science & Business Media (2012)
[45] Liu, J., Ji, S., Ye, J.: Multi-task feature learning via efficient 2, 1-norm minimization. In: Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, UAI 2009. pp. 339-348, AUAI Press (2009)
[46] Liu, S., Chen, T., Atashgahi, Z., Chen, X., Sokar, G., Mocanu, E., Pechenizkiy, M., Wang, Z., Mocanu, D.C.: Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. In: International Conference on Learning Representations (2022)
[47] Liu, S., der Lee, T.V., Yaman, A., Atashgahi, Z., Ferraro, D., Sokar, G., Pechenizkiy, M., Mocanu, D.C.: Topological insights into sparse neural networks. In: Joint European conference on machine learning and knowledge discovery in databases. pp. 279-294. Springer (2020)
[48] Liu, S., Mocanu, D.C., Matavalam, A.R.R., Pei, Y., Pechenizkiy, M.: Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications 33, 2589-2604 (2020)
[49] Liu, S., Mocanu, D.C., Pei, Y., Pechenizkiy, M.: Selfish sparse rnn training. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, Pro- ceedings of Machine Learning Research, vol. 139, pp. 6893-6904. PMLR (18-24 Jul 2021)
[50] Liu, S., Yin, L., Mocanu, D.C., Pechenizkiy, M.: Do actually need dense over- parameterization? in-time aver-parameteriiztionn in sparse training. In: International Conference on Machine Learning. pp. 6989-7000. PMLR (2021)
[51] Miao, J., Niu, L.: A survey on feature selection. Procedia Computer Science 91, 919-9199999 926 (2016)
[52] Min, F., Hu, Q., Zhu, W.: Feature selection with test cost constraint. International Journal of Approximate Reasoning 55(1), 167-179 (2014)
[53] Mocanu, D.C., Mocanu, E., Pinto, T., Curei, S., Nguyen, P.H., Gibescu, M., Ernst, D., Vale, Z.A.: Sparse training theory for scalable and efficient agents. In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. pp. 34-38 (2021)
[54] Mocanu, D.C., Mocanu, E., Stone, P., Nguyen, P.H., Gibescu, M., Liotta, A.: Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications 9(1), 1-12 (2018)
[55] Nene, S.A., Nayar, S.K., Murase, H., et al.: Columbia object image library (coil-100) (1996)
[56] Nie, F., Huang, H., Cai, X., Ding, C.: Efficient and robust feature selection via joint 2, 1-norms minimization. Advances in neural information processing systems 23 (2010)
[57] Özdenizei, O., Legenstein, R.: Training adversarially robust sparse networks via bayesian
connectivity sampling. In: International Conference on Machine Learning. pp. 8314-8324.
PMLR (2021)
13

---

[58] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury. J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024-8035.
Curran Associates, Inc. (2019)
[59] Raihan, M.A., Aamodt, T.: Sparse weight activation training. In: Larochelle, H.., Ranzato, M.. Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 15625-15638. Curran Associates. Inc. (2020)
[60] Sheikhpour, R., Sarram, M.A., Gharaghani, S., Chahooki, M.A.Z..: A survey on semi-supervised feature selection methods. Pattern Recognition 64, 141-158 (2017)
[61] Singh, D., Srimente-Gonzäle,, H., Petrovich, M., Kawakami, E., Yamada, M.: Fsnet: Feature selection network on high-dimensional biological data. arXiv preprint arXiv:2001.085 1000020822 (2020)
[62] Sokar, G., Mocanu. D.C., Pechenizkiy, M.: Spacenet: Make free space for continual learning. Neurocomputing 439, 1-11 (2021)
[63] Sokar, G., Mocanu, E., Mocanu, D.C., Pechenizkiy, M., Stone, P.: Dynamic sparse training for deep reinforcement learning. International Joint Conference on Artificial Intelligence (2022)
[64] Spira, A., Beane, J.E., Shah, V., Steiling, K., Liu, G., Schembri, F,, Gilman, S., Dumas, Y.M., Calner, P., Sebastiani, P., et al:: Airway epithelial gene expression in the diagnostic evaluation of smokers with suspect lung cancer. Nature medicine 13(3), 361-366 (2007)
[65] Sun, L.,, Hui, A.M., Su, Q., Vortmeyer, A., Kotliarov, Y., Pastorino, S., Passaniti, A., Menon, J., Walling, J. Bailey, R., et al.: Neuronal and glioma-derived stem cell factor induces angiogenesis within the brain. Cancer cell 944,, 287-300 (2006)
[66] Wang, P., Ji, Y., Hong, C., Lyu, Y., Wang, D., Xie, Y.: Snrram: an efficient sparse neural network computation architecture based on resistive random-access memory. In: 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC). pp. 1-6. IEEE (2018)
[67] Wang. S., Ding, Z., Fu, Y.: Feature selection guided auto-encoder. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 31 (2017)
[68] Weinberger, K., Dasgupta, A., Langford, J., Smola, A., Attenberg, J.: Feature hashing for large scale multitask learning. In: Proceedings of the 26th annual international conference on machine learning, pp. 111--1120 (2009)
[69] Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnis novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708 708.07777777777 (2017)
[70] Yamada, M., Jitkrittum, W., Sigal, L., Xing, E.P., Sugiyama, M.: High-dimensional feature selection by feature-wise kernelized lasso. Neural computation 26(1), 185-207 (2014)
[71] Yamada, Y., Lindenbaum, O.. Negahban, S., Kluger, Y.: Feature selection using stochastic gates. In: International Conference on Machine Learning. pp. 10648-10659, PMLR (2020)
[72] Yang, Y., Shen, H.T., Ma, 7., Huang, 7., Zhou, X.: L.2, 1-norm regularized discriminative feature selection for unsupervised. In: Twenty-second international joint conference on artificial intelligence (2011)
[73] Yuan, G., Ma. X., Niu, W., Li. Z., Kong, Z., Liu, N., Gong, Y., Zhan, Z., He. C., Jin, Q. et al.: Mest: Accurate and fast memorr-cconomic sparse training framework on the edge. Advances in Neural Information Processing Systems 34 (2021)
[74] Zhang, R., Nie, F., Li, X., Wei, X.: Feature selection with multi-view data: A survey. Information Fusion 50. 158-167 (2019)
[75] Zhao, 7.,, Liu, H.: Semi-supervised feature selection via spectral analysis. In: Proceedings of the 2007 SIAM international conference on data mining. pp. 641-646. SIAM (2007)
14

---

[76] Zhou, A., Ma, Y. Zhu, J., Liu, J., Zhang, Z.. Yuan, K., Sun, W., Li, H.: Learning n: M fine-grained structured sparse neural networks from scratch. In: International Conference on learning Representations (2020)
[77] Zhu, H., Jin, Y: Multi-objective evolutionary federated learning. IEEE Transactions on Neural Networks and Learning Systems 31(4), 1310-1322 (2019)
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A] No potential negative societal impacts have been found. See Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.3 and Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Tables 2 and Figure
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Sections 4.3 and
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sections 4.2 and
4
(b) Did you mention the license of the assets? [Yes] See Section 4.3
(c) Did you include any new assets either the supplemental material or as URL? [Ys]] The code of this work is included in the supplemental material.
(d) Did you discuss whether and how consent obtained from people whose data you're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "zHQvPF8B0Kw", "reviewer": "Reviewer_U347", "summary": "The authors propose a method for unsupervised feature selection which is based on Dynamic Sparse Training applied to an autoencoder with a single hidden dimension and trained with reconstruction loss. The number of selected features is given as an hyperparameter along with sparsity level of the NN. Fixing the number of epochs to be 10 for a proposed method while the other methods are trained with 100 epochs, produces comparable results in terms of accuracy to both supervised and unsupervised methods. ", "questions": "* Could you please provide a more detailed comparison to the QS method. Although your method outperforms it, the improvement is marginal in 7 out of 10 datasets (in t.o. accuracy)?\n* Is it a typo in Table 2 (AEFS has the maximal accuracy on FashinMNIST)?\n* Have you re-implemented other methods by yourself or used them as-is? I didn’t find them in your supplementary.\n* Line 230 (Consistency and Stability) - I think it is better to provide a numeric comparison to support your claim. \n* Why are some results missing from Table 2?\n* Once you compare your method to classical unsupervised FS, I think it could be interesting to compare the total training time between non-* NNet and NNet-based methods.\n* Have you conducted experiments on sparsity level hyper parameter, how does it affects the performance of your method (accuracy/sparsity tradeoff)?\n", "limitations": "The limitations are mentioned, also the social impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The method reduces the number of epochs x10 less than other methods\n+ The experiments conducted on datasets that come from different domains: image, speech, time series, text, bio\n+ Simple method, with clear description, easy to follow\n+ Single loss term without additional reguralizations\n+ Outperforms almost in all datasets selected for the experiments\n\nWeaknesses:\n - Some related recent works are missed, f.e. DUFS [1] - the method outperforms both CAE and MCFS on real datasets\n - The author has a claim on “robustness of our method in extremely noisy environments and its effectiveness for datasets with very high dimensional feature space and a few training samples”, which is misleading. Noisy environment is defined both in terms of the number of noise features which are unrelated to dataset labels, and also in terms of noise in data samples. This work doesn’t provide any experimental results on datasets with noisy samples. \n- The datasets that were chosen to validate the effectiveness of the method are not challenging since the number of samples in the datasets larger or near the same as the number of features.\n- The supervised methods chosen for comparison are mostly classical non-NNet methods so they couldn’t serve as an upper bound (in terms of accuracy) for the proposed unsupervised method, while only a single NN-based method was chosen.\n- The number of target selected features should be provided as a hyperparameter. \n- The experiments run only 5 times for each method and dataset, I think it should be at leaset 10 times.\n\n[1] Lindenbaum, Ofir, et al. \"Differentiable unsupervised feature selection based on a gated laplacian.\" Advances in Neural Information Processing Systems 34 (2021): 1530-1542.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "xXIfanKOdea", "reviewer": "Reviewer_cbLE", "summary": "The authors propose an extension to a recent auto-encoder-based sparse network approach for feature selection/extraction. Specifically, they extend the previously proposed drop and grow approach for inducing sparsity by leveraging the contribution of an input or output neuron to the reconstructed output. Experiments on 10 different datasets show that the proposed approach provides statistically significant gains in computational cost compared to the previous approach(es).", "questions": "1. Beyond the classical classification tasks, how would the proposed approach fare against the state-of-the-art in CV, NLP, and graph ML tasks?\n2. Why are only the input and output neurons subject to the drop and grow process but not the hidden neurons?\n3. I can understand the improvements in efficiency but why does the classification accuracy improve with the use of WAST? Section 5.1 describes this additionally but doesn't explain the root causes.\n", "limitations": "Yes, the authors have addressed many of the limitations and societal impact. They might additionally wish to consider the questions above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Easy to read and follow paper.\n2. Lucid experimental details and crisp presentation.\n3. Fair coverage of competing classical approaches.\n4. Offer to make the code publicly available.\n\nGrowth opportunities\n1. The work is an incremental improvement over the previous approach. The proposed approach is intuitively motivated with several judgement calls (on leveraging neuronal weights differently) that are not theoretically justified.\n2. The field is moving away from explicit feature selection. Hence, the impact of this work might be limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mRuf-f_3FGa", "reviewer": "Reviewer_XbZ9", "summary": "This paper proposes a novel method for unsupervised feature selection called WAST, which is an adaptation of Quick Selection (Atashgahi et al 2021) to give a more efficient search. The general idea of this and related methods is to pass the features through a simple autoencoder and to keep features which are more important. The main difference between WAST and QS is that, on each epoch, after dropping low-importance weights from the autoencoder, QS regrows random weights, WAST adds weights in a way which takes into account the importance of the weights in the previous epoch.", "questions": "QS assigns some importance to the fact that they use denoising autoencoders (because adding noise is supposed to encourage the network to find the most important features), but WAST does not appear to add any noise. Do you have any comment on this? What does this tell us about the approach and why it works?\n\nThere is also something quite disturbing about the fact that this method is competitive with supervised methods, given that their objectives are so different. What does this tell us about the approach and why it works?", "limitations": "The discussion seems well thought out.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper demonstrates good performance (for K=50) on a large number of benchmarks, often beating supervised baselines. More importantly, the paper demonstrates much better performance than *comparable* benchmarks over many values of K, including QS. The paper also shows that WAST is slightly more efficient than QS (which is already much more efficient than some other related methods). \n\nThe benchmarks in question are evaluated on whether or not they retain accuracy. However, typical supervised variable selection is evaluated on whether it selects the *right* features, on artificial datasets for which the true features are known. While one could argue that the objectives of unsupervised feature selection are not the same, I think it is quite reasonable, given that the method presented here is competitive with supervised methods, to ask whether it can also go farther than just picking out useful features to picking the correct ones. \n\nThe technical advance is somewhat minor, given how close the algorithm is to QS, but on the other hand, the gains are excellent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PyKvQ0rdeQf", "reviewer": "Reviewer_y8xW", "summary": "This paper presents an effective unsupervised method on feature selection via auto-encoders, inspired by the attention mechanism. It acquires the idea of the attention framework and applies it onto the weight-zeroing scheme of the autoencoder recursively, resulting in better training efficiency and promising performance on multiple tasks.", "questions": "1. In equation 3), how did you select the \\lamda hyper parameter to balance the two components?\n\n2. In the network propagation, is the output processed with a softmax at the end?\n\n3. What do you think on your work's difference with [Dropconnect](https://proceedings.mlr.press/v28/wan13.html)? I know there definitely are some, but as a popular method, you may want to address it.", "limitations": "I think the authors have adequately addressed the limitations and potential societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. This paper clearly addresses its difference from earlier state-of-the-art level work and identifies its main contributions clearly.\n2. The presentation is very clear, with solid statements and references. At least I did not spot any typo or grammatical mistakes there.\n3. The novelty is clearly confirmed and proven in the paper, with rigorous mathematical proof.\n4. This paper clearly addresses the potential limitation and societal impact.\n\n## Weaknesses\n1. The dataset used for benchmarking is a bit out-dated. For example, for spoken digit recognition task, we may want to replace it with a little bit more popular and up-to-date sets such as TIMIT and [yesno](https://www.openslr.org/1/). They are not necessarily harder than the one covered in the paper, but clearly has more work pieces at state-of-the-art level.\n2. Also, instead of spoken digit recognition, voice command recognition might be a better choice as tasks, as this paper addresses a lot on real-time application and usage.\n3. The work lacks comparison with ad-hoc/hand-crafted features and task-specific features. For example, for spoken digit recognition, one additional experiment with common acoustic features such as [MFCC](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) may help. But this is trivial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept