Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks
Authors: Zahra Atashgahi
Xuhao Zhang
Neil Kichler
Shiwei Liu
Lu Yin
Mykola Pechenizkiy
Raymond Veldhuis
Decebal Constantin Mocanu
Venue: Accepted by TMLR ()

Abstract: Feature selection that selects an informative subset of variables from data not only enhances the model interpretability and performance but also alleviates the resource demands. Recently, there has been growing attention in feature selection using neural networks. However, existing methods usually suffer from high computational costs when applied to high-dimensional datasets. In this paper, inspired by evolution processes, we propose a novel resource-efficient supervised feature selection method using sparse neural networks, named "NeuroFS". By gradually pruning the uninformative features from the input layer of a sparse neural network trained from scratch, NeuroFS derives an informative subset of features efficiently. By performing several experiments on $11$ low and high-dimensional real-world benchmarks of different types, we demonstrate that NeuroFS achieves the highest ranking-based score among the considered state-of-the-art supervised feature selection models. We will make the code publicly available on GitHub after acceptance of the paper.

Published in Transactions on Machine Learning Research (02/2023)
Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks
Zahra Atashgahi*
z.ctashgooss@hdueente..t
Xuhao Zhang!
r.shang- utwente.nl
Neil K
weil.iccher@rrut--uuche de
Shiwei Liu
s.iu8@@ue.nl
Lu Yin?
Lyin@tue.nl
Mykola Pechenizkiy²
mwppechenizkk@@laa...l
Raymond Veldhuis
r.n.j. www.
Decebal Constantin Mocanul.2
d. A.c.moomu@utuente.. ml
Deepormennt of Electrical Engineering, Mathematics and Computer Science, University of Twente, The Netherlands DDpertment of Mathematics and Computer Science, Eindhoven University of Technology, The Netherlands
Reviewed OpenReview: otppe://oe net/ forum? KcD6ugrLKKp
Abstract
Feature selection that selects an informative subset of variables from data not only enhances the model interpretability and performance but also alleviates the resource demands. Recently, there has been growing attention on feature selection using neural networks. However, exist- ing methods usually suffer from high computational costs when applied to high-dimensional datasets. In this paper, inspired by evolution processes, we propose novel resource-effiient supervised feature selection method using sparse neural networks, named "NeuroFS"". By gradually pruning the uninformative features from the input layer of sparse neural network trained from scratch, NeuroFS derives informative subset of features efficiently. By perform- ing several experiments on 11 low and high-dimensional real-world benchmarks of different types, we demonstrate that NeuroF'S achieves the highest ranking based score among the con- sidered state-of-tho-art supervised feature selection modols. The code is available on GitHul
1 Introduction
Feature selection has been gaining increasing importance due to the growing amount of big data. The high dimensionality of data can give rise to issues such as the curse of dimensionality, over-fitting, and high memorr and computation demands Li et al. (2018). By removing the irrelevant and redundant attributes in dataset, feature selection combats these issues while increasing data interpretability and potentially improving the accuracy Chandrashekar Sahin (2014).
The literature feature selection can be stratified into three major categories: filter, wrapper, and embedded methods Chandrashekar & Sahin (2014). Unlike filter methods that perform feature selection before the learning task and wrapper methods that use a learning algorithm to evaluate a subset of the features, embedded
methods use learning algorithms to determine the informative features Zhang et al. (009)). Since embedded
https: http://gggtttbtt// http:///ttttuuboooo/oesaee/ hwwp//aaaaaatttaaggh////ttrrrr

---

Published in Transactions on Machine Learning Research (02/2023)
methods combine feature selection and the learning task into unified problem, they usually perform more effectively than the other two categories in terms of the quality of the selected features Han et al. (0188) Balm
et al. 2019p. Therefore, this paper focuses on embedded feature selection due to its superior performance.
In recent years, there has been a growing interest in using artificial neural networks (ANNs) to perform embedded feature selection. This due to their favorable characteristic of automatically exploring non-linear dependencies among input features, which is often neglected in traditional embedded feature selection methods Tibshirani (1996). In addition, the performance of ANNs scales with the dataset size Hestness et al. (017)) while most feature selection methods do not scale well on large datasets Li et al.
(0118). Moreover, many works have demonstrated the efficacy of neural network-based feature sclection in both supervised Wojtas & Chen
Lu et al. (2018); Lemhadri et al. (2021); ;; Yamada et al. (2020); Singh & Yamada (2020);
(2020) and unsupervised Han et al. (2018); Chandra & Sharma (0055] Balm et al. (0019 Doquet Sebag
(019 Atashgahi et al. (2021) ; Shaham et al. (2022) settings.
However, while being effective in terms of the quality of the selected features, feature selection with ANNs is still challenging task. pvee-aarameterizaioon of neural networks results in nigh-computational and memory costs, which make their deployment and training low-resource devices infeasible Hoefler et al. very few works have tried to increase the scalability of feature selection using neural networks on low-resource devices. F.g., Atashgahi et al. (2021) proposes, for the first time, that sparse neural networks Hoefler et al. (021)) can be exploited to perform efficient feature selection. Their proposed method, QuickSelection, which is designed for unsupervised feature selection, trains a sparse neural network from scratch to derive the ranking of the features using the information of the corresponding neurons in the neural network.
2021 Only
In this paper, by introducing dynamic input neurons evolution into the training of a sparse neural network, propose to use the sparse neural networks to perform supervised feature selection and introduce an efficient feature selection method, named Feature Selection with Neuron Evolution (NeuroFS). Our contributions can be summarized as follows:
We introduce dynamic neuron pruning and regrowing in the input layer of sparse neural networks during training.
Based on the newly introduced dynamic neuron updating process, we propose a novel efficient supervised feature selection algorithm named "NeuroFS".
We evaluate NeuroF'S on 11 real-world benchmarks for feature selection and demonstrate that NeuroFS achieves the highest average ranking among the considered feature selection methods on low and high-dinensional datasets.
2 Background
In this section, we provide background information on feature selection and sparse neural networks.
2.1
Feature Selection
2.1.1 Problem Formulation
In this section, we first describe the general supervised feature selection problem. Consider a dataset X containing samples (zx0), y()), where (f) Id is the i-th sample in data matrix X Rmxd d is the dimensionality of the dataset or the number of the features, and y(t is the corresponding label for supervised learning. Feature selection aims to select a subset of the most discriminative and informative features of X as F. C such that FF,1 = K, where is the original feature set, and K is a hyperparameter of the algorithm which indicates the number of features to be selected.
Objective function: In supervised feature selection, we seek to optimize:
F
arg min
J(f(2.) 0),y
(1)
F, FF, = 1=0
2

---

Published in Transactions on Machine Learning Research (02/2023)
where FF is the final selected feature set, J is a desired loss function, and [ (0 0) is classification function parameterized by 0 aiming at estimating the target for the i-th sample using subset of features Tp.-
Solving this optimization problem can be a challenging task. As the choice of feature subset F, grows exponentially with incrcasing number of features d, solving Equation I is a NP-hard problem. Additionally, it is important that function f that can learn a fruitful representation and complex data dependencies Lemhadri et al. (2021). We choose artificial neural networks due to their high expressive power; a simple one-hidden layer ford-forward neural network is known to be a universal approximator Goodfellow et al. (2016). Finally, as we aim to select leal ures in a computationally efficient manner, in this paper, we choose sparse neural networks to represent the data and perform feature selection.
2.1.2 Related Work
Feature selection methods are classified into three main categories: filter, wrapper, and embedded methods. Filter methods use criteria such as correlation Guyon & Elissooff (2003), mutual information Chandrashckar
& Sahin
2014). Laplacian score He et al. (2006), to rank the features independently from the learning task,
which makes them fast and efficient. However, they are prone to selecting redundant features Chandrashekai
& Sahin (2014). Wrapper methods find a subset of features that maximize an objective function Zhang
ct al. (2019) using various search strategics such as tree structures Kohavi & John (1997) and cvolutionary
algorithms Liu et. al. (1966). However, these methods are costly in terms of compulation. Embedded methods aim to address the drawbacks of the filter and wrapper approaches by integrating feature selection and training tasks to optimize the subset of features. Various approaches have been used to perform embedded feature selection including, mutual information [Battiti (1994] Peng et al. (0055), the SVM
classifier Guyon et al. (2002), and neural networks Setiono & Liu 1997).
Recently, neural network- based feature selection in both supervised Lu et al. 20188 Lemhadri et al. 202 ;
Yamada et al. (2020); (Singh & Yamada (2020); Wojtas Chen (2020) and unsupervised Atashgahi al. 2021 ;
Baln et al. )2 Han et al. (2018): Chandra & Sharma (2015); Doquet & Sebag (2019) settings have gained
incrcasing attention due to their favorable advantages of capturing non-lincar dependencies and showing good
performance on large datasets. However, most of these methods suffer from wver-parammeeerization which leads to high computational costs, particularly on higl-dimensional datasets. QuickSelection Atashgahi et al. (2021) addresses this issue by exploiting sparse neural networks; however, due to the random growth of connections in its topology update stage, might not be able to detect fastly enough the informative features on high- dimensional datasets due to the large search space. As we show in the following sections, we address this issue by gradually pruning uninformative input neurons and exploiting gradients to speed up the learning process.
2.2 Sparse Neural Networks
Sparse neural networks have been proposed to address the high computational costs of dense neural networks Hoefler et al. (2021). They aim to reduce the parameters of a dense neural network while preserving decent. level of performance on the task of interest.
There are two main approaches to obtain a sparse neural network: dense-to-sparse and methods Mocanu et al. p2021).
Dense-to-sparse algorithms start with a dense network and prune the unimportant connections to obtain sparse network LeCun et al. (1990); Hassibi & Stork (1993): Han et al. (2015): Lee et al. (2019); Frankle & Carbin 2018 Molchanov et al. (2017 2019); Gale al. (2019). As they start with dense network, they need the memory and computational resources to fit and train the dense network for at least a couple of iterations. Therefore, they are mostly efficient during the inference phase.
Sparse-to-sparse algorithms aim to bring computational efficiency both during the training and inference. These methods use static Mocanu et al.] (2016) or dynamic Mocanu et al. (2018): : Belloc et al. (2018) sparsity pattern during training. In the following, we will elaborate on sparse training with dynamic sparsity (or started to be known in the literature as dynamic sparse training (DST)), which usually outperforms the static approach.
3

---

Published in Transactions on Machine Learning Research (02/2023)
2.2.1 Dynamic Sparse Training (DST)
DST is a class of methods to train sparse neural networks sparsely from scratch. DST methods aim at optimizing the sparse connectivity a sparse neural network during training, such that they never use dense network matrices during training Mocanu et al. (2021). Formally, DST methods start with sparse neural network (æx 0.) with a sparsity level of S. We have S = 1 T0l
||0_1|a where Og is subset of parameters of the
equivalent dense network parameterized by 0, ||0=llo and ||0||, are the number of parameters of the sparse and dense network, respectively. They aim optimize the following problem:
1 1I m
0- arg min
j(f(z(); 02), ()),
(2)
O l0lo, 110.11
where = called density level. During training, DST methods periodically update the sparse connectiv- ity of the network; e.g., in Mocanu et al. (0018) Evci al. (2020) authors remove fraction of the parameters 0g and add the same number of parameters to the network to keep the sparsity level fixed. In the literature, usu- ally, weight magnitude has been used as a criterion for dropping the connections. However, there exists various approaches for weight regrowth including, random MMoanu et al. (2018); Mostafa & Wang 2201)), gradient- based Evci et al. (202 :: Dai et al. (2019); Dettmers & Zettlemoyer (2019): Jayakumar et al. (2020) locality- based Hoefler et al. 2021 and similarity-
AAaashahii et al. (2019). It has been shown that in many cases, they can match or cven outperform their dense counterparts Frankle & Carbin (2018); Mocanu et al. (2018); Liu al. (2021a b) Evei al. (2022) have discussed in-depth that DST methods improve the gradient flow in the network by updating the sparse connectivity that eventually leads to good performance. In this paper, we exploit sparse neural network training from scratch to design an efficient supervised feature selection method.
3 Proposed Method
In this section, we present our proposed methodology for feature selection using sparse neural networks, named Feature Selection with Neuron evolution (NeuroFS). We start by describing our proposed sparse training algorithm. Then, we explain how the introduced sparse training algorithm can be used to perform feature selection.
3.1 Dynamic Neuron Evolution
Inspired by the weights update policy in DST, we introduce dynamic neuron evolution in the framework of DST to perform efficient feature selection. While existing DST methods update only the connections or the hidden neurons Dai el al. (2019) to evolve the topology of sparse neural networks, we propose to update also the input neurons of the network to dynamically derive a of relevant features of the given input data.
Our proposed neuron evolution process has two steps. Consider network in which only fraction of input neurons have non-zero connections. We periodically update the input layer connectivity by first dropping a fraction of unimportant neurons (neuron removal) and then adding a number of unconnected neurons back to the network (newron regrowth):
Neuron Removal. The criterion used for dropping the neurons is strength, which is introduced in Atashgahi et al. (021). Strength the summation of the absolute weights of existing conncctions for an input neuron. A higher strength of neuron indicates that the corresponding input feature has higher importance in the data. Therefore, drop fraction of low-strength neurons at each epoch. We call the neurons with at least one non-zero weight connection, active, and the neurons without any non-zero connections, inactive.
Neuron Regrowth. After removing unimportant neurons, we explore the inactive neurons. We activate a number of neurons with the highest potential to enhance the learned data representation. We exploit the gradient magnitude of the non-existing connections for cach neuron as a critcrion choose the most important inactive neurons. It has been shown in Evei et al.) (2020) that adding the zero-connections with the largest gradients magnitude in the DST process accelerates the learning and improves the accuracy. Evei
4

---

Published in Transactions on Machine Learning Research (02/2023)
Training
Feature Selection
Epeen
Comsugged
Fredsfowen and Book propagelio
Updating roue lnyer
Updcting Hitlln Layers
grow
Selected Noueniiaarare
Fox
Upokding
Figure Overview of "NeuroFS". At initialization, a sparse MLP is initialized (Section 3.2.1 During training, at each training epoch, after standard feed-forward and back-propagation, input and hidden layers are updated such that large fraction of unimportant inpul. neurons are gradually dropped while giving chance to the removed input neurons for regrowth (Section 3.2.2). After convergence, NeuroFS selects the corresponding features to K active neurons with the highest strength (Section 3.2.3
et al. (022)) also have shown that picking the connections with the highest gradient magnitude increases the gradient flow, which eventually leads to a decent performance. We hypothesize that adding inactive neurons connected to the zero-connections with the highest gradient magnitude to the network would improve the data representation and increase the likelihood of finding an informative set of features.
Dynamic neuron evolution is loosely inspired by evolutionary algorithms Stanley & Miikkulainen 2002 Still, due to the large search space, the latter cannot be directly applied to our problem without signilicantly increased computational time. To alleviate this, we seek inspiration in the dynamics of the evolution process from the bio- logical brain at the epigenetic level, which performs cellular changes (seconds to days time scale) Kowaliw et al. (2014), and not at the phylogenic level (generations time scale) as it is usually performed in evolutionary comput- ing. Accordingly, NeuroFS removes and regrows neurons in the input layer sparsely trained neural network based on chosen criteria at each epoch until reduced optimal set of input neurons remains active in the network. In the next section, will explain how NeuroFS uses dynamic neuron evolution to perform fcature selection.
3.2 NeuroFS
Our proposed algorithm is briefly sketched in Figure 1 In short, NeuroF'S aims at efficiently selecting a subset of features that can learn an effective representation of the input data in sparse neural network. In the following, we describe the algorithm in more details.
3.2.1 Problem Setup
We first start by describing the network structure and problem setup.
Network Architecture. We exploit supervised deep neural network, Multi-Layer Perceptron (MLP). We initialize a sparse MLP fæx 0.), with L layers and sparsity level of S.
Initialization. The sparse connectivity is initialized randomly an Erdos-Reny" random graph Mocanu ct al. (2018). Sparsity level S is determined by hyperparameter of the model, named E, such that the density of layer is s(n! (nn"" 1 Uxn()), and the total number of parameters is equal to 10. 1 where E 21,2...., E} is the layer index and nl)) is number of neurons layer l. The number of connections at each layer is computed as |!o!"]| =e(n!
=e(n! (),
3.2.2 Training
After initializing the network, we start the training process. In summary, we start with a sparse neural network and aim to optimize the topology of the network and the selected subset of features simultancously. During training, we gradually remove the input neurons while giving chance for the inactive neurons to
5

---

Published in Transactions on Machine Learning Research (02/2023)
be re-added to the network. Finally, when the training is finished, select the important features from limited number of active neurons. In the following, we describe the training algorithm in more detail.
At each training epoch, NeuroF'S performs the following three steps:
1. Feel-forward and propppprop.aatoo.. At each epoch, first, standard leed-forward and back-propagation are performed to train the weights of the sparse neural network.
2. Updating Input Layer. After each training epoch, update the input layer. The novelty of our proposed algorithm lies mainly in updating the input layer. During training, NeuroFS gradually decreases the number of active input features. In short, epoch t, gradually prunes a number of input neurons (cprune) and regrows a number of unconnected neurons (Cgrow) back to the network. Updating the input layer in NeuroF'S consists of two phases:
Removal Phase. From the beginning of the training until hremoval updating the input layer is at the removal phase. this phase, the total number of active neurons decreases at cach epoch such that Cprune > Corow, if & & fremoval- We have fremoval fatmar) where O < < 1 is hyperparameter of NeuroF'S determining the neuron removal phase duration, [] is the ceiling function, and tma is the total number of epochs.
Update Phase. From fremoval until the end of training, the number of connected neurons remains fixed in the network and only a fraction of neurons are updated. In other words, Cprune = Cgrow: if t Iyemoval-
Formally, we compute cprune at epoch (t) as follows:
Cremone + Cgrow: & tremoval
Oprune
Cgrows
(t)
otherwise
(3)
Grenne in the removal phase consists of two parts: Cremove and Cgrow- As the overall number of active neurons
(t)
is decreasing in this phase, ( extra neurons to the updated ones will be removed at each epoch. cremove is computed
(f)
Algorithm NeuroFS
Input: Dataset X, sparsity hyperparameter drop fractions Çin and Çh. neuron removal phase duration hyperparameter a, number training epochs mmaz number of features to select K. Initialization: Initialize the network with sparsity level S determined by e (Section 3.2.1 for ( }tmas) do
I. Standard feed-forward and back-propagation
II. Update Input Layer:
0. Compute Cprune (Equation 3 and offo (Equation 7
1. Drop oprune neurons with the lowest strength.
(t)
2. Drop fraction Sin of connections with the lowest magnitude.
3. Select Corow inactive neurons (that have connections with the highest gradient magnitude), to activated.
(t)
4. Regrow as many connections as have been removed to the active neurons.
III. Update Hidden Layers:
for I E (1...., L} do
1. Drop a fraction Ça of connections with the lowest magnitude from layer h'.
2. Regrow many connections as have been removed in layer M/.
end for
end for Feature Selection:
Seloct K features corresponding to the active neurons with the highest strength in the input layer.

---

Published in Transactions on Machine Learning Research (02/2023)
R(()
(0) remoue
(4)
tremoval
t-1 R
(6) removes
(5)
=
R [11 Six ) K7.
(6)
where R(O) is the total number of inactive neurons at epoch t, R is the total number of neurons to be removed in the removal phase, and Çin E R. 0 < Sin < 1 is the update fraction of the input layer. In other words, the total number of active neurons after the removal phase is Sind K. We keep God neurons extra to the number of selected features K, so that the update phase does not disturb the already found important features.
Finally, the number of neurons to grow epoch computed as:
t
[Sin
(7)
tmax
In other words, at each epoch, we add a fraction Çin of the inactive neurons back the network. However, as the number of inactive neurons increases during training, the number of updated neurons will increase consequently. A large number of updated neurons might diverge the network training. Therefore, we decrease the update fraction linearly during training. At epoch t, we update Çin(1 .....) proportion of the total inactive neurons.
After computing ofrune and Cgrow: the input layer is updated as follows:
1. Neuron pruning: (e)
Sprune neurons with lowest strength are dropped from the input layer. The strength of imput neuron i is computed as 8 = (l ||, where un(") is the weights vector of neuron Weight pruning: a fraction Sin of connections with the lowest magnitudes are dropped from the active input features.
3. Neuron regrowth: (t)
Cgrow neurons are selected for being activated and added to the network. As 3.1 these neurons are the ones connected to the connections with the largest absolute gradient among all non-existing connections inactive neurons.
discussed in Section
Weight growing: the same number as the number of removed connections will be added to the network so that the sparsily remains fixed during training. These connections the ones with the largest absolute gradient among all non-existing connections of the active neurons at the current epoch.
3. Updating Hidden Layers. Hidden layers will be updated by updating the sparse connectivity. which is the standard approach in the DST process. We use gradients for weight regrowth Evei et al. (2020). For each hidden layer 1,) NeuroFS performs the following two steps:
1. Weight pruning: a fraction ÇΛ of connections with the lowest magnitude are dropped from layer (!))
Weight growing: the same number as the number removed connections will be added to layer h(), These connections are the ones with the largest absolute gradient among all non-existing connections.
3.2.3 Feature Selection
After the training process is finished, we perform feature selection. We select K neurons with the highest strength out of the {nnd+ + K remained active neurons. The corresponding feature to these K neurons are the most informative and relevant features in our dataset. NeuroFS schematically described Figure and the corresponding pscudocode is available at Algorithm I
4 Experiments and Results
In this section, we first describe the experimental settings and then analyze the performance of NeuroFS and compare it with several state-of-the-art feature selection methods.
7

---

Published in Transactions on Machine Learning Research (02/2023)
Table 1: Datasets characteristics.
Text
CCIL-20 USFS
1024
1440
1150
285
MNIST Tinii
Image
206
766
706400
00500
74266
190000
1580))
10
784
70000
00000
10000
10
Isolet
Speech
S17
1737
6207
1560
KAR
Time
10209
7362
2N47
Spectrumatry
SMK--AN-11B
Biological
187
140
36
2
GLA-BRA-188
40151
180
144
36
4.1 Settings
This section describes the experimental settings, including, dalasets, compared methods, hyperparameters, implementation, and the evaluation metric.
Datasets. We evaluate the effectiveness of NeuroFS on cleven datasets described in Table
Comparison, We have selected seven state-of-theearr feature selection methods for comparison as follows:
Embedded methods: LassoNet Lemhadri ct al. (2021) h exploits a neural network with residual connections to the imput layer and solves a two-component (linear and non-linear) optimization problem to find the feature importance. STG Yamada et al. (2020) exploits a continuous relaxation of Bernoulli distribution in neural network to perform feature selection. QuickSelection Atashgahi et al. (2021) (denoted as QS in the Figures) selects features using the strength of input neurons of a sparse neural network. RFS Nie et al. (2010) employs a joint f2,1nrrm minimization on the loss function and regularization to select features.
Filter methods: Fisher score Gu al. (2011) selects features that maximizes similarity of feature values among the same class. CIFE Lin & Tang (2006) maximizes the conditional redundancy between unselected and selected features given the class labels. Finally, ICAP Jakulin (0055) iteratively selects features maximizing the mutual information with the class labels given the selected features.
Typerparameters. The architecture of the network used in the experiments is a 3-layer sparse MLP with 1000 neurons in each hidden layer. The activation function used for the hidden layers is Tanh (except for Isolet dataset where Relu used), and the output layer activation function is Softmaz. The values for the hyperparam- eters were found through grid search among a small set of values. We have used stochastic gradient descent. (SGD) with momentum of 0.9 as the optimizer. The parameters for training neural network- based methods, including batch size, learning rate, and the number of epochs (tmaz), have been set to 100, 0.01, and 100, respectively. However, the batch size for datasets with few samples (m S 200) was set to 20. The hyperparame- ter determining the sparsity level e is set to 30. Update fraction for the input layer Çin and hidden layer Çn have been set to 0.2 and 0.3 respectively. Neuron removal duration hyperparameter a is set to 0.65. Gin and a are the only hyperparamoters particular to NeuroFS. We use min-max scaling for data preprocessing for all methods except for the BASEHOCK dataset, where we perform standard scaling with zero mean and unit variance.
Implementation. We implemented our proposed method using Keras Chollet et al. (015)). The starting point of our implementation is based on the sparse evolutionary training introduced as SET in Mocanu et al. (2018) 3 to which we added the gradient- gradient-based connections growth proposed in RigL Evci et al. Fisher score, CIFE, ICAP, and RFS, we have used the implementations provided by the Scilit-Feature Feature library Li et al. 2018 The hyperparameter of RFS (7) has been set to 10 (searched among [0.01, 0.1, 0.5, 1, 10).. We implemented QuickSelection Atashgahi et al. (2021) in our code; we adapted it supervised feature selection, as this was not done in the paper proposing QuickSelection. We have used a similar structure and sparsity level (c = 30) to our method for fair comparison. For QuickSelection,
(020).. For
set Ç = 0.3. For STG and LassoNet, we used the implementation provided by the author For STG, we used a 3-layer MLP with 1000 hidden neurons in each layer and set the hyperparameter X = 0.5 (searched among [0.001, 0.01, 0.5, 1, 10))). For LassoNet, we used a 1-layer MLP with 1000 hidden neurons and set M = 10, as suggested by the authors Lemhadri et al. 2021). Please note that we have also tried using a 3-layer MLP for LassoNet. However, it significantly
Available at httttpp:////udong/l///// github. ittpoooo///ooooooooo ht//aciiktt-feeet////tttee httpe://gitbub.co wtt.bl..oo///aaaaaeeee www.w/owww/..//cccooooooooocace ioo
httpe:// /aa::://jnndnnn...m (iuuub,bbuuu wt//cikit feature/
httpa: http://ggghuub..cooooott////////
8

---

Published in Transactions on Machine Learning Research (02/2023)
COIL-20
MNIST
Fa Tesii
USPS
Isolet
MAR
Asurage uccurasy 195)
Asstago ucsurasy 261
Asstage accurasy 761
Aec uge ascurasy (Sl
Avcruge sscurocy (16)
Average uccurocy (h)
8
73
123
(a) Low-dimensional datasets
BASEHOCK
Prostate GE
Arcene
SMK
GLA-BRA-180
92
crage accuracy
Anorage accuracy
arerage accuracy
Average accuracy
Average accuracy
60
200
150
201
150
200
200
100
355
200
LassoNet
Q5
F_score
CIFE
ICAP
RFS
Baseline
(b) higl-dimensional datasets
Figure 2: Supervised feature selection comparison for low (a) and high-dimensional (b) datasets, including accuracy for various values of K (below) and average accuracy over K (above).
increased the rumning time, and particularly large datasets, it exceeded the 12 hours rumning time. In addition, in the other cases, it did not lead to significantly different results than LassoNet with 1-layer MLP. To have a fair comparison, for NN-based methods (NeuroFS, LassoNet, STG, and QS), we used similar training hyperparameters, including learning rate (0.01), optimizer (SGD), batch size (100, except 20 for datasets with few samples (m<))00)), and training epoch (100). We consider a 12 hours limit on the running time of each experiment. The results of the experiments that exceed this limit are discarded. We used a Dell R730 processor to run the experiments. We neural network-based methods using Tesla-P100 GPU with 16G memory.
Evaluation Metrics. For evaluating the methods, use classification accuracy of a. SVM classifier Keerthi et al. (2001) with RBF kernel implemented by Scikit-Learn library and used the default hyperparameters of this library. As some of the compared methods do not exploit neural networks to perform feature selection, we intentionally use a non-ncural network- based classifier to ensure that the evaluation process is objective and does not lake advantage of the same underlying mechanisms as our method. We first find the K important features using each method. Then, we train a SVM classifier on the selected features subset of the training set. We report the classification accuracy
the test set mcasure of performance. We have also evaluated the methods using two other classifiers including KNN and ExtraTrees in Appendix C We have considered classification accuracy using all features the baseline method.
www.nsc................ SVC..tt.l
Table Supervised feature selection comparison (average classification accuracy over various values (%)).
Empty entries show that the corresponding experiments exceeded the time limit (12 hours). Bold and italic fonts indicate the best and second-best performer, respectively.
Low-dimensional Pushion-MN1ST
Datasets USPS
Hiigh-dinensional Datasets
Method
COIL-20
MNIST
Leolet
HAR
BASEHOCK Prostate
CE
Arcene
SMK
GLA-BRA-150
Baseine
100.0
88.3
57
95.03
90.00
91.
S6.ML
72.22
S
71.771 29
85.
0.34
85.03 à: 0.15
96.68
0.16
03.22
0.11
0.221
86.70
78
82:30
0.98
80.46 0.90
Noe.
98.00
80.00.00000
85.812 0.
1:1000
31
1
94.03
0.10
800
89 8026
71 11:33
67.67
2.00
77. 2022 1.8)
STG
00.30
94.16
2: 0.47 0.47
63.74 2
96.558
17
80.13
13
91.87
0.63
85.45
0.78
85.41 2 2.72
73.75
2.78
81.34
2.88
71.19 =: 2.03
Q25
97.23
1.34
94.57
0 2:33 0.35
82.00 2 0.24
90.22
0.20
90.20
92.70
0.37
S7.93 0.46
76.30 -: 7:44
77.08
1.50
82
52.0142.62
72.91 2: 0.69
Fisber
acorn
70.02
0.00
86
0.00
15.85 2 0.00
03.12.:5
0.60
75.58
0.08200000 0.00
62.10
0.00
80.72 0.00
90.50 0.00
66 25
0.00
75
1:0.0.00 0.00
65:45 2: 0.00
CIFE
64.18
0.00
92.07
0.00
10.27 2 0.00
73.90
0.00
74.15
0.00
86.55
0.00
71.865 0.00
65.46 0.00
61.87
0.00
60.27 0.00
65.60 =: 0.60
ICAP
08.65
0.00
92.00
0.00
70.12 2 0.00
04.75
0.00
60.72
0.00
90.20
0.00
92.300 0.00
50.00 0.00
76
0.00
74.57 0.00
70.90 - 0.00
KPS
97
0.00
96.68
0.00
91
04 9.8.0.00000
85.
0.000
90.50 - 0.00
79.17
0.000
9

---

Published in Transactions on Machine Learning Research (02/2023)
4.2 Feature Selection Evaluation
In this section, we evaluate the performance of NeuroFS and compare it with several feature selection algorithms. We run all the methods on the datasets described in Section [.1 and for several values of K E (25, 50, 75, 100, 150, 200). Then, we evaluate the quality of the selected set of features by measuring the classification accuracy on an unseen test set described in Section 4.1 The results are an average of five different seeds. The detailed results for low and high-dimensional dataset, including accuracy for various values of K (below) and average accuracy over K (above), are demonstrated in Figure 2 We have also presented the detailed results for each value of K in Table [ in Appendix D To summarize the results and have a general overview of the performance of each method independent of a particular K' value, we have shown the average accuracy over the different values of K in Table 2
As presented in Figure 2 and Table 2 NeuroFSS is the best performer in 6 datasets out of 11 considered datasets in terms of average accuracy, while performing very closely to the best performer in the remaining cases. Filter methods, such as ICAP, CIFE, and F-score, have been outperformed by embedded methods most datasets considered, as they select. feat independently from the learning task. Among these methods, ICAP performs well on the text dataset (BASEHOCK); this can show that mutual information is informative in feature selection from the text datascts. Among the considered embedded mothods, RFS fails to find the informative Teatures on datasets with a high number of samples (e.g.. MNIST, Fashion-MNIST) or dimensions (e.g., SMK, GLA-BRA-IS0) within the considered time limit.
By looking into the results of all considered methods, it can be observed that neural network-based feature selection methods outperform classical feature selection methods in most cases. Therefore, it can be concluded that the complex non-linear dependencies extracted by the neural network are beneficial for the feature selection task. However, as will be discussed in Section 5.2 the weer-parameterization in dense neural networks, as used for STG and LassoNet, leads to high computational costs and memory requirements, particularly on high-dimensional datasets. NeuroFS and QuickSelection address this issue by exploiting sparse layers instcad of dense ones.
NeuroFS outperforms QuickSelection, which is the sparse competitor of NeuroFS, in terms of average accuracy, particularly on the high-dimensional datasets. This is because, for high-dimensional datasets, QuickSelection needs more training time to find the optimal topology in the large connections search space duc to the random search. NeuroFS alleviale this problem by exploiting the gradient of the connections to find the informative paths in the network while removing the uninformative neurons gradually to reduce the search space.
To summarize the results and have general overview of the methods' performance, we use a ranking- nanking-basod score. For each dataset and value of K, we rank the methods based on their classification accuracy and give a score of 0 to the worst performer, and the highest score (#methods-1) 1) to the best performer. For each method, we compute the average score for different values of K and different datasets. The results are summarized in Figure 3 NeuroFS achieves the highest average ranking on both low and high- sigh-dimensional datasets.
Overall, it can be concluded that inspired by the evolutionary process, NeuroFS can find an effective subset of features by dynamically changing the sparsity pattern in both input neurons and connections. By dropping the unimportant input neurons (based on magnitude) and adding new neurons based on the incoming gradient, it can mostly outperform its direct competitors, LassoNet, STG, and QuickSelection, in terms of accuracy while being efficient by using sparse layers instead of dense over-parameterized layers.
KCP
2.
8.CIFE
CCPP
Lassehict
1
(a) Low- now-dimensional dataset
(b) high-dimensional dataset
Figure 3: Average ranking score over all datasets and K values.
10

---

Published in Transactions on Machine Learning Research (02/2023)
Initialization
Epoch 5
Epoch
Epoch 20
Epoch 50
Epoch
10
10
5
20
25
10
20
25
10
15
20
25
5
10
15
20
25
5
10
15
20
25
10
115
20
5
10
15
20
25
Figure 4: Feature importance visualization on the MNIST dataset (number selected features K-50).
4.3 Feature Importance Visualization
In order to gain a better understanding of the NeuroFS algorithms, in this section, we analyze the feature importance during the training of the network. We run NeuroFS on the MNIST datasct and for K 50 and visualize the strength of input neurons as a heat-map at several epochs in Figure 4
As shown in Figure 4 at the initialization, all the neurons have very close strength/ tongggthimmootaace.. This stems from the random initialization of the weights to small random value. During training, the number of active neurons gradually decreases. The removed neurons This pattern is similar to the MNIST digits dataset, where most digits appear in the middle of the image. Finally, at the last epoch, a limited number of neurons have remained active. We select the most important features out of the active features. In conclusion, this experiment shows that NeuroFS can determine the most important region in the features accurately.
mostly located lowards the edges of this picture.
5 Discussion
In this section, we present the results of several analyses on the performance of NeuroF', including robustness evaluation and hyperparamcter's effect. We have additionally analyzed weight/neuron neuron growth policy Appendix A and compared NeuroFS with two HSICLasso-based feature selection methods and RigL in Appendix and respectively.
5.1 Robustness Evaluation:
Topology Variation
In this section, we analyze the robustness of NeuroFS to variation in the topology. We aim to explore if different runs of NeuroFS converge to similar or distant topologies and whether NeuroFS performance remains stable for these different topologies.
To achieve this aim, conduct two experiments. In the first experiment, we analyze the topology of five networks that trained and initialized with different random seeds. In other words, they start with different sparse connectivities at initialization and have different training paths. In the second experiment, we analyze the topology of five networks initialized with the same sparse connectivity (using similar random seed) and trained with different random seeds. For both experiments, we measure the topology distance among networks using a metric introduced in Liu et al. (2020). called NNSTD, It measures the distance of two sparse networks; NNSTD of means that two networks are identical, and means completely different.
Eacch
Irtekottinn
booth
toechs
tree 12
250d.25 trcch
20222 2.222 22222
22322 222 12222
22322
22222
(a) Different initial sparse connectivity
(b) Similar initial sparse connectivity
Figure 5: Topology distance of five MLPs with (a) different and (b) similar initial sparse connectivity (topology). Input layers converge to relatively similar topologies in both cases, while hidden layers remain distant. N; refers to the network trained with 4th random seed.
11

---

Published in Transactions on Machine Learning Research (02/2023)
Table 3: NeuroFSS Classification Accuracy (%) on the MNIST dataset for five networks (K = =5).
No
N;
Ng
Ng
Na
NeuroFS (Dulloost csitial
95.6
85.3
95,2 06.2
86.5
96.6
N
98.6
94.4
05.4
95.8
We perform both experiments on the MNIST dataset to find the K = 50 most important features. The topology distance of the networks at different epochs are depicted in Figure 5 as 2d heat Each row depicts the distances for one layer of different networks. Each tile in the heatmaps refers to the distance between two layers of two networks. In these figures, N₂ refers to the network trained with 1th random seed. The corresponding accuracies are shown in Table
In Figure 5a the networks are very distant at the beginning as their sparse connectivity (topology) initialized differently. During training, while their hidden layers remain distant, their input layers become more similar. Considering these figures and comparing them with the results in Table 3 it can be observed that while the feature selection remains almost the same, the network topologies do not. This indicates that NeuroFS can find several well-performing networks.
The similarity of the network topologies in Figure 5b almost match the pattern of Figure 5a While the networks start from the same sparse connectivity, they become distant the next epoch when they start training with different random seeds. This indicates that NeurolS explores various connectivities during training. Interestingly, in the end, the converged input layers more similar to each other than the experiment due to the similar sparse connectivity at initialization. As shown in Table 3 the corresponding accuracies are close together. Experiment 2 confirms the observations in experiment 1, where NeuroFS finds distant topologies with very close feature selection performance.
To conclude, NeuroFS is robust to changes in topology. While it finds very different topologies overall, the input layers converge to relatively similar topologies, resulting in close feature selection performance.
5.2 Computational Efficiency of NeuroFS
In this section, we analyze the computational efficiency of NeuroFS. We present the number of training FLOPs and the number of parameters of NeuroFS and compare it with its neural network-based competitors.
Estimating the FLOPs (loating-point operations) and parameter count is commonly used approach to analyze the efficiency gained by a sparse neural network compared to its dense equivalent network Evei ct al. 2020): Sokar et al. (2021). Number of parameters indicates the size of the model, which directly affects the memory consumption and also computational complexity. FLOPs estimates the time complexity of an algorithm independently of its implementation. In addition, since existing deep learning hardware is not optimized for sparse matrix computations, most methods for obtaining sparse neural networks only simulate sparsity using binary mask over the weights. Consequently, the running time of these methods does not reflect their elficiency. Besides, developing proper pure sparse inplementations for sparse neural networks is currently a highly researched topic pursued by the community Hooker (2021). Thus, as our paper is, in its essence, theoretical, we decided to let this engineering research aspect for future work. Therefore, we also use parameter and FLOPs count to analyze efficiency.
To give an intuitive overview of the efficiency of NouroFS, we compare NeuroFS with its noural network-basod competitors. We compute the FLOPs and number of parameters of two dense MLPs with one (Dense and three hidden layers (Densez). These are the architectures used by LassoNet and STG, respectively. However, it should be noted that LassoNet might require several rounds of training for the dense model. Therefore, we have also computed the actual training FLOPs for LassoNet. In addition, as the computational cost of QuickSelection is similar to our method, we refer to both NeuroFS and QuickSelection as Sparse.
As explained in Scction 3.2.1 the sparsity/densti level determined by the e. The density level of the network (D), the number of paraneters and FLOP count of NeuroFS, and the compared methods are shown in Table 4 We estimate the FLOP count. for the considered methods, using the implementation provided by Evei et al.
12

---

Published in Transactions on Machine Learning Research (02/2023)
Neuren removal duration
Sparsity level hypersaramster
Update fraction
C
60
64
31
11
25-
0.5
0.03
855
0:30 0.20
BASBHOCK Fruran
Figure 6: Effect of hyperparameters on the performance of the algorithm (K = 100).
As can be scen from Table NeuroFS and QuickSelection (Sparse) have the least number of parameters and FLOPs among the considered architectures on all considered datasets, particularly on high-dimensional datasets. In addition, discussed in Section 4.2 NeuroFS outperforms LassoNet, STG, and QuickSelection, in terms of accuracy most cases considered. In short, NeuroFSS is efficient in terms of memory requirements and computational costs while finding the most informative subset of the features on real-world benchmarks, including low and high-dimensional datasets.
5.3 Hyperparameters Effect
In this section, we analyze the effect of hyperparametors of NeuroFS on the quality of the selected features. The hyperparaneters include neuron removal duration fraction a, hyperparameler determining sparsity level e, and the update fraction of the input layer Sin- We try different sets of values for each of these hyperparameters and measure the performance of NeuroF'S when selecting K- 100 features. The results are presented in Figure 6
The results of most datasets are stable for different sets of hyperparameter values. However, high-dimensional datasets with few samples (d 2 10000 and m < 200) are sensitive to the sparsity level hyperparameter. The feature selection performance decreases for higher densities; this might come from over-fitting of the network for large parameter count and a low number of training samples. We select a = 0.65, & = 30, and Çan = 0.2 as the final values for the other experiments.
6 Conclusion
This paper proposes a novel supervised feature selection method named NeuroFS. NeuroFS introduces dynamic neuron evolution in the training process of sparse neural network to find informative set. of features. By evalusting NeuroF'S on real-world benchmark datasets, we demonstrated that it achieves the highest ranking-based score amonn the considered stale-of-the-art supervised feature selection models. However, due to the general lack of knowledge on optimally implementing sparse neural networks during training, NeuroFS does not take full advantage of its thcoretical high computational and memory advantages. We let the development of this challenging research direction for future work, hopefully, in a greater joint effort of the community.
Table 4: Number of parameters (x10²) and Number of training FLOPs (x1012) of NeuroFS (Sparse) and the oquivalent dense MLPs on different datasets.
#parumatic (x10") Denae,
#PLOPs (x10"))
Datased
Denxity
Sparas
Denan
Sporse
Denoee
Densey LawseNat
CDLL-20
6.29%%
1.91
10.34
30.34
0.13
9.72
2.13
4.5
MNEST
6.57%
1.84
7.54
27.94
6.66
19
(8.000
1001
371.0
Fushion-XNNST
6.57%
6.68
10.12
BASEHOCK
3.77%
120.02
0.05
Prodiate 63B
4.15%
0.02
CANI--
3.42%
7.52
199.83 95
210.96
0.07
1.70
1.97
4.4
GLA- BRA--88-
3.15%
16.29
491.55
511.56
0.18
5.31
5.62
12.6
13

---

Published in Transactions on Machine Learning Research (02/2023)
References
Zahra Atashgahi, Joost Picterse, Shiwei Liu, Decebal Constantin Mocamu, Raymond Veldhuis, and Mykola Pechenizkiy. A brain-inspired algorithm for training highly sparse neural networks. arv preprinl WVXi:1108..07188, 2019.
Zahra Atashgahi, Ghada Sokar, Tim van der Lee, Elena Mocanu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pechenizkiy. Quick and robust feature selection: the strength of energy-effciinnt sparse training for autocncoders. Machine Learning, pp. 1-38, 2021.
Muhammed Fatih Balm,, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference Machine Learning, pp. 444 444-453, 2019.
Roberto Battiti. Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on neural networks, 5(4):537 550, 1994.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Represenlations, 2018.
B Chandra and Rajesh K Sharma. Exploring autoencoders for unsupervised feature selection. In 2015 International Joint Conference Neural Networks (IJCNN), pp. 1-6. IEEE, 2015.
Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Computers & Electrical Engincering, 40(1):16-28, 2014.
François Chollet et al. Keras. kttps://keras.// 10 2015.
Xiaoliang Dai, Hongxu Yin, and Niraj Jha. Nest: A neural network synthesis tool based on a grow-and-prune paradigm. IEEE Transactions on Computers, 68(10): 1497, 2019.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint ar NKk::1907.04... 2019.
Guillaume Doquet and Michèle Scbag. Agnostic fcature selection. In Joint European Conference on Machine Springer, 2019.
Learning and Knowledge Discovery in Dalabuses, pp.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943 2952. PMLR, 2020.
Utku Evci, Yani Toannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 6577-6586, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks. orXiv preprint D.Xiv:1902.09514 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Quanquan Gu, Zhenhui Li, and Jiawei Han. Generalized fisher score for feature selection. In 27th Conference on Uncertainty in Artificial Intelligence, UAI 2011, pp. 266-20 273, 2011.
Isabelle Guyon and André Elissoeff. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157 1182, 2003.
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Machine learning, 46(1) (3)9))390 422, 2002.
14

---

Published in Transactions on Machine Learning Research (02/2023)
Kai Han, Yunhe Wang, Chao Zhang, Chao Li, and Chao Xu. Autoencoder inspired unsupervised feature selection. In 2018 TEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2941 2945. IEEE, 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advonces in neural information processing systems, pp. 1135-1143, 2015.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances neural information processing systems, pp. 164-171, 1993.
Xiaofci Hc, Dong Cai, and Partha Niyogi. Laplacian score for feature selection. In Advances neural informalion processing systems, pp. 507 514, 2006.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Pat- wary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint ar 1i::171.. 00409, 2017.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 2021.
Sara Hooker. The hardware lottery. Communications of the ACM, 64(12):58 65, 2021.
Alcks Jakulin. Machine learning based on attribute interactions. PhD thesis, University of Ljubljana, 2005.
Siddhant Jayakumar, Razvan Pascanu, Jack Rac, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744 33:20744-20754, 2020.
S. Sathiya Keerthi, Shirish Krishnaj Shevade, Chiranjib Bhattacharyya, and Karuturi Radha Krishna Murthy. Improvements to platt's smo algorithm for svm classifier design. Neural computation, 13(3):637 649, 2001.
Ron Kohavi and George H. John. Wrappers for feature subset selection. Artificial intelligence, 97(1-2):273 324, 1997.
Taras Kowaliw, Nicolas Bredeche, Sylvain Chevallier, and René Doursat. Artificial neurogenesis: An introduction and selective review. Growing Adaptive Machines, pp. 1-60, 2014.
Kazuki Koyama, Keisuke Kiritoshi, Tomomi Okawachi, and Tomonori lzumitani. Effective nonlinear feature selection method based on hsic lasso and with variational inference. In International Conference on Artificial Intelligence and Statistics, pp. 10407-1000 10421. PMLR, 2022.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruming based on connection sensitivity. In International Conference Learning Representations, 2019.
Ismael Lemhadri, Feng Ruan, Louis Abraham, and Robert Tibshirani. Lassonet: A neural network with feature sparsity. Journal of Machine Learning Research, 22(127):1 29, 2021.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys (CSUR), 50(6):94, 2018.
Dahua Lin and Xiaoou Tang. Conditional infomax learning: An integrated framework for feature extraction and fusion. In European conference on computer vision, pp. 68-82. Springer, 2006.
Huan Liu, Rudy Setiono, et al. A probabilistic approach feature selection-a filter solution. In ICML, volume 96, pp. 319-30 327. Citeseer, 1996.
15

---

Published in Transactions on Machine Learning Research (02/2023)
Shiwei Liu, Tim van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Topological insights into sparse neural networks. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD) 2020., 2020.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neurorogeneration. Advances in Neural Information Processing Systems (NeurIPS 2021), 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over- parameterization? in-time over- prramettttttinnoooonnn in sparse training. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6989 6989-7000. PMLR, 18-24 Jul 2021b.
Yang Lu, Yingying Fan, Jinchi Lv, and William Stafford Noble. Deeppink: reproducible feature selection deep neural networks. In Advances in Newrol Information Processing Systems, pp. 8676 8686, 2018.
topological insight into restricted boltzmann machines. Machine Learning, 104(2-3):243 270, 2016. Decebal Constantin Mocanu, Elena Mocanu, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. A
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018.
Decebal Constantin Mocanu, Flena Mocanu, Tiago Pinto, Selima Curci, Phuong H Nguyen, Madeleine Gibescu, Damien Ernst, and Zita A Vale. Sparse training theory for scalable and efficient agents. In Proceedings of the 20th International Conference on Aulonomous Agents and MulliAgend Systems, pp. 34-38, 2021.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. International Conference on Learning Representations, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Turi Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the TEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Hesham Mostafa and Xin Wang. Parameter efficient. training of deep convolutional neural networks by dynamic sparse reparameterization. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4646-4655. PMLR, 09-15 Jun 2019.
Feiping Nie, Heng Huang, Xiao Cai, and Chris Ding. Efficient and robust feature selection via joint 12, 1-norms minimization. Advances in neural information processing systems, 23, 2010.
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of max-dependencyy max-relevance, and min-redundancy. IEEE Trunsactions on pallern analysis and machine intelligence, 27(1): 1238, 2005.
Rudy Setiono and Huan Liu. Neural-network feature selector. IEFE Imnsacttins on neural networks, 8(3): 654 662, 1997.
Uri Shaham, Ofir Lindenbaum, Jonathan Svirsky, and Yuval Kluger. Deep unsupervised feature selection by discarding muisance and corrclated features. Neural Networks, 152:34-43, 2022.
Dinesh Singh and Makoto Yamada. Fsnet: Feature selection network on high-dimensional biological data. arXiv preprint arXiv:2001.08920 2020.
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkly, and Peter Stone. Dynamic sparse training for deep reinforcement learning. ar preprint orXiv:2106.042) 2021.
16

---

Published in Transactions on Machine Learning Research (02/2023)
Kenneth O Stanley and Risto Milkkulainen. Evolving neural networks through augmenting topologies. Evolulionary compulation, 10(2):99-11 127, 2002.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society:
Series B (Methonooooiical) 58(1):267-28) 288, -8822888,8
1996.
Maksymilian Wojtas and Ke Chen. Feature importance ranking for deep learning. Advances in Newral Information Processing Systems, 33:5105 5114, 2020.
Makoto Yamada, Wittawat Jitkrittum, Leonid Sigal, Eric P Xing, and Masashi Sugiyama. High-dimensional feature selection by feature-wise kernelized lasso. Neural computation, 26(1):185 207, 2014.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference Machine Learning, pp. 10648-10659. PMLR, 2020.
Rui Zhang, Feiping Nie, Xuelong Li, and Xian Wei. Feature selection with multi-view data: A survey. Information Fusion, 50:158 167, 2019.
17

---

Published in Transactions on Machine Learning Research (02/2023)
A
Ablation Study: Gradient vs Random Policy for Weight and Neuron Selection
This Appendix discusses the effect of gradient-based weights and neuron selection in NeuroF'S by performing an ablation study. We use random growth instead of the gradient to measure the importance of weights and neurons. We call this method MarroFS/w/oGrrddde
The settings this experiment is similar to Section
4.2 The results are presented in Figure
In Figure NeuroFS outperforms NeuroFS[Jw /o/////ien/rr in most cases. While the results of these methods are relatively close some datasets, the Coil-20, SMK, GLA-BRA-180, and Arcene datasets, there is large gap between the results. It can be concluded that NeuroFS performs more stable than NeuroFS[[w: ////addieen While random growth of weights and neurons might lead to better results in some cases, it can not ensure a stable performance across different datasets.
NeuroFS
NeuroFS(wioGradien)
100
100
95
95
Accuracy (%)
90
Accuracy (%)
90
85
85
80
80
75
75
25
50
75
100
150
200
25
50
75
100
150
200
K
K
COIL-20
Fashion-MNIST
BASEHOCK
Prostate_Co
Arcene
SMK
BLA-BRA-180
Figure 7: Gradient (left) vs. random (right) weight and neuron growth policy comparison.
B Comparison with bsssedd Feature Selection Methods
In this section, we compare NeuroFS with two HSIC-based feature selection methods. We select two algorithms including, HSICLassc R8 Yamada et al. (204 and HSICLasso V Koyama et al. (2022) and used the default hyperparameters used the corresponding code repositorics. The results are presented in Table 5 As can be seen in this table, NeuroFS oulperforms these methods in seven cases while performing very close to the best performer in the other cases (less than 1% difference in accuracy).
Table 5: Supervised feature selection comparison (average classification accuracy for K values in [25, 50, 75, 100, 150, 200] (%)) with HSICLasso- Laase--based methods. Empty entries show that the corresponding experiments ran into error. Bold and italic fonts indicate the best and second-best. performer, respectively.
Low-dimensional Dutascts Pashism--NNNST
High-dineensional Datascts Arcene
Method
COIL-20
MNIST
USPS
Isnlet:
HAR
RASEHOCK
Prostate GR
SMK GLAA-BRA-110
Bascline
100.0
07.35 52
$8.3
07.58
96.03
96.05
91.98
$0.96
77.5
85.84
72.22
NouroFS
88 225-0.85
85.48 # 0.34
$5.03 0.15
96.68
4: 0.16
93.22 + 0.11
92.74 è 0.23
90.42 + 0.80
89. 70 0. 72
78.00 # 1.78
82.861 98
$0.161 0.99
IISCCasso
99.32-2 + 0.00
29. 60 00
08.75200.00
96.03 4 0.00
02.0200000
53
03.00800000
58.625.0 03..2250000
00.50 0.00
77.585
0.00
70.00
0.00
80.55 2 0.00
92. 33 1 0.00
75.72 0.00
96.1550.60
75.13
0.00
60.10
1: 0.00
63.35 2 0.00
C Supervised Feature Selection Comparison using Different Classifiers for Evaluation
To show that the evaluation results are not. biased by the chosen classifier, we measure the classification accuracy by using two other widely-used classifiers including KNN TO and ExtraTreed IT The classification
httpe://gitbul htt.a//b..oooatttcoa///sseeeelle
http:///sl........///////////aaa ensembla. hta.aa/aaa.......o.ooo.eoeeeeear
18

---

Published in Transactions on Machine Learning Research (02/2023)
Table 6: Supervised feature selection comparison (classification accuracy for K = 50 (%)) using different classifiers. Empty entries show that the corresponding experiments exceeded the time limit (12 hours). Bold and italic fonts indicate the best and second-best, performer, respectively.
Low-ditetasional Dutuscts
High-dinensionul Datusets
Method
CO11.20
Pashion-MNIST
Tenlet
BASRHOCK
GR
GLA-BRA-110
BVM
Baseiine NeuroFS S
180.0
07:00
B8-33
U7:
95.00
91. 11
40-90
17.5
96.84
72.22
54
05.30
0.41
3:
0.04
96.78
02.82
0.40
86
0.79
86.062..466
90.50 2: 0.00
76.501.3
81 383: 7
80.54 à: 4.96
LessoNes
07.
22
40:00.21
32.69
0.20
55.
94.90
0.22
33.74400.30
88.0920.33
71.00
20.52
74.4934.78
STG
82.35
0.62
95.82
91.22
1:23
84.78 3.56
71.00
20:25
4.05
1.04
22:22-
1.22
70.20
74.38
72:
2,80
Fishor
0.00
73.20
50
50.30 0.00
0.00
69.80
0.00
0.00
61.00
0.00
ICAP P
60
80.00
50.50
0.00
96.
75.10
55.70
0.00
57.10
0.00
70:00
73.70
0.00
RFS
98
96.
87. 02.50266.00
94.00
0.00
85.00
0.00
90.50
à 0.00
77.50
0.00
KNN
S
96.18
1:77
78.97
LessoNes
98.
22.
79.300 0.20
06.70
79.22
0.55
38.96-
0.80
74. 04.724:6633
68.90 4.07
STC
8T.16
0.64
17.05 0.48
55.
88.16
0.39
0.00
75.004-55
71.08
58.90 7.52
QS
96.
0.58
80.30
0.76
16.65
0:51
96
0.45
82.38
3.12
85.88
1.13
86.02
65.47
8.37
75.001:3.5
68.00 68.08
2.87
66. 66.701:0.00
Fluber
96
0.00
50.20
0.00
65.70
0.00
88
0.00
74.10
0.00
81.10
0.00
89.55
0.00
65.70
0.00
20.00
0.00
65.50
0.00
50.00 4: 0.00
CIFE
71 20
1:00::0000
0.00
82.00
0.00
61:60 = 0.00
30.60
0.00
44.60
0.00
71.50
0.00
05.
0.00
67.10 5 0.00
70.00
0.00
71.10
0.00
44.40 +: 0.00
ICAP
98. 00
1:002:000 0.00
83.40
0.00
59.30
0.00
94.00
0.00
58.00
0.00
02:70
2: 0.00
91.70
0.00
66
70 8: 0.00
63-
0.00
71.10
0.00
61. 10 0.00
RFS
97. 20
0.00
95. 00..0:33
10 0.00
87.20
0.00
90.30
0.00
78.70
0:00
90.50 2 0.00
85.00
0.00
Extra Trocs
Hoselne
100.00
96.9
87.
05:55
91.04
50.00
96. 01
85.71
62.5
05
00.38
N
99
01..11:0011
03.68
0.43
84:26 à 0.55
96.44
0:27
01.46
0.78
85.45
2: 1-16
89.95 1.89
90.50 2: 0.00
75.00
5.31
81.75
4.16
75.1616
LesseNes
06. 08.7662::3 0.12
22
96:00.1
35.688:0.25
94
0.26
84.94
0.62
91.12
0.30
92.08 0.36
82. 82.548 1.92
73.50
4.64
77.88
6.77
76.12 + 3.80
STG
100.00 0.00
90.38
0:42
52.05 = 0:48
94
04.32-
0.21
98.50
2.15
85.68
0.42
53.50 1.62
53.54 : 3.20
79.002-5
89
70.78
5.30
71:08 4: 2.24
QS
06: 25 0.07
91.95
0.58
81:28 0 0.54
94. 94.28
0.50
68.78
1.8
87.86
0.72
66. 0.91
77:38 2: 5.19
73.75
4.15
75.00
1:50
75.00 0.00
Fisher Jeure
06. 26 0.45
64.80
0.15
72.06 8 0.08
00.94
0.24
61.42
0.59
85.50
0.30
92. 59 0.00
90.50 + 0.00
60.00
1.58
74.22
1.95
63. 00 0.00
CIFE
74.70
0.00
67.60
0.00
68.40 = 0.00
62: 62.70-1
0.00
65.40
0.00
85.30
0.00
65.
0.00
62.00 = 0.00
50.00
:: 0.00
81.
06
60 00-40-11 0.00
ICAP
99.
87.
0.00
65.50
0.00
98.
0.00
70.60
0.00
89.25
0.00
93.
0.00
81 00 2: 0.00
80.00
0.00
78. 90
00
63.98 60.90 0.00
RPS
98.
0.00
94.
70.70:11
0.00
30
(022
66. 792.
96. 06.50.2 0:00
90.50 =: 0.00
75.00
0.00
accuracy results are presented in Table G for K= 50. As can be seen in this table, the overall performance of
all methods is consistent across different classifiers in most cases.
D Comparison to RigL
In this section, we compare NeuroFS with RigT. Evci et al. (022)) which is a DST method mainly designed for classification; it uses gradient for weight regrowth when updating the sparse connectivity in the DST framework. To adapt RigL to perform feature selection, while trying to keep a fair comparison, we take the most straightforward approach: at the end of the training process with RigL, we use neuron strength (same as in NeuroFS) on the trained network to derive the indices of the important features. We train 3-layer MLP with Rigl for 100 epochs. RigL training algorithm updates the sparse connectivity at each epoch by removing So of the weights with the lowest magnitude and adding the same mumber of connections as the dropped ones to the network, among the non-existing connections with the highest gradient magnitude. When the training is finished we select the top K features corresponding to the neurons with the highest neuron strength as the selected features. The main difference ween NeuroFS and feature selection using RigL, is the input. layer neuron removal and addition. While RigL updates only the sparse connectivity in all layers, NeuroF'S updates also the neurons in the input layer to gradually decrcase the number of active neurons (neurons with at least one non-zero connection) to be suited for feature selection. All the experimental settings are similar to NeuroFS (Section 4.2j, such as 6 = 30, Ça = 0.2, training epochs, activation functions, batch size, learning rate, and etc. We measure the performance of feature selection using RigL for several values of K E {25, 50, 75, 100, 150, 2003. The results are presented in Table and Figure
As can be seen Table 7 NeuroFS outperforms feature selection with RigT. most cases in terms of classification accuracy. On low-dimensional datasets, RigL performs closely to NeuroF'S and even outperforms it in some cases, particularly for large values of K; while for small values of K, e.g., 25 or 50, the performance gap is larger than the larger K values. On the other hand on high-dimensional datasets, NeuroFS outperforms RigL with large gap in most datascts considered except SMK where they perform very closcly. It can be concluded that the performance gap is usually high in cases where the proportion of selected features to the total number of features is low, e.g., selecting a low number of features in low-dimensional datasets and feature selection from high-dimenssonal datasets. Relatively similar behavior exists in feature selection using QuickSelection, particularly in sigh-dimensional datasets such as BASEHOCK, Prostate GE, and GLA-BRA-180 (See Table 8) The reason behind this is that when the search space becomes large (all input.
19

---

Published in Transactions on Machine Learning Research (02/2023)
NeuroFS
RigL
100
100
95-
95
Accuracy (%)
Accuracy (%6
85
85
80
80
75
75
70
25
50
100
150
200
25
100
150
200
K
COIL
Fashion MNIST
Isolet HAR
BASEHOCK Prostate, GE
Arcene
GLA BRA
MNIST
USPS
SMK
Figure NeuroF'S (left) vs. feature selection using RigL (right) comparison.
features), finding a low fraction of informative features becomes difficult for QuickSelection and RigL and the neuron strength might not be informative its own. Therefore, NeuroFS reduces the scarch space by removing uninformative features during training, thus allowing the high-magnilude weights to be assigned to limited set of the most informative features. This indicates the importance of the neuron removal scheme in NeuroFS.
Table 7: Supervised feature selection comparison (classification accuracy (%)) with RigL for K e
{25, 50, 75, 100, 150, 200). Bold fonts indicate the best performer for cach dataset.
Method
75
100
CO1L--20
Ne
0.29
0..4
NewoFFS
07
06.
0.09
MINSST
NewoPFs
Righ.
NS Righ.
0.17
0.15
07.22 + 0.12
97.48
Isclet
Negpot' Righ.
86.22 à
02.62 0.4
94.04 0.94
96. 08.00-2 0.31
05.58
06.82
2.25
0.56 02.52
96.9
JAAK
Newrok' Higl.
87.46 4:
01.40
10.15
94.18 0.29
85.02
95.14
1.47 80.46
01.82 0.3
03.34 0.47
05.08
BASEHOCK
NeurFFS Rigl.
83.86 + 5.98
89.06 + 2.46
90.64 2: 2.56
02.72 1.5
08.44 0.91
$2.52 1.18
63:38
2.85
85.86 1 3.04
55.85 9.30
60. 1.54
88.14
us 1.91
Prostate GB
NowroFS Righ.
88.58
2.35
90.5 0.0
80.54 1.02
80.54 # 1.52
89.54 1.92
90.5 0.0
78.06 6.46
70.06 7:11
79.06 =: 8.83
61.92 8: 8.15
81.02 6.53
$4.76 1.88
Arcene
N Rigl.
63.00:: 4.M0
76.5 1: 2.55
82.0 d:
89.0 1.87
80.5 4.3
$4.0 3.39
74.5 d: 4.3
77.0 a 3.39
$1.5 4.04
80.00 1: 4.07
71.5
50.0 0:002:00 4.18
SMK
NeuroFS Higl:
75.52 4: 1.68
81.58 # 1.68
$2.62 2.12
83. 16 1.27
88.68
$4.2 0.0
81.04 1.99
70.08 1 4.51
83.14 3: 3:15
84.72 4: 1.95
84.2 d 3.73
83.05 7.55
GLA-BRA-180
NeurFFS Rigl.
73.88
80.54 4.96
82.24 9.91
81.12 2.05
62.22 1.32
82.76 2.71
06.1 2: 3.22
70.54 : 4.16
72.22 4.98
73.9 3.76
70.0 ::
76.1 4.1G
E
Comparison Results
The detailed results for each K value are presented in Table
20

---

Published in Transactions on Machine Learning Research (02/2023)
Table 8: Supervised feature selection comparison (classification accuracy for various K values (%)). Empty entries show that the corresponding experiments exceeded the time limit (12 hours). Bold and italic fonts indicate the best and second-best performer, respectively.
(a)
K = 25
Low-dinensional Inshion-MNNSST
Datasets USPS
Hligh-dinensional Datasets
Metbod
COIL-20
MNIST
Isolet
HAR
BASEHOCK Prostate.
GE
Arcene
SMK
GLA--BAA-180
Baselan
100.0
66.3
97.58
96.05
91.96
60.84
72:22
N
95.861
87.86
1.77
0.90
59.96:0.87
-0.79
68 6.35
63.00
18
18.92411.68
75.88:33.60
Net
92.77
86
1:56
76. 08.688.0.55
94,041.
18.48-1
93.00
0:31
85.08
88 18.5823.35
09 00
2.55
76.84
5:31
76.12 1: 4.19
STG
97.02
1.41
86.33
77.04 5 0:53
94-041.9
56
77.
57.4820.880
62:36
1.30
65.72 2300
69 09:00
5.15
77.38
5.57
67.22 1.78
91.00
4.21
85.25
1.47
71.57 2 1.97
50.00
0.51
72.56
6.53
67.14
1.74
63.80 1.61
71.15 1: 12
25.
25.2528.80
76.97
1- 7.52
69.45 2: 2.75
Fister score
24.70
2 0.00
74.40
0.00
03.10 + 0.00
52.00
0.00
57.40
0.00
77.10
0.00
85. 06.509:0.000
90.50 0.00
03.00
0.00
08. 40
1.0:::0..00 0.00
08.30 = 0.00
CIFE
50:70
2: 0.00
20.90
0:00
63.30
6 0.00
50:20
0.00
56.00
0:000
50.20
0.000
76.20 8: 0.06
61.90 0:00
07 50
0.00
11.00
01.10 2: 0.00
1CAP
94.40
0.00
61.60
0:000 0.00
50.10 2 0.00
50.00
0.00
67.10
0:000
86.50
0.00
89.20 = 0.00
47.60 0.00
77.50
+ 0.00
78.90
3: 0.00
89:40 2: 0.00
RFS
55.20
5: 0.00
94.50
+ 0.00
76.50:00.00
88. 0.000
60.20 0.00
90.50 = 0.00
77.50
+ 0.00
(b) K 50
Low-dinendooal
Datasets USPS
Highhdineessoonll Datasets
Method
COIL-20
MNIST
Fachion-MNIST
Tenlet
HAR:
BASRHOCK Prodate GR Arcene
SMK GIAA-BRA-110
Baseline
100.0
97 92
88.3
97.58
95.03
55.06
91.96
50.55
77.5
$0.84
72.22
NeurSFS
98.78
1 0.28
95.30
2::30:0000
0:41
83.78 2 0.61
96-78
2 0.17
92.82
0-40
91:06 2 0.73
88. 06..6..2.44
90.50 2 0.00
76.
76.50112.35
81.5827.00
7.68
80.54 :: 4.96
Nae
97.16
1.06
94
06200.8
82. 0.2882
0.30
04.912
0.15
61.00
88.7420.39
87.18 2: 0.38
88 8..5
71 10
2.00
50
52
1: 2.80
7(-182 (.78
STG
99.32
0.40
90.22
20
0.02
82.36 2 0.52
06.624-0.8
85
85.2
2.83
91.22
1:20
85.12 2: 1.366
84.78 = 3.55
71 11.00-
2.55
50
20
+ 2.95
70.00 = 4.08
QS
96.22 :: 1.53
90 0010022
80.82 2 0.51
35.52
0.27
80.7 78
90
91.96
1.04
87 2: 1.22
76.20 = 7.53
74.38
4:80
50
90
2 2:20
72:30 =: 2,80
Vister
score
74:00
1: 0.00
09
67.80
0.00
91:00
0:00
07
40
0.00
70.80
0.00
90.20 2: 0.00
00.50
=: 0.00
67
50
0:00
73
70
0:000
65.90 1: 0.00
50.40
20.30
0.00
06.90
0.00
68.50
0.00
84.20
0.00
77.40 0.00
47.60 0.00
52
81.00
0.00
58.90 0.00
ICAP
09.80t:0.00
20.00 0.00
69.50
0.00
90.20
0.00
76.10
0.00
58.70
0.00
67.10 0.00
0.00
70.70
0.00
72.20 0.00
(c) K 75
Low-dimensionol Fushion-MNIST
Datzeetx USPS
High-dimensionll Dataseta
Metbod
COIL-20
MNIST
Isolet
HAR
BASEHOCK Prostate CE
Arecue
CLA-BRA-180
Roselime
100.0
88.3
97.58
95.05
91.98 $1.98
77.5
86.81
72:22
NeunFF
99.06 +: 0.12
96.78 0.22
85.70 + 0.28
97.061
94.04
0.34
85.16
0.79
50 60.642:8.05
60 5/
82.62
2.12
+ 3.31
LassoNes
00150.
0.13
90.35
91.00
06.6220.17
0.27
0.00
78.94
01.61::0 0.44
STG
90.30
84.142.0
1:11
1.22
4:21
1:37
Fisker
0.00
76.00
0.00
00.50 0.00
0.00
CIFE
0.00
74.30
84.80
0.00
74.70
47.60 0.00
72.50
0.00
ICAP
90.00
0.00
90. 10 0.00
67.20 5 0.00
05
03.39-1
0.00
79.70
60.20
0.00
93.50
0.00
57.10 0.00
92.50 0.00
71.10 0.00
72.20 2: 0.00
RFS
99.70 0.00
97.20
0.00
03.90020000 00
94.90 =: 0.00
86.50
0.00
90.50 = 0.00
66.095.0.00
(d) K = 100
Low.dInensional
Datasets USPS
High-dincostonal Datesets
Metbod
COIL-20
MNIST
Fashion-MINIST
Isolet
HAR
BASEHOCK Prostate. GK
Arcene
SMK
GLA-BBAA-180
Baeeline
100.0
56.3
97.58
96.03
91.58
86.54
99
0.50
38
0.17
86.64 1: 0.21
97.
06
0:31
94.18
0.21
710 1.50
69. 51
1.87
89:
16:.1.87
81.12 1 2.05
Not
50 49:30
0.00
96.64
81.08
0.18
97.
83.18
0.22
65.
65.1
00.95
1.36
90.50 2 0.00
72.00
6.30
78
1.20
79. 70.161.2.00
STG
96.38
0.35
85.30
0.58
97. 08
0.18
02.61
0.56
02.82
2: 0.74
85. 65.06
1.24
85.72 5 3.00
75.50
5.67
20.08
3.8.87
20 1: 3.07
Q5
96 2:
1.15
26.
0.200.09
35.526.0.15
97.
07.00:111
0.14
94.22
0:28
94.05 2: 0.48
89.02
1.26
78.58 2 9.82
78.32
2 1.08
84.85
2.16
73.00 1: 1.40
Fister
score
90
0.00
900.70
0.00
79:00 = 0.00
96.00 0.00
70.80
2: 0.00
53.50
0.00
99.70 0.00
90.50 2 0.00
03.00
0.00
78.00
2: 0.00
05 70 2: 0.00
CIFE
67. 70
0.00
95.10
10:10:20000 0.00
80:20 = 0.00
78.00
1 0.00
61.20
0.00
85.50
0.00
74.00
0.00
71.00
2: 0.00
65.00
0.00
51.80
0:00
55. 60
2: 0.00
ICAP
100.00
0.00
95.00 2: 0.00
77.70 = 0.60
96. 40 0.00
62.80
2: 0.00
92.10
= 0.00
94.00 2: 0.00
52.
0.00
82.50
2: 0.00
76.30
2: 0.00
69.40 0.00
RFS
100.00
:: 0.00
97.40
+ 0.00
24 24.40:20.00
95.40
+: 0.00
96.70
0.00
90.50 + 0.00
50.00
0 0.00
(e) K 150
Low-dImensioonll Fushion-MNIST
Datasets USPS
High-dinraddnal Datasets
Method
COIL-20
MNIST
Lsolet
HAR
BASEHOCK Prostate_ CE
Arcene
SMK CLA-BRA-150
Bnseline
130.0
97:592
88.3
HT. 58
95.03
95.05
91.98
50:95
T7.5
95.84
72.32
NeursFS
56 a6 0.25
97.72
2: 0.10
87.18 à:
0.16
97.48
0.04
25
5.26.29
85.02
=: 0:35
$3.44 à: 0.91
68:5
1.92
80.50
à: 4.30
83
7-02
82.22 $ 1.52
LassoNec
59.35 03.5442
2 0.20
57.42
0.07
56.02
0.15
97.48
0.07
54.96
2 0.15
95.72
0.20
32.05
0.62
90.50 2 0.00
72.00
1.87
78.08
1.04
70.1881.37
STG
100.00 0.00
07.14
1:144:2000 0.08
86.25 = 0.36
57
28
0.12
54,20
0.35
93.56
- 0.59
96.42 2 1.74
84.76
4.67
75.00
2 3.16
22.00-
4.27
72 T6
3.27
qS
07
70:66.12
36. 08.88....0
07.495-
05.48
0:32
94.50
- 0.24
99.50
0.53
79.
6.19
76.75
1.25
84.22
3.93
75.00 10:00
0.00
Fisbar
scure
81
05.10
0.00
0.00
97
65.00
0.00
84.60
0.00
0.60
90.50
0.00
75.00
0.00
0.00
71
96.80 0.00
75.60
0.00
80.60
85.70
85.00
0.00
79.20
0.00
76.20
0.00
55.00
81.00
0.00
0.00
71.10
RFS
(I) K 200
Low-dincusioual Datasets
High--dincusional Datusets
Metbod
COIL-20
MNIST
Fashion-MNIST
USPS
Isolet
HAR
BASEHOCK Prostate CE
Arceue
GLA-BRA-180
Roselime
100.0
88.3
97. 57.58
96.03
96.05
91.88
77.5
72.22
NowrFS
100.00
0.00
27.
0.7
07.0060017
07.52-b-
60.30=6.31
95.14 0:21
92.82
= 0.00
54.00+
34,30 0.00
LassoNot
0.00
0.14
07.58
U5.34
95.0820.12
98. 93.952
0.00
75.92
50.021.2.00
STG
97.
0.07
10..594-0.53
0.07
0.15
06.14
0.20
6.19
50.624:3.25
82.6815.44
4 1.40
Fister
54
0.00
97.50 1.000:0.000
50.90
85.50
0.00
0.00
78.50
0.00
C1FR
73
0.00
97.60
0.00
15.60
0.00
96.30
0.000
0.00
67.80
0.00
85.00 -: 0.00
79.20
0.00
16.20 2: 0.00
57.50
0.00
78.80
2.000
72. 72.21-12 0.00
ICAP
06.
0.00
07.60 0.00
84.50 0.00
96.
90 0.000
90.30
0.00
05.50
=2 0.00
93.77 à 0.00
62.00 2 0.00
80.00 0.00
76.30
0:000
72.20 0.00
RFS
100.00 2: 0.00
97.50
0.000
0.00
85.70
2:00
0.00
95.80
2: 0.00
80.00 0.00
90.50 2 0.00
80.00 0.00
21


Reviews:
[{"review_id": "PXm9ovsrnY", "reviewer": "Reviewer_42sy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strong aspects\n\nI could not find any strong aspects of the paper.\nCurrently, to me, the proposed method looks like a special case of RigL Evci et al. (2020) with pruning and regrowth applied only to the first layer (see Weakness 1).\n\n\n### Weak aspects\n\n#### Weakness 1: There is a critical concern regarding the novelty of the paper.\nThe idea of the proposed method is on the dynamic pruning and regrowth of the features during training.\nAs the authors mention in the paper, such an idea itself is not novel.\nFor example, Mocanu et al. (2018) considers the similar pruning criterion with random regrowth.\nRigL Evci et al. (2020) considers the similar strategy as the proposed method; the pruning based on the weight magnitudes and the regrowth based on the gradient magnitudes (not only in the input layer, but also the entire networks).\nWith these prior studies in mind, the question is what is the advantage of the proposed strategy over these approaches.\nIn the current paper, the authors merely states that \"most of these methods suffer from over-parameterization, which leads\nto high computational costs, particularly on high-dimensional datasets\" without clear evidence (see Weakness 2 below).\nThere is no discussion on why the proposed method is more advantageous over the prior methods, e.g., why and when the proposed pruning/regrowth strategies can be considered better than the previous ones.\nCurrently, to me, the proposed method looks like a special case of RigL Evci et al. (2020) with pruning and regrowth applied only to the first layer.\n\n#### Weakness 2: There is a critical concern regarding the novelty of the paper.\nThe authors claimed that \"most of these methods suffer from over-parameterization, which leads to high computational costs, particularly on high-dimensional datasets\" based on their experimental results.\nThe concern here is on the tuning of these prior methods.\nAccording to Section 4.1, the authors said \"we used the implementation provided by the authors\" without specifying how their hyperparameters are selected.\nHere, I wonder whether the authors used the implementations with their default hyperparameters, the ones far from optimal.\nIf this is the case, all the claims in the paper based on the experiments can be meaningless.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a feature selection method for sparse three layer perceptrons.\nThe proposed method consists of successive pruning of the input features as well as regrowth.\nFor pruning, the authors proposed using the magnitude of the connection weights related to each of active features.\nFor regrowth, the authors proposed using the magnitude of the gradient related to each of inactive features.\nThe proposed method then combines these pruning are regrowth steps with carefully tuned scheduling so that the number of total active features to decrease during the training.\nIn the experiments, the authors showed that the proposed method outperformed some of the existing feature selection methods for neural networks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "#### Request 1.\nPlease describe the difference and the advantage of the proposed method from the prior methods that also use pruning and regrowth.\nWhy and when the proposed pruning/regrowth strategies are considered better than the previous ones?\n\n#### Request 2.\nPlease describe the choice of hyperparameters used in the baseline methods in the experiments.\nI wonder whether the authors used the implementations with their default hyperparameters, the ones far from optimal.", "broader_impact_concerns": "There is no ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DsHNFOyYPB", "reviewer": "Reviewer_6znL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n- The topic is obviously important for a wide range of the community.  \n\n- The intuition behind the proposed method is easy to understand and simple to implement.\n\n- The paper is well-organized and easy to follow.\n\nWeaknesses\n\n- Both criteria for the magnitude based pruning of weights and the gradient based activation are quite common. The scheduling strategy that consists of the removal and growth phases can be seen as an original contribution, but the basic strategy is somewhat straightforward. Since the scheduling procedure are heuristically designed, it is not fully motivated why the proposed specific way of the scheduling procedure stably provides good performance.\n\n- Filter based compared methods are somewhat old such as RFS (2010), Fisher score (2011), ICAP (2005) and CIFE (2006). The authors said their main focus is in embed approaches, and so, I guess the authors selected classical methods for filter approaches. However, more modern approaches should have been included for clarifying advantage of the proposed framework as far as filter methods are also used as baselines. For example, well known HSIC based non-linear feature selection strategies can be important baselines, though it is not even mentioned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a neural based feature selection method. The selection is performed during training. At each neural network training epoch, neurons and connections are pruned and activated based on magnitude of weight and gradient values. A scheduling strategy for determining how many neurons and connections are removed/added are proposed, which consists of removal phase (in which # of active neurons decreases) and update phase (in which # of active neurons is fixed). The authors empirically shows the proposed method can maintain high prediction performance even when only a small number of features are used. The performance is compared with several neural based feature selection methods and classical filter methods. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical\n\n- The authors define that the proposed method is an 'embed' approach of the feature selection. On the other hand, experiments only shows performance of SVM with the selected features. Although the authors mention that a non-neural model is intentionally selected for a fair evaluation, this evaluation metric confuses me. Obviously, each embed method optimizes a set of features so that a criterion derived from a specific prediction model. For me, it is quite unclear how it can be justified that only using SVM for evaluating features selected based on neural models. It should be clarified why the SVM based evaluation is valid for the embed feature selection evaluation in more detail, or change the evaluation strategy (such as using original models of each method, or adding prediction models other than SVM to avoid that the result and discussion are unconsiously biased to SVM).\n\n- In introduction, the following two statement are somewhat exaggerated.\n  + 'we propose for the first time to use the sparse neural networks to perform supervised feature selection'\n  + As a main contribution, 'introduce dynamic neuron pruning and regrowing in the input layer of sparse neural networks during training'.\n\n  Hefler et al., JMLR2021 the authors cited mentioned 'most neuron sparcification schemes can be used for feature selection' (sec2.5), by which I think claiming novelty on applying the neuron sparcification to the supervised feature selection scenario itself is not convincing. \n\n- Is using LassoNet with 1-layer MLP while the proposed method employs a 3-layer MLP fair comparison? \n\n- What criterion is used to determine hyper-parameters of networks? \n\n- The settings of SVM in evaluation metric should have been clarified (kernel, hyper-parameters, and so on).\n\n- In the 'Implementation' paragraph, what does 'similar hyperpameters' mean? Describe precisely.\n\nNot Critical:\n\n- The average accuracy plot of Figure 2 is too small.", "broader_impact_concerns": "I have no concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B8Ux1zGC8J", "reviewer": "Reviewer_Cn4v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The problem is well-studied but still important for the community. The English level is satisfactory, and overall I enjoyed reading the paper. The problem statement and solutions are clear. The evaluation seems clear, and they further provide an evaluation of the sensitivity of the approach to hyperparameters and stability to different initializations. Related work is mostly well-written. The method is simple and leads to performance improvements while reducing flops.\n\nWeakness: The method is embedded but is evaluated as a wrapper. Namely, the standard way to evaluate an embedded method is to use the prediction learned by the classifier/regressor that is learned in an end-to-end fashion. Instead, the authors train an additional classifier after their model selects features. While I understand the intuition behind this procedure, it is not the correct procedure to evaluate the capabilities of an embedded approach. Since they already put substantial effort into this procedure, I would recommend adding at least one or two evaluations in this suggested setting and comparing their model to other embedded approaches like LASSO, LASSO net, and STG… Furthermore, they haven’t presented any example that demonstrates that the model can accurately recover informative features in a noisy environment. This could be done with synthetic data or semi-synthetic like MADELLON or GISSETTE.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address a challenging and important problem of feature selection. The new approach is an embedded-type method that relies on ideas from sparse neural networks to train the model for prediction and FS. The main advantage of the approach is that it does not require training a dense network but rather sparsifies and regrows neurons to learn the prediction model while simultaneously identifying the best subset of features. They demonstrate the effectiveness of the approach using 11 real-world datasets. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here, I point out some suggestions to improve the paper:\n\nThe second sentence in the abstract is very long; try to rephrase it.\n\nUnimportant-> this is a weird term; maybe you mean uninformative?\nIn the intro, you claim embedded methods are more efficient than other types, but then you evaluate the method as a wrapper. Furthermore, some filter methods are highly efficient, scaling linearly with the data, so I recommend relaxing your statement.\n\nP2 “the proposed method, QuikSelection” this is confusing, quickselection is not proposed here\n\nEquation 1, k should be capitalized\n\nFor unsupervised FS methods, it is worthwhile to cite\n[1] Abid et al.  Concrete Autoencoders for Differentiable Feature Selection and Reconstruction\n[2] Shaham et al. Deep Unsupervised Feature Selection by Discarding Nuisance and Correlated Features\n\nSection 3.2, please add more details about the figure. Also, please move it to the same/adjacent page.\nSection 3.2 more detail-? Details\n \nHyperparameters- Your use a 3-layer sparse MLP with 1000 neurons. Is this the architecture for all NN baselines? This is not clear\n\nIt is not clear how the parameters are optimized for all methods, do you have a validation set?\n\nIn table 2, what is the number of features selected by each method? This is crucial for understanding each method. If this is an average overall value of K it is really not informative. \n\nYou train SVM; what type? Kernel SVM? How is it tuned?\n\nP12, STG, and LassoNet are not really dense NNs, since they sparsify the first layer during training; the number of flops they use varies and needs to be evaluated empirically by running the method on each data. This means that your statement is misleading, and the results of STG and LASSONet should be evaluated and added to table 4 as another column.\n \n", "broader_impact_concerns": "No ethical concerns ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a neural-based feature selection method called NeuroFS, where feature selection for NN is an important machine learning research topic. More specifically,  NeuroFS gradually prunes the uninformative features from the input layer of a sparse neural network trained from scratch. 

The critical concern for this paper is the difference from the RigL (Evci 2020). As pointed out by Reviewer 42sy, "the proposed method looks like a special case of RigL Evci et al. (2020) with pruning and regrowth applied only to the first layer." Then, the authors respond to the question as "While Mocanu et al. (2018) and Evci et al. (2020) focus on the typical classification task, our work focuses on the feature selection task." I agree that Evci et al. (2020) focus on training sparse neural networks. However, RigL can be potentially used for feature selection, as written in its appendix Fig 7 (https://arxiv.org/pdf/1911.11134.pdf). Then, the important point of this paper is that NeuroFS can select a certain number of features, while RigL may not select. I agree that this is an important property for feature selection.

However, it remains a crucial question; how feature selection technique is significant over RigL + simple heuristic-based feature selection algorithm? For example, after we train RigL, we can select features by ranking feature importance (magnitude of features in the first layer) and selecting top-K features. 

Thus, I request the authors to the comparison with RigL for the camera-ready version. Adding RigL for all the experiments is preferable. The similar experiments in the section "C Comparison with HSICLasso-based Feature Selection Methods" is acceptable.
Decision: Accept (poster)