SAND: One-Shot Feature Selection with Additive Noise Distortion
Authors: Pedram Pad
Hadi Hammoud
Mohamad Dia
Nadim Maamari
Liza Andrea Dunbar
Venue: ICML 2025 poster (2025)
Keywords: feature selection
machine learning
neural networks
Primary area: deep_learning->algorithms
TL;DR: Novel and efficient end-to-end feature selection for NN. Simple implementation, no alteration of loss function or model architecture, full control over the number of features without tuning or search, outperforms SOTA, theoretical insights provided.

Abstract: Feature selection is a critical step in data-driven applications, reducing input dimensionality to enhance learning accuracy, computational efficiency, and interpretability. Existing state-of-the-art methods often require post-selection retraining and extensive hyperparameter tuning, complicating their adoption. We introduce a novel, non-intrusive feature selection layer that, given a target feature count $k$, automatically identifies and selects the $k$ most informative features during neural network training. Our method is uniquely simple, requiring no alterations to the loss function, network architecture, or post-selection retraining. The layer is mathematically elegant and can be fully described by:
\begin{align}
\nonumber
\tilde{x}_i = a_i x_i + (1-a_i)z_i
\end{align}
where $x_i$ is the input feature, $\tilde{x}_i$ the output, $z_i$ a Gaussian noise, and $a_i$ trainable gain such that $\sum_i{a_i^2}=k$.
This formulation induces an automatic clustering effect, driving $k$ of the $a_i$ gains to $1$ (selecting informative features) and the rest to $0$ (discarding redundant ones) via weighted noise distortion and gain normalization. Despite its extreme simplicity, our method achieves competitive performance on standard benchmark datasets and a novel real-world dataset, often matching or exceeding existing approaches without requiring hyperparameter search for $k$ or retraining. Theoretical analysis in the context of linear regression further validates its efficacy. Our work demonstrates that simplicity and performance are not mutually exclusive, offering a powerful yet straightforward tool for feature selection in machine learning.

SAND: One-Shot Feature Selection with Additive Noise Distortion
Pedram Pad Hadi Hammoud 12
Mohamad Dia
Nadim Maamari
L. Andrea Dunbar 12
Abstract
1. Introduction
Feature selection is fundamental problem in high- dimensional statistics and machine learning (Guyon & Elis- seeff, 2003; Li et al., 2017). Unlike feature extraction tech- niques that alter features' semantics by creating new ones in a lower dimensional space, feature selection involves the identification and retention of the most informative fea- tures while discarding irrelevant or redundant ones. This preservation enhances the interpretability and explainability of predictive models, particularly critical in domains like medicine and biology where gene selection is focal ap- plication (Guyon et al., 2002). By retaining the original features, researchers can directly relate model outputs to the underlying data, facilitating insights and hypothesis gener- ation. Furthermore, feature selection not only contributes to storage reduction by eliminating unnecessary data points, optimizing memory usage, and enhancing computational efficiency, but also aids in reducing model size and com- plexity. By selecting a subset of input features, models can improve performance and generalization capabilities crucial for mitigating overfitting and addressing the curse of dimen- sionality. Moreover, in applications where sensing hardware costs energy consumption are major concerns, such as in IoT devices sensor-based systems, feature selection can inform the design of simpler and more cost-effective hardware by ensuring that only relevant features are sensed or measured, thereby conserving resources without compro- mising performance.
Feature selection is critical step in data-driven applications, reducing input dimensionality to enhance learning accuracy, computational effi- ciency, and interpretability.. Existing state-of-the- art methods often require post-selection retrain- ing and extensive hyperparameter tuning, com- plicating their adoption. We introduce a novel, non-intrusive feature selection layer that, given target feature count k, automatically identifies and selects the k most informative features during neural network training. Our method is uniquely simple, requiring no alterations to the loss func- tion, network architecture, or post-selection re- training. The layer is mathematically elegant and can be fully described by:
T: 0,24 + (1 - (1-a:)z:
where 2; is the input feature, ã; the output, z; a Gaussian noise, and a, trainable gain such that a = k. This formulation induces an auto- matic clustering effect, driving is of the a; gains to 1 (selecting informative features) and the rest 0 (discarding redundant ones) via weighted noise distortion and gain normalization. Despite its ex- treme simplicity, our method achieves competitive performance on standard benchmark datasets and a novel real-world dataset, often matching or ex- ceeding existing approaches without requiring hy- perparameter search for k or retraining. Theoreti- cal analysis in the context of linear regression fur- ther validates its efficacy. Our work demonstrates that simplicity and performance are not mutually exclusive, offering a powerful yet straightforward tool for feature selection in machine learning.
Feature selection methods can be broadly categorized into two main groups: unsupervised and supervised. Unsuper- vised methods often involve analysis of the relations between input features through methods like clustering (He et al., 2005), matrix factorization (Wang et al., 2015), and the use of autoencoder neural networks (Balm et al., 2019). These methods are useful when labeled data searce or unavailable, allowing for the exploration of inherent data structures and patterns. On the other hand, supervised meth- ods leverage the availability of labeled data to guide the selection process. Within the realm of these methods, there exist model-independent and model-dependent approaches. Model indeeeenddene methods, also known as based, rely on statistical tests and cetrmaation-thhoreee metrics to evaluate feature relevance with respect to the target vari- able, irrespective of the underlying machine learning model
'CSEM, Neuchâtel, Switzerland ?EPFL, Lausanne, Switzer- land. Correspondence to: Pedram Pad <pedram.pan @add@csoo @csem.ch>>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
(Yang & Pedersen, 1997: Chandrashekar & Sahin, 2014). While these methods are computationally efficient and can handle high-dimensional data, they may overlook complex interactions between features. Model-dependent methods, on the other hand, tailor feature selection to specific ma- chine learning models or architectures. This category can be further divided into wrapper and embedded methods. Wrap- per methods (Kohavi & John, 1997) involve search process guided by the final performance of learning model, such as classifier accuracy. Examples include greedy sequential feature selection (Das & Kempe, 201 1), SHAP (SHapley Additive ex exPanatiions) values calculation (Lundberg & Lee, 2017), in addition to combinatorial optimization and meta- heuristic search algorithms (Zadeh et al., 2017; Dokeroglu et al., 2022). Wrapper methods offer the advantage of considering feature interactions but may suffer from high computational costs due to intensive search over the input space, which is highly impractical for complex models and large feature dimensions. In contrast, embedded methods rank features based on metrics intrinsically learned during model training. seamlessly integrating feature selection into the learning process. Examples include feature importance for tree-based algorithms (Breiman, 2001), Recursive Fea- ture Elimination for Support Vector Machine (RFE-SVM) (Guyon et al., 2002), sparsity-promotinn models (Tibshirani, 1996), and deep learning techniques (Simonyan et al., 2014; Wang et al., 2014). Such methods enable an automatic selec- tion of relevant features during training and can effectively handle non-trivial relationships in data.
selection. Alternatively, impose sparsity and overcome the limitations of applying gradient descent & regular- ized objective functions, (Yamada et al., 2020) introduces continuous relaxation of Bermoulli gates that are attached to the input features. A Gaussian-based regularization is then added to the objective function and grid-search over the regularization parameter is applied to select the required number of features. While (Yanada et al., 2020) employs a similar stochastic approach as ours, it modifies the loss fune- tion by adding an additional term and lacks direct control over the number of selected features, requiring a grid search over a model hyperparameter. In contrast, our stochastic framework does not alter or rely on the loss function, elim- inating the need for retraining or hyperparameter tuning. Additionally, our method allows direct specification of the desired number of features as an input. Lately, the attention mechanism is being employed to relate trainable softmax mask to feature importance, and hence perform embedded feature selection by adaptively estimating marginal feature gains over multiple rounds (Yasuda et al., 2023).
Contributions
The existing methods mentioned above typically necessitate alterations to the objective function or significant modifica- tions to the neural network architecture involving the addi- tion of new connections. Consequently, feature selection often a separate phase followed by a retraining phase on the selected features, it requires some kind of hyperparameter tuning to control the number of selected features (Yamada et al., 2020; Lemhadri et al., 2021; Yasuda et al., 2023). In this work, we propose a novel, yet exceptionally simple, method for one-shot feature selection. It involves the in- tegration of a simple constrained weighted additive noise layer at the neural network's 's input. The constrained stochas- ticity helps the network generate polarized input space and effectively select the desired number of features during training. As a result, the network architecture inherently converges to its final form, which can be used for inference without necessitating any additional retraining. The con- straint on the weights is imposed by construction through a normalization operation and requires no regularization terms in the objective function. The proposed layer imposes negligible computational overhead and can be seamlessly incorporated, akin to the addition of Dropout or Batch Nor- malization layers. Through this layer, direct control over the number of selected features is enabled without the need for additional grid search or further tuning of regularization terms. The simplicity of our method does not compromise the final prediction performance of the neural network. In this work, conduct extensive benchmarking study against state-of-the-arr feature selection methods using com- mon datasets and a novel real-world dataset, showcasing our method's s effective competition against existing approaches
Related works
Given the pervasive adoption of deep learning in recent years, this work concentrates on embedded feature selection techniques tailored to neural networks. Within this domain, multitude of approaches have emerged, predominantly centered around various adaptations of LASSO-basee regu- larization (Luo & Chen, 2014; Zhao et al., 2015; Li et al., 2016; Lemhadri et al., 2021; Cancela et al., 2023), the addi- tion of stochastic gates (Srinivas et al., 2017; Borisov et al., 2019; Yamada et al., 2020), the use of attention mechanisms (Liao et al., 2021: Yasuda et al., 2023), and the applica- tion of saliency maps (Cancela al., 2020) to solve the feature selection problem non-linear models. For in- stance, Sequential LASSO (Luo & Chen, 2014) provides an efficient implementation of greedy LASSO to recursively select input features, while Group LASSO (Zhao et al., 2015; Scardapane et al., 2017) further modifies the objec- tive function to encourage sparsity at the group level. In LassoNet (Lemhadri et al., 2021), skip linear connection is added to the neural network with two types of regularization parameters. A continuous search is then applied using hierarchical proximity algorithm, which combines a prox- imal gradient descent method with a hierarchical feature
2

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
while highlighting its practicality and application-driven nature. Furthermore, we provide theoretical insights by demonstrating that our method, when applied to linear re- gression, promotes the selection of a predefined number of features on an equivalent problem.
addition of n extra parameters (a," s) to optimize. As training progresses, observe that k of the a 's cluster around 1, indicating the selected features, while the remaining at's as's cluster around O, indicating the neglected features. We refer to this approach as SAND,, which stands for Selection with Additive Noise Distortion.
2. Selection with Additive Noise Distortion (SAND)
Remark 2.1. The two operations of the SAND layer in (1) and (2) are implemented together. This means the constraint (2) is enforced by construction where we normalize the a 's by their la -nrm inside the layer, without adding any regularization term to the loss function. Hence, the SAND layer takes this form in practice:
In a typical supervised learning problem, we are tasked to map a set of input vectors to predefined outputs. These input vectors consist of various features. However, not all features are equally important in determining the output. Some may be irrelevant, while others might contain redundant infor- mation. This leads us to the concept of feature selection: the quest to identify the subset of features that provide suf- ficient information to determine the output accurately. In real-world applications, the number of features to select is typically pre-defined due to constraints on data acquisition burden, computation cost, or memory footprint.
a Ialla k O (1-
a
k= |alla
O z.
(3)
Notice that if there is no noise z (i.e., o = 0), the az's would be absorbed in the weights of the first layer of the neural network. Additionally, if = n, all a, can become 1 and then E will be identical to E without any noise. Given that the noise independent of the data and lacks information about the output, the network naturally adjusts to mitigate its impact during training.
Consider an n-dimensional feature vector (1), ,,, =..) (which can be of any shape; but for the sake of simplicity in notations, we assume it to be n x 1) to be mapped to the output vector y'. Figure ((a) depicts a typical neural network solution to this problem. Now, assume we are interested in finding the & dimensions that yield the highest performance, with k n.
The first non-trivial property is that there is always an op- timal a whose entries are between 0 and 1. Here, optimal means with respect to the loss function of the network. To prove that, assume there is an optimal a that has an aj < O. By replacing aj by -aj. while the constraint (2) still holds, ăy is a less noisy version of æz. On the other hand, if there is an optimal a that has an aj > 1, which results in ăJ becoming noiseless copy of æj. and increase other a,'s 's that are less than 1 towards 1 to satisfy the condition (2); it decreases the noise added to those fea- tures as well. Therefore, we can confine the search space of a to the vectors that have all entries between and 1, i.e.
Our idea is to multiply each feature x; with gain a, and add zero-mean Gaussian noise with the standard deviation of (1 - a,) ) to it before feeding it to the neural network. Here, a is a fixed scalar. Moreover, we constrain the vector a = -a11 .2.............................. an) to have the -o-norm equal to k for a pre-selected a > Thus, we define
decrease a to 1,
Σ = ==0@z T
+ (1 - Oz
(1)
a (a1, a2 .e....,.))
an) that have
where
0 H a 1 for i i 1....,,77 n.
(4)
La l k.
(2)
Now, we analyze what is happening during the training. In- tuitively, a more informative feature Iy will get gain aj closer to 1 so that it will be passed to the neural network with less noise. Due to the constraint (2), this automatically yields smaller gains ay for other features which are less informative. Consequently, the less informative features become noisier, which makes them even less informative and pushes them to get even smaller gains. This leaves more room, due (2), for the more informative features to get their gains closer to 1 and become less noisy. This reinforcing loop, summarized in Figure 2, results in polar- ization of the gains around 1 and 0. Ideally, we will end up with a vector a which has k entries equal to 1, indicating the selected features, and the rest equal to 0, indicating the neglected features. However, since in practice the smallest
We then feed ä to the neural network during the training phase instead of æ as illustrated in Figure 1(b)). Here, z is a Gaussian vector with i.i.d. entries with zero-mean and standard deviation o. In this setting, when a, is close to 1, ã; is close to noiseless I,, and when o is close to O, ã; becomes almost pure noise (the signal-to-noise ratio is proportional to (1=a:)=0=). During the training phase, we allow the a, 's to be trained alongside the other parameters of the network. The architecture of the neural network and the loss function remain unchanged; the only difference is the
Throughout the paper, we use small characters to denote scalars, underlined characters indicate vectors, capital characters for matrices and bold font for random objects
3

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
SAND
ox)
(1 og) x2
Netuaal Network
Neural Network W
11 - 0x) Rq.
W. argmin (w/ (z))
2 1a
g. W argmin &
(2 W (z.1)
(a) Vanilla Neural Network
(b) Neural Network with the SAND layer
Figure 1. Neural network architecture and the loss function before and after adding the SAND layer. Here, L and W indicate the loss function and the trainable parameters of the neural network respectively.
values have not necessarily converged to absolute zero, at the end of the training phase, we keep the top l gains intact and manually set the n & smallest gains to O, effectively removing those features. Notably, features with small gains noisy and hence inherently ignored by other parts of the neural network.
where Wj is the 22-norm of the zth column of W. Define the matrix W to be the matrix W that its 1th column is multiplied by ag for = I,..., n. Rewriting (9), we obtain
C (a, W.
(9)
2
= Ez {11-- Wx - -1112] +
Linear Regression
Here, we mathematically show that in the case of linear regression, adding the SAND layer introduced above is equivalent to adding term in the loss function that promotes selection of k features.
where w; is the norm of the 1th column of W. Using the Lagrange multiplier method for constrained optimization, we obtain
Da, Ca. Caa, W.b) = à
|
(10)
In linear regression problem, the loss function
da,
L (W, b) = Eg..y
Wx b1l2]
(5)
where À is scalar and the right side of the equation is from the constraint in (7). Thus, we have
where denotes the expected value, is the coefficient matrix and b is the bias vector. Thus, optimal solution is
1 (11) aj
2 a3
> sgn (a;)
la,1a-1 laz)
(11)
W b" = argmin C(W,)).
(6)
We
which leads to
Now, we add the SAND layer in the beginning. i.e.,
2 |a,1 (1 az). (12)
-11
E -=cOx+ Oz such that |g] k. (7)
We get
By summing over j and incorporating the constraint (7), get
L Ca, W.
-Ezy
y yy w [
A
-
Ag a (1
az)
Ezyz
y W(c02+(1-0) b1l2]
aj
2
A
n
Exy
W( -1 Σ 3 (1 0₂)² 202 2
a
h: Σ
aj
la
(13)
2
j=1

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
The more informative Seatures seck higher gains best noise
Higher contrast the information content
Due the gain normelization:
lall
Noise further supprosecs the formattin content in the lees informative festures
The less informativv features receive goins
Initialization:
Noise addition mechanismm: a G z + (1- 19
Inderuaiis
Z
Figure Reinforeing loop that results polarization of the gains with n = and l:
Combining (13) and (10), we obtain
3. Experiments
(a. W)
(14)
Feature Selection for Neural Networks
A EEz 11y-Wz Wz [1b]] -ak
We explored the performance of SAND through experiments standard benchmark datasets used for feature selection in neural networks. Specifically, we utilized nine datasets, seven of which were used in previous studies by (Balin et al., 2019; Lemhadri et al., 2021; Yamada et al., 2020; Yasuda et al., 2023). The additional real and synthetic datasets were California Housing (Torgo, 1997) and HAR70 (Logacjov & Ustad, 2023). California (CA) Housing is real dataset con- sisting of 20640 samples with 8 interpretable features, with the task of regressing the price of the house with the least features. Additionally, HAR70 is a large synthetic dataset comprising 2.3 million samples. It includes informative features, which we augment with 100 nuisance features sam- pled from A' 0.1), and the task involves classifying the activity being performed. The testing metric for all datasets was accuracy, except for the CA Housing dataset, where the mean absolute error (MAF) was used. The datasets were normalized to have zero mean and unit standard deviation for each feature. Furthermore, we evaluated SAND on a novel multi-spectral image dataset, introduced in this paper for the first time, which has never been used before for fea- ture selection. Details of this dataset and our evaluation provided in Appendix C.
at la
2
Remember that we can confine the search space to the a; 's between 0 and 1. Hence, according to (12), we have À 2 0. Therefore, the term at the end of (14) achieves its minima when there are k of a is equal to 1 and & of them equal to 0, which completes the proof.
Remark 2.2. There are three hyper parameters in the SAND layer, k and
1 is the number of features to be selected. It will initially straightforwardly.
a indicates how firmly we would like to restrict the number of features to A higher value of sigma places greater emphasis on precisely achieving & fea- tures, resulting in faster binarization (polarization to- ward 0 and 1) of the gains (a, (s).
a indicates which norm to be used normalize the gain vector during training.
We will see in the experiments that the method is not sensi- tive the choice of and In fact, setting within the range of standard deviation of the input features, and = 2, yields nearly optimal results across all datasets. Thus, there no need to tee-tune a and a. Moreover, to ensure stable training, we apply clipping to the a, values throughout the training process, keeping them within the range [0, 1).
We implemented a neural network with one hidden layer and a ReLU activation. Given the variation in hidden layer widths across cited works, we opted for a width equal to n/3, where n represents the dimensionality of the input data. Please refer to Table of Appendix for comprehensive overview of the nine datasets, the corresponding number of
5

---

of the model intact. Hence, from this point of view. our method offers two key benefits. Firstly, it demands fewer epochs (33% fewer epochs in our experiments). Secondly, SAND: One-Shot Feature Selection with Additive Noise Distortion
NAST
538
6.64
6397
5A4
COI-90
1.00
0.11
1:5 100
033
0354
0.97
Madeton
46070
HoHodin
8.50
0.43
5600
Figure 3. Test metries datasets 10 trials. The metric is accuracy (f) for all except MAE (4) for CA Housing being regression problem. SA = Sequential Attention, LLY = Batch-wise Attenuation, GL = Group LASSO, SL Sequential LASSO, and STG Stochastic Gates.
epochs and batch size used for training, along with the mean accuracy/aboolute-erro of the model with all features.
Our evaluation included a comparison between SAND and five established feature selection algorithms, namely Se- quential Attention (SA) (Yasuda al., 2023), Sequential LASSO (SL) (Luo & Chen, 2014), Btach-wiss Attenuation (LLY) (Liao et al., 2021), Group LASSO (GL) (Zhao et al., 2015), and Stochastic Gates (STG) (Yamada et al., 2020).² For all methods except ours, training comprised a feature se- lection phase followed by fitting phase wherein the neural network was retrained on the selected features. Although STG often eliminates the need for retraining thanks weight polarization, the weights fail to polarize when k 60, even after exhaustive search over the regularization parameter which indirectly governs the number of selected features, and so we resort to retraining in some cases. As for SAND, the fitting phase was omitted and the weights learned during the selection phase were directly utilized for inference. In other words, the gains corresponding to the non-selected features where set to zero while keeping all other weights
provides a streamlined pipeline where both selection and inference are handled by the same model.
Across all experiments, with learning rate of 10-3, and we partitioned the datasets into 70- (0-10-20 splits for training, validation, and testing, respectively. For hyperparameters of the SAND layer, we used o 1.5 and a = 2 consistently. Unless otherwise specified, we selected = 60 features for all datasets by default, except for the following: k for the Madelon dataset, to = 3 for the CA Housing dataset, and k = 6 for the Har70 dataset. The comparative results are summarized in Figure 3, and the exact numerical values are shared in Table of Appendix B. The error bars were calculated using the standard deviation over 10 trials. Drawing from the results in Figure SAND competes effectively with other feature selection methods, while offering a streamlined pipeline that requires fewer iterations. also exhibits very consistent behavior as shown in Table of Appendix B.
employed the Adam optimizer
some datasets (e.g. MICE Protein)
"The code reproduce experiments available https: //github. com/ csem/ SAND
To provide additional insights, we varied the number of selected features k under consistent settings and evaluated performance on the test set. Results are shown in Figure 4, Notably, SAND demonstrates its strength in feature selec- tion, showcasing results that outperform comparable
'These algorithms represent the top-performing methods re- ported in the literature. A comparison with other approaches (Atashgahi et al., 2022: Lemhadri al.. 2021; Sokar al., 2022) demonstrates that SAND achieves better performance.
6

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
MINIST
Fashion
0.330
3:22
012
0.04
115
182
076
118
ISOLET
COIL-20
Activity
014
135
0.4
032
1.1
078
20
30
MADELON
HAR70
CA Housing
3.8
0.5
SA
LLY
GL
SEQL
SAND
Figure 4. Test metrics for different k's. Accuracy (t) for all except MAE (L) for CA Housing.
to other methods across different feature counts. This ad- vantage is particularly significant given the fact that the best method is changing from dataset to dataset. Thus, there is need for algorithms that deliver value beyond marginal accu- racy improvements, prioritizing enhancements in computa- tional demand and simplicity-qualln emplifies. It important to note that all SAND experiments were conducted using fixed values of a 1.5 and a = 2, without any fine-tuning. We omitted STG from Figure due to the high computational cost of performing exhaus- tive search over the regularization parameter À for each k. Moreover, on the MICE Protein dataset (77 features), STG fails to identify a clear feature subset once k exceeds 37. For k < 37, STG produces a polarized solution solution-exactly & features attain importance weight at almost 1, while the re- mainder are almost zero. However, for > 37, the selection process stalls: irrespective of X, only the same 37 features remain at weight I, and the other 40 features converge to an almost uniform, nonzero weight. Adjusting À cannot induce the selection of additional features; it only uniformly increases the weights of these 40 unselected features above zero.
We recorded the gains every 10 epochs. Figure 5 presents the sorted gains for selected epochs. We observe that while the gains tend to cluster around 1 and 0 in both plots, this clustering occurs at higher rate for larger a.. Moreover, to assess SAND's ' sensitivity to we replicated the initial experiment while varying a E {1.0, 2.0, 3.0). Re- sults are presented in Table 1. As evident in the table, our approach demonstrates high robustness to the selection of a, which underscores a positive aspect of the proposed method.
We
that our method ex-
Effect of the choice of a in E.-Norralization
To have an insight of the effect of a, we conducted a du- plicate experiment, this time employing a 1.0 (with E {0.5, 1.5)). The outcomes are presented in Table 2. As shown in the table, the performance remains highly consis- tent, despite variations in a.
4. Summary and Future Works
In this paper, we introduced novel feature selection method. Specifically, we presented new layer (SAND) that inte- grates into neural network, enabling automatic feature selection during the training phase. The benefits of this approach include:
Role of a
As discussed in Remark 2.2, the parameter a influences the rate of gain polarization. To demonstrate this, we trained SAND model on MICE dataset for 2000 epochs, using o values of 1.0 and 2.0, while keeping other settings fixed.
On par with the sfate-of-the-ar performance: Through extensive experiments, we showed that the proposed method has effectively ofte-oo-thh-ar performance.

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
1.0
==1
0
Gain of SAND layer
430
02
100
0.0
20
40
60
10
20
30
40
50
60
70
Input features sorted by gain
Input features sorted by gain
Figure 5, Polarization of the feature gains in SAND layer for le 30.
Table Test metrics for feature selection with SAND using different
Dataset
o 1.0
σ = 1.5
2.0
2.5
d 3.0
Mice Protein
0.988 = 0.006
0.988 + 0.006
0.988 + 0.007
0.988 = 0.006
0.987 + 0.006
MNIST
0.953 = 0.007
0.962 + 0.002
0.958
0.002
0.953
0.002
0.948 + 0.004
MNIST-Fashion
0.830 + 0.007
0.832 + 0.007
0.833 + 0.003
0.831 = 0.004
0.825 1 0.003
ISOLET
0.913 = 0.009
0.926 + 0.005
0.922
0.007
0.921 0.006
0.916 0.006
COIL-20
0.991 = 0.005
0.998 + 0.003
0.998 + 0.002
0.998 = 0.003
0.996 1 0.004
Activity
0.929 = 0.006
0.923 + 0.004
0.922 0.006
0.924 0.004
0.922 0.005
Madelon
0.741 = 0.022
0.732 + 0.021
0.708 + 0.023
0.689 = 0.017
0.646 + 0.025
HAR70
0.901 = 0.002
0.897 + 0.005
0.889 + 0.002
0.881 0.005
0.873 + 0.006
CA Housing 0.514 = 0.026
0.521 + 0.027
0.532 + 0.030
0.557 = 0.022
0.566 + 0.007
Low computational and memory burden: The layer introduces only n trainable parameters, along with n multiplication-addel addiiiits and a single n--iimensional fz-normalization, where n is the number of features.
network, preserving the original architecture and loss function. With only input gradients required to train the layer gains, the network architecture or loss function can be treated as a black box.
One-shot feature selection and network training: There is no need for selecting the features in one phase of the training and then retrain the network with the selected features. Once the training phase has finished, the features are selected and the neural network is trained for the selected features.
Tailored features to the application and the neural net- work architecture: Since SAND layer is an integral component the base model, the features selected are automatically adapted for the specific application hand and the chosen model architecture.
Remarkably simple both conceptually and practically: the mathematical model of our method involves only entrywise multiplication, addition with Gaussian noise, followed by f2-norm normalization, rendering it re- markably simple in theory and in practice.
Control on the number of selected features: The num- ber of features can be directly set in the algorithm in contrast to the main stream methods which require sweeping over hyper parameter to be able to obtain the desired number of features.
It worth mentioning that the proposed SAND layer works in very similar way to the Dropout layer but with op- posing effect. In the Dropout layer, randomization leads to an even distribution of information across all neurons. Con- versely, randomization in SAND, due to weights' constraint, selects only neurons with the highest information content.
Considerably faster: ing phase, and due to the low computational overload, the method is considerably faster than the competitors.
As there is no need for the retrain-
Handy Integration of Feature Selection in Neural Net- works: Our feature selection method seamlessly inte- grates as an additional layer the outset of the neural
A straightforward continuation of this work is explore
8

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
SAND layer's performance for network pruning by incor- porating it into intermediate layers, akin to how Dropout and Batch Normalization layers are utilized. Additionally, studying the effect of different noise distributions and rig- orous understanding of the effect of a and a for different network architectures (dense, convolutional, transformers, etc.) are interesting lines for future researches. Furthermore, considering features relation structure during the selection is another valuable avenue explore.
International Conference on Artificial Neural Networks, 2019.
Breiman, Random forests. Machine Learning. 45(1): 5-32, 2001.
Cancela, B., Bolón-Canedo, V., Alonso-Betanzos, A., and Gama, J. A scalable saliency- baasdd feature selection method with instance-level information. Knowledge- Based Systems, 192:105326, 2020.
Cancela, B., Bolón-Canedo, V., and Alonso-Betanza A. E2e-fs: An end-to-end feature selection method for neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8311-8322 2023.
Table 2. Effect of a on SAND layer.
Dataset
a = 1.0
a= 2.0
0.5
1.5
Mice Protcin
0.987 + 0.005 0.988 + 0.007 0.988 + 0.006
Chandrashekar, G. and Sahin, F. A survey feature selec- tion methods. Computers & Electrical Engineering, 40 (1):16-22 2014.
MNIST
0.959 + 0.002 0.956 + 0.002 0.962 + 0.002
MNIST-Fashion
0.828 + 0.007 0.830 0,006 0.832 + 0.007
ISOLET
0.922 + 0.008 0.910 0.926 + 0.005
COIL-20
0.997 + 0.003 0.996 + 0.005 0.998 + 0.003
Activity
0.922 + 0.004 0.907 + 0.008 0.923 + 0.004
Das, A. and Kempe, D. Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, pp. 1057-1064, 2011.
Madelon
0.756 + 0.039 0.675 + 0.029 0.732 + 0.021
HAR70
0.854 + 0.036 0.814 + 0.038 0.897 + 0.005
CA Housing
0.538 0.063 0.560 0.023 0.521 + 0.027
Acknowledgements
Dokeroglu, T., Deniz, A., and Kiziloz, H. E. A comprehen- sive survey on recent metaheuristics for feature selection. Neurocomputing, 494:269-290 2022.
The present work developed within the AGRARSENSE Project that has received Chips JU funding (Grant Agree- ment No. 101095835). It has received funding from the Swiss State Secretariat for Education, Research and Inno- vation (SERI) and is co-funded by the Innosuisse Swiss Innovation Agency. More information: info@agrarsense.co
Dunbar, A. L.., Blanc, S., Pad, P., Beuchat, P.-A., Papathana- siou, A., Nikitakis, A., and Makantasis, K. A fast simple- to-use and inexpensive multispectral camera to detect skin conditions (Conference Presentation). In Photonic Instrumentation Engineering VII, volume 11287. SPIE, 2020.
Impact Statement
Dupont, M. F., Elbourne, A., Cozzolino, D., Chapman, J., Truong, V. K., Crawford, R. J., and Latham, K. Chemo- metrics for environmental monitoring: review. Analyti- cal Methods, 12(38):4597-4620, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
ElMasry, G., Mandour, N., Al-Rejaie, S., Belin, E., and Rousseau, D. Recent applications of multispectral imag- ing in seed phenotyping and quality monitoring-an overview. Sensors, 19(5), 2019.
References
Atashgahi, Z., Sokar, G., van der Lee, T., Mocanu, E., Mo- canu, D., Veldhuis, R., and Pechenizkiy, M. Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. Machine Learning, 111: 377-414, 2022.
Guyon, I. and Elisseeff, A. An introduction to variable and feature selection. J. Mach. Learn. Res., 3(null); 1157-1182, mar 2003. ISSN 1532-4435.
Guyon, I., Weston, J., Barnhill, S., and Vapnik, V. Gene selection for cancer classification using support vector machines. Mach. Learn., 46(1-3):389 mar 2002. ISSN 0885-6125.
Balın M. F., Abid, A., and Zou, J. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference on machine learning, pp. 444- 453, PMLR, 2019.
He, X., Cai, D.,, and Niyogi, P. Laplacian score for feature selection. In Advances in Neural Information Processing Systems, volume 18. MIT Press, 2005.
Borisov, V., Haug. J.. and Kasneci, G. Cancelout: A layer for feature selection in deep neural networks. In 28th

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
Kohavi, R. and John, G. H. Wrappers for feature subset selection. Artif. Intell., 97:273- 324, 1997.
Srinivas, S., Subramanya, A., and Babu, R. V. Training sparse neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 455 455-462, 2017.
Lemhadri, I., Ruan, F., and Tibshirani, R. Lassonet: Neural networks with feature sparsity. In International Confer- ence on Artificial Intelligence and Statistics, pp. 10-18. PMLR, 2021.
Tibshirani, R. Regression shrinkage and selection via the lasso. Jourmal of the Royal Statistical Society. Series (Methodological), 58(1):667-288 1996.
Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., and Liu, H. Feature selection: A data perspective. ACM Comput. Surv, 50(6), dec 2017. ISSN 0360-0300.
Torgo, I. California housing dataset, 1997. URL
https://wwtt dcc. fc. wtt..f.......ttttorree.
Regression/ ca www.lhhouiing.taa. html. Accessed: 2023-10-10.
Li, Y., Chen, C.-Y., and Wasserman, W. W. Deep feature selection: Theory and application to identify enhancers and promoters. Journal of Computational Biology, 23(5): 322-336, 2016.
Wang, Q. Zhang, J., Song, S., and Zhang. Z. Attentional neural network: Feature selection using cognitive feed- back. In Neural Information Processing Systems, 2014.
Liao, Y., Latty, R., and Yang, B. Feature selection using batch-wise attenuation and feature mask normalization. In 2021 International Joint Conference on Neural Net- works (IJCNN), pp. 1-9. IEEE, 2021.
Wang, S., Pedrycz, W., Zhu, Q., and Zhu, W. Subspace learning for unsupervised feature selection via matrix factorization. Pattern Recognition, 48(1):10-19, 2015.
Yamada, Y., Lindenbaum, O., Negahban, S., and Kluger, Y. Feature selection using stochastic gates. In Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pp. 10648-10659) PMLR, 13-18 Jul 2020.
Logacjov, A. and Ustad, A. Har70+ [dataset]. UCI Machine Learning Repository, 2023. URL https::: : /doi. org/ 10.
Accessed: 2023-10-10.
Lundberg. S. M. and Lee, S.-I. A unified approach to in- terpreting model predictions. Proceedings of the 31 International Conference on Neural Information Process- ing Systems, NIPS' 17, pp. 4768-4487468 11668477,,22 2017.
Yang, Y. and Pedersen, J. O. A comparative study on fea- ture selection in text categorization. In Proceedings of the Fourteenth International Conference on Machine Learn- ing. ICML '97, pp. 412-420. San Francisco, CA, USA, 1997. Morgan Kaufmann Publishers Inc.
Luo, S. and Chen, Z. Sequential Lasso cum EBIC for fea- ture selection with ultra-high dimensional feature space. Journal of the American Statistical Association, 109(507); 1229-1240, 2014.
Yasuda, T., Bateni, M., Chen, L., Fahrbach, M., Fu, G., and Mirrokni, V. Sequential attention for feature selection. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Ortega, S., Halicek, M., Fabelo, H., Callico, G. M., and Fei, B. Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review. Biomed. Opt. Express, 11(6):3195-3233 Jun 2020.
Zadeh, S., Ghadiri, M., Mirrokni, V., and Zadimoghaddam, M. Scalable feature selection via distributed diversity maximization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Scardapane, S., Comminiello, D., Hussain, A., and Uncini, A. Group sparse regularization for deep neural networks. Neurocomputing, 241:81-89, 2017.
Zhao, L., Hu, Q., and Wang, W. Heterogeneous feature selection with multi-modal deep neural networks and sparse group lasso. IEEE Transactions on Multimedia, 1964191996-1948, 2015.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations (ICLR), 2014.
Sokar, G., Atashgahi, Z., Pechenizkiy, M., and Mocanu, D. C. Where to pay attention in sparse training for feature selection? In Advances in Neural Information Processing Systems, 2022.
niques and Applications. CRC Press, 2024. Spigulis, J. Aitra-Narrowband Multispectral Imaging: Tech-
10

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
A. Experimental set-up
We provide Table containing details about all datasets utilized the feature selection experiments. Additionally, the table includes the epochs employed during training for each dataset, identifying the ones used for feature selection and the ones used retrain/fit the model on the selected features. As indicated the experiments (Section 3), fitting epochs are only utilized by models other than SAND, whereas SAND employs only the 'Select Epochs', The table also presents the test accuracy of the base model trained using all features.
Table 3. Dataset characteristies, experiment parameters, and all features performance metrics."
Dataset
(n, d)
# Classes
Select Epochs
Fit Epochs
Batch Size
All Features
Mice Protein
(1,080, 77)
400
200
64
0.987 + 0.006
MNIST
(70,000, 784)
10
100
50
64
0.978 + 0.001
MNIST-Fashion
(70,000, 784)
10
200
100
64
0.878 + 0.003
ISOLET
(7,797, 617)
26
400
200
64
0.958 + 0.002
COII--20
(1,440, 400)
1000
500
0.996 0.003
Activity
(10,299, 561)
200
100
64
0.941 + 0.002
Madelon
(2,600, 500)
500
250
64
0.575 + 0.017
HAR70
(2,259,597, 106)
6
3
64
0.890 +: 0.002
CA Housing
(20,640, 8)
N/A
200
100
64
0.440 +: 0.011
Moreover, the experiments were executed on a machine equipped with an NVIDIA GeForce RTX 4090 GPU with 24GB of RAM, paired with an AMD Ryzen 9 5900X 12-Core Processor featuring 24 threads, The code to reproduce our experiments available at https: pettps://githun /ttth/ggggtt..tt/////g hwtmm//ce///e/eeee SAND.
B. Experimental results
We present in Table 4 the benchmarking results of SAND alongside other methods on the nine datasets discussed in the experiments (Section 3). The intervals were calculated using the standard deviation across 10 trials.
Table 4. Test metrics over 10 trials (mean + standard deviation): Accuracy (f) for all except MAE (4) for CA Housing.
Dataset
SA
L.Y
GL.
SI.
STG
SAND
Mice Protein
0.986 + 0.005
0.983 + 0.011
0.988 + 0.007
0.987 + 0.007
0.988 0.007
0.988 + 0.006
MNIST
0.961 + 0.002
0.949 1 0.003
0.930 + 0.006
0.963 + 0.002
0.962 = 0.002
0.962 + 0.002
MNIST-Fashion
0.836 + 0.003
0.830 + 0.004
0.828 + 0.004
0.829 + 0.003
0.856 + 0.002
0.836 + 0.005
ISOLET
0.927 + 0.004
0.908 = 0.011
0.924 0.003
0.924 = 0.007
0.936 = 0.005
0.926 # 0.005
COIL-20
0.996 = 0.005
0.998 + 0.003
+ 0 0.003
0.996 = 0.003
0.997 = 0.004
0.998 = 0.003
Activity
0.920 + 0.009
0.894 + 0.050
0.930 0.004
0.915 + 0.005
0.922 + 0.006
0.923 + 0.004
Madelon
0.786 + 0.008
0.862 0.027
0.612 0.010
0.603 0.011
0.793 0.03
0.732 0.021
HAR70
0.901 + 0.001
0.910 + 0.001
0.910 + 0.003
0.908 + 0.004
0.897 + 0.002
0.897 + 0.005
CA Housing
0.639 + 0.063
0.636 + 0.063
0.590 + 0.097
0.589 + 0.097
0.577 = 0.119
0.521 + 0.027
Table 5 presents the consistency analysis for the HAR70 dataset, which comprises informative features augmented with 100 synthetically generated noisy ones. This dataset was specifically chosen because SAND demonstrates its lowest relative performance here compared other methods, shown in Table (with SAND's classification accuracy being 0.013% lower than the best-performing method, LLY). Nevertheless. SAND exhibits high consistency, consistently selecting 5 out of the 6 informative features and misidentifying the remaining feature only 20% of the time. Compared to SA and SL, SAND is more reliable in selecting the informative features. However, its slightly lower performance can be attributed to the inherent randomness of the experiments and the fact that other methods benefit from retraining after feature selection, unlike SAND, which operates without this additional training phase.
'The metric is classification accuracy (f) for all except MAE (4) for CA Housing. Our study utilizes the entire MNIST and Fashion datasets, unlike related works. Additionally, the Activity dataset sourced from (Lemmharri et al., 2021)'s Google Drive and (Yasuda al., 2023)'s repository contains 10,299 samples, as opposed to the 5,744 samples reported in the referenced papers.

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
Table Selection consistency on HAR70: feature selection frequency of useful features (0 5) and misselected ones over 10
runs for k = 6.
Feature
SA
SL
SAND
STG
GL
LLY
10
10
10
10
10
10
8
9
8
10
10
10
10
10
10
10
10
7
10
10
10
10
2
10
10
10
10
10
Misselected
10
4
2
2
0
C. Benchmarking on real-world multi-spectral image dataset
In this section, introduce novel real-world multi-specttal image dataset, where feature selection is crucial for developing efficient and cost-effective acquisition hardware. We apply SAND to this dataset and demonstrate that it outperforms state-of-the-ar methods.
Multi-spectral imaging a powerful technology that captures images across multiple wavelengths, unlike conventional RGB imaging, which may fail reveal critical spectral information. This approach has shown significant promise in various applications, including cancer detection in medical imaging (Ortega al., 2020), precision agriculture (EIMasry al., 2019). and chemometries (Dupont et al., 2020). There multiple ways to acquire multi-spectral data, including snapshot cameras, push-broom scanners, and filter wheel systems (Spigulis, 2024). However, more cost-effective and flexible approach involves actively controlling the illumination. This method uses monochromatic camera while sequentially activating lights at different wavelengths, enabling multi-spectral acquisition without requiring specialized optical components (Dunbar et al., 2020).
Figure 6. Demo of real-time segmentation on the MSI Grain dataset using multi-spectral imaging.
The "MSI Grain" dataset was acquired using active LED light control across 15 narrow spectral bands ranging from 360 nm to 940 nm. It contains spectral data for four distinct classes: Lentils, Wood, Stones, and Background. The goal is to leverage multi-spectrall imaging to classify these visually similar classes, which are difficult to distinguish with the naked eye or conventional imaging. The dataset consists of 50,000 spectra per class, extracted from several multi-spectral image cubes (see Figure 6). Each spectrum corresponds to 15-dimensional pixel within the image, representing its spectral signature
12

---

SAND: One-Shot Feature Selection with Additive Noise Distortion
the measured wavelengths. The dataset intentionally includes only spectral information, excluding spatial context. This design choice ensures that classification relies solely on the material's spectral response rather than its shape. The dataset is available in the code repository at https: httptttttbbccccoo com/ csem/ SAND.
A key challenge in multi-spectral imaging is reducing the number of spectral bands, or features. to optimize hardware efficiency. Fewer bands help minimize heat dissipation, lower hardware costs for commercial adoption, and enable fast real-time image acquisition. As a result, feature selection not just a theoretical challenge but a practical necessity in this domain. SAND was developed to address this real-world need by jointly optimizing machine learning- maaning---sedddd spectral detection and hardware design. Conventional feature selection methods often fall short in terms of performance and consistency, while state-of of-he-of approaches tend be complex and less user-friendly. In contrast, SAND provides practical, application-driven solution inspired by these challenges, reinforcing its effectiveness in real-world scenarios.
We present in Table 6 the benchmarking results of SAND alongside other methods on the MSI Grain dataset. The target number of selected features is k 9 and the experiments were run across 5 trials. All other experimental set-ups and pre-processing steps are the same as discussed Section 3. Along with remarkable simplicity compared to other methods, SAND achieves the highest performance, as demonstrated below.
Table 6. Test accuracy over 5 trials MSI Grain (mean + standard deviation).
Method
SA
LLY
GL
SL
STG
SAND 0.919
Accuracy
0.917 + 0.003
0.917
0.001
0.916 0.002
0.918 + 0.002
0.911
0.005
0.002
All Features Accuracy
0.924 + 0.001
It is important to note that the segmentation in Figure 6 occurs in real-time, with the trained neural network running directly on the handheld edge device capturing the image. No cloud computing is involved the tablet is used solely for visualization. Real-time edge processing was made possible by feature selection, which reduced the number of relevant spectral bands to nine, significantly decreasing both the model size and the number of spectral acquisitions. This optimization enabled the algorithm to be deployed on the edge device and operate in real-time.
13


Reviews:
[{"review_id": "NCqdecnE5A", "reviewer": "Reviewer_gNnB", "summary": "The key contribution of this paper is to introduce a method for feature selection during training of a neural network. A key feature of this method is to be able to do the feature selection in a very simple way, without substantive modifications to the architecture itself, while still being able to retain good performance. The main idea is to have a trainable noise that can be added to the feature, which forces the features to be clustered at the end of the training process.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful reading of our paper and your kind compliments. Regarding your question about learnable k, for the moment what comes to our mind is the simple idea to sweep over k and see where the loss exhibits a drastic change.", "claims_and_evidence": "I would say so, it feels like the paper is well-written and thought through. The examples chosen are perhaps a bit small-scale but still provide an illustration of the efficacy of the method.", "methods_and_evaluation_criteria": "The methods and the selected data sets used in the evaluation are appropriate.", "theoretical_claims": "I did. The authors presented a nice illustration with linear regression showing how their proposed method effectively promotes the selection of six features.", "experimental_designs_or_analyses": "I did. I don't think with the experiments as presented there are any issues.", "relation_to_broader_scientific_literature": "I think in the crowded and extensively studied problem of feature selection, the authors managed, to the best of my knowledge, make a nice contribution to the literature.", "essential_references_not_discussed": "Not to the best of my knowledge.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Is it possible to make k a learnable parameter, somehow?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "uWcbn0PJ0p", "reviewer": "Reviewer_PP3t", "summary": "This paper proposes SAND (Stochastic Additive Noise Decoupling), a method for feature selection that avoids adding any explicit loss or regularization term. Instead, it uses a simple noise-injection mechanism where each input feature is blended with noise according to a learnable gating parameter $a$. The gating vector is constrained via an $\\ell_\\alpha$-norm, and features with higher $a_i$ values are preserved with less noise. The method is evaluated across several datasets and shows performance that is comparable to existing feature selection approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful reading of the paper and the valuable comments.\n\nConcerning the first question, as mentioned by the reviewer, mathematically, the optimum a_i is guaranteed to be between 0 and 1. In the code, to be on the safe side, we always clip the values between 0 and 1 at each iteration when the vector a_i is normalized. Now that the issue is cleverly raised as a concern by the reviewer, we will add it to the text of the paper as well.\n\nConcerning the second question, we tried to compare our method with other state-of-the-art methods, some of which were proposed very recently. Furthermore, a quick comparison was already done with other methods, including stochastic gate and LassoNet, as mentioned in footnote 3 (page 6, lines 326-328). However, this comparison was not quantitatively elaborated in the current version of the paper. We agree that a direct quantitative comparison with the stochastic gates method might be useful, given the similarity with SAND (note that SAND is much simpler with no need for loss function alteration or hyperparameter search for k as in stochastic gates ). For this end, we have now included the stochastic gates (STG) in our previously shared anonymized code repo (https://anonymous.4open.science/r/SAND-6BB1), which allows us to compare it with SAND and the 4 other methods used in our paper for benchmarking. Furthermore, the readme file of the repo now has a table comparing SAND and STG on 5 classification datasets and one regression dataset. SAND outperforms STG on 3 classification datasets and on the regression dataset (lower MAE is better). Note that for the 2 datasets on which SAND is worse, SAND was already not presented as the top performer (in terms of accuracy) in the current version of the paper.\nThese 6 datasets were chosen for their moderate size (less than 70k samples) and for fast experimentation in this rebuttal. This is because STG needs a computationally costly fine-grained tuning to select the specified number of parameters. In the final version of the paper, we can include the comparison with STG on all 9 datasets.\n \n\nFor the third question, a quick robustness/consistency experiment was done in Table 5 of Appendix B (supplementary material). The experiment was purposely done on one dataset with known correct features and on which SAND performs the worst compared to other methods (as explained in the appendix). Nevertheless, SAND shows a high consistency in choosing the correct features: SAND has better consistency and more accurate feature selection than the two methods that seem to outperform it in terms of classification accuracy (note that SAND is trained with 33% less number of epochs compared to other methods since it does not require post-selection training; that might be the reason it appears to undeperform in some scenarios as explained in the appendix). Such robustness analysis can indeed be extended to other datasets (even if the correct features are not known a priori) to track and confirm the robustness/consistency of SAND. We can run the experiment more thoroughly and add it to the supplementary material (or the paper, should we have enough space).\n\nThanks for the productive suggestions. We hope these clarifications prove useful, and please feel free to reach out during the rebuttal phase if further discussion is needed; we kindly ask that you reconsider your evaluation in light of these explanations.", "claims_and_evidence": "Most claims are supported by experiments and theoretical reasoning. However, an important gap is the lack of discussion around the training behavior of $a_i$ values. While the paper shows that optimal solutions lie in $[0,1]$, there are no guarantees or constraints during training that enforce this. Since the model is trained with gradient-based optimization, it's unclear how $a_i$ values are kept within $[0,1]$ in practice. This should be clarified, especially since negative or $>1$ values could affect the noise interpolation behavior.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criterion are appropriate. The datasets and architectures used are standard, and the comparisons to existing feature selection methods are relevant.", "theoretical_claims": "I have verified the derivation provided for the linear regression case, where the authors show that adding the SAND layer is equivalent to introducing a term in the loss function that promotes the selection of $k$ features. The steps and conclusions appear to be correct and are consistent with the behavior expected from the formulation.", "experimental_designs_or_analyses": "The experiments are straightforward and use commonly adopted architectures and datasets.", "relation_to_broader_scientific_literature": "The work is related to prior methods in differentiable feature selection, such as stochastic gates and other gating mechanisms that involve regularization. The key difference is that SAND avoids adding terms to the loss and instead relies on noise injection and norm constraints, making it a simpler alternative.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n 1. The approach is simple and can be easily integrated into standard training pipelines.\n 2. Avoids the need for tuning additional loss weights or regularization terms.\n\nWeaknesses:\n1. It is unclear how the model ensures that $a_i \\in [0,1]$ during training. The paper relies on theoretical reasoning that such a solution exists, but does not explain how it's maintained during optimization. Since gradient updates can move parameters outside this range, it’s important to address this empirically or with a constraint mechanism.\n2. Empirical comparison to stochastic gates would make the evaluation more complete, as they also perform embedded feature selection using a gating mechanism, but through loss regularization. This would help clarify how SAND compares in practice and in formulation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Are the $a_i$ values explicitly constrained or reparameterized during training to ensure they stay within $[0,1]$? If not, how is it ensured that values do not become negative or exceed 1 during optimization?\n2. How does SAND compare to stochastic gates in practice, especially since both aim to select features using differentiable mechanisms? Including this comparison would help contextualize the method.\n3. An analysis of the stability or sparsity of the learned $a$ vectors across different runs or initializations could inform the robustness of the approach. Including such an analysis would enhance the completeness of the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EtRzXEczIr", "reviewer": "Reviewer_UP58", "summary": "The paper introduces SAND (Selection with Additive Noise Distortion), a novel feature selection layer for neural networks that automatically selects $k$ informative features during training. SAND operates by multiplying each input feature by a trainable gain $a_i$ while adding Gaussian noise weighted by $1-a_i$. Such a design encourages the informative $k$ gains to 1 (selecting features) and the rest to 0 (discarding features) without altering the loss function or network architecture. Unlike traditional methods, SAND requires no post-selection retraining or extensive hyperparameter tuning. Experimental results on nine benchmark datasets (e.g., MNIST, ISOLET) as well as a new real-world multi-spectral imaging dataset (MSI Grain) demonstrate that SAND matches or exceeds state-of-the-art methods in accuracy and efficiency. A theoretical analysis in linear regression further validates its approach, positioning SAND as a simple yet powerful tool for feature selection.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful reading of the paper and the valuable comments.\n\nWe agree that finding out the required number of features for a desired performance is a very valuable question to answer, but it is outside the scope of this work. By the sentence “SAND directly controls the number of selected features without hyperparameter tuning”, we mean that once the number of desired features k is given, the algorithm directly gives a solution for that many features. This is in contrast with other approaches where, even when k is given, there is another hyper parameter by which we can push the algorithm to have more or less number of features, and there is no direct control on the number of features to select in one shot. In LASSO-based approaches, stochastic gates, and many other methods, there is no direct control over the number of features. There, we can only achieve the exact (predefined) number of desired features by sweeping over the other hyper parameter (which is usually a regularization term in the loss). In short, for a given k (whether defined by the hardware constraints as explained in the dataset of Appendix C, or by model complexity/size constraint,...), SAND finds/selects the k features in one shot without a sweeping or line search, in contrast with many other methods. The fact that k is a predefined number in the context considered here is mentioned in the abstract, but this might get confusing later on in the paper. We will make sure to clarify this confusion in the final version of the paper by stressing that this is the standard feature selection problem where k is given and we are after selecting the k features (not identifying k itself).\n\nWe acknowledge that expanding the range of datasets and experimental scenarios can further strengthen the evaluation of a newly proposed method. In our work, we deliberately selected datasets that are widely recognized as standard benchmarks for feature selection, ensuring a fair and consistent basis for comparison within a reasonable computational timeframe. Moreover, we compared our approach against state-of-the-art methods, including those acclaimed in the literature, such as the recent work by Google on sequential attention feature selection. In addition to these benchmarks, we introduced a novel real-world dataset (detailed in Appendix C) to demonstrate the practical applicability of our method. We believe that the breadth of datasets and competitive methods included in our study represents one of the most comprehensive evaluations in the current literature. This selection was carefully designed to balance thoroughness with computational feasibility while ensuring a robust comparison with leading approaches.\n\nWe appreciate your feedback and hope that the clarifications provided have been helpful. Should you need any further details during the rebuttal phase, please feel free to reach out, and we kindly invite you to reconsider our work in light of these explanations.", "claims_and_evidence": "This paper has several claims. While majority of them are well-supported. The claim that *SAND directly controls the number of selected features without hyperparameter tuning* is bit overclaimed. SAND requires to explicitly define the number of selected features $k$, which is usually missing in real-world dataset. Other approaches from feature interpretation domains can only explicitly define the number $k$ to conduct feature selection.", "methods_and_evaluation_criteria": "Both the methods and evaluation criteria are generally reasonable.", "theoretical_claims": "I quickly check its correctness, but I am not an expert in theory.", "experimental_designs_or_analyses": "The dataset selection is a bit questionable. The only large-scale dataset is Har70, which has a number of well-known informative features. Based on the hyper-parameter selection in Section 3, such information helps authors better determine $k$. Evaluating SAND on large-scale datasets without a number of informative features pre-known, such as Criteo or ImageNet (could be compressed), can better showcase SAND's ability in real-world scenarios. The paper also misses baselines from discrete optimization domains, which can naturally be adapted for feature selection.", "relation_to_broader_scientific_literature": "One-shot feature selection without post-selection retraining is a very interesting topic that can be applicable to the real world. The reviewer finds this point being the most important contribution of this paper.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "Pros:\n1. This work tackles an interesting and applicable problem.\n2. The writing of this paper is generally easy to follow.\n\nCons:\n1. The experimental setup could be questionable, which limits its contribution and applicability.\n2. Certain claims are a bit over-claimed and not unique.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aVB3Ii4iL2", "reviewer": "Reviewer_x6bu", "summary": "This paper proposes SAND, a new feature selection method which modifies the original input to a linear combination of additive zero-mean Gaussian noises and the original input. The weight of linear combination $a\\in \\mathbb{R}^d$ measures the feature importance and the constraint of the number of selected features is imposed in a soft way by requiring the $\\alpha$-norm of $a$ equaling $k$. By doing so, there is no need to add regularization term to the original loss function, which can be optimized using popular gradient descent methods like Adam. In the inference stage, only features within the top-$k$ weights $a$ are retained and used as the inputs to the trained model. Experiments on synthetic and real datasets demonstrate the effectiveness of the proposed method compared to existing methods including sequential attention, batch-wise attention, sequential Lasso and group Lasso.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful reading of the paper and your valuable feedback.\n\nRegarding being SOTA, our work emphasizes that although no single method consistently outperforms all others across every dataset, our approach is notably simpler and competitive—especially when accuracy/MAE metrics are saturated (see Table 4 in Appendix B). We believe that a method that is the best in a significant number of cases and nearly optimal in the remaining ones merits the designation of state-of-the-art—an acknowledgment that can apply to multiple methods when performance differences are marginal (SAND performs better or is comparable with other methods although it is extremely simpler conceptually and considerably faster in practice: no tuning/line-search to get to k features, no alteration of loss function, no post-selectin training). Nonetheless, we recognize that the wording may have been misleading, and we will revise it in the final version of the paper to better reflect these nuances.\n\nFor small k, no state-of-the-art method consistently outperforms the rest. Specifically, SAND excels on three datasets for small k’s; for example, on the CA Housing dataset (regression, lower MAE is better), SAND shows superior performance for small k. Moreover, SAND is trained with about 33% fewer epochs than competitors, as it requires no post-selection training. This efficiency may explain its underperformance in some cases, as detailed in Appendix B (e.g., the analysis on Har70, chosen for its known correct features and relatively lower SAND performance). Although SAND’s classification accuracy on Har70 is 0.013% lower than the best (LLY), it achieves better feature selection accuracy (correct features out of 6) and greater stability than the two methods (SA and SL) that slightly outperform it in classification\n\nThank you for the suggested references. We can add them to the final version of the paper.\n\nConcerning the proof in linear regression, since $a_i$ s are between 0 and 1,  for each $i$, $a_i |a_i|^\\alpha <= |a_i|^\\alpha$ with equality holding only for $a_i=0$ or $a_i=1$. Thus, we have $\\sum_i{ a_i |a_i|^\\alpha } <= \\sum_i{|a_i|^\\alpha } = k$ with euqlity holds only if all $a_i$ s are either 0 or 1. Since in the first two terms of Eq. (14), there’s no a_i, and \\lambda is positive, the whole expression will be minimized when $\\sum_i{ a_i |a_i|^\\alpha }$ gets its maximum value (k) and this happens when all of $a_i$ s are either 0 or 1 . Since we have the constraint $\\sum_i{|a_i|^\\alpha } = k$, we conclude that exactly k of $a_i$ s are 1 and (n-k) of $a_i$ s are 0 . We can add this explanation to the final version of the paper and improve the writing, while respecting the page limit.\n\nConcerning the comparison with STG: we tried to compare our method with other state-of-the-art methods, some of which were proposed very recently. Furthermore, a quick comparison was already done with other methods, including STG and LassoNet, as mentioned in footnote 3 (page 6, lines 326-328). However, this comparison was not quantitatively elaborated in the current version of the paper. We agree that a direct quantitative comparison with STG method is useful, given the similarity with SAND (SAND is much simpler with no need for loss function alteration or hyperparameter search for k as in SGT). For this end, we have now included the STG in our previously shared anonymized code repo (https://anonymous.4open.science/r/SAND-6BB1), which allows us to compare it with SAND and the 4 other methods used in our paper for benchmarking. Furthermore, the readme file of the repo now has a table comparing SAND and STG on 5 classification datasets and one regression dataset. SAND outperforms STG on 3 classification datasets and on the regression dataset (lower MAE is better). Note that for the 2 datasets on which SAND is worse, SAND was already not presented as the top performer (in terms of accuracy) in the current version of the paper.\nThese 6 datasets were chosen for their moderate size (less than 70k samples) and for fast experimentation in this rebuttal. This is because STG needs a computationally costly fine-grained tuning to select the specified number of parameters. In the final version of the paper, we can include the comparison with STG on all 9 datasets.\n\nWe kindly ask the reviewer to consider these clarifications and to reach out during the discussion phase if further explanation is needed. We reiterate the potential of this simple, practical, and efficient method, which may impact areas beyond feature selection (e.g., NN pruning). Our goal is not to introduce a model that dramatically outperforms all baselines but to present a lightweight model that at least matches baseline performance while offering distinct advantages: simplicity, low computational and memory burden, one-shot feature selection and network training, control over the number of selected features, and easy integration with neural networks.", "claims_and_evidence": "The claim that SAND achieves SOTA performance on benchmark datasets is not well supported. SAND performs worse than other methods especially in the case with small $k$ in experiments.", "methods_and_evaluation_criteria": "The proposed method is well motivated and the evaluation criteria makes sense. However, SAND is very similar to STG [34] since both of them introduces randomness in training, but the comparison to STG is missed in experiments.", "theoretical_claims": "I have checked the analysis of SAND in the linear regression case. The last term in Eq. (14) achieves its minima when there are k of $a_i$'s equal to 1 is not obvious and detailed analysis is needed.", "experimental_designs_or_analyses": "I have check the soundness of experimental designs.", "relation_to_broader_scientific_literature": "Apart from feature selection, the idea of introducing noise to the input variables has been investigated in prior feature noising method, which introduces either multiplicative or additive noise to the input variables to enhance the generalization ability and robustness of the model. \n\n[1] Maaten L, Chen M, Tyree S, et al. Learning with marginalized corrupted features[C]//International Conference on Machine Learning. PMLR, 2013: 410-418.\n[2] Wager S, Wang S, Liang P S. Dropout training as adaptive regularization[J]. Advances in neural information processing systems, 2013, 26.\n[3] Zhuo J, Zhu J, Zhang B. Adaptive Dropout Rates for Learning with Corrupted Features[C]//IJCAI. 2015, 24: 4126-4133.\n[4] Li Z, Gong B, Yang T. Improved dropout for shallow and deep learning[J]. Advances in neural information processing systems, 2016, 29.\n[5]", "essential_references_not_discussed": "To the best of my understanding, key references are reviewed in this paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method is well motivated.\n\nWeakness:\n1. On the benchmark datasets, the performance of SAND is not the SOTA compared to other methods, especially when $k$ is small.\n2. Writing can be further improved, especially the analysis of SAND in the linear regression case.", "other_comments_or_suggestions": "None", "questions_for_authors": "My concerns are listed in the weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed stochastic additive noise decoupling (SAND) for feature selection which can avoid adding explicit loss or regularization term via simple noise-injection mechanism. Reviewers praised that the proposed approach is simple, can be easily integrated into standard training pipelines and avoid tuning additional regularization parameter or feature selection parameter. Reviewers also raised some concerns such as the comparison to stochastic gates, the sensitivity of parameter a_i, and the selection of datasets. Authors’ responses to these concerns are satisfactory which addressed critical issues like competitive results against stochastic gate on 6 datasets and the reported sensitivity analysis of a_i.. As a novel alternative to stochastic gate approach, the acceptance is recommended.
Decision: Accept (poster)