A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning
Authors: Valeriia Cherepanova
Roman Levin
Gowthami Somepalli
Jonas Geiping
C. Bayan Bruss
Andrew Gordon Wilson
Tom Goldstein
Micah Goldblum
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: tabular deep learning
tabular data
feature selection
deep lasso
lasso
TL;DR: We construct a challenging feature selection benchmark evaluated on downstream tabular deep learning models and propose an input-gradient-based analogue of LASSO for neural networks.

Abstract: Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent over-fitting in subsequent downstream modeling, practitioners commonly use automated feature selection methods that identify a reduced subset of informative features. Existing benchmarks for tabular feature selection consider classical downstream models, toy synthetic datasets, or do not evaluate feature selectors on the basis of downstream performance. We construct a challenging feature selection benchmark evaluated on downstream neural networks including transformers, using real datasets and multiple methods for generating extraneous features. We also propose Deep Lasso -- an input-gradient-based analogue of LASSO for neural networks that outperforms classical feature selection methods on challenging problems such as selecting from corrupted or second-order features.

A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning
Valeriia Cherepanova University of Maryland
Roman Levin" University of Washington
Gowthami Somepalli University of Maryland
Jonas Geiping University of Maryland
C. Bayan Bruss Capital One
Andrew Gordon Wilson New York University
Tom Goldstein
Micah Goldblum New York University
University of Maryland
Abstract
Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent overfit- ting in subsequent downstream modeling, practitioners commonly use automated feature selection methods that identify reduced subset of informative features. Existing benchmarks for tabular feature selection consider classical downstream models, toy synthetic datasets, or do not evaluate feature selectors on the basis of downstream performance. Motivated by the increasing popularity of tabular deep learning, we construct challenging feature selection benchmark evaluated downstream neural networks including transformers, using real datasets and multiple methods for generating extraneous features. We also propose Deep Lasso input-gradient gradient-based analogue of Lasso for neural networks that outperforms classical feature selection methods on challenging problems such as selecting from corrupted second-ordes features.
Introduction
Tabular data is ubiquitous across scientific and industrial applications of machine learning. Practi- tioners often curate tabular datasets by including exhaustive sets of available features or by hand- engineering additional features. Under such procedure, real-world tabular datasets quickly accumulate a large volume of features, many of which are not useful for downstream models. Training on such large number of features, including noisy or uninformative ones, can cause overfitting. To avoid overfitting, practitioners filter out and remove features using automated feature selection
methods [Guyon and Elisseeff 2003 Liu and Yu, 2005
The literature contains a wide body of work proposing or evaluating feature selection approaches that classical machine learning algorithmss in their selection criteria which select features for training classical machine learning algorithms downstream [Tibshirani] 1996 Kohavi and John 1997]. Over the past few years, deep learning for tabular data has become competitive and increasingly adopted by practitioners [Ank and Pfister 2021 Gorishniy et al.) [021] (omepalli et al.) 2021] While the community has identified that neural networks are especially prone overfitting noisy features
TTee substantive contributions of the author to the work described in the paper were done prior to the author joining Amazon.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Grinsztajj et al. 2022], thorough evaluation of feature selection methods for downstream tabular neural networks is lacking.
To address this absence, we benchmark feature selection methods in tabular deep learning setting by evaluating the selected features via the performance of neural networks trained on them downstream. In addition to popular existing methods, we select features using the attention maps of tabular transformer models (Gorishniy et al. 2021]. and we further propose Deep Lasso input-gradient- based analogue of Lasso (Tibshirani, 1996] for deep tabular models. Whereas many previous works
on feature selection use entirely synthetic datasets Waa al. 2018 Passemiers al. 2023 Bolón Canedo et al. 2013 or create extraneous features by concatenating random noise onto existing features [Dinh and Ho 2020 [Borisov et al., 2019), we conduct our benchmark on real datasets, and explore three different ways to construct extraneous features: random noise features, corrupted features, and second-order features that serve as a prototypical example of feature engineering.
In our experiments, we find that while many feature selection methods can differentiate between informative features and noise within reason, they may fail under more challenging settings. No- tably, Deep Lasso selects features which achieve significantly better downstream neural network performance than previous methods when selecting from corrupted or second-order features.
Our primary contributions are summarized as follows:
We construct a challenging feature selection benchmark comprising real-world datasets with extraneous uninformative, corrupted, and redundant features.
We benchmark different feature selection algorithms for deep tabular models, including recent tabular transformer architectures, on the basis of downstream performance.
We propose a generalization of Lasso for deep neural networks, which leverages input gradients to train models robust to changes in uninformative features. We show that Deep Lasso outperforms other feature selection methods, including rree-based methods, in the most challenging benchmark setups.
Related Work
In the following section, we provide brief overview of recent developments in tabular deep learning, and of feature selection in machine learning more broadly.
2.1 Tabular Deep Learning
Tabular data is the dominant format of data in real-world machine learning applications. Until recently, these applications were primarily solved using classical decision tree models, such as gradient boosted decision trees (GBDT). However, modern deep tabular neural networks started to bridge the gap to conventional GBDTs, which turn unlocked new use cases for deep learning in tabular domain. Recent work includes the development of novel tabular architectures, for example based on transformer models (Huang et al.) [2020] Gorishniy et al.) [202T] (Somepalli et al., 2021 Arık and Pfister [202T]. and ensembles of differentiable learners [Popov et al. [019]] Kontschieder iet al.
et al. 2020 Badirli et al. 2020] as well as modifications and regularizations for MLP-based architectures [Kadra al. [202] Gorishniy al. 2022]. Other works explore new capabilities that are enabled by tabular deep learning, such as self-supervised pre-training [Ucar
ct al.
102 Somepalli et al. 2021 Rubachev al.
2022
Kossen et al. 2021 Agarwal et al. 2022
transfer learning [Levin et al. 2023 Wang and Sun
2022
Zhu et al. 2023 few-shot learning
Nam
ct al.
2023 and data generation Kotelnikov et al.
2022
2.2 Feature Selection
A comerstone of applied machine learning is feature selection, where data science practitioners carefully curate and select features for predictive tasks. As a result, there has been considerable interest in automating this process.
'Our code for benchmark and Deep Lasso is available at

---

Existing approaches for feature selection can be categorized into three main types: filter, wrapper and embedded methods. Filtering algorithms rank features based on their individual characteristics and relevance to target variables, without considering any specific learning algorithm. Examples of
filter methods include univariate statistical tests, variance filters, and mutual information scores. comprehensive overview of existing filter methods can be found in [Lazar et al. [012]. Wrapper methods, on the other hand, are algorithn-dependen and involve iteratively re-training a machine a subset of features to identify the subset that yields the best performance. These include greedy sequential algorithms [Kitter] 1978]. recursive feature elimination [Guyon et al.
learning algorithm
2002
Huang al. 2018] well as evolutionary algorithms (Xue et al., 2015 Siedlecki and Sklansky 1989 Kennedy and Eberhart 1995 Gheyas and Smith 2010]. Embedded methods incorporate the
task of feature selection into the training process, allowing the model to learn which features most relevant while training. Lasso [Tibshirani 1996] is a classical embedded feature selection algorithm. which has been also applied to deep neural networks in the form of Adaptive Group Lasso (Dinh and Ho 2020)_ Additionally, tree-based algorithms like Random Forests [Breiman] 2001] and Gradient Boosted Decision Trees [Friedman 2001] employ built-in feature importance measures, enabling automatic feature selection. A few recent works propose specialized neural network architectures with embedded feature selection through knockoff filters (Lu et al. 2018 Zhu et al. [202T]. auto-encoders [Balm et al, 2019 Zhu et al. [02T] and specialized gating layers Lemhadri et al. [022].
We find wrapper methods generally too computationally expensive to be useful for the deep neural network models we consider in this study, especially when hyperparameter optimization is performed. Therefore, in our study we focus on established filter and embedded approaches, both classical and modern, that can be applied to generic tabular architectures.
2.3 Feature Selection Benchmarks
So far, it is unclear which strategy, whether classical or modern, would be optimal for feature selection with deep tabular models. Existing benchmark studies focus primarily on classical downstream
al. 2014] do not optimize for downstream performance at all [Lu et al. 2018]. A few works evaluate feature selection
models (Bolón-Canedo al. 2013 Bommert al., 2020 Bolón-Canedo
methods on synthetic datasets (Bolón-Canedo al. 2013 Wah et al. 2018 Passemiers et al. 2023
Sánchez-Maroñoo et al. 2007] or on domain specific datasets such as high-dimensional genomics data
containing small number of samples [Bommert et al., 2020 Bolón-Canedo et al. [014]. malware detection data [Darshan and Jaidhar| [2018] and text classification [Darshan and Jaidhar [2018] Forman
ctt al. 2003). Passemiers et al. [2023] evaluates neural network interpretability methods in a feature selection setup on synthetic datasets, using small MLP model, and finds them to be less effective than classical feature selection methods such as random forest importance.
In contrast, in this work we provide an extensive benchmark of feature selection methods for modern deep tabular models. We construct our feature selection benchmark on large-scale tabular datasets with different types of extraneous features and investigate a range of representative feature selectors, both classical and deep learning based. Our benchmark evaluates feature selection methods based on performance of the downstream deep tabular models.
3 Experimental Setup
We construct a challenging feature selection benchmark that uses real datasets and includes multiple approaches for the controlled construction of extraneous features. In all cases, we evaluate feature selection methods based on downstream neural network performance, i.e. the practically relevant met- ric of success for feature selection procedure. We conduct experiments with both MLP architectures and the recent transformer-based deep tabular FT- Traanssffrreer architecture [Gorishniy et al.) [021] as downstream models. Our benchmark comprises 12 datasets with types of additional features. These datasets are collected and adapted based on recent tabular benchmark papers [Gorishnly et al.
2021 |021]2022] 2022] Rubachev et al. 2022] and include ALOI (AL), California Housing (CA), Covertype (CO). Eye Movements (EY), Gesture (GE), Helena (HE), Higgs 98k (HI), House 16K (HO), Jannis (JA), Otto Group Product Classification (OT), Year (YE) and Microsoft (MI). Among these datasets, there are eight classification datasets and four regression datasets. We measure downstream model performance using accuracy for the classification tasks and RMSE for the regression tasks. Additional details concerning these datasets can be found in Appendix B.
3

---

ALOI
Californi Housing
Covertype
Eye Movements
0.97
0.950
004225
0.925
-0.450
0.96
0.000
-0.475
0.875
-0.500
0.850
-0.525
0.94
Gesture
Helena
Higgs
House
0.65
0.39
0.82
-0.550
Perforance
0.00
-0.575
0.38
0.95
0.37
0.78
-0.000
FT-Fansformer
0.50
0.76
-0.625
0.36
XGBoost
0.45
-0.675
Jannis
-0.86
0.82
-0.78
0.72
-0.88
0.80
0.70
-0.90
-0.80
0.76
-0.81
-0.92
% of uninformative features in the dataset
Figure I: Performance of FT- T'T-Transformer, MLP and XGBoost models when trained on data with random extra features. The X-axes indicates the percentage of uninformative features in the dataset and the Y-axes depict accuracy for classification problems and negative RMSE for regression datasets, so higher values always indicate better performance. Overall, find MLP models to be more susceptible to noise than either XGBoost or TT-Transformer models.
For each benchmarking experiment, we perform extensive hyperparameter tuning for both feature selection algorithms and downstream models with respect to the downstream model performance using the Bayesian hyperparameter optimization engine Optuna [Akiba et al., 2019]. We select the best hyperparameters based validation metrics and report test metrics computed over 10 random model initializations (seeds). Details concerning final hyperparameters can also found in the Appendix sections
In the following section, we present a motivating experiment. In Section 5 we discuss our benehmark design in detail. Section presents experimental benchmark results using feature selection methods.
4 Are Deep Tabular Models More Susceptible to Noise than GBDT?
Recent contributions to the ongoing competition between tabular neural networks and gradient boosted decision trees (GBDT) have found that neural networks are more susceptible to noise than
GBDT on small to medium datasets (up to 10,000 samples) [Grinsztajn et al.) 2022]. We scale this experiment to larger datasets and showease as motivating example for feature selection methods and benchmarks specific to deep tabular models.
We explore the influence of uninformative features on tabular neural networks and assess the perfor- mance of MLP and FT-Transformer models on datasets containing varying numbers of uninformative Gaussian noise features. For reference, we also include GBDT model into our comparison as implemented in the popular XGBoost package [Chen and Guestrin 2016]. Figure F fillustrates the relationship between the performance of the three models and the proportion of uninformative features in these datasets. Similarly to Grinsztajn et al. [2022), we observe that the MLP architecture, on average, exhibits more overfitting to uninformative features compared to XGBoost, motivating the need for careful feature selection with tabular neural networks. Interestingly, as seen from the slope of the blue and green curves in Figure [ the FT-Transformer model is roughly as robust to noisy features the XGBoost model. The fact that the performance of the FT- Trannsffore model is not as

---

Random Features
Corrupted Features
Second- Order Features
% of extra features selected
1.0
/mportance correlation is 0.89
1.0
Importance rank correlation 0.63
1.0
Importance correlation
0.8
0.8
0.0
0.6
0.6
0.6
0.4
Univariate
Lasso
Lasso
Forest
XGBoost
Deep Lasso
Univariate
Lasso
Lasso
Forest
XGBoost
Deep Lasso
Univariate
Lasso
Lasso
Forest
xGBoost
Deep Lasso
Figure 2: Percent random, corrupted, and second-order features selected by different feature selection algorithms. Importance rank correlation refers to pair-wise feature importance Spearman correlation averaged across feature selection algorithms and datasets. Random features are less often ranked as important compared to corrupted and second-order features, and feature selection algorithms have higher agreement when selecting from random features.
severely affected by noisy features could be attributed to the ability of the transformer architecture to filter out uninformative features through its attention mechanism. Inspired by this observation, we further investigate the effectiveness of utilizing the attention map importance within FT-Transformer Gorishmiy et al. 2021] as a feature selection method in our benchmark study.
5 Feature Selection Benchmark
It is not common for real-world datasets contain completely random noise features with predic- tive power whatsoever, although engineered features often exhibit varying degrees of redundancy and noise. Nonetheless, feature selection algorithms are often evaluated on datasets containing spurious features generated from Gaussian noise. Not only does this differ significantly from real-world feature selection scenarios, but it also presents relatively straightforward task for many feature selection algorithms to eliminate these random features. In light of these considerations, we propose an alternative approach establish more challenging and realistic feature selection benchmark by introducing three distinct methods for crafting additional features:
Random Features. In the simplest scenario, we sample uninformative features from Gaussian distribution and concatenate them with the original dataset features.
Corrupted Features. To simulate a scenario with noisy, yet still relevant features, we sample extraneous features from the original ones and corrupt them with Gaussian noise. In addition, we conduct experiments with Laplace noise corruption and report experimental results in Appendix F.]
Recond-Order Features. To simulate feature engineering scenario with redundant information contained in engineered features, we add second-order features, i.e. products randomly selected original features.
Note that higher-order features are not spurious and are often used by data scientists precisely because they can contain useful information. Therefore, selecting higher order features instead of original ones may not be a bad thing. Such fcature selection algorithms must be evaluated terms of downstream model performance
do in the following section.
To gauge the difficulty of the proposed benchmark, we explore how often different feature selection algorithms rank extraneous features among the top-k most important features, where i represents the number of original features the datasets. From Figure we observe that all methods select fewer random features than they do corrupted or second-ord -rderr features. Additionally, to quantify the overall agreement between different feature selection methods, we analyze average pair-wise Spearman correlation between the rankings of features generated by different selection algorithms. Notably, the setup involving random extra features exhibits the highest correlation, indicating that filtering out random features is relatively straightforward and all feature selection algorithms behave similarly. In
5

---

contrast, the setup with second-order extra features has the lowest rank correlation implying greater disparity in selection preferences among the algorithms.
We note, that different feature selection algorithms may select similar features, but rank features dif- ferently within "important" and "unimportant" groups, which would results in lower rank correlation even in the settings where feature selection is straightforward (i.e. selecting from random features).
Benchmarking Feature Selection Methods
In this section, we benchmark various feature selection methods. particular, we consider the following feature selection approaches:
Univariate Statistical Test. This classical analysis checks the linear dependence between the predictors and the target variable. It selects features based on the ANOVA F-values for classification problems and univariate linear regression
F-values for regression problems.
Lasso uses L1 regularization to encourage spansity in a linear regression model [Tibshirani [996]. After this sparse regression, features are ranked with respect to the magnitudes of their coefficients in the model.
First-Layer Lasso (1L Lasso) is an extension of Lasso for MLPs with multiple layers. It applies Group Lasso penalty to the weights of the first layer parameters:
min aC (X, + (1
|1
where W is the j-th column of weight matrix of the first hidden layer corresponding to the j-th feature. Similarly to Lasso, First-Layer Lasso ranks features with respect to their grouped weights in the first layer.
Adaptive Group Lasso (AGL) an extension of the Group Lasso regularization method (Dinh and Ho 2020]. Similarly to the First-Layer Lasso, it applies a Group Lasso penalty to the weights of the first layer parameters, however each group of coefficients is weighted with an adaptive weight parameter:
min + (1-a))
Σ
|W
|WOI
where W is the Group Lasso estimate of W. Adaptive Group Lasso then ranks features with respect to their grouped weights in the first layer.
LassoNet is neural network architecture that incorporates feature selection [Lemhadri et al. 2021 LassoNet achieves feature sparsity by adding a skip (residual) layer and allowing the features to participate only if their skip-layer connection active.
Random Forest (RF) bagging ensemble of decision trees, and it ranks features with respect to their contribution to the ensemble [Breiman] 2001]. In particular, importance is calculated by measuring the decrease in impurity when that feature used for splitting each node of the decision trees in the forest.
XGBoost is popular implementation for gradient boosted decision tree [Chen and Guestrin, 2016 XGBoost computes feature importance as the average gain across all splits in trees where a feature was used.
Attention Map Importance (AM) is computed for FT-Transformer model from one forward pass on the validation set. We follow Gorishniy et al. 2021 and calculate feature importance the average attention map for the [CLS] token across all layers, heads, and validation samples.
Deep Lasso is our generalization of Lasso to deep tabular models (and, in fact, any differentiable model). Deep Lasso encourages feature gradient sparsity for deep tabular models by applying Group Lasso penalty to gradients of the loss with respect input features during training. Intuitively, this makes the model robust to changes in unimportant features. For train data (X, Y) with samples and m features, the Deep Lasso penalty is given by

---

Table 1: Benchmarking feature selection methods for MLP and FT-Transformer downstream models on datasets with random extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers percent of extra features in the dataset: either 50% or 75% features are random. Bold font indicates the best numbers dataset-wise and lower rank indicates better overall result.
FS method
AL
CH
CO
EY
GE
50
NS + MLP
0.941
-0.450
0.961
0,538
0.466
0.366
0.798
40.622
0.703
-0.911
0.773
-0.801
8.08
50
Univariate MLP
0.96
-0.447
0.970
0.575
0.515
0.379
0.811
-0.549
0.715
-0.891
0.508
-0.776
2.66
50
Lasso + MLP
0.949
-0.454
0.969
0.547
0.458
0.380
0.812
-0.599
0.715
-0,907
0.805
-0.787
5.91
50
Lasso MLP
0,952
-0.451
0.969
0.564
0.474
0.375
-0.568
0.715
-0.897
0.796
-0.773
4.91
50
AGL +LP
0.958
-0.512
0.969
0,578
0.473
0.386
0.810
-0.557
0.718
-0.898
0.799
-0,778
4.33
50
LassoNet MLP
0.954
-0.445
0,969
0.552
0.495
0.385
0.811
-0.557
0.715
-0.907
0.783
-0,787
5.16
50
AM + MLP
0.953
-0.444
0.968
0.554
0.498
0.382
0.813
-0.566
0.722
-0,904
0.801
-0.777
3.83
50
RF + MLP
0.955
00.533
0.969
0.589
0.594
0.386
0.814
005522
0.720
-0.904
0.806
-0,786
2.91
50
XGBoost ML.P
0.956
-0.444
0.969
0.59
0.502
0.385
0.812
-0.560
0.72
-0.893
808
-0.777
2.33
50
Deep Lasso MLP
0.959
-0.443
0.968
0,573
0.485
0,383
0,814
-0,549
0.72
-0,894
0.802
-0,776
2.33
75
FS MLP
0.925
-0.527
0.955
0.502
0.417
0,348
0,778
-0.674
0.671
-0,917
0.749
-0.812
7.41
75
Univariate MLP
0.96
-0.447
0.970
0.575
0.502
0.381
0.810
-0,549
0.713
-0.89
0.806
-0,776
2.50
75
Lasso ML.P
0,959
-0.454
0.967
0.543
0.491
0.381
0
-0.612
0.716
-0,907
0.801
-0.789
4.33
75
IL Lasso + MLP
0.957
-0.448
0.968
0,555
0.432
0,380
0.809
-0.572
0.717
-0,903
0.799
-0.775
4.41
75
AGL MLP
0.954
-0.447
0.968
0.561
0,429
-0.571
0.719
-0,901
4.33
75
LassoNet MLP
0.958
-0.452
0.966
0.528
0.475
0.383
0,809
40,555
0.705
-0,913
0.768
-0.794
4.75
75
RF+ MLP
0.949
-0.453
0,968
0.584
0.61
0.386
0.814
-0.585
0,718
-0.902
0.808
-0,784
2.91
75
XGBoost MLP
0.958
-0.451
0.969
0.576
0.583
0,382
0,810
-0.568
0.72
-0.892
0.804
-0.774
2.08
75
Deep Lawso MLP
0.957
-0.446
0.969
0.569
0.479
0.387
0.814
40.559
0.721
.0.893
0.800
-0.774
2.33
50
FS
0.959
-0.443
0.966
0.673
0.500
0.384
0.817
-0.577
0.730
-0,902
0.813
-0.792
6.58
50
Univaristo + FT
0.963
-0.424
0,970
0,700
0.519
0.389
0.819
-0.554
0,733
-0.897
0.819
-0,789
2.83
50
Lasso FT
0.952
-0.419
0.960
0.682
0.489
0,388
0,819
-0.599
0.728
-0,999
0.817
-0.998
6.33
50
H. Lasso + FT
0.963
-0.423
0.969
0.72
0.489
0.382
0.818
-0.577
0.732
-0.904
0.819
-0,791
5.16
50
AGL FT
0.899
-0.42
0.969
0.701
0.480
0.393
0.822
-0.586
0.733
-0,915
0.814
-0.832
5.25
50
LassoNer FT
0.963
-0.426
0.970
0.670
0.505
0.392
0.818
-0.559
0.733
-0,904
0.808
-0.791
4.83
50
0.962
-0.425
0.968
0.657
0.505
-0.554
0.735
-0,903
3.75
50
RF + FT
0.963
-0.42
0.969
0.718
0.591
0.395
0.821
-0.558
0.737
-0,900
0.82
-0.791
1.75
50
XGBeost + FT
0.963
-0.42
0.969
0,725
0.572
0.392
0.820
40.558
0,734
-0.898
0.82
-0,789
1.91
50
Deep Lasso FT
0.962
-0.419
0.969
0.703
0.504
0.392
0.817
40.560
0.733
-0,900
0.817
-0.788
3.66
m min ( a1- Σ
(
axt)
where, for j-th feature (column of X), (C,) OX
(Cx(X.))
OC,(X.Y)
(C,
Once
d21,
D22j
the model is trained with the above regularizer, the corresponding feature importance is provided by
OX,
importance of the j-th feature-
(2)
ax
Note that in the linear regression case, the classical Lasso is equivalent to the proposed input-gradien sparsity regularizer applied to model output since in the linear case input gradients are the weights of the linear model. We provide a formal proof of equivalence between Deep Lasso, classical Lasso, and
First-Layer Lasso in the linear regression case in Appendix Section
E We also note that Deep Lasso is
related to methods used to promote network explainability by leveraging input gradients (Sundararajan
et al. 2017 Shrikumar al. 2017 Smilkov et al. 2017 Levin et al. 2022). In addition, Liu 202 examines input gradients for feature selection from a Bayesian perspective. This work uses Bayesian Neural Networks (BNNs) along with associated credible intervals to estimate uncertainty surrounding input gradients, choosing features based on the plausibility that their corresponding gradients are zero. On the hand, this principled work comes with theoretical guarantees, but on the other hand it requires Hamiltonian Monte Carlo, an expensive sampler which does not scale to large datasets and models.
6.1 Results
We benchmark feature selection methods for downstream MIP and FT-Transformer models. In the case of the MLP downstream model, we explore scenarios where either 50% or 75% of the features
7

---

Table 2: Benchmarking feature selection methods for MLP and FT-Transformer downstream models on datasets with corrupted extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of extra features in the dataset: either 50% or 75% features are corrupted. Bold font indicates the best numbers dataset-wise and lower rank indicates better overall performance.
16
FS method + Model
AL
CH
CO
EY
GE
HE
HI
HO
JA
M1
OT
YE
rank
50
No FS + MLP
0,946
-0.475
0.965
0.557
0.525
0.370
0.802
-0.607
0.703
-0.909
0.778
-0.797
8.00
50
Univariate MLP
0,955
-0.451
0.966
0.556
0.514
0.346
0,810
-0.620
U.717
-0.920
0.795
-0.828
7.33
50
Lasso MLP
0.955
-0,449
0.968
0,548
0.512
0.382
0.813
-0.602
0.713
-0.903
0.796
-0.795
5.42
50
Lasso MLP
0,955
-0.447
0.968
0,566
0.515
0.382
0.812
-0.581
0
-0.902
0.795
-0,780
4.75
50
AGL MLP
0.953
0.968
0.538
0.386
0,813
-0.561
0.722
00.902
0.796
.0.780
3.00
50
LassoNet ML.P
0.955
-0.452
0.969
0.570
0.556
0.382
0,811
-0.551
0.719
-0.905
0.795
-0.777
3.83
50
AM + MLP
0.955
-0.449
0.967
0,583
0.527
0,381
0.814
-0.555
0.722
-0.905
0.797
-0.780
3.58
50
RF MLP
0.951
-0:453
0.967
0,574
0.568
0.383
0.810
-0.565
0.724
-0.904
0.788
-0.786
4.67
50
XGBoost MLP
0.954
-0.454
0.969
0.583
0.510
0,385
0.815
-0.553
0.722
-0.892
0.803
-0.779
2.67
50
Deep Lasso+ MLP
0.955
0.525
0.388
0.815
-0.567
-0.895
0.801
-0.776
2.58
75
FS + MLP
0
-0.516
0.956
0,518
0.503
0.356
0.788
-0.632
0.686
-0.913
0.762
-0.808
7.58
75
Univariate MLP
0.955
-0.569
0.941
0,510
0.495
0.347
0.742
-0.620
0.686
-0.921
0.779
-0.838
7.50
75
Lasso + MLP
0,948
-0.454
0.963
0.565
0.490
0.373
0,810
-0.593
U.717
-0.903
0.795
-0.791
5.00
75
Lasso MLP
0.955
-0.444
0.967
0,549
0.495
0,380
0.811
-0.576
0.715
-0.903
0.797
-0,779
3.25
75
AGL + MLP
0.928
-0.566
0.967
0,548
0.490
0.382
0,811
-0.574
U,714
-0.904
0.7888
-U,780
4.92
75
LassoNet MLP
0.947
0,539
0.533
0.383
0.805
-0.572
0.708
0..08
0.791
.0.785
4.33
75
RF + ML.
0.952
-0.450
0.963
0,547
0.533
0.372
0,805
-0.573
0.716
-0.903
0.765
-0.788
4.92
75
XGBeost MLP
0,954
-0.515
0.968
0.571
0.53
0,381
0.811
-0.571
0.721
-0.895
0.800
-0.784
2.58
75
Deep Lasso MLP
0.959
-0,441
0.968
0,554
0.517
0,386
0.813
-0,563
0,718
-0.898
0,804
-0.778
1.42
50
No FT
0,960
0.386
0,818
-0.901
0.809
-0.793
6.08
50
Univariate FT
0.963
-0.422
0.965
0.681
0.574
0.345
0,812
-0.628
0,733
-0.920
0.812
-0.826
6.17
50
Lasso + FT
0,952
-0.422
0.936
0.697
0.556
0.387
0.820
-0.586
0.732
-0.937
0.812
-0.915
6.42
50
Lasso FT
0.962
-0,419
0.969
0,718
0.571
0,389
0.820
-0.570
0.731
-0.899
0,816
-0,795
3,50
50
AGL FT
0,906
-0.426
0.969
0.697
0.591
0.392
0,820
-0.552
0.735
-0.914
0.816
-0.830
4.08
50
LassoNct
0.962
-0.426
0.97
0.578
0.393
0.814
-0.572
-0.903
0.813
3.92
50
+M
0.962
-0.424
0.969
0,680
0.572
0.392
0,820
-0.549
U.734
-0.901
0,817
-0,790
3.42
50
RT+
0,962
0.969
0.711
0.387
0,819
-0.557
+0.8%
0.806
.0.793
3.42
50
XGBeost + FT
0.963
-0.422
0.97
0,706
0.564
0.392
0.821
-0.548
0.735
-0.899
0.816
-0.790
2.42
50
Deep Lasso FT
0,961
-0.422
0.968
0.725
0.577
0.393
0.821
-0.561
0.736
-0.89
0.809
-0.788
2.67
in the dataset are extraneous. For the FT-Transformer, we focus on evaluating FS methods solely on datasets with 50% added features. For simplicity. we train the downstream models on the top-k important features determined by the feature selection algorithms, where corresponds to the original number of features in the datasets. We include more details on our experimental setup in Section C
We report dateset-wise downstream performance based on the proposed benchmark, as well overall rank of the methods in Tables [ and and we include results with standard errors computed across seeds in Tables T6 17 18 While did not discover universally optimal feature selection method that performed the best across all datasets and extra feature types, we did identify several noteworthy patterns, which we will discuss below.
Random Features. In the easy scenario where extraneous features are Gaussian noise, we observe in T XGBoost, Random Forest, univariate statistical test and Deep Lasso perform on par for MLP downstream models, while for FT-Transformer downstream models, Random Forest and XGBoost outperform other methods. Conversely, Lasso, IL Lasso, AGL, LassoNet and Attention Map Importance are less competitive. These findings align with the results depicted in Figure 2 which highlight the high similarity in importance rank between the methods and the greater tendency of Lasso based methods to assign higher ranks to random features. In addition to downstream performance, we report ROC-AUC and precision scores in Tables
Corrupted Features. In a more challenging scenario involving corrupted extra features, both Deep Lasso and XGBoost significantly outperform the other feature selection methods. Specifically, Deep Lasso exhibits superior performance for the MLP downstream model, while XGBoost performs slightly better for the FT-Trassformer downstream model, see Table 2
Second-Order Features. Finally, in the most challenging scenario of choosing among original and second-order features, Deep Lasso demonstrates significant performance advantage over the other methods, see [ Interestingly, we discover that the relative rank of Deep Lasso lower when 75% of all features were generated, indicating that Deep Lasso excels in more challenging feature
8

---

Table 3: Benchmarking feature selection methods for MLP and FT-Transformer downstream models on datasets with second-order extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of extra features in the dataset: either 50% or 75% features are second-order. Bold font indicates the best numbers dataset-wise and lower rank indicates better overall result.
18
FS method
AL
CI1
CO
BY
GE
IIE
III
I10
MI
OT
YE
rank
50
ES + ML.
0,960
-0.445
0.969
0.631
0.605
0.383
0.811
-0.549
0.719
-0.891
0.800
-0.786
4.50
50
Univariste MLP
0.961
-0.439
0.959
0.584
0.582
0.357
0.817
-0.614
0.724
-0,902
0.798
-0.810
6.58
50
Lasso + MLP
0.955
-0.443
0.966
0,608
0.590
0.366
0.816
-0.564
0.724
-0.891
0.806
-0,783
5,33
50
Lasso MLP
0.959
00.455
0.969
0.634
0.571
0.380
0.815
40.565
0.728
-0.89
0.808
-0.780
3.92
50
AGI. ML.P
0
-0.443
0.953
0.637
0.594
0.383
0.807
-0.565
0.730
-0.89
0.806
-0,776
3.25
50
LassoNet MLP
0,959
-0.442
0.969
0.641
0.611
0.379
0.816
-0.595
0.724
-0.893
0.797
-0.784
4,50
50
AM + MLP
0.961
-0.439
0.968
0.622
0.604
0
0.819
-0.566
0.730
-0.892
0.802
-0,778
3.50
50
RF+ MLP
0.958
-0.437
0.969
0.639
0.619
0.370
0.818
-0.586
0.735
-0.89
0.801
-0.781
3.25
50
XGBoost+ MLP
0,870
-0.438
0.97
0.635
0.604
0,373
0.818
-0.579
0.734
-0.891
0.805
-0.786
3.83
50
Deep Lasso MLP
0.961
-0.441
0.969
0.648
0.600
0.384
0.815
-0.572
0.733
-0.89
0.805
-0.776
2.67
75 No FS + MLP
0.952
-0.451
0.969
0.630
0.598
0.388
0.808
-0,542
0.717
-0,900
0.792
-0.792
4,92
75
Univariate MLP
0.960
-0.530
0.488
0.583
0.531
0.352
-0.608
0.720
-0,908
0.785
-0.820
7.25
75
Lasso MLP
0.96
-0.434
0.968
0.612
0.519
0.363
0.820
-0.555
0.739
-0.894
0,807
-0.793
3.67
75
1L Lasso + MLP
0.96
-0.452
0.966
0.654
0.579
0.375
0.818
-0.549
0.741
-0.893
0.805
-0,782
3.33
75
AGL MLP
0.958
-0.438
0.968
0.647
0.601
0.384
0.819
-0.545
0.736
-0.893
0.800
-0.781
2.67
75
LassoNet ML.P
0.958
-0.454
0.968
0.633
0.615
0.362
0.813
-0.569
0.726
-0.895
0.793
-0,786
4.83
75
RF MLP
0,956
-0.445
0.968
0.627
0.566
0.339
0.819
-0.615
0.728
-0.892
0.794
-0.789
5.08
75
XGBoost+ MLP
0.459
-0.495
0.968
0.627
0.555
0.358
0.821
-0.588
0.738
-0.892
0.803
-0.795
5.00
75
Deep Lasso MLP
0.959
-0.448
0.969
0.647
0.582
0.378
0.821
-0.568
0.74
-0.89
0.805
-0.78
2.17
50
No FS + FT
0.962
-0.425
0.968
0.733
0.558
0.391
0.819
.0,552
0,732
-0.901
0.818
-0,790
4.08
50
Univariate FT
0.963
-0.419
0.942
0.615
0.573
0.351
0.820
-0.613
0.728
-0,910
0.810
-0,819
6.25
50
Lasso FT
0.962
-0.422
0.967
0.720
0.560
0.381
0.823
-0.561
0.733
-0,905
0.816
-0.874
5.08
50
Lasso FT
0.964
-0.42.
0.970
0.748
0.604
-0.557
0.736
-0.896
0.817
-0.795
2.83
50
AGL FT
0.935
-0.421
0.950
0.747
0.540
0,388
0.821
-0.557
0.730
-0,907
0.821
-0.821
4.16
50
LassoNet + FT
0.963
-0.422
0.970
0,729
0.612
0.378
0.817
-0.592
0.733
-0.901
0.808
-0,792
5,00
50
+M+
0,963
-0.418
0.970
0.719
0.597
0.387
0.820
.0.579
0.736
-0,902
0.819
-0.791
3.83
50
RF FT
0.963
-0.422
0.970
0.733
0.615
0.365
0.822
-0.600
0.738
-0.895
0.816
-0,793
3,66
50
XGBoost FT
0.871
-0.424
0.962
0.737
0.594
0.379
0.822
-0.587
0.736
-0.8%
0.812
-0.792
4,08
50
Deep Lisso + FT'
0.963
-0.422
0.970
0.726
0.608
0.388
0.822
40.558
0.738
-0.897
0.819
-0.789
2.41
selection problems involving a significant proportion of spurious or redundant features, as observed in both corrupted and second-order extra features.
7 Similarity between Feature Selection Methods
In this section we analyze which feature selection algorithms are more similar to each other in terms of feature ranking. In Figure we present a heatmap of wair-wise Spearman correlation between feature rankings of different FS methods averaged across datasets. We focus on the setup involving second-order features, as we discovered the lowest level of agreement between methods within this configuration. It is worth noting that we provide results for methods fine-tuned for both MLP and FT- FT-nransformer models, ensuring comprehensive comparison.
We find related similarity patterns for both downstream models. In particular, we observe that Random Forest and XGBoost rankings exhibit high correlation, indicating a strong agreement between these both tree-based methods. We find that First-Layer Lasso is highly correlated with Adaptive Group Lasso, which is an extension of First-Layer Lasso. Additionally, Random Forest results are also correlated with results from the Univariate Test. Deep Lasso is most correlated with Lasso-based methods, such as classical Lasso, First-Layer Lasso and Adaptive Group Lasso and with Attention Map Importance.
8 Discussion
In this paper, we present a challenging feature selection benchmark for deep tabular models, address- ing the need for more realistic scenarios in feature selection evaluation. Our benchmark encompasses real-world datasets with extraneous uninformative, corrupted, and redundant features. Through comprehensive experimentation, we compare various feature selection methods on our proposed benchmark. Our findings reveal that while classical feature selection methods, including tree-based

---

MLP
FT-Transformer
Univariate
Univariate
Lasso
Lasso
1L Lasso
1L Lasso
AGL
AGL
LassoNet
LassoNet
AM
AM
Forest
Forest
XGBoost
XGBoost
Deep Lasso
Deep Lasso
Univariate
Lasso
1L Lasso
AGL
LassoNet
AM
Forest
XGBoost
Deep Lasso
Univariate
Lasso
Lasso
AGL
LassoNet
AM
Forest
XGBoost
Deep Lasso
Figure 3: Similarity between feature selection methods tuned downstream MLP and FT-
Transformer models. Heatmap depicts pair-wise Spearman correlation of feature importances for the setup with second-order features. Correlations are averaged across the datasets.
algorithms perform competitively in the random and corrupted setups, specialized deep tabular feature selection methods, like the proposed Deep Lasso outperform other methods selecting from second-orde features. This indicates the benefits of neural network inductive biases in feature selection algorithms. Overall, our study contributes a systematic new benchmark with analysis, new feature selection method, and insights into improving the performance and robustness of deep tabular models. The benchmark code and Deep Lasso implementation are made available to facilitate reproducibility and practical usage.
Acknowledgements
This work was made possible by the ONR MURI program and the AFOSR MURI program. Com- mercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IS-2212182), and by the NSF TRAILS Institute (2229885).
References
R. Agarwal, A. Muralidhar, A. Som, and H. Kowshik. Self-supervised representation learning across sequential and tabular features using transformers. In NewrlPS 2022 First Table Representation Workshop, 2022. URL (tthhtt:::. httttp:://ppenrvieex wttp:a/wwwe...ttteoooooooood.ddd
1Dsk
Akiba, Sano, T. Yanase, T. Ohta, and M. Koyama. Optuma: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery data mining, pages 2623-2631, 2019.
O. Arık and T. Pfister. Tabnet: Attentive interpretable tabular learning. AAAI, volume 35, pages 6679 1779-6687,7 6687, 2021.
S. Badirli, X. Liu, Z. Xing, A. Bhowmik, K. Doan, and S. S. Keerthi. Gradient boosting neural networks: Grownet. arXiv preprint arXiv:2002.07971, 2020.
P. Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):4308, 2014.
M. Balın, Abid, and Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International conference machine learning, pages 444-453, PMLR, 2019.
10

---

T. Bertin- Mahieux, D. P. Ellis, B. Whitman, and P. Lamere. The million song dataset. 2011.
J. A. Blackard and J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 24(3):131-151) 1999.
V. Bolón-Canede N. Sánchez- Marooño, and methods on synthetic data. Knowledge and information systems, 34:483-519 2013.
review of feature selection
V. Bolón-Canedo, N. Sánchez- A. of microarray datasets and applied feature selection methods. Information sciences, 282:111-133 11-11,, 2014.
J. M. Benítez, and F. Herrera. A review
A. Bommert, X. Sun, B. Bischl, J. Rahnenführer, and M. Lang. Benchmark for filter methods for feature selection in high-dimennional classification data. Computational Statisties & Data Analysis, 143:106839, 2020.
V. Borisov, Haug, and G. Kasneci. Cancelout: A layer for feature selection in deep neural networks. In Artificial Neural Networks and Machine Learning-ICANN 2019: Deep Learning: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part Il 28, pages 72 83. Springer, 2019.
Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
S. S. Darshan and C. Jaidhar. Performance evaluation of filter-based feature selection techniques in classifying portable executable files. Procedia Computer Science, 125:3463366, 2018.
V. C. Dinh and L. S. Ho. Consistent feature selection for analytic deep neural networks. Advances in Neural Information Processing Systems, 33:2420-244 2231, 2020.
G. Forman et al. An extensive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res., ((Maa)):2899133)) 2003.
H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
J.-M. Geusebroek, G. J. Burghouts, and A. W. Smeulders. The amsterdam library of object images. International Journal of Computer Vision, 61:103-112, 2005.
I. A. Gheyas and L. S. Smith. Feature subset selection in large dimensionality domains. Pullern recognition, 43(1):5-13, 2010.
Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data. arXiv preprint arXiv:2106.1 1959, 2021.
Y. Gorishniy, I. Rubachev, and A. Babenko. On embeddings for numerical features tabular deep learning, arXiv preprint arXiv:2203.0 05.55656..6. 2022.
L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL http:////pppeeeeeeeeeeeeettutuue Fp7 PhQeznn
I. Guyon and A. Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 315ar):1157-1182, 2003.
I. Guyon, J. Weston, S. Barnhill, and V. Vapnik. Gene selection for cancer classification using support vector machines. Machine learning, 46:389-422, 2002.
Guyon, Sun-Hosoya, M. Boullé, Escalante, S. Escalera, Liu, D. Jajetic, Ray, M. Saeed, M. Sebag, al. Analysis of the automl challenge series, Automated Machine Learning, 177, 2019.

---

H. Hazimeh, N. Ponomareva, P. Mol, Z. Tan, and R. Mazumder, The tree ensemble layer: Differen- tiability meets conditional computation. In International Conference on Machine Learning, pages 4138 4148. PMLR, 2020.
X. Huang, L.. Zhang, B. Wang, F. Li, and Z. Zhang. Feature clustering based support vector machine recursive feature elimination for gene selection. Applied Intelligence, 48:594-607, 2018.
X. Huang, A. Khetan, M. Cvitkovie, and Z. Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
A. Kadra, M. Lindauer, F. Hutter, and J. Grabocka. Regularization is all you need: Simple neural nets can excel on tabular data. arXiv preprint arXiv:2 1/189, 2021.
J. Kennedy and R. Eberhart. Particle swarm optimization. In Proceedings of ICNN"95-interraationa conference on neural networks, volume 4, pages 1942-1948. IEEE, 1995.
J. Kittler. Feature set search algorithms. Pattern recognition and signal processing, pages 41-60, 1978.
R. Kohavi and G. H. John. Wrappers for feature subset selection. Artificial Intelligence, 97(1): 273-324, Dec. 1997. ISSN 0004-3702. doi: 10.10166500043377077000003333333 URL //www.
P. Kontschieder, M. Fiterau, A. Criminisi, and S. R. Bulo. Deep neural decision forests. In Proceedings of the IEEE international conference on computer vision, pages 1467-1475, 2015.
J. Kossen, N. Band, C. Lyle, N. Gomez, T. Rainforth, and Y. Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems, 34, 2021.
A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko. Tabddpm: Modelling tabular data with diffusion models. arXiv preprint 2rXiv:2209.15221, 2022.
C. Lazar, J. Taminau, S. Meganck, D. Steenhoff, A. Coletta, C. Molter, V. de Schaetzen, R. Duque, H. Bersini, and A. Nowe. survey on filter techniques for feature selection in gene expression microarray analysis. IEEE/ACM transactions computational biology and bioinformatics, 94): 1106-1119, 2012.
I. Lemhadri, F. Ruan, L. Abraham, and R. Tibshirani. Lassonet: A neural network with feature sparsity. The Journal of Machine Learning Research, 22(1)55633,-661, 2021.
R. Levin, M. Shu, E. Borgnia, F. Huang, M. Goldblum, and Goldstein. Where do models go wrong? parameter-space saliency maps for explainability. Advances in Neural Information Processing Systems, 15:15602-15615, 2022.
R. Levin, V. Cherepanova, A. Schwarzschild, A. Bansal, C. B. Bruss, T. Goldstein, A. G. Wilson, and M. Goldblum. Transfer learning with deep tabular models. The Eleventh International Conference on Learning Representations, 2023. URL bORuGUYošpA
wwt.e/wwwweeeeeemmmemeeeeeeeeeee wtt.:/rumteeetttt/to/me forum?id
H. Liu and I.. Yu. Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on knowledge and data engineering. 17(4)499-502, 2005.
J. Liu. Variable selection with rigorous uncertainty quantification using deep bayesian neural networks: Posterior concentration and bernstein- von mises phenomenon. In International Conference on Artificial Intelligence and Statistics, pages 3124-3132, PMLR, 2021.
Y. Lu, Y. Fan, J. Lv, and W. Stafford Noble. Deeppink: reproducible feature selection in deep neural networks. Advances in neural information processing systems, 31, 2018.
R. C. Madeo, C. A. Lima, and S. M. Peres. Gesture unit segmentation using support vector machines: segmenting gestures from rest positions. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, pages 46-52, 2013.
12

---

H. B. Mann and D. R. Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50-60, 1947.
Nam, J. Tack, K. Lee, H. Lee, and J. Shin. Stunt: Few-shot tabular learning with self-generated tasks from unlabeled tables. arXiv preprint Stt::23030009188, 2023.
R. K. Pace and R. Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3): 291-297, 1997.
A. Passemiers, Folco, D. Raimondi, G. Birolo, Y. Moreau, and P. Fariselli. How good neu- ral networks interpretation methods really are? quantitative benchmark. arXiv preprint arXiv:2304.02383, 2023.
S. Popov. S. Morozov, and A. Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv: wwXh::199.00612. 2019.
T. Qin and T.-Y. Liu. Introducing letor 4.0 datasets. arXiv preprint arXiv: 2011
I. Rubachev, A. Alekberov, Y. Gorishniy, and A. Babenko. Revisiting pretraining objectives for tabular deep learning. arXiv preprint arXiv:2207.088 0032008 2022.
J. Salojärvi, K. Puolamäki, J. Simola, L. Kovanen, I. Kojo, and S. Kaski. Inferring relevance from eye movements: Feature extraction. In Workshop al NIPS 2005, in Whistler, BC, Canada, on December 10, 2005., page 45, 2005.
N. Sánchez--aroño, A. Alonso-Betanzos, and M. Tombilla-Sanromáá. Filter methods for feature selection-a comparative study. Lecture notes computer science, 4881:178- 2007.
A. Shrikumar, Greenside, and Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pages 1145-3153. PMLR, 2017.
W. Siedlecki and J. Sklansky. A note on genetic algorithms for large-scale feature selection. Pattern recognition letters, 10(5):335-347, 1989.
D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 1706::/700603822: 2017.
G. Somepalli, M. Goldblum, A. Schwarzschild, C. B. Bruss, and T. Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01 2021.
M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR, 2017.
R. Tibshirani. Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267-288 1996. ISSN 0035-9246. doi: 10.1111/j.
URL httph:h/ts//ww.... jstor 1at.soo.rrr/staale/26666/1
T. Ucar, E. Hajiramezanali, and L.. Edwards. Subtab: Subsetting features of tabular data for self- supervised representation learning. Advances in Neural Information Processing Systems, 34, 2021.
Y. Wah, N. Ibrahim, H. A. Hamid, Abdul-Rahman, and Fong. Feature selection methods: Case of filter and wrapper approaches for maximising classification accuracy. Pertanika Journal of Science Technology, 26(1), 2018.
Z. Wang and J. Sun. Transtab: Learning transferable tabular transformers across tables. arXiv preprint arXiv:2205.09338, 2022.
F. Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80 1945.
ISSN 00994987. http://ttt////w/jjstttt httpsstoorrr/g/taaaoaeeeoo strg/stable/3001963
B. Xue, M. Zhang, W. N. Browne, and X. Yao. survey on evolutionary computation approaches to feature selection. IEEE Transactions evolutionary computation, 2004):00-662 2015.
13

---

B. Zhu, X. Shi, N. Erickson, M. Li, G. Karypis, and M. Shoaran. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305.06090 2023.
Z. Zhu, Y. Fan, Y. Kong, J. Lv, and F. Sun. Deeplink: Deep learning inference using knockoffs with applications to genomics. Proceedings of the National Academy of Sciences, 118(36)::2104681111 18, 2021.
14

---

A Limitations
In this section we discuss limitations of our work. We note that Deep Lasso feature selection is more computationally demanding than using GBDT-based feature selection methods, especially when large-scale hyperparameter optimization is performed. In fact, for this work we ran over 80 thousand experiments. Combined with computationally intense methods, this may have resulted in excessive carbon emissions. In addition, while introduce challenging feature selection tasks and use sizeable datascts, the scale of these datasets still smaller than truly large-scale industrial tabular data with billions of samples.
B Benchmark Detail
Our benchmark comprises of datasets used in [Gorishniy et al. 2021 2022). We select datasets with up to 150 numerical features and no categorical Teatures.
B.1
Datasets
ALOI Geusebroek et al.
2005 (AL, image data)
California Housing
Pace and Barry
1997
(CH, real estate data)
Covertype Blackard and Dean
1999
(CO, forest characteristics)
Eye Movements SSaloarrvi et al.
2005
(EY, eye movement trajectories)
Gesture Phase Prediction
Madeo et
2013 (GE, gesture phase segmentation)
Helena Guyon et al.
2019 (HE, anonymized)
Higgs 98
Baldi et al.
2014 (HI, simulated physical particles)
House 16H
(HO, house pricing)
Jannis Guyon al.
2019
(JA, anonymized)
Microsoft
QQn and Liu
2013 (MI, search queries)
Otto (T,, product categories)
Year [Bertin-Mahieux et al. 2011 (YE, audio features)
Table 4: Dataset Details. Number of classes is reported for classification problems.
AI.
CH
CO
EY
GF
HE
HI
HO
JA
MI
OT
YE
# samples
108000
20640
581012
10936
9873
65196
98050
22784
83733
1200192
61878
515345
# features
128
8
54
26
32
27
28
16
54
136
93
90
# classes
1000
7
3
5
100
2
4
B.2 Extraneous Features
In the random features setup draw extraneous features from the Gaussian distribution. After preprocessing original features with quantiler transformation. both original and extraneous features follow normal distribution. In the corrupted features setup randomly sample original features and corrupt them with Gaussian noise with the same standard deviation I. = 0.5z + 0.5e, N N/00, a). For generating second-order features we randomly sample two subsets of original features and compute their products.
bttpss//www.kggle.co//cc/
15

---

B.3 Data Preprocessing
We follow [Gorishniy et al. 2021 and apply quantiler transformation to normalize numerical features in all datasets, except for Helena and ALOI, where standardization applied instead. For the regression tasks we also standardize the target variable. Due to limited computational resources, we restrict the number of samples in the Microsoft dataset 500,000. We reserve 20% of samples for test set and 15% of samples as a validation set for tuning the hyperparameters of feature selection and downstream models.
B.4 Code
We include the benchmark code as well instructions for reproducing results in the supplementary. We will release our code as a GitHub Repository under the MIT License.
C Experimental Detail
C.1
Hardware
We ran our experiments on NVIDIA GeForce RTX 2080 Ti machines. Each hyperparameter tuning experiment took from to 72 hours depending on the model and the dataset size. Overall ran 756 experiments, each with hyperparameter tuning for 100 trials and 10 subsequent seed runs, which resulted 83160 training runs.
C.2 Implementation Licenses
For the model implementations adapt the code from the following publicly available repositories and libraries:
MLP and FT-Transformer models from RTDL repository under MIT License
GBDT model from XGBoost library under Apache License
Random Forest, Lasso and Univariate statistical test models from Seikit-Leean library
under BSD 3-Clause License.
C.3
Training Details
train all deep tabular models for 200 epochs with early stopping after 20 epochs, meaning that training stops if the validation accuracy does improve after 20 consecutive epochs. XGBoost models are trained with a maximum of 2000 estimators and 50 early stopping rounds. We utilize the AdamW optimizer and upply Linear learning rate scheduler after epochs 40 and 80. The batch size for experiments is set to 512. Other training parameters, including learning rate and weight decay, determined through hyperparameter tuning.
C.4 Statistical Significance
We run each experiment with 10 random model initializations (seeds) after we find the optimal hyperparameters for feature selection and downstream models. We report average test metries in Tables 3 as well as model ranks. When calculating the average ranks, we consider the statistical significance performance differences among the models. Specifically, we assign rank to top-performing models that do not exhibit statistically significant differences. We determine statistical significance with the one-sided Wilcoxon Rank-Sum test Wilcoxon 1945). Mann and Whitney [144 with p 0.05.
httpa://zgboost.at
httpe://sciktt-lermm.orgg//ee
16

---

D Hyperparameter Tuning
We carefully tune the parameters of all models for each experiment using the Bayesian hyperparame- ter optimization library Optuna Akiba et al. 2019]. For the feature selection experiments we tune parameters for both feature selection and downstream models simultaneuslyy to optimize for down- stream validation accuracy. We run 100 trials to identify the optimal parameters for each experiment.
adapt the hyperparameter search spaces from the original papers with slight modifications, which are provided below.
Hyperparameters for the MLP model are provided in Table
Hyperparameters for the FT-Transformer model are provided in Table Hyperparameters for the XGBoost feature selection model are provided in Table 7 Hyperparameters for Random Forest feature selection model are provided in Table Hyperparameters for feature selection regularizers (Lasso, Deep Lasso, First-Layer Lasso, Adaptive Group Lasso) are provided in Table
Table 5: Optuna hyperparameter search space for MLP
Parameter
Search Space
Number of layers
UniformInt(1. [1, 8]
Layer size Dropout
UniformInt 1,512
{0, Uniform 0.5
Learning rate
Log le 5, le 2]
Weight decay
{0, LogUniform[1e 6. 3]}
Table Optuna hyperparameter search space and default configuration for FT-Transformer
Parameter
Search Space
Default
Number of layers
Uniformint(1,
Residual dropout
{0, Uniform [0, 0.2]]
0.0
Attention dropout
Uniform [0, 0.5)
0.2
FFN dropout
Uniform [0, 0.5]
0.1
FFN factor
Uniform 2/3, 8/3]
4/3
Learning rate
LogUniform le le
le
Weight decay
LogUniform le 6, le 3]
2e - 4
Table 7: Optuna hyperparameter search space for XGBoost
Parameter
Search Space
Max depth Min child weight Subsample
UniformInt 10]
LogUniform|| le 8, 1e5]
Uniform (0.5. 1]
Learning rate
LogUniform[[L 5, 1]
Col sample by tree
Uniform 1]
Gamma
{0, LogUniformi [e - 8, 1e2
Lambda
(0, LogUniform le 8, 1e2
Table Optuna hyperparameter search space for Random Forest
Parameter
Search Space
Num estimators
UniformInt 10, 2000) Uniformint's 3, 10)
Max depth
17

---

Table 9: Optuna hy hyperparameter search space for Feature Selection regularizers penalty weights
Feature Selection
Penalty Search Space
Lasso
Uniform [Le 3, 5e 1]
Deep Lasso
LogUniform [ee 2, 5e
First-Layer Lasso
LogUniform le 2, 5e
Adaptive Group Lasso
LogUniform (1m 1e 3, 5e
For certain datasets training XGBoost (ALOI dataset) and FT- #T-Transformer (Microsoft, Year, ALOI, Covtype) models can be excessively time-consuming. Due to these time constraints, we not perform parameter tuning for these particular pairs of models and datasets and instead utilize the default configurations of the models. We also note that the computational complexity of attention mechanisms grows quadratically as the number of features in the dataset increases. Finally, LassoNet implementation performs hyperparameter tuning automatically, therefore we do not use Optuna to tune LassoNet model, and only tune parameters of the downstream models.
E
Deep Lasso for Linear Regression
Consider the case of linear regression, where the model is
(w, = Xw +
We follow the Syorch-styye notation and w is vector with entries where is the number of features columns of X. We show that Deep Lasso applied to the model output (rather than loss) is equivalent regular Lasso and First-Layer Lasso.
Proof:
Deep Lasso applied to the model output is given by:
min (Xx
Σ
(3)
ax
Now, for j-th feature (column of X), we have
(X)
0(Xw b
w. w.
ax
Ox(i
That is, get:
Of..b(X)
Σ
Σ
I In |1
Σ [wyl
In 1l F ||llivn,
(4)
axu
Finally, we get the equivalence of Deep Lasso to regular Lasso (with the adjustment of the Lasso penalty by vn):
o df Oxo)
Σ
m al
(5)
Additionally, we show the equivalence to First-Layer Lasso too. First-Layer Lasso applies Group Lasso penalty the weights of the first layer parameters:
min aL ao (1
Σ
1
where W is the j-th column of weight matrix of the first hidden layer corresponding to the j-th feature. However, in the case of linear regression, the first layer output coincides with the model
18

---

Table 10: Benchmarking feature selection methods for MLP and FT-Transformer downstream
models on datasets with corrupted with Laplace noise extra features.
FS method + Model
AL
CH
GE
HI
III
H1O
JA
M1
OT
YE
runk
50 No ES + MLP
0.942
-0.471
0.546
0.525
0.372
0.804
-0.616
0.704
-0.909
0.781
-0.797
8.27
50
Univariate MLP
0,955
-0.448
0.552
0.531
0.345
0.810
-0.627
0.717
-0.920
0.791
-0.827
7.00
50
Lasso MLP
0,953
-0.456
0.560
0.507
0.374
0.810
-0.594
0.718
-0.903
0.797
-0.789
6.55
50
Lasso + MLP
0,955
0.445
0.594
0.521
0.378
0.816
-0.722
0.721
00.902
0.798
-0.776
3.09
50
A MI.
0.955
-0,443
0.570
0.518
0.384
0.810
-0.577
0.721
-0,900
0.8
-0.782
3.36
50
LassoNet MLP
0,951
-0.451
0.554
0.551
0.387
0.812
-0.55
0.716
-0.904
0.797
-0.778
4.45
50
-0.905
0.795
50
MLP
0.953
-0.449
0,563
0,558
0.379
0.810
-0.583
0,723
-0.905
0.789
-0,787
5,00
50
XGBoost MLP
0.956
-0.454
0.570
0.385
0.813
-0.560
0.720
-0.894
0.8
-0.787
2.64
50
Deep Lasso MLP
0.957
-0.441
0.589
0.539
0.383
0.816
-0.556
0.720
-0,900
0.801
-0,776
1,91
75
0.765
7.73
75 Univariate MLP
0,950
-0.612
0,515
0.493
0.338
0.739
-0.621
0.688
-0.921
0.779
-0.838
7.64
75
Lasso + MLP
0,952
-0.447
0.545
0.482
0.376
0.811
-0.568
0.714
-0.902
0.796
-0.794
4.36
75
IL Lasso MLP
0,955
-0.4447
0.543
0.519
0.372
0.814
-0.596
0.714
-0.901
0.791
-0,781
3.73
75
AGL MLP
0.954
-0.443
0.550
0.519
0.382
0.810
-0.566
0.722
0.900
0.792
-0.784
3.09
75
LassoNet MT.P
0.947
-0.460
0.548
0.543
0.375
0.807
-0.575
0.712
-0.909
0.791
-0.787
5.09
75
RF + MLP
0.951
-0.450
0.564
0.524
0.367
0.805
-0.561
0.716
-0.902
0.770
-0.790
4.45
75
XGBoost
-0.896
0.797
75
Deep Lasso MLP
0.953
-0.443
0,552
0,527
0.377
0.812
40,557
0.72
-0.898
0.798
-0.78
2.00
output and has only a single logit. That is, the weight matrix W is in fact a single row vector w with m entries giving us the equivalence between the First-Layer Lasso and regular Lasso, and therefore between First-Layer Lasso and Deep Lasso too in the case of linear regression:
min (1 a) Σ 0
mjn min + (1 a) 11]ll
0
F
More Results
Tables 16
17 18 are similar to the tables in the main body, but they include the standard deviations computed across seeds for each experiment. We also report performance XGBoost baseline model for comparison, however we exclude it from rank computations.
F1 Corrupting features with Laplace noise
In addition the experiments with Gaussian noise corruption, conduct experiments with corrupt- ing features with Laplace noise. In particular, we sample additional features from the original ones and corrupt them with Laplace noise with the standard deviation original features:
Te = 0.5x + 0.5ore, E ~ Laplace (0,
V2
We include results for this setup in Table 10 corruption we observe that Deep Lasso along with XGBoost feature selection achieve the best downstream performance in terms of average rank.
Similarly to the experiments with Gaussian noise
F.2 ROC-AUC, Precision and Recall Metrics
One common way of measuring performance of feature selection methods is using ROC-AUC, precision and recall scores, which indicate how often these methods choose "correct" and "incorrect" features. We instead focus on the downstream performance metric since in the case of corrupted and more prominently second-order extraneous features, it is not possible to treat feature selection as binary classification problems because additional features may contain useful information. However in this section we report precision, recall and ROC-AUC scores for the setup with random features, which can be reasonably treated as binary classification problem. Tables [2 present ROC-AUC. and precision scores correspondingly with ranks computed taking importance into account. We note
19

---

Table 11: ROC-AUC scores for feature selection methods trained on datasets with random
extraneous features Bold font indicates the best numbers dltaset-wise and lower rank indicates better overall result.
%
FS mothod
AL
CH
C0
BY
GE
HE
HI
HO
JA
MI
OT
YE
rank
50
Univariate
1.0
1.0
1.0
0.916
0.926
0.982
0.866
1.0
0.706
0,993
1.0
0.990
2.50
50
Lasso
0.988
1.0
1,000
0.795
0.371
0.976
0.847
0,920
0.735
0.596
0.991
0.965
5.00
50
Lasso
0.785
0.995
0.969
0.789
0.513
0,980
0.858
0.879
0.716
0.947
0.952
0.998
4.58
50
AGI.
0.972
0.827
0.862
0.870
0.471
0.999
0.786
1.0
0.746
0.925
0
0.999
4.25
LassoNet
1.0
0.999
0.998
0.862
0.979
50
AM
0.848
0.992
0.752
0.765
0.770
1.0
0.876
0,980
0.759
0.704
0.972
0.988
4.33
0.906
1.0
0.898
0.884
0.998
1.0
0.874
0,934
0.729
0.559
0,995
0.891
4.17
50
XGBoost
0.945
1.0
0.944
0,993
0,943
1.0
0.890
1.0
0.75
0,968
0,992
0.988
2.33
50
Deep Lasso
1.0
0.998
0.981
0.927
0.482
1.0
0.922
0.993
0.699
0.957
1.0
1.0
2.33
75
Univariate
1.0
1.0
1.0
0.918
0.872
0.986
0.846
1.0
0,731
0.993
1.0
0.991
2.08
Lasso
1.0
1.0
0.976
0.781
0.976
0.837
0.858
0.77
0.597
0
0.943
4.08
75
Lasso
0.979
0.998
0.932
0.841
0.477
0.983
0.784
0.910
0.739
0.830
0,963
1.0
4.33
75
AGL
0.922
1.0
0.862
0.815
0.492
0.986
0.791
0,908
0.726
0.891
0.831
1.000
4.92
75
LassoNet
1.0
1.0
0.958
0.582
0.531
0.965
0.821
0.974
0.613
0.448
0.589
0.931
5.08
75
Forest
0.927
1.0
0.957
0.916
1.0
1.0
0.88
0.990
0.730
0.700
0.998
0.942
3.50
1.0
0.948
0.700
0.997
0.737
0
0.999
2.92
75
Deep Lasso
1,000
0.994
0.985
0.919
0.273
1.0
0.895
0.995
0.762
0.959
1.000
1.0
2.25
Table 12: Precision scores for feature selection methods trained on datasets with random extraneous features Bold font indicates the best numbers dataset-wiss and lower rank indicates better overall result.
mothod
AL
CH
CO
HE
H1
HO
rank
Univariate
1.0
1.0
1.0
0.808
0.844
0.963
0.786
1.0
0.648
0.986
0.967
2.75
50
Lasso
0.987
1.0
0.989
0.704
0.403
0.963
0.750
0.875
0.661
0.588
0,969
0.922
5.00
50
Lasso
0.776
0,988
0.954
0.719
0.497
0.937
0.811
0,856
0.644
0,882
0.915
0.958
4.67
AGL
0.946
0.738
0.794
0.777
0.989
0.704
1.0
0.665
0.870
0.875
0.987
4.50
50
LassoNet
1.0
0,988
1.0
0.619
0.625
0.985
0.771
0,931
0.644
0.582
0.692
0.908
5.00
AM
0.768
0.962
0.715
0.789
0,938
50
RF
0.901
1.0
0.893
0.885
0.984
1.0
0.807
0,875
0.661
0.553
0.985
0.809
4.25
XGBoost
0.945
1.0
0.944
0.969
0.894
1.0
0.818
1.0
0.981
0.956
0,992
0.977
2.17
50
Deep Lasso
1.0
0.988
0.974
0.835
0.488
1.0
0,861
0.969
0.641
0,950
1.0
1.0
2.67
Univariale
1.0
1.0
75
Lasso
1.0
1.0
0.957
0.608
0.766
0.963
0.679
0,812
0.622
0.401
0.963
0.878
3.75
Lasso
0.882
0.975
0.909
0.251
0.948
0.632
0.881
0,531
0.721
0.953
0.996
5.17
75
AGL
0.856
1.0
0.743
0.685
0.253
0.952
0.654
0.869
0.541
0,775
0,704
0.987
4.75
75
LussNNet
1.0
1.0
0.976
0.458
0.488
0.919
0.650
0.912
0.415
0.331
0.524
0.770
5.08
75
0.918
0.789
0.912
0.559
0.836
75
XGBoost
0.938
1.0
0.946
0,873
0.994
1.0
0.643
0.938
0.567
0.993
0,974
0.981
2.67
Deep Lasso
0.990
0.957
0.775
0.956
0.576
0.946
that in our setup the recall numbers are identical to the precision numbers, since we always "select" top-k features, where k is fixed. Therefore, the number of false positives is equal to the number of false negatives. We observe similar trends to results in the Table where we evaluated algorithms with respect to the downstream performance. In particular, GBDT based approaches, univariate statistical test and Deep Lasso outperforming other considered feature selection methods.
F.3 Does feature selection help bridging the gap with GBDT models?
In Section 4
we demonstrated that neural networks are more sensitive to noise than GBDT models, particularly the MLP architecture. In this section, we analyze whether feature selection helps narrow the gap between neural networks and GBDT models in the presence of noisy features. In Tables [3]]
15 14 we present results for the baseline XGBoost models, neural networks trained without feature selection, and neural networks trained with Deep Lasso feature selection, based on our benchmark. We observe that with feature selection, MLP models perform comparably to the XGBoost models in second-order setup and significantly better in random and corrupted features setup, while transformer models consistently outperform GBDT models in terms of average rank.
20

---

Table 13: Benchmarking Neural Networks with feature selection against XGBoost models
datasets with random extra features.
FS method
AL
CH1
CO
EY
GE
11E
111
110
M1
OT
YE
rank
50
XGBoost Baseline
0.838
-0.416
0.951
0.665
0.613
0.360
0.728
-0.562
0.725
-0.885
0.801
-0.797
1.83
50
0.941
-0.480
0.961
0.538
0.466
-0.622
0,703
-0.91
0.773
50
Doep Lasso MLP
0.959
-0.443
0.968
0.573
0.485
0.383
0.814
-0.549
0.720
-0.894
0.802
-0.776
1.42
75
XGBoost Baseline
0.839
-0.42
0.941
0.626
0.583
0.349
0.726
-0.57
0.718
-0.894
0.796
-0,802
1.92
15
No FS MLP
0.925
-0.527
0.955
0,502
0.417
0.348
0.778
-0.674
0.671
-0,917
0.749
-0.812
2.67
75
Deep Lasso
-0.446
0.569
0.479
50
XGBoost Baseline
0.858
-0.416
0.951
0.665
0.613
0.360
0.728
40.562
0.725
40.88
0.801
00.797
2.25
50
No
0.959
-0.432
0.966
0.673
0.500
0.384
0.817
-0.577
0.730
-0,902
0.813
-0.792
2.08
50
Deep Lasso FT
0.962
-0.419
0.969
0.703
0.504
0.392
0.817
-0.56
0.733
-0,900
0.817
-0.788
1.17
Table 14: Benchmarking Neural Networks with feature selection against XGBoost models on datasets with corrupted extra features.
%
PS method
AL.
CH
CO
EY
GF
HF
HI
HO
OT
YE
rank
50
XGBoost Baseline
0.854
-0.415
0.946
0.666
0.618
0.359
0.726
40.559
0.723
-0.891
0.796
-0.797
1.83
0.557
00607
0.703
-0.797
50
Deep Lasso MLP
0.955
-0.447
0.968
0.577
0.525
0.388
0.815
40.567
0.721
-0.895
0.801
-0.776
1.58
75
XGBoost Baseline
0.830
-0.425
0.929
0.625
0.58
0.721
-0.573
0.719
-0.894
0.795
75
No ES MLP
0.921
-0.516
0.956
0,518
0.503
0.356
0.788
00.632
0.686
-0,913
0.762
-0,808
2.67
Deep Lasso
0.554
0.718
50
XGBoost Baseline
0.854
-0.415
0.946
0.666
0.618
0.359
0.726
40.599
0.723
-0.891
0.796
-0.797
2.25
50
No
-0.430
0.967
0.686
0.576
0.386
-0.574
0.731
-0,901
0.809
2.17
50
Deep Lasso + FT'
0.961
-0.422
0.968
0.725
0.577
0.393
0.821
-0.561
0.736
-0.898
0.809
-0.788
1.25
Table 15: Benchmarking Neural Networks with feature selection against XGBoost models on datasets with secondorder extra features.
%
ES method
AL
CH
CO
EY
GE
HE
HI
HО
MI
OT
YE
rank
XGBoost Baseline
-0.429
0.969
0.709
0.683
-0.541
0.726
-0.846
0.825
50
No FS MLP
0.960
00.443
0.969
0.631
0.605
0.383
0.811
00.49
0.719
.0,891
0.800
.0.786
2.42
50
Deep Lasso
-0.441
0.969
0.648
0.600
-0.572
75
XGBoost Baseline
0.917
-0.414
0.968
0.74
0.691
0.376
0.731
-0,542
0.725
-0.849
0.825
-0.789
1.67
0.952
0.969
-0.542
75
Deep Lasso MLP
0.959
00.488
0.969
0.647
0.582
0.378
0.821
0.568
0.74
00,890
0.805
-0.78
1.67
50
XGBoost Baseline
0.923
-0.429
0.969
0.709
0.683
0.374
0.729
40.541
0.726
-0.846
0.825
-0.781
2.00
50
No FS + FT
0.962
-0.425
0.968
0.733
0.558
0.391
0.819
-0.552
0.732
-0,901
0.818
-0.790
2.25
Doep
-0.422
0.726
0.608
0.738
21

---

Table 16: Benchmarking feature selection methods for MLP and FT- Transformer downstream models on datasets with random extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to pervent of extra features in the dataset; either 50% or 75% features are second sccond-order -order. Bold font indicates the best numbers dataset- wwise--wsee-eeee and lower rank indicates better overall result.
rank
8.08
2.66
5.91
4.91
4.33
3,83
2.91
2,33
2.33
7.41
2.50
4.35
4.41
4.33
4.75
2.91
2,08
2.35
6,58
2,33
6,33
5.16
5.25
4.83
3,75
1.75
1.91
3.66
-0,781 +000
-0,801
-0. 4..77650 001
0.767400D0)
-0. 0.773360.00
-0. 40.778500000 001
00.767A00002
-0.77760.000
002
0.786:2000)
0..77160.00 001
0.776668.0)
40.3 10.799000
-0.8 2240 00)
-0. 0,776+00
-0.79919.00)
-0.7 -0.775 2012
0.777=0000)
-0. 0..99420 002
0.764:0001
-0. 00.77460.00)
-0. 0..77450.00)
0000
-0. 0..79210.002
00.789240001
-0.998+0
-0.791
-0.8.2210.0022
-0.791 0.791
40789=50000
-0. -0.79 001000
0.709940.00)
-0.78880.000
000
-0.
YE
0.825e0 00)
0.773+0
0.808550000
0,805540 0000
0.796160 008
0.79940.000 0OS
0.7K366.00 00G
0.801 0.801 180 O03
0.80000 002
0.805L0 003
0.802+0.000 0000
0.825 001
0,74940 003
0.50GE0.000 0003
0.80210.00 003
0.79968.003 OO
0.762+0 035
0.768E0 OUS
0.80N00.00
0,80440 002
0.8:00 0022
0.81 0.81320000 OUS
0.81940.00 000
0.81 0.81 151700 001
0.81900002
0.8 1.81419.00 0.00
0.808640000 0002
0.81560.00 002
0.82:0.001
0,82400022
0.81 002.
OT
-0.84600000
-0.91 0020
-0.991
(.9074D.000
-0 897=0.0
-0 8..9000000
001
-..07=0.0001
-..90400000 DG3
-0.9040000
-0.993L..00 001
0.894E:.00) 000
-0.84920.0
-0.91 197000000 001
-0 -0.8P 001
-0.90710.00
-09903 IXI3
-0.00
-0.9 10150000 000
00.02EDD
-0.89320.00 0000
-0,90220 04
TORH01002 -000000000
-0.999000
2003 00.1510.096
004
2 -0.90330
001 -0.9=0.004
-0.9=0.005 000
-0
MI
0.733EM SOII
0.726600000 001
0.703=0.000
0.715a0.002
.002
0.71540.003
0.71540.000 000
0.7 0.718:00000 003
0.715An,104
0.722=0.000 0012
0.72+9 001
0.7240.000 0GG
0,72+00003
0.725
0.671 651
M0.002
0.7 0.713+0.00 001
0.71610.902
0.717
0.71
105H0000
0.75m0 002
0.727
10S 0.
1002 0
0,737
0.734
0.733H0.002
0.7
-0.541 002
-0,6220 007
-0. 90.549250.000 0MS
0.599400003 DE3
-0 0050840 DOS
-0.55740 003
-0.557ENDDNS
0156660012
40.7229
00.5620.0144
40.54H0A005
-0.54220000
002
-0,67400
007
-0.5HH0
008
1S11:10011 -0.112100
03.899 1001 40.52HH0
2
03909-0 003 10.550
-0 M.SSHLOTT
009
-0
113777 000
00:000 1.
1 SOSHIIS
0. 002 -0
| 40
005
40.86000122
101 -0.55
-0.
0.81::000 -0.
Hо
0.729=0.0m 001
0.798=0 00)
0.81 0.81 1:: 001
0.812:000
0.81 100010000000 00)
0.8 0011000011
0.81 0.81 181120000
1.813:00022
0.814:0001
0.812:0.0
0,814:0000
100
0.731=0.000
000
0.778-0 002
0.81=00001
00
0.80233
10MEERT11
0.315000
1
001
0.81
E00 0222220000
0.2200011
0.821 001
0,82000011 001
0.817-0.00 005
03.90
0.8
0.81
0
0.374:0.001 001
0.306=0.000 001
0.379:0 002
0.38400022
037550.003
1230OZ0003
0. 02392-00002
0012
0330220002
13N5510022
03363400002 0012
03376297.01 0010
0. 0.34840.002
002
0.38) 181=0000 002
000
005 000
0=7900.007 7 30720001
033-221-0
00392:0-000
000000000
033925000
002
HE
0
20/12 0
0.68340.000 003
0.406+0
0,515220.00 000
0.45:40.015
0.474+0.015
0:473:0.018
0.49520.01
0.498=0.00
0,594:00023
0.302
0.48=00.066
0.691 +0.008
0.1 1417400000
00.000
0502:0 005
0,29110.009
0.32:0005 05
OF
055340001
05:0000
1506500000
040 05505000
0504000011
0013 0:
0672000 1150
0
0,70940.003 003
0.538+1 005
0.57580.01 01
0.47700016
0,5640.0119
0.57800017
0.55240.006
0.55450.016 0
0.589=00011
0.59200155
0.573=0.007
0.7430.002
0.50240.000
007
0.57550 008
0.51100013
0.555500015
0,50-40003
0.22800000
000
0,57640000
007
0.569:000
0.67390015
0,7440000
0187740088
0.2200111
0,5013001
0,701-000
1
0.55:40044
0.718
0.725
0,70300011
0
0.909=000
0.96 (:06:00000
0.97a0.0
0.96=00001
109000000
0000000
119691000)
O. 0096000001
0:900000
0.969100
0.968=0001
0.968=00
0.955=0.001
0.97+0.0
0.967-00
1.96800001
0.968-0000
0.96620002
968600000
0,900000
0000000
0.966:0000
0.9780111
0.964101
0. 900-00
(19691000)
0.9780.001
0090000000
0,9600000
0.969-00001
009000000)
40 00422s0.000 001
40 007
-0 00.447500000 004
0 00454:0.004
-0 4545 1400000000 003
-0.51 0.51220.064 064
41.455000004
-1. A1444400IN
4.453200003
-0. 41.441000022
A1.415000002
40.4 420.0300000
006
40 0-52740.00
0.44750.004
00002
9-45410.90
4.447=00003
8045220.000
000
-0 0:5510000
0.446200004
40.43220.002
0,41930000
-0 0-4220000
4042500000
40-2200003
M-42200002
4.41950.000
-0.43
CH
0.923-0.0
0.9620.000 001
094940.001
0.55=00.00
0.958:0.001 001
0.954a0.00)
095320.001 001
0.955=0.001
0.956.10.011 001
0.959a0.001
0.91700.0
0925=0.0072 002
0.96H-0.00 001
0.95910.00
0.957 2010
0.54.0..02
0.95 001
0999200001
0958-0.011
0.95700001
0.95920.00 002
8.63200.011 0011
0952=0.003
0963:00001 001
0.89910.02 023
8963200.00 001
0.96200.001
0.96320.001 OU
0.96340.00)
1997-00001
0.94
AL
method
XGBoost Baseline
Univariaee + MLP
+ + MLP
IL. Lasso MLP
AGL + MLP
LassoNt MLP
AM + MLP
RF MLP
XGBood + MLP
Deep Lawa + ML.P
XGBoodt Baseline
No IS MLP
Univariae + MLP
Laso MLP
IL LaNNO + MLP
AGL + MLP
LassoNet+ + MLP
RF MLP
XGBoest MLP
Deep Lasso - MLP
No FS POFS-T FT
Univariate FT
Lasso FT 50
IL Lasso FT
AGL FT
LassoNo FT
AM + FT
RF+ FT
XGBood FT
No PS
Deep
50
50
50
50
50
50
50
50
50
75
75
75
75
75
75
75
75
75
50
50
50
50
50
50
50
50
22

---

Table 17: Beuchmarking feature selection methods for MLP and FT-Iransforme downstream models on datasets with second second-ordes -order extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of extra features in the dataset: either 50% or 75% features are -orde..orddr. Bold font indicates the best numbers datasel dataset-wise lower rank indicates better overall result.
wwie and
rank
8:00
7.33
5-42
4.75
3.00
3
3.58
4.67
2.67
2.58
7.58
7.50
5.00
3.25
4.92
4.33
4.92
2.58
1.42
6,08
6.17
6.42
3.50
4,08
3.92
3.42
3.42
2,42
2.67
-0. 00,97760.0
00.797=00000 00)
-0. -0.828850 001
0.795:+0.00)
-0.78=0.004 0.000
-0. 0..7ED00001
-0.774
-0.7Ne0.002 0.00
-0. 0.786G20000
-0. .0.77910.000
001
.0.77/660.000
-0.800 =0.800
-0. -0.808-00 00)
-0. 0.8.38::00 001
-0.779 0001
-0.78=0.001
00.7855000000 002
0.708440000
-0.78440 002
40.778600000
-0. 0001
-0. 0.793200 001
-0.8266000
.0.91540 00)
0..955+00004
004
-0..5-0.015
-0.79%0.MM 2011
-0.79=0.000
-0. -0.793150000
002
.0.79.0.00)
-0.788860.00
000
-0.79
YE
-0.
0.796.0 00)
0.778=0 004
0.795 001
0.796:0 0.000
0.795=0 002
0.796: 002
(1795.0.004
0.797=00003
0.788 002
0.8013
0.801=00002
0.795
0.762-0 003
0.779:0 002
0.595 0002
0.797-0000
0.788=0000 0002
001 0.79 0003
0.765-9000
2003
0.800
0002
0.812 0002
0.817:00000
002
0.816:0.00
0.809=0 003
0.8+0.003
0.809
0.812
MA 0.806
Or
-0.891 #0.00)
-0,909000000
0011
-0.9240000
009003900
-0.90240.000
000
0.72240002 0012 -0,90220.00) 1010
-0.95599 001
-0.0022000
0.12210082 2002 10.92H0NI 101
2000 -0
40,89450.00 XX1
-0 100
-0
0O7R:1100 -9.80
-0.91
000
0 2012 SOREENA
HN
SOTEI1 0 002 S03PRREAN
TEM 0.736H0000 -0.5%50000
40.91
0 003 1
MI
0.72360.00 001
0.703=0.00
0000
0.71720.002
0022
0.71340.00
0.71840.033
0.79::.001
0. 0.68660 20
0.71720100
0.731201011
0
0.7
-0.55930.00 002
-0,007+
005
0,622000004
002
006
10 00
-0.5T
M.SSSHIIT
2112
-0.56
004
-0 -0.55810 INN
-0.5770 00S
40.57350000 001
40,63220 DIS
-0.020
004
0:81 1=0001 11.7720
A0 COD 1
-0.58100133
0.60220
-0.58150
-0.561
-0
HО
-0
1000 -0.50
1 -0.
0.726=00 001
0.502=0.002 002
0.81 0.81000000
0.813=0.00) 000000
0.812=0000
0.813:0.00 001
0.81 100,002
0.81
0.81200002
0.815
(L815=0.001
0.721=00002
0. 0.768=0
O. 0,742
001
0.81
0.81
0.81--0001
0.36200000 0.812200
0. 1 0088112
0. 002 002 0-
1
1
0.31
0.30
0.
0. 0.359#0.00)
009
0 0.37=00001
0. 1.346600000 003
0. 1.366200000 0012
0. 00382-41 082-00000 002
0. 3..662000001
0. 0.30220.002
0. .8 100000002
0 (.388390000)
0. 0.3655.000003
01.08800.002
0. 0.355:500000 0011
0. 0.35640.002 002
0. 0.347:00000 002
0.37310.00
0. 1.332600022
003
0. 383320
002
0.3722 2013
0. 38 150
0.30020000000 02000000
0. 20220.002
0. 044450000
0. 0
0.87-00000
000000 000
0 0000000000
11. 1RR3EE0
1992000011
0.3935500002
0
HE
0.61 R40 .00
0.525+00000
0.51420 005
0.51240.011
0,5 511.510.11 08
0.38810.000
000
0.55640.009
0527+0.009
099
0.568:0011
0311100009
0.52540.015
0.5Re00002
0 0.50340.005
005
0-005+0.005
005
0,2910011
0.49550.022
0199:0000
051340/011
0,532000 1
0.54-00013 031750.000
0.5762)).222
G57440101
05500
100000
0.8400117 7 (5572000
80
06200000
6664000099
0.564401000
057740.014
1000111 0
0577
0,50
0.666600000 003
0.557 007
0.53620.00
007
0548400117
0,56640.01
0.588:00015
0.5730016
0,583200012
0.574:1100
0.583200111
0.57740011
0.625-6..00
0,51840.01
0.5 051=00008
0.56EHA0
0.5-9
101
0.38--1000
0.47500111
0.571
0.68620000
0.936-0005 037 1197-100222
0718-0000
0:11100116
0,70650011
0,72520008
0.968:00 0.55-2005
0.990- 001 07
01960900000 1000010011 0
0.9466:0.0
0.965=0 001
0,966:20 001
0.968:00.00 001
0.908=00 00)
0.966350 001
0.99
0:967=0.00 001
0.967=0.00) (0)
0,9692000
0.968:000
0.9294000
0.956-0 001
0.94 00)
0.965 000
0,9674901
0.267 000
0:969=0001
0.9633:00000 100000
0,988:00 00)
0.967 001
1001
0.965550000
01902-000
0,969 001
0:97400
0.988-0.00
00)
CO
00415H0.00
001
-0.475:0
004
-0,451 004
-0447740.00
000
00,55:0.005
-0.43300000 003
40.45410.00 063
11447500003
-0.425540000
-0.51 00.16600000 M4
-0.5000 004
-045510
0.928=0.0155 005000
199
0.94730.001 100 -0,45200
006
015550000
038
-0-41:000
-0.13:0000
0 062200000 001 10
N.96220.00 40442220000
1 -0-2669166
0002
-0,42240
0. 0.96220.001 -0-42HD
004
2002
-0,42250000
-0.41
CH
0 0.85000000
0.946=0.001
0.95520.00) 001
0.955a0.000 0000
0.985=0.000 2012
0.955:0.000 001
0L95540.00 1011
0.95520.000 001
0.951=0,001
0.954100001
0.955s0.001
0.92100.001
9555000
0.9081100001
955
99524000011
0. 0.5-000001
0.95920.001 0011
0.96E0. 00]
0.96320.011
95250000 00
0.90610.021
0.962=0.001
0.96340.001
0 969 00.00000001
0.8 0.8360.0
A1
XGBoost Baseline
Univarisee + MLP
IL. Lasso MLP
LasooNt+ MLP
XGBood + MLP
Deep Lassa + ML.
XGBoodt Baseline
Univariae + MLP
IL LaNNo + MLP
LassoNec+ + MLP
XGBoost MLP
Deep Lasso - MLP
method
No PS MLP
Lasso + MLP
AGL + MLP
AM + MLP
RF MLP
NS IS MLP
Laso MLP
AGL + MLP
RF MLP
No FS POFS-T FT
Univariate
Lasso FT 50
IL Lasso FT FT
AGL FT
LassoNNt FT
AM + FT
RF+FT FT
XGBood + FT
Deep Lasso
50
50
50
50
50
50
50
50
50
75
75
75
75
75
75
75
75
75
50
50
50
50
50
50
50
50
50
23

---

Table 18: Beuchmarking feature selection methods for MLP and FT-Transtormn downstream models on datasets with second second-ordes -order extra features. We report performance of models trained on features selected by different FS algorithms in terms of accuracy for classification and negative RMSE for regression problems. % refers to percent of cxtra features in the dataset: either 50% or 75% features are -orde.. Bold font indicates the best numbers datasel dataset-wise wwie and lower rank indicates better overall result.
rank
4,50
6,58
5.33
3,92
3.25
4.50
3.50
3.25
3,83
2.67
4.92
7.25
3,67
3,33
2.67
4.85
5.08
5,00
2.17
4.17
6.67
5.25
2.83
4.17
5.00
3.83
3.67
4,33
242
-0.781 00.0
0.776++00000 001
-0.81 0000
0.783+00
-0.78=0.00)
-0. 4..77620.00)
0001
0..764100002
-0. -0.778560000
002
-0.781+00000
-0. 0..86660 001
-0.77666
-0.3 40.7
-0. 0.792240000
002
-0. 0.82200001
-0.79419.00
-0.7N2 1000
-0.781
-0. 0..86650 002
0.719940000
-0.79540 00)
-0. -0.79e0.00 0001
0.7920.001
10.81990000
001
0.816200 -0.7991
0002 -0.87400
1817- 002 0.851199.11
004
1
-0.79
0022 00.79240000
002
10210001
-0.79240000
1000
-0.78990.000
-0.
YE
0.825 00)
0.8=0 05=0,003
0.798:0.000 002
0.8006:0.001
0.808
0.806:000000
0003
0.797-9.000 00002
0.8012500000 003
0.801 002
0.80S TT 0022
0,805550.000
0.825=0.000 000
0.792-0 003
0.785:0 002
0.80740001
0,805
8032000 0.8+0
0.793 003
01794-0000
0. 2002
0.803:0 003
0.805 0002
0.818
002
200
2002
1XXH10 202 -0000000000000000 00 0.816
0.81950000
002
0.812 000003
0.819-0 00)
H 0.8
OT
-0884600.0
-0.891 000000
-0.90240.0 0
00.911200
-0.99:000
-0.89
1001
-0.89299
OX22
408000
-0.891
000 1891110001
001
40,34940.0
-09-00001
0. 6.720 002 -
409012000
S800E 00)
001 - 2
101 -0.99500000
3
-
01 0.74:0000 100000 -0
MI
-0
0.726680.00 001
0.719:0.002
002
0. 0.724g0 002
0.72640007
6.728-0.00)
0.73 000
9.72480,033
2012
0-735:000
0.
0.725
0.71440002
2NN
0
0.73890.00
0.7346
-0 004 0
POS 424 0
0.7
-0.541 002
0.549+0.000
-0.61420
009
DN
00.6440
00585500 01
-0.50549
007
033
00.7910011
-0.57200000
40.54220000
002
-0. 0.54LM
(017
1001 10.599E0
02 AASISEEE
100 CO00 CONAIIAA SASSS
-0 SALTAA 01
SORZEE 000
DOS
40.58M
0.812:0 001 -0
HО
-0
-0.
0. 0.729=0
0.81 1=00
0.81750.00 001
0.816:0.000
0.815=0.077
O. 1.807000000 0007
0.81G=00002
(1.1 1.819:00011
0.818=0001
0.81810.011
0.815=0.00) 000
0.731=0.001 000
0.808=0 00)
0.82101001
0:818:00001 1000000
1
0
II
033-0000
10000001
(001
0,22200000
00822-000
0.81
0.881
600 00
0.
0.
0.
0
0.
0
0. 0.374+0.00
001
0.383=
0. 0.357:00000 0011
0. 0.366:000077 0007
0. 0.38+00.03
O. 1..35::50000 0002
(.379500000
0. 18 1000000000 0013
0 0.37200002
0. 0.373400000
0.384=0.001
0. 0.376e00000)
0000
0. 0.N55=00001
0 0.35200 002
0. 1.363400000
1 0600- 003330 2000
0 18220
035500000
0555500012 0..770000000
1100000
000000111 002
0.38340.00
HE
0 35
0.
0
0,08340.000 003
0.005+00012
0.38240.01
0.59:0022
0,57 40.06
0.59400.019
061110110
060=+000
1001
0,691 0,691 +0.000
0. 0.59840.000 006
40.005
0579201000
1
060050.011
064400
0.53 0.53
GE
20000 0
2 O
0,709 003
0.63 0,631+10000 000
0.58420.011
013
0.508500122
0.634-0.014
0487:00015
0115
0,64110011
0.62241100 0199
0.639:t WWS
0.635 100
0,7400002
0.63-0006
0.53-0008 008
085120022
0647:0003
CUURREEE
2
1
02262000
100000
TI
0.909=0.0 0
0.909>=0 0000
0.959
0.986:0.000 0004
0,969-0000 000
0:953:0052
01969910
0.988 001
0.9002000
1M911061
(1969:40000
(001
0.968:000
0.909=000
0.488:000
0.9081000
1196650000
0998--000
0.998 001
01963550000
007 00000000000000000000000000000000
100850000
000
0.988
CO
10442900000 001
-0.443:0 002
0.439920000 002
0-44740.00 004
00445550000 00S
-0.443::0 008
-0-457
-0,438::00022
40.41 15:111550000 AM33
-0,45 003
-0.5 0.5.:00.00
00+
-0443400.00
1002
-0.45250. D0
00438=0 OON
40.455300000 027
-0-445 MNOS
-0.49550000 050
0.:48:50.000
-0.425:0
HO
001 0141854NN0
0.963 001 40-22200
0.964=0401 10T -
0.871=0101 -0
CH
-0.4
0.923-0.0
0.96+= 00)
0.96120.0o 001
9555=0.001
0. 8.95000.0
0.961 0090) 001
0.961 001
0.958:00,001
0.8 0.8710.001 001
0.961
0.917=0.0
0.95200.01 100
0 96:0 001
0.9611000 001
N.9664L7001 1001
0.958=00000
0.938000011
0995600.001
0450-0.001 1
0.95220001 1
0.96200000 001
0.96320.001
0 9867-100
0.963=1000
0.963=00.00 1
0.
AL
XGBoost Baseline
Univarisee + MLP
IL. Lasso MLP
LasooNt+ MLP
XGBood + MLP
Deep Lassa + ML.
XGBoodt Baseline
Univariae + MLP
1L LaNs + MLP
LassoNet+ + MLP
XGBoost MLP
Deep Lasso - MLP
method
No FS -MLP
Lasso + MLP
AGL + MLP
AM + MLP
RF MLP
No IS MLP
Laso MLP
AGL + MLP
RF MLP
No FS POFS-T FT
Univariate
Lasoo FT
IL Lasso- + FT
AGL+ FT
LassoNNt FT
AM + FT
+F+ FT
XGBoost+ FT
Deep Lasso
50
50
50
50
50
50
50
50
50
75
75
75
75
75
75
75
75
75
75
50
50
50
50
50
50
50
50
50
50
24


Reviews:
[{"review_id": "LplBb60kDB", "reviewer": "Reviewer_Lpp4", "summary": "", "questions": "", "limitations": "There are no negative societal impacts.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors go beyond the standard Gaussian error variable to assess the importance of variable selection. The proposition of other sources of erroneous variables opens the avenue for describing nuisance variables. Also the deep lasso idea is an interesting approach to design variable importance.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "frtvA9or8c", "reviewer": "Reviewer_rBeS", "summary": "", "questions": "", "limitations": "I dont see a limitations subsection.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "\n* **[Application Domain]** Tabular Datasets are very common in many practical situations (e.g., raw data from students, patients, financial records, and many more). While most modern ML methods are geared towards modality-rich data, we should encourage research on tabular datasets that are common in-practice. \n\n* **[Comparison of Feature Selection methods]** The paper develops code to compare a number of feature selection methods. They releases their code. This allows future researchers to easily compare their own feature-selection method with any method implemented in the provided code.\n\n* **[New Feature Selection Algorithm]**. Paper proposes a new method for feature selection. Specifically, a regularizer is added to the main (e.g., classification) objective that encourages the sum of L2 norm of columns of gradients to reduce. They mention that their method becomes equivalent to first-layer lasso in case the network is only 1 layer deep (though a theoretical proof is warranted, even if it was straight-forward)", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "V1VnWl57BL", "reviewer": "Reviewer_rx9X", "summary": "", "questions": "", "limitations": "The authors provide details on the limitation of the proposed algorithm from a computational perspective. I would add that the size of the benchmark might potentially be sub-optimal (12 datasets), as related work shows that there is no consistently outperforming method on tabular data, and model rankings might widely differ from one dataset to another. As tabular datasets are generally diverse, a larger collection of datasets might capture a broader range of their properties. I suggest the extension of the proposed benchmark to more datasets as a future work step.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed benchmark takes into consideration different types of extraneous features, as well as including varied datasets in terms of number of samples and feature size, for both classification and regression tasks.\n2. The feature selection methods used in the evaluation cover a wide range: statistical tests, tree model-based feature importance, attention-based and regularization coefficients.\n3. The proposed Deep Lasso, although a more computationally demanding method compared to the other evaluated methods, provides a consistent improvement in scenarios considered to be \"more challenging\".", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Most structured databases tabular data. Feature selection is of significance to the ML community (e.g., for compute efficiency and reducing overfitting). Paper releases a dataset of tabular data. In addition, they propose a Deep Lasso Regularizer which can be added to the objective function for training deep networks, proving its equivalence to classical Lasso when the network is 1 layer deep.

Overall, this is a nice contribution of a dataset and a feature selection algorithm.
Decision: Accept (Poster)