SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Valeriia Cherepanova  
University of Maryland
Roman Levin*  
University of Washington
Growthami Somepalli  
University of Maryland
Jonas Geiping  
University of Maryland
C. Bayan Bruss  
Capital One
Andrew Gordon Wilson New York University
Tom Goldstein  
University of Maryland
Micah Goldblum  
New York University
Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent overfitting in subsequent downstream modeling, practitioners commonly use automated feature selection methods that identify a reduced subset of informative features. Existing benchmarks for tabular feature selection consider classical downstream models, toy synthetic datasets, or do not evaluate feature selectors on the basis of downstream performance. Motivated by the increasing popularity of tabular deep learning, we construct a challenging feature selection benchmark evaluated on downstream neural networks including transformers, using real datasets and multiple methods for generating extraneous features. We also propose Deep Lasso – an input-gradient-based analogue of Lasso for neural networks that outperforms classical feature selection methods on challenging problems such as selecting from corrupted or second-order features.
Tabular data is ubiquitous across scientific and industrial applications of machine learning. Practitioners often curate tabular datasets by including exhaustive sets of available features or by hand-engineering additional features. Under such a procedure, real-world tabular datasets can quickly accumulate a large volume of features, many of which are not useful for downstream models. Training on such a large number of features, including noisy or uninformative ones, can cause overfitting. To avoid overfitting, practitioners filter out and remove features using automated feature selection methods [Guyon and Elisseeff, 2003, Liu and Yu, 2005].
The literature contains a wide body of work proposing or evaluating feature selection approaches that use classical machine learning algorithms in their selection criteria or which select features for training classical machine learning algorithms downstream [Tibshirani, 1996, Kohavi and John, 1997]. Over the past few years, deep learning for tabular data has become competitive and is increasingly adopted by practitioners [Arik and Pfister, 2021, Gorishniy et al., 2021, Somepalli et al., 2021]. While the community has identified that neural networks are especially prone to overfitting on noisy features
*The substantive contributions of the author to the work described in the paper were done prior to the author joining Amazon.
[Grinsztajn et al., 2022], a thorough evaluation of feature selection methods for downstream tabular neural networks is lacking.
To address this absence, we benchmark feature selection methods in tabular deep learning setting by evaluating the selected features via the performance of neural networks trained on them downstream. In addition to popular existing methods, we select features using the attention maps of tabular transformer models [Gorishniy et al., 2021], and we further propose Deep Lasso - an input-gradient-based analogue of Lasso [Tibshirani, 1996] for deep tabular models. Whereas many previous works on feature selection use entirely synthetic datasets [Wah et al., 2018, Passemiers et al., 2023, Bolon-Canedo et al., 2013] or create extraneous features by concatenating random noise onto existing features [Dinh and Ho, 2020, Borisov et al., 2019], we conduct our benchmark on real datasets, and explore three different ways to construct extraneous features: random noise features, corrupted features, and second-order features that serve as a prototypical example of feature engineering.
In our experiments, we find that while many feature selection methods can differentiate between informative features and noise within reason, they may fail under more challenging settings. Notably, Deep Lasso selects features which achieve significantly better downstream neural network performance than previous methods when selecting from corrupted or second-order features.
Our primary contributions are summarized as follows:
- We construct a challenging feature selection benchmark comprising real-world datasets with extraneous uninformative, corrupted, and redundant features.
- We benchmark different feature selection algorithms for deep tabular models, including recent tabular transformer architectures, on the basis of downstream performance.
- We propose a generalization of Lasso for deep neural networks, which leverages input gradients to train models robust to changes in uninformative features. We show that Deep Lasso outperforms other feature selection methods, including tree-based methods, in the most challenging benchmark setups.
In the following section, we provide a brief overview of recent developments in tabular deep learning, and of feature selection in machine learning more broadly.
2.1 Tabular Deep Learning
Tabular data is the dominant format of data in real-world machine learning applications. Until recently, these applications were primarily solved using classical decision tree models, such as gradient boosted decision trees (GBDT). However, modern deep tabular neural networks started to bridge the gap to conventional GBDTs, which in turn unlocked new use cases for deep learning in tabular domain. Recent work includes the development of novel tabular architectures, for example based on transformer models [Huang et al., 2020, Gorishniy et al., 2021, Somepalli et al., 2021, Arik and Pfister, 2021], and ensembles of differentiable learners [Popov et al., 2019, Kontschieder et al., 2015, Hazimeh et al., 2020, Badirli et al., 2020], as well as modifications and regularizations for MLP-based architectures [Kadra et al., 2021, Gorishniy et al., 2022]. Other works explore new capabilities that are enabled by tabular deep learning, such as self-supervised pre-training [Ucar et al., 2021, Somepalli et al., 2021, Rubachev et al., 2022, Kossen et al., 2021, Agarwal et al., 2022], transfer learning [Levin et al., 2023, Wang and Sun, 2022, Zhu et al., 2023], few-shot learning [Nam et al., 2023] and data generation [Kotelnikov et al., 2022].
2.2 Feature Selection
A cornerstone of applied machine learning is feature selection, where data science practitioners carefully curate and select features for predictive tasks. As a result, there has been considerable interest in automating this process.
<sup>1</sup>Our code for benchmark and Deep Lasso is available at https://github.com/vcherepanova/tabular-feature-selection
Existing approaches for feature selection can be categorized into three main types: filter, wrapper and embedded methods. Filtering algorithms rank features based on their individual characteristics and relevance to target variables, without considering any specific learning algorithm. Examples of filter methods include univariate statistical tests, variance filters, and mutual information scores. A comprehensive overview of existing filter methods can be found in [Lazar et al., 2012]. Wrapper methods, on the other hand, are algorithm-dependent and involve iteratively re-training a machine learning algorithm on a subset of features to identify the subset that yields the best performance. These include greedy sequential algorithms [Kittler, 1978], recursive feature elimination [Guyon et al., 2002, Huang et al., 2018] as well as evolutionary algorithms [Xue et al., 2015, Siedlecki and Sklansky, 1989, Kennedy and Eberhart, 1995, Gheyas and Smith, 2010]. Embedded methods incorporate the task of feature selection into the training process, allowing the model to learn which features are most relevant while training. Lasso [Tibshirani, 1996] is a classical embedded feature selection algorithm, which has been also applied to deep neural networks in the form of Adaptive Group Lasso [Dinh and Ho, 2020]. Additionally, tree-based algorithms like Random Forests [Breiman, 2001] and Gradient Boosted Decision Trees [Friedman, 2001] employ built-in feature importance measures, enabling automatic feature selection. A few recent works propose specialized neural network architectures with embedded feature selection through knockoff filters [Lu et al., 2018, Zhu et al., 2021], auto-encoders [Balin et al., 2019, Zhu et al., 2021] and specialized gating layers [Lemhadri et al., 2021].
We find wrapper methods generally too computationally expensive to be useful for the deep neural network models we consider in this study, especially when hyperparameter optimization is performed. Therefore, in our study we focus on established filter and embedded approaches, both classical and modern, that can be applied to generic tabular architectures.
2.3 Feature Selection Benchmarks
So far, it is unclear which strategy, whether classical or modern, would be optimal for feature selection with deep tabular models. Existing benchmark studies focus primarily on classical downstream models [Bolón-Canedo et al., 2013, Bommert et al., 2020, Bolón-Canedo et al., 2014] or do not optimize for downstream performance at all [Lu et al., 2018]. A few works evaluate feature selection methods on synthetic datasets [Bolón-Canedo et al., 2013, Wah et al., 2018, Passemiers et al., 2023, Sánchez-Marono et al., 2007] or on domain specific datasets such as high-dimensional genomics data containing small number of samples [Bommert et al., 2020, Bolón-Canedo et al., 2014], malware detection data [Darshan and Jaidhar, 2018] and text classification [Darshan and Jaidhar, 2018, Forman et al., 2003]. Passemiers et al. [2023] evaluates neural network interpretability methods in a feature selection setup on synthetic datasets, using a small MLP model, and finds them to be less effective than classical feature selection methods such as random forest importance.
In contrast, in this work we provide an extensive benchmark of feature selection methods for modern deep tabular models. We construct our feature selection benchmark on large-scale tabular datasets with different types of extraneous features and investigate a range of representative feature selectors, both classical and deep learning based. Our benchmark evaluates feature selection methods based on performance of the downstream deep tabular models.
We construct a challenging feature selection benchmark that uses real datasets and includes multiple approaches for the controlled construction of extraneous features. In all cases, we evaluate feature selection methods based on downstream neural network performance, i.e. the practically relevant metric of success for a feature selection procedure. We conduct experiments with both MLP architectures and the recent transformer-based deep tabular FT-Transformer architecture [Gorishniy et al., 2021] as downstream models. Our benchmark comprises 12 datasets with 3 types of additional features. These datasets are collected and adapted based on recent tabular benchmark papers [Gorishniy et al., 2021, 2022, Rubachev et al., 2022] and include ALOI (AL), California Housing (CA), Covertype (CO), Eye Movements (EY), Gesture (GE), Helena (HE), Higgs 98k (HI), House 16K (HO), Jannis (JA), Otto Group Product Classification (OT), Year (YE) and Microsoft (MI). Among these datasets, there are eight classification datasets and four regression datasets. We measure downstream model performance using accuracy for the classification tasks and RMSE for the regression tasks. Additional details concerning these datasets can be found in Appendix B.1.
Figure 1: Performance of FT-Transformer, MLP and XGBoost models when trained on data with random extra features. The X-axes indicate the percentage of uninformative features in the dataset and the Y-axes depict accuracy for classification problems and negative RMSE for regression datasets, so higher values always indicate better performance. Overall, we find MLP models to be more susceptible to noise than either XGBoost or FT-Transformer models.
For each benchmarking experiment, we perform extensive hyperparameter tuning for both feature selection algorithms and downstream models with respect to the downstream model performance using the Bayesian hyperparameter optimization engine Optuna [Akiba et al., 2019]. We select the best hyperparameters based on validation metrics and report test metrics computed over 10 random model initializations (seeds). Details concerning final hyperparameters can also be found in the Appendix sections D, C.3.
In the following section, we present a motivating experiment. In Section 5 we discuss our benchmark design in detail. Section 6 presents experimental benchmark results using feature selection methods.
4 Are Deep Tabular Models More Susceptible to Noise than GBDT?
Recent contributions to the ongoing competition between tabular neural networks and gradient boosted decision trees (GBDT) have found that neural networks are more susceptible to noise than GBDT on small to medium datasets (up to 10,000 samples) [Grinsztajn et al., 2022]. We scale this experiment to larger datasets and showcase it as a motivating example for feature selection methods and benchmarks specific to deep tabular models.
We explore the influence of uninformative features on tabular neural networks and assess the performance of MLP and FT-Transformer models on datasets containing varying numbers of uninformative Gaussian noise features. For reference, we also include a GBDT model into our comparison as implemented in the popular XGBoost package [Chen and Guestrin, 2016]. Figure 1 illustrates the relationship between the performance of the three models and the proportion of uninformative features in these datasets. Similarly to Grinsztajn et al. [2022], we observe that the MLP architecture, on average, exhibits more overfitting to uninformative features compared to XGBoost, motivating the need for careful feature selection with tabular neural networks. Interestingly, as seen from the slope of the blue and green curves in Figure 1, the FT-Transformer model is roughly as robust to noisy features as the XGBoost model. The fact that the performance of the FT-Transformer model is not as
Figure 2: Percent of random, corrupted, and second-order features selected by different feature selection algorithms. Importance rank correlation refers to pair-wise feature importance Spearman correlation averaged across all feature selection algorithms and datasets. Random features are less often ranked as important compared to corrupted and second-order features, and feature selection algorithms have higher agreement when selecting from random features.
severely affected by noisy features could be attributed to the ability of the transformer architecture to filter out uninformative features through its attention mechanism. Inspired by this observation, we further investigate the effectiveness of utilizing the attention map importance within FT-Transformer Gorishniy et al. [2021] as a feature selection method in our benchmark study.
5 Feature Selection Benchmark
It is not common for real-world datasets to contain completely random noise features with no predictive power whatsoever, although engineered features often exhibit varying degrees of redundancy and noise. Nonetheless, feature selection algorithms are often evaluated on datasets containing spurious features generated from Gaussian noise. Not only does this differ significantly from real-world feature selection scenarios, but it also presents a relatively straightforward task for many feature selection algorithms to eliminate these random features. In light of these considerations, we propose an alternative approach to establish a more challenging and realistic feature selection benchmark by introducing three distinct methods for crafting additional features:
Random Features. In the simplest scenario, we sample uninformative features from a Gaussian distribution and concatenate them with the original dataset features.
Corrupted Features. To simulate a scenario with noisy, yet still relevant features, we sample extraneous features from the original ones and corrupt them with Gaussian noise. In addition, we conduct experiments with Laplace noise corruption and report experimental results in Appendix F.1.
Second-Order Features. To simulate a feature engineering scenario with redundant information contained in engineered features, we add second-order features, i.e. products of randomly selected original features.
Note that higher-order features are not spurious and are often used by data scientists precisely because they can contain useful information. Therefore, selecting higher order features instead of original ones may not be a bad thing. Such feature selection algorithms must be evaluated in terms of downstream model performance as we do in the following section.
To gauge the difficulty of the proposed benchmark, we explore how often different feature selection algorithms rank extraneous features among the top- $k$  most important features, where  $k$  represents the number of original features in the datasets. From Figure 2, we observe that all methods select fewer random features than they do corrupted or second-order features. Additionally, to quantify the overall agreement between different feature selection methods, we analyze the average pair-wise Spearman correlation between the rankings of features generated by different selection algorithms. Notably, the setup involving random extra features exhibits the highest correlation, indicating that filtering out random features is relatively straightforward and all feature selection algorithms behave similarly. In
contrast, the setup with second-order extra features has the lowest rank correlation implying greater disparity in selection preferences among the algorithms.
We note, that different feature selection algorithms may select similar features, but rank features differently within "important" and "unimportant" groups, which would results in lower rank correlation even in the settings where feature selection is straightforward (i.e. selecting from random features).
6 Benchmarking Feature Selection Methods
In this section, we benchmark various feature selection methods. In particular, we consider the following feature selection approaches:
Univariate Statistical Test. This classical analysis checks the linear dependence between the predictors and the target variable. It selects features based on the ANOVA F-values for classification problems and univariate linear regression test F-values for regression problems.
Lasso uses  $L_{1}$  regularization to encourage sparsity in a linear regression model [Tibshirani, 1996]. After this sparse regression, features are ranked with respect to the magnitudes of their coefficients in the model.
First-Layer Lasso (1L Lasso) is an extension of Lasso for MLPs with multiple layers. It applies a Group Lasso penalty to the weights of the first layer parameters:
$$
\min  _ {\theta} \alpha \mathcal {L} _ {\theta} (X, Y) + (1 - \alpha) \sum_ {j = 1} ^ {m} | | W ^ {(j)} | | _ {2},
$$
where  $W^{(j)}$  is the  $j$ -th column of weight matrix of the first hidden layer corresponding to the  $j$ -th feature. Similarly to Lasso, First-Layer Lasso ranks features with respect to their grouped weights in the first layer.
Adaptive Group Lasso (AGL) is an extension of the Group Lasso regularization method [Dinh and Ho, 2020]. Similarly to the First-Layer Lasso, it applies a Group Lasso penalty to the weights of the first layer parameters, however each group of coefficients is weighted with an adaptive weight parameter:
$$
\min  _ {\theta} \alpha \mathcal {L} _ {\theta} (X, Y) + (1 - \alpha) \sum_ {j = 1} ^ {m} \frac {1}{| | \hat {W} ^ {(j)} | | _ {2} ^ {\gamma}} | | W ^ {(j)} | | _ {2},
$$
where  $\tilde{W}$  is the Group Lasso estimate of  $W$ . Adaptive Group Lasso then ranks features with respect to their grouped weights in the first layer.
LassoNet is a neural network architecture that incorporates feature selection [Lemhadri et al., 2021]. LassoNet achieves feature sparsity by adding a skip (residual) layer and allowing the features to participate only if their skip-layer connection is active.
Random Forest (RF) is a bagging ensemble of decision trees, and it ranks features with respect to their contribution to the ensemble [Breiman, 2001]. In particular, importance is calculated by measuring the decrease in impurity when that feature is used for splitting at each node of the decision trees in the forest.
XGBoost is a popular implementation for gradient boosted decision tree [Chen and Guestrin, 2016]. XGBoost computes feature importance as the average gain across all splits in trees where a feature was used.
Attention Map Importance (AM) is computed for FT-Transformer model from one forward pass on the validation set. We follow Gorishniy et al. [2021] and calculate feature importance as the average attention map for the [CLS] token across all layers, heads, and validation samples.
Deep Lasso is our generalization of Lasso to deep tabular models (and, in fact, any differentiable model). Deep Lasso encourages feature gradient sparsity for deep tabular models by applying a Group Lasso penalty to gradients of the loss with respect to input features during training. Intuitively, this makes the model robust to changes in unimportant features. For train data  $(X,Y)$  with  $n$  samples and  $m$  features, the Deep Lasso penalty is given by
$$
\min  _ {\theta} \alpha \mathcal {L} _ {\theta} (X, Y) + (1 - \alpha) \sum_ {j = 1} ^ {m} \left\| \frac {\partial \mathcal {L} _ {\theta} (X , Y)}{\partial X ^ {(j)}} \right\| _ {2}, \tag {1}
$$
where, for  $j$ -th feature (column of  $X$ ),  $\frac{\partial \mathcal{L}_{\theta}(X,Y)}{\partial X^{(j)}} = \left(\frac{\partial \mathcal{L}_{\theta}(X,Y)}{\partial x_{1j}}, \frac{\partial \mathcal{L}_{\theta}(X,Y)}{\partial x_{2j}}, \ldots, \frac{\partial \mathcal{L}_{\theta}(X,Y)}{\partial x_{nj}}\right)$ . Once the model is trained with the above regularizer, the corresponding feature importance is provided by
$$
\text {i m p o r t a n c e} = \left\| \frac {\partial \mathcal {L} _ {\theta} (X , Y)}{\partial X ^ {(j)}} \right\| _ {2}. \tag {2}
$$
Note that in the linear regression case, the classical Lasso is equivalent to the proposed input-gradient sparsity regularizer applied to model output since in the linear case input gradients are the weights of the linear model. We provide a formal proof of equivalence between Deep Lasso, classical Lasso, and First-Layer Lasso in the linear regression case in Appendix Section E. We also note that Deep Lasso is related to methods used to promote network explainability by leveraging input gradients [Sundararajan et al., 2017, Shrikumar et al., 2017, Smilkov et al., 2017, Levin et al., 2022]. In addition, [Liu, 2021] examines input gradients for feature selection from a Bayesian perspective. This work uses Bayesian Neural Networks (BNNs) along with associated credible intervals to estimate uncertainty surrounding input gradients, choosing features based on the plausibility that their corresponding gradients are zero. On the one hand, this principled work comes with theoretical guarantees, but on the other hand it requires Hamiltonian Monte Carlo, an expensive sampler which does not scale to large datasets and models.
6.1 Results
We benchmark feature selection methods for downstream MLP and FT-Transformer models. In the case of the MLP downstream model, we explore scenarios where either  $50\%$  or  $75\%$  of the features
in the dataset are extraneous. For the FT-Transformer, we focus on evaluating FS methods solely on datasets with  $50\%$  added features. For simplicity, we train the downstream models on the top-k important features determined by the feature selection algorithms, where k corresponds to the original number of features in the datasets. We include more details on our experimental setup in Section C.