An Inductive Bias for Tabular Deep Learning
Authors: Ege Beyazit
Jonathan Kozaczuk
Bo Li
Vanessa Wallace
Bilal H Fadlallah
Venue: NeurIPS 2023 poster (2023)
Keywords: Tabular Deep Learning
Spectral Bias
Neural Networks
TL;DR: Reducing target function frequency improves neural network performance on tabular data

Abstract: Deep learning methods have achieved state-of-the-art performance in most modeling tasks involving images, text and audio, however, they typically underperform tree-based methods on tabular data. In this paper, we hypothesize that a significant contributor to this performance gap is the interaction between irregular target functions resulting from the heterogeneous nature of tabular feature spaces, and the well-known tendency of neural networks to learn smooth functions. Utilizing tools from spectral analysis, we show that functions described by tabular datasets often have high irregularity, and that they can be smoothed by transformations such as scaling and ranking in order to improve performance. However, because these transformations tend to lose information or negatively impact the loss landscape during optimization, they need to be rigorously fine-tuned for each feature to achieve performance gains. To address these problems, we propose introducing frequency reduction as an inductive bias. We realize this bias as a neural network layer that promotes learning low-frequency representations of the input features, allowing the network to operate in a space where the target function is more regular. Our proposed method introduces less computational complexity than a fully connected layer, while significantly improving neural network performance, and speeding up its convergence on 14 tabular datasets.

An Inductive Bias for Tabular Deep Learning
Ege Beyazit"
Jonathan Kozaczuk"
Amazon beyazi Deyazitt@mazon.comm
Amazon
jonkozac@amzzon.comm com
Bo Li
Vanessa Wallace
Bilal Fadlallah
Amazon
Amazon wwallGamazon, com
Amazon bhf famazoz
mooliaamazon.com
Abstract
Deep learning methods have achieved state-of-the-an performance in most model- ing tasks involving images, text and audio, however, they typically underperform tree-based methods on tabular data. In this paper, hypothesize that signifi- cant contributor to this performance gap is the interaction between irregular target functions resulting from the heterogeneous nature of tabular feature spaces, and the well-known tendency of neural networks to learn smooth functions. Utilizing tools from spectral analysis, have high irregularity, and that they can be smoothed by transformations such as scaling and ranking in order to improve performance. However, because these trans- formations tend to lose information or negatively impact the loss landscape during optimization, they need to be rigorously fine-tuned for each feature achieve performance gains. To address these problems, we propose introducing nntroducing_rrequeeccy frequency reduction an inductive bias. We realize this bias as neural network layer that promotes learning low-frequency representations of the input features, allowing the network to operate space where the target function is more regular. Our proposed method introduces less computational complexity than fully connected layer, while significantly improving neural network performance, and speeding up its convergence on 14 tabular datasets.
show that functions described by tabular datasets often
Introduction
Despite being one of the simplest and most natural ways describe data, tabular data with heteroge- neous columns poses significant challenge for deep learning models. Recently, [5] 13 benchmarked various deep learning approaches against tree-based models and showed that tree-based models tend to perform the best on tabular data. Nevertheless, novel neural network architectures provide several ad advantages differentiable loss functions, are able to learn meaningful and high- high-dimensional data representations, and can scale to large datasets. In addition, the rapid advances of neural networks in generative image and language modeling clearly indicate their capability to encode complex information [8] 18]. In order fully utilize these benefits on tasks that include tabular datasets, identifying and addressing the reasons behind neural networks' lack of performance on this form of data is essential.
classical machine learning methods. Specifically, neural networks operate
Several influential research efforts on synthetic and image data point out that, despite being universal approximators, neural networks have a spectral bias, i.e., they tend to learn the low frequency components of function much more easily than higher frequency components 12 27 In addition, recent empirical studies on tabular deep learning provide insights on why neural networks typically
'These authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

10
2.-
10
10-2
P-0k
3.-
10-1
cBrib-nnal
10
2.0
frequency mi
10 10
10
10-
Teaturm Valle
Figure 1: (Left) Due to their heterogeneous nature, tabular datasets tend to describe higher frequeney target functions compared to images. The spectra corresponding to image datasets (curves in color) tend to feature lower Fourier amplitudes at higher frequencies than hetergoneous tabular datasets (cyan region). (Top Right) Data with labels varying over small distances are difficult for neural networks to learn, but easy for tree-based methods. (Bottom Right) Ranking is transformation that redistributes the data and increases the scales over which the feature labels vary. This transformation effectively lowers the frequency of the target function, making easier for neural network to learn.
underperform on tabular data, including the latter's tendency to feature irregular target functions K3 5). Notably, the feature spaces of the tabular datasets considered these studies are heterogeneous. Unlike images in tabular form where columns correspond to the same signal on different sensors, columns of tabular deep learning tasks typically represent features of different nature. These features are engineered to be informative as individuals (or subsets) and tend to have different statistical properties than each other.
In this work, we connect these lines of inquiry and claim that the spectral bias of neural networks may prevent them from learning the information present in the high frequency components of heterogeneous tabular datasets. We provide evidence to support this claim by analyzing the Fourier components of image and tabular datasets, well as neural networks. To best of our knowledge, we present the first analysis on the relationship between spectral bias and tabular deep learning. We suggest that transforming tabular features into low-frequency representations may alleviate the negative impact of spectral bias. Notably, such transformations need to be performed in a way that balances the benefits of reduced frequency against potential information loss and additional negative effects optimization (e.g., creating large gradients along certain directions) that can arise when altering the representation of the data. Figure Iillustrates an example of the analyses we conduct. From the left panel, it can be observed that many tabular datasets tend to have higher power in their high frequency components, compared to some of the popular image datasets. We extend this comparison and present additional observations in Appendix (C, The right panel depicts the impact of frequency-reducing transformation on the functions learned by multilayer perceptron (MIP) and Extreme Gradient Boosting (XGB) [7] on synthetic data. From this panel, it can be observed that the MLP is able to fit the target function better after the frequency-reduccng transformation. Details of this analysis are provided in Appendix
Driven by these observations, we propose using frequency reduction as an inductive bias during neural network training. We apply this bias by introducing a trainable layer that promotes learning lower frequency feature mappings. We show that several commonly-use used data processing methods impact the frequency spectrum of the target function as well as the resulting neural network learned from the data, however, their impact on performance varies with the dataset. Our proposal leads to the best performance, converges faster across all datasets considered, and introduces minimal computational complexity.
Related Work
Spectral Analysis of Neural Networks. Spectral properties of neural networks have been extensively studied. [22 27
rely on Fourier analysis to investigate the learning behavior of neural networks,

---

observing that neural networks learn lower frequency components of target functions first. This phenomenon is referred to as spectral bias or the frequency principle. 3 study spectral bias on data with non-uniform density by examining the NTK for deep fully-connected networks, observing that both data density and frequency impact the gradient descent behavior. [6] show that during neural network training, error terms from different frequencies are controlled by the eigenvalues of the NTK. [66] decompose the reconstruction error of neural networks into low and high frequency components with respect to cutoff frequency, and use these components to define metric that quantifies spectral bias. [26] study the spectral bias of coordinate- Using NTKs, they show that mapping imput coordinates to Fourier features [23] enables learning higher frequency information. Notably, they focus on the frequency described by pixels of a given image, rather than the frequency of the target function described by the whole dataset. The existing studies mainly focus on advancing the understanding of spectral bias on synthetic and image datasets, however, woal-world tabular datasets are not considered.
MLPs for modeling functions low dimensions.
Tabular Deep Learning. Due to neural networks' inferior performance on tabular data compared to tree-based models, various approaches have been proposed to close this performance gap. [I] take raw tabular data as input and use sequential attention to choose the set of features to use for every step of inference. This instance-wise feature selection is shown to improve performance and enable interpretability. [28] [ extend self and semi-supervised learning to tabular data by introducing pretext tasks that aim to optimize model to recover input samples from their corrupted variants. [25 leverage row- and column-wise wwin attention combined with contrastive pre-traine traininnn.. [12] embed numerical features using linear transformations and categorical features using lookup tables, and feed these embeddings to a transformer. [15 show that using hyperparameter tuning, one can find a strong combination of 13 regularizers to be used with simple MLP to outperform other tabular deep learning methods, as well as tree-based methods such as XGB. [1] propose individually embedding numerical features onto high-dimensional spaces before jointly using them in a model. The authors explore approaches such as piecewise linear embeddings that use bins selected by decision trees, and periodic activations transform numerical features into high-dimensional representations.
Discussion. In most learning tasks that include images, audio or text, deep neural networks provide tfat-oof-hhe-r performance. Breakthroughs in these domains strongly connected to the design choices that exploit the salient characteristics of the corresponding data modality (c.g.. convolutional layers in computer vision). On the other hand, existing studies tabular deep learning mainly focus on applying approaches that are successful in other domains to improve performance [2] 25). These studies do not rigorously investigate the fundamental reasons behind the performance gap between neural networks and tree-based methods. As a result, in order to gain performance, these approaches tend sacrifice simplicity by complicating training and hyper-parameter tuning significantly. Such complexity causes these models to perform inconsistently across different studies, as utilizing the full potential of these models can be challenging. For example, [I3] provide an extensive set of benchmarks, showing that tree-based approaches significantly outperform various tabular deep learning methods that originally claimed to outperform tree-based models. Similarly, (55 show that simple MLPs perform better than TabNet, an architecture that is tailored for tabular deep learning. As result, instead of focusing on designing complex models to improve performance, we aim to identify and address the fundamental weaknesses of neural networks when learning from tabular data. We show that our approach improves performance of neural networks significantly without introducing additional complexity for fine-tuning, and helps them to converge faster.
3 Methodology
3.1 Background
Consider classification tasks defined on a tabular dataset D= ((x;,:)) where x; E R² and y: {0, 13"H is the corresponding label. The task is to learn classifier f R + {0, 1mm For simplicity, set m = 1 (i.e., binary classification), and let y = 1 correspond to the minority class in all cases. In Section 4 we provide experimental results and analyses for multi-class datasets as well, by
extending this approach via one-vs-rest.
We are interested in the frequency spectra of various functions, including target functions implicitly defined by the data, for which f(xx, = yi- Since we only have the labels defined at the datapoints supplied in each dataset, we rely on spectral analysis methods that take an arbitrary set of points as
3

---

input and output a measure of the strength of fluctuations at particular scales. One such method is generalized Fourier transform known as the non non-uniform discrete Fourier transform (NUDFT) [[O]. For function f(x) and set of points X = {x,)==1> the NUDFT defines function of frequency vector k E TR." as
N Σ f(x,)e 2
(k) N
Equation T can be interpreted as an approximation of the multidimensional continuous Fourier transform (FT) of fxx) weighted by the empirical distribution represented by the datapoints X, further discussed in Appendix A [27 demonstrate the spectral bias of neural networks in terms of the NUDFT.
The Fourier transform approximated in Equation D evaluated a given k, corresponds to the femm coefficient of the 22xieex term in projection onto complex exponentials. Therefore, a natural definition of one function being higher frequency than another corresponds the former having karger-magnituce Fourier amplitudes away from a specified low-frequency regime than the latter. We can make this comparison concrete by evaluating amplitudes along specific directions in k-space. Defining k lkk, with k unit vector in IR, we can compare the size of the squared NUDFT amplitudes, Py (k) = fx (k) [x) for functions along given k direction by computing the sum
2 (h. - ko
S
Σ
Pj (1ko + ndkik) Ak,
where f is function of interest. Here, is the mean of f(x) (i.e., y) evaluated on the the points in X, so that the scales at low frequencies are comparable for any given dataset (see Appendix (A.2). Equation (2) can then be used to compare spectra: if Sglfi] > [2 for two functions fı and fz defined on R", we say that f2 is a higher frequency function than fi along the direction k, in the sense that it has more normalized signal energy (variance) in the region of interest, and refer to frequency reduction as reducing Sy (k). In the above expression, ky defines the boundary between the low- and high-fie figh-freuuency T regions. and k. is a cutoff frequency, geeeee, discussed in Appendix [A.4] Notably, Equation 2 only considers the magnitude of Fourier amplitudes along one dimension of I"" at a time. In Appendices A..3 and A.6] we discuss an alternative approach that works in the full feature space Ih" and show that using (2) to evaluate along principal components (PCs) yields similar results to working in R.,, while being more computationally efficient. Therefore, we rely on evaluations along PCs.
3.2 Spectral Analysis of a Class of Target Functions
The NUDFT, when evaluated using the labels f(x;) Vs. can be understood in terms of the continuous Fourier transform of a class of target functions fitting the data. Given a binary classification task described by dataset D {(x,, b) a a typical way train neural network is minimizing the empirical risk 1 1/N E Eyy, f(x,)), where l denotes the cross-entropy loss. Assuming that D maps each unique x, to single yı, the cross-entropy loss without explicit regularization terms exactly minimized for any function fx) such that f (x,) = yivi E {1..... .N. This defines a set of equalities satisfied by any ideal target function perfectly fitting the training set but does not further specify the form of the function away from the training instances. Indeed, there is no unique definition of the target function given only a finite set of points. However, we can parameterize class of target functions minimizing the loss by convolving the target values at known data points with given envelope function, C(x, x'), that falls off sufficiently rapidly away from the points {x;}
N
/ y,G(x, (x'-))'''"'"'x))))) xx) f[x, G = Σ JR.
where 8" (x) is the Dirac delta function in R" and a; denotes the parameters defining the envelope for the point x; A typical loss function (neglecting regularization terms) will be exactly minimized
LI practice, some form of regularization will need to be utilized to avoid overfitting, but this set of equalities should still approximately hold. We verify empirically that the spectra of classifier functions learned in realistic settings approach the spectra of the target functions defined in this idealized sense.

---

provided x;;,) and G X3; = 04,,, E D. Perhaps the simplest such envelope a multi-dimensional generalization of the boxcar function
B(x A,) II rect
(4)
A,
where < <
a' A/2, and vanishes otherwise. Notably, A corresponds to an upper bound on the Do distance between and a', in order for them to be assigned to the same value y. The target function defined by inserting Equation (4) into Equation is then
N
f(x)
Σ
rect
8(0') - (xx),)dx'.
(5)
A
j=1
R
Provided that A satisfies
xj oc
A VX3 D,
(6)
for each x, this function minimizes the cross-entropy loss on the training set, since its values at each point are simply the corresponding label. This corresponds to riling the feature space with n-dimensional hypercubes edge lengths V,Vi, centered on each datapoint x; and taking the target function to be constant within that region with value This is the simplest possible choice for the envelope function in that does not assume any spectral features present in the target functions on distances smaller than A,
We can now use the continuous Fourier transform to analyze the frequency spectrum of the target function defined in Equation (5). Denote the Fourier transform of a function g(x) as [x] By the convolution theorem, we have
n II sinc(
(x)((x)((k) = Σ y/c 2nrkxx A,
where sinc(x) = sin(T=)/((r)) Different choices for the A; correspond to different assumptions
about the target function away from the datapoints. One simple class of target functions obtained
by taking common A; = A min {|x, x2)1 +1
N-1. This choice satisfies Eq. 6 and in
the limit k/ < 1/(=A),, yields
N
(x) R A Σ Uie 2ark-xx NA fx(k)
(8)
where the approximation holds up to terms ((k)2777)) (here || denotes Euclidean distance). The RHS above the NUDFT up to a constant dimensionful scaling factor, which will drop out when using a procedure like Equation ( that normalizes the amplitudes to the value at k = 0. Therefore, the NUDFT reproduces the spectral properties of a simple class of target functions defined by 3 for frequencies below the scale set by the inverse minimum nearest-neighbor separation of points in the dataset. We implicitly enforce the latter restriction by cutting off our analysis at ki, as discussed in Appendix(A.A Note that other choices for the A; result in different target functions with spectra that can be analyzed by appropriately modifying the NUDFT.
Equation
In order to compare against the same quantity in all analyses, we also utilize Equation ( for analyzing the frequency spectra of classifiers. A similar argument holds, and the corresponding results can be interpreted as analyzing the spectra of classifier coarse -rrained by convolving the neural network predictions at the datapoints with narrow envelope as in Equation (5). all cases, the classifier spectra defined in this way converge to that of the target function as training progresses, up to deviations caused by spectral bias (see also (27 for similar results). This suggests that high-frequency target functions, defined as above, are best fit by high-frequency classifiers. and that transformations affecting the target function spectrum will impact the frequency spectrum of the corresponding classifier.
5

---

electricity
phoneme
MagicTelescope
bankMarkeiing
MiniBooNT:
alhert
california
unit length
1.31
1,46
1.66
0.13
2.01
16.24
1.74
1.5
0.18
0.44
0.74
0.34
scale
0.01
0.18
0.06
0.01
0.17
0.35
0.04
0.03
solrank
0.11
0.22
0.58
0.07
1.28
0.34
0.29
0.32
electricity
phoneme
MugicTelescope
bankMarketing
MiniBoaNT:
california
unit longth
88, = 0.17
85, = 0.12
87.78 + 0.28
87.84 = 0.18
90.56 0.11
90.78 + 0.04
66.35 0.04
88.51 = 0.15
rank
89.61
0.14
0.19
88.0 0.46
90.41
95.01
0.17
scale
89
: 0.16
85. 0.13
88.64 0.21
87,88 = 0.23
90.56 2 0.13
94.0 0.04
66.48 = 0.05
88.53 2 0.22
solrank
89,91 0.16
87.04 = 0.17
88.15
88.14 0.27
90.56 0.13
94.94 0.05
66,28 0.06
88.48 0.2
Table 1: (Top) Normalized sum of high-frequency Fourier amplitudes, Sg for various binary classification datasets and transformations, averaged over the 1st principal component direction after each transform is applied. Ranking and scaling individual features tends to significantly reduce the high-frequency energy of tabular datasets relative to features normalized to unit scale. For an extended discussion, see Appendix[A.d (Bottom) Accuracy measurements corresponding to different transformations. Overall, lower frequencies are correlated with better performance. However, the benefits of frequency reduction is transformation- and dataset-dependent.
3.3 Case Study: Impact of Transformations on Target Function Spectra
Most neural networks need their input to be transformed in order to converge faster and generalize better. [14) show that these transformations serve to regulate layer input/output gradient statistics and/or the conditioning of input/gradient covariance matrices, A typical example of such transfor- mations is scaling the features so that they vary over the unit interval [0, 1], i.e., min-max scaling, Different choices of transformations can dramatically impact the target function frequency spectrum, and therefore the spectrum of the network that best fits the data with given loss function. Due to the spectral bias of neural networks, we expect that using transfornations that reliably reduce the high frequency components of the target functions result in better performance (see Appendix D for further discussion and illustration of this point).
Let us consider two types of such transformations, rank and scale, that are applied to each feature independently. Consider a vector z' E R. with elements (z')r {(x1),,...... (xx))} such that (z'), I (z'), vr S Then
rank((x)))) = N min N(2),
scale((x:)))
(x:)e,
(10)
for x; E X and a E R. Equation 9 simply maps feature to the range [0, 1]. proportional to its position when ranked by magnitude along with the other values the dataset. For any (x)e not in the original dataset, rank returns value interpolated between rank(x,)e and rank(x;)e for the (xe)e, )e E X nearest to (x)x such that (xx)² < (x); < ). Both scale and rank may alter the frequency spectrum of the target function described by a dataset by either stretching/squeezing fluctuations while preserving relative distances between points or neglecting distance information altogether and forcing the points to be uniformly distributed along direction, respectively.
we present the high-frequency Fourier amplitudes (i.e., |k > 0.5) of the target functions of 8 tabular binary classification datasets with different transformations and their impact on performance across 10 random seeds in Table Here, we implement scale as standardization. such that a in Equation (TO) is the inverse standard deviation of the feature in the training set population. From the top panel of Table can be observed that scale and rank consistently reduce the frequency of the target functions of all datasets, although by different relative amounts.
Using Equation (
The bottom panel of Table M shows that, overall, lowering target function frequency is correlated with better NN performance: converting the averaged Sj and NN performance for unit length, rank, and scale to their ranks within each dataset and comparing yields a Spearman's rank correlation coefficient of ~ -0.7 with a p-value of = 3 x 10-4, However, this correlation is not exact: large reduction in high-frequency Fourier amplitudes may not always result in significant performance improvements. This is because the amount of high frequency information encoded by each feature different across datasets. Also, scaling up the feature values by a significant amount reduces the frequnery energy, but can also negatively impact optimization by generating large gradients

---

during training. Finally, rank discards relative distance information between points in favor of frequency reduction, and also impacts optimization produces non-centered features. Therefore, it is important to simultaneously balance the effect of a given reqqeency-reduciin transformation with other potentially negative impacts on the loss landscape or due to information loss.
These observations indicate that there is not a simple one-size-fits -its-alll transformation that reliably and consistently reduces the high-frequency spectral energy that results improved network performance. However, with this trade-off between frequency reduction and potential loss of information or effects on the loss landscape in mind, we can take a first step improving over the rank and scale transformations by doing them selectively. For example, to balance the aforementioned trade-off we consider a selective version of rank defined as
sclrank((x)) - rank((x)))) Sk, (Hrank(()))) 26) M < @ (5xx)((x))) Vi Nr1
(11)
(x)e, otherwise
where the datapoints in the conditions are those in the training set and θ R. selrank applies rank only if it reduces the high frequency energy of the target function defined for the starting data representation by certain relative amount, @. Otherwise, selrank leaves given feature as-is. The threshold θ parameterizes how significant the frequency reduction must be in order to outweigh the potentially detrimental loss of information impact on the loss landscape. From Table []] it can be observed that selrank performs comparably or better than its counterpart rank across the datasets considered, motivating the use of frequency-informed transformations.
3.4 Frequency Reduction as Inductive Bias
Reducing the frequency of the target function may improve neural network performance by reducing the impact of spectral bias. On the other hand, the way we reduce frequency may impact other factors that play a significant role neural network training. Since the interactions between transformations that reduce frequency and other learning dynamics are unique to how a feature is originally distributed, we can loosely formulate finding a transformation, y parameterized by 0g that reliably and beneficially reduces the frequency as constrained optimization problem:
J Pron <
argmin 8,.0g
Elye, ((g(::0)))) s.t.
Py. Py. (k)dk,
(12)
where Of denotes the trainable parameters of neural network f. P can be evaluated on arbitrary (including unlabeled) points, K denotes a high frequency region of k-space, and f denotes a neural ii=1
network that minimizes the empirical loss 1 I/NE L(v. (x;; 0-))). Equation 12) suggests that a beneficial freduingy-reducing transformation can be found by limiting the space of acceptable solutions represented by f to yield lower spectral energy in its high frequency components, compared to its analogous model f trained on the raw data. One way to attempt solving this optimization problem is simplify the frequency reduction constraint and directly use Py(k) to regularize the training loss towards finding a low frequency solution. However, this approach is computationally expensive, would in general require transformations multiple features simultaneously, and would require careful pre-determination of the region K to represent the spectral energy in high-frequency components of the learned function.
To alleviate these challenges we make the following simplifications. First, to avoid the computational complexity of working in the full feature space, we assume that all input features are conditionally independent given the target y. hence, they do not have interactions that significantly impact the frequency of the high dimensional decision boundary described by the data. Notably, we only use this assumption to limit our consideration to transformations that reduce the high frequency spectral energy by acting individual features one at time instead of jointly. After the transformation is conducted, consideration of feature interactions for the learning task is dictated by the underlying network architecture. Second, instead of relying on S₁ as a proxy for Jg Py (k)dk to impose frequency reduction constraint on f a during training, we constrain to be a composition of the two frequency-reducing transformations rank (9) and scale (10) (understood be acting on the individual components of the feature vector x):
9 (x: 0g+ : {0), 07, 0g)) = 8 scale(rank( (x; (7): 05) + (1 (1-61 scalexx; 03)
s.t. 0 e [0, 1], (13)
where Or represents the set of parameters specifying the mapping from original feature values to their ranked counterparts (c.g., how the interpolation is done), and 0r denotes the set of coefficients
7

---

used to scale the input features. Note that oy is not variable of this optimization problem as it only depends on the order of feature values training set. Equation (I3) describes a linear trajectory between scaled features and their ranked counterparts, while the degree of scaling applied to original and ranked features can be different. Notably, although we describe a linear relationship, one can encode more complex relationships to enable exploration of a larger space of frequency reducing transformations. With 9 applying frequency reduction as a soft constraint, we drop the constraint from Equation (12), and set (0,, 0g} to the minimizers of the empirical loss. We implement Equation (3)) as the input layer of neural network and let 0g get updated jointly with the rest of the network parameters during backpropagation. We satisfy the constraint 6 E [0. 1] by clamping these weights after each update. Note that, computational complexity introduced by Equation (3 is linear in the number of imput features (see Appendix (F)
is important to note that rank and scale are used to individually reduce the high frequency spectral energy of the empirical distribution function p(x), even though our objective is to reduce the spectral energy for fx; fy) P(y) x) instead. This is motivated by the observation that (13) designed as an inductive bias that exploits the neural network f's spectral bias, and g's parameters are selected towards minimizing the empirical loss. Given that neural networks have strong spectral bias towards learning low frequency functions, when jointly optimized during training, / prefers mapping the training instances to an informative alternative in order to minimize the loss further. In Section 4 we show empirical evidence that neural networks trained with g indeed have reduced spectral energy in their Sigh-frequeccy components compared to the networks trained on the same input data.
frequenqycyy form using g and fitting to this uww-freuency
4 Experiments and Results
Our evaluation focuses on 3 key metries: performance, rate of convergence and the irregularity of functions learned. Performance is evaluated using accuracy and the area under the receiver operating characteristic curve (AUROC). Rate of convergence is evaluated using the mean number of training epochs required to minimize validation loss. Finally, irregularity of functions learned by the neural network models are measured using total
power in Equation 2 aallng top principal components (PCs). We evaluate our proposed approach using 14 benchmark classification datasets listed in Table 2 These datasets are used by [I3] to demonstrate the performance gap between tree- based models and neural networks. However, unlike [33, we use these datasets without truncating any samples, and we drop id and date type features as the approaches we compare are not designed to utilize the information conveyed by them. Because we consider a wide variety of datasets, the evaluation metries we collect for the basclines and the proposed method highly fluctuate. This makes it challenging to draw conclusions for the overall behavior of these methods. Based on these observations, similar to 13] we normalize our measurements and aggregate them across datasets. In Appendix we provide the raw measurements and additional details on data preparation and training. Implementation details to reproduce our results are provided in Appendix H
Frequency Reduction with Neural Networks. We present normalized and aggregated statistics across 14 datasets to highlight the general behavior of different transformations. Figure 2 depicts these statistics, where boxes represent quartiles, notches represent 95% confidence intervals and whiskers represent the full range except outliers. From the figure, it can be observed that our proposed
Name
#Samples
Al Prenluues
Source
eleciricity g
45312
httpe:/ //oppn// org/e/151
house. 16Fl
22784
17
01:05 // opanni or // 7/7
pol
htige // /opennl oz ofg////22
kdd. ipams la 97-small
2019
61
h1.228 // // /opannt orgg///950
Magic Telescope
19020
htlpe /oppnnl
bank--mrkketing
45211
n11.8: 7ogenn1 www////
phoneme
5404
Btbps 7/ openin G2g/ 0/ 1899
MiniBooNE
130064
eye ro__movements
X4
10936
httpe (stpp-/7opemp 01e/ 2043
jannis
83733
68
califernia
19
206-10
https://wwx.coo doc.
up. wttpppppppp..... htt//toorg//leee htw.ooogooooeeesssioooo
www.bouunggg... htmi
alhert
425240
(atrpp.7/55eeee /ttp///p//..... 017/ 61147
credit card clients
30010
bttps:/ /archire..... ice uci. edl.edd//ll. /tt.a///aaaaaaaaaaaaaaaaaaeeeeee
Diahetes
Attpa:7/4vx. www.eooeen...................... www.serrc////yyeee....
Table 2: 14 Tabular datasets used in the experiments
8

---

1.0
0.0
12T
methed
Figure 2: Normalized measurements of performance, convergence and frequency across datasets and seeds. The simple frequency- frequency-informed informed approach, selrank, with rank implemented as quantile transform improves upon rank alone. Additionally, our proposed methods significantly outperform all baselines, while converging faster.
100
202
Figure 3: Weights learned by the proposed approach electricity dataset. Lines represent values taken by the weights 9g throughout 250 epochs, and shaded regions represent the 95% confidence intervals. The weights consistently converge similar values across 10 random seeds that vary data splits and neural network initializations. Due to heterogeneous nature of tabular datasets, scaling and ranking weights that minimize the loss for different features different.
method provides significantly higher normalized accuracy and AUROC, while taking significantly less epochs to converge. All of the transformations considered result in reduction of frequency of the function learned, compared the neural network trained on unit scaled data. However, as discussed in Section 3.3 existing methods do not benefit from frequency reduction much as our proposed methods, as they may have side effects such as losing relevant information (i.e., rank), or negatively impacting optimization (i.e., robust, standard and rank). On the other hand, our proposed method able to balance the tradeoff between performance gain from frequency reduction and loss due to the side effects of the transformations used, providing an effective inductive bias for tabular deep learning.
Investigating the Learned Weights. Figure depicts the change of the weights of our proposed approach across 10 random seeds on electricity dataset. Appendix(E.JJetennn Across different seeds, initialization of our proposed layer's weights stay the same (i.e., 0 = 1 and 05 == 0.5), however, the network weights 0, and data splits change. The first two columns of the figure correspond to the scaling weights 8r for the raw features and ranked features, and the third column corresponds to the weights 8 that combine these features from Equation 13). Shaded
this plot to all datasets.
regions in the figure represent 95% confidence intervals. From the figure, can be observed that the confidence intervals of the learned weights are small, relative their change over epochs. This implies that the values of these weights stay consistent across random seeds. Consequently, network consistently learns similar representations for the same feature and the representations do not overfit to the underlying network architecture. The figure also shows that the amount of scaling and ranking that can be applicd gain performance is data-dependent. Therefore, it is computationally expensive to exhaustively find the best transformations for each feature.
Frequency Reduction with Tree-based Models. XGB typically outperforms all other tabular learning approaches and known in order to isolate the performance improvements observed by our approach to spectral bias of neural networks, important to confirm that our approach does not improve XGB performance. In
be impacted by irregularity target functions [3]. Therefore,
Appendix we provide empirical evidence that our approach's impact to XGB is insignificant. Additionally, y, even though we do not intend this study be benchmarking of state-of-the-ar methods, see that using approach, simple MLP with regularization and limited hyperparameter

---

propeioed
26-
Figure 4: AUROCS of MLPs using the baseline transforms and the proposed method over multiple scale factors. As the scale factor increases, the target function frequency of the dataset also increases. Shaded regions correspond to 95% confidence intervals across 10 random seeds.
tuning can outperform XGB in 4 of the datasets that were identified to be tree- fre-friendly by previous studies [33. while providing competitive performance in 2 datasets.
Frequency Reduction with other Network Architectures. Frequency reduction as inductive bias can improve the performance of more sophisticated deep learning architectures as well. For example, our proposed layer's 's mappings can be used as an input TabNet to significantly reduce the amount of hyperparameter tuning required. As discussed in Section 2 it is challenging to utilize TabNet's 's full potential due its hyperparameter sensitivity, causing inconsistent results across different studies. However, this model also offers additional benefits beyond performance improvements such as interpretabilityy. Therefore, we believe TabNet is good candidate to evaluate our inductive bias on beyond the vanilla MLP. Similar to our other experiments, we limit the number of hyperparameter configurations swept for tuning to 100. For each dataset, we train an MLP with our proposed layer until convergence and use the layer to generate low-frequency versions of the corresponding dataset. We compare the performance our approach to TabNet trained by following the author's 's suggestions [] We find that approach improves TabNet's AUROC by 4% + 1.6% (sce Appendix E.2)
Performance over Varying Target Function Frequency. Due to the diverse nature of the datasets considered in our experiments, it may appear challenging to reliably draw conclusions about the effect of the proposed method. Different dataset characteristics naturally lead to variation in the performance gain observed from our methods. To address these concern,, we provide additional results using synthetic data. Specifically, we generate synthetic datasets of varying target function frequency by applying common scale factor to k; in Equation (24) while keeping all other parameters fixed. We train the same 2-hidden-layer MI.P for 200 epochs on the unit-scaled, ranked, and scaled (via standardization) features. Figure Sesure/depicc depicts the AUROCS of these MI.s over different target function frequencies. From the figure we observe that for low target function frequencies, all methods perform comparably. Also, performance decreases with increasing target function frequency due to NN's spectral bias. Our proposed method consistently outperforms the others and is the most robust to high frequency target functions.
5 Conclusion
In this paper, we studied the impact of spectral bias of neural networks on tabular datasets. We showed that tabular datasets tend describe irregular target functions and this irregularity negatively impacts neural network performance. We proposed incorporating frequency reduction as an inductive bias when training neural networks reduce the irregularity of these functions. We showed that our proposed method significantly improves neural network performance while reducing the number of training epochs needed to converge. Although we propose the new direction of using frequency reduction as an inductive bias for tabular deep learning, we merely scratched the surface when it comes realizing said bias. For example, our analyses can be extended to other tabular deep learning approaches to determine if the performance improvements they offer can be explained from the lens of spectral analysis. We discuss other future directions as well the limitations of our work in Appendix G
10

---

Acknowledgements
We thank Tian Tan, Xuan Li, Man-Ho Marco Tang, Mihir Pendse and Jiajian Lu for valuable discussions and feedback. We are also grateful to the anonymous reviewers for providing constructive feedback on this manuscript.
References
[1] Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679-6687, 2021.
[2] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15147 2021.
[3] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, pages 685-694. PMLR, 2020.
[4] James Bergstra, Daniel Yamins, and David Cox. Making science of model search: Hyper- parameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 115-123. PMLR, 2013.
[5] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[6] Yuan Cao, Zhiying Fang, Yue Wu, Ding- Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv: 19/2.01 198, 2019.
[7] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[8] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[9] Dheeru Dua and Casey Graff. UCI machine learning repository. 2017.
[10] Jeffrey A Fessler and Bradley P Sutton. Nonuniform fast fourier transforms using min-max interpolation. IEEE transactions on signal processing, 51(2):560-574, 2003.
[11] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991-25004, 2022.
[12] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:1 14:18932- 18943, 2021.
[13] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[14] Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[15] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928-23944 2021.
[16] Jonas Kiessling and Filip Thor. computable definition of the spectral bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7168-7175, 2022.

---

[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] OpenAl. Gpt-4 technical report. arXiv, 2023.
[19] Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33(3):291-297, 1997.
[20] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS-W, 2017.
[21] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning Python, Journal of Machine Learning Research, 12:2825-2830 2011.
[22] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. International Conference Machine Learning, pages 5301-5310. PMLR, 2019.
[23] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[24] Jarkko Salojärvi, Kai Puolamäki, Jaana Simola, Lauri Kovanen, Ilpo Kojo, and Samuel Kaski. Inferring relevance from eye movements: Feature extraction. In Workshop NIPS 2005, in Whistler, BC, Canada, on December 10, 2005., page 45, 2005.
[25] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. 2021.
arXiv preprint
[26] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Keill, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[27] Zhi-Qin John Xu, Yaoyu Zhang. Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 28(5):1746-177 2020.
[28] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and superiupere learning to tabular domain. Advances in Neural Information Processing Systems, 33:1 1033-11043, 2020.
12

---

Supplementary Material
A Fourier Analysis Details
In this section, we provide additional details regarding Fourier analysis.
A.1 The NUDFT and its continuous limit
The primary starting point for our spectral analysis is the non- non-uniform discrete Fourier transform defined in Equation IT. This particular transform conveys spectral information about functions sampled at arbitrary data points and its adjoint is commonly used in the medical imaging domain to reconstruct images from measurements taken in Fourier space. Equation ① can be interpreted as an approximation of the multidimensional continuous Fourier transform (FT) of [(x) weighted by the empirical distribution represented by the datapoints X. continuous limit as N + oo the Fourier integral
[x((());
/ R
2ak- u(dx),
(14)
where j(dx) is the probability measure representing the distribution from which the points in the dataset are sampled.
The NUDFT (as well as any spectral analysis method relying solely on information at finite number of points without additional interpolation or assumptions) reflects the spectral information of not only the target function, but also the underlying distribution of the datapoints itself. It is in principle possible to separate out these contributions using density estimation methods, e.g., a multidimensional generalization of the technique in [16]. Presently, however, simply define the frequency spectra of interest as that reflected in Equation 14 (and its approximations) and frame our analysis self cefffconsistently in terms of this definition. In particular,
empirically observe spectral bias (see also (277), see that certain transformation reduce high frequency energy, and propose an inductive bias that consistently outperforms other methods, all using this definition.
A.2 Limiting behavior of the NUDFT with respect |
Some of the limiting behavior of the NUDFT amplitude spectrum can be understood from Equation 1 Specifically, consider the expression for (kk))
N Σ f(x,)e
N-1 N
Pý(k) = N2
2 2n
f(x;)[(x)) cos (2rk (x; xÿ)) j=it1
Σ
N
N2
(15)
The overline represents the mean of quantity evaluated over the x,, At small [k], the RHS of Equation 15 is approximately j", the mean squared of the function or labels. This is the reason for the 1/ʃ scaling front of Equation ( otherwise, the scales of the Fourier amplitudes would not be directly comparable across datasets. As [k| increases from O, at some point, P3 starts to deviate from the mean squared. This happens when the cos term the expression above begins to deviate from for a significant number of points. This occurs schematically when |k| ~ Az), where AT is a typical separation between datapoints. In the other limit, once | >>>>> 1/(2nAx), the NUDFT is probing distances much smaller than the separations of points in the dataset. At this point, the NUDFT ceases to provide meaningful spectral information, as the dataset does not convey information about the target function such small scales. The spectrum becomes noisy, since the last term on the RHS of Equation (15) features many cancellations, resulting in fluctuations around the first term, /N,, on the RHS. The most interesting region from the standpoint of our analysis is that between these two regimes, describing collective oscillatory behavior, which is the region focus on in our analysis.
There is another way to see that the 1/N asymptotic behavior of Py at large frequencies is an artifact of the discrete distribution of datapoints. The NUDFT approximates the integral 1 p(x))(x)) (oxx))-2rikx 2rik dx. For fxx) continuous, bounded, and compactly supported, the in- tegrated spectral energy density will be finite. However, the corresponding integral of the squared
13

---

NUDFT amplitudes, which should yield the same result as N - oo by the Plancherel theorem. does exist for finite N. The problem again arises from the behavior discussed in the previous paragraph. To obey the Plancherel theorem, the result in the continuous limit needs to fall off fast enough in |k] to be integrable, meaning that the /²/N asymptotic behavior discussed above is an artifact of the discrete approximation. We therefore limit our analysis to below this range, which motivates the upper cutoff k. appearing in several of our expressions.
A.3 Approximating the high-dimensional spectral energy integral
Our spectral analysis primarily relies on the quantity Sy defined as sum of squared NUDFT amplitudes as in Equation (2). This method considers spectral energy along single dimensions of I time. Here we describe an alternative convolution method for approximating the full n--diennsionl spectral energy integral over finite high-frequency region in R". In Appendix A.6 we show that both methods yield similar results for the datasets analyzed.
The method here is similar to that in [16]. Consider the approximate energy spectral density, (y(k), for function f integrated over high- fighffreueeccy region K R".
SK = 1/e Py(k)dk.
(16)
Rather than attempting to perform this high-dimensional integral in Fourier space, we can make of the convolution theorem and work directly in feature space. Consider region K defined by Ko < 1x S k., where oo is the Los norm, kg is a threshold determining the start of the high-frequency region, and k. a cutoff of the high frequency region introduced for the same reason as discussed in the previous section. We can then write
SK f Py(k) [k] Sk. dk
[ Pj(k) (kl=) Sko
(17)
Defining the function
g(x) = / [(x') II sine (2k. x'):) p(dx'),
(18)
the convolution theorem dictates that
Fxlg(x);(((kk)) (xl 11k 1/
(19)
so that
12(x) dx / [=(((())
/
(kkk)d
(20)
Ikl..
Approximating the x' integral in Equation (18) with Monte Carlo integral sampling from the density defined by μ., substituting the resulting expression into Equation (20) and using the definition in Equation we arrive at
[K( ~
(2h:.))
Σ {(x)s(())) +
N
II
sinc [2k,(x; - x,)e
II
sine [2ko
xx)el] (21)
N
f=1
This estimate of the spectral integral in Equation (16 is evaluated for binary classification datasets in Appendix A.6 below.
A.4 Defining the high-frequency regime
Both the NUDFT and fully-integrater energy spectrum approach rely on two different cutoffs to define the high-frequency range. The lower cutoff, ko, sets the start what we consider the high-frequency range. We take ko = 0.5 in our analyses, motivated by two observations. First, in our conventions, and for features sealed to unit length, | < 0.5 corresponds to frequency modes that fit less than half a period in any given dimension of the feature space. This corresponds to long-distance (i.e., global) fluctuations and so the high-frequency regime should be considered for |k| larger than this value. The second motivation comes from the behavior of the normalized sine function, which describes
14

---

the Fourier transform of the unit rectangle function. Say we had dataset with all points having the same label distributed uniformly in the unit interval. The NUDFT would then approximate the normalized sine function, which begins to decay when its argument is around 0.5. For larger [k], the envelope of the squared NUDFT amplitudes will decay as 1/lk|², consistent with the fact that the corresponding target function does not have high-frequency components. Of course we are also free to choose values larger than 0.5. We have verified that the other choices of ko ~ do not affect our conclusions.
As pointed out in Appendix A.2 at high frequencies the behavior of our methods relying on discrete points fail to reproduce the correct decaying high-frequency behavior of the continuous Fourier amplitudes. Care must therefore be taken to properly select k,, especially for computing Sk: for Sg the high-frequency JzzN behavior adds a constant, but this behavior leads to SK о (2k,)" which would dominate the result numerically if we did not account for this behavior. As result we implement the following procedure in computing SK to compare between datasets: for each dataset, we first compute Sx for several different values of k., and choose value near where the integral begins to scale as (2k..)". We then manually enforce that the contributions to SK above this value vanish. The corresponding integral then has the expected behavior, and we can directly compare Sy between datasets at arbitrarily large k, since the integral will saturate once it comes close to scaling as (2k.)". Finally, for computing Sg. we simply cut off the sum in Equation (2) at given value near where Sg approaches the Ja/N. In practice, we take k. = 20 when computing Sg. but have verified that the results do not significantly change when varying this value.
A.5 Evaluating Fourier transforms along principal components
In several of our analyses, we evaluate the NUDFT (x) along specific directions, such as those defined by the principal components (PCs) of the data. This approach was also used in [27] and we provide further details about this procedure here for evaluating along general directions in R"
The principal component decomposition (i.e., PCA) typically takes one point x the original feature space and yields transformed vector XPc with entries corresponding to the projection of the mean-centered features onto the PC directions. Focusing on one PC direction, we have
EPC
êPc (x ()
(22)
where m is the vector defined by the mean of each n dimensions of the feature space over the dataset, and êpc is a unit vector pointing along the PC of interest in the original feature space. To take the
Fourier transform along the direction êpc, kpc êpc, and plug this value of kpc into Equation| Using Equation 22
define kpc kyckre. where kpc k êpc and
we have
NV
Σ f(xy)e 2Tikpp - 22rkpp.mm
Σ
f(xy)e 2Tikpcare.
(23)
The phase factor in front of the RHS will drop out when take amplitudes. The above expression tells us that to obtain the FT along the direction êPc defined by given principal component, we can simply plug in the values of ZPC corresponding to a given datapoint and treat the Fourier transform as dieeemmensional with frequency kpC (up to an overall phase factor). However, this is not a one-dimensional Fourier transform: it the full (approximate) n-dimensional Fourier transform, evaluated along given one-dimensional subset of R
The above method applies when evaluating dataset along of its own PCs. In computing Sj. e.g., in [able we typically average over the PCs defined for different transformations. In this case, one should directly substitute the corresponding kPC == êPc into the LHS of Equation (23
A.6 Spectra of binary classification datasets and correlation between Sg and Sx
Here, we provide supplementary details related to our spectral analysis of the various binary classifi- cation datasets considered in our experiments.
In Figure 5 we show squared NUDFT amplitudes for the binary classification datasets of interest. The squared amplitudes (k) are evaluated for unit-scale, ranked and scaled data, with rank implemented as quantile transform and scale as standardization. The amplitudes are evaluated
15

---

house 16H _
Mectricity-normaalize
10'2
P.lker)
10-4
10"6
10
10a
10a
10
10a
10a
[k]
[k]
pol
phoneme
10-2
P,0kac)
10
10
10a
10a
10-1
10
10a
[k]
MagicTelescope
bankMarketing
10-2
length
10-4
ranked
scaled
10
10a
10a
10
10a
101
[k]
[k]
MiniBooNE
kdd-small
10-2
kkke
10
10-6
10
10a
10:
10
10
10a
[k]
kk]
albert
callfornia callfornla_housing
Pylkec]
10-1
10-9
10-1
10a
10a
10-1
10a
10a
[k]
[k]
Figure 5: Spectra for binary classification training datasets determined by Equation 11 for the rank and scale transformations, compared to the datasets with unit length features. Results are marginalized over 10 different random seeds for the train-test split, and over kpc corresponding to the 1st PC after each transformation applied. The solid lines are the resulting mean values, with shaded regions representing the 95% confidence intervals. Different transformations significantly affect target function frequency spectra.
along the first after each transformation and for 10 different random training splits of the data. Solid curves represent the mean and shaded regions the corresponding 95% confidence intervals after
16

---

10a
10a
Size
10
10
10
10
10a
10
10a
SK
Figure 6: Comparison of different methods for quantifying the spectral energy at high frequencies. The horizontal axis corresponds to Equation (2T), and the vertical axis Equation (2). Squares, circles, and diamonds correspond to unit-scaled, rank, and scale features, and the different colors correspond to different datasets. The two methods are highly correlated.
marginalizing over all random seeds and different kFc directions. From the figure, it can be observed that the various transformations affect the -eggerrey composition of these datasets in different ways, and in some more than others.
The corresponding values of Sy for several of these datasets presented in Table These computed the same way as the results in Figure ( avee.aver over random seeds and PCs), however still correspond to examining the spectrum along one dimension at time. To assess the extent to which the one-dimensional results represent the full spectral properties in R", in Figure 6 we compare the resulting values of Sg against the quantity SK of Appendix A.3 Results for all binary classification datasasets except for alberr, which features k. below Ko for some methods when computed using the full Fourier integral. In order to compare the different datasets similar scales, we compute s1//". since the dimensions of the volume integral over k will be different for datasets with different feature space dimension. In computing SK. , due to the ( computational complexity, subsample each dataset to 10000 records if larger than this size. As seen in Figure 6 the two methods are strongly correlated: each dataset individually has Spearman rank correlation coefficient of when comparing Sg and Sk. From the figure, also the general behavior demonstrated in the spectral plots previously: the unit-scaled representation tends to imply the most figuefrre target functions, followed by rank, implemented as a quantile transform, and scale, implemented as standardization.
shown
B Spectral Analysis of Neural Networks on Synthetic Data
In this section, provide additional details and experimental results related the synthetic data experiments reflected on the RHS of Figure (reproduced in Figure 7 for convenience).
We consider binary classification synthetic dataset consisting of a single numerical feature, for simplicity. Heterogeneous tabular datasets used for classification tend to feature rapid variations of the class labels over short distances in feature space. To simulate this scenario, consider normally- distributed feature g, and an oscillatory target function that depends on a function g(x) of the feature x, generating the class label
11 y(z) E (1 A, sin (kgg(=) + pi)) 2.
(24)
When sufficiently high frequency components k; are considered, simple multilayer perceptron (MLP) will have difficulty learning the target function, as expected from the spectral bias phenomenon. This is illustrated in the top panel of Figure For this example, take A/(0,5, 0.01).
17

---

Unit Length
1.0-
0.8
Prediction
0.6
data
0.2
0.0
Ranked
1.0
0.8
Prediction
0.6
0.4
0.0
0.0
0.6
0.8
1.0
Feature value
Figure 7: (Top) Data with labels varying small distances are difficult for neural networks to learn, but easy for tree- tree-based based methods. (Bottom) Transformations that redistribute the data can increase the scales over which the feature labels vary, effectively lowering the frequency of the target function and making it easier for a neural network to learn.
k (1,5, 5, 10, 20} 2n and randomly sample the corresponding amplitudes A; and phases sp: from the continuous uniform distribution on [o, 1]. We also take g(æ) as the rank transformation of Equation (9) for illustration. order to compare NN's with features at a common scale, we first min-max normalize the features to have unit length. The corresponding target function has S x 0.02. A 2-hidden-layer MLP with width 256 is then trained classify 5,000 points using binary cross-entropy loss for 10000 epochs after an initial 250 epoch tuning stage to select the best learning rate. The resulting predictions are shown in black on the top panel of Figure 7 and do not match the training data very well, particularly in regions where the tag varies more rapidly (i.e., the spectral bias phenomenon). The NN evaluated on validation set 1000 points distributed in the same way as the training set yields an AUROC of 0.935. On the other hand, we also train an XGBoost classifier using default hyperparameters for 100 boosting iterations (the performance converges quickly so boosting for additional iterations does not improve performance). The corresponding predictions are shown in orange. This tree-based classifier performs well and is able to reproduce the rapid variations of the target function, since, long the algorithm's 's histogram binning is fine enough, straightforward to recursively split this one-dimensional feature space into regions with y = 0 and 1. The corresponding AUROC is 0.998.
There can several approaches taken to overcome this performance gap. One approach is focus primarily the underlying model: perhaps tuning the hyperparameters extensively or incorporating more sophisticated architecture could lead a better-performing configuration. However, this process quickly becomes computationally intensive and is dataset-dependent. An alternative approach, and the one we study in this work, is to focus on the representation of the data. The frequency characteristics of function defined on a dataset are inherently tied to the representation of the data itself. Stretching or warping the feature space while maintaining the label values will in general change the length scales over which the function varies.
To investigate this effect, consider the impact of re-arranging the datapoints such that rather than following their original distribution, they are now spaced as far apart from each other as possible within fixed interval, which we take to be (0, 1]. This familiar transformation: mapping the original distribution to a uniform distribution along the feature direction, in the rank transformation of Equation (g). However, the resulting effect on the example Figure is dramatic:
this re-distribution of points reduces the frequency of the target function, as clearly seen in the bottom
18

---

panel (the rank-tran transformed target function has Sg ~ 0.01). We train the same two models (MLP and XGBoost) with the same hyperparameters as before on the transformed data. The tree-based model performance does not change significantly. but the MLP's performance improved dramatically, with the AUROC evaluated on the transformed validation set increasing from 0.935 to 0.989.
For this synthetic example, the simple inductive bias of seeking a representation of the data that "smooths out" the target function in a way that does not negatively effect optimization significantly boosts neural network performance and reduces the performance gap with respect to the GBDT-basee model. Our analyses utilizing realistic heterogeneous tabular datasets yield similar results.
B.1 Connection to computable spectral bias
Since we know the underlying density of the synthetic datapoints, we can directly compute the spectral bias of the MLPs trained on the pre- and post-transformed data following the computable definition of spectral bias proposed by [16]. In particular, quantitative definition of the spectral bias, SB. can be provided in terms of the fraction of variance unexplained (FVU) compared between a predefined low- and eighffreuency region. A value of SB > 0 indicates a larger fraction of FVU in the high-freqqence region, hence a spectrally- biased NN, since the NN fits the data worse at high frequencies. To compare two NNs, consider epoch for which the performance of the two NNs comparable: the epoch with largest validation AUC for the unit length NN and the epoch with the closest corresponding performance for the ranked NN. The FVUs are approximately equal for both networks, however the spectral bias of the ranked NN, computed as in [16], is lower than that of the NN trained unit-length features: SB 0.58 for unit length, SB 0.42 for rank. This indicates that for NNs achieving similar performance, the rank transformation has produced network with lower fraction of its error in high frequencies, and therefore has reduced the negative impact of spectral bias on NN training, since the relatively larger low-frequency errors can be effectively reduced by further epochs of training, in contrast to high-frequency errors, which are more difficult for the NN to reduce. This allows for significantly better performance overall for the mankttransformed data. Similar effects are suggested by our analysis of realistic multidimensional tabular datasets: transformations resulting in Nower-frequeng NNs for a fixed performance metric can typically be further trained to reduce their correspondingly lower- twerr-frequecy errors, hence mitigating the impact of spectral bias in learning the corresponding target function. Notably, this analysis is an attempt to establish the initial connection between our approach and [T6.. A more extensive analysis of the computable spectral bias of all the methods we consider in this paper will be included in future work.
C Frequency of Target Functions in Tabular and Image Datasets
In this section, we provide details related to the results on the LHS of Figure I contrasting the target function spectra of image tabular datasets.
Our study focuses on mitigating the effects of spectral bias in the context of tabular deep learning. One of the empirical motivations for our approach is the observation that target function spectra of tabular datasets tend to be of higher-frequenee
than similar examples from popular image datasets. To compare frequencies across datasets, the features should be normalized in a consistent way. We take unit length normalization as the baseline for our comparisons. We consider several popular benchmark datasets from computer vision applications: cifar- 10-arll (https://oopeml. org/ a fashionn.fs (nttps://openn) d/554). svhn (https://openni) wwtp////g////////e 4/188) ) and scene (https: Nittpa://openm.. http://ppamm.oo/////4///1// org/d/41471) After min-max normalizing, we treat these image datasets in the same way as our tabular multi-class datasets: for each of 10 random seeds, we consider a training set sampled from the full dataset, and compute the NUDFT along the first PC direction, assigning binary labels for all 1-vs-rest class assignments, and computing the mean over all seeds and binary label assignments. The resulting squared NUDFT amplitudes. Py (k), are plotted in color on the LHS of Figure [ To compare against tabular dataset spectra, we use the same procedure. The envelope containing all such spectra In this figure, all
org/ a/ (/40996), mnist-784 (https:
org/
computed for our 14 tabular benchmarks is shaded in blue on the LHS of Figure |
amplitudes shown are normalized with an implicit I/T" to account for the different levels of class imbalance across the datasets.
19

---

From these results, we immediately see that, using our definitions, the image datasets considered have significantly smaller high-frequency Fourier amplitudes when all features are normalized to the same scale. While in our study we simply use this observation as an empirical motivation for addressing the high-frequency nature of tabular dataset target functions, would be interesting to extend this analysis to more examples and to study the underlying causes of these differences in future work.
D Illustration: frequency reduction and the effects of spectral bias
In this appendix, we provide informal analytic arguments, based on the results of [22], illustrating how frequency reduction can improve NN performance given their inherent spectral bias. Theorem 1 of [22] provides the analytic form of the Fourier amplitudes of a general ReLU network, f(x). The authors also show that along each direction of k space, these amplitudes are upper bounded as [k)] S (
Here, Ly is the Lipschitz constant of the NN for given set of parameters 0, N2 is the number of linear regions, and < A S d depends on the orientation of k with respect to the polytope faces represented by the NN. In any realistic setting, there is maximum My that can be achieved through training. and therefore the amplitude of the NN Fourier coefficient for a fixed k is bounded from above. Along each k direction, assume that the target function behavior at high frequencies falls off more slowly than t --A- Then, for large enough & the target function Fourier amplituds,, 7kk,, cannot be fit by the neural network with fixed architecture and parameters. Let Ω2 denote the corresponding high-frequency region of R", Reducing the 1.2 norm of the target function Fourier amplitude over S2, So [77 [k]] relative the corresponding integral over IR" (assuming the target function Fourier amplitudes are jguure-ittegrable) will tend to reduce the corresponding error arising from this spectral bias when evaluated on particular set of data points. Our definition of frequency reduction in terms of the NUDFT amplitudes can be viewed as proxy for reducing this _ norm.
Applying the scale transformation with scale factor a > 1 can lead to the aforementioned reduction of relative spectral energy over Q2 The Fourier amplitudes for function g (xx are related to those for the corresponding function, Iscaled (x) acting on scaled inputs as g(k) 1/a x Iscated (k/a). This relationship shows directly that scale with 1 maps a given Fourier component of the original function to a component at reduced frequency (and reduced overall magnitude) after applying the scaling transformation. Thus, denoting the corresponding high-frequency domain of the scaled target function as Qscaled. in general Iscaled C S2 (again assuming a > and that the original target function decays more slowly at large i than the spectral bias bound). The smaller domain of integration, together with the relationship between ÿ and Iscaled, can reduce the relative amplitude of the spectral integral. The corresponding argument for rank further depends on the underlying distribution of the feature. Consider the simple example of uniformly nniformly-disributee feature x over an interval [21. z2). Then rank(z) acts in precisely the same way as scale with a 1/(xz 21). and the same effect is seen (assuming X22 Ii < 1). Similar arguments can be made with other assumptions about the underlying distribution.
These arguments suggest how the transformations we consider can mitigate the impact of NN spectral bias, however they do not account for the other potentially adverse effects of rank and scale on the optimization process. In practice, these effects are important and motivate the learnable convex combination of rank and scale we propose, as it is implicitly regularized by the loss function itself. We plan to analyze the corresponding training dynamics analytically, and further formalize the illustrations above, in future work.
E
Experiment Details
Datasets and Training. We train and evaluate all models across 10 random seeds, and report mean metrics with their 95% confidence intervals. We use 70% of each dataset for training, 15% for validation and 15% for testing. We report evaluation metrics on the test set that correspond to the lowest validation set loss for each model. We use simple target-encoding (1 to convert categori- cal variables to numerics, in order to avoid introducing potential interactions between categorical embedding techniques and our metrics of interest.
Hyperparameter Spaces. We conduct hyperparameter tuning for every model across 100 configura- tions using the validation set. We use HyperOpt [4] to tune the hyperparameters of all approaches
20

---

Parameter
Distribution
depth
Uniformint 2,
width
UniformInt [28, 256, 512]
learning rate
LogUniform (10 10 -
Table 3: Hyperparameter space for MLPs. Batch size is set to 128 and is not tuned. PyTorch's 20 implementation of Adam [17) with its default parameters is used for optimization.
Parameter max depth num. estimators learning rate
Distribution
UniformInt [5, 10, 20)
UniformInt [100, 500, 1000]
LogUniforas [10-5, 10=
Table 4: Hyperparameter space for XGBoost. The rest of the XGBoost parameters the defaults of the DMLC XGBoost package 77
Parameter
Distribution
1dd The
Uniformint [56, 60, 64
Msttps
Uniform [1, 3]
learning rate
LogUniform [10 10
Table Hyperparameter space for TabNet. Following the suggestion of the authors, we use the same value for ny and Ra. and set the batch size to 1024. PyTorch's [20) implementation of Adam [17 with its default parameters used for optimization.
considered. Since our main objective is to surface the impact reqquency-reducingg inductive bias and not perform extensive benchmarking, focus tuning the parameters that control the model complexity and learning rates. Specifically, the parameter space considered for each model are shown in Tables 5nd
Baselines. As baseline, we evaluate the impact of common normalization techniques on our metrics of interest and present our results in Figure 2 Specifically, study MLPs with standardization. robust scaling and quantile normalization (2I]. We choose these transformations to compare our methods against for reasons. First, these the most common transformations used in the tabular deep learning domain. Second, these transformations are instantiations rank and scale. Therefore, besides normalization, they impact the frequency spectra. We use scikit-leane (21) implementations of these transformations with their default hyperparameters. In our experiments we observe that, among these baselines, there is not single transformation that consistently performs the best across all datasets. This observation consistent with the existing tabular deep learning literature, where studies that different datasets find different transformations working the best. In these studies, the quantile transform, standardization and min-max scaling methods are the most commonly used.
Frequency Reduction and XGBoost. In addition to our various MLP experiments, we also study the impact of frequency reduction XGBoost performance. We first train an MLP until convergence, then use the weights 6g to transform input data to a lower-frequency form. We then train an XGBoost model using the transformed data. From Table 6 it can be observed that reduction does not improve XGBoost performance. However, it improves simple MLPs to provide better performance than XGB on 3 datasets, and comparable performance 2 datasets. The proposed method does not improve upon simple MI.Ps the baseline performance is already high (i.e., > 99%), which is expected as these datasets do not feature fighfrreuueny information that cannot be learned by MLPs with unit scale.
E. Weights learned by the proposed approach on 14 datasets
We investigate the weights 0g learned by our proposed approach on the 14 datasets. Since these weights get updated jointly with the neural network weights 0y, this investigation is essential to
21

---

dataset
XGB
XGB
MLP
MagicTelescope
93.74 0.14
93.67 0.17
MiniBooNE
98.7
9.07
98.79
bankMarketing
93.58 # 0.12
9.16 0.12
california housing
97.22 0.1
97.22
96.01
electricity- nectriitty-oormalized
98.81 + 0.04
9.151 0.00
house 16H
95.62 + 0.15
0.152
95.68 0.12
kdd-small
99.28 + 0.05
99.28
99.17 # 0.08
Diabetes
71.02 0.09
71.03
68.88 # 0.13 0.13
albert
76.47 + 0.06
0.04
72.8 0.04
credit-card-clentss
78.12 0.21
0.21
78.16 0.23
yve_movements
91.38 + 0.25
0.25
80.34 0.3
jannis
88.14 0.13
0.15
87.77 0.11
phoneme
96.74 + 0.15
# 0.16
pol
99.94 + 0.01
99.94
99.94 0.01
Table 6: Un-normalized AUROCs for XGBoost (XGB), XGBoost with low-frequency embeddings (XGB*) and MIP with lowffrequency embeddings (MIP**).
understand whether or not fg merely works as a set of additional trainable parameters that adapt to changes the rest of the network parameters. If theta 0g depends on 0y, then the low-frequency mappings produced using Og may not be useful general purpose representations. Figures a depict the weights learned by our proposed approach. Shaded areas represent the 95% confidence intervals across 10 random seeds. Notably, neural network weights f get initialized differently across the seeds, while 0g gets initialized to the same values. Specifically, we initialize all scaling weights as 1, and all ranking weights to 0.5. More advanced initialization techniques may be explored to further improve performance. From the figures, we observe that 0g converges to similar values across different random seeds all the datasets considered. Additionally, observe that for the datasets mank-marketing (bank) and diabetes, some of the scaling weights converge to zero, eliminating the of the corresponding features and causing the combining weights not get updates (i.e., staying at 0.5). This behavior may be interpreted as a frequency-reduucng feature selection mechanism: feature's contribution to the performance is not sufficiently high, compared to its contribution the frequency of the decision boundary, the neural network may choose to drop it. This behavior should be further investigated and compared methods that were designed features. Overall, the results in Figures { and 9 along with Figure 2 suggest that our proposed method is indeed enabling the network learn useful representations of the data biased towards lower-frequency target functions.
select informative
E.2 Un-normalized AUROCS on all dataset and approaches considered
We provide un-normalized AUROCs of all approaches considered in Table 7 The methods unit, standard, robust, quantile and selective-quantile are applied to MLPs. Based on our experimental results, observe that the best performing baseline varies across different datasets. As result, in order to produce a comparison of the overall performance, we provide average normalized metrics in Figure 2
F Computational Complexity
Our proposed method introduces the additional parameters 0, {0>, or. 0²) The scaling and combining weights, 07 and 0g are learned together with the neural network weights 0,. The parameters 8 are set by fitting the ranking transformer to the training set as preprocessing step (i.e., only once), and extracting mapping from reference points in the training sets to their ranks. We sample 1000 points to extract this mapping.
In all experiments, we allocate two scaling weights (for raw and ranked features), and one combining weight for each feature. Therefore, the additional parameter complexity introduced to neural network with input size N is O(N). In comparison, additional fully connected layer introduces
22

---

dataset
XGB
XGB*
TabNet
TabNet*
MLP
MagicTelescope
93.74 0.14
9.17
90.08 # 1.54
91.56 0.59
93.67 + 0.15
MiniBooNE
0.02
9.07
91.74 89.47 0.65
98.59 0.03
98.79
bankMarketing
93.58 0.12
9.15
89.22 0.51
92.66
california housing
0.1
9..2220.1
NA
NA
96.01 #.1
98,81 0.04
9..0
84.45 + 2.58
84.72 1.97
94,51
house_ 16H
95.62
9..62
89.24 + 1.17
94.54 0.13
0.12
kdd-small
99.28
96.62 + 0.73
98.72 0.15
99.17
Diahetes
71.02 0.09
7.003
NA
albert
7.04 NA
NA
credit-card-clints
78.12 0.21
78.1 NA
NA
78.16
eye_movee movements
91.38 0.25
9.22
NA
NA
80.34
jannis
88.14 0.13
8..58 0.16
NA 1 0.28
87.77 0.11
phoneme
96.74 0.15
81.28 2.9
94.84 0.12
pol
99.94 # 0.01
99.94 0.01
77.18 + 6.46
94.41 2.45
99.94 0.01
dataset
unit
standard
robust
quantile
sclective-quantic
MagicTelescope
93.32 # 0.16
93.27 0.18
9
93.27 0.18
93.37 + 0.18
MiniBooNE
96.47 = 0.02
9.02
98.74 98.77
98.72
bankMarketing
92.71 0.14
0.15
92.11 9118
0.14
california housing
95.56 0.1
95.75
9.06
9.12
95.63
0.09 0.09
olectricity
93.04
9.12
02.
93.85
0.11
house_160 16H
9
0.13
0.15
0.12
0.16
kdd-small
99.32
0.06
99.01
0.05
9.05
Diabetes
69.65
0.02
0.11
6.11
6.09
albert
7
0.06
7.09
7.02
7.027
credit-card-clicnt
0.25
0.26
0.2
7.23
7.25
eye mee_meeeentss
79.86
8.36
7.25
0.3
76.98
jannis
86.67 # 0.11
8.11
07.1
86.99 0.13 8.12 0.23
phoneme
94.13 0.25
94.71
94.11 = 0.26
94.12
pol
99.96 =
0.0
99.94
99.32 0.03
99.94 + 0.01
Table 7: Un-normalized AUROCs of all approaches considered. An asterisk (*) indicates that the corresponding model uses the proposed www-freuency transformations. Note that our TabNet experiments only consider binary classification datasets.
(d,,NN, where d, is the width of the subsequent layer. Notably, our approach be extended to high-dimensional setting similar to [1]] to map each feature to a k-dimennsioal vector that scales and combines the same feature with its ranked counterpart in different ratios. Although this extension is out of the scope of this work, the complexity of this approach would O(kN), where our current approach ;=
G Limitations
In this paper, we focus on building the connection between the spectral bias of neural networks and its impact on tabular deep learning performance. We show that heterogeneous tabular datasets tend to feature high-frequency target functions as defined by the Fourier analysis methods presented. We also observe that with frequency-reducing transformations, these target functions can be smoothed to alleviate the negative impact spectral bias. Consequently, we propose using frequency reduction inductive bias while learning with neural networks. We discuss some of the limitations of our methods and analysis below.
Limitations of the proposed method. We implement the proposed inductive bias as neural network layer that learns scale and map points along a linear trajectory between (scaled) input features and their ranked counterparts. This implementation is motivated by our empirical analyses of commonly-used transformations in tabular deep learning, and an attempt to generalize them while retaining their frequency-reducin properties. However, our implementation only covers subset of such transformations. Indeed, one may carefully design transformations that guarantee frequency reduction with fewer negative side effects the other training dynamics. Also, limit
23

---

our experiments to independently transform each feature. More beneficial transformations might be found by instead first grouping features with informative interactions (i.e., subsets that are compatible to form informative higher-order features), then attempting to transform them to a low- low-frequency form. As tabular datasets feature heterogeneous feature spaces, identification of such compatible feature sets may be challenge. It is possible that, for example, pair of features may meaningfully interact in certain subregions (i.e., subdomains). Then, one may choose to identify these regions and represent them in different dimensions of the feature space. Such mapping of features onto high-dimensional spaces to may help reduce target function frequency by transforming the feature space to be more homogeneous. We do not consider such approaches in this work. Finally, our methods only transform the inputs to the network, while more generally one may apply our frequency reduction inductive bias to all network activations to ensure that learned representations also have low-frequency form.
Limitations of the experiments. Our experiments mainly focus on observing the dataset-dependent effects of existing transformations that reduce target function frequency, and demonstrating how this dependency can be alleviated by using frequency reduction an inductive bias during learning. In doing so, we limited our analysis to classification problems, although our methods can be applied in regression settings as well. Also, although our experimental setup enables fair performance comparison between MLPs with different transformations, the corresponding results may not reflect the best performance achievable by MLPs. This is because do not techniques such as batch normalization, residual connections and weight decay regularizers that are shown to improve MLP performance tabular data. This scope will be extended in future work. Our experiments with other neural network architectures are also limited to TabNet on binary classification datasets. Future work will also include extensive experiments with additional tabular data-specific deep learning architectures in order to explore the relationship between the improvements they offer and the frequency spectra of the functions they learn. Finally, although we qualitatively observe that our proposed methods alleviate the negative effects of spectral bias through our analysis of Fourier spectra and the corresponding impact on performance for all datasets considered, our quantification of spectral bias (see Appendix [BI) is limited to synthetic datasets due to its computational complexity. Such quantification can be extended beyond synthetic data experiments in the future.
H Implementation
In this section, we demonstrate how to implement the key components of our analyses and proposed method using PyTorch (0)
H.1 Proposed method as a network layer
The module CustomLayer implements our frequency-reducing transformation. In our implemen- tation, in addition to scaling and combining, we also incorporate shifting factors in order to center features when necessary. We empirically observed that addition of the shifting factors did not change the performance of the neural network significantly. Notably, this implementation assumes that the input is in the form x [xare, Xergl = |Xraw Xyankedl where ranked is generated using scikit- -earn's 's quantile transform. Also, for the "raw" features, our implementation uses their scaled version as input to support training stability.
class CustomLayer Modduleee) # proposed layer def isemif size):
self size size weights torch. Tensor.eenrrre (torch.
oneorchh.oee(( ones ((1, size)) 0.5)
scaliing_actors__ torch. Tensor (trncorttrchoones((11 ones((1, size)))
shiftiig_ src torch. Twwsor (torch.zeros((..
size)))
scaling.facttrs_.tz torch. torch.Tensor
ones ((1, size)))
shifting.factors_tttt trg = torch. Tensor (torch. zeros size)))
self weights Parameter (weights) wal.fsssaling_wwihhtts..oo
nn. Parameter
sell solfsshiftiing.wshhts.eco nn. Parameter (shiftting_
self haaf.scaliin_wweigtte ttr = nn. Parameter (scaling_factors_ttg
self wwwsshiiiiggwwwighheee nn. Parameter shhiftinn_facttrrte
24

---

def forward(seff, x):
x_src x(;, :x.size [1]//2] raw input
x_txg x(:, x.size() [1]//2:] # ranked input
aelf weights
c_arc self self.
u_arc self. wwt.sshhttingg.igghtt....c.
c_trg self saalsssaainng.eightts.
tig self www.stiig..wiigitss..tt
x [rc + u_arc) c_src) w) + (((x_tgg u_trg) c_trg) (1 x))
return x
class MLP (nn. Moduul)):
def _.init_..seff, layers, activation,layer p_dropout, bias, num cum_casses)
layer_ - (in.LinearrlayersSSll [i+1l, bias-bias) for i in
range (11
self. layers mn. Moduuell
self. calffcaaseeeirr mn. Linear (layers mum_classes)
self .drpoott = nn. www.Drooot(ppp..pppoo.
self activattivvi self. celf.cuustmlayee Customlayer (layerss
activation_layer
def forward(self,, x):
x self. (xl..cccsstmml
for i in range(1en(self (aayea())). .layers)):
self layers [i] (x)
self aattvattion_laeeet
self. duf..droput (x)
yhat self classifier(x)
return yhat
H.2 Functions to calculate spectra
To efficiently compute NUDFT amplitudes, utilize the nfft_ fff__ajjontt function from the third
party package nfft (https://githir) cttp:tuubb.comm/aake//////tt The implementation is as follows:
!pip install nfft
from nfft import afft_adjoint
def (xDFT(x, y, kmax, nk): return (1/1en(x))
y, (nk+1)) [nk+1:]
def Drnctrum_NUDFTICO
kmax=50, nk=1000);
kvals np. linspacee,, kmax, nk+1)
mufft NUDFT y, kmax, nk)
return [kvals, np. arraaynuufff,, ctype="complex, ")]
H.3 Selective-rank
In our experiments with selective rank, we set transformer to scikit-learn's quantile transform.
def wtw_.aaturre_wtttthrdddddddd (X_tr, y_tr, transformer, threshold-0.755
features []
25

---

X_tr_quantile = transformer. .ftttmrrrff....e..... trantrttraofoomccct
for range (X_tr. shapeee [1]): [1]):
v_raw (p_ctrumX__tr__ i] y_tr) [lower_ _ound_d
v_qnt spectrum (aatrum(X_tttaaaaae tet.ruuuanileeei... i], y_tr) Lower_bound:
metric - np. mean (abs (v__qq)) ** 2) /np meap.mmennneee (abs (v_raa) 2)
if metric threshold: features.append
else:
fww.ssorr_ffatuuresooo (1pend
return features
def selective selective.tramsfooeeoooeeee tww.ssoommdaaaaseeeeee..e scaler, config, transformer):
x_tr, y_tr dataset ['tr'] [X]], dataset['tr']]'>>
X_va, y-va dataset ['va'][[X] dataset dataset["va]]
X_te, y_te dataset (ates
d ['y]]
transform cols get_features. direehool.roorrigg....hheeaaaa)"
(repuuccdrreeuueee(XXX.... y_tr, transformer,
ColumnTramsformer ([""sel-rank", transformer, transform_cola)])
ft. y-tr)
X_tr[; transform_cols] x_va[; transform_com cols] ct transform va)
ct. transforrm (X_tr)
X_te(; caanforr_coos).
ct. transforr (X_te)
return X_tr, X_va, X_te
26

---

1.40
0.75
2.5
0.93
0.55
100
23
40
63
18C
30
60
100
kpoth
4020h
bank
0.0
100
40053
#0402
80201
creditcard-ccleene
3.4
0.99
0.35
0.5
110
100
clobetes
1.59
0.75
0.99
0.59
1.53
1.3
0.75
opoch
opoch
opoch
cyp, movemennsssss
1.59
0.50
0.35
0.58
100
Rouse__1h
Sooke 093
0.7
1.50
2.2
-25
Scontand
0.75
0.31
0.23
100
e00ch
0900h
Figure 8: Weights learned by the proposed approach on all 14 datasets, part
27

---

NyScceeem 0003
80
0.
100
kpocc
spacch
spoch
0.5
0.09
minibonne
0.6
0.2
Doallo
2.00
0.995
0.4
opoch
o50ch
cpoch
0.4
Figure 9: Weights learned by the proposed approach on all 14 datasets, part 2.
28


Reviews:
[{"review_id": "LWK49was45", "reviewer": "Reviewer_uyTw", "summary": "This paper proposes an inductive bias for tabular deep learning to bridge the performance gap between deep learning and tree-based methods on tabular data by reducing the frequency of irregular target functions through scaling and ranking transformations. Deep learning methods underperform tree-based methods on tabular data due to the interaction between irregular target functions and the tendency of neural networks to learn smooth functions. Spectral analysis tools can be used to identify the irregularity of functions described by tabular data sets and the potential for smoothing through scaling and ranking transformations. The proposed inductive bias of frequency reduction through scaling and ranking can significantly improve the performance of neural networks on tabular data without introducing additional complexity.", "questions": " - How does the proposed method compare with other existing methods for improving the performance of neural networks on tabular data, such as feature engineering or model ensembling?\n - Can the proposed method be extended to handle highly irregular target functions or noisy data sets?\n - How does the proposed method affect the interpretability of the learned models, and how can this be addressed?", "limitations": "- Investigate the impact of the proposed method on the interpretability of the learned models and explore ways to improve interpretability without sacrificing performance.\n- Investigate the potential limitations of the proposed method for highly irregular target functions or noisy datasets and develop alternative methods to overcome these challenges.\n- Investigate the potential of combining the proposed method with other existing methods for improving the performance of neural networks on tabular data, such as feature engineering or model ensembling.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a clear and concise explanation of the performance gap between deep learning and tree-based methods on tabular data.\n- The proposed inductive bias is simple and easy to implement, without requiring additional hyperparameter tuning or complex model architectures.\n- The paper provides empirical evidence of the effectiveness of the proposed method on various tabular datasets and neural network architectures.", "weaknesses": "- The paper does not compare the proposed method with other existing methods for improving the performance of neural networks on tabular data.\n- The paper does not provide a detailed analysis of the impact of scaling and ranking transformations on the interpretability of the learned models.\n- The paper does not explore the potential limitations of the proposed method on highly irregular target functions or noisy data sets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Please see our responses below. \n\n**Q1: “How does the proposed method compare with other existing methods for improving the performance of neural networks on tabular data, such as feature engineering or model ensembling?”**\n\nWe believe that the domain of tabular deep learning is in need of theoretical advancements, on top of the existing strong empirical studies (see [Related Work/Discussion/L94]). Consequently, in this work, rather than benchmarking our proposed method to compare against other existing approaches, we focus on showing that (1) NN spectral bias is an important factor contributing to the lack of performance on tabular data, and (2) learnings from this analysis can be applied to NNs straightforwardly (e.g., using selective rank (Eq. 11), or the proposed learnable convex combination (Eq. 13)). \n\nAdditionally, we would like to emphasize that our proposed method is developed to introduce minimal complexity (i.e., less than a single feedforward layer) and no additional hyper-parameters, therefore, it is expected to work synergistically with other existing approaches (see Experiments/Frequency Reduction with other Network Architectures). As a follow-up study, we plan to extend our work to (1) investigate the behavior of other approaches designed towards improving neural networks on tabular data through the lens of spectral analysis, (2) evaluate the effectiveness of the proposed inductive bias on these approaches and (3) provide more extensive benchmarking.\n\n**Q2: “Can the proposed method be extended to handle highly irregular target functions or noisy data sets?”**\n\nOur method is designed to improve the NN’s ability to learn highly irregular target functions, where the irregularity is expected to convey relevant information to the corresponding task (i.e., irregularity is not due to noise). For this case, we provide additional experimental results in the PDF submitted for rebuttal, where we evaluate the performance of our method and baselines on 16 synthetic datasets with increasing target function frequency (i.e., irregularity). From the figure, we observe that our method consistently performs the best among the methods considered. Nevertheless, its performance does degrade as the frequency of the target function increases, since even with reduced relative target function frequency, the NN still is subject to spectral bias. \n\nIf the highly irregular behavior of the target function is caused by noise, we expect the spectral bias of neural networks to act as a regularizer and help with generalization [1]. Although we have not explicitly studied the behavior of our method on noisy datasets, since it does not distinguish between high-frequency noise and high-frequency information, we do not expect the learned high-frequency patterns to improve generalization further. To extend our methods to noisy datasets, additional mechanisms to differentiate informative high-frequency components from noise will likely be required.\n\n[1] Fridovich-Keil, S., Gontijo Lopes, R. and Roelofs, R., 2022. Spectral bias in practice: The role of function frequency in generalization. Advances in Neural Information Processing Systems, 35, pp.7368-7382.\n\n**Q3: “How does the proposed method affect the interpretability of the learned models, and how can this be addressed?”**\n\nIn this work, we focus on identifying the fundamental reasons behind the lack of performance of neural networks on tabular data. Although we believe investigating the interpretability of neural networks is important (e.g., to argue for replacing tree-based ensembles with neural networks, as they are widely accepted as more performant and interpretable alternatives), it is not in the scope of this study, however, it can be a promising future direction given the observations we make in Appendix D1 (L731). Specifically, since the proposed method independently learns a set of weights for each feature, it may be possible to interpret these weights to understand feature importances (e.g., scaling weights that converge to zero for some features may point out irrelevant/redundant features) and types of information they contribute to the decision (e.g., features with high ranking weights may indicate the feature can be compressed into quantile/percentiles instead of raw values).\n"}, {"review_id": "G6AbzMqO0U", "reviewer": "Reviewer_24qS", "summary": "# Summary\n\nThe paper introduces a hypothesis that tabular datasets are best described by functions with high frequency. They connect this finding to existing empirical knowledge in the tabular space, and introduce formal tools to measure spectral properties of target functions in tabular data. The authors propose a simple novel neural network layer to transform tabular datasets such that they are better fit by neural networks, reducing the frequency of the function required to fit these datasets. The authors present results on a suite of datasets.\n\nOverall, the paper is well-written and provides what seems to be a \"missing link\" in understanding performance gaps between neural and non-neura methods in the tabular space. The main contribution of the work seems to be the introduction of formal tools for analyzing spectral bias in tabular classifiers and their application to real-world datasets. The empirical results are more limited, but definitely encouraging. I suggest some minor revisions to the paper and a possible addition of a more controlled exeriment in the form of simulated-data experiments but believe that there is a solid case for acceptance.\n\n# Major Comments\n\n* A major limitation of the empirical results in the experiments section is that the authors only show results on an assortment of tabular datasets that are themselves heterogeneous and have a number of impossible to detect differences. As such, it is hard to really assess whether the proposed method is doing what it claims to do, even though it does lead to some minor but nontrivial gains in performance. I would suggest that the authors conduct a more controlled study, possibly on synthetic data, to demonstrate how the NNs with the proposed modifications perform on a dataset constructed to have a specific form of spectral bias. I am not sure of the exact design of such an experiment, but believe that, if desgned correctly, it would make a stronger and more direct case for the authors' claimed mechanisms than the current results (better accuracy and faster convergence) do.\n\n* The \"discussion\" section at the end of the related work is a nice synthesis of the state of the field of tabular machine learning. It also makes some helpful and pointed claims about adapting methods to data modalities, and how tabular researchers often \"trade\" complexity and hyperparameter tuning for performance gains, which provide useful guideposts for readers.\n\n* Table 1 is helpful, but a bit more information about the NN model used would be helpful. Additionally, since a major axis of comparison here would be neural vs. non-neural (i.e. GBM) methods, it would be useful to add a comparison with XGBoost to this table (I expect that the transformations would have no effect on such a model, again making a clearer case). This could also go in the supplement.\n\n* Despite a persuasive thesis and some data analysis that seems to support it, the empirical results are more mixed than expected (perhaps part of this is poor data visualization; the \"whiskers\" in the plot are extremely wide and it would be more helpful to display all of the data, ideally separated by dataset). I wonder whether the authors could comment on why the empirical impacts seem to vary so much. Could one factor be that the proposed method ignores feature interactions?\n\n# Minor Comments\n\n* The empirical experiments are not sufficiently described in the main text; at the very least, it would be useful for the authors to clearly list which datasets are used in the experiments.\n\n* The main takeaways from Figure 3 are not entirely clear in the main text. IIUC the takeaway is that the CI bars are narrower in the center and right columns? For each \"conclusion\" the authors make about Figure 3, it would be helpful to explain how/where this is demonstrated in the figure. Also, why is there a feature with such large variance and instability in the rightmost column?\n\n# Typos etc.\n\nL325: \"the first two rows\" --> the first two columns", "questions": "See above.", "limitations": "See above.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "See above.", "weaknesses": "See above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Please see our responses below.\n\n**Comment 1: “A major limitation of the empirical results in the experiments section is that the authors only show results on an assortment of tabular datasets that are themselves heterogeneous and have a number of impossible to detect differences. As such, it is hard to really assess whether the proposed method is doing what it claims to do, even though it does lead to some minor but nontrivial gains in performance...”**\n\nWe agree with the Reviewer that relying heavily on empirical results generated from a heterogeneous collection of datasets makes it more difficult to isolate the effects of frequency reduction as inductive bias. To address this limitation, we have extended the results of our synthetic data experiment, showing results for the various frequency-altering transformations while directly varying the underlying frequency spectrum of the target function in the attached pdf (more details in the comments for all reviewers). The results directly substantiate several claims made elsewhere in the paper, namely (1) low frequency target functions are easier for NNs to learn, (2) NN performance degrades with increasing frequency, a consequence of spectral bias, (3) our proposed method is the most robust against increased target function frequency of those considered, and (4) the difference in performance metrics (e.g., AUC) can be significantly larger numerically in datasets designed to exhibit a particular form of high-frequency behavior. In these synthetic dataset results, only the frequency scale factor and the random seeds for initializing the neural network were changed, showing that indeed our proposed inductive bias mitigates the adverse effects of NN spectral bias in these experiments.\n\n**Comment 2: “Table 1 is helpful, but a bit more information about the NN model used would be helpful. Additionally, since a major axis of comparison here would be neural vs. non-neural (i.e. GBM) methods, it would be useful to add a comparison with XGBoost to this table (I expect that the transformations would have no effect on such a model, again making a clearer case). This could also go in the supplement.”**\n\nWe thank the reviewer for this suggestion. If accepted, we will add the corresponding details and results for the final version of the paper.\n\n**Comment 3: “Despite a persuasive thesis and some data analysis that seems to support it, the empirical results are more mixed than expected ...”**\n\nThere are a few considerations relevant to the Reviewer’s comment and question. For one, although the numerical size of the corresponding performance changes between methods is small in some cases, our aggregated empirical results show significant and consistent performance improvement using the proposed method. Because we built the proposed method to realize the inductive bias out of existing frequency-altering transformations, the expectation is not necessarily to see unprecedented performance, but rather consistent top performance across datasets (see “Focus of our work and its impact on the experiment design” in the general comments for more in-depth discussion). Furthermore, most datasets for which the proposed method does not achieve the highest central value AUC either already have very high baseline performance metrics for all methods (kdd-small, pol), implying that spectral bias is not as significant an issue for these datasets, or poor performance for all NN methods (Diabetes, eye_movements), that may indicate other pathologies present in these datasets that our method is not designed to address. Finally, as the reviewer suggests, more sophisticated methods incorporating frequency reduction as an inductive bias, such as those accounting for interactions between features, could further improve performance in some cases.\n\n**Comment 4: “The empirical experiments are not sufficiently described in the main text; at the very least, it would be useful for the authors to clearly list which datasets are used in the experiments.”**\n\nWe will add this discussion to the main text if accepted.\n\n**Minor Comments. “The main takeaways from Figure 3 are not entirely clear in the main text. IIUC the takeaway is that the CI bars are narrower in the center and right columns? For each \"conclusion\" the authors make about Figure 3, it would be helpful to explain how/where this is demonstrated in the figure. Also, why is there a feature with such large variance and instability in the rightmost column?\"**\n\nWe agree that this can be clarified further in the main text. If accepted, we will provide additional details on interpreting the figure. Different columns in the figure correspond to different weights (scale, rank, combine) for a select dataset ( electricity), and the intended message is the same for all three columns: across different random seeds (i.e., initializations), the weights converge to the same directions (the reviewer’s confidence interval interpretation is correct). This implies that the network learns similar representations across different random initializations, and does not overfit to the rest of the parameters. Connected to this, we observe that these representations can be reused across models: representations learned via MLPs can be used to improve TabNet (L343).\n"}, {"review_id": "RsG52oAbkf", "reviewer": "Reviewer_H86d", "summary": "the paper “An Inductive Bias for Tabular Deep Learning” presents an interesting exploration of inductive biases for deep learning applied to tabular data. The paper introduces a novel inductive bias, named frequency reduction, which is specifically designed for tabular data. The authors propose a novel approach that leverages domain knowledge to improve the performance of models on tabular datasets.", "questions": "1. Lack of theoretical analysis: The paper lacks a deeper theoretical analysis of the proposed inductive bias. While the empirical results are convincing, a more thorough theoretical explanation of why and how the approach works would enhance the paper’s contribution. Including a theoretical analysis could provide more insights into the underlying mechanisms and generalization capabilities of the proposed approach.\n2. Dataset limitations: The paper primarily focuses on benchmark tabular datasets. However, real-world tabular datasets often exhibit diverse properties and complexities that may not be fully captured by the selected benchmarks. Including additional, more diverse datasets in the evaluation would provide a more comprehensive understanding of the generalizability of the proposed approach.\n3. Writing quality: The paper is not well-structured. The section of experiments is too short while most experiment results are included in the appendix.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strengths:\n1. Novel approach: The paper introduces a novel inductive bias, named frequency reduction, which is specifically designed for tabular data. By incorporating domain knowledge into the learning process, this approach offers a unique perspective on improving the performance of models for tabular datasets. This contribution adds to the existing literature by presenting a fresh perspective on the application of inductive biases. The idea is inspiring to the community.\n2. Rigorous evaluation: The authors provide a comprehensive evaluation of their proposed approach using various benchmark datasets. They compare the performance of their method against existing state-of-the-art techniques for tabular data. The evaluation metrics used are well-established and allow for a fair comparison. The empirical results provided demonstrate the effectiveness and superiority of the proposed approach.\n3. Clear motivation: The paper effectively communicates the motivation behind the proposed approach and the reasons for its effectiveness on tabular datasets. The use of visual aids and examples further enhances the clarity of presentation.", "weaknesses": "Weaknesses:\n1. Lack of theoretical analysis: The paper lacks a deeper theoretical analysis of the proposed inductive bias. While the empirical results are convincing, a more thorough theoretical explanation of why and how the approach works would enhance the paper’s contribution. Including a theoretical analysis could provide more insights into the underlying mechanisms and generalization capabilities of the proposed approach.\n2. Dataset limitations: The paper primarily focuses on benchmark tabular datasets. However, real-world tabular datasets often exhibit diverse properties and complexities that may not be fully captured by the selected benchmarks. Including additional, more diverse datasets in the evaluation would provide a more comprehensive understanding of the generalizability of the proposed approach.\n3. Writing quality: The paper is not well-structured. The section of experiments is too short while most experiment results are included in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1: “Lack of theoretical analysis: The paper lacks a deeper theoretical analysis of the proposed inductive bias. While the empirical results are convincing, a more thorough theoretical explanation of why and how the approach works would enhance the paper’s contribution. Including a theoretical analysis could provide more insights into the underlying mechanisms and generalization capabilities of the proposed approach.”**\n\nWe agree with the reviewer that our study could have benefited from a more thorough theoretical exploration of our inductive bias. In response, we briefly outline a theoretical argument that illustrates the underlying mechanism for improving NN performance on tabular data via frequency reduction.\n\nTheorem 1 of Rahaman et al [1] provides the analytic form of the Fourier amplitudes of a general ReLU network, $f(\\mathbf{x})$. The authors also show that along each direction of $\\mathbf{k}$ space, these amplitudes are upper bounded as \n\n$|\\tilde{f}_\\theta(\\bf{k})| \\leq N_f L_f(\\theta) k^{-\\Delta-1}$. \n\nHere, $L_f$ is the Lipschitz constant of the NN for a given set of parameters $\\theta$, $N_f$ is the number of linear regions, and $1\\leq \\Delta \\leq d$ depends on the orientation of $\\mathbf{k}$ with respect to the polytope faces represented by the NN. In any realistic setting, there is a maximum $N_f L_f$ that can be achieved through training, and therefore the amplitude of the NN Fourier coefficient for a fixed $\\mathbf{k}$ is bounded from above. For a given $\\mathbf{k}$ direction, we can therefore define a high-frequency region, $\\Omega$, in which the target function Fourier amplitudes, $\\widetilde{y}(\\mathbf{k})$, cannot be fit by the neural network. Reducing the $L^2$ norm of the target function Fourier amplitude over $\\Omega$,  $\\int_{\\Omega} |\\widetilde{y}(\\mathbf{k})|^2 d\\mathbf{k}$, relative to the corresponding integral over $\\mathbb{R}^n$ will tend to reduce the corresponding error arising from this spectral bias when evaluated on a particular set of data points (assuming the target function Fourier amplitudes are square-integrable). \n\nIt is straightforward to show that applying our $\\mathrm{scale}$ transformation with scale factor $a>1$ directly leads to this reduction of spectral energy over $\\Omega$, since the Fourier amplitudes for a function $g(\\mathbf{x})$ are related to those for the corresponding function, $g_{\\rm scaled}(\\mathbf{x})$ acting on scaled inputs as $\\widetilde{g}(\\mathbf{k}) = 1/a \\times  \\widetilde{g}_{\\rm scaled}(\\mathbf{k}/a)$. This relationship shows directly that $\\mathrm{scale}$ with $a>1$ maps a given Fourier component of the original function to a component at reduced frequency (and reduced overall magnitude) after applying the scaling transformation. The corresponding argument for $\\mathrm{rank}$ further depends on the underlying distribution of the feature. Consider the simple example of a uniformly-distributed feature $x$ over an interval $[x_1,x_2]$. Then $\\mathrm{rank}(x)$ acts in precisely the same way as $\\mathrm{scale}$ with $a=1/(x_2-x_1)$, and the same effect is seen. Similar arguments can be made with other assumptions about the underlying distribution. This analysis illustrates how the transformations we consider can mitigate the impact of NN spectral bias, however they neglect the corresponding impact on the optimization process itself. In practice, these effects are important and motivated the learnable convex combination of $\\mathrm{rank}$ and $\\mathrm{scale}$ we propose, as it is implicitly regularlized by the loss function itself. We plan to analyze the corresponding training dynamics analytically in future work, however its observed behavior is empirically consistent with our claims both on synthetic (see also results above) and real-world data experiments.\n\n**Comment 2: “Dataset limitations: The paper primarily focuses on benchmark tabular datasets. However, real-world tabular datasets often exhibit diverse properties and complexities that may not be fully captured by the selected benchmarks. Including additional, more diverse datasets in the evaluation would provide a more comprehensive understanding of the generalizability of the proposed approach.”**\n\nWith regards to dataset limitations, please see our general comments to all reviewers, clarifying the nature of the various real-world datasets we considered and highlighting their diversity.\n\n**Comment 3: “Writing quality: The paper is not well-structured. The section of experiments is too short while most experiment results are included in the appendix.”**\n\nWe thank the reviewer for this suggestion. If accepted, we will work on rebalancing the content to include more experiments on the main body for camera ready.\n\n\n[1]Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301–5310. PMLR, 2019."}, {"review_id": "co2u2ksYRz", "reviewer": "Reviewer_CXbH", "summary": "The authors address the problem of fitting deep nets to tabular datasets. This is a challenging task due to the heterogeneity of tabular datasets. Following a recent work, the authors first demonstrate that tabular data require learning prediction functions with nonnegligible high-frequency components. Since deep nets have an inductive bias towards learning a low frequency, they may struggle in modeling prediction functions over tabular data. To solve this limitation, the authors evaluate several transformations applied to the input features designed to reduce the relevance of high-frequency components of the prediction function. Finally, the authors present an optimization to minimize the risk of the model while transforming the input features using a convex combination of predefined transformations. The merits of the new approach are demonstrated using several real-world datasets. ", "questions": "How is the convex combination of rank and scale significantly outperforming both of them?\nNotations are quite confusing, for instance n is the dimension and N is the number of samples, why not use a different letter for the dimension?\nAlso the notation in page 6 is confusing, both upper case and lower cases are used to indicate the index of a vector.\n", "limitations": "The limitations of the method are not discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Progressing the capabilities of deep nets on tabular data is of high importance for the ML community. The paper provides value both in terms of the empirical evidence on the inductive bias and the practical solution presented by the authors. The paper is well-written and easy to follow, and the motivation and solution are intuitive and clear. ", "weaknesses": "My biggest concern is the empirical evaluations conducted in the paper. It is focused on relatively low dimensional datasets, and with a large sample size, this regime is typically less challenging for NN than the high dimensional low sample size (see [1]). More importantly, the evaluation is performed using normalized accuracy (and AUC) in the main text; the reason for using these metrics is not explained in the main text. In the supplemental, the authors use the more standard un normalized metrics, which demonstrate that the differences between most normalization schemes is subtle. It is unclear why the authors use normalized in the main text and unnormalized in the supplemental material; this should be clarified. The comparison to other tree-based models should be moved to the main text, and the authors should use one metric. Do the tree-based methods significantly outperform the proposed approach using the normalized metric? \nThe comparison presented in the paper is limited to some normalizations and does not include other tree-based models or architectures recently proposed for tabular data, for example:\n[1] Yang et al. \"Locally sparse neural networks for tabular biomedical data.\" In International Conference on Machine Learning, pp. 25123-25153. PMLR, 2022.\n[2] Ke, Guolin, et al. \"TabNN: A universal neural network solution for tabular data.\" (2018).\nFurthermore, additional transformations proposed for tabular data should be included.\n[3] Alexander, Yotam, et al. \"What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglement.\" arXiv preprint arXiv:2303.11249 (2023).\n[4] Zhu, Yitan, et al. \"Converting tabular data into images for deep learning with convolutional neural networks.\" Scientific reports 11.1 (2021): 11325.\nI would also include trees in the main text.\n\nI have a concern regarding the preprocessing normalization performed in the paper; how is it performed with respect to train/test splits? The authors do not detail this. Unproper normalization could lead to bias in classification results, as demonstrated in:\nMoscovitch and Rosset, \"On the Cross-Validation Bias due to Unsupervised Preprocessing\", Journal of the Royal Statistical Society Series B: Statistical Methodology\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Please see our responses below.\n\n**Comment 1: “My biggest concern is the empirical evaluations conducted in the paper. It is focused on relatively low dimensional datasets, and with a large sample size, this regime is typically less challenging for NN than the high dimensional low sample size (see [1]). ”**\n\nThe motivation behind our dataset choices is discussed in the global response (Choice of Datasets). We agree with the Reviewer that high-dimensional low-sample-size datasets would be good additions for extensive benchmarking against tree-based approaches. They are explicitly excluded from the benchmarking study of Grinsztajn et al, from which our datasets were selected. On the other hand, we believe that our experiments with the current datasets are sufficiently representative to validate our claims (i.e., frequency reduction as inductive bias) as it can be observed that the proposed approach significantly improves the performance of other NNs.\n\n**Comment 2: “More importantly, the evaluation is performed using normalized accuracy (and AUC) in the main text; the reason for using these metrics is not explained in the main text. In the supplemental, the authors use the more standard un normalized metrics, which demonstrate that the differences between most normalization schemes is subtle. It is unclear why the authors use normalized in the main text and unnormalized in the supplemental material; this should be clarified. ”**\n\n\nWe agree with the reviewer that a discussion in the main text to motivate the use of normalized metrics would help the reader, and we will be adding this to camera ready version if accepted. \n\nWith our experiments, we observed that the performance of baseline approaches highly fluctuated across different datasets. This implies that finding a suitable approach to used with a given dataset requires exhaustive search, and existing approaches do not generalize well across different tabular datasets. On the other hand, our proposed method consistently performed among the best. Based on these observations, we decided use the normalized metrics in order to aggregate measurements across datasets (since the ranges of measurements are different) and convey information on the overall behavior (i.e., performance , convergence speed, frequencies) of different approaches. Notably, this approach is also used in other studies [1, 2]. We provide the raw measurements in Appendix as they do not convey information on overall behavior and need to be broken down by dataset (hence take larger space).\n\nPlease also see our global response (Additional Synthetic Data Experiments).\n\n[1] Wistuba, M., Schilling, N. and Schmidt-Thieme, L., 2015, October. Learning hyperparameter optimization initializations. In 2015 IEEE international conference on data science and advanced analytics (DSAA) (pp. 1-10). IEEE.\n[2] Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M. and Hutter, F., 2022. Auto-sklearn 2.0: Hands-free automl via meta-learning. The Journal of Machine Learning Research, 23(1), pp.11936-11996.\n\n**Comment 3: “The comparison to other tree-based models should be moved to the main text, and the authors should use one metric. Do the tree-based methods significantly outperform the proposed approach using the normalized metric? ”**\n\nYes - tree-based models are still the highest-performing overall. However, our proposed method closes the gap significantly.\n\n**Comment 4: “The comparison presented in the paper is limited to some normalizations and does not include other tree-based models or architectures recently proposed for tabular data, for example...”**\n\n\nWe thank the reviewer for pointing out these interesting studies. Please see our global response (Focus of our work and its impact on the experiment design) for a detailed discussion on this matter.\n\n**Comment 5: “I have a concern regarding the preprocessing normalization performed in the paper; how is it performed with respect to train/test splits?...”**\n\nFor each cross-validation fold (i.e., train/validation/test splits under different random seeds), normalizers are re-fit (i.e., to collect statistics) from training set only. Then, they are used to transform validation and test splits.\n\n**Comment 6: “How is the convex combination of rank and scale significantly outperforming both of them?”**\n\nWe observe that although rank and scale reduce frequency, they may impact NN performance negatively as well (Section 3.3/line 224). For example, ranking may lose relevant information when the relative distances among pairs of points are important, and scaling may impact the conditioning of the loss landscape (i.e., hindering the optimization). The convex combination can be seen as a training loss-driven dial that lets the NN to learn the amount of ranking and scaling that should be conducted to find a low-frequency mapping that does not suffer as much from the negative impacts of the individual transformations.\n\n**Comment 7: “Notations are quite confusing, for instance n is the dimension and N is the number of samples, why not use a different letter for the dimension? Also the notation in page 6 is confusing, both upper case and lower cases are used to indicate the index of a vector.”**\n\n\nWe thank the reviewer for this feedback. If accepted, we will work on simplifying the notation.\n\n**Comment 8: “Limitations: The limitations of the method are not discussed in the paper.”**\n\nPlease see Appendix F for an extensive discussion of the limitations.\n"}]

Author rebuttal:
Dear reviewers, we thank you for all of your constructive feedback. In this general response, we would like to address some questions/concerns that arose from multiple reviewers. We address specific questions further on their corresponding threads, 

**Focus of our work and its impact on the experiment design**

First, we feel it may be worth re-emphasizing the focus of our work and how it informs the metrics we choose to evaluate. The domain of tabular deep learning has been progressing rapidly, where we see extremely creative and effective ways of improving NN performance. On the other hand, there is little work that focuses on identifying the fundamental reasons behind the typical gap between tree-based and NN-based techniques on tabular data. We see our work as one of the early steps towards building a lens that can help analyzing this performance gap. We believe it opens up a new direction for researchers to study and understand the behavior of NNs on tabular data, and eventually entirely address the NN performance gap (i.e., compared to tree-based ensembles). We believe our central contribution is showing, through both theoretical and empirical analysis, that (1) NN spectral bias is an important factor contributing to the performance gap between trees and NNs, and (2) learnings from this analysis can be applied to NNs straightforwardly (using e.g., selective rank, or the proposed learnable convex combination (refer to sections)). Since we ultimately design a technique that utilizes existing transformations such as rank and scale, the expectation from these learnings is not necessarily to see unprecedented performance gains; rather, it is to see that approaches that make use of existing practices for dealing with tabular datasets (e.g., scaling, ranking) but with spectral bias in mind consistently perform better and converge faster across different datasets. In effect, the proposed approach we motivate through our analysis can be seen as alleviating the need for exhaustively searching for the correct transformation to apply to each feature in each different dataset to reduce the impact of spectral bias. For this reason, although in some instances the magnitude of performance gained on individual metrics with our proposed method appears numerically small, the overall performance improvement and convergence speed up is significant and consistent across datasets, which can be clearly seen through the normalized metrics we consider in the main text. This strongly suggests that extensions of our methods beyond the simple frequency-altering transformations we consider could provide even numerically larger gains. Note that our approach can also be used in any other NN-based method, and the benefits may vary depending on how much various architectures are impacted by spectral bias (c.f., our TabNet experiment).

**Choice of Datasets**

Second, we would like to clarify our choice of datasets for our experiments. We chose all datasets from Grinsztajn et al (NeurIPS’22), which considered several criteria for selection in order to benchmark tree-based vs NN-based model performance (Grinsztajn et al / Appendix 3). . Notably, we limited our study to numerical feature-heavy classification datasets (14) among the regression and classification datasets (45) provided in Grinsztajn et al. They are all real-world datasets that span fundamentally different problems and data characteristics, from classification for particle physics experiments (e.g., MiniBooNE) to predicting credit card defaults in Taiwan (e.g., credit card clients), containing combinations of both raw data and engineered features. Furthermore, in contrast to Grinsztajn et al, we do not remove complexities such as missing features, low/high-cardinality categorical features, and class imbalance, or discard any samples from the datasets, in order to retain the diverse properties present in these datasets

**Additional Synthetic Data Experiments**

Due to the diverse nature of the datasets considered in our experiments, there were some concerns voiced about being able to reliably draw conclusions about the effect of our proposed inductive bias from our experiments. Also, the different characteristics naturally lead to variation in the performance gain observed from our methods. To address these concerns, in this reply we also provide additional results from the synthetic data experiment in our original submission that support several of our observations made on the real-world datasets. In particular, in the attached pdf we show results obtained by varying the overall frequency of the target function in Eq. 24, keeping all other parameters fixed. For each choice of scale factor, we trained the same two-hidden-layer MLP on the raw (unit-scaled) data, and with rank, scale (standardization), and our proposed convex combination layer for 200 epochs (25 epochs were used to tune the learning rate). The results in the attached figure clearly show (1) low frequency target functions are easier for NNs to learn, (2) NN performance degrades with increasing frequency, a consequence of spectral bias, (3) our proposed method is the most robust against increased target function frequency, and (4) the difference in performance metrics (e.g., AUC) can be significantly larger numerically in datasets designed to exhibit a particular form of high-frequency behavior. Taken together, these results further substantiate our claims that frequency reduction as an inductive bias can significantly improve performance on datasets with high-frequency target functions.


Meta-review: This paper introduces a novel inductive bias called "frequency reduction", reducing the frequency of irregular target functions, for improving deep learning on tabular data, to bridge the performance gap between deep learning and tree-based methods.

Four reviewers assessed this paper, and all four unanimously recommended acceptance. In particular, they all agreed on the excellence of this paper in providing a clear explanation of the performance gap between deep learning and tree-based models in tabular data and proposing an easily implementable inductive bias to bridge this gap. However, there were concerns from reviewers regarding the validation in the experiment section such as verification of simulation data and validation on higher-dimensional real datasets with diverse features and so on. I believe that including results from a variety of datasets in the final camera-ready version, including additional experiments in the rebuttal, would make this paper even stronger.
Decision: Accept (poster)