SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ege Beyazit*
beyazit@amazon.com
Jonathan Kozaczuk*
jonkozac@amazon.com
Bo Li
booli@amazon.com
Vanessa Wallace
vwall@amazon.com
Bilal Fadlallah
bhf@amazon.com
Deep learning methods have achieved state-of-the-art performance in most modeling tasks involving images, text and audio, however, they typically underperform tree-based methods on tabular data. In this paper, we hypothesize that a significant contributor to this performance gap is the interaction between irregular target functions resulting from the heterogeneous nature of tabular feature spaces, and the well-known tendency of neural networks to learn smooth functions. Utilizing tools from spectral analysis, we show that functions described by tabular datasets often have high irregularity, and that they can be smoothed by transformations such as scaling and ranking in order to improve performance. However, because these transformations tend to lose information or negatively impact the loss landscape during optimization, they need to be rigorously fine-tuned for each feature to achieve performance gains. To address these problems, we propose introducing frequency reduction as an inductive bias. We realize this bias as a neural network layer that promotes learning low-frequency representations of the input features, allowing the network to operate in a space where the target function is more regular. Our proposed method introduces less computational complexity than a fully connected layer, while significantly improving neural network performance, and speeding up its convergence on 14 tabular datasets.
Despite being one of the simplest and most natural ways to describe data, tabular data with heterogeneous columns poses a significant challenge for deep learning models. Recently, [5, 13] benchmarked various deep learning approaches against tree-based models and showed that tree-based models tend to perform the best on tabular data. Nevertheless, novel neural network architectures provide several advantages over classical machine learning methods. Specifically, neural networks operate on differentiable loss functions, are able to learn meaningful and high-dimensional data representations, and can scale to large datasets. In addition, the rapid advances of neural networks in generative image and language modeling clearly indicate their capability to encode complex information [8, 18]. In order to fully utilize these benefits on tasks that include tabular datasets, identifying and addressing the reasons behind neural networks' lack of performance on this form of data is essential.
Several influential research efforts on synthetic and image data point out that, despite being universal approximators, neural networks have a spectral bias, i.e., they tend to learn the low frequency components of a function much more easily than higher frequency components [22, 27]. In addition, recent empirical studies on tabular deep learning provide insights on why neural networks typically
Figure 1: (Left) Due to their heterogeneous nature, tabular datasets tend to describe higher frequency target functions compared to images. The spectra corresponding to image datasets (curves in color) tend to feature lower Fourier amplitudes at higher frequencies than heterogeneous tabular datasets (cyan region). (Top Right) Data with labels varying over small distances are difficult for neural networks to learn, but easy for tree-based methods. (Bottom Right) Ranking is a transformation that redistributes the data and increases the scales over which the feature labels vary. This transformation effectively lowers the frequency of the target function, making it easier for a neural network to learn.
underperform on tabular data, including the latter's tendency to feature irregular target functions [13, 5]. Notably, the feature spaces of the tabular datasets considered in these studies are heterogeneous. Unlike images in tabular form where columns correspond to the same signal on different sensors, columns of tabular deep learning tasks typically represent features of different nature. These features are engineered to be informative as individuals (or subsets) and tend to have different statistical properties than each other.
In this work, we connect these lines of inquiry and claim that the spectral bias of neural networks may prevent them from learning the information present in the high frequency components of heterogeneous tabular datasets. We provide evidence to support this claim by analyzing the Fourier components of image and tabular datasets, as well as neural networks. To best of our knowledge, we present the first analysis on the relationship between spectral bias and tabular deep learning. We suggest that transforming tabular features into low-frequency representations may alleviate the negative impact of spectral bias. Notably, such transformations need to be performed in a way that balances the benefits of reduced frequency against potential information loss and additional negative effects on optimization (e.g., creating large gradients along certain directions) that can arise when altering the representation of the data. Figure 1 illustrates an example of the analyses we conduct. From the left panel, it can be observed that many tabular datasets tend to have higher power in their high frequency components, compared to some of the popular image datasets. We extend this comparison and present additional observations in Appendix C. The right panel depicts the impact of a frequency-reducing transformation on the functions learned by a multilayer perceptron (MLP) and Extreme Gradient Boosting (XGB) [7] on synthetic data. From this panel, it can be observed that the MLP is able to fit the target function better after the frequency-reducing transformation. Details of this analysis are provided in Appendix B.
Driven by these observations, we propose using frequency reduction as an inductive bias during neural network training. We apply this bias by introducing a trainable layer that promotes learning lower frequency feature mappings. We show that several commonly-used data processing methods impact the frequency spectrum of the target function as well as the resulting neural network learned from the data, however, their impact on performance varies with the dataset. Our proposal leads to the best performance, converges faster across all datasets considered, and introduces minimal computational complexity.
Spectral Analysis of Neural Networks. Spectral properties of neural networks have been extensively studied. [22, 27] rely on Fourier analysis to investigate the learning behavior of neural networks,
observing that neural networks learn lower frequency components of target functions first. This phenomenon is referred to as spectral bias or the frequency principle. [3] study spectral bias on data with non-uniform density by examining the NTK for deep fully-connected networks, observing that both data density and frequency impact the gradient descent behavior. [6] show that during neural network training, error terms from different frequencies are controlled by the eigenvalues of the NTK. [16] decompose the reconstruction error of neural networks into low and high frequency components with respect to a cutoff frequency, and use these components to define a metric that quantifies spectral bias. [26] study the spectral bias of coordinate-based MLPs for modeling functions in low dimensions. Using NTKs, they show that mapping input coordinates to Fourier features [23] enables learning higher frequency information. Notably, they focus on the frequency described by pixels of a given image, rather than the frequency of the target function described by the whole dataset. The existing studies mainly focus on advancing the understanding of spectral bias on synthetic and image datasets, however, real-world tabular datasets are not considered.
Tabular Deep Learning. Due to neural networks' inferior performance on tabular data compared to tree-based models, various approaches have been proposed to close this performance gap. [1] take raw tabular data as input and use sequential attention to choose the set of features to use for every step of inference. This instance-wise feature selection is shown to improve performance and enable interpretability. [28, 2] extend self and semi-supervised learning to tabular data by introducing pretext tasks that aim to optimize a model to recover input samples from their corrupted variants. [25] leverage row- and column-wise attention combined with contrastive pre-training. [12] embed numerical features using linear transformations and categorical features using lookup tables, and feed these embeddings to a transformer. [15] show that using hyperparameter tuning, one can find a strong combination of 13 regularizers to be used with a simple MLP to outperform other tabular deep learning methods, as well as tree-based methods such as XGB. [11] propose individually embedding numerical features onto high-dimensional spaces before jointly using them in a model. The authors explore approaches such as piecewise linear embeddings that use bins selected by decision trees, and periodic activations to transform numerical features into high-dimensional representations.
Discussion. In most learning tasks that include images, audio or text, deep neural networks provide state-of-the-art performance. Breakthroughs in these domains are strongly connected to the design choices that exploit the salient characteristics of the corresponding data modality (e.g., convolutional layers in computer vision). On the other hand, existing studies on tabular deep learning mainly focus on applying approaches that are successful in other domains to improve performance [1, 12, 2, 25]. These studies do not rigorously investigate the fundamental reasons behind the performance gap between neural networks and tree-based methods. As a result, in order to gain performance, these approaches tend to sacrifice simplicity by complicating training and hyper-parameter tuning significantly. Such complexity causes these models to perform inconsistently across different studies, as utilizing the full potential of these models can be challenging. For example, [13] provide an extensive set of benchmarks, showing that tree-based approaches significantly outperform various tabular deep learning methods that originally claimed to outperform tree-based models. Similarly, [25] show that simple MLPs perform better than TabNet, an architecture that is tailored for tabular deep learning. As a result, instead of focusing on designing complex models to improve performance, we aim to identify and address the fundamental weaknesses of neural networks when learning from tabular data. We show that our approach improves performance of neural networks significantly without introducing additional complexity for fine-tuning, and helps them to converge faster.
Consider classification tasks defined on a tabular dataset  $D = \{(\mathbf{x}_i, y_i)\}_{i=1}^N$  where  $\mathbf{x}_i \in \mathbb{R}^n$  and  $y_i \in \{0,1\}^m$  is the corresponding label. The task is to learn a classifier  $f: \mathbb{R}^n \to \{0,1\}^m$ . For simplicity, set  $m = 1$  (i.e., binary classification), and let  $y = 1$  correspond to the minority class in all cases. In Section 4, we provide experimental results and analyses for multi-class datasets as well, by extending this approach via one-vs-rest.
We are interested in the frequency spectra of various functions, including target functions implicitly defined by the data, for which  $f(\mathbf{x}_i) = y_i$ . Since we only have the labels defined at the datapoints supplied in each dataset, we rely on spectral analysis methods that take an arbitrary set of points as
input and output a measure of the strength of fluctuations at particular scales. One such method is a generalized Fourier transform known as the non-uniform discrete Fourier transform (NUDFT) [10]. For a function  $f(\mathbf{x})$  and a set of points  $X = \{\mathbf{x}_i\}_{i=1}^N$ , the NUDFT defines a function of frequency vector  $\mathbf{k} \in \mathbb{R}^n$  as
$$
\widetilde {f} _ {X} (\mathbf {k}) = \frac {1}{N} \sum_ {i = 1} ^ {N} f \left(\mathbf {x} _ {i}\right) e ^ {- 2 \pi i \mathbf {k} \cdot \mathbf {x} _ {i}}. \tag {1}
$$
Equation (1) can be interpreted as an approximation of the multidimensional continuous Fourier transform (FT) of  $f(\mathbf{x})$  weighted by the empirical distribution represented by the datapoints  $X$ , further discussed in Appendix A. [27] demonstrate the spectral bias of neural networks in terms of the NUDFT.
The Fourier transform approximated in Equation 1 evaluated at a given  $\mathbf{k}$ , corresponds to the coefficient of the  $e^{2\pi i\mathbf{k} \cdot \mathbf{x}}$  term in a projection onto complex exponentials. Therefore, a natural definition of one function being higher frequency than another corresponds to the former having larger-magnitude Fourier amplitudes away from a specified low-frequency regime than the latter. We can make this comparison concrete by evaluating amplitudes along specific directions in  $k$ -space. Defining  $\mathbf{k} = k\hat{\mathbf{k}}$ , with  $\hat{\mathbf{k}}$  a unit vector in  $\mathbb{R}^n$ , we can compare the size of the squared NUDFT amplitudes,  $P_f(\mathbf{k}) = \left|\widetilde{f}_X(\mathbf{k})\right|^2$ , for functions along a given  $\mathbf{k}$  direction by computing the sum
$$
S _ {\hat {\mathbf {k}}} [ f ] = \frac {2}{\bar {f} ^ {2}} \sum_ {n = 0} ^ {(k _ {*} - k _ {0}) / \Delta k} P _ {f} ((k _ {0} + n \Delta k) \hat {\mathbf {k}}) \Delta k, \tag {2}
$$
where  $f$  is a function of interest. Here,  $\overline{f}$  is the mean of  $f(\mathbf{x})$  (i.e.,  $y$ ) evaluated on the points in  $X$ , so that the scales at low frequencies are comparable for any given dataset (see Appendix A.2). Equation (2) can then be used to compare spectra: if  $S_{\hat{\mathbf{k}}} [f_1] > S_{\hat{\mathbf{k}}} [f_2]$  for two functions  $f_1$  and  $f_2$  defined on  $\mathbb{R}^n$ , we say that  $f_2$  is a higher frequency function than  $f_1$  along the direction  $\hat{\mathbf{k}}$ , in the sense that it has more normalized signal energy (variance) in the region of interest, and refer to frequency reduction as reducing  $S_{\hat{\mathbf{k}}} (\mathbf{k})$ . In the above expression,  $k_0$  defines the boundary between the low- and high-frequency regions, and  $k_*$  is a cutoff frequency, discussed in Appendix A.4. Notably, Equation (2) only considers the magnitude of Fourier amplitudes along one dimension of  $\mathbb{R}^n$  at a time. In Appendices A.3 and A.6, we discuss an alternative approach that works in the full feature space  $\mathbb{R}^n$  and show that using (2) to evaluate along principal components (PCs) yields similar results to working in  $\mathbb{R}^n$ , while being more computationally efficient. Therefore, we rely on evaluations along PCs.
3.2 Spectral Analysis of a Class of Target Functions
The NUDFT, when evaluated using the labels  $f(\mathbf{x}_i) = y_i$ , can be understood in terms of the continuous Fourier transform of a class of target functions fitting the data. Given a binary classification task described by a dataset  $D = \{(\mathbf{x}_i, y_i)\}_{i=1}^N$ , a typical way to train a neural network  $f$  is minimizing the empirical risk  $1/N\sum_{i=1}^{N}\ell(y_i, f(\mathbf{x}_i))$ , where  $\ell$  denotes the cross-entropy loss. Assuming that  $D$  maps each unique  $\mathbf{x}_i$  to a single  $y_i$ , the cross-entropy loss without explicit regularization terms is exactly minimized for any function  $f(\mathbf{x})$  such that  $f(\mathbf{x}_i) = y_i\forall i \in \{1, \dots, N\}$ . This defines a set of equalities satisfied by any ideal target function perfectly fitting the training set<sup>2</sup>, but does not further specify the form of the function away from the training instances. Indeed, there is no unique definition of the target function given only a finite set of points. However, we can parameterize a class of target functions minimizing the loss by convolving the target values at known data points with a given envelope function,  $G(\mathbf{x}, \mathbf{x}')$ , that falls off sufficiently rapidly away from the points  $\{\mathbf{x}_i\}$ :
$$
\hat {f} [ \mathbf {x}, G ] \equiv \sum_ {i = 1} ^ {N} \int_ {\mathbb {R} ^ {n}} y _ {i} G (\mathbf {x}, \mathbf {x} ^ {\prime}; \alpha_ {i}) \delta^ {n} \left(\mathbf {x} ^ {\prime} - \mathbf {x} _ {i}\right) d ^ {n} \mathbf {x} ^ {\prime}, \tag {3}
$$
where  $\delta^n (\mathbf{x})$  is the Dirac delta function in  $\mathbb{R}^n$  and  $\alpha_{i}$  denotes the parameters defining the envelope for the point  $\mathbf{x}_i$ . A typical loss function (neglecting regularization terms) will be exactly minimized
In practice, some form of regularization will need to be utilized to avoid overfitting, but this set of equalities should still approximately hold. We verify empirically that the spectra of classifier functions learned in realistic settings approach the spectra of the target functions defined in this idealized sense.
provided  $G(\mathbf{x}_i, \mathbf{x}_i; \alpha_i) = 1$  and  $G(\mathbf{x}_i, \mathbf{x}_j; \alpha_i) = 0 \forall \mathbf{x}_i, \mathbf{x}_j \in D$ . Perhaps the simplest such envelope is a multi-dimensional generalization of the boxcar function
$$
B \left(\mathbf {x} ^ {\prime}, \mathbf {x}; \Delta_ {i}\right) = \prod_ {j = 1} ^ {n} \operatorname {r e c t} \left(\frac {x _ {j} ^ {\prime} - x _ {j}}{\Delta_ {i}}\right), \tag {4}
$$
where  $\mathrm{rect}(\frac{a' - a}{\Delta}) = 1$  if  $|a' - a| < \Delta / 2, = 0.5$  if  $|a' - a| = \Delta / 2$ , and vanishes otherwise. Notably,  $\Delta$  corresponds to an upper bound on the  $L_{\infty}$  distance between  $a$  and  $a'$ , in order for them to be assigned to the same value of  $y$ . The target function defined by inserting Equation (4) into Equation (3) is then
$$
f (\mathbf {x}) = \sum_ {i = 1} ^ {N} y _ {i} \prod_ {j = 1} ^ {n} \int_ {\mathbb {R} ^ {n}} \operatorname {r e c t} \left(\frac {x _ {j} - x _ {j} ^ {\prime}}{\Delta_ {i}}\right) \delta \left(x _ {j} ^ {\prime} - \left(\mathbf {x} _ {i}\right) _ {j}\right) d \mathbf {x} ^ {\prime}. \tag {5}
$$
Provided that  $\Delta_{i}$  satisfies
$$
\left| \mathbf {x} _ {i} - \mathbf {x} _ {j} \right| _ {\infty} \geq \Delta_ {i} \forall \mathbf {x} _ {j} \in D, \tag {6}
$$
for each  $\mathbf{x}_i$ , this function minimizes the cross-entropy loss on the training set, since its values at each point are simply the corresponding label. This corresponds to tiling the feature space with  $n$ -dimensional hypercubes of edge lengths  $\Delta_i \forall i$ , centered on each datapoint  $\mathbf{x}_i$  and taking the target function to be constant within that region with value  $y_i$ . This is the simplest possible choice for the envelope function in that it does not assume any spectral features present in the target functions on distances smaller than  $\Delta_i$ .
We can now use the continuous Fourier transform to analyze the frequency spectrum of the target function defined in Equation (5). Denote the Fourier transform of a function  $g(\mathbf{x})$  as  $\mathcal{F}_{\mathbf{x}}[g(\mathbf{x})](\mathbf{k})$ . By the convolution theorem, we have
$$
\mathcal {F} _ {\mathbf {x}} [ f (\mathbf {x}) ] (\mathbf {k}) = \sum_ {i = 1} ^ {N} y _ {i} e ^ {- 2 \pi i \mathbf {k} \cdot \mathbf {x} _ {i}} \Delta_ {i} ^ {n} \left(\prod_ {j = 1} ^ {n} \operatorname {s i n c} \left(k _ {j} \cdot \Delta_ {i}\right)\right), \tag {7}
$$
where  $\mathrm{sinc}(x) = \sin (\pi x) / (\pi x)$ . Different choices for the  $\Delta_{i}$  correspond to different assumptions about the target function away from the datapoints. One simple class of target functions is obtained by taking a common  $\Delta_{i} = \Delta = \min \left\{\left|\mathbf{x}_{i} - \mathbf{x}_{j}\right|_{\infty}\right\}_{i = 1,j = i + 1}^{N - 1,N}$ . This choice satisfies Eq. (6) and in the limit  $|\mathbf{k}|_{\infty}\ll 1 / (\pi \Delta)$ , yields
$$
\mathcal {F} _ {\mathbf {x}} [ f (\mathbf {x}) ] (\mathbf {k}) \approx \Delta^ {n} \sum_ {i = 1} ^ {N} y _ {i} e ^ {- 2 \pi i \mathbf {k} \cdot \mathbf {x} _ {i}} = N \Delta^ {n} \widetilde {f} _ {X} (\mathbf {k}) \tag {8}
$$
where the approximation holds up to terms  $\mathcal{O}(|\mathbf{k}|^2\Delta^2\pi^2)$  (here  $||$  denotes Euclidean distance). The RHS above is the NUDFT up to a constant dimensionful scaling factor, which will drop out when using a procedure like Equation (2) that normalizes the amplitudes to the value at  $k = 0$ . Therefore, the NUDFT reproduces the spectral properties of a simple class of target functions defined by Equation (3) for frequencies below the scale set by the inverse minimum nearest-neighbor separation of points in the dataset. We implicitly enforce the latter restriction by cutting off our analysis at  $k_*$ , as discussed in Appendix A.4. Note that other choices for the  $\Delta_i$  result in different target functions with spectra that can be analyzed by appropriately modifying the NUDFT.
In order to compare against the same quantity in all analyses, we also utilize Equation (1) for analyzing the frequency spectra of classifiers. A similar argument holds, and the corresponding results can be interpreted as analyzing the spectra of a classifier coarse-grained by convolving the neural network predictions at the datapoints with a narrow envelope as in Equation (5). In all cases, the classifier spectra defined in this way converge to that of the target function as training progresses, up to deviations caused by spectral bias (see also [27] for similar results). This suggests that high-frequency target functions, defined as above, are best fit by high-frequency classifiers, and that transformations affecting the target function spectrum will impact the frequency spectrum of the corresponding classifier.
house	electricity	phoneme	MagicTelescope	bankMarketing	MiniBooNE	albert	california
unit length	1.31	1.46	1.66	0.13	2.01	16.24	1.74	1.5
rank	0.11	0.18	0.44	0.07	0.74	0.34	0.17	0.3
scale	0.01	0.18	0.06	0.01	0.17	0.35	0.04	0.03
selrank	0.11	0.22	0.58	0.07	1.28	0.34	0.29	0.32
house	electricity	phoneme	MagicTelescope	bankMarketing	MiniBooNE	albert	california
unit length	88.94 ± 0.17	85.28 ± 0.12	87.78 ± 0.28	87.84 ± 0.18	90.56 ± 0.11	90.78 ± 0.04	66.35 ± 0.04	88.51 ± 0.15
rank	89.61 ± 0.14	86.1 ± 0.19	88.0 ± 0.46	87.9 ± 0.24	90.41 ± 0.08	95.01 ± 0.06	66.21 ± 0.08	88.51 ± 0.17
scale	89.65 ± 0.16	85.19 ± 0.13	88.64 ± 0.21	87.88 ± 0.23	90.56 ± 0.13	94.0 ± 0.04	66.48 ± 0.05	88.53 ± 0.22
selrank	89.91 ± 0.16	87.04 ± 0.17	88.15 ± 0.32	88.14 ± 0.27	90.56 ± 0.13	94.94 ± 0.05	66.28 ± 0.06	88.48 ± 0.2
Table 1: (Top) Normalized sum of high-frequency Fourier amplitudes,  $S_{\hat{\mathbf{k}}}$ , for various binary classification datasets and transformations, averaged over the 1st principal component direction after each transform is applied. Ranking and scaling individual features tends to significantly reduce the high-frequency energy of tabular datasets relative to features normalized to unit scale. For an extended discussion, see Appendix A.6. (Bottom) Accuracy measurements corresponding to different transformations. Overall, lower frequencies are correlated with better performance. However, the benefits of frequency reduction is transformation- and dataset-dependent.
3.3 Case Study: Impact of Transformations on Target Function Spectra
Most neural networks need their input to be transformed in order to converge faster and generalize better. [14] show that these transformations serve to regulate layer input/output gradient statistics and/or the conditioning of input/gradient covariance matrices. A typical example of such transformations is scaling the features so that they vary over the unit interval  $[0, 1]$ , i.e., min-max scaling. Different choices of transformations can dramatically impact the target function frequency spectrum, and therefore the spectrum of the network that best fits the data with a given loss function. Due to the spectral bias of neural networks, we expect that using transformations that reliably reduce the high frequency components of the target functions result in better performance (see Appendix D for further discussion and illustration of this point).