SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 7: (Top) Data with labels varying over small distances are difficult for neural networks to learn, but easy for tree-based methods. (Bottom) Transformations that redistribute the data can increase the scales over which the feature labels vary, effectively lowering the frequency of the target function and making it easier for a neural network to learn.
$k_{i} \in \{1, 5, 10, 20\} \times 2\pi$  and randomly sample the corresponding amplitudes  $A_{i}$  and phases  $\varphi_{i} / 2\pi$  from the continuous uniform distribution on  $[0, 1]$ . We also take  $g(x)$  as the rank transformation of Equation (9), for illustration. In order to compare NN's with features at a common scale, we first min-max normalize the features to have unit length. The corresponding target function has  $S_{\hat{\mathbf{k}}} \approx 0.02$ . A 2-hidden-layer MLP with width 256 is then trained to classify 5,000 points using binary cross-entropy loss for 10000 epochs after an initial 250 epoch tuning stage to select the best learning rate. The resulting predictions are shown in black on the top panel of Figure 7, and do not match the training data very well, particularly in regions where the tag varies more rapidly (i.e., the spectral bias phenomenon). The NN evaluated on a validation set of 1000 points distributed in the same way as the training set yields an AUROC of 0.935. On the other hand, we also train an XGBoost classifier using default hyperparameters for 100 boosting iterations (the performance converges quickly so boosting for additional iterations does not improve performance). The corresponding predictions are shown in orange. This tree-based classifier performs well and is able to reproduce the rapid variations of the target function, since, as long as the algorithm's histogram binning is fine enough, it is straightforward to recursively split this one-dimensional feature space into regions with  $y = 0$  and 1. The corresponding AUROC is 0.998.
There can be several approaches taken to overcome this performance gap. One approach is to focus primarily on the underlying model: perhaps tuning the hyperparameters extensively or incorporating a more sophisticated architecture could lead to a better-performing configuration. However, this process quickly becomes computationally intensive and is dataset-dependent. An alternative approach, and the one we study in this work, is to focus on the representation of the data. The frequency characteristics of a function defined on a dataset are inherently tied to the representation of the data itself. Stretching or warping the feature space while maintaining the label values will in general change the length scales over which the function varies.
To investigate this effect, consider the impact of re-arranging the datapoints such that rather than following their original distribution, they are now spaced as far apart from each other as possible within a fixed interval, which we take to be  $[0,1]$ . This is a familiar transformation: mapping the original distribution to a uniform distribution along the  $x$  feature direction, as in the rank transformation of Equation (9). However, the resulting effect on the example in Figure 7 is dramatic: this re-distribution of points reduces the frequency of the target function, as clearly seen in the bottom
panel (the rank-transformed target function has  $S_{\hat{\mathbf{k}}} \approx 0.01$ ). We train the same two models (MLP and XGBoost) with the same hyperparameters as before on the transformed data. The tree-based model performance does not change significantly, but the MLP's performance improved dramatically, with the AUROC evaluated on the transformed validation set increasing from 0.935 to 0.989.
For this synthetic example, the simple inductive bias of seeking a representation of the data that "smooths out" the target function in a way that does not negatively effect optimization significantly boosts neural network performance and reduces the performance gap with respect to the GBDT-based model. Our analyses utilizing realistic heterogeneous tabular datasets yield similar results.
B.1 Connection to computable spectral bias
Since we know the underlying density of the synthetic datapoints, we can directly compute the spectral bias of the MLPs trained on the pre- and post-transformed data following the computable definition of spectral bias proposed by [16]. In particular, a quantitative definition of the spectral bias,  $SB$ , can be provided in terms of the fraction of variance unexplained (FVU) compared between a predefined low- and high-frequency region. A value of  $SB > 0$  indicates a larger fraction of FVU in the high-frequency region, hence a spectrally-biased NN, since the NN fits the data worse at high frequencies. To compare our two NNs, we consider an epoch for which the performance of the two NNs is comparable: the epoch with largest validation AUC for the unit length NN and the epoch with the closest corresponding performance for the ranked NN. The FVUs are approximately equal for both networks, however the spectral bias of the ranked NN, computed as in [16], is lower than that of the NN trained on unit-length features:  $SB = 0.58$  for unit length,  $SB = 0.42$  for rank. This indicates that for NNs achieving similar performance, the rank transformation has produced a network with a lower fraction of its error in high frequencies, and therefore has reduced the negative impact of spectral bias on NN training, since the relatively larger low-frequency errors can be effectively reduced by further epochs of training, in contrast to high-frequency errors, which are more difficult for the NN to reduce. This allows for significantly better performance overall for the rank-transformed data. Similar effects are suggested by our analysis of realistic multidimensional tabular datasets: transformations resulting in lower-frequency NNs for a fixed performance metric can typically be further trained to reduce their correspondingly lower-frequency errors, hence mitigating the impact of spectral bias in learning the corresponding target function. Notably, this analysis is an attempt to establish the initial connection between our approach and [16]. A more extensive analysis of the computable spectral bias of all the methods we consider in this paper will be included in future work.
C Frequency of Target Functions in Tabular and Image Datasets
In this section, we provide details related to the results on the LHS of Figure 1, contrasting the target function spectra of image vs tabular datasets.
Our study focuses on mitigating the effects of spectral bias in the context of tabular deep learning. One of the empirical motivations for our approach is the observation that target function spectra of tabular datasets tend to be of higher-frequency than similar examples from popular image datasets. To compare frequencies across datasets, the features should be normalized in a consistent way. We take unit length normalization as the baseline for our comparisons. We consider several popular benchmark datasets from computer vision applications: cifar-10-small (https://openml.org/d/40926), fashion-mnist (https://openml.org/d/40996), mnist-784 (https://openml.org/d/554), svhn (https://openml.org/d/41081), and scene (https://openml.org/d/41471). After min-max normalizing, we treat these image datasets in the same way as our tabular multi-class datasets: for each of 10 random seeds, we consider a training set sampled from the full dataset, and compute the NUDFT along the first PC direction, assigning binary labels for all 1-vs-rest class assignments, and computing the mean over all seeds and binary label assignments. The resulting squared NUDFT amplitudes,  $P_{y}(\mathbf{k})$ , are plotted in color on the LHS of Figure 1. To compare against tabular dataset spectra, we use the same procedure. The envelope containing all such spectra computed for our 14 tabular benchmarks is shaded in blue on the LHS of Figure 1. In this figure, all amplitudes shown are normalized with an implicit  $1 / \overline{f}^2$ , to account for the different levels of class imbalance across the datasets.
From these results, we immediately see that, using our definitions, the image datasets considered have significantly smaller high-frequency Fourier amplitudes when all features are normalized to the same scale. While in our study we simply use this observation as an empirical motivation for addressing the high-frequency nature of tabular dataset target functions, it would be interesting to extend this analysis to more examples and to study the underlying causes of these differences in future work.
D Illustration: frequency reduction and the effects of spectral bias
In this appendix, we provide informal analytic arguments, based on the results of [22], illustrating how frequency reduction can improve NN performance given their inherent spectral bias. Theorem 1 of [22] provides the analytic form of the Fourier amplitudes of a general ReLU network,  $f(\mathbf{x})$ . The authors also show that along each direction of  $\mathbf{k}$  space, these amplitudes are upper bounded as  $|\widetilde{f}_{\theta}(\mathbf{k})| \leq N_{f}L_{f}(\theta)k^{-\Delta -1}$ . Here,  $L_{f}$  is the Lipschitz constant of the NN for a given set of parameters  $\theta$ ,  $N_{f}$  is the number of linear regions, and  $1 \leq \Delta \leq d$  depends on the orientation of  $\mathbf{k}$  with respect to the polytope faces represented by the NN. In any realistic setting, there is a maximum  $N_{f}L_{f}$  that can be achieved through training, and therefore the amplitude of the NN Fourier coefficient for a fixed  $\mathbf{k}$  is bounded from above. Along each  $\mathbf{k}$  direction, assume that the target function behavior at high frequencies falls off more slowly than  $k^{-\Delta -1}$ . Then, for large enough  $k$  the target function Fourier amplitudes,  $\widetilde{y} (\mathbf{k})$ , cannot be fit by the neural network with fixed architecture and parameters. Let  $\Omega$  denote the corresponding high-frequency region of  $\mathbb{R}^n$ . Reducing the  $L^2$  norm of the target function Fourier amplitude over  $\Omega$ ,  $\int_{\Omega}|\widetilde{y} (\mathbf{k})|^2 d\mathbf{k}$ , relative to the corresponding integral over  $\mathbb{R}^n$  (assuming the target function Fourier amplitudes are square-integrable) will tend to reduce the corresponding error arising from this spectral bias when evaluated on a particular set of data points. Our definition of frequency reduction in terms of the NUDFT amplitudes can be viewed as a proxy for reducing this  $L^2$  norm.
Applying the scale transformation with scale factor  $a > 1$  can lead to the aforementioned reduction of relative spectral energy over  $\Omega$ . The Fourier amplitudes for a function  $g(\mathbf{x})$  are related to those for the corresponding function,  $g_{\mathrm{scaled}}(\mathbf{x})$  acting on scaled inputs as  $\widetilde{g} (\mathbf{k}) = 1 / a\times \widetilde{g}_{\mathrm{scaled}}(\mathbf{k} / a)$ . This relationship shows directly that scale with  $a > 1$  maps a given Fourier component of the original function to a component at reduced frequency (and reduced overall magnitude) after applying the scaling transformation. Thus, denoting the corresponding high-frequency domain of the scaled target function as  $\Omega_{\mathrm{scaled}}$ , in general  $\Omega_{\mathrm{scaled}}\subset \Omega$  (again assuming  $a > 1$  and that the original target function decays more slowly at large  $k$  than the spectral bias bound). The smaller domain of integration, together with the relationship between  $\widetilde{y}$  and  $\widetilde{y}_{\mathrm{scaled}}$ , can reduce the relative amplitude of the spectral integral. The corresponding argument for rank further depends on the underlying distribution of the feature. Consider the simple example of a uniformly-distributed feature  $x$  over an interval  $[x_1,x_2]$ . Then  $\mathrm{rank}(x)$  acts in precisely the same way as scale with  $a = 1 / (x_2 - x_1)$ , and the same effect is seen (assuming  $x_{2} - x_{1} < 1$ ). Similar arguments can be made with other assumptions about the underlying distribution.
These arguments suggest how the transformations we consider can mitigate the impact of NN spectral bias, however they do not account for the other potentially adverse effects of rank and scale on the optimization process. In practice, these effects are important and motivate the learnable convex combination of rank and scale we propose, as it is implicitly regularized by the loss function itself. We plan to analyze the corresponding training dynamics analytically, and further formalize the illustrations above, in future work.
E Experiment Details
Datasets and Training. We train and evaluate all models across 10 random seeds, and report mean metrics with their  $95\%$  confidence intervals. We use  $70\%$  of each dataset for training,  $15\%$  for validation and  $15\%$  for testing. We report evaluation metrics on the test set that correspond to the lowest validation set loss for each model. We use simple target-encoding [21] to convert categorical variables to numerics, in order to avoid introducing potential interactions between categorical embedding techniques and our metrics of interest.
Hyperparameter Spaces. We conduct hyperparameter tuning for every model across 100 configurations using the validation set. We use HyperOpt [4] to tune the hyperparameters of all approaches
Parameter	Distribution
depth	UniformInt[2, 4, 8]
width	UniformInt[128, 256, 512]
learning rate	LogUniform[10-6, 10-2]
Table 3: Hyperparameter space for MLPs. Batch size is set to 128 and is not tuned. PyTorch's [20] implementation of Adam [17] with its default parameters is used for optimization.
Parameter	Distribution
max depth	UniformInt[5, 10, 20]
num. estimators	UniformInt[100, 500, 1000]
learning rate	LogUniform[10-5, 10-1]
Table 4: Hyperparameter space for XGBoost. The rest of the XGBoost parameters are set to the defaults of the DMLC XGBoost package [7].
Parameter	Distribution
nd = na	UniformInt[56, 60, 64]
nsteps	UniformInt[1, 2, 3]
learning rate	LogUniform[10-5, 10-2]
Table 5: Hyperparameter space for TabNet. Following the suggestion of the authors, we use the same value for  $n_d$  and  $n_a$ , and set the batch size to 1024. PyTorch's [20] implementation of Adam [17] with its default parameters is used for optimization.
considered. Since our main objective is to surface the impact of a frequency-reducing inductive bias and not to perform extensive benchmarking, we focus on tuning the parameters that control the model complexity and learning rates. Specifically, the parameter space considered for each model are shown in Tables 3, 4, and 5.
Baselines. As a baseline, we evaluate the impact of common normalization techniques on our metrics of interest and present our results in Figure 2. Specifically, we study MLPs with standardization, robust scaling and quantile normalization [21]. We choose these transformations to compare our methods against for two reasons. First, these are the most common transformations used in the tabular deep learning domain. Second, these transformations are instantiations of rank and scale. Therefore, besides normalization, they impact the frequency spectra. We use scikit-learn [21] implementations of these transformations with their default hyperparameters. In our experiments we observe that, among these baselines, there is not a single transformation that consistently performs the best across all datasets. This observation is consistent with the existing tabular deep learning literature, where studies that use different datasets find different transformations working the best. In these studies, the quantile transform, standardization and min-max scaling methods are the most commonly used.
Frequency Reduction and XGBoost. In addition to our various MLP experiments, we also study the impact of frequency reduction on XGBoost performance. We first train an MLP until convergence, then use the weights  $\theta_{g}$  to transform input data to a lower-frequency form. We then train an XGBoost model using the transformed data. From Table 6, it can be observed that reduction does not improve XGBoost performance. However, it improves simple MLPs to provide better performance than XGB on 3 datasets, and comparable performance on 2 datasets. The proposed method does not improve upon simple MLPs if the baseline performance is already high (i.e.,  $>99\%$ ), which is expected as these datasets do not feature high-frequency information that cannot be learned by MLPs with unit scale.
E.1 Weights learned by the proposed approach on 14 datasets
We investigate the weights  $\theta_{g}$  learned by our proposed approach on the 14 datasets. Since these weights get updated jointly with the neural network weights  $\theta_{f}$ , this investigation is essential to
dataset	XGB	XGB*	MLP*
MagicTelescope	93.74 ± 0.14	93.74 ± 0.14	93.67 ± 0.15
MiniBooNE	98.7 ± 0.02	98.7 ± 0.02	98.79 ± 0.02
bankMarketing	93.58 ± 0.12	93.58 ± 0.12	92.66 ± 0.17
california_housing	97.22 ± 0.1	97.22 ± 0.1	96.01 ± 0.1
electricity-normalized	98.81 ± 0.04	98.81 ± 0.04	94.51 ± 0.1
house_16H	95.62 ± 0.15	95.62 ± 0.15	95.68 ± 0.12
kdd-small	99.28 ± 0.05	99.28 ± 0.05	99.17 ± 0.08
Diabetes	71.02 ± 0.09	71.03 ± 0.09	68.88 ± 0.13
albert	76.47 ± 0.06	76.47 ± 0.06	72.8 ± 0.04
credit-card-clients	78.12 ± 0.21	78.11 ± 0.21	78.16 ± 0.23
eye Movements	91.38 ± 0.25	91.38 ± 0.25	80.34 ± 0.3
jannis	88.14 ± 0.13	88.08 ± 0.15	87.77 ± 0.11
phoneme	96.74 ± 0.15	96.73 ± 0.16	94.84 ± 0.12
pol	99.94 ± 0.01	99.94 ± 0.01	99.94 ± 0.01
Table 6: Un-normalized AUROCs for XGBoost (XGB), XGBoost with low-frequency embeddings (XGB*) and MLP with low-frequency embeddings (MLP*).
understand whether or not  $\theta_{g}$  merely works as a set of additional trainable parameters that adapt to the changes in the rest of the network parameters. If theta  $\theta_{g}$  depends on  $\theta_{f}$ , then the low-frequency mappings produced using  $\theta_{g}$  may not be useful as general purpose representations. Figures 8 and 9 depict the weights learned by our proposed approach. Shaded areas represent the  $95\%$  confidence intervals across 10 random seeds. Notably, neural network weights  $\theta_{f}$  get initialized differently across the seeds, while  $\theta_{g}$  gets initialized to the same values. Specifically, we initialize all scaling weights as 1, and all ranking weights to 0.5. More advanced initialization techniques may be explored to further improve performance. From the figures, we observe that  $\theta_{g}$  converges to similar values across different random seeds on all the datasets considered. Additionally, we observe that for the datasets bank-marketing (bank) and diabetes, some of the scaling weights converge to zero, eliminating the use of the corresponding features and causing the combining weights to not get updates (i.e., staying at 0.5). This behavior may be interpreted as a frequency-reducing feature selection mechanism: if a feature's contribution to the performance is not sufficiently high, compared to its contribution to the frequency of the decision boundary, the neural network may choose to drop it. This behavior should be further investigated and compared to methods that were designed to select informative features. Overall, the results in Figures 8 and 9, along with Figure 2, suggest that our proposed method is indeed enabling the network to learn useful representations of the data biased towards lower-frequency target functions.
E.2 Un-normalized AUROCs on all dataset and approaches considered
We provide un-normalized AUROCs of all approaches considered in Table 7. The methods unit, standard, robust, quantile and selective-quantile are applied to MLPs. Based on our experimental results, we observe that the best performing baseline varies across different datasets. As a result, in order to produce a comparison of the overall performance, we provide average normalized metrics in Figure 2.
F Computational Complexity
Our proposed method introduces the additional parameters  $\theta_{g}:\{\theta_{g}^{s},\theta_{g}^{r},\theta_{g}^{c}\}$ . The scaling and combining weights,  $\theta_g^s$  and  $\theta_g^c$  are learned together with the neural network weights  $\theta_{f}$ . The parameters  $\theta_{g}^{r}$  are set by fitting the ranking transformer to the training set as a preprocessing step (i.e., only once), and extracting a mapping from reference points in the training sets to their ranks. We sample 1000 points to extract this mapping.
In all experiments, we allocate two scaling weights (for raw and ranked features), and one combining weight for each feature. Therefore, the additional parameter complexity introduced to a neural network with input size  $N$  is  $\mathcal{O}(N)$ . In comparison, an additional fully connected layer introduces
dataset	XGB	XGB*	TabNet	TabNet*	MLP*
MagicTelescope	93.74 ± 0.14	93.74 ± 0.14	90.08 ± 1.54	91.56 ± 0.59	93.67 ± 0.15
MiniBooNE	98.7 ± 0.02	98.7 ± 0.02	89.47 ± 0.65	98.59 ± 0.03	98.79 ± 0.02
bankMarketing	93.58 ± 0.12	93.58 ± 0.12	91.74 ± 0.19	89.22 ± 0.51	92.66 ± 0.17
california_housing	97.22 ± 0.1	97.22 ± 0.1	NA	NA	96.01 ± 0.1
electricity-normalized	98.81 ± 0.04	98.81 ± 0.04	84.45 ± 2.58	84.72 ± 1.97	94.51 ± 0.1
house_16H	95.62 ± 0.15	95.62 ± 0.15	89.24 ± 1.17	94.54 ± 0.13	95.68 ± 0.12
kdd-small	99.28 ± 0.05	99.28 ± 0.05	96.62 ± 0.73	98.72 ± 0.15	99.17 ± 0.08
Diabetes	71.02 ± 0.09	71.03 ± 0.09	NA	NA	68.88 ± 0.13
albert	76.47 ± 0.06	76.47 ± 0.06	NA	NA	72.8 ± 0.04
credit-card-clients	78.12 ± 0.21	78.11 ± 0.21	NA	NA	78.16 ± 0.23
eye movements	91.38 ± 0.25	91.38 ± 0.25	NA	NA	80.34 ± 0.3
jannis	88.14 ± 0.13	88.08 ± 0.15	NA	NA	87.77 ± 0.11
phoneme	96.74 ± 0.15	96.73 ± 0.16	89.14 ± 0.28	81.28 ± 2.9	94.84 ± 0.12
pol	99.94 ± 0.01	99.94 ± 0.01	77.18 ± 6.46	94.41 ± 2.45	99.94 ± 0.01
dataset	unit	standard	robust	quantile	selective-quantile
MagicTelescope	93.32 ± 0.16	93.27 ± 0.18	93.52 ± 0.17	93.27 ± 0.18	93.37 ± 0.18
MiniBooNE	96.47 ± 0.02	98.29 ± 0.02	98.77 ± 0.02	98.74 ± 0.02	98.72 ± 0.02
bankMarketing	92.71 ± 0.14	92.26 ± 0.15	91.82 ± 0.16	92.11 ± 0.15	92.4 ± 0.14
california_housing	95.56 ± 0.1	95.75 ± 0.09	95.66 ± 0.09	95.65 ± 0.12	95.63 ± 0.09
electricity-normalized	93.04 ± 0.12	92.99 ± 0.12	92.65 ± 0.16	93.85 ± 0.12	94.64 ± 0.11
house_16H	94.68 ± 0.13	95.23 ± 0.13	95.53 ± 0.14	95.25 ± 0.14	95.5 ± 0.16
kdd-small	99.32 ± 0.05	99.16 ± 0.06	99.21 ± 0.07	99.28 ± 0.05	99.34 ± 0.05
Diabetes	69.65 ± 0.09	69.29 ± 0.09	69.57 ± 0.11	67.96 ± 0.11	69.54 ± 0.09
albert	72.16 ± 0.06	72.39 ± 0.06	71.99 ± 0.04	72.27 ± 0.04	72.27 ± 0.05
credit-card-clients	77.14 ± 0.25	77.67 ± 0.24	77.73 ± 0.2	77.39 ± 0.21	77.56 ± 0.27
eye movements	79.86 ± 0.29	80.62 ± 0.34	79.53 ± 0.28	75.77 ± 0.3	76.98 ± 0.29
jannis	86.67 ± 0.11	87.32 ± 0.11	87.39 ± 0.11	86.84 ± 0.12	86.99 ± 0.13
phoneme	94.13 ± 0.25	94.71 ± 0.13	94.18 ± 0.23	94.11 ± 0.26	94.12 ± 0.22
pol	99.96 ± 0.0	99.96 ± 0.0	99.94 ± 0.01	99.32 ± 0.03	99.94 ± 0.01
Table 7: Un-normalized AUROCs of all approaches considered. An asterisk (*) indicates that the corresponding model uses the proposed low-frequency transformations. Note that our TabNet experiments only consider binary classification datasets.
$\mathcal{O}(d_hN)$ , where  $d_h$  is the width of the subsequent layer. Notably, our approach can be extended to a high-dimensional setting similar to [11] to map each feature to a  $k$ -dimensional vector that scales and combines the same feature with its ranked counterpart in different ratios. Although this extension is out of the scope of this work, the complexity of this approach would be  $\mathcal{O}(kN)$ , where our current approach sets  $k \coloneqq 1$ .
G Limitations
In this paper, we focus on building the connection between the spectral bias of neural networks and its impact on tabular deep learning performance. We show that heterogeneous tabular datasets tend to feature high-frequency target functions as defined by the Fourier analysis methods presented. We also observe that with frequency-reducing transformations, these target functions can be smoothed to alleviate the negative impact of spectral bias. Consequently, we propose using frequency reduction as inductive bias while learning with neural networks. We discuss some of the limitations of our methods and analysis below.