SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let us consider two types of such transformations, rank and scale, that are applied to each feature independently. Consider a vector  $\mathbf{z}^{\ell}\in \mathbb{R}^{N}$  with elements  $(\mathbf{z}^{\ell})_r\in \{(\mathbf{x}_1)_\ell ,\ldots ,(\mathbf{x}_N)_\ell \}$  such that  $(\mathbf{z}^{\ell})_r\leq (\mathbf{z}^{\ell})_s\forall r\leq s$  . Then
$$
\operatorname {r a n k} \left(\left(\mathbf {x} _ {i}\right) _ {\ell}\right) = \frac {1}{N} \min  \left\{r \in \mathbb {N} \mid \left(\mathbf {z} ^ {\ell}\right) _ {r} = \left(\mathbf {x} _ {i}\right) _ {\ell} \right\}, \tag {9}
$$
$$
\operatorname {s c a l e} \left(\left(\mathbf {x} _ {i}\right) _ {\ell}\right) = a \times \left(\mathbf {x} _ {i}\right) _ {\ell}, \tag {10}
$$
for  $\mathbf{x}_i\in X$  and  $a\in \mathbb{R}$ . Equation (9) simply maps a feature to the range [0, 1], proportional to its position when ranked by magnitude along with the other values in the dataset. For any  $(\mathbf{x})_{\ell}$  not in the original dataset, rank returns a value interpolated between  $\mathrm{rank}(\mathbf{x}_i)_\ell$  and  $\mathrm{rank}(\mathbf{x}_j)_\ell$  for the  $(\mathbf{x}_i)_\ell$ ,  $(\mathbf{x}_j)_\ell \in X$  nearest to  $(\mathbf{x})_\ell$  such that  $(\mathbf{x}_i)_\ell < (\mathbf{x})_\ell < (\mathbf{x}_j)_\ell$ . Both scale and rank may alter the frequency spectrum of the target function described by a dataset by either stretching/squeezing fluctuations while preserving relative distances between points or neglecting distance information altogether and forcing the points to be uniformly distributed along a direction, respectively.
Using Equation (2), we present the high-frequency Fourier amplitudes (i.e.,  $|\mathbf{k}| > 0.5$ ) of the target functions of 8 tabular binary classification datasets with different transformations and their impact on performance across 10 random seeds in Table 1. Here, we implement scale as standardization, such that  $a$  in Equation (10) is the inverse standard deviation of the feature in the training set population. From the top panel of Table 1, it can be observed that scale and rank consistently reduce the frequency of the target functions of all datasets, although by different relative amounts.
The bottom panel of Table 1 shows that, overall, lowering target function frequency is correlated with better NN performance: converting the averaged  $S_{\hat{\mathbf{k}}}$  and NN performance for unit length, rank, and scale to their ranks within each dataset and comparing yields a Spearman's rank correlation coefficient of  $\approx -0.7$  with a  $p$ -value of  $\approx 3 \times 10^{-4}$ . However, this correlation is not exact: large reduction in high-frequency Fourier amplitudes may not always result in significant performance improvements. This is because the amount of high frequency information encoded by each feature is different across datasets. Also, scaling up the feature values by a significant amount reduces the high-frequency energy, but can also negatively impact optimization by generating large gradients
during training. Finally, rank discards relative distance information between points in favor of frequency reduction, and also impacts optimization as it produces non-centered features. Therefore, it is important to simultaneously balance the effect of a given frequency-reducing transformation with other potentially negative impacts on the loss landscape or due to information loss.
These observations indicate that there is not a simple one-size-fits-all transformation that reliably and consistently reduces the high-frequency spectral energy that results in improved network performance. However, with this trade-off between frequency reduction and potential loss of information or effects on the loss landscape in mind, we can take a first step in improving over the rank and scale transformations by doing them selectively. For example, to balance the aforementioned trade-off we consider a selective version of rank defined as
$$
\operatorname {s e l r a n k} \left(\left(\mathbf {x}\right) _ {\ell}\right) = \left\{ \begin{array}{l} \operatorname {r a n k} \left(\left(\mathbf {x}\right) _ {\ell}\right), \quad S _ {\hat {\mathbf {k}} _ {\ell}} \left[ \left\{\operatorname {r a n k} \left(\left(\mathbf {x} _ {i}\right) _ {\ell}\right), y _ {i} \right\} _ {i = 1} ^ {N _ {\mathrm {t r}}} \right] \leq \Theta S _ {\hat {\mathbf {k}} _ {\ell}} \left[ \left\{\left(\mathbf {x} _ {i}\right) _ {\ell}, y _ {i} \right\} _ {i = 1} ^ {N _ {\mathrm {t r}}} \right] \\ \left(\mathbf {x}\right) _ {\ell}, \quad \text {o t h e r w i s e} \end{array} \right. \tag {11}
$$
where the datapoints in the conditions are those in the training set and  $\Theta \in \mathbb{R}$ . Selrank applies rank only if it reduces the high frequency energy of the target function defined for the starting data representation by a certain relative amount,  $\Theta$ . Otherwise, selrank leaves a given feature as-is. The threshold  $\Theta$  parameterizes how significant the frequency reduction must be in order to outweigh the potentially detrimental loss of information or impact on the loss landscape. From Table 1, it can be observed that selrank performs comparably or better than its counterpart rank across the datasets considered, motivating the use of frequency-informed transformations.
3.4 Frequency Reduction as Inductive Bias
Reducing the frequency of the target function may improve neural network performance by reducing the impact of spectral bias. On the other hand, the way we reduce frequency may impact other factors that play a significant role in neural network training. Since the interactions between transformations that reduce frequency and other learning dynamics are unique to how a feature is originally distributed, we can loosely formulate finding a transformation  $g$  parameterized by  $\theta_g$  that reliably and beneficially reduces the frequency as a constrained optimization problem:
$$
\underset {\theta_ {f}, \theta_ {g}} {\operatorname {a r g m i n}} \frac {1}{N} \sum_ {i = 1} ^ {N} \ell \left(y _ {i}, f \left(g \left(\mathbf {x} _ {i}; \theta_ {g}\right)\right); \theta_ {f}\right) \quad \text {s . t .} \quad \int_ {K} P _ {f \circ g} (\mathbf {k}) d \mathbf {k} <   \int_ {K} P _ {f ^ {*}} (\mathbf {k}) d \mathbf {k}, \tag {12}
$$
where  $\theta_{f}$  denotes the trainable parameters of neural network  $f$ ,  $P$  can be evaluated on arbitrary (including unlabeled) points,  $K$  denotes a high frequency region of  $k$ -space, and  $f^{*}$  denotes a neural network that minimizes the empirical loss  $1 / N\sum_{i = 1}^{N}\ell (y_{i},f^{*}(\mathbf{x}_{i};\theta_{f^{*}}))$ . Equation (12) suggests that a beneficial frequency-reducing transformation can be found by limiting the space of acceptable solutions represented by  $f$  to yield lower spectral energy in its high frequency components, compared to its analogous model  $f^{*}$  trained on the raw data. One way to attempt solving this optimization problem is to simplify the frequency reduction constraint and directly use  $P_{f}(\mathbf{k})$  to regularize the training loss towards finding a low frequency solution. However, this approach is computationally expensive, would in general require transformations on multiple features simultaneously, and would require careful pre-determination of the region  $K$  to represent the spectral energy in high-frequency components of the learned function.
To alleviate these challenges we make the following simplifications. First, to avoid the computational complexity of working in the full feature space, we assume that all input features are conditionally independent given the target  $y$ , hence, they do not have interactions that significantly impact the frequency of the high dimensional decision boundary described by the data. Notably, we only use this assumption to limit our consideration to transformations that reduce the high frequency spectral energy by acting on individual features one at a time instead of jointly. After the transformation is conducted, consideration of feature interactions for the learning task is dictated by the underlying network architecture. Second, instead of relying on  $S_{\hat{\mathbf{k}}}$  as a proxy for  $\int_K P_f(\mathbf{k}) d\mathbf{k}$  to impose a frequency reduction constraint on  $f \circ g$  during training, we constrain  $g$  to be a composition of the two frequency-reducing transformations rank (9) and scale (10) (understood to be acting on the individual components of the feature vector  $\mathbf{x}$ ):
$$
g \left(\mathbf {x}; \theta_ {g}: \left\{\theta_ {g} ^ {s}, \theta_ {g} ^ {r}, \theta_ {g} ^ {c} \right\}\right) = \theta_ {g} ^ {c} \operatorname {s c a l e} \left(\operatorname {r a n k} \left(\mathbf {x}; \theta_ {g} ^ {r}\right); \theta_ {g} ^ {s}\right) + \left(1 - \theta_ {g} ^ {c}\right) \operatorname {s c a l e} \left(\mathbf {x}; \theta_ {g} ^ {s}\right) \quad \text {s . t .} \quad \theta_ {g} ^ {c} \in [ 0, 1 ], \tag {13}
$$
where  $\theta_g^r$  represents the set of parameters specifying the mapping from original feature values to their ranked counterparts (e.g., how the interpolation is done), and  $\theta_g^s$  denotes the set of coefficients
used to scale the input features. Note that  $\theta_g^r$  is not a variable of this optimization problem as it only depends on the order of feature values in training set. Equation (13) describes a linear trajectory between scaled features and their ranked counterparts, while the degree of scaling applied to original and ranked features can be different. Notably, although we describe a linear relationship, one can encode more complex relationships to enable exploration of a larger space of frequency reducing transformations. With  $g$  applying frequency reduction as a soft constraint, we drop the constraint from Equation (12), and set  $\{\theta_f,\theta_g\}$  to the minimizers of the empirical loss. We implement Equation (13) as the input layer of a neural network and let  $\theta_{g}$  get updated jointly with the rest of the network parameters during backpropagation. We satisfy the constraint  $\theta_g^c\in [0,1]$  by clamping these weights after each update. Note that, computational complexity introduced by Equation (13) is linear in the number of input features (see Appendix F).
It is important to note that rank and scale are used to individually reduce the high frequency spectral energy of the empirical distribution function  $\hat{p}(\mathbf{x})$ , even though our objective is to reduce the spectral energy for  $f(\mathbf{x};\theta_f) \sim P(y|\mathbf{x})$  instead. This is motivated by the observation that (13) is designed as an inductive bias that exploits the neural network  $f$ 's spectral bias, and  $g$ 's parameters are selected towards minimizing the empirical loss. Given that neural networks have strong spectral bias towards learning low frequency functions, when jointly optimized during training,  $f$  prefers mapping the training instances to an informative low-frequency form using  $g$  and fitting to this low-frequency alternative in order to minimize the loss further. In Section 4, we show empirical evidence that neural networks trained with  $g$  indeed have reduced spectral energy in their high-frequency components compared to the networks trained on the same input data.
Our evaluation focuses on 3 key metrics: performance, rate of convergence and the irregularity of functions learned. Performance is evaluated using accuracy and the area under the receiver operating characteristic curve (AUROC). Rate of convergence is evaluated using the mean number of training epochs required to minimize validation loss. Finally, irregularity of functions learned by the neural network models are measured using total high-frequency power in Equation 2 along top principal components (PCs). We evaluate our proposed approach using 14 benchmark classification datasets listed in Table 2. These datasets are used by [13] to demonstrate the performance gap between tree-based models and neural networks. However, unlike [13], we use these datasets without truncating any samples, and we drop id and date type features as the approaches we compare are not designed to utilize the information conveyed by them. Because we consider a wide variety of datasets, the evaluation metrics we collect for the baselines and the proposed method highly fluctuate. This makes it challenging to draw conclusions for the overall behavior of these methods. Based on these observations, similar to [13], we normalize our measurements and aggregate them across datasets. In Appendix E, we provide the raw measurements and additional details on data preparation and training. Implementation details to reproduce our results are provided in Appendix H.
Frequency Reduction with Neural Networks. We present normalized and aggregated statistics across 14 datasets to highlight the general behavior of different transformations. Figure 2 depicts these statistics, where boxes represent quartiles, notches represent  $95\%$  confidence intervals and whiskers represent the full range except outliers. From the figure, it can be observed that our proposed
Name	#Samples	#Features	Source
electricity [9]	45312	9	https://openml.org/d/151
house_16H	22784	17	https://openml.org/d/821
pol	15000	49	https://openml.org/d/722
kdd_ipums_ta_97-small	7019	61	https://openml.org/d/993
MagicTelescope [9]	19020	11	https://openml.org/d/1120
bank-marketing [9]	45211	17	https://openml.org/d/1461
phoneme	5404	6	https://openml.org/d/1489
MiniBooNE [9]	130064	51	https://openml.org/d/41150
eye movements [24]	10936	28	https://openml.org/d/1044
jannis	83733	55	https://openml.org/d/41168
california [19]	20640	8	https://www.dcc.fc.up.pt/ltorgo/Regression/cal_housing.html
albert	425240	79	https://openml.org/d/41147
credit card clients [9]	30000	24	https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients
Diabetes [9]	768	9	https://www.openml.org/search?type=data&sort=runs&id=37
Table 2: 14 Tabular datasets used in the experiments
Figure 2: Normalized measurements of performance, convergence and frequency across datasets and seeds. The simple frequency-informed approach, selrank, with rank implemented as quantile transform improves upon rank alone. Additionally, our proposed methods significantly outperform all baselines, while converging faster.
Figure 3: Weights learned by the proposed approach on electricity dataset. Lines represent values taken by the weights  $\theta_{g}$  throughout 250 epochs, and shaded regions represent the  $95\%$  confidence intervals. The weights consistently converge to similar values across 10 random seeds that vary data splits and neural network initializations. Due to heterogeneous nature of tabular datasets, scaling and ranking weights that minimize the loss for different features are different.
method provides significantly higher normalized accuracy and AUROC, while taking significantly less epochs to converge. All of the transformations considered result in reduction of frequency of the function learned, compared to the neural network trained on unit scaled data. However, as discussed in Section 3.3, existing methods do not benefit from frequency reduction as much as our proposed methods, as they may have side effects such as losing relevant information (i.e., rank), or negatively impacting optimization (i.e., robust, standard and rank). On the other hand, our proposed method is able to balance the tradeoff between performance gain from frequency reduction and loss due to the side effects of the transformations used, providing an effective inductive bias for tabular deep learning.
Investigating the Learned Weights. Figure 3 depicts the change of the weights of our proposed approach across 10 random seeds on electricity dataset. Appendix E.1 extends this plot to all datasets. Across different seeds, initialization of our proposed layer's weights stay the same (i.e.,  $\theta_g^s \coloneqq 1$  and  $\theta_g^c \coloneqq 0.5$ ), however, the network weights  $\theta_f$  and data splits change. The first two columns of the figure correspond to the scaling weights  $\theta_g^s$  for the raw features and ranked features, and the third column corresponds to the weights  $\theta_g^c$  that combine these features from Equation (13). Shaded regions in the figure represent  $95\%$  confidence intervals. From the figure, it can be observed that the confidence intervals of the learned weights are small, relative to their change over epochs. This implies that the values of these weights stay consistent across random seeds. Consequently, the network consistently learns similar representations for the same feature and the representations do not overfit to the underlying network architecture. The figure also shows that the amount of scaling and ranking that can be applied to gain performance is data-dependent. Therefore, it is computationally expensive to exhaustively find the best transformations for each feature.
Frequency Reduction with Tree-based Models. XGB typically outperforms all other tabular learning approaches and is known to not be impacted by irregularity of target functions [13]. Therefore, in order to isolate the performance improvements observed by our approach to spectral bias of neural networks, it is important to confirm that our approach does not improve XGB performance. In Appendix E, we provide empirical evidence that our approach's impact to XGB is insignificant. Additionally, even though we do not intend this study to be a benchmarking of state-of-the-art methods, we see that using our approach, a simple MLP with no regularization and limited hyperparameter
Figure 4: AUROCs of MLPs using the baseline transforms and the proposed method over multiple scale factors. As the scale factor increases, the target function frequency of the dataset also increases. Shaded regions correspond to  $95\%$  confidence intervals across 10 random seeds.
tuning can outperform XGB in 4 of the datasets that were identified to be tree-friendly by previous studies [13], while providing competitive performance in 2 datasets.
Frequency Reduction with other Network Architectures. Frequency reduction as an inductive bias can improve the performance of more sophisticated deep learning architectures as well. For example, our proposed layer's mappings can be used as an input to TabNet to significantly reduce the amount of hyperparameter tuning required. As discussed in Section 2, it is challenging to utilize TabNet's full potential due to its hyperparameter sensitivity, causing inconsistent results across different studies. However, this model also offers additional benefits beyond performance improvements such as interpretability. Therefore, we believe TabNet is a good candidate to evaluate our inductive bias on beyond the vanilla MLP. Similar to our other experiments, we limit the number of hyperparameter configurations swept for tuning to 100. For each dataset, we train an MLP with our proposed layer until convergence and use the layer to generate low-frequency versions of the corresponding dataset. We compare the performance of our approach to TabNet trained by following the author's suggestions [1]. We find that our approach improves TabNet's AUROC by  $4\% \pm 1.6\%$  (see Appendix E.2).
Performance over Varying Target Function Frequency. Due to the diverse nature of the datasets considered in our experiments, it may appear challenging to reliably draw conclusions about the effect of the proposed method. Different dataset characteristics naturally lead to variation in the performance gain observed from our methods. To address these concerns, we provide additional results using synthetic data. Specifically, we generate synthetic datasets of varying target function frequency by applying a common scale factor to  $k_{i}$  in Equation (24) while keeping all other parameters fixed. We train the same 2-hidden-layer MLP for 200 epochs on the unit-scaled, ranked, and scaled (via standardization) features. Figure 4 depicts the AUROCs of these MLPs over different target function frequencies. From the figure we observe that for low target function frequencies, all methods perform comparably. Also, performance decreases with increasing target function frequency due to NN's spectral bias. Our proposed method consistently outperforms the others and is the most robust to high frequency target functions.
In this paper, we studied the impact of spectral bias of neural networks on tabular datasets. We showed that tabular datasets tend to describe irregular target functions and this irregularity negatively impacts neural network performance. We proposed incorporating frequency reduction as an inductive bias when training neural networks to reduce the irregularity of these functions. We showed that our proposed method significantly improves neural network performance while reducing the number of training epochs needed to converge. Although we propose the new direction of using frequency reduction as an inductive bias for tabular deep learning, we merely scratched the surface when it comes to realizing said bias. For example, our analyses can be extended to other tabular deep learning approaches to determine if the performance improvements they offer can be explained from the lens of spectral analysis. We discuss other future directions as well as the limitations of our work in Appendix G.
We thank Tian Tan, Xuan Li, Man-Ho Marco Tang, Mihir Pendse and Jiajian Lu for valuable discussions and feedback. We are also grateful to the anonymous reviewers for providing constructive feedback on this manuscript.