SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679-6687, 2021.
[2] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15147, 2021.
[3] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, pages 685-694. PMLR, 2020.
[4] James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 115-123. PMLR, 2013.
[5] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[6] Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv:1912.01198, 2019.
[7] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[8] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[9] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[10] Jeffrey A Fessler and Bradley P Sutton. Nonuniform fast fourier transforms using min-max interpolation. IEEE transactions on signal processing, 51(2):560-574, 2003.
[11] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991-25004, 2022.
[12] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-18943, 2021.
[13] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[14] Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[15] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928-23941, 2021.
[16] Jonas Kiessling and Filip Thor. A computable definition of the spectral bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7168-7175, 2022.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] OpenAI. Gpt-4 technical report. arXiv, 2023.
[19] R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291-297, 1997.
[20] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
[21] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaopeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[22] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[23] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[24] Jarkko Salojärvi, Kai Puolamäki, Jaana Simola, Lauri Kovanen, Ilpo Kojo, and Samuel Kaski. Inferring relevance from eye movements: Feature extraction. In Workshop at NIPS 2005, in Whistler, BC, Canada, on December 10, 2005., page 45, 2005.
[25] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342, 2021.
[26] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
[27] Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 28(5):1746-1767, 2020.
[28] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033-11043, 2020.
A Fourier Analysis Details
In this section, we provide additional details regarding our Fourier analysis.
A.1 The NUDFT and its continuous limit
The primary starting point for our spectral analysis is the non-uniform discrete Fourier transform defined in Equation (1). This particular transform conveys spectral information about functions sampled at arbitrary data points and its adjoint is commonly used in the medical imaging domain to reconstruct images from measurements taken in Fourier space. Equation (1) can be interpreted as an approximation of the multidimensional continuous Fourier transform (FT) of  $f(\mathbf{x})$  weighted by the empirical distribution represented by the datapoints  $X$ . Its continuous limit as  $N\to \infty$  is the Fourier integral
$$
\mathcal {F} _ {\mathbf {x}} [ f (\mathbf {x}); \mu ] (\mathbf {k}) = \int_ {\mathbf {R} ^ {n}} f (\mathbf {x}) e ^ {- 2 \pi \mathbf {k} \cdot \mathbf {x}} \mu (d \mathbf {x}), \tag {14}
$$
where  $\mu (d\mathbf{x})$  is the probability measure representing the distribution from which the points in the dataset are sampled.
The NUDFT (as well as any spectral analysis method relying solely on information at a finite number of points without additional interpolation or assumptions) reflects the spectral information of not only the target function, but also the underlying distribution of the datapoints itself. It is in principle possible to separate out these contributions using density estimation methods, e.g., a multidimensional generalization of the technique in [16]. Presently, however, we simply define the frequency spectra of interest as that reflected in Equation 14 (and its approximations) and frame our analysis self-consistently in terms of this definition. In particular, we empirically observe spectral bias (see also [27]), see that certain transformation reduce high frequency energy, and propose an inductive bias that consistently outperforms other methods, all using this definition.
A.2 Limiting behavior of the NUDFT with respect to  $|\mathbf{k}|$
Some of the limiting behavior of the NUDFT amplitude spectrum can be understood from Equation (1). Specifically, consider the expression for  $P_{f}(\mathbf{k})$ :
$$
P _ {f} (\mathbf {k}) = \frac {1}{N ^ {2}} \left| \sum_ {i = 1} ^ {N} f \left(\mathbf {x} _ {i}\right) e ^ {- 2 \pi i \mathbf {k} \cdot \mathbf {x} _ {i}} \right| ^ {2} = \frac {1}{N} \overline {{f ^ {2}}} + \frac {2}{N ^ {2}} \sum_ {i = 1} ^ {N - 1} \sum_ {j = i + 1} ^ {N} f \left(\mathbf {x} _ {i}\right) f \left(\mathbf {x} _ {j}\right) \cos \left(2 \pi \mathbf {k} \cdot \left(\mathbf {x} _ {i} - \mathbf {x} _ {j}\right)\right). \tag {15}
$$
The overline represents the mean of a quantity evaluated over the  $\mathbf{x}_i$ . At small  $|\mathbf{k}|$ , the RHS of Equation (15) is approximately  $\overline{f}^2$ , the mean squared of the function or labels. This is the reason for the  $1 / \overline{f}^2$  scaling in front of Equation (2); otherwise, the scales of the Fourier amplitudes would not be directly comparable across datasets. As  $|\mathbf{k}|$  increases from 0, at some point,  $P_f$  starts to deviate from the mean squared. This happens when the  $\cos$  term in the expression above begins to deviate from 1 for a significant number of points. This occurs schematically when  $|\mathbf{k}| \sim 1/(2\pi \Delta x)$ , where  $\Delta x$  is a typical separation between datapoints. In the other limit, once  $|\mathbf{k}| \gg 1/(2\pi \Delta x)$ , the NUDFT is probing distances much smaller than the separations of points in the dataset. At this point, the NUDFT ceases to provide meaningful spectral information, as the dataset does not convey information about the target function on such small scales. The spectrum becomes noisy, since the last term on the RHS of Equation (15) features many cancellations, resulting in fluctuations around the first term,  $\overline{f^2}/N$ , on the RHS. The most interesting region from the standpoint of our analysis is that between these two regimes, describing collective oscillatory behavior, which is the region we focus on in our analysis.
There is another way to see that the  $1 / N$  asymptotic behavior of  $P_{f}$  at large frequencies is an artifact of the discrete distribution of datapoints. The NUDFT approximates the integral  $\int_{-\infty}^{\infty}f(\mathbf{x})\rho (\mathbf{x})e^{-2\pi ik\cdot \mathbf{x}}d\mathbf{x}$ . For  $f(\mathbf{x})\rho (\mathbf{x})$  continuous, bounded, and compactly supported, the integrated spectral energy density will be finite. However, the corresponding integral of the squared
NUDFT amplitudes, which should yield the same result as  $N \to \infty$  by the Plancherel theorem, does not exist for finite  $N$ . The problem again arises from the behavior discussed in the previous paragraph. To obey the Plancherel theorem, the result in the continuous limit needs to fall off fast enough in  $|\mathbf{k}|$  to be integrable, meaning that the  $\overline{f^2} / N$  asymptotic behavior discussed above is an artifact of the discrete approximation. We therefore limit our analysis to below this range, which motivates the upper cutoff  $k_*$  appearing in several of our expressions.
A.3 Approximating the high-dimensional spectral energy integral
Our spectral analysis primarily relies on the quantity  $S_{\hat{\mathbf{k}}}$  defined as a sum of squared NUDFT amplitudes as in Equation (2). This method considers spectral energy along single dimensions of  $\mathbb{R}^n$  at a time. Here we describe an alternative convolution method for approximating the full  $n$ -dimensional spectral energy integral over a finite high-frequency region in  $\mathbb{R}^n$ . In Appendix A.6 we show that both methods yield similar results for the datasets analyzed.
The method here is similar to that in [16]. Consider the approximate energy spectral density,  $P_{f}(\mathbf{k})$  for a function  $f$  integrated over a high-frequency region  $K\subset \mathbb{R}^n$ ,
$$
S _ {K} = \int_ {K} P _ {f} (\mathbf {k}) d \mathbf {k}. \tag {16}
$$
Rather than attempting to perform this high-dimensional integral in Fourier space, we can make use of the convolution theorem and work directly in feature space. Consider a region  $K$  defined by  $k_0 < |\mathbf{k}|_{\infty} \leq k_*$ , where  $||_{\infty}$  is the  $L_{\infty}$  norm,  $k_0$  is a threshold determining the start of the high-frequency region, and  $k_*$  is a cutoff of the high frequency region introduced for the same reason as discussed in the previous section. We can then write
$$
S _ {K} = \int_ {\mathbb {R} ^ {n}} P _ {f} (\mathbf {k}) \mathbb {1} _ {| \mathbf {k} | _ {\infty} \leq k _ {*}} d \mathbf {k} - \int_ {\mathbb {R} ^ {n}} P _ {f} (\mathbf {k}) \mathbb {1} _ {| \mathbf {k} | _ {\infty} \leq k _ {0}} d \mathbf {k}. \tag {17}
$$
Defining the function
$$
g (\mathbf {x}) = \int_ {\mathbb {R} ^ {n}} f \left(\mathbf {x} ^ {\prime}\right) \left(2 k _ {*}\right) ^ {n} \prod_ {i = 1} ^ {n} \operatorname {s i n c} \left(2 k _ {*} \left(\mathbf {x} - \mathbf {x} ^ {\prime}\right) _ {i}\right) \mu \left(d \mathbf {x} ^ {\prime}\right), \tag {18}
$$
the convolution theorem dictates that
$$
\mathcal {F} _ {\mathbf {x}} [ g (\mathbf {x}); \mu ] (\mathbf {k}) = \mathcal {F} _ {\mathbf {x}} [ f (\mathbf {x}); \mu ] (\mathbf {k}) \mathbb {1} _ {| \mathbf {k} | _ {\infty} \leq k _ {*}}, \tag {19}
$$
$$
\int_ {\mathbb {R} ^ {n}} | g (\mathbf {x}) | ^ {2} d \mathbf {x} = \int_ {\mathbb {R} ^ {n}} | \mathcal {F} _ {\mathbf {x}} [ g (\mathbf {x}); \mu ] (\mathbf {k}) | ^ {2} d \mathbf {k} = \int_ {| \mathbf {k} | _ {\infty} \leq k _ {*}} P _ {f} (\mathbf {k}) d \mathbf {k}. \tag {20}
$$
Approximating the  $\mathbf{x}^{\prime}$  integral in Equation (18) with a Monte Carlo integral sampling from the density defined by  $\mu$ , substituting the resulting expression into Equation (20) and using the definition in Equation (17), we arrive at
$$
S _ {K} [ f ] \approx \frac {\left(2 k _ {*}\right) ^ {n}}{N ^ {2}} \sum_ {i, j = 1} ^ {N} f \left(\mathbf {x} _ {i}\right) f \left(\mathbf {x} _ {j}\right) \left(\prod_ {\ell = 1} ^ {n} \operatorname {s i n c} \left[ 2 k _ {*} \left(\mathbf {x} _ {i} - \mathbf {x} _ {j}\right) _ {\ell} \right] - \prod_ {\ell = 1} ^ {n} \operatorname {s i n c} \left[ 2 k _ {0} \left(\mathbf {x} _ {i} - \mathbf {x} _ {j}\right) _ {\ell} \right]\right), \tag {21}
$$
This estimate of the spectral integral in Equation (16) is evaluated for binary classification datasets in Appendix A.6 below.
A.4 Defining the high-frequency regime
Both the NUDFT and fully-integrated energy spectrum approach rely on two different cutoffs to define the high-frequency range. The lower cutoff,  $k_{0}$ , sets the start of what we consider the high-frequency range. We take  $k_{0} = 0.5$  in our analyses, motivated by two observations. First, in our conventions, and for features scaled to unit length,  $|\mathbf{k}| < 0.5$  corresponds to frequency modes that fit less than half a period in any given dimension of the feature space. This corresponds to long-distance (i.e., global) fluctuations and so the high-frequency regime should be considered for  $|\mathbf{k}|$  larger than this value. The second motivation comes from the behavior of the normalized sinc function, which describes
the Fourier transform of the unit rectangle function. Say we had a dataset with all points having the same label distributed uniformly in the unit interval. The NUDFT would then approximate the normalized sinc function, which begins to decay when its argument is around 0.5. For larger  $|\mathbf{k}|$ , the envelope of the squared NUDFT amplitudes will decay as  $1 / |\mathbf{k}|^2$ , consistent with the fact that the corresponding target function does not have high-frequency components. Of course we are also free to choose values larger than 0.5. We have verified that the other choices of  $k_0 \sim 1$  do not affect our conclusions.
As pointed out in Appendix A.2, at high frequencies the behavior of our methods relying on discrete points fail to reproduce the correct decaying high-frequency behavior of the continuous Fourier amplitudes. Care must therefore be taken to properly select  $k_*$ , especially for computing  $S_K$ : for  $S_{\hat{\mathbf{k}}}$  the high-frequency  $\overline{f^2} / N$  behavior adds a constant, but this behavior leads to  $S_K \propto (2k_*)^n$  which would dominate the result numerically if we did not account for this behavior. As a result we implement the following procedure in computing  $S_K$  to compare between datasets: for each dataset, we first compute  $S_K$  for several different values of  $k_*$ , and choose a value near where the integral begins to scale as  $(2k_*)^n$ . We then manually enforce that the contributions to  $S_K$  above this value vanish. The corresponding integral then has the expected behavior, and we can directly compare  $S_K$  between datasets at arbitrarily large  $k_*$ , since the integral will saturate once it comes close to scaling as  $(2k_*)^n$ . Finally, for computing  $S_{\hat{\mathbf{k}}}$ , we simply cut off the sum in Equation (2) at a given value near where  $S_{\hat{\mathbf{k}}}$  approaches the  $\overline{f^2} / N$ . In practice, we take  $k_* = 20$  when computing  $S_{\hat{\mathbf{k}}}$ , but have verified that the results do not significantly change when varying this value.
A.5 Evaluating Fourier transforms along principal components
In several of our analyses, we evaluate the NUDFT  $\tilde{f}_X(\mathbf{k})$  along specific directions, such as those defined by the principal components (PCs) of the data. This approach was also used in [27] and we provide further details about this procedure here for evaluating along general directions in  $\mathbb{R}^n$
The principal component decomposition (i.e., PCA) typically takes one point  $\mathbf{x}$  in the original feature space and yields a transformed vector  $\mathbf{x}_{\mathrm{PC}}$  with entries corresponding to the projection of the mean-centered features onto the PC directions. Focusing on one PC direction, we have
$$
x _ {\mathrm {P C}} = \hat {\mathbf {e}} _ {\mathrm {P C}} \cdot (\mathbf {x} - \mathbf {m}), \tag {22}
$$
where  $\mathbf{m}$  is the vector defined by the mean of each  $n$  dimensions of the feature space over the dataset, and  $\hat{\mathbf{e}}_{\mathrm{PC}}$  is a unit vector pointing along the PC of interest in the original feature space. To take the Fourier transform along the direction  $\hat{\mathbf{e}}_{\mathrm{PC}}$ , we can define  $\mathbf{k}_{\mathrm{PC}} \equiv k_{\mathrm{PC}} \hat{\mathbf{k}}_{\mathrm{PC}}$ , where  $k_{\mathrm{PC}} = \mathbf{k} \cdot \hat{\mathbf{e}}_{\mathrm{PC}}$  and  $\hat{\mathbf{k}}_{\mathrm{PC}} = \hat{\mathbf{e}}_{\mathrm{PC}}$ , and plug this value of  $\mathbf{k}_{\mathrm{PC}}$  into Equation 1. Using Equation 22, we have
$$
\sum_ {j = 1} ^ {N} f \left(\mathbf {x} _ {j}\right) e ^ {- 2 \pi i \mathbf {k} _ {\mathrm {P C}} \cdot \mathbf {x} _ {j}} = e ^ {- 2 \pi i \mathbf {k} _ {\mathrm {P C}} \cdot \mathbf {m}} \sum_ {j = 1} ^ {N} f \left(\mathbf {x} _ {j}\right) e ^ {- 2 \pi i k _ {\mathrm {P C}} x _ {\mathrm {P C}, j}}. \tag {23}
$$
The phase factor in front of the RHS will drop out when we take amplitudes. The above expression tells us that to obtain the FT along the direction  $\hat{\mathbf{e}}_{\mathrm{PC}}$  defined by a given principal component, we can simply plug in the values of  $x_{\mathrm{PC}}$  corresponding to a given datapoint and treat the Fourier transform as one-dimensional with frequency  $k_{\mathrm{PC}}$  (up to an overall phase factor). However, this is not a one-dimensional Fourier transform: it is the full (approximate)  $n$ -dimensional Fourier transform, evaluated along a given one-dimensional subset of  $\mathbb{R}^n$ .
The above method applies when evaluating a dataset along one of its own PCs. In computing  $S_{\hat{\mathbf{k}}}$ , e.g., in Table 1, we typically average over the PCs defined for different transformations. In this case, one should directly substitute the corresponding  $\hat{\mathbf{k}}_{\mathrm{PC}} \coloneqq \hat{\mathbf{e}}_{\mathrm{PC}}$  into the LHS of Equation (23).
A.6 Spectra of binary classification datasets and correlation between  $S_{\hat{\mathbf{k}}}$  and  $S_K$
Here, we provide supplementary details related to our spectral analysis of the various binary classification datasets considered in our experiments.
In Figure 5 we show squared NUDFT amplitudes for the binary classification datasets of interest. The squared amplitudes  $P_{y}(\mathbf{k})$  are evaluated for unit-scale, ranked and scaled data, with rank implemented as a quantile transform and scale as standardization. The amplitudes are evaluated
Figure 5: Spectra for binary classification training datasets determined by Equation (1) for the rank and scale transformations, compared to the datasets with unit length features. Results are marginalized over 10 different random seeds for the train-test split, and over  $\hat{\mathbf{k}}_{\mathrm{PC}}$  corresponding to the 1st PC after each transformation is applied. The solid lines are the resulting mean values, with shaded regions representing the  $95\%$  confidence intervals. Different transformations significantly affect target function frequency spectra.
along the first PC after each transformation and for 10 different random training splits of the data. Solid curves represent the mean and shaded regions the corresponding  $95\%$  confidence intervals after
Figure 6: Comparison of different methods for quantifying the spectral energy at high frequencies. The horizontal axis corresponds to Equation (21), and the vertical axis to Equation (2). Squares, circles, and diamonds correspond to unit-scaled, rank, and scale features, and the different colors correspond to different datasets. The two methods are highly correlated.
marginalizing over all random seeds and different  $\hat{\mathbf{k}}_{\mathrm{PC}}$  directions. From the figure, it can be observed that the various transformations affect the high-frequency composition of these datasets in different ways, and in some more than others.
The corresponding values of  $S_{\hat{\mathbf{k}}}$  for several of these datasets are presented in Table 1. These are computed in the same way as the results in Figure 5 (i.e., averaged over random seeds and PCs), however still correspond to examining the spectrum along one dimension at a time. To assess the extent to which the one-dimensional results represent the full spectral properties in  $\mathbb{R}^n$ , in Figure 6, we compare the resulting values of  $S_{\hat{\mathbf{k}}}$  against the quantity  $S_K$  of Appendix A.3. Results are shown for all binary classification datasets except for albert, which features  $k_*$  below  $k_0$  for some methods when computed using the full Fourier integral. In order to compare the different datasets at similar scales, we compute  $S_K^{1/n}$ , since the dimensions of the volume integral over  $\mathbf{k}$  will be different for datasets with different feature space dimension. In computing  $S_K$ , due to the  $\mathcal{O}(N^2)$  computational complexity, we subsample each dataset to 10000 records if larger than this size. As seen in Figure 6, the two methods are strongly correlated: each dataset individually has a Spearman rank correlation coefficient of 1 when comparing  $S_{\hat{\mathbf{k}}}$  and  $S_K$ . From the figure, we also see the general behavior demonstrated in the spectral plots previously: the unit-scaled representation tends to imply the most high-frequency target functions, followed by rank, implemented as a quantile transform, and scale, implemented as standardization.
B Spectral Analysis of Neural Networks on Synthetic Data
In this section, we provide additional details and experimental results related to the synthetic data experiments reflected on the RHS of Figure 1 (reproduced in Figure 7 for convenience).
We consider binary classification on a synthetic dataset consisting of a single numerical feature,  $x$ , for simplicity. Heterogeneous tabular datasets used for classification tend to feature rapid variations of the class labels over short distances in feature space. To simulate this scenario, consider a normally-distributed feature  $x$ , and an oscillatory target function that depends on a function  $g(x)$  of the feature  $x$ , generating the class label as
$$
y (x) = \left\lfloor \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(1 + A _ {i} \sin \left(k _ {i} g (x) + \varphi_ {i}\right)\right) + \frac {1}{2} \right\rfloor . \tag {24}
$$
When sufficiently high frequency components  $k_{i}$  are considered, a simple multilayer perceptron (MLP) will have difficulty learning the target function, as expected from the spectral bias phenomenon. This is illustrated in the top panel of Figure 7. For this example, we take  $x \sim \mathcal{N}(0.5, 0.01)$ ,