SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vivien Cabannes
Stefano Vigogna
University of Rome Tor Vergata
A core principle in statistical learning is that smoothness of target functions allows to break the curse of dimensionality. However, learning a smooth function seems to require enough samples close to one another to get meaningful estimate of high-order derivatives, which would be hard in machine learning problems where the ratio between number of data and input dimension is relatively small. By deriving new lower bounds on the generalization error, this paper formalizes such an intuition, before investigating the role of constants and transitory regimes which are usually not depicted beyond classical learning theory statements while they play a dominant role in practice.
The current practice of machine learning consists in feeding a machine with many samples for it to infer a useful rule. In supervised learning, the samples are input/output (I/O) pairs, and the rule is a relationship to predict outputs from inputs. Once learned, the I/O mapping can be deployed in the wild to infer useful information from new inputs. Because it was not engineered by hand, one can question the existence of unwanted behaviors. Classical guarantees regarding the mapping's correctness are offered by statistics: assuming that the training samples are independent and identically distributed according to the future use cases, it is possible to derive theorems akin to the central limit theorem.
While many statistical learning principles offer practical insights, theoretical results often appear somewhat obscure, and forming intuition about them is often challenging, which limits their impact. In this paper, we focus on one simple principle: "smoothness allows us to break the curse of dimensionality". The curse of dimensionality is a generic term referring to a set of high-dimensional phenomena with significant practical consequences. In supervised learning, it manifests as follows: without a good prior on the I/O mapping to be learned, one can only get good estimates of the mapping close to the observed examples; as a consequence, to obtain a good global estimate, one needs to collect enough data points to finely cover the input space, which implies that the number of data points should scale exponentially with the dimension of the input space. Yet, when provided with the information that the mapping has some structure, one might need significantly less examples to learn from. This is notably the case when the mapping is assumed to be smooth. The goal of this paper is to better understand how and when we can expect to get much better convergence guarantees when the target function is known to be smooth.
Related work. Nonparametric local estimators were introduced as soon as the field of learning began to form in the 50's [11], and their consistency was studied extensively in the second half of the twentieth century (see Stone [29], Devroye et al. [8] and references therein). Introduced for scatter plots [7], local polynomials were the first estimators to leverage smoothness to improve regression [12]. They were later replaced by kernel methods, which are a powerful way to adapt to smoothness without much fine-tuning, and were widely regarded as state-of-the-art before the deep learning era [25]. Convergence results for kernel methods can be understood through the size of their associated functional spaces [13, 32], how those sizes relate to generalization guarantees [31], and how those spaces adhere to  $L^2$  [22, 26]. For least-squares regression, relations between the size of those spaces
Figure 1: Log-log-log-log plots of excess risk (in color) with respect to the number of samples ( $z$ -axis), the regularizer  $\lambda$  ( $x$ -axis) and the bandwidth  $\sigma$  ( $y$ -axis) when  $f^{*}(x) = \mathrm{sign}(\langle x, e_{1} \rangle)$  and  $\rho_{\mathcal{X}}$  is uniform on  $[-1, 1]^{2}$ . Asymptotically, there exists some  $(\lambda_{n}, \sigma_{n})$  to ensure that the excess risk decreases in  $O(n^{-\gamma})$  for a  $\gamma$  predicted by theory. However, for finite values of  $n$ , the excess risk can present different power law decays. The dark blue line on each plot indicates the transition between low- and high-sample regime: it corresponds to the graph  $\{(\sigma, \lambda, n) \mid n = \mathcal{N}_{1}(\sigma, \lambda)\}$  (see (10) for the definition of  $\mathcal{N}_{1}$ ). The figure illustrates a double descent phenomenon where excess risk peaks are reached when  $n = \mathcal{N}_{1}$  before a second descent takes place. Those peaks can be avoided in practice by computing the effective dimension  $\mathcal{N}$  and tuning hyperparameters to ensure it to be smaller than  $n$ .
and generalization guarantees are usually derived through operator concentration [27, 6]. More recently, transitory regime behaviors were described in Mei et al. [16], Mei and Montanari [15], and high-dimensional phenomena that lift the need to have more data than dimensions were studied in [23, 14], with Bach [3] relating the latter analyses with the former ones.
Contributions. This work focuses on the role of smoothness in breaking the curse of dimensionality for typical supervised learning problems. At a high-level, our main contribution is to showcase the importance of transitory regimes where one does not have enough samples to leverage high-order smoothness. In such transitory regimes, the behavior of the excess risk can be quite different than its asymptotic "stationary" behavior. Usually not well described by theory, they might be the dominating regimes in applied machine learning, where the number of samples is often relatively small compared to the input dimension (see Figure 1 for an illustration). This arguably explains the poor performance of kernel methods without strong kernel engineering in the deep learning era: they try to leverage smoothness, but usually do not access enough samples to meaningfully estimate high-order derivatives. More precisely, our contributions are twofold:
- We provide two generic "minimax" lower bounds that illustrate how the curse of dimensionality can not be fully beaten under smoothness assumptions alone.
- We delve more specifically into guarantees offered by algorithms that are built to leverage smoothness assumptions, and get a fine-grained picture of some of the transitory regimes where learning takes place in practice.
All results are illustrated by numerical experiments, some of them to be found in Appendix C.
2 The Significance of Constants
This section reviews classical results in learning theory, before providing generic lower bounds when relying solely on smoothness assumptions.
2.1 Established Upper Bounds
Supervised learning is concerned with learning a function from an input space  $\mathcal{X}$  to an output space  $\mathcal{Y}$  from a dataset  $\mathcal{D}_n = (X_i, Y_i)_{i \in [n]}$  of  $n \in \mathbb{N}$  examples. For simplicity, we will assume  $\mathcal{Y} = \mathbb{R}$  and  $\mathcal{X} = \mathbb{R}^d$ , or  $\mathcal{X} = \mathbb{T}^d$  being the torus. A learning rule, or learning algorithm, is a mapping  $\mathcal{A}: \mathcal{D}_n \mapsto f_n$  that builds a function  $f_n: \mathcal{X} \to \mathcal{Y}$  based on the dataset  $\mathcal{D}_n$ , with the goal of capturing
<sup>1</sup>We use  $[n] = \{1, \dots, n\}$  to denote the set of integer from one to  $n$ .
the underlying I/O relation. To discuss generalization to unseen examples, it is standard to model both the already collected and the future examples as independent realizations of a random pair  $(X,Y)$ .
Assumption 1. There exists a distribution  $\rho \in \Delta_{\mathcal{X} \times \mathcal{Y}}$  that has generated  $n$  independent training samples  $\mathcal{D}_n = (X_i, Y_i)_{i \in [n]} \sim \rho^{\otimes n}$ , and will generate future test samples independently.
Under Assumption 1, the quality of a mapping  $f: \mathcal{X} \to \mathcal{Y}$  is measured through the excess risk
$$
\mathcal {E} (f) = \mathcal {R} (f) - \mathcal {R} \left(f ^ {*}\right) = \| f - f ^ {*} \| _ {L ^ {2} \left(\rho_ {\mathcal {X}}\right)} ^ {2}, \tag {1}
$$
where  $\rho_{\mathcal{X}}$  is the marginal distribution of  $\rho$  on  $\mathcal{X}$  and, assuming that  $(Y|X = x)$  has a second order moment for every  $x \in \mathcal{X}$ ,
$$
f ^ {*} (x) = \mathbb {E} [ Y | X = x ], \quad \text {w h i c h m i n i m i z e s} \quad \mathcal {R} (f) = \mathbb {E} [ | f (X) - Y | ^ {2} ]. \tag {2}
$$
The risk  $\mathcal{R}(f)$  represents the average error of guessing  $f(X)$  in place of  $Y$  when the error is measured through the least-squares loss. From a statistical viewpoint, it is useful to model  $f_{n} = \mathcal{A}(\mathcal{D}_{n})$  as a random function (inheriting its randomness from the samples), so to study the expectation or the upper tail of the excess risk  $\mathcal{E}(f_n)$ . Provided that  $f^{*}$  is measurable, it is possible to find methods such that  $\mathcal{E}(f_n)$  converges to zero in probability. Without additional assumptions on  $f^{*}$ , it is not possible to give any guarantee on the speed of this convergence [12, Theorem 3.1]. However, when  $f^{*}$  is assumed to be smooth, the picture improves consequently.
Theorem 1 (Breaking the curse of dimensionality [12, 4]). Under Assumption 1, when  $f^{*}$  is  $\alpha$ -smooth for  $\alpha > 0$  in the sense that it admits  $\lfloor \alpha \rfloor$  derivatives that are regular, more precisely if  $f^{*} \in C^{\alpha}$  (i.e.  $f$  is  $\alpha$ -Hölder regular), or  $f^{*} \in H^{\alpha} = W^{\alpha,2}$  (i.e.  $f$  is  $\alpha$ -Sobolev regular), there exists a learning rule  $f_{n} = \mathcal{A}(\mathcal{D}_{n})$  that guarantees
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} \left(f _ {n}\right) \right] \leq c n ^ {- 2 \alpha / (2 \alpha + d)}, \tag {3}
$$
where  $c$  is a constant independent of  $n$ . Moreover, the bound (3) is minimax optimal, in the sense that for any rule  $f_{n} = \mathcal{A}(\mathcal{D}_{n})$  there exists a distribution  $\rho$  such that  $f^{*}\in \mathcal{C}^{\alpha}$ , or  $f^{*}\in H^{\alpha}$ , and the upper bound (3) holds as a lower bound with a different constant  $c$ .
Why do constants matter? At first glance, when two algorithms  $\mathcal{A}_1$  and  $\mathcal{A}_2$  guarantee two different upper bounds  $O(n^{-\gamma_1})$  and  $O(n^{-\gamma_2})$  on the expected excess risk,  $\mathcal{A}_1$  will be deemed superior to  $\mathcal{A}_2$  if  $\gamma_1 \geq \gamma_2$ , since after a certain number of samples, we will have that  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{A}_1(\mathcal{D}_n)] \leq \mathbb{E}_{\mathcal{D}_n}[\mathcal{A}_2(\mathcal{D}_n)]$ . However, the constants hidden in the front of the big  $O_s$  might lead to a different picture when given a small number of samples:  $\mathcal{A}_1$  might actually be a so-called "galactic algorithm", similarly to Strassen's algorithm for matrix multiplication, that might not be worth using without an indecently large number of samples.
2.2 Minimax Lower Bounds
The following lower bounds show how classical algorithms that reach the minimax optimal convergence rates based on smoothness assumptions (3) necessarily present constants that are growing fast with respect to the input dimension. Our analysis holds in noisy settings.2
Assumption 2 (Homoscedasticity). The noise in the label  $(Y\mid X = x)$  is assumed to be independent of  $x\in \mathcal{X}$  with var  $(Y\mid X = x) = \varepsilon^{2}$
Theorem 2. Under Assumptions 2, for any algorithm  $\mathcal{A}$ , there exists a target function  $f^{*}$  such that  $f^{*(\alpha +1)} = 0$ , and
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} \left(\mathcal {A} \left(\mathcal {D} _ {n}\right)\right) \right] \geq \frac {\varepsilon^ {2}}{n} \binom {d + \alpha} {d}. \tag {4}
$$
Theorem 3. Under Assumption 2, for any algorithm  $\mathcal{A}$ , there exists a target function  $f^{*}$  such that its Fourier transform is compactly supported on the  $\ell^{\infty}$ -disk of radius  $\omega$ , i.e.,  $\widehat{f}^{*}(\omega') = 0$  for all  $\| \omega' \|_{\infty} > \omega$ , and
$$
\mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} \left(\mathcal {A} \left(\mathcal {D} _ {n}\right)\right) \right] \geq \frac {\varepsilon^ {2} (2 \omega + 1) ^ {d}}{n}. \tag {5}
$$
2This contrasts with interpolation regimes where other phenomena might appear, requiring different analysis tools.
Figure 2: (Left) Convergence rates as a function of the input dimension  $d$  and the number of samples  $n$  when the target function is  $f^{*}(x) = x_{1}^{5}$  and  $k(x,y)\propto (1 + x^{\top}y)^{5}$ . We observe that convergence rates depend heavily on the dimension, which is mainly due to "changing constants". (Middle) Theoretical lower bound. As the number of samples increases and we enter the high-sample regime, the lower bound resembles the real convergence rates. (Right) Illustration of  $\alpha \mapsto \mathcal{N}(\alpha) = \binom{d + \alpha}{\alpha}$  for  $d = 100$ , which corresponds to the dimension of the space of polynomials with  $d$  variables of degree at most  $\alpha$ . Given a number of samples, Taylor expansion can only be estimated meaningfully up to the order  $\alpha$  such that  $\mathcal{N}(\alpha)\simeq n$ . In particular, it shows that in dimension one-hundred, one millions samples ( $n = 10^{6}$ ) only allow to leverage no more than fourth-order smoothness ( $\alpha = 4$ ). See Appendix C.2 for details.
Proof Sketch (detailed in Appendix). The proofs of those two new theorems consist in retaking the standard lower bound in  $\varepsilon^2 D / n$  when performing linear regression with  $D$  orthogonal features. In Theorem 2,  $D$  corresponds to the number of polynomials of degree less than  $\alpha$  with  $d$  variables; in Theorem 3,  $D$  is the number of integer vector  $m\in \mathbb{Z}^d$  whose norm is smaller than  $\omega$ .
Theorems 2 and 3 illustrate how the usage of strong smoothness assumptions on  $f^{*}$  can not guarantee an excess risk lower than  $\varepsilon^2$  without accessing an indecently large number of samples with respect to the input dimension (e.g.,  $n \geq \binom{d + \alpha}{d} \geq (1 + \alpha / d)^d$  or  $n \geq 2^d\omega^d$  respectively). Empirical validations are offered by Figure 2. In their inner-workings, those theorems capture how the rates derived through Theorem 1 are deceptive when one does not have enough samples compared to the size of the hypothesis space that  $f^{*}$  is assumed to belong to; and that the size of smooth functions spaces grows quite fast with respect to the dimension of the input space. Similar lower bound theorems can be proven with rates in  $n^{-2\alpha /(2\alpha +d)}$  under more detailed assumptions, as illustrated with Theorem 7 in Appendix. Efficient learning beyond the limits imposed by those theorems can only take place when leveraging other priors: for example, sparsity priors would reduce the minimax rates from  $\varepsilon^2 D / n$  to  $\varepsilon^2 s\log (D) / n$  where  $s$  is the sparsity index of  $f^{*}$  [9].
3 Crisp Picture in RKHS Settings
To provide a fine-grained analysis of the phenomena at stake, this section presents stylized settings where constants and transitory regimes can be studied precisely, allowing to get a better picture of convergence rates in practical machine learning setups.
3.1 Backbone Analysis
In the following, we shall consider a feature map  $\varphi : \mathcal{X} \to \mathcal{H}$ , with  $\mathcal{H}$  a Hilbert space and  $\varphi \in L^{2}(\rho_{\mathcal{X}})$ . The map  $\varphi$  linearly parameterizes the space of functions
$$
\mathcal {F} = \left\{f _ {\theta}: x \mapsto \langle \varphi (x), \theta \rangle_ {\mathcal {H}} \mid \theta \in \mathcal {H} \right\} \subset L ^ {2} \left(\rho_ {\mathcal {X}}\right). \tag {6}
$$
For example,  $\mathcal{H}$  could be  $\mathbb{R}^k$ ,  $\varphi$  seen as defining  $k$  features  $\varphi_i(x)$  on inputs  $x \in \mathcal{X}$ . This model can be used to estimate  $f^*$  through the empirical risk minimizer
$$
f _ {n, 0} \in \underset {f \in \mathcal {F}} {\arg \min } \sum_ {i \in [ n ]} | f (X _ {i}) - Y _ {i} | ^ {2}. \tag {7}
$$
In order to ensure that  $\mathcal{F}$  can learn any function  $f^{*}$ , the features can be enriched by concatenating an infinite countable number of features together. In this setting, it is more convenient to describe the geometry induced by  $\mathcal{F}$  through the (reproducing) kernel  $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$  defined as  $k(x,x^{\prime}) =$
$\langle \varphi (x),\varphi (x^{\prime})\rangle$  . When  $\mathcal{F}$  can fit too many functions, the estimate (7) needs to be refined to avoid overfitting. This paper will focus on Tikhonov (also known as ridge) regularization
$$
f _ {n} \in \underset {f \in \mathcal {F}} {\arg \min } \sum_ {i \in [ n ]} | f (X _ {i}) - Y _ {i} | ^ {2} + n \| f \| _ {\mathcal {F}} ^ {2}, \tag {8}
$$
where the norm is defined from (6) as  $\| f\|_{\mathcal{F}} = \inf \left\{\| \theta \| \mid f = f_{\theta}\right\}$ , but can also be expressed with the sole usage of  $k$  through the integral operator  $K:L^{2}(\rho_{\mathcal{X}})\to L^{2}(\rho_{\mathcal{X}})$ ,
$$
K f (x) = \int_ {\mathcal {X}} k \left(x, x ^ {\prime}\right) f \left(x ^ {\prime}\right) \rho_ {\mathcal {X}} \left(\mathrm {d} x ^ {\prime}\right) = \mathbb {E} _ {X} [ k (x, X) f (X) ], \tag {9}
$$
as  $\| f\|_{\mathcal{F}} = \left\| K^{-1 / 2}f\right\|_{L^2 (\rho_{\mathcal{X}})}$ , with the convention  $K^{-1}(\ker K) = \{+\infty \}$ .
The statistical quality of  $f_{n}$  (8) depends on two central quantities, defined as
$$
\mathcal {N} _ {a} (K) = \operatorname {T r} \left(K ^ {a} (K + 1) ^ {- a}\right) \quad \text {a n d} \quad \mathcal {S} (K) = \left\| (K + 1) ^ {- 1} f ^ {*} \right\| _ {L ^ {2} (\rho_ {\mathcal {X}})} ^ {2}, \tag {10}
$$
where  $a = 2$ . The first term, known as the effective dimension, quantifies the size of the space  $\mathcal{F}$  in which  $f^{*}$  is searched for. It relates to the variance of the estimator as a function of the dataset  $\mathcal{D}_n$ . It will capture the estimation error, the error due to the finite number of accessed samples, related to the risk of overfitting. The second term quantifies the adherence of  $f^{*}$  to  $\mathcal{F}$ . It can be understood as the proximal distance between  $f^{*}$  on  $\mathcal{F}$  since  $(K + 1)^{-1} = I - K(K + 1)^{-1}$  is a proximal projector. It will capture the approximation error, the error due to the fact that our model does not exactly fit the target function, related to the risk of underfitting.
Theorem 4 (High-sample regime learning behavior). Under Assumptions 1 and 2, as well as two mild technical Assumptions 3 and 4, when  $f^{*}$  is in the closure of  $\mathcal{F}$  in  $L^2(\rho_{\mathcal{X}})$ , there exists a constant  $c$  such that the estimate (8) verifies
$$
\left| \mathbb {E} _ {\mathcal {D} _ {n}} \left[ \mathcal {E} \left(f _ {n}\right) \right] - \frac {\varepsilon^ {2} \mathcal {N} _ {2} (K)}{n} - \mathcal {S} (K) \right| \leq c \mathcal {N} _ {1} (K) \left(a _ {n} \cdot \frac {\varepsilon^ {2} \mathcal {N} _ {1} (K)}{n} + a _ {n} ^ {1 / 2} \mathcal {S} (K)\right) \tag {11}
$$
where  $a_{n} = \mathcal{N}_{+}(K) / n$ , and  $\mathcal{N}_{+}(K) = \operatorname{ess}\sup_{x\sim \rho_{\mathcal{X}}}\mathrm{Tr}(K_x(K + 1)^{-1})$  with  $K_{x}$  the rank-one operator on  $L^2 (\rho_{\mathcal{X}})$  that maps  $f$  to the constant function equal to  $\mathbb{E}[f]k(x,x)$ . The different notions of search space size are always related by  $\mathcal{N}_2\leq \mathcal{N}_1\leq \mathcal{N}_+$ . Moreover, under the interpolation property  $K^{p}(L^{2}(\rho_{\mathcal{X}}))\hookrightarrow L^{\infty}(\rho_{\mathcal{X}})$ , i.e.  $\| K^p f\|_{\infty}\leq \| f\|_{L^2 (\rho_\mathcal{X})}$ , it holds that  $\mathcal{N}_{+}(\lambda^{-1}K) = O(\lambda^{-2p})$ ; while under the source condition  $f^{*}\in K^{r}(L^{2}(\rho_{\mathcal{X}}))$ , it holds that  $\mathcal{S}(\lambda^{-1}K) = O(\lambda^{2r})$ .
While the excess risk upper bound deriving from Theorem 4 is somewhat standard, the lower bound is new. This theorem states that the generalization error  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)]$  behaves as  $A(n,K) \coloneqq \varepsilon^2 \mathcal{N}_2(K) / n + \mathcal{S}(K)$  up to higher order terms specified in the right-hand side. Theorem 4 also holds for ridge-less regression (7) with  $\mathcal{N}_1(K,0) = \mathcal{N}_2(K,0) = \dim \mathcal{F}, \mathcal{N}_+(K,0) = \| K^{-1} \|^{-1} \| \varphi \|_{\infty}$ , and  $\mathcal{S}(K,0) = \| f^* - \pi_{\mathcal{F}} f^* \|^2$ , where  $\pi_{\mathcal{F}}$  is the  $L^2(\rho_{\mathcal{X}})$ -orthogonal projection onto  $\mathcal{F}$ . In this setting,  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_{n,0})] = A(n,K,0)(1 + O(n^{-1}))$ . More in general, we conjecture the right-hand side of Theorem 4 to be improvable with the removal of  $\mathcal{N}_1(K)$  in front of the rates (which is due to our usage of concentration inequalities on operators rather than on scalar values), the change of the second  $\mathcal{N}_1$  into  $\mathcal{N}_2$ , and the substitution of  $a_n^{1/2}$  by  $a_n$ . This would show that  $\mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)]$  behaves as  $A(n,K)(1 + O(a_n))$ . In the following, we will call very high-sample regimes situations where  $a_n \leq 1$ , and high-sample regimes situations where  $\mathcal{N}_2(K) \leq n$ .
3.2 Approach Generality
Despite their apparent simplicity, reproducing kernels  $k$  describe rich spaces of functions  $\mathcal{F}$ , known as reproducing kernel Hilbert space (RKHS), namely any Hilbert space of functions with continuous pointwise evaluations [2]. Classical examples are provided by subspaces of analytical functions  $C^\omega$
In practice, it is usual to add a regularization parameter  $\lambda$  in front of  $\| f\|_{\mathcal{F}}^2$  in (8). This parameter  $\lambda$  can be incorporated as a specific hyperparameter of the kernel by replacing  $k$  by  $\lambda^{-1}k$ . This is useful to unify the study of the different hyperparameters that might define a kernel.
<sup>4</sup>When  $\rho_{\mathcal{X}}$  does not present heavy tails behaviors, those three quantities actually behaves similarly.
Samples  $n$
Figure 3: Illustration of transitory regimes. In essence, Theorem 4 states that  $\mathcal{E}_n\coloneqq \mathbb{E}_{\mathcal{D}_n}[\mathcal{E}(f_n)] = A(n,K)(1 + h(n,K))$  for  $h = O(\mathcal{N}_{+}(K) / n)$ . We illustrate our upper-lower bound when  $A(n,K) = n^{-1 / 2}$  and  $nh(n,K)$  is known to be in  $[-10^{2},10^{2}]$ . The upper-lower bound forces  $\mathcal{E}_n$  to behave in  $n^{-1 / 2}$  when  $n$  goes to infinity, yet when  $n$  is small, it can showcase quite different "transitory" behaviors.
through the Gaussian kernels  $k(x, x') = \exp(-\|x - x'\|^2 / \sigma^2)$ , and by the Sobolev space  $H^{(d+1)/2}$  through the exponential kernel  $k(x, x') = \exp(-\|x - x'\| / \sigma)$ . Reproducing kernels encompass several approaches and algorithms that have been suggested to leverage smoothness of the target function  $f^*$ .
The first approach consists in estimating local Taylor expansion through local polynomials, defined through
$$
\varphi (x) = \lambda^ {- 1 / 2} \left(\mathbf {1} _ {x \in A} x ^ {i}\right) _ {i \leq \beta , A \in \mathcal {P}}, \tag {12}
$$
for  $\mathcal{P}$  a partition on  $\mathcal{X}$ ,  $\beta \in \mathbb{N}$  a degree,  $\lambda > 0$  a regularization parameter. Intuitively in high-dimension problems, where  $d = \dim(\mathcal{X})$  is big, leveraging local properties may not be very reasonable, since the covering of  $\mathcal{X}$  with local neighborhoods grows exponentially with the dimension  $d$  (when  $\mathcal{X}$  has unit volume, and neighborhoods have a fixed radius), meaning that if one wants to have enough samples per neighborhood,  $n$  should scale exponentially with  $d$ .
Rather than local properties, the second approach consists in leveraging global smoothness properties, through the estimation of Fourier coefficients. Such estimators can be built implicitly from translation-invariant kernels, defined as
$$
k \left(x, x ^ {\prime}\right) = \lambda^ {- 1} q \left(\left(x - x ^ {\prime}\right) / \sigma\right), \tag {13}
$$