First, we note that for any $h_2 \le h_0$, $\varphi \in \mathcal{F}$ and $n_{r-1} \le n \le n_r$,

$$
\begin{aligned}
& \left\| nh_2^p \mathbb{E} \vartheta_{\varphi,n,h_1,h_2,r}^{(R)} \right\|_{\mathbf{I}} \\
& \le \|K\|_{\infty} \|\alpha\|_{\infty}^p n_r \mathbb{E} [F(\mathbf{Y}) \mathbf{1}_{\{F(\mathbf{Y}) \ge (n_r/r)^{1/v}\}}] + \|L\|_{\infty} \|\max \alpha\|_{\infty}^{p+1} n_r \mathbb{E} [F(\mathbf{Y}) \mathbf{1}_{\{F(\mathbf{Y}) \le (n_r/r)^{1/v}\}}] \\
& \quad + \left[ \|K\|_{\infty} \|\delta_1\|_{\infty} + \|\alpha\|_{\infty}^p \|\delta_2\|_{\infty} + \|L_1\|_{\infty} \|\alpha\|_{\infty}^p \|\delta\|_{\infty}^2 \right] n_r \mathbb{E} [F(\mathbf{Y}) \mathbf{1}_{\{F(\mathbf{Y}) \ge (n_r/r)^{1/v}\}}] \\
& \quad + \left[ \|L_1\|_{\infty} \|\delta\|_{\infty} + \|\delta_1\|_{\infty} \|\delta_2\|_{\infty} + \|\alpha\|_{\infty}^p \|\delta\|_{\infty}^2 \|\delta_2\|_{\infty} \right] n_r \mathbb{E} [F(\mathbf{Y}) \mathbf{1}_{\{F(\mathbf{Y}) \ge (n_r/r)^{1/v}\}}] \\
& \quad + \|\delta_2\|_{\infty} \|\delta_1\|_{\infty} n_r \mathbb{E} [F(\mathbf{Y}) \mathbf{1}_{\{F(\mathbf{Y}) \ge (n_r/r)^{1/v}\}}] \\
& \le \mathcal{C}_{c,p,C,K} n_r \mathbb{E} [F(\mathbf{Y}) \mathbf{1}_{\{F(\mathbf{Y}) \ge (n_r/r)^{1/v}\}}],
\end{aligned}
$$

for some constant $\mathcal{C}_{c,p,C,K}$. Here, the constant $c$ and the function $p$ are applied in the definition of $\alpha(\cdot)$ in (2.3). We further have by (F.v)

$$ \mathbb{E}F^v(\mathbf{Y})\mathbf{1}\{\mathbf{X} \in \mathcal{J}\} \le \infty, $$

and we see that uniformly in $n_{r-1} \le n \le n_r$, $h_2 \le h_0$ and $\varphi \in \mathcal{F}$,

$$ \left\| nh_2^p \mathbb{E} \vartheta_{\varphi,n,h_1,h_2,r}^{(R)} \right\|_{\mathbf{I}} = o(n_r^{1/v} r^{1-1/v}) = o\left(\sqrt{n_r a_r \log(1/a_r)}\right), $$

as $r \to \infty$, where

$$ \tilde{a}_r = c(\log n_r / n_r)^{1-2/v}. $$

We readily obtain that, by the monotonicity of the function $h \to h \log(1/h)$, $h \le 1/e$,

$$ \lim_{r \to \infty} \max_{n_{r-1} \le n \le n_r} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_r \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^p} \left\| \mathbb{E} \vartheta_{\varphi,n,h_1,h_2,r}^{(R)} \right\|_{\mathbf{I}}}{\sqrt{(\log(1/h_2) \vee \log \log n)}} = 0. \quad (9.39) $$

It remains to establish that, with probability 1,

$$ \lim_{r \to \infty} \max_{n_{r-1} \le n \le n_r} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_r \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^p} \left\| \vartheta_{\varphi,n,h_1,h_2,r}^{(R)} \right\|_{\mathbf{I}}}{\sqrt{(\log(1/h_2) \vee \log \log n)}} = 0. \quad (9.40) $$

Similarly, as above, we have

$$ \max_{n_{r-1} \le n \le n_r} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_r \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \left\| \vartheta_{\varphi,n,h_1,h_2,r}^{(R)} \right\|_{\mathbf{I}} \le C \sum_{i=1}^{n_r} F(\mathbf{Y}_i) \mathbf{1}\{F(\mathbf{Y}_i) > (n_r/r)^{1/v}\}. $$

According to the proof of Lemma 1 of Einmahl and Mason (2000), we see that the argument there also applies if we set $h_{2,n_r} = c(\log n_r / n_r)^{1-2/v}$ in equation (2.10) of Einmahl and Mason (2000) to give as $r \to \infty$, with probability 1,

$$ \sum_{i=1}^{n_r} F(\mathbf{Y}_i) \mathbf{1}\{F(\mathbf{Y}_i) > (n_r/r)^{1/v}\} = o\left(\sqrt{n_r a_r \log(1/a_r)}\right), $$

and we see by the same argument as in (9.39) that (9.40) holds. Hence the proof of the lemma is complete. $\square$

## 9.3 Proof of Theorem 3.1.3

We first remark the following decomposition

$$
\begin{aligned}
& \mathbb{E} U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) \\
& = \frac{1}{h_2^p} \mathbb{E} \left[ \varphi(\mathbf{Y}) K \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \right] \\
& \quad + \frac{1}{h_2^p} \mathbb{E} \left[ \varphi(\mathbf{Y}) L \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \delta(\mathbf{X}) \right] \\
& \quad + \mathbb{E} \left[ \frac{1}{nh_2^p} \sum_{i=1}^n \varphi(\mathbf{Y}_i) \left[ K \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \right]
\end{aligned}
$$
$$
\begin{align}
&+pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}}\alpha(f(\mathbf{X}_i))\right)\alpha^p(f(\mathbf{X}_i))\delta^2(\mathbf{X}_i)\right] \nonumber \\
&+\mathbb{E}\left[\frac{1}{nh_2^p}\sum_{i=1}^n\varphi(\mathbf{Y}_i)\left[L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2}\alpha(f(\mathbf{X}_i))\right)\delta(\mathbf{X}_i)\delta_1(\mathbf{X}_i)+p\alpha^p(f(\mathbf{X}_i))\delta(\mathbf{X}_i)\delta_2(\mathbf{t},\mathbf{X}_i)\right]\right] \nonumber \\
&+\mathbb{E}\left[\frac{1}{nh_2^p}\sum_{i=1}^n\varphi(\mathbf{Y}_i)\delta_2(\mathbf{t},\mathbf{X}_i)\delta_1(\mathbf{X}_i)\right]. \tag{9.41}
\end{align}
$$

In the setting of the proof of Proposition 10.0.6, the inverse function theorem yields the existence and differentiability of $V_t(\mathbf{u})$, the inverse function of $U_t(\mathbf{v}) = \mathbf{v}\alpha(f(\mathbf{t}-\mathbf{v}))$ in a neighborhood of zero independent of $\mathbf{t}$. This, together with the facts that $K$ has bounded support, $\alpha$ is bounded away from zero and $h_{2,n} \to 0$, we obtain, by the change of variables $h\mathbf{z} = (\mathbf{t}-\mathbf{s})\alpha(f(\mathbf{t}-(\mathbf{t}-\mathbf{s})))$, that the first term in (9.41) is equal to

$$
\begin{align*}
& \frac{1}{h_2^p} \mathbb{E} \left[ \varphi(\mathbf{Y}) K \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \right] \\
&= \frac{1}{h_2^p} \int_{\mathbb{R}^p} \int_{\mathbb{R}^p} \varphi(\mathbf{u}) K \left( \frac{\mathbf{t} - \mathbf{s}}{h_2} \alpha(f(\mathbf{s})) \right) \alpha^p(f(\mathbf{s})) f_{XY}(\mathbf{s}, \mathbf{u}) \, d\mathbf{s} \, d\mathbf{u} \\
&= \frac{1}{h_2^p} \int_{\mathbb{R}^p} \mathbb{E} [\varphi(\mathbf{u}) \mid \mathbf{X} = \mathbf{s}] f_{\mathbf{X}}(\mathbf{s}) \alpha^p(f(\mathbf{s})) K \left( \frac{\mathbf{t} - \mathbf{s}}{h_2} \alpha(f(\mathbf{s})) \right) \, d\mathbf{s} \\
&:= \frac{1}{h_2^p} \int_{\mathbb{R}^p} \psi(\mathbf{s}) \alpha^p(f(\mathbf{s})) K \left( \frac{\mathbf{t} - \mathbf{s}}{h_2} \alpha(f(\mathbf{s})) \right) \, d\mathbf{s} \\
&= - \int_{\mathbb{R}^p} \psi(\mathbf{t} - V_t(h_2 \mathbf{z})) \alpha^p(f(\mathbf{t} - V_t(h_2 \mathbf{z}))) \frac{dV_t(h_2 \mathbf{z})}{d(h_2 \mathbf{z})} K(\mathbf{z}) \, d\mathbf{z} \\
&= \sum_{k=0}^{\ell} a_{k,h_2}(\mathbf{t}) h_2^k + o(h_2^{\ell}),
\end{align*}
$$

where as $h_2 \to 0$, uniformly in $\mathbf{t} \in \mathbf{I}$ and the set of functions $a_{k,\delta}$, which are uniformly bounded and equicontinuous, are defined as

$$a_{2k+1,\delta}(\mathbf{t}) = 0, \quad a_{2k,\delta}(t) = \sum_{|v|=2k} \frac{\tau_v}{v!} D_v \left( \frac{\mathbb{E}[\varphi(\mathbf{u}) | \mathbf{X} = \mathbf{t}] f_{\mathbf{X}}(\mathbf{t})}{\alpha^{2k}(f(\mathbf{t}))} \right),$$

for $k \le l/2$, in particular, $a_{0,\delta}(\mathbf{t}) = \mathbb{E}[\varphi(\mathbf{Y}) | \mathbf{X} = \mathbf{t}] f_{\mathbf{X}}(\mathbf{t})$. Now, we turn to the second term in (9.41). By using the expansion (9.4) of $\delta$, we get

$$
\begin{align*}
& \frac{1}{h_2^p} \mathbb{E} \left[ \varphi(\mathbf{Y}) L \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \delta(\mathbf{X}) \right] \\
&= \frac{1}{h_2^p} \int_{\mathbb{R}^p} \mathbb{E} [\varphi(\mathbf{u}) \mid \mathbf{X} = \mathbf{s}] f_{\mathbf{X}}(\mathbf{s}) \alpha^{p-1}(f(\mathbf{s})) (\alpha(f_{\mathbf{X};n}(\mathbf{s};h_1)) - \alpha(f(\mathbf{s}))) L \left( \frac{\mathbf{t} - \mathbf{s}}{h_2} \alpha(f(\mathbf{s})) \right) d\mathbf{s} \\
&= \frac{1}{h_2^p} \int_{\mathbb{R}^p} \mathbb{E} [\varphi(\mathbf{u}) \mid \mathbf{X} = \mathbf{s}] f_{\mathbf{X}}(\mathbf{s}) \alpha^{p-1}(f(\mathbf{s})) \alpha(f_{\mathbf{X};n}(\mathbf{s};h_1)) L \left( \frac{\mathbf{t} - \mathbf{s}}{h_2} \alpha(f(\mathbf{s})) \right) d\mathbf{s} \\
&\quad - \frac{1}{h_2^p} \int_{\mathbb{R}^p} \mathbb{E} [\varphi(\mathbf{u}) \mid \mathbf{X} = \mathbf{s}] f_{\mathbf{X}}(\mathbf{s}) \alpha^p(f(\mathbf{s})) L \left( \frac{\mathbf{t} - \mathbf{s}}{h_2} \alpha(f(\mathbf{s})) \right) d\mathbf{s} \\
&:= - \int_{\mathbb{R}^p} \psi(\mathbf{t} - V_t(h_2 \mathbf{z})) \alpha^{p-1}(f(\mathbf{t} - V_t(h_2 \mathbf{z}))) \alpha(f_{\mathbf{X};n}(\mathbf{t} - V_t(h_2 \mathbf{z})); h_1) \frac{dV_t(h_2 \mathbf{z})}{d(h_2 \mathbf{z})} L(\mathbf{z}) d\mathbf{z} \\
&\quad - \sum_{k=0}^{\ell} a_{k,h_2}(\mathbf{t}) h_2^k + o(h_2^{\ell}) \\
&= \int_{\mathbb{R}^p} F(h_2 \mathbf{z}) G(h_2 \mathbf{z}) L(\mathbf{z}) d\mathbf{z} - \sum_{k=0}^{\ell} a_{k,h_2}(\mathbf{t}) h_2^k + o(h_2^{\ell}).
\end{align*}
$$

Then, using that $\int L(\mathbf{z}) d\mathbf{z} = \int \mathbf{z}_i L(\mathbf{z}) d\mathbf{z} = 0$ and expanding, we obtain

$$
\begin{align*}
& \frac{1}{h_2^p} \mathbb{E} \left[ \varphi(\mathbf{Y}) L \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \delta(\mathbf{X}) \right] \\
&= -\frac{h_2^2}{2} \int_{\mathbb{R}^p} \left( \sum_{i,j=1}^d (F_{i,j}'' G + F_i' G'_j + F_j' G'_i + F G_{i,j}'')(\theta(h_2 \mathbf{z})) \mathbf{z}_i \mathbf{z}_j \right) L(\mathbf{z}) d\mathbf{z}
\end{align*}
$$
$$ -\sum_{k=0}^{\ell} a_{k,h_2}(\mathbf{t})h_2^k + o(h_2^\ell). $$

Since $F(\cdot)$ and its partial derivatives are bounded and $L(\cdot)$ is also bounded and has bounded support, as a result of Giné and Sang (2013) and Einmahl and Mason (2005), we obtain that

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \frac{1}{h_2^p} \mathbb{E} \left[ \varphi(\mathbf{Y}) L \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \delta(\mathbf{X}) \right] = O(h_1^2 h_2^2). $$

From the above results, it can be deduced directly that

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \left| \mathbb{E} U_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n}) - r^{(1)}(\varphi, \mathbf{t}) f_{\mathbf{X}}(\mathbf{t}) \right| = O(h_2^4 + h_1^2 h_2^2). $$

□

## 9.4 Proof of Corollary 3.1.4

The proof of Corollary 3.1.4 is based on Theorems 3.1.2 and 3.1.3. □

□

## 9.5 Proof of Corollary 3.1.7

It is easy to show that the following relation holds

$$ \begin{aligned} & \left| \tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - \hat{\mathbb{E}} \tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) \right| \\ & \le \frac{|r_n(\varphi, \mathbf{t}; h_1, h_2) - \mathbb{E}(r_n(\varphi, \mathbf{t}; h_1, h_2))|}{|f_n(\mathbf{t}; h_{1,n}, h_{2,n})|} \\ & + \frac{|\mathbb{E}(r_n(\varphi, \mathbf{t}; h_1, h_2))|}{|f_n(\mathbf{t}; h_1, h_2)|} \frac{|f_n(\mathbf{t}; h_1, h_2) - \mathbb{E}(f_n(\mathbf{t}; h_1, h_2))|}{|f_n(\mathbf{t}; h_1, h_2)|} \end{aligned} $$

From Theorem 3.1.1 with $\varphi \equiv 1$, we get that

$$ \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathbf{I}} |f_n(\mathbf{t}; h_1, h_2) - \mathbb{E} f_n(\mathbf{t}; h_1, h_2)| = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n \tilde{a}_n}} \right) \quad \text{a.s.} $$

and from Theorem 3.1.1 also, we have

$$ \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} |r_n(\varphi, \mathbf{t}; h_1, h_2) - \mathbb{E} r_n(\varphi, \mathbf{t}; h_1, h_2)| = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n \tilde{a}_n}} \right) \quad \text{a.s.} $$

So, it follows, with probability 1, that

$$ \sup_{a_n \le h_{1,n} \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \left| \tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - \hat{\mathbb{E}} \tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) \right| = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n \tilde{a}_n}} \right) \quad \text{a.s.} $$

□

## 9.6 Proof of Corollary 3.1.8

As we saw before, we have

$$ \begin{aligned} \mathbb{E}(f_n(\mathbf{t}; h_1, h_2)) &= \frac{1}{h_2^p} \mathbb{E} \left[ K \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f_{\mathbf{X};n}(\mathbf{X}; h_1)) \right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}; h_1)) \right] \\ &= \frac{1}{h_2^p} \mathbb{E} \left[ K \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \right] \end{aligned} \quad (9.42) $$

$$ + \frac{1}{h_2^p} \mathbb{E} \left[ L \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^p(f(\mathbf{X})) \delta(\mathbf{X}) \right] \quad (9.43) $$

$$ + \mathbb{E} \left[ \frac{1}{nh_2^p} \sum_{i=1}^{n} \left[ K \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \right] $$
$$+pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}}\alpha(f(\mathbf{X}_i))\right)\alpha^p(f(\mathbf{X}_i))\delta^2(\mathbf{X}_i)]\quad (9.44)$$

$$+\mathbb{E}\left[\frac{1}{nh_2^p}\sum_{i=1}^n\left[L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2}\alpha(f(\mathbf{X}_i))\right)\delta(\mathbf{X}_i)\delta_1(\mathbf{X}_i) + p\alpha^p(f(\mathbf{X}_i))\delta(\mathbf{X}_i)\delta_2(\mathbf{t},\mathbf{X}_i)\right]\right] \quad (9.45)$$

$$+\mathbb{E}\left[\frac{1}{nh_2^p}\sum_{i=1}^n \delta_2(\mathbf{t}, \mathbf{X}_i)\delta_1(\mathbf{X}_i)\right]. \quad (9.46)$$

We know that all the terms (9.43), (9.44), (9.45) and (9.46) are of order

$$O_{a.s.}\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right).$$

Therefore, we get

$$|\mathbb{E}(f_n(\mathbf{t}; h_1, h_2)) - f_X(\mathbf{x})| \le \left|\frac{1}{h_2^p} \int K\left(\frac{\mathbf{t}-\mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \alpha^p(f(\mathbf{x}))f_X(\mathbf{x})d\mathbf{x} - f_X(\mathbf{t})\right| \quad (9.47)$$

$$+O_{a.s.}\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right). \quad (9.48)$$

Thus, it suffices to bound (9.47). Notice that, by using the change of variables $h_2\mathbf{u} = (\mathbf{t}-\mathbf{x})\alpha(f(\mathbf{t}-(\mathbf{t}-\mathbf{x}))$,

$$\begin{aligned} & \left|\frac{1}{h_2^p} \int K\left(\frac{\mathbf{t}-\mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \alpha^p(f(\mathbf{x}))f_X(\mathbf{x})d\mathbf{x} - f_X(\mathbf{t})\right| \\ &= \left| - \int \left(\alpha^p(f(\mathbf{t}-V_t(h_2\mathbf{u})))f_X(\mathbf{t}-V_t(h_2\mathbf{u}))\frac{dV_t(h_2\mathbf{u})}{d(h_2\mathbf{u})} - f_X(\mathbf{t})\right) K(\mathbf{u})d\mathbf{u} \right| \\ &=: \left| \int H(h_2\mathbf{u})K(\mathbf{u})d\mathbf{u} \right|. \end{aligned}$$

We see that $H(\cdot)$ and its partial derivatives are bounded and $K(\cdot)$ is bounded and has bounded support. We obtain that

$$\left|\frac{1}{h_2^p} \int K\left(\frac{\mathbf{t}-\mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \alpha^p(f(\mathbf{x}))f_X(\mathbf{x})d\mathbf{x} - f_X(\mathbf{t})\right| = O(h_2^4). \quad (9.49)$$

So, we conclude that,

$$|\mathbb{E}(f_n(\mathbf{t}; h_1, h_2)) - f_X(\mathbf{x})| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) + O(h_2^4) \quad \text{a.s.} \quad (9.50)$$

Similarly, we get

$$\left|\mathbb{E}(r_n(\varphi, \mathbf{t}; h_1, h_2)) - r^{(1)}(\varphi, \mathbf{t})f_X(\mathbf{x})\right| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) + O(h_2^4) \quad \text{a.s.} \quad (9.51)$$

By combining (9.50) with (9.51), we obtain the desired result. $\square$

## 9.7 Proof of Theorem 3.3.2: The bounded case

Recall the Hoeffding (1948) decomposition

$$u_n(\varphi, h_2, \tilde{\mathbf{t}}) = \sqrt{n} \sum_{\ell=1}^{k} \binom{k}{\ell} U_n^{(\ell)} \left( \pi_{\ell} G_{\varphi, h_2, \tilde{\mathbf{t}}} (\cdot, \cdot) \right) \quad (9.52)$$

where the $\ell$ th Hoeffding projection for a (symmetric) function $H: S^\ell \times S^\ell \to \mathbb{R}$ is defined for $\mathbf{x}_\ell = (x_1, \dots, x_\ell) \in S^\ell$ and $\mathbf{y}_\ell = (y_1, \dots, y_\ell) \in S^\ell$ as

$$\pi_{\ell} H(\mathbf{x}_{\ell}, \mathbf{y}_{\ell}) := (\delta_{(\mathbf{x}_{\ell}, \mathbf{y}_{\ell})} - \mathbb{P}) \times \dots \times (\delta_{(\mathbf{x}_{\ell}, \mathbf{y}_{\ell})} - \mathbb{P}) \times \mathbb{P}^{k-\ell}(H), \quad (9.53)$$
where $\mathbb{P}$ is any probability measure on $(S, \mathcal{S})$, dropped whenever no confusion is possible). This decomposition follows easily by expanding

$$f(\mathbf{x}_1, \dots, \mathbf{x}_k) = \delta_{\mathbf{x}_1} \times \dots \times \delta_{\mathbf{x}_k} f = ((\delta_{\mathbf{x}_1} - \mathbb{P}) + \mathbb{P}) \times \dots \times ((\delta_{\mathbf{x}_k} - \mathbb{P}) + \mathbb{P}) f$$

into terms of the form

$$(\delta_{\mathbf{x}_{l_1}} - \mathbb{P}) \times \dots \times (\delta_{\mathbf{x}_{l_k}} - \mathbb{P}) \times \mathbb{P}^{k-l} f.$$

It is very simple to check that $f(\cdot)$ symmetric is $\mathbb{P}$-degenerate of order $r-1^2$ iff $r = \min\{\ell > 0 : \pi_\ell f \neq 0\}$. Therefore, $f(\cdot)$ is degenerate of order $r-1 \ge 0$ iff its Hoeffding expansion, except for the constant term, starts at term $r$, that is,

$$U_n(f) - \mathbb{P}^m f = \sum_{k=r}^{m} \binom{m}{k} U_n^{(k)}(\pi_k f).$$

Considering $(\mathbf{X}_i, \mathbf{Y}_i), i \ge 1$, i.i.d.- $\mathbb{P}$ and assuming $L(\cdot)$ is in $L_2(\mathbb{P}^k)$, this is an orthogonal decomposition and

$$\mathbb{E}[\pi_{\ell} L(\mathbf{X}_{\ell}, \mathbf{Y}_{\ell}) | (\mathbf{X}_2, \mathbf{Y}_2), \dots, (\mathbf{X}_{\ell}, \mathbf{Y}_{\ell})] = 0, \ell \ge 1,$$

where we denote $\mathbf{X}_\ell$ and $\mathbf{Y}_\ell$ for $(\mathbf{X}_1, \dots, \mathbf{X}_\ell)$ and $(\mathbf{Y}_1, \dots, \mathbf{Y}_\ell)$, respectively. Thus, the kernels $\pi_\ell L$ are canonical for $\mathbb{P}$ (or completely degenerate, or completely centered). Also, $\pi_\ell, \ell \ge 1$, are nested projections, that is, $\pi_\ell \circ \pi_\ell = \pi_\ell$ if $\ell \le l$, and

$$\mathbb{E}[(\pi_{\ell} H)^2(\mathbf{X}_{\ell}, \mathbf{Y}_{\ell})] \le \mathbb{E}[(H - \mathbb{E}H)^2(\mathbf{X}, \mathbf{Y})] \le \mathbb{E}H^2(\mathbf{X}, \mathbf{Y}). \quad (9.54)$$

On the other hand, since we assume $\mathcal{F}$ to be of VC-type with envelope function $F(\cdot)$ and $\mathcal{H}$ to be of VC-type with envelope $\kappa$, it is readily checked (via Lemma A.1 in Einmahl and Mason (2000)) that the class of functions on $\mathbb{R}^{pk} \times \mathbb{R}^{kq}$ given by

$$\mathcal{G} := \left\{ h_2^{pk} G_{\varphi, h_2, \tilde{\mathbf{t}}}(\cdot, \cdot) : \varphi \in \mathcal{F}, 0 < h_2 < 1, \tilde{\mathbf{t}} \in \mathbb{R}^{pk} \right\} \quad (9.55)$$

is of VC-type, for which we denote the VC-type characteristics by $A_1$ and $v_1$, and the envelope function by

$$\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) \equiv \tilde{\mathbf{F}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) = \kappa^k \|\alpha\|_{\infty}^{kp} \sum_{\sigma \in I_k^k} F(\tilde{\mathbf{y}}_{\sigma}), \quad \tilde{\mathbf{y}} \in \mathbb{R}^{kq}. \quad (9.56)$$

Next, for $\ell = 1, \dots, k$, introduce the following classes of functions on $\mathbb{R}^{pk} \times \mathbb{R}^{kq}$:

$$\mathcal{G}^{(\ell)} := \left\{ h^{kp} \pi_{\ell} G_{\varphi, h_2, \tilde{\mathbf{t}}}(\cdot, \cdot) : \varphi \in \mathcal{F}, 0 < h_2 < 1, \tilde{\mathbf{t}} \in \mathbb{R}^{pk} \right\}. \quad (9.57)$$

According to Giné and Mason (2007a), each class $\mathcal{G}^{(k)}$ is of VC-type with characteristics $A_1$ and $v_1$ and envelope function

$$\mathbf{F}_{\ell} \le 2^{\ell} \|\tilde{\mathbf{F}}\|_{\infty}. \quad (9.58)$$

We start by treating the first term of (9.52), namely, the linear term,

$$k\sqrt{n}U_n^{(1)}(\pi_1 G_{\varphi, h_2, \tilde{\mathbf{t}}}(\cdot, \cdot)) = \frac{k}{\sqrt{n}} \sum_{i=1}^{n} \pi_1 G_{\varphi, h_2, \tilde{\mathbf{t}}}(\mathbf{X}_i, \mathbf{Y}_i).$$

### Linear term of (9.52).

According to the definition of the Hoeffding projections and remembering that the sample $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$ is i.i.d, for all $(\mathbf{x}, \mathbf{y}) \in \mathbb{R}^2$

$$\pi_1 G_{\varphi, h_2, \tilde{\mathbf{t}}}(\mathbf{x}, \mathbf{y}) = \mathbb{E} \left[ G_{\varphi, h_2, \tilde{\mathbf{t}}} \left( (\mathbf{x}, \mathbf{X}_2, \dots, \mathbf{X}_k), (\mathbf{y}, \mathbf{Y}_2, \dots, \mathbf{Y}_k) \right) \right] - \mathbb{E} G_{\varphi, h_2, \tilde{\mathbf{t}}}(\mathbf{X}, \mathbf{Y})$$

2

**Definition 9.7.1.** A $\mathbb{P}^k$-integrable symmetric function of $k$ variables, $f: S^k \mapsto \mathbb{R}$, is $\mathbb{P}$-degenerate of order $r-1, 1 < r \le k$, if $\int f(\mathbf{x}_1, \dots, \mathbf{x}_k) d\mathbb{P}^{k-r+1}(\mathbf{x}_r, \dots, \mathbf{x}_k) = \int f d\mathbb{P}^k$ for all $\mathbf{x}_1, \dots, \mathbf{x}_{r-1} \in S$ whereas

$$\int f(\mathbf{x}_1, \dots, \mathbf{x}_k) d\mathbb{P}^{m-r}(\mathbf{x}_{r+1}, \dots, \mathbf{x}_k)$$

is not a constant function. If $f$ is $\mathbb{P}^k$-centered and is $\mathbb{P}$-degenerate of order $k-1$, that is, if

$$\int f(\mathbf{x}_1, \dots, \mathbf{x}_k) d\mathbb{P}(\mathbf{x}_1) = 0 \quad \text{for all } \mathbf{x}_2, \dots, \mathbf{x}_k \in S,$$

then $f(\cdot)$ is said to be canonical or completely degenerate with respect to $\mathbb{P}$. If $f(\cdot)$ is not degenerate of any positive order, we say it is nondegenerate or degenerate of order zero.
$$ = \mathbb{E} [G_{\varphi,h_2,\tilde{\mathbf{t}}}(\mathbf{X}, \mathbf{Y}) | (\mathbf{X}_1, \mathbf{Y}_1) = (\mathbf{x}, \mathbf{y})] - \mathbb{E} G_{\varphi,h_2,\tilde{\mathbf{t}}}(\mathbf{X}, \mathbf{Y}). $$

We need to introduce the following function on $\mathbb{R} \times \mathbb{R}$

$$ S_{\varphi,h_2,\tilde{\mathbf{t}}} : \mathbb{R} \to \mathbb{R} \\ (\mathbf{x}, \mathbf{y}) \mapsto kh_2^{pk} \mathbb{E} [G_{\varphi,h_2,\tilde{\mathbf{t}}}(\mathbf{X}, \mathbf{Y}) | (\mathbf{X}_1, \mathbf{Y}_1) = (\mathbf{x}, \mathbf{y})]. $$

We write, by utilizing this last notation,

$$ kh_2^{pk} \pi_1 G_{\varphi,h_n,\tilde{\mathbf{t}}} = S_{\varphi,h_2,\tilde{\mathbf{t}}} - \mathbb{E} S_{\varphi,h_2,\tilde{\mathbf{t}}}, $$

and hence for all $\varphi \in \mathcal{F}, h_2 \in [\tilde{a}_n, \tilde{b}_0]$ and $\tilde{\mathbf{t}} \in \mathbf{I}^k$, we can see that the linear term of the decomposition in (9.52) times $h_2^{pk}$ is given by

$$ kh_2^{pk} \sqrt{n} U_n^{(1)} (\pi_1 G_{\varphi,h_2,\tilde{\mathbf{t}}}) = \frac{1}{\sqrt{n}} \sum_{i=1}^{n} \{ S_{\varphi,h_2,\tilde{\mathbf{t}}} (\mathbf{X}_i, \mathbf{Y}_i) - \mathbb{E} S_{\varphi,h_2,\tilde{\mathbf{t}}} (\mathbf{X}_i, \mathbf{Y}_i) \} \\ =: \alpha_n (S_{\varphi,h_2,\tilde{\mathbf{t}}}), $$

where this last expression is an empirical process $\alpha_n$ based on the sample $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$ and indexed by the class of functions on $\mathbb{R}^p \times \mathbb{R}^q$,

$$ \mathcal{S}_n = \{ S_{\varphi,h_2,\tilde{\mathbf{t}}} (\cdot, \cdot) : \varphi \in \mathcal{F}, \tilde{a}_n \le h_2 \le \tilde{b}_0, \tilde{\mathbf{t}} \in \mathbf{I}^k \} $$

Clearly, $\mathcal{S}_n \subset k\mathcal{G}^{(1)}$ and the class $k\mathcal{G}^{(1)}$ has envelope function $k\mathbf{F}_1$, where $\mathbf{F}_1$ is the envelope function of the class $\mathcal{G}^{(1)}$ defined in (9.57). This class is of VC-type with the same characteristics as $\mathcal{G}$.

Hence, the proof of this part is similar to that of the Theorem 3.1.1 and then, for some $0 < \tilde{\mathcal{C}} < \infty$, we get

$$ \limsup_{n \to \infty} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{k \sqrt{nh_2^{pk}} |U_n^{(1)}(\pi_1 G_{\varphi,h_2,\tilde{\mathbf{t}}})|}{\sqrt{|\log h_2| \vee \log \log n}} \le \tilde{\mathcal{C}} \quad \text{a.s.} \quad (9.59) $$

### The other terms of (9.52).

Now, we must move on to the nonlinear part of the Hoeffding decomposition. Accordingly, the goal is to prove, for $2 \le l \le k$, that:

$$ \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\left( \frac{k}{l} \right) \sqrt{nh_2^{pk}} |U_n^{(l)}(\pi_l G_{\varphi,h_2,\tilde{\mathbf{t}}})|}{\sqrt{|\log h_2| \vee \log \log n}} = O(1) \quad \text{a.s.} \quad (9.60) $$

Since $n\tilde{a}_n^{kp} = c^{kp} \log n$, this will be accomplished if we can prove that for each $l=2, \dots, k$,

$$ \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |U_n^{(l)}(\pi_l G_{\varphi,h_2,\tilde{\mathbf{t}}})|}{\sqrt{(|\log h_2| \vee \log \log n)^l}} = O\left(\frac{1}{\sqrt{\tilde{a}_n^{kp} n^{l-1}}}\right) \quad \text{a.s.} \quad (9.61) $$

In order to achieve our result, we shall need a blocking argument and a decomposition of the interval $[\tilde{a}_n, \tilde{b}_0]$ into smaller intervals. For this, set $n_r = 2^r, r \ge 0$ and consider the intervals $\mathcal{H}_{2,r,j} := [h_{2,r,j-1}, h_{2,r,j}]$, where the boundaries are given by $h_{2,r,j}^{kp} := 2^j \tilde{a}_n^{kp}$. Set

$$ E(r) = \max \{ j : h_{2,r,j} \le 2\tilde{b}_0 \}, $$

remark that

$$ [\tilde{a}_{n_r}, \tilde{b}_0] \subseteq \bigcup_{r=1}^{E(r)} \mathcal{H}_{2,r,j} \quad \text{and} \quad E(r) \sim \log \left( \frac{n_r \tilde{b}_0}{c \log n_r} \right) / \log 2, \quad (9.62) $$

this implies that $E(r) \le 2 \log n_r$. Then, for $1 \le j \le E(r)$, consider the class of functions on $\mathbb{R}^{pk} \times \mathbb{R}^{kq}$,

$$ \mathcal{G}_{r,j} := \{ h^{kp} G_{\varphi,h_2,\tilde{\mathbf{t}}} (\cdot, \cdot) : \varphi \in \mathcal{F}, h_2 \in \mathcal{H}_{2,r,j}, \tilde{\mathbf{t}} \in \mathbf{I}^k \}, $$
as well as the class on $\mathbb{R}^{pk} \times \mathbb{R}^{kq}$,

$$ \mathcal{G}_{r,j}^{(\ell)} := \left\{ \frac{h_2^{pk} \pi_{\ell} G_{\varphi,h_2,\tilde{\mathbf{t}}}(\cdot, \cdot)}{M_{\ell}} : \varphi \in \mathcal{F}, h_2 \in \mathcal{H}_{2,r,j}, \tilde{\mathbf{t}} \in \mathbf{I}^k \right\}, $$

where $M_\ell = 2^\ell \|\alpha\|_\infty^{kp} \kappa^k M$. Absolutely, each class $\mathcal{G}_{r,j}$ is of VC-type with the same characteristics and envelope function as $\mathcal{G}$ and $\mathcal{G}_{r,j}^{(\ell)}$ is of VC-type with the same characteristics as $\mathcal{G}^{(\ell)}$ (and thus as $\mathcal{G}$) with envelope function $M_\ell^{-1} \mathbf{F}_\ell$, where $\mathbf{F}_\ell$ is the envelope function of $\mathcal{G}^{(\ell)}$. Notice that from (F.iv) and (9.58),

$$ \sup_{(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) \in \mathbb{R}^{pk} \times \mathbb{R}^{kq}} \left\{ \left| \pi_{\ell} G_{\varphi,h_2,\tilde{\mathbf{t}}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) \right| : \varphi \in \mathcal{F}, 0 < h_2 < 1, \tilde{\mathbf{t}} \in \mathbf{I}^k \right\} \le M_\ell $$

and hence each function in $\mathcal{G}_{r,j}^{(\ell)}$ is bounded by 1. Define now for $n_{r-1} < n \le n_r, r = 1, 2, \dots$,

$$ \mathcal{Y}_n(j, \ell, r) = n_r^{-\ell/2} \sup_{H \in \mathcal{G}_{r,j}^{(\ell)}} \left| \sum_{\mathbf{i} \in \mathcal{I}_n^k} H(\mathbf{X}_{\mathbf{i}}, \mathbf{Y}_{\mathbf{i}}) \right|. \qquad (9.63) $$

Due to the Theorem 10.0.7 in Appendix, we have for $c = 1/2, r' = 2$ and all $\mathbf{x} > 0$ that for any $r \ge 1$,

$$ \mathbb{P} \left\{ \max_{n_{r-1} < n \le n_r} \mathcal{Y}_n(j, \ell, r) > \mathbf{x} \right\} \le \frac{2}{\mathbf{x}} \mathbb{P} \left\{ \mathcal{Y}_{n_r}(j, \ell, r) > \mathbf{x}/2 \right\}^{1/2} \mathbb{E} \left[ \mathcal{Y}_{n_r}^2(j, \ell, r) \right]^{1/2}. \qquad (9.64) $$

We are going to apply an exponential inequality and a moment bound for U-statistics represented in de la Peña and Giné (1999) and Giné and Mason (2007b) on the class $\mathcal{G}_{r,j}^{(\ell)}$ to bound (9.64). In order to utilize the above results, we have first to derive some bounds. First of all, it is readily checked that

$$ \begin{aligned} \mathcal{Y}_n(j, \ell, r) &= n_r^{-\ell/2} \sup_{H \in \mathcal{G}_{r,j}^{(\ell)}} \left| \sum_{\mathbf{i} \in \mathcal{I}_n^k} H(\mathbf{X}_{\mathbf{i}}, \mathbf{Y}_{\mathbf{i}}) \right| \\ &= n_r^{-\ell/2} \left( \begin{array}{c} k \\ \ell \end{array} \right) \left( \begin{array}{c} k \\ \ell \end{array} \right)^{-1} \sup_{H \in \mathcal{G}_{r,j}^{(\ell)}} \left| \sum_{\mathbf{i} \in \mathcal{I}_n^k} H(\mathbf{X}_{\mathbf{i}}, \mathbf{Y}_{\mathbf{i}}) \right| \\ &= n_r^{-\ell/2} \left( \begin{array}{c} k \\ \ell \end{array} \right) \| U_n^{(\ell)}(\pi_{\ell} G_{\varphi,h_2,\tilde{\mathbf{t}}}) \|_{\mathcal{G}_{r,j}^{(\ell)}} \\ &\le n_r^{-\ell/2} \| U_n^{(\ell)}(\pi_{\ell} G_{\varphi,h_2,\tilde{\mathbf{t}}}) \|_{\mathcal{G}_{r,j}^{(\ell)}}, \end{aligned} \qquad (9.65) $$

for all $n_{r-1} < n \le n_r$. Next, notice that in Assumptions (K.i) and (K.ii), the kernel $K(\cdot)$ is assumed to be bounded by $\kappa$ and have support in $[-1/2, 1/2]$, so, making use assumption (F.iv) and $M_\ell = 2^\ell \kappa^k \|\alpha\|_\infty^{kp} M$, for $H \in \mathcal{G}_{r,j}^{(\ell)}$, we obtain by (9.54)

$$ \begin{aligned} \mathbb{E}H^2(\mathbf{X}, \mathbf{Y}) &\le M_\ell^{-2} h_2^{2kp} \mathbb{E}G_{\varphi,h_2,\tilde{\mathbf{t}}}^2(\mathbf{X}, \mathbf{Y}) \\ &= M_\ell^{-2} \mathbb{E} \left[ \varphi^2(\mathbf{Y}) \tilde{K}^2 \left( \frac{\tilde{\mathbf{t}} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \boldsymbol{\alpha}^2(\mathbf{X}) \right] \\ &\le h_2^{pk} 4^{-\ell} \|f\|_{\infty}^k. \end{aligned} \qquad (9.66) $$

For $B_{k,\ell} = 4^{-\ell} \|f\|_{\infty}^k$, this gives that

$$ \sup_{H \in \mathcal{G}_{r,j}^{(\ell)}} \mathbb{E}H^2(\mathbf{X}, \mathbf{Y}) \le B_{k,\ell} h_{2,r,j}^{pk} := \sigma_{2,r,j}^2. \qquad (9.66) $$

Since $\pi_\ell \pi_\ell H = \pi_\ell H$ for all $\ell \ge 1$, so, by applying Theorem 10.0.8 to the class $\mathcal{G}_{r,j}^{(\ell)}$ with $\sigma_{2,r,j}^2$ as in (9.66), we obtain easily, for some constant $A_\ell$, that

$$ \mathbb{E} \mathcal{Y}_{n_r}^2(j, \ell, r) \le n_r^\ell \mathbb{E} \| U_{n_r}^{(\ell)}(\pi_\ell H) \|_{\mathcal{G}_{r,j}^{(\ell)}}^2 \le 2^\ell A_\ell h_{2,\ell,j}^{pk} |\log(h_{2,r,j})|^\ell. $$
To control the probability term in (9.64), we shall apply an exponential inequality to the same class $\mathcal{G}_{r,j}^{(\ell)}$, remember that each $H \in \mathcal{G}_{r,j}^{(\ell)}$ is bounded by 1. For this, set

$$w^* = \mathcal{D}_{1,\ell} (|\log h_{2,r,j}| \lor \log \log n_r)^{\ell/2} =: \mathcal{D}_{1,\ell} \xi_{j,\ell}(r), \qquad (9.67)$$

where $\mathcal{D}_{1,\ell} < \infty$. According to the Theorem 5.3.14 of de la Peña and Giné (1999), we obtain constants $\mathcal{D}_{2,\ell}$, $\mathcal{D}_{3,\ell}$ and $\mathcal{D}_{4,\ell}$ such that, for $j = 1, \dots, E(r)$ and for any $\beta > 1$,

$$ \begin{aligned} \mathbb{P}\{\mathcal{Y}_{n_r}(j, \ell, r) > \beta^{\ell/2} w^*\} &\le \mathcal{D}_{2,\ell} \exp\{-\mathcal{D}_{3,\ell} \beta w^{*2/\ell}\} \\ &\le \exp\{-\mathcal{D}_{4,\ell} \beta \log \log n_r\}. \end{aligned} \qquad (9.68) $$

By plugging the bounds (9.67) and (9.68) into (9.64), we get, for some $\mathcal{D}_{5,\ell} > 0$, and $\beta \ge 2$ and $r$ large enough, that

$$ \begin{aligned} \mathbb{P}\left\{\max_{n_{r-1}<n \le n_r} \mathcal{Y}_n(j, \ell, r) > 2\beta^{\ell/2} w^*\right\} &\le \frac{(\log n_r)^{-\beta \mathcal{D}_{4,\ell}/2} \sqrt{2^\ell A_\ell h_{2,r,j}^{pk} |\log(h_{2,r,j})|^\ell}}{\mathcal{D}_{1,\ell} \sqrt{\beta^\ell (|\log h_{2,r,j}| \lor \log \log n_r)^\ell}} \\ &\le \sqrt{h_{2,r,j}^{pk} \log n_r}^{-\beta \mathcal{D}_{5,\ell}}. \end{aligned} \qquad (9.69) $$

Finally, we note also

$$ \begin{aligned} n_r^{\ell/2} \|U_n^{(\ell)}(\pi_\ell G)\|_{\mathcal{G}_{r,j}} &= n_r^\ell \binom{k}{\ell}^{-1} n_r^{-\ell/2} \sup_{H \in \mathcal{G}_{r,j}^{(\ell)}} \left| \sum_{\mathbf{i} \in \mathcal{I}_n^k} H(\mathbf{X}_i, \mathbf{Y}_i) \right| \\ &= n_r^\ell \binom{k}{\ell}^{-1} \mathcal{Y}_n(j, \ell, r) \\ &\le \ell! \prod_{j=0}^{\ell-1} \left(\frac{n_r}{n-j}\right) \mathcal{Y}_n(j, \ell, r) \\ &\le \mathcal{D}_\ell \mathbb{M}_\ell \mathcal{Y}_n(j, \ell, r), \end{aligned} \qquad (9.70) $$

for some $\mathcal{D}_\ell > 0$. Therefore, by (9.62), for each $\ell = 2, \dots, k$ and $r$ large enough, we have

$$ \begin{aligned} \max_{n_{r-1}<n \le n_r} A_{n,\ell} &:= \max_{n_{r-1}<n \le n_r} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\phi \in \mathcal{F}} \sup_{\tilde{\mathbf{I}} \in \mathcal{I}^k} \frac{\sqrt{nh_2^{pk}} |U_n^{(\ell)}(\pi_\ell G_{\phi,h_2,\tilde{\mathbf{I}}})|}{\sqrt{(|\log h_2| \lor \log \log n)^\ell}} \\ &\le \max_{n_{r-1}<n \le n_r} \max_{1 \le j \le E(r)} \sup_{h_2 \in \mathcal{H}_{2,r,j}} \sup_{\phi \in \mathcal{F}} \sup_{\tilde{\mathbf{I}} \in \mathcal{I}^k} \frac{\sqrt{n_r h_2^{pk}} |U_n^{(\ell)}(\pi_\ell G_{\phi,h_2,\tilde{\mathbf{I}}})|}{\sqrt{(|\log h_2| \lor \log \log n_r)^\ell}} \\ &\le \frac{\mathcal{D}_\ell \mathbb{M}_\ell}{\sqrt{n_r^{\ell-1}}} \max_{n_{r-1}<n \le n_r} \max_{1 \le j \le E(r)} \frac{\mathcal{Y}_n(j, \ell, r)}{\xi_{j,\ell}(r)} \\ &\le \frac{\mathcal{D}_\ell \mathbb{M}_\ell}{\sqrt{\tilde{a}_{n_r}^{pk} n_r^{\ell-1}}} \max_{n_{r-1}<n \le n_r} \max_{1 \le j \le E(r)} \frac{\mathcal{Y}_n(j, \ell, r)}{\xi_{j,\ell}(r)}, \end{aligned} $$

where $\xi_{j,\ell}(r)$ was represented in (9.67). Now, remember that $h_{2,r,j} \le 2\tilde{b}_0 < 2$ for $j = 1, \dots, E(r)$ and that $E(r) \le 2\log n_r$. Then, by applying (9.69) with $\beta \ge (2+\delta)/\mathcal{D}_{5,\ell}$, $\delta > 0$ and in combination with the above inequality and the obvious bound

$$\sqrt{\tilde{a}_n^{pk} n^{\ell-1}} A_{n,\ell} \le \sqrt{\tilde{a}_{n_r}^{pk} n_r^{\ell-1}} A_{n,\ell}$$

valid for all $n_{r-1} < n \le n_r$, implies for $\mathcal{D}_{6,\ell} \ge 2\beta^{\ell/2} \mathcal{D}_\ell \mathbb{M}_\ell \mathcal{D}_{1,\ell}$ that for $\ell = 2, \dots, k$,

$$ \begin{aligned} \mathbb{P}\left\{\max_{n_{r-1}<n \le n_r} \sqrt{\tilde{a}_n^{pk} n^{\ell-1}} A_{n,\ell} > \mathcal{D}_{6,\ell}\right\} &\le \sum_{j=1}^{E(r)} \sqrt{h_{2,r,j}^{pk}} (\log n_r)^{-\beta \mathcal{D}_{5,\ell}} \\ &\le E(r) \sqrt{2^k} (\log n_r)^{-\beta \mathcal{D}_{5,\ell}} \\ &\le \sqrt{2^{k+2}} (r \log 2)^{-(1+\delta)}. \end{aligned} $$

This demonstrates, through some elementary bounds and an application of the Borel-Cantelli lemma, that (9.61) holds, which implies (9.60), and this completes the proof of Theorem 3.3.2. $\square$
## 9.8 Proof of Theorem 3.3.3: The unbounded case

We must truncate the conditional U-statistic to demonstrate this theorem. So, taking advantage of the Condition (F.v), we have, remembering that $n_r = 2^r, r \ge 0$, for each $\lambda > 0$ and $\gamma_r = n_r/\log n_r$,

$$ 
\begin{aligned}
G_{\varphi,h_2,\tilde{\mathbf{t}}} (\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) &= G_{\varphi,h_2,\tilde{\mathbf{t}}} (\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) \mathbb{1}_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) \le \lambda \gamma_r^{1/v}\}} + G_{\varphi,h_2,\tilde{\mathbf{t}}} (\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) \mathbb{1}_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) > \lambda \gamma_r^{1/v}\}} \\
&=: G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)} (\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) + G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)} (\tilde{\mathbf{x}}, \tilde{\mathbf{y}}),
\end{aligned}
$$

which means that we truncate each function $\varphi \in \mathcal{F}$, either the envelope function as follows :

$$ 
\begin{aligned}
\varphi(\tilde{\mathbf{y}}) &= \varphi(\tilde{\mathbf{y}}) \mathbb{1}_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) \le \lambda \gamma_r^{1/v}\}} + \varphi(\tilde{\mathbf{y}}) \mathbb{1}_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) > \lambda \gamma_r^{1/v}\}} \\
&= \varphi^{(T)}(\tilde{\mathbf{y}}) + \varphi^{(R)}(\tilde{\mathbf{y}}).
\end{aligned}
$$

We see also, from (3.21), $u_n(\varphi, h_2, \tilde{\mathbf{t}})$ can be decomposed as follows,

$$ 
\begin{aligned}
u_n(\varphi, h_2, \tilde{\mathbf{t}}) &= \sqrt{n} \left\{ U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)}) - \mathbb{E}U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)}) \right\} \\
&\quad + \sqrt{n} \left\{ U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)}) - \mathbb{E}U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)}) \right\} \\
&=: u_n^{(T)}(\varphi, h_2, \tilde{\mathbf{t}}) + u_n^{(R)}(\varphi, h_2, \tilde{\mathbf{t}}).
\end{aligned}
$$

We call the linear term of the right side $u_n^{(T)}(\varphi, h_2, \tilde{\mathbf{t}})$ truncated part and the second one $u_n^{(R)}(\varphi, h_2, \tilde{\mathbf{t}})$ remainder part. Let's begin with $u_n^{(T)}(\varphi, h_2, \tilde{\mathbf{t}})$. First of all, we consider bandwidths lying, as before, in a slightly smaller interval $\mathcal{H}_{2,n_r}' = [a_{n_r}', b_0]$ which can be composed into the sub-intervals

$$ \mathcal{H}_{2,r,j}' := [h_{2,r,j}' - 1, h_{2,r,j}'] \quad \text{with} \quad h_{2,r,j}' := 2^j \tilde{a}_{n_r}'^{pk}. \qquad (9.71) $$

We still also have, in particular, $E(r) \le 2 \log n_r$ such that $E(r)$ is defined, now, as

$$ E(r) := \max\{j : h_{2,r,j}' \le 2\tilde{b}_0\}. $$

### Truncated Part:

As we saw before, we need to decompose our U-process $u_n^{(T)}(\varphi, h_2, \tilde{\mathbf{t}})$ according to Hoeffding's decomposition, so we get

$$ 
\begin{aligned}
u_n^{(T)}(\varphi, h_2, \tilde{\mathbf{t}}) &= \sqrt{n} \sum_{\ell=1}^{k} \binom{k}{\ell} U_n^{(\ell)} \left( \pi_{\ell} G_{\varphi, h_2, \tilde{\mathbf{t}}}^{(T)} (\cdot, \cdot) \right) \\
&= k \sqrt{n} U_n^{(1)} \left( \pi_1 G_{\varphi, h_2, \tilde{\mathbf{t}}}^{(T)} \right) + \sqrt{n} \sum_{\ell=2}^{k} \binom{k}{\ell} U_n^{(\ell)} \left( \pi_{\ell} G_{\varphi, h_2, \tilde{\mathbf{t}}}^{(T)} \right).
\end{aligned}
\qquad (9.72) $$

Let us start with the linear term in this decomposition.

### Linear term of (9.72).

We can see, by following the same technique as in the previous case ($m=1$), that $\pi_1 G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)}$ is a centered conditional expectation and that the first term of (9.72) can be written as an empirical process based on the sample $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$ and indexed by the class of functions

$$ \mathcal{T}_r := \left\{ H_{\varphi,h_2,\tilde{\mathbf{t}}}^{(r)}(\cdot, \cdot) : \varphi \in \mathcal{F}, h_2 \in \mathcal{H}_{2,n_r}', \tilde{\mathbf{t}} \in \mathbf{I}^k \right\}, $$

where

$$ H_{\varphi,h_2,\tilde{\mathbf{t}}}^{(r)}(\mathbf{x}, \mathbf{y}) = k h_2^{pk} \mathbb{E} \left[ G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)}(\mathbf{X}, \mathbf{Y}) \mid (\mathbf{X}_1, \mathbf{Y}_1) = (\mathbf{x}, \mathbf{y}) \right] $$

and $\mathcal{H}_{2,n_r}'$ was defined in (9.71). Now, introduce the following class of functions of $(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) \in \mathbb{R}^{pk} \times \mathbb{R}^{kq}$

$$ \tilde{\mathcal{F}} := \left\{ h_2^{pk} G_{\varphi,h_2,\tilde{\mathbf{t}}} (\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) \mathbb{1}_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) \le c\}}, \quad \text{for } 0 \le h_2 \le 1 \text{ and } \varphi \in \mathcal{F} \right\}. \qquad (9.73) $$

This class is a VC-type class of functions (since both $\mathcal{F}'$, defined as

$$ \mathcal{F}' := \left\{ h_2^{pk} G_{\varphi,h_2,\tilde{\mathbf{t}}} (\cdot, \cdot) \quad \text{for } 0 \le h_2 \le 1 \text{ and } \varphi \in \mathcal{F} \right\}, $$
and the class of functions given by

$$ \mathcal{F} = \{1_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) \le c\}} : c > 0\} $$

are of VC-type, according to Lemma A.1 in Einmahl and Mason (2000), with some characteristics. Therefore, so is the class of functions in the class of functions $k\tilde{\mathcal{F}}^{(1)}$ on $\mathbb{R}^2$, where $\tilde{\mathcal{F}}^{(1)}$ consists of the $\pi_1$-projections of the functions in the class $\tilde{\mathcal{F}}$. Thus, we see that $\mathcal{F}_r \subset k\tilde{\mathcal{F}}^{(1)}$ and hence $\mathcal{F}_r$ is of VC-type with the same characteristics as $k\tilde{\mathcal{F}}^{(1)}$. Now, we shall find an envelope function for $\mathcal{F}_r$, set $\tilde{\mathbf{t}}_j := (\mathbf{t}_1, \dots, \mathbf{t}_{j-1}, \mathbf{t}_{j+1}, \dots, \mathbf{t}_k) \in \mathbb{R}^{p(k-1)}$ and $\mathbb{Z}_j(\mathbf{u}) := (\mathbf{Z}_1, \dots, \mathbf{Z}_{j-1}, \mathbf{u}, \mathbf{Z}_{j+1}, \dots, \mathbf{Z}_k) \in \mathbb{R}^{kq}$ for $\mathbf{u} \in \mathbb{R}$, and $\mathbb{Z} \in \mathbb{R}^{kq}$. We can then rewrite the function $H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) \in \mathcal{F}_r$ as

$$ H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)}(\mathbf{x}, \mathbf{y}) = K\left(\frac{\mathbf{t}_1 - \mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \mathbb{E}\left[\varphi(\mathbf{Y}_1(\mathbf{y})) \tilde{K}\left(\frac{\tilde{\mathbf{t}}_1 - \mathbb{X}^*}{h_2}\alpha(f(\mathbb{X}^*))\right) 1_{\{\tilde{\mathbf{F}}(\mathbf{Y}_1(\mathbf{y})) \le \lambda \gamma_r^{1/\nu}\}} \boldsymbol{\alpha}(\mathbb{X})\right] \\ + K\left(\frac{\mathbf{t}_2 - \mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \mathbb{E}\left[\varphi(\mathbf{Y}_2(\mathbf{y})) \tilde{K}\left(\frac{\tilde{\mathbf{t}}_2 - \mathbb{X}^*}{h_2}\alpha(f(\mathbb{X}^*))\right) 1_{\{\tilde{\mathbf{F}}(\mathbf{Y}_2(\mathbf{y})) \le \lambda \gamma_r^{1/\nu}\}} \boldsymbol{\alpha}(\mathbb{X})\right] \\ + \dots + K\left(\frac{\mathbf{t}_k - \mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \mathbb{E}\left[\varphi(\mathbf{Y}_k(\mathbf{y})) \tilde{K}\left(\frac{\tilde{\mathbf{t}}_k - \mathbb{X}^*}{h_2}\alpha(f(\mathbb{X}^*))\right) 1_{\{\tilde{\mathbf{F}}(\mathbf{Y}_k(\mathbf{y})) \le \lambda \gamma_r^{1/\nu}\}} \boldsymbol{\alpha}(\mathbb{X})\right], $$

where $\mathbb{X}^* = (\mathbf{X}_2, \dots, \mathbf{X}_k) \in \mathbb{R}^{p(k-1)}$ and where the product kernel is now defined for $p(k-1)$-dimensional vectors, that is,

$$ \tilde{K}(\mathbf{u}) = \prod_{i=1}^{k-1} K(\mathbf{u}_i), \quad \mathbf{u} \in \mathbb{R}^{p(k-1)}. $$

Accordingly, we can bound $H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)} \in \mathcal{F}_r$ as follows

$$ \begin{aligned} |H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)}(\mathbf{x}, \mathbf{y})| & \le K\left(\frac{\mathbf{t}_1 - \mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \mathbb{E}\left[F(\mathbf{y}, \mathbf{Y}_2, \dots, \mathbf{Y}_k) \tilde{K}\left(\frac{\tilde{\mathbf{t}}_1 - \mathbb{X}^*}{h_2}\alpha(f(\mathbb{X}^*))\right) \boldsymbol{\alpha}(\mathbb{X})\right] \\ & \quad + K\left(\frac{\mathbf{t}_2 - \mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \mathbb{E}\left[F(\mathbf{Y}_2, \mathbf{y}, \mathbf{Y}_3, \dots, \mathbf{Y}_k) \tilde{K}\left(\frac{\tilde{\mathbf{t}}_2 - \mathbb{X}^*}{h_2}\alpha(f(\mathbb{X}^*))\right) \boldsymbol{\alpha}(\mathbb{X})\right] \\ & \quad + \dots + K\left(\frac{\mathbf{t}_k - \mathbf{x}}{h_2}\alpha(f(\mathbf{x}))\right) \mathbb{E}\left[F(\mathbf{Y}_2, \dots, \mathbf{Y}_k, \mathbf{y}) \tilde{K}\left(\frac{\tilde{\mathbf{t}}_k - \mathbb{X}^*}{h_2}\alpha(f(\mathbb{X}^*))\right) \boldsymbol{\alpha}(\mathbb{X})\right] \\ & =: G_k(\mathbf{x}, \mathbf{y}). \end{aligned} $$

Now, we can apply the moment bound in Proposition 10.0.2 to the subclasses

$$ \mathcal{T}_{r,j} := \{H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)} : \varphi \in \mathcal{F}, h_2 \in \mathcal{H}_{2,r,j}', \tilde{\mathbf{t}} \in \mathbf{I}^k\}, \quad 1 \le j \le E(r), $$

Since $\mathcal{T}_{r,j} \subset \mathcal{T}_r$ for $j=1, \dots, E(r)$, all of these subclasses are of VC-type, with the same envelope function and characteristics as the class $k\tilde{\mathcal{F}}^{(1)}$ (which is independent of $r$), this is confirming (ii) in Theorem 10.0.2. To show (i), remember that although all terms of the envelope function $G_k(\mathbf{x}, \mathbf{y})$ differ, their expectations are the same. Thus, taking advantage of the condition (F.v) and an application of Minkowski's inequality followed by Jensen's inequality, we get the following upper bound for the second moment of the envelope function:

$$ \begin{aligned} \mathbb{E}G_k^2(\mathbf{X}, \mathbf{Y}) &= \kappa^{2k} \|\boldsymbol{\alpha}\|_{\infty}^{2kp} \mathbb{E}_{\mathbf{Y}}\{\mathbb{E}_{\mathbf{Y}^*}[F(\mathbf{Y}, \mathbf{Y}_2, \dots, \mathbf{Y}_k)] \\ & \quad + \mathbb{E}_{\mathbf{Y}^*}[F(\mathbf{Y}_2, \mathbf{Y}, \mathbf{Y}_3, \dots, \mathbf{Y}_k)] + \dots + \mathbb{E}_{\mathbf{Y}^*}[F(\mathbf{Y}_2, \dots, \mathbf{Y}_k, \mathbf{Y})]\}^2 \\ & \le k^2 \kappa^{2k} \|\boldsymbol{\alpha}\|_{\infty}^{2kp} \mathbb{E}F^2(\mathbf{Y}_1, \dots, \mathbf{Y}_k) \\ & \le k^2 \kappa^{2k} \|\boldsymbol{\alpha}\|_{\infty}^{2kp} \theta_v^{2/\nu}, \end{aligned} $$

where we use the notation $\mathbb{Y}^*$ for $(\mathbf{Y}_2, \dots, \mathbf{Y}_k)$. Remark, additionally, that by the symmetry of $\tilde{\mathbf{F}}$,

$$ \mathbb{E}G_{\phi,h_2,\tilde{\mathbf{t}}}^{(T)}(\mathbf{X}, \mathbf{Y}) = h_2^{-kp} \mathbb{E}\left[\varphi(\mathbf{Y}) \tilde{K}\left(\frac{\mathbf{t} - \mathbf{X}}{h_2}\alpha(f(\mathbf{X}))\right) 1_{\{\tilde{\mathbf{F}}(\mathbf{Y}) \le \lambda \gamma_r^{1/\nu}\}} \boldsymbol{\alpha}(\mathbf{X})\right], $$

by applying Jensen's inequality, utilizing the change of variable $s = (\mathbf{t} - \mathbf{x})/h_2$ and making use of the condition (F.v), we obtain the following upper bound for the second moment of any function in $\mathcal{F}_r$:

$$ \mathbb{E}(H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)}(\mathbf{X}, \mathbf{Y}))^2 \le k^2 \mathbb{E}\left[\varphi^2(\mathbf{Y}) \tilde{K}^2\left(\frac{\mathbf{t} - \mathbf{X}}{h_2}\alpha(f(\mathbf{X}))\right) 1_{\{\tilde{\mathbf{F}}(\mathbf{Y}) \le \lambda \gamma_r^{1/\nu}\}} \boldsymbol{\alpha}(\mathbf{X})\right] $$
$$
\begin{align}
&\le k^2 \mathbb{E} \left[ \tilde{K}^2 \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \mathbb{E} \left[ F^2(\mathbf{Y}) \mid \mathbf{X} = \mathbf{x} \right] \boldsymbol{\alpha}(\mathbf{X}) \right] \nonumber \\
&\le k^2 \|\boldsymbol{\alpha}\|_{\infty}^{2pk} \theta_v^{2/v} \mathbb{E} \left[ \tilde{K}^2 \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \right] \nonumber \\
&= k^2 \|\boldsymbol{\alpha}\|_{\infty}^{2pk} \theta_v^{2/v} \boldsymbol{\kappa}^{2k} \|f\|_{\infty}^k h_2^{pk}. \tag{9.74}
\end{align}
$$

In consequence, our preceding calculations, with $\beta = k\kappa^k \theta_v^{1/v} \|\alpha\|_{\infty}^{kp} (1 \vee \|f\|_{\infty}^{k/2})$, give us that

$$ \mathbb{E}G_k^2(\mathbf{X}, \mathbf{Y}) \le \beta^2 $$

and

$$ \sup_{H \in \mathcal{H}_{r,j}} \mathbb{E}H^2(\mathbf{X}, \mathbf{Y}) \le \beta^2 h_{2,r,j}^{kp} := \sigma_{r,j}^2, $$

validing condition (iii) as well. Finally, recall, from (9.56), that since $\mathcal{S}$ has envelope function $\tilde{F}(\tilde{\mathbf{y}})$, we deduce, for all $\mathbf{x}, \mathbf{y} \in \mathbb{R}$, that

$$
\begin{align*}
|H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)}(\mathbf{x}, \mathbf{y})| &\le k \mathbb{E} \left[ \tilde{F}(\mathbf{Y}) \mathbf{1}_{\{\tilde{F}(\mathbf{Y}) \le \lambda \gamma^{1/v}\}} \mid (\mathbf{X}_1, \mathbf{Y}_1) = (\mathbf{x}, \mathbf{y}) \right] \\
&\le k \lambda \gamma^{1/v},
\end{align*}
$$

so, the Theorem 10.0.2 is now applicable by taking $\lambda > 0$ small enough. Therefore, for an absolute constant $\mathcal{A}_1 < \infty$, we obtain

$$
\begin{align}
\mathbb{E} \left\| \sum_{i=1}^{n_r} \varepsilon_i H(\mathbf{X}_i, \mathbf{Y}_i) \right\|_{\mathcal{H}_{r,j}} &\le \mathcal{A}_1 \sqrt{n_r h_{2,r,j}^{pk} |\log h_{2,r,j}'|} \nonumber \\
&\le \mathcal{A}_1 \sqrt{n_r h_{2,r,j}^{pk} \left( |\log h_{2,r,j}'| \vee \log \log n_r \right)} \nonumber \\
&=: \mathcal{A}_1 \zeta_j(r), \tag{9.75}
\end{align}
$$

where $\varepsilon_1, \dots, \varepsilon_{n_r}$ are independent Rademacher variables that are independent of $(\mathbf{X}_i, \mathbf{Y}_i)$, $1 \le i \le n_r$. As a result, we obtain, for an absolute constant $\mathcal{A}_2 < \infty$ and all $\mathbf{u} > 0$, by applying the exponential inequality of Talagrand (1994) represented in Theorem 10.0.4 to the class $\mathcal{T}_{r,j}$ with $\mathbf{M} = k\lambda \gamma_r^{1/v}$, $\sigma_{\mathcal{T}_{r,j}}^2 = \beta^2 h_{2,r,j}^{pk}$ and the moment bound in (9.75), that

$$
\begin{align}
& \mathbb{P} \left\{ \max_{n_{r-1} < n \le n_r} \left\| \sqrt{n} \alpha_n \right\|_{\mathcal{T}_{r,j}} \ge \mathcal{C}_1 (\mathcal{A}_1 \zeta_j(r) + \mathbf{u}) \right\} \nonumber \\
& \le 2 \left[ \exp \left( - \frac{\mathcal{A}_2 \mathbf{u}^2}{n_r \beta^2 h_{2,r,j}^{pk}} \right) + \exp \left( - \frac{\mathcal{A}_2 \mathbf{u}}{k \lambda \gamma_r^{1/v}} \right) \right]. \tag{9.76}
\end{align}
$$

Concerning the application of this inequality with respect to $\mathbf{u} = \rho \zeta_j(r)$, $\rho > 1$, notice that, keeping in mind $\tilde{a}_n^{pk} = c^{pk}(\log n/n)^{1-2/v}$, it is evident, from the definitions of $h_{2,r,j}$, $\gamma_r$ and $\zeta_j(r)$, follows, for all $j \ge 0$,

$$ \frac{\zeta_j^2(r)}{n_r h_{2,r,j}^{pk}} = |\log h_{2,r,j}'| \vee \log \log n_r \ge \log \log n_r, $$

$$ \frac{\zeta_j^2(r)}{\gamma_r^{2/v}} = 2^j c^{kp} \log n_r (|\log h_{2,r,j}'| \vee \log \log n_r) \ge c^{kp} (\log \log n_r)^2. $$

Then, an application of (9.76) with $\mathbf{u} = \rho \zeta_j(r)$ and any $\rho > 1$ with $r$ large enough, give us, for suitable constants $\mathcal{A}'_2, \mathcal{A}''_2$ and $\mathcal{A}_3$, the inequality

$$
\begin{align}
& \mathbb{P} \left\{ \max_{n_{r-1} < n \le n_r} \left\| \sqrt{n} \alpha_n \right\|_{\mathcal{T}_{r,j}} \ge \mathcal{C}_1 (\mathcal{A}_1 + \rho) \zeta_j(r) \right\} \nonumber \\
& \le 2 \left[ \exp(-\mathcal{A}_2' \rho^2 \log \log n_r) + \exp(-\mathcal{A}_2'' \rho \log \log n_r) \right] \nonumber \\
& \le 4 (\log n_r)^{-\mathcal{A}_3 \rho}. \tag{9.77}
\end{align}
$$

Taking into consideration that $kh_2^{pk} \sqrt{n} U_n^{(1)}(\pi_1 G_{\phi,h_2,\tilde{\mathbf{t}}}^{(T)})$ is the empirical process $\alpha_n(H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)})$ indexed by the class $\mathcal{T}_r$ and remembering (9.62), we obtain, for $r \ge 1$, that

$$ \max_{n_{r-1} < n \le n_r} \mathcal{A}_{n,r}' := \max_{n_{r-1} < n \le n_r} \sup_{\tilde{a}_n' \le h_2 \le \tilde{b}_0} \sup_{\phi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{k \sqrt{n h_2^{pk}} |U_n^{(1)}(\pi_1 G_{\phi,h_2,\tilde{\mathbf{t}}}^{(T)})|}{\sqrt{|\log h_2| \vee \log \log n}} $$
$$
\begin{align*}
&\le \max_{n_{r-1} < n \le n_r} \max_{1 \le j \le E(r)} \sup_{h_2 \in \mathcal{H}_{r,j}'} \sup_{\phi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{2\sqrt{2^{kp}} |\sqrt{n}\alpha_n (H_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)})|}{\sqrt{n_r h_{2,r,j}^{pk}} (|\log h_{2,r,j}| \vee \log \log n_r)} \\
&\le \max_{n_{r-1} < n \le n_r} \max_{1 \le j \le E(r)} \sup_{H \in \mathcal{F}_{r,j}} \frac{3|\sqrt{n}\alpha_n(H)|}{\lambda_j(r)}.
\end{align*}
$$

Thus, by using $E(r) \le 2 \log n_r$ once again, we can deduce from (9.77) that, for some constant $\mathcal{C}_5(\rho) \ge 3\mathcal{C}_1(\mathcal{A}_1 + \rho)$,

$$
\begin{align*}
\mathbb{P} \left\{ \max_{n_{r-1} < n \le n_r} \mathcal{A}_{n,r}' > \mathcal{C}_5(\rho) \right\} &\le \sum_{j=1}^{E(r)} \mathbb{P} \left\{ \max_{n_{r-1} < n \le n_r} \| \sqrt{n} \alpha_n \|_{\mathcal{F}_{r,j}} > \mathcal{C}_1 (\mathcal{A}_1 + \rho) \zeta_j(r) \right\} \\
&\le 8 (\log n_r)^{1-\mathcal{A}_3 \rho}.
\end{align*}
$$

So, by combining this last inequality for $\rho \ge (2+\delta)/\mathcal{A}_3$, $\delta > 0$ with the choice $n_r = 2^r$ and the Borel-Cantelli lemma, it follows that, for some $\mathcal{C}' < \infty$ and with probability 1,

$$
\limsup_{r \to \infty} \max_{n_{r-1} < n \le n_r} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\phi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{k \sqrt{n h_2^{pk}} |U_n^{(1)}(\pi_1 G_{\phi,h_2,\tilde{\mathbf{t}}}^{(T)})|}{\sqrt{|\log h_2| \vee \log \log n}} \le \mathcal{C}', \quad (9.78)
$$

this achieves the first term in (9.72). Now, we turn to the nonlinear terms in (9.72).

**The other terms of (9.72).**

The goal is to show that, for $\ell = 2, \dots, k$ and with probability 1, all of the other terms of (9.72) are asymptotically bounded or go to zero at the proper rate, that is

$$
\max_{n_{r-1} < n \le n_r} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\phi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^{pk}} \frac{\sqrt{n h_2^{pk}} |U_n^{(\ell)}(\pi_\ell G_{\phi,h_2,\tilde{\mathbf{t}}}^{(T)})|}{\sqrt{|\log h_2| \vee \log \log n}} = O(\gamma_r^{1-\ell/2}). \quad (9.79)
$$

First of all, like the bounded case, we begin by representing the classes of functions on $\mathbb{R}^{pk} \times \mathbb{R}^{pk}$ and $\mathbb{R}^{p\ell} \times \mathbb{R}^{p\ell}$,

$$
\begin{align*}
\mathcal{G}_{r,j}^t &:= \left\{ h_2^{pk} G_{\phi,h_2,\tilde{\mathbf{t}}}^{(T)}(\cdot, \cdot) : \phi \in \mathcal{F}, h_2 \in \mathcal{H}_{2,r,j}', \tilde{\mathbf{t}} \in \mathbf{I}^k \right\} \\
\mathcal{G}_{r,j}^{t(\ell)} &:= \left\{ h_2^{pk} \left( \pi_\ell G_{\phi,h_2,\tilde{\mathbf{t}}}^{(r)}(\cdot, \cdot) \right) / \left( 2^\ell \lambda \gamma_r^{1/\nu} \right) : \phi \in \mathcal{F}, h_2 \in \mathcal{H}_{2,r,j}', \tilde{\mathbf{t}} \in \mathbf{I}^k \right\}.
\end{align*}
$$

Then, it is easy to show that these classes are of VC-type with characteristics that are independent of $\ell$ and with envelope functions $\tilde{\mathbf{F}}$, defined as in (9.56), and $(2^\ell \lambda \gamma_r^{1/\nu})^{-1} \mathbf{F}_\ell$, determined as in the proof of Theorem 1 of Giné and Mason (2007a), respectively. Set, like in (9.63) and (9.58),

$$
\mathcal{Y}_n'(j, \ell, r) := \sup_{H \in \mathcal{G}_{r,j}^{(\ell)}} \left| \frac{1}{n_r^{\ell/2}} \sum_{\mathbf{i} \in t_n^k} H(\mathbf{X}_{\mathbf{i}}, \mathbf{Y}_{\mathbf{i}}) \right|, \quad n_{r-1} < n \le n_r. \quad (9.80)
$$

We can see, for all $\ell = 2, \dots, k$ and $n_{r-1} < n \le n_r$, that

$$
\mathcal{Y}'_n(j, \ell, r) \le n_r^{\ell/2} \| U_n^{(\ell)} (\pi_\ell G) \|_{\mathcal{G}_{r,j}^{(\ell)}}.
$$

Accordingly, an application of Theorem 10.0.7 with $c = 1/2$ and $r' = 2$ gives us exactly (9.64) with $\mathcal{Y}_n(j, \ell, r)$ and $\mathcal{Y}_{n_r}(j, \ell, r)$ replaced by $\mathcal{Y}'_n(j, \ell, r)$ and $\mathcal{Y}'_{n_r}(j, \ell, r)$, respectively. Thus, we will follow the same strategy as in the bounded case. Remark that all the functions in $\mathcal{G}_{r,j}^{(\ell)}$ are bounded by 1, as held for the functions in $\mathcal{G}_{r,j}'$, and have second moments, by arguing as in (9.74) and (9.66), that can be bounded by $h_2^{pk} B_{k,\ell}$ for a suitable $B_{k,\ell}$. Hence, the expression in (9.66) is also satisfied for functions in $\mathcal{G}_{r,j}^{(\ell)}$, that is,

$$
\sup_{H \in \mathcal{G}_{r,j}^{(\ell)}} \mathbb{E}H^2(\mathbf{X}, \mathbf{Y}) \le B_{k,\ell} h_{2,r,j}^{pk} := \tilde{\sigma}_{2,r,j}^2.
$$

Now, we have all the needed conditions of Theorems 10.0.8 and 10.0.9 satisfied. So, after some obvious changes of notation and modifications, the second part of the proof of Theorem 3.1.1 (and (9.69) in particular) shows us, for some $\mathcal{C}_{7,\ell} > 0$, all $j = 1, \dots, E(r)$ and any $\rho > 2$,

$$
\mathbb{P} \left\{ \max_{n_{r-1} < n \le n_r} \mathcal{Y}'_n(j, \ell, r) > 2\rho^{\ell/2} w'^* \right\} \le \sqrt{h_{2,r,j}^{pk}} (\log n_r)^{-\rho \mathcal{C}_{7,\ell}}, \quad (9.81)
$$
by taking $w'^* = \mathcal{C}_{1,\ell}' \xi_{j,\ell}'(r)$ for some $\mathcal{C}_{1,\ell}' > 0$ and where $\xi_{j,\ell}'(r)$ is represented as in (9.67) with $h_{2,r,j}$ replaced by $h_{2,r,j}'$, that is,

$$ \xi_{j,\ell}'(r) = (|\log h_{2,r,j}'| \lor \log \log n_r)^{\ell/2}. $$

To finish the proof of (9.79), note, as the same argument in (9.70), that, for some $\mathcal{D}_\ell > 0$, for $n_{r-1} < n \le n_r$,

$$ n_r^{\ell/2} \|U_n^{(\ell)}(\pi_\ell G)\|_{\mathcal{Q}_{r,j}'} \le 2^\ell C_\ell \lambda \gamma_r^{1/\nu} \gamma_n'(j, \ell, r). $$

This follows that, for some $c_\ell > 0$,

$$ \begin{aligned} \max_{n_{r-1}<n \le n_r} \mathcal{A}_{n,r,\ell}' &:= \max_{n_{r-1}<n \le n_r} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |U_n^{(\ell)}(\pi_\ell G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)})|}{\sqrt{(|\log h_2| \lor \log \log n)^\ell}} \\ &\le \frac{2^\ell c_\ell \lambda \gamma_r^{1/\nu}}{\sqrt{n_r^{\ell-1}}} \max_{n_{r-1}<n \le n_r} \max_{1 \le j \le E(r)} \frac{\gamma_n'(j, \ell, r)}{\xi_{j,\ell}'(r)} \\ &\le \frac{2^\ell c_\ell \lambda \gamma_r^{1/\nu}}{\sqrt{\tilde{a}_{n_r}^{pk} n_r^{\ell-1}}} \max_{n_{r-1}<n \le n_r} \max_{1 \le j \le E(r)} \frac{\gamma_n'(j, \ell, r)}{\xi_{j,\ell}'(r)}. \end{aligned} $$

We now see, by using $\tilde{a}_n'^{pk} = c^{pk}(\log n/n)^{1-2/\nu}$, that

$$ \gamma_r^{2/\nu} / \tilde{a}_{n_r}^{pk} n_r^{\ell-1} = c^{-k} n_r^{2-\ell} / \log n_r. $$

Moreover, taking into account that $\log n/n^{2-\ell}$ is monotone increasing in $n \ge 2$ whenever $\ell \ge 2$ so, we have, for some constant $\mathcal{D}_{8,\ell} > 0$, that

$$ \begin{aligned} & \mathbb{P} \left\{ \max_{n_{r-1}<n \le n_r} \sqrt{\frac{\log n}{n^{2-\ell}}} \mathcal{A}_{n,r,\ell}' > \mathcal{D}_{8,\ell} \right\} \\ & \le \mathbb{P} \left\{ \max_{n_{r-1}<n \le n_r} \max_{1 \le j \le E(r)} \frac{\gamma_n'(j, \ell, r)}{\xi_{j,\ell}'(r)} > \frac{\mathcal{D}_{8,\ell}}{2^\ell c_\ell \lambda \gamma_r^{1/\nu}} \sqrt{\frac{n_r^{2-\ell} \tilde{a}_{n_r}^{pk} n_r^{\ell-1}}{\log n_r}} \right\} \\ & \le \sum_{j=1}^{E(r)} \mathbb{P} \left\{ \max_{n_{r-1}<n \le n_r} \gamma_n'(j, \ell, r) > \frac{\mathcal{D}_{8,\ell} c^{pk/2}}{2^\ell c_\ell \lambda} \xi_{j,\ell}'(r) \right\}. \end{aligned} $$

Thus, by selecting

$$ \mathcal{D}_{8,\ell} > 2^{\ell+1} c^{-k/2} \lambda c_\ell \mathcal{A}_{1,\ell}' \left( (2+\delta) / \mathcal{D}_{7,\ell} \right)^{\ell/2} $$

and remarking that by definition $E(r) \le 2 \log n_r$ and $h_{2,r,j}' < 2$ for all $j = 1, \dots, E(r)$, we can infer, from (9.81) with

$$ \rho = (2 + \delta) / \mathcal{D}_{7,\ell}, $$

that

$$ \begin{aligned} & \mathbb{P} \left\{ \max_{n_{r-1}<n \le n_r} \sqrt{\frac{\log n}{n^{2-\ell}}} \mathcal{A}_{n,r,\ell}' > \mathcal{D}_{8,\ell} \right\} \\ & \le \sum_{j=1}^{E(r)} \mathbb{P} \left\{ \max_{n_{r-1}<n \le n_r} \gamma_n'(j, \ell, r) > 2 \left( \frac{2+\delta}{\mathcal{D}_{7,\ell}} \right)^{\ell/2} \mathcal{D}_{1,\ell}' \xi_j' \right\} \\ & = \sum_{j=1}^{E(r)} \mathbb{P} \left\{ \max_{n_{r-1}<n \le n_r} \gamma_n'(j, \ell, r) > 2 \left( \frac{2+\delta}{\mathcal{D}_{7,\ell}} \right)^{\ell/2} w'^* \right\} \\ & \le E(r) \sqrt{h_{2,r,j}^{pk}} (\log n_r)^{-\rho \mathcal{D}_{7,\ell}}, \\ & \le 2 \sqrt{2^{pk}} (\log n_r)^{-(1+\delta)}. \end{aligned} $$

This immediately implies, according to the Borel-Cantelli lemma, that for all $\ell = 2, \dots, k$ and $r \ge 1$,

$$ \max_{n_{r-1}<n \le n_r} \sup_{\tilde{a}_n' \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |U_n^{(\ell)}(\pi_\ell G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)})|}{\sqrt{(|\log h_2| \lor \log \log n)^\ell}} = O \left( \sqrt{\frac{n_r^{2-\ell}}{\log n_r}} \right) \quad \text{a.s.,} $$
this, in turn, implies that (9.79). Finally, keeping in mind the decomposition of Hoeffding (9.52) and (9.78), we obtain for some
$\mathcal{C}'' > 0$ and with probability 1,

$$
\limsup_{r \to \infty} \max_{n_{r-1} < n \le n_r} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)}) - \mathbb{E}U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(T)})|}{\sqrt{|\log h_2| \vee \log \log n}} \le \mathcal{C}'' . \quad (9.82)
$$

## Remainder part

Let us investigate, now, the remainder part that is the U-process $u_n^{(R)}(\varphi, h_2, \tilde{\mathbf{t}})$ based on the unbounded kernel given by :

$$
G_{\varphi, h_2, \tilde{\mathbf{t}}}^{(R)}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) = G_{\varphi, h_2, \tilde{\mathbf{t}}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) 1_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) > \lambda \gamma_r^{1/\nu}\}}
$$

Our main goal is to prove that the U-process $u_n^{(R)}(\varphi, h_2, \tilde{\mathbf{t}})$ is negligible, more precisely

$$
\limsup_{r \to \infty} \max_{n_{r-1} < n \le n_r} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)}) - \mathbb{E}U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)})|}{\sqrt{|\log h_2| \vee \log \log n}} = o(1) \quad \text{a.s.} \quad (9.83)
$$

For all $\varphi \in \mathcal{F}$, $h_2 \in [\tilde{a}'_n, \tilde{b}_0]$ and $\tilde{\mathbf{t}}, \tilde{\mathbf{x}} \in \mathbb{R}^{pk}$ we know that

$$
\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) \geq h_2^{pk} |G_{\varphi, h_2, \tilde{\mathbf{t}}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}})|.
$$

So, from the symmetry of $\tilde{\mathbf{F}}$, we get that

$$
|U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)})| \le h_2^{-pk} U_n^{(k)}(\tilde{\mathbf{F}}1_{\{\tilde{\mathbf{F}} > \lambda \gamma_r^{1/\nu}\}}),
$$

where $U_n^{(k)}(\tilde{\mathbf{F}}1_{\{\tilde{\mathbf{F}} > \lambda \gamma_r^{1/\nu}\}})$ is a U-statistic based on the positive and symmetric kernel

$$
\tilde{\mathbf{y}} \to \tilde{\mathbf{F}}(\tilde{\mathbf{y}}) 1_{\{\tilde{\mathbf{F}}(\tilde{\mathbf{y}}) > \lambda \gamma_r^{1/\nu}\}}.
$$

We can infer, keeping in mind that $\tilde{a}_n^{pk} = c^{pk}(\log n/n)^{1-2/\nu}$, that, for all $\varphi \in \mathcal{F}$, $h_2 \in [\tilde{a}', \tilde{b}_0]$, $\tilde{\mathbf{t}} \in \mathbf{I}^k$ and some $\mathcal{C} > 0$,

$$
\begin{align*}
\max_{n_{r-1} < n \le n_r} \frac{\sqrt{nh_2^{pk}} |U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)})|}{\sqrt{|\log h_2| \vee \log \log n}} &\le \frac{\sqrt{n_r} U_{n_r}^{(k)} (\tilde{\mathbf{F}}1_{\{\tilde{\mathbf{F}} > \lambda \gamma_r^{1/\nu}\}})}{\sqrt{\tilde{a}_{n_r}^{pk} (|\log \tilde{a}_{n_r}'| \vee \log \log n_r)}} \\
&\le \mathcal{C} \gamma_r^{1-1/\nu} U_{n_r}^{(k)} (\tilde{\mathbf{F}}1_{\{\tilde{\mathbf{F}} > \lambda \gamma_r^{1/\nu}\}}).
\end{align*}
$$

Similarly, we get, uniformly in $\varphi \in \mathcal{F}$, $h_2 \in [\tilde{a}'_n, \tilde{b}_0]$ and $\tilde{\mathbf{t}} \in \mathbf{I}^k$, that

$$
\begin{align}
\max_{n_{r-1} < n \le n_r} \frac{\sqrt{nh_2^{pk}} |\mathbb{E}U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)})|}{\sqrt{|\log h_2| \vee \log \log n}} &\le \mathcal{C} \gamma_r^{1-1/\nu} \mathbb{E}U_{n_r}^{(k)}(\tilde{\mathbf{F}}1_{\{\tilde{\mathbf{F}} > \lambda \gamma_r^{1/\nu}\}}) \nonumber \\
&\le \mathcal{C}_1 \mathbb{E}[\tilde{\mathbf{F}}^\nu(Y)1_{\{\tilde{\mathbf{F}}(Y) > \lambda \gamma_r^{1/\nu}\}}]. \tag{9.84}
\end{align}
$$

Based on (9.84), we obtain that,

$$
\limsup_{r \to \infty} \max_{n_{r-1} < n \le n_r} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |\mathbb{E}U_n^{(k)}(G_{\varphi,h_2,\tilde{\mathbf{t}}}^{(R)})|}{\sqrt{|\log h_2| \vee \log \log n}} = o(1).
$$

Therefore, to complete the proof of (9.83), it is only to prove that

$$
U_{n_r}^{(k)}(\tilde{\mathbf{F}}1_{\{\tilde{\mathbf{F}} > \lambda \gamma_r^{1/\nu}\}}) = o(\gamma_r^{1/\nu - 1}) \quad \text{a.s.} \quad (9.85)
$$
By applying Chebyshev's inequality and another well-known inequality, we get, for any $\delta > 0$:

$$ \begin{aligned} & \mathbb{P} \left\{ \left| U_{n_r}^{(k)} \left( \tilde{\mathbf{F}} \mathbf{1} \left\{ \tilde{\mathbf{F}} > \lambda \gamma_r^{1/v} \right\} \right) - \mathbb{E} U_{n_r}^{(k)} \left( \tilde{\mathbf{F}} \mathbf{1} \left\{ \tilde{\mathbf{F}} > \lambda \gamma_r^{1/v} \right\} \right) \right| > \delta \gamma_r^{-(1-1/v)} \right\} \\ & \le \delta^{-2} \gamma_r^{2-2/v} \mathrm{Var} \left( U_{n_r}^{(k)} \left( \tilde{\mathbf{F}} \mathbf{1} \left\{ \tilde{\mathbf{F}} > \lambda \gamma_r^{1/v} \right\} \right) \right) \\ & \le k \delta^{-2} \frac{n_r^{1-2/v}}{(\log n_r)^{2-2/v}} \mathbb{E} \left[ \tilde{\mathbf{F}}^2(\mathbf{Y}) \mathbf{1} \left\{ \tilde{\mathbf{F}}(\mathbf{Y}) > \lambda \gamma_r^{1/v} \right\} \right]. \end{aligned} $$

Next, to determine the finite convergence of the series as mentioned above of probabilities, we divide the indicator function 1{$\tilde{\mathbf{F}}(\mathbf{Y})>\lambda\gamma_r^{1/v}$}

into two separate parts determined by whether $\tilde{\mathbf{F}}(\mathbf{Y}) > n_r^{1/v}$ or $\lambda\gamma_r^{1/v} < \tilde{\mathbf{F}}(\mathbf{Y}) \le n_r^{1/v}$, and consider the corresponding second moments in (9.86) separately. For the second case, we have, using the condition (F.v) and (9.56), that $\mathbb{E}\tilde{\mathbf{F}}^v(\mathbf{Y}) \le \theta_v \kappa^{pk} \|\alpha\|_{\infty}^{pk} (k!)^v$ and remark that since $v > 2$ and $n_r = 2^r$,

$$ \sum_{r=1}^{\infty} \frac{n_r^{1-2/v}}{(\log n_r)^{2-2/v}} \mathbb{E} \left[ \tilde{\mathbf{F}}^2(\mathbf{Y}) \mathbf{1} \left\{ \tilde{\mathbf{F}}(\mathbf{Y}) > n_r^{1/v} \right\} \right] \le \mathbb{E} \left[ \tilde{\mathbf{F}}^v(\mathbf{Y}) \right] \sum_{r=1}^{\infty} (\log n_r)^{-(2-2/v)} < \infty. $$

To treat the first case, we will need the following fact of Einmahl and Mason (2000).

**Fact 9.8.1.** Let $(c_n)_{n \ge 1}$ be a sequence of positive constants such that $c_n/n^{1/s} \nearrow \infty$ for some $s > 0$ and let $\mathbf{Z}$ be a random variable satisfying

$$ \sum_{n=1}^{\infty} \mathbb{P} \{|\mathbf{Z}| > c_n\} < \infty. $$

We then have, for any $q > s$,

$$ \sum_{k=1}^{\infty} 2^k \mathbb{E} \left[ |\mathbf{Z}|^q \mathbf{1}_{\{|\mathbf{Z}| \le c_{2k}\}} \right] / (c_{2k})^q < \infty. $$

Notice that for any $v < \tau \le 2v$,

$$ \begin{aligned} & \sum_{r=1}^{\infty} \frac{n_r^{1-2/v}}{(\log n_r)^{2-2/v}} \mathbb{E} \left[ \tilde{\mathbf{F}}^2(\mathbf{Y}) \mathbf{1} \left\{ \lambda \gamma_r^{1/v} < \tilde{\mathbf{F}}(\mathbf{Y}) \le n_r^{1/v} \right\} \right] \\ & \le \lambda^{\tau-2} \sum_{r=1}^{\infty} \frac{n_r \mathbb{E} \left[ \tilde{\mathbf{F}}^{\tau}(\mathbf{Y}) \mathbf{1} \left\{ \tilde{\mathbf{F}}(\mathbf{Y}) \le n_r^{1/v} \right\} \right]}{n_r^{\tau/v}}. \end{aligned} $$

Now, set $\mathbf{Z} = \tilde{\mathbf{F}}(\mathbf{Y})$, $c_n = n^{1/v}$, $k = r$ and $q = \tau$ in Fact 9.8.1 and note that $c_n/n^{1/s} \nearrow \infty$ for any $s$ such that $q = \tau > s > v$. Since $q = \tau > s$, we can conclude from Fact 9.8.1 that this last bound is finite. Finally, note that the bound leading to (9.86) implies that

$$ \gamma_r^{1-1/v} \mathbb{E} U_{n_r}^{(k)} \left( \tilde{\mathbf{F}} \mathbf{1} \left\{ \tilde{\mathbf{F}} > \lambda \gamma_r^{1/v} \right\} \right) = o(1). $$

Accordingly, the above results with (9.86) imply, by applying Borel-Cantelli lemma and the arbitrary choice of $\delta > 0$, that (9.85) holds, which, when combined with (9.84) and (9.86), completes the proof of (9.83). Therefore, Theorem 3.3.3 is proved since the desired result for the truncated part in (9.82) was also established. $\square$

## 9.9 Proof of Theorem 3.3.4

Notice that

$$ \begin{aligned} & |\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - \mathbb{E}\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})| \\ & \le |\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}})| \\ & \quad + |\mathbb{E}\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - \mathbb{E}U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}})| \\ & \quad + |U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}}) - \mathbb{E}U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}})|. \end{aligned} \quad (9.86) $$

We begin, first, by the first term in (9.86). We have

$$ |\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}})| $$
$$
\le \frac{(n-k)!}{n!} \sum_{i \in I_h^k} \left| \varphi(\mathbf{Y}_{i_1}, \dots, \mathbf{Y}_{i_k}) \left\{ \prod_{j=1}^k K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) \right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) \right. \\
\left. - \prod_{j=1}^k K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_j})) \right) \alpha^p(f(\mathbf{X}_{i_j})) \right\} \right|.
$$

Recall the following useful lemma.

**Lemma 9.9.1.** Let $a_i$, $i = 1, \dots, k$, $b_i$ $i = 1, \dots, k$ be real numbers

$$
\prod_{i=1}^{k} a_i - \prod_{i=1}^{k} b_i = \sum_{i=1}^{k} (a_i - b_i) \prod_{j=1}^{i-1} b_j \prod_{h=1+i}^{k} a_h.
$$

By applying the last lemma we get that :

$$
\begin{align*}
& \prod_{j=1}^{k} K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) \right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) - \prod_{j=1}^{k} K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_j})) \right) \alpha^p(f(\mathbf{X}_{i_j})) \\
&= \sum_{j=1}^{k} \left[ \left\{ K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) \right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) - K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_j})) \right) \alpha^p(f(\mathbf{X}_{i_j})) \right\} \right. \\
& \qquad \left. \times \prod_{s=1}^{j-1} K \left( \frac{\mathbf{t}_s - \mathbf{X}_{i_s}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_s}; h_{1,n})) \right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_s}; h_{1,n})) \prod_{s'=j+1}^{k} K \left( \frac{\mathbf{t}_{s'} - \mathbf{X}_{i_{s'}}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_{s'}})) \right) \alpha^p(f(\mathbf{X}_{i_{s'}})) \right] \\
&= \sum_{j=1}^{k} \left[ \left\{ K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) \right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) - K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_j})) \right) \alpha^p(f(\mathbf{X}_{i_j})) \right\} \right. \\
& \qquad \left. \times \prod_{s=1, s \neq j}^{k} K \left( \frac{\mathbf{t}_s - \mathbf{X}_{i_s}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_s})) \right) \alpha^p(f(\mathbf{X}_{i_s})) \right] + O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n \tilde{a}_n}} \right).
\end{align*}
$$

By utilizing the Taylor expansion, we obtain that

$$
\begin{align*}
& \prod_{j=1}^{k} K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) \right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n})) \\
& \quad - \prod_{j=1}^{k} K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_j})) \right) \alpha^p(f(\mathbf{X}_{i_j})) \\
& = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n \tilde{a}_n}} \right).
\end{align*}
$$

We conclude that

$$
|\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}})| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) \quad \text{a.s.}
$$

This implies the second term in (9.86), so, we deduce that

$$
| \mathbb{E} \tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - \mathbb{E} U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}}) | = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n \tilde{a}_n}} \right) \quad \text{a.s.}
$$

Thus, the previous results in combination with (3.3) lead to the desired result.

<figure><img src="image_30.png" /></figure>

**9.10 Proof of Theorem 3.3.6**

Observe that

$$
| \check{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - \hat{\mathbb{E}}[\check{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})] |
$$
$$
\begin{align*}
&= \left| \frac{\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})}{\tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})} - \frac{\mathbb{E}[\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})]}{\mathbb{E}[\tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})]} \right| \\
&\le \frac{\left| \tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - \mathbb{E}[\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})] \right|}{\left| \tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) \right|} \\
&\qquad + \frac{\left| \mathbb{E}[\tilde{\mathbf{U}}_n(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})] \cdot \left| \tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - \mathbb{E}[\tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})] \right|}{\left| \tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) \right| \cdot \left| \mathbb{E}[\tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})] \right|} \\
&=: \mathbf{I} + \mathbf{II}.
\end{align*}
$$

Under the imposed hypothesis and the previous obtained results, since $f$ is bounded away from zero, we get, for some $\omega_1, \omega_2 > 0$ and $c$ large enough in $\tilde{a}'_n = c(\log n/n)^{1/k}$,

$$
\sup_{a_n \le h_1 < b_n} \sup_{\tilde{a}'_n \le h_2 < \tilde{b}_n} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \left| \tilde{\mathbf{U}}_n(1, \tilde{\mathbf{t}}; h_1, h_2) \right| = \omega_1 \quad \text{a.s.}
$$

$$
\sup_{a_n \le h_1 < b_n} \sup_{\tilde{a}'_n \le h_2 < \tilde{b}_n} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \left| \mathbb{E} \left[ \tilde{\mathbf{U}}_n (1, \tilde{\mathbf{t}}; h_1, h_2) \right] \right| = \omega_2 \quad \text{a.s.}
$$

$$
\sup_{\varphi \in \mathcal{F}} \sup_{a_n \le h_1 < b_n} \sup_{\tilde{a}'_n \le h_2 < \tilde{b}_n} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \left| \tilde{\mathbf{U}}_n (\varphi, \tilde{\mathbf{t}}; h_1, h_2) \right| = O(1).
$$

Now, we are going to apply Theorems 3.3.4 and 3.3.5 to deal I, depending on whether the class $\mathcal{F}$ satisfies (F.iv) or (F.v), and
Theorem 3.3.4 to deal II. By taking everything together, we conclude that for some $\tilde{C}'' > 0$, we have, with probability 1,

$$
\begin{align*}
& \limsup_{n \to \infty} \sup_{a'_n \le h_1 < b_n} \sup_{\tilde{a}'_n \le h_2 < \tilde{b}_n} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) - \hat{\mathbb{E}}\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)|}{\sqrt{|\log h_2| \vee \log \log n}} \\
& \le \limsup_{n \to \infty} \sup_{a'_n \le h_1 < b_n} \sup_{\tilde{a}'_n \le h_2 < \tilde{b}_n} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} (\mathbf{I})}{\sqrt{|\log h_2| \vee \log \log n}} \\
& \qquad + \limsup_{n \to \infty} \sup_{a'_n \le h_1 < b_n} \sup_{\tilde{a}'_n \le h_2 < \tilde{b}_n} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} (\mathbf{II})}{\sqrt{|\log h_2| \vee \log \log n}} \\
& \le C''.
\end{align*}
$$

Hence the proof is complete.

□

**9.11 Proof of Theorem 3.3.7**

We can see, by the decompositions (9.3) and (9.5), that

$$
\begin{align*}
& \mathbb{E} \tilde{\mathbf{U}}_n (1, \tilde{\mathbf{t}}; h_1, h_2) \\
&= \int \tilde{K}_{h_2} ((\tilde{\mathbf{t}} - \tilde{\mathbf{x}}) \alpha(f_{\mathbf{X};n}(\tilde{\mathbf{x}}; h_{1,n}))) \boldsymbol{\alpha}(f_{\mathbf{X};n}(\tilde{\mathbf{x}}; h_1)) f_{\mathcal{X}\mathcal{Y}} (\mathbf{x}_1, \mathbf{y}_1) \cdots f_{\mathcal{X}\mathcal{Y}} (\mathbf{x}_k, \mathbf{y}_k) \mathrm{d}\tilde{\mathbf{x}} \mathrm{d}\tilde{\mathbf{y}} \\
&= \int \prod_{i=1}^k \left[ K \left( \frac{\mathbf{t} - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i)) \right) + \sum_{j=1}^d K'_j \left( \frac{\mathbf{t} - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i)) \right) \frac{(\mathbf{t} - \mathbf{x}_j)_j}{h_{2,n}} \alpha(f(\mathbf{x}_i)) \delta(\mathbf{x}_i) + \boldsymbol{\delta}_2(\mathbf{t}; \mathbf{x}_i) \right] \\
&\quad \times \prod_{i=1}^k [\alpha^p(f(\mathbf{x}_i)) + d\alpha^p(f(\mathbf{x}_i))\delta(\mathbf{x}_i) + \alpha^p(f(\mathbf{x}_i))\delta_1(\mathbf{x}_i)] \tilde{f}(\tilde{\mathbf{x}}) \mathrm{d}\tilde{\mathbf{x}} \\
&=: \int \prod_{i=1}^k \left[ K \left( \frac{\mathbf{t} - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i)) \right) + \mathbf{G}(\mathbf{x}_i) \right] \prod_{i=1}^k [\alpha^p(f(\mathbf{x}_i)) + \mathbf{H}(\mathbf{x}_i)] \tilde{f}(\tilde{\mathbf{x}}) \mathrm{d}\tilde{\mathbf{x}} \\
&\le \int \left[ \prod_{i=1}^k K \left( \frac{\mathbf{t} - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i)) \right) + \sum_{r=0}^{k-1} \binom{k}{r} \prod_{i=1}^r K \left( \frac{\mathbf{t} - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i)) \right) \prod_{i=1}^{k-r} \mathbf{G}(\mathbf{x}_i) \right] \\
&\quad \times \left[ \prod_{i=1}^k \alpha^p(f(\mathbf{x}_i)) + \sum_{r'=0}^{k-1} \binom{k}{r'} \prod_{i=1}^{r'} \alpha^p(f(\mathbf{x}_i)) \prod_{i=1}^{k-r'} \mathbf{H}(\mathbf{x}_i) \right] \tilde{f}(\tilde{\mathbf{x}}) \mathrm{d}\tilde{\mathbf{x}}
\end{align*}
$$
$$
\le \int \prod_{i=1}^{k} K\left(\frac{\mathbf{t}-\mathbf{x}_{i}}{h_{2}} \alpha\left(f(\mathbf{x}_{i})\right)\right) \prod_{i=1}^{k} \alpha^{p}\left(f(\mathbf{x}_{i})\right) \tilde{f}(\tilde{\mathbf{x}}) \mathrm{d} \tilde{\mathbf{x}} \\
+ \int \sum_{r=0}^{k-1}\left(\frac{k}{r}\right) \prod_{i=1}^{r} K\left(\frac{\mathbf{t}-\mathbf{x}_{i}}{h_{2}} \alpha\left(f(\mathbf{x}_{i})\right)\right) \prod_{i=1}^{k-r} \mathbf{G}(\mathbf{x}_{i}) \prod_{i=1}^{k} \alpha^{p}\left(f(\mathbf{x}_{i})\right) \tilde{f}(\tilde{\mathbf{x}}) \mathrm{d} \tilde{\mathbf{x}} \\
+ \int \prod_{i=1}^{k} K\left(\frac{\mathbf{t}-\mathbf{x}_{i}}{h_{2}} \alpha\left(f(\mathbf{x}_{i})\right)\right) \sum_{r'}\left(\frac{k}{r'}\right) \prod_{i=1}^{r'} \alpha^{p}\left(f(\mathbf{x}_{i})\right) \prod_{i=1}^{k-r'} \mathbf{H}(\mathbf{x}_{i}) \tilde{f}(\tilde{\mathbf{x}}) \mathrm{d} \tilde{\mathbf{x}} \\
+ \int \sum_{r=0}^{k-1}\left(\frac{k}{r}\right) \prod_{i=1}^{r} K\left(\frac{\mathbf{t}-\mathbf{x}_{i}}{h_{2}} \alpha\left(f(\mathbf{x}_{i})\right)\right) \prod_{i=1}^{k-r} \mathbf{G}(\mathbf{x}_{i}) \sum_{r'}\left(\frac{k}{r'}\right) \prod_{i=1}^{r'} \alpha^{p}\left(f(\mathbf{x}_{i})\right) \prod_{i=1}^{k-r'} \mathbf{H}(\mathbf{x}_{i}) \tilde{f}(\tilde{\mathbf{x}}) \mathrm{d} \tilde{\mathbf{x}} \quad (9.87)
$$

We know that the second, the third, and the fourth term in (9.87) is of order

$$
O_{a.s.}\left(\sqrt{\frac{\log(1/\tilde{a}_n) \lor \log \log n}{n\tilde{a}_n}}\right).
$$

Therefore, we have that

$$
\left| \mathbb{E} \tilde{\mathbf{U}}_n (1, \tilde{\mathbf{t}}; h_1, h_2) - \prod_{i=1}^{k} f_X(\mathbf{x}_i) \right| \\
\leq \left| \frac{1}{h_2^{pk}} \int \prod_{i=1}^{k} K \left( \frac{\mathbf{t}_i - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i)) \right) \prod_{i=1}^{k} \alpha^p(f(\mathbf{x}_i)) \tilde{f}(\tilde{\mathbf{x}}) d\tilde{\mathbf{x}} - \tilde{f}(\tilde{\mathbf{t}}) \right| \\
+ O_{a.s.} \left( \sqrt{\frac{\log(1/\tilde{a}_n) \lor \log \log n}{n\tilde{a}_n}} \right). \tag{9.88}
$$

Thus, it suffices to bound (9.88). Notice that, by using the change of variables $h_2\mathbf{u}_i = (\mathbf{t}_i - \mathbf{x}_i)\alpha(f(\mathbf{t}_i - (\mathbf{t}_i - \mathbf{x}_i)))$,

$$
\left| \frac{1}{h_2^{pk}} \int \prod_{i=1}^{k} K\left(\frac{\mathbf{t}_i - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i))\right) \prod_{i=1}^{k} \alpha^p(f(\mathbf{x}_i)) \tilde{f}(\tilde{\mathbf{x}}) d\tilde{\mathbf{x}} - \tilde{f}(\tilde{\mathbf{t}}) \right| \\
= \left| - \int \left( \prod_{i=1}^{k} \alpha^p(f(\mathbf{t}_i - V_{\mathbf{t}_i}(h_2 \mathbf{u}_i))) f_X(\mathbf{t}_i - V_{\mathbf{t}_i}(h_2 \mathbf{u}_i)) \frac{dV_{\mathbf{t}_i}(h_2 \mathbf{u}_i)}{d(h_2 \mathbf{u}_i)} - \tilde{f}(\tilde{\mathbf{t}}) \right) \prod_{i=1}^{k} K(\mathbf{u}_i) d\mathbf{u}_i \right|.
$$

We will use the following Taylor expansion

$$
\prod_{i=1}^{k} \alpha^p(f(\mathbf{t}_i - V_{\mathbf{t}_i}(h_2 \mathbf{u}_i))) f_X(\mathbf{t}_i - V_{\mathbf{t}_i}(h_2 \mathbf{u}_i)) \frac{dV_{\mathbf{t}_i}(h_2 \mathbf{u}_i)}{d(h_2 \mathbf{u}_i)} = \sum_{i=0}^{l} \phi^{(i)}(0) h_2^{m_i},
$$

where

$$
\phi^{(i)}(s) = \frac{1}{i!} \sum_{m_1+\cdots+m_k=i} \left( \binom{i}{m_1:k} \prod_{j=1}^{k} \sum_{i_1, \ldots, i_m=1}^{p} (-1)^{m_j} u_{i_1}^{v_1} \cdots u_{i_m}^{v_m} D_{m_j} \left( \frac{f_X}{\alpha^{2ip}(f(\mathbf{x}_j))} \right) \left( \mathbf{x}_j - \frac{h_2 s_{\mathbf{x},t} \mathbf{t}_j}{\alpha^p(f(\mathbf{x}_j))} \right) \right)
$$

and

$$
\binom{i}{m_1:k} := i! / \left( \prod_{j=1}^{k} (m_j!) \right).
$$

Notice that $\phi^{(0)}(0) = \prod_{i=1}^{k} f_X(\mathbf{x}_i)$. From this we infer that

$$
\left| \frac{1}{h_2^k} \int \prod_{i=1}^{k} K\left(\frac{\mathbf{t} - \mathbf{x}_i}{h_2} \alpha(f(\mathbf{x}_i))\right) \prod_{i=1}^{k} \alpha^p(f(\mathbf{x}_i)) \tilde{f}(\tilde{\mathbf{x}}) d\tilde{\mathbf{x}} - \tilde{f}(\tilde{\mathbf{t}}) \right| = O(\tilde{b}_n^4).
$$

So, we deduce that

$$
\sup_{a_n \le h_{1,n} \le b_n} \sup_{\tilde{a}_n \le h_{2,n} \le \tilde{b}_n} \sup_{\tilde{\mathbf{t}} \in \mathcal{T}^k} \left| \mathbb{E} \tilde{\mathbf{U}}_n (1, \tilde{\mathbf{t}}; h_1, h_2) - \tilde{f}(\tilde{\mathbf{t}}) \right| = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \lor \log \log n}{n\tilde{a}_n}} \right) + O(\tilde{b}_n^4) \quad \text{a.s.} \quad (9.89)
$$
In a similar way, we get that

$$
\sup_{a_n \le h_{1,n} \le b_n} \sup_{\tilde{a}_n \le h_{2,n} \le \tilde{b}_n} \sup_{\tilde{t} \in \mathbf{I}^k} \left| \mathbb{E} \tilde{\mathbf{U}}_n (\varphi, \tilde{\mathbf{t}}; h_1, h_2) - r^{(k)} (\varphi, \tilde{\mathbf{t}}) \tilde{f}(\tilde{\mathbf{t}}) \right| = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n \tilde{a}_n}} \right) + O(\tilde{b}_n^4) \quad \text{a.s.} \quad (9.90)
$$

Keeping in mind the definition of $\hat{\mathbb{E}}\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})$ in (3.30), it is clear that since $f_X$ is bounded away from zero, (9.89) and (9.90) imply that

$$
\sup_{a_n \le h_{1,n} \le b_n} \sup_{\tilde{a}_n \le h_{2,n} \le \tilde{b}_n} \sup_{\tilde{\varphi} \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \left| \hat{\mathbb{E}} \tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - r^{(k)}(\varphi, \tilde{\mathbf{t}}) \right| = O(1). \quad (9.91)
$$

□

**10 Appendix**

This appendix contains supplementary information that is an essential part of providing a more comprehensive understanding of the paper.

**Theorem 10.0.1** (Dony and Mason (2008)). Let $I = [a, b]$ be a compact interval. Suppose that $\mathcal{H}$ is a uniformly equicontinuous family of real-valued functions $\varphi$ on $J = [a - \eta, b + \eta]^d$ for some $d \ge 1$ and $\eta > 0$. Further, assume that $K$ is an $L_1$-kernel with support in $[-B, B]^d$, with $B > 0$ satisfying $\int_{\mathbb{R}^d} K(\mathbf{u})\mathrm{d}\mathbf{u} = 1$. Then, uniformly in $\varphi \in \mathcal{H}$ and for any sequence of positive constants $b_n \to 0$,

$$
\sup_{0<h<b_n} \sup_{\mathbf{z} \in \mathcal{I}^d} |\varphi * K_h(\mathbf{z}) - \varphi(\mathbf{z})| \to 0 \quad \text{as } n \to \infty
$$

where $K_h(\mathbf{z}) = h^{-d}K(\mathbf{z}/h)$ and

$$
\varphi * K_h(\mathbf{z}) := h^{-d} \int_{\mathbb{R}^d} \varphi(\mathbf{x}) K\left(\frac{\mathbf{z} - \mathbf{x}}{h}\right) \mathrm{d}\mathbf{x}.
$$

**Proposition 10.0.2** (Proposition 1 of Einmahl and Mason (2005)). Let $\mathcal{F}$ be a pointwise measurable class of bounded functions such that for some constants $C, \mathbf{v} \ge 1$ and $0 < \sigma \le \beta$ and $F$ as above, the following conditions hold:

$$
(i) \quad \mathbb{E} [F(\mathbf{x})^2] \le \beta^2;
$$

$$
(ii) \quad N(\epsilon, \mathcal{F}) \le C\epsilon^{-\nu}, \quad 0 < \epsilon < 1;
$$

$$
(iii) \quad \sigma_0^2 := \sup_{g \in \mathcal{F}} \mathbb{E} [g(\mathbf{x})^2] \le \sigma^2;
$$

$$
(iv) \quad \sup_{g \in \mathcal{F}} \|g\|_{\infty} \le \frac{1}{4\sqrt{\nu}} \sqrt{n\sigma^2 / \log(C_1\beta/\sigma)}, \text{ where } C_1 = C^{1/\nu} \vee e.
$$

Then we have for some absolute constant $A$,

$$
\mathbb{E} \left\| \sum_{i=1}^{n} \varepsilon_i g(\mathbf{X}_i) \right\|_{\mathcal{F}} \le A \sqrt{n \sigma^2 \log (C_1 \beta / \sigma)}.
$$

**Corollary 10.0.3.** Let $g$ be as in Proposition 10.0.2 satisfying 1-3, and instead of 4 assume that

$$
(v) \quad \sup_{g \in \mathcal{F}} \|g\|_{\infty} \le U, \text{ where } \sigma_0 \le U \le C_2 \sqrt{n\beta}, \text{ and } C_2 = \frac{1}{4\sqrt{\nu \log C_1}}.
$$

Then we have

$$
\mathbb{E} \left\| \sum_{i=1}^{n} \varepsilon_i g(\mathbf{X}_i) \right\|_{\mathcal{F}} \le A \left\{ \sqrt{vn\sigma_0^2 \log(C_1\beta/\sigma_0)} + 2vU \log(C_3n(\beta/U)^2) \right\},
$$

where

$$
C_3 = C_1^2 / 16\nu.
$$

Let $\alpha_n$ be the empirical process based on the sample $\mathbf{X}_1, \dots, \mathbf{X}_n$, that is, if $g: \mathcal{X} \to \mathbb{R}$, we have

$$
\alpha_n(g) = \sum_{i=1}^{n} \frac{(g(\mathbf{X}_i) - \mathbb{E}g(\mathbf{x}))}{\sqrt{n}}
$$

and set, for any class $\mathcal{F}$ of such functions

$$
\|\sqrt{n}\alpha_n\|_{\mathcal{F}} = \sup_{g \in \mathcal{F}} |\sqrt{n}\alpha_n(g)|.
$$
**Theorem 10.0.4** (Talagrand (1994)). Let $\mathcal{F}$ be a pointwise measurable class of functions satisfying for some $0 < \mathbf{M} < \infty$,

$$ \|g\|_{\infty} \leq \mathbf{M}, \quad g \in \mathcal{F} $$

Then we have for all $\mathbf{u} > 0$,

$$ \mathbb{P} \left\{ \begin{aligned} & \max_{1 \le m \le n} \| \sqrt{m} \alpha_m \|_{\mathcal{F}} \ge A_1 \left( \mathbb{E} \left\| \sum_{i=1}^n \varepsilon_i g(\mathbf{X}_i) \right\|_{\mathcal{F}} + \mathbf{u} \right) \\ & \le 2 \left\{ \exp \left( -\frac{A_2 \mathbf{u}^2}{n \sigma^2} \right) + \exp \left( -\frac{A_2 \mathbf{u}}{\mathbf{M}} \right) \right\}, \end{aligned} \right. $$

where

$$ \sigma_g^2 = \sup_{g \in \mathcal{F}} \mathrm{Var}(g(\mathbf{x})) $$

and $A_1, A_2$ are universal constants.

**Theorem 10.0.5** (Theorem 4.7 of Lax and Terrell (2017)). (Taylor) Suppose $f$ from $\mathbb{R}^d$ to $\mathbb{R}$ is a $C^{m+1}$ function on an open ball centered at $\mathbf{A}$. Then for $\mathbf{A} + \mathbf{H}$ in the ball,

$$ f(\mathbf{A} + \mathbf{H}) = f(\mathbf{A}) + \sum_{i_1=1}^{d} h_{i_1} f_{x_{i_1}}(\mathbf{A}) + \frac{1}{2} \sum_{i_1, i_2=1}^{d} h_{i_1} h_{i_2} f_{x_{i_1} x_{i_2}}(\mathbf{A}) \\ + \dots + \frac{1}{m!} \sum_{i_1, \dots, i_m=1}^{d} (h_{i_1} h_{i_2} \cdots h_{i_m}) f_{x_{i_1} x_{i_2} \cdots x_{i_m}}(\mathbf{A}) + R_m(\mathbf{A}, \mathbf{H}), $$

where

$$ |R_m(\mathbf{A}, \mathbf{H})| \leq k \|\mathbf{H}\|^{m+1} $$

for some constant $k$. The remainder goes to zero faster than $\|\mathbf{H}\|^m$ in the sense that

$$ 0 \le \frac{|R_m(\mathbf{A}, \mathbf{H})|}{\|\mathbf{H}\|^m} \le k \|\mathbf{H}\|. $$

**Proposition 10.0.6** (Giné and Sang (2013); McKay (1993a)). Let the kernel $K: \mathbb{R}^d \to \mathbb{R}$ be symmetric about zero separately in each coordinate, have bounded support and integrate to 1. Assume the density $f$ is in $C^1(\mathbb{R}^d)$. Assume $\gamma_\delta(t) \ge c > 0$ for some $c > 0$ and all $t \in \mathbb{R}^d$ and $0 \le \delta \le \delta_0$, for some $\delta_0 > 0$, and that the function $\gamma(t, \delta) := \gamma_\delta(t)$ is in $C^{l+1}(\mathbb{R}^d \times [0, \delta_0])$. Then we have

$$ E \bar{f}_n(t; h, \delta) = \sum_{k=0}^{l} a_{k, \delta}(t) h^k + o(h^l) $$

as $h \to 0$, uniformly in $t \in \mathbb{R}^d$ and $0 \le \delta \le \delta_1$ for some $\delta_1 > 0$, and the set of functions $a_{k, \delta}$, which are uniformly bounded and equicontinuous, are defined as

$$ a_{2k+1, \delta}(t) = 0, \quad a_{2k, \delta}(t) = \sum_{|v|=2k} \frac{\tau_v}{v!} D_v \left( \frac{f(t)}{\gamma_{\delta}^{2k}(t)} \right), $$

for $k \le 1/2$, in particular,

$$ a_{0, \delta}(t) = f(t). $$

Here $|v|, v!$, $\tau_v$ and $D_v$ are defined in (2.12).

**Theorem 10.0.7** (Theorem 4 of Giné and Mason (2007a)). Let $\mathbf{X}_1, \mathbf{X}_2, \dots$ be i.i.d. $S$-valued with probability law $\mathbb{P}$. Let $\mathcal{H}$ be a $\mathbb{P}$-separable collection of measurable functions $f: S^k \to \mathbb{R}$ and assume that $\mathcal{H}$ is $\mathbb{P}$-canonical (which means that every $f$ in $\mathcal{H}$ is $\mathbb{P}$-canonical). Further, assume that

$$ \mathbb{E} \| f(\mathbf{X}_1, \dots, \mathbf{X}_k) \|_{\mathcal{H}}^{r'} < \infty $$

for some $r' > 1$ and let $s$ be the conjugate of $r'$. Then, with $S_n$ Uniform in bandwidth consistency of conditional $U$-statistics defined as

$$ S_n = \sup_{f \in \mathcal{H}} \left| \sum_{i \in I_n^k} f(\mathbf{X}_{i_1}, \dots, \mathbf{X}_{i_k}) \right|, \quad n \ge k, $$

we have, for all $x > 0$ and $0 < c < 1$,

$$ \mathbb{P} \left\{ \max_{k \le m \le n} S_m > x \right\} \le \frac{\mathbb{P} \{ S_n > cx \}^{1/s} (\mathbb{E} S_n^{r'})^{1/r'}}{x(1-c)}. $$
**Theorem 10.0.8** (Corollary 1 of Giné and Mason (2007a)). Let $\mathcal{F}$ be a collection of measurable functions $f: S^m \to \mathbb{R}$, symmetric in their entries, with absolute values bounded by $M > 0$, and let $P$ be any probability measure on $(S, \mathcal{F})$ (with $X_i$ i.i.d.-$P$). Assume that $\mathcal{F}$ is of VC-type with envelope function $\bar{F} \equiv M$ and with characteristics $A$ and $v$. Then, for every $m \in \mathbb{N}, A \ge e^m, v \ge 1$, there exist constants $C_1 := C_1(m, A, v, M)$ and $C_2 = C_2(m, A, v, M)$ such that for $k = 1, \dots, m$

$$n^k \mathbb{E} \left\| U_n^{(k)} (\pi_k f) \right\|_{\mathcal{F}}^2 \le C_1^2 2^k \sigma^2 \left( \log \frac{A}{\sigma} \right)^k,$$

assuming

$$n\sigma^2 \ge C_2 \log(A/\sigma),$$

where $\sigma^2$ is any number satisfying

$$\|\mathbb{P}^m f^2\|_{\mathcal{F}} \le \sigma^2 \le M^2.$$

**Theorem 10.0.9** (Theorem 5.3.14 of de la Peña and Giné (1999)). Let $\mathcal{H}$ be a VC-subgraph class of uniformly bounded measurable real-valued kernels $H$ on $(S^m, \mathcal{F}^m)$, symmetric in their entries. Then, for each $1 \le k \le m$, there exist constants $c_k, d_k \in ]0, \infty[$ such that, for all $n \ge m$ and $t > 0$,

$$\left\{ \left\| n^{k/2} U_n^{(k)} (\pi_k H) \right\|_{\mathcal{H}} > t \right\} \le c_k \exp \left\{ -d_k t^{2/k} \right\}.$$

## Funding

This research received no external funding.

## Data Availability Statement

Not applicable.

## Conflicts of Interest

The authors declare no conflict of interest.

## References

Abramson, I. S. (1982). On bandwidth variation in kernel estimates—a square root law. *Ann. Statist.*, **10**(4), 1217–1223.

Abrevaya, J. and Jiang, W. (2005a). A nonparametric approach to measuring and testing curvature. *J. Bus. Econom. Statist.*, **23**(1), 1–19.

Abrevaya, J. and Jiang, W. (2005b). A nonparametric approach to measuring and testing curvature. *J. Bus. Econom. Statist.*, **23**(1), 1–19.

Almanjahie, I. M., Bouzebda, S., Kaid, Z., and Laksaci, A. (2022). Nonparametric estimation of expectile regression in functional dependent data. *J. Nonparametr. Stat.*, **34**(1), 250–281.

Arcones, M. A. and Giné, E. (1993). Limit theorems for $U$-processes. *Ann. Probab.*, **21**(3), 1494–1542.

Arcones, M. A. and Wang, Y. (2006). Some new tests for normality based on $U$-processes. *Statist. Probab. Lett.*, **76**(1), 69–82.

Bellet, A. and Habrard, A. (2015). Robustness and generalization for metric learning. *Neurocomputing*, **151**, 259–267.

Bellet, A., Habrard, A., and Sebban, M. (2013). A survey on metric learning for feature vectors and structured data. *CoRR*, abs/1306.6709.

Bergsma, W. and Dassios, A. (2014). A consistent test of independence based on a sign covariance related to Kendall’s tau. *Bernoulli*, **20**(2), 1006–1028.

Bierens, H. J. (1983). Uniform consistency of kernel estimators of a regression function under generalized conditions. *J. Amer. Statist. Assoc.*, **78**(383), 699–707.

Blum, J. R., Kiefer, J., and Rosenblatt, M. (1961). Distribution free tests of independence based on the sample distribution function. *Ann. Math. Statist.*, **32**, 485–498.

Blundell, R., Gosling, A., Ichimura, H., and Meghir, C. (2007). Changes in the distribution of male and female wages accounting for employment composition using bounds. *Econometrica*, **75**(2), 323–363.

Borovkova, S., Burton, R., and Dehling, H. (1999). Consistency of the Takens estimator for the correlation dimension. *Ann. Appl. Probab.*, **9**(2), 376–390.
Borovkova, S., Burton, R., and Dehling, H. (2001). Limit theorems for functionals of mixing processes with applications to U-statistics and dimension estimation. *Trans. Amer. Math. Soc.*, **353**(11), 4261–4318.

Borovskikh, Y. V. (1996). *U-statistics in Banach spaces*. VSP, Utrecht.

Bouzebda, S. (2023a). General tests of conditional independence based on empirical processes indexed by functions. *Jpn. J. Stat. Data Sci.*, **6**(4), 1–63.

Bouzebda, S. (2023b). On the weak convergence and the uniform-in-bandwidth consistency of the general conditional *U*-processes based on the copula representation: multivariate setting. *Hacet. J. Math. Stat.*, **52**(1), 1–46.

Bouzebda, S. and Chaouch, M. (2022). Uniform limit theorems for a class of conditional Z-estimators when covariates are functions. *J. Multivariate Anal.*, **189**, Paper No. 104872, 21.

Bouzebda, S. and El-hadjali, T. (2020). Uniform convergence rate of the kernel regression estimator adaptive to intrinsic dimension in presence of censored data. *J. Nonparametr. Stat.*, **32**(4), 864–914.

Bouzebda, S. and Elhattab, I. (2011). Uniform-in-bandwidth consistency for kernel-type estimators of Shannon's entropy. *Electron. J. Stat.*, **5**, 440–459.

Bouzebda, S. and Nemouchi, B. (2019). Central limit theorems for conditional empirical and conditional *U*-processes of stationary mixing sequences. *Math. Methods Statist.*, **28**(3), 169–207.

Bouzebda, S. and Nemouchi, B. (2020). Uniform consistency and uniform in bandwidth consistency for nonparametric regression estimates and conditional *U*-statistics involving functional data. *J. Nonparametr. Stat.*, **32**(2), 452–509.

Bouzebda, S. and Nemouchi, B. (2023). Weak-convergence of empirical conditional processes and conditional *U*-processes involving functional mixing data. *Stat. Inference Stoch. Process.*, **26**(1), 33–88.

Bouzebda, S. and Nezzal, A. (2022). Uniform consistency and uniform in number of neighbors consistency for nonparametric regression estimates and conditional U-statistics involving functional data. *Jpn. J. Stat. Data Sci.*, **5**(2), 431–533.

Bouzebda, S. and Nezzal, A. (2023). Asymptotic properties of conditional *U*-statistics using delta sequences. *Comm. Statist. Theory Methods*, pages 1–56.

Bouzebda, S. and Soukarieh, I. (2022). Renewal type bootstrap for *U*-process Markov chains. *Markov Process. Related Fields*, **28**(5), 673–735.

Bouzebda, S. and Soukarieh, I. (2023). Non-parametric conditional *U*-processes for locally stationary functional random fields under stochastic sampling design. *Mathematics*, **11**(1), 1–70.

Bouzebda, S. and Taachouche, N. (2023a). Rates of the strong uniform consistency for the kernel-type regression function estimators with general kernels on manifolds. *Math. Methods Statist.*, **32**(1), 27–81.

Bouzebda, S. and Taachouche, N. (2023b). Rates of the strong uniform consistency with rates for conditional *U*-statistics estimators with general kernels on manifolds. *Math. Methods Statist.*, pages 1–65.

Bouzebda, S., Elhattab, I., and Nemouchi, B. (2021). On the uniform-in-bandwidth consistency of the general conditional *U*-statistics based on the copula representation. *J. Nonparametr. Stat.*, **33**(2), 321–358.

Bouzebda, S., Nezzal, A., and Zari, T. (2023a). Uniform consistency for functional conditional u-statistics using delta-sequences. *Mathematics*, **11**(1), 1–39.

Bouzebda, S., El-hadjali, T., and Ferfache, A. A. (2023b). Uniform in bandwidth consistency of conditional *U*-statistics adaptive to intrinsic dimension in presence of censored data. *Sankhya A*, pages 1–59.

Breiman, L., Meisel, W., and Purcell, E. (1977). Variable kernel estimates of multivariate densities. *Technometrics*, **19**(2), 135–144.

Bretagnolle, J. (1983). Lois limites du bootstrap de certaines fonctionnelles. *Ann. Inst. H. Poincaré Sect. B (N.S.)*, **19**(3), 281–296.

Cao, Q., Guo, Z.-C., and Ying, Y. (2016a). Generalization bounds for metric and similarity learning. *Mach. Learn.*, **102**(1), 115–132.

Cao, Q., Guo, Z.-C., and Ying, Y. (2016b). Generalization bounds for metric and similarity learning. *Mach. Learn.*, **102**(1), 115–132.

Carbonez, A., Györfi, L., and van der Meulen, E. C. (1995). Partitioning-estimates of a regression function under random censoring. *Statist. Decisions*, **13**(1), 21–37.

Chacón, J. E., Montanero, J., and Nogales, A. G. (2007). A note on kernel density estimation at a parametric rate. *J. Nonparametr. Stat.*, **19**(1), 13–21.

Chen, X. and Kato, K. (2020). Jackknife multiplier bootstrap: finite sample approximations to the *U*-process supremum with applications. *Probab. Theory Related Fields*, **176**(3-4), 1097–1163.
Chen, Y. and Datta, S. (2019). Adjustments of multi-sample *U*-statistics to right censored data and confounding covariates. *Comput. Statist. Data Anal.*, **135**, 1–14.

Clarkson, J. A. and Adams, C. R. (1933). On definitions of bounded variation for functions of two variables. *Trans. Amer. Math. Soc.*, **35**(4), 824–854.

Clémençon, S. (2014). A statistical view of clustering performance through the theory of *U*-processes. *J. Multivariate Anal.*, **124**, 42–56.

Clémençon, S. and Robbiano, S. (2015). The TreeRank Tournament algorithm for multipartite ranking. *J. Nonparametr. Stat.*, **27**(1), 107–126.

Clémençon, S., Lugosi, G., and Vayatis, N. (2008). Ranking and empirical minimization of *U*-statistics. *Ann. Statist.*, **36**(2), 844–874.

Clémençon, S., Robbiano, S., and Vayatis, N. (2013). Ranking data with ordinal labels: optimality and pairwise aggregation. *Mach. Learn.*, **91**(1), 67–104.

Clémençon, S., Colin, I., and Bellet, A. (2016a). Scaling-up empirical risk minimization: optimization of incomplete *U*-statistics. *J. Mach. Learn. Res.*, **17**, Paper No. 76, 36.

Clémençon, S., Colin, I., and Bellet, A. (2016b). Scaling-up empirical risk minimization: optimization of incomplete *U*-statistics. *J. Mach. Learn. Res.*, **17**, Paper No. 76, 36.

Cover, T. M. and Thomas, J. A. (2006). *Elements of information theory*. Wiley-Interscience [John Wiley & Sons], Hoboken, NJ, second edition.

Csiszár, I. (1962). *Informationstheoretische Konvergenzbegriffe im Raum der Wahrscheinlichkeitsverteilungen*. *Magyar Tud. Akad. Mat. Kutató Int. Közl.*, **7**, 137–158.

Csörgő, M. and Révész, P. (1981). *Strong approximations in probability and statistics*. Probability and Mathematical Statistics. Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London.

Cybis, G. B., Valk, M., and Lopes, S. R. C. (2018). Clustering and classification problems in genetics through *U*-statistics. *J. Stat. Comput. Simul.*, **88**(10), 1882–1902.

Datta, S., Bandyopadhyay, D., and Satten, G. A. (2010). Inverse probability of censoring weighted *U*-statistics for right-censored data with an application to testing hypotheses. *Scand. J. Stat.*, **37**(4), 680–700.

de la Peña, V. H. and Giné, E. (1999). *Decoupling*. Probability and its Applications (New York). Springer-Verlag, New York. From dependence to independence, Randomly stopped processes. *U*-statistics and processes. Martingales and beyond.

Deheuvels, P. (2011). One bootstrap suffices to generate sharp uniform bounds in functional estimation. *Kybernetika (Prague)*, **47**(6), 855–865.

Deheuvels, P. and Mason, D. M. (2004). General asymptotic confidence bands based on kernel-type function estimators. *Stat. Inference Stoch. Process.*, **7**(3), 225–277.

Denker, M. and Keller, G. (1983). On *U*-statistics and v. Mises' statistics for weakly dependent processes. *Z. Wahrsch. Verw. Gebiete*, **64**(4), 505–522.

Derumigny, A. and Fermanian, J.-D. (2019). On kernel-based estimation of conditional Kendall's tau: finite-distance bounds and asymptotic behavior. *Depend. Model.*, **7**(1), 292–321.

Devroye, L. and Lugosi, G. (2001). *Combinatorial methods in density estimation*. Springer Series in Statistics. Springer-Verlag, New York.

Dony, J. and Mason, D. M. (2008). Uniform in bandwidth consistency of conditional *U*-statistics. *Bernoulli*, **14**(4), 1108–1133.

Dudewicz, E. J. and van der Meulen, E. C. (1981). Selection procedures for the best binomial population with generalized entropy goodness. *Tamkang J. Math.*, **12**(2), 205–208.

Dudley, R. M. (1999). *Uniform central limit theorems*, volume 63 of *Cambridge Studies in Advanced Mathematics*. Cambridge University Press, Cambridge.

Dwass, M. (1956). The large-sample power of rank order tests in the two-sample problem. *Ann. Math. Statist.*, **27**, 352–374.

Dynkin, E. B. and Mandelbaum, A. (1983). Symmetric statistics, Poisson point processes, and multiple Wiener integrals. *Ann. Statist.*, **11**(3), 739–745.

Ebrahimi, N., Habibullah, M., and Soofi, E. (1992). Testing exponentiality based on Kullback-Leibler information. *J. Roy. Statist. Soc. Ser. B*, **54**(3), 739–748.
Efromovich, S. (1999). *Nonparametric curve estimation*. Springer Series in Statistics. Springer-Verlag, New York. Methods, theory, and applications.

Einmahl, U. and Mason, D. M. (2000). An empirical process approach to the uniform consistency of kernel-type function estimators. *J. Theoret. Probab.*, **13**(1), 1–37.

Einmahl, U. and Mason, D. M. (2005). Uniform in bandwidth consistency of kernel-type function estimators. *Ann. Statist.*, **33**(3), 1380–1403.

Ellison, G. and Ellison, S. F. (2011). Strategic entry deterrence and the behavior of pharmaceutical incumbents prior to patent expiration. *American Economic Journal: Microeconomics*, **3**(1), 1–36.

Epanechnikov, V. A. (1969). Nonparametric estimation of a multidimensional probability density. *Teor. Verojatnost. i Primenen.*, **14**, 156–162.

Faivishevsky, L. and Goldberger, J. (2009). Ica based on a smooth estimation of the differential entropy. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, *Advances in Neural Information Processing Systems*, volume 21. Curran Associates, Inc.

Filippova, A. A. (1959). Mises theorem on the limit behaviour of functionals derived from empirical distribution functions. *Dokl. Akad. Nauk SSSR*, **129**, 44–47.

Földes, A. and Rejtő, L. (1981). A LIL type result for the product limit estimator. *Z. Wahrsch. Verw. Gebiete*, **56**(1), 75–86.

Frees, E. W. (1989). Infinite order U-statistics. *Scand. J. Statist.*, **16**(1), 29–45.

Gao, J. and Gijbels, I. (2008). Bandwidth selection in nonparametric kernel testing. *J. Amer. Statist. Assoc.*, **103**(484), 1584–1594.

Ghosal, S., Sen, A., and van der Vaart, A. W. (2000). Testing monotonicity of regression. *Ann. Statist.*, **28**(4), 1054–1082.

Giné, E. and Mason, D. M. (2007a). Laws of the iterated logarithm for the local U-statistic process. *J. Theoret. Probab.*, **20**(3), 457–485.

Giné, E. and Mason, D. M. (2007b). On local U-statistic processes and the estimation of densities of functions of several sample variables. *Ann. Statist.*, **35**(3), 1105–1145.

Giné, E. and Nickl, R. (2016). *Mathematical foundations of infinite-dimensional statistical models*. Cambridge Series in Statistical and Probabilistic Mathematics, [40]. Cambridge University Press, New York.

Giné, E. and Sang, H. (2010). Uniform asymptotics for kernel density estimators with variable bandwidths. *J. Nonparametr. Stat.*, **22**(5-6), 773–795.

Giné, E. and Sang, H. (2013). On the estimation of smooth densities by strict probability densities at optimal rates in sup-norm. In *From probability to statistics and back: high-dimensional models and processes*, volume 9 of *Inst. Math. Stat. (IMS) Collect.*, pages 128–149. Inst. Math. Statist., Beachwood, OH.

Gokhale, D. V. and Kullback, S. (1978). *The information in contingency tables*, volume 23 of *Statistics: Textbooks and Monographs*. Marcel Dekker, Inc., New York.

Györfi, L., Kohler, M., Krzyżak, A., and Walk, H. (2002). *A distribution-free theory of nonparametric regression*. Springer Series in Statistics. Springer-Verlag, New York.

Hall, P. and Marron, J. S. (1988). Variable window width kernel estimates of probability densities. *Probab. Theory Related Fields*, **80**(1), 37–49.

Hall, P., Hu, T. C., and Marron, J. S. (1995). Improved variable window kernel estimates of probability densities. *Ann. Statist.*, **23**(1), 1–10.

Halmos, P. R. (1946). The theory of unbiased estimation. *Ann. Math. Statistics*, **17**, 34–43.

Hardy, G. H. (1905). On double fourier series and especially those which represent the double zeta-function with real and incommensurable parameters. *Quart. J. Math*, **37**(1), 53–79.

Harel, M. and Puri, M. L. (1996). Conditional U-statistics for dependent random variables. *J. Multivariate Anal.*, **57**(1), 84–100.

Heilig, C. and Nolan, D. (2001). Limit theorems for the infinite-degree U-process. *Statist. Sinica*, **11**(1), 289–302.

Hobson, E. W. (1958). *The theory of functions of a real variable and the theory of Fourier's series*. Vol. II. Dover Publications, Inc., New York, N.Y.

Hoeffding, W. (1948). A class of statistics with asymptotically normal distribution. *Ann. Math. Statistics*, **19**, 293–325.

Hollander, M. and Proschan, F. (1972). Testing whether new is better than used. *Ann. Math. Statist.*, **43**, 1136–1146.
Horowitz, J. L. and Spokoiny, V. G. (2001). An adaptive, rate-optimal test of a parametric mean-regression model against a nonparametric alternative. *Econometrica*, **69**(3), 599–631.

Huber, P. J. (1964). Robust estimation of a location parameter. *Ann. Math. Statist.*, **35**, 73–101.

Janson, S. (1990). A functional limit theorem for random graphs with applications to subgraph count statistics. *Random Structures Algorithms*, **1**(1), 15–37.

Jaynes, E. T. (1957). Information theory and statistical mechanics. *Phys. Rev.* (2), **106**, 620–630.

Jin, R., Wang, S., and Zhou, Y. (2009). Regularized distance metric learning:theory and algorithm. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, *Advances in Neural Information Processing Systems*, volume 22. Curran Associates, Inc.

Joly, E. and Lugosi, G. (2016). Robust estimation of U-statistics. *Stochastic Process. Appl.*, **126**(12), 3760–3773.

Jones, M. (1990). Variable kernel density estimates and variable kernel density estimates. *Australian Journal of Statistics*, **32**(3), 361–371.

Jones, M. C. (1995). On higher order kernels. *J. Nonparametr. Statist.*, **5**(2), 215–221.

Jones, M. C., McKay, I. J., and Hu, T.-C. (1994). Variable location and scale kernel density estimation. *Ann. Inst. Statist. Math.*, **46**(3), 521–535.

Jones, M. C., Linton, O., and Nielsen, J. P. (1995). A simple bias reduction method for density estimation. *Biometrika*, **82**(2), 327–338.

Kaplan, E. L. and Meier, P. (1958). Nonparametric estimation from incomplete observations. *J. Amer. Statist. Assoc.*, **53**, 457–481.

Kendall, M. G. (1938). A new measure of rank correlation. *Biometrika*, **30**(1/2), 81–93.

Klebanov, I. (2018). Axiomatic approach to variable kernel density estimation. *arXiv preprint arXiv:1805.01729*.

Koenker, R. and Bassett, Jr., G. (1978). Regression quantiles. *Econometrica*, **46**(1), 33–50.

Kohler, M., Máthé, K., and Pintér, M. (2002). Prediction from randomly right censored data. *J. Multivariate Anal.*, **80**(1), 73–100.

Kosorok, M. R. (2008). *Introduction to empirical processes and semiparametric inference*. Springer Series in Statistics. Springer, New York.

Krause, M. (1903). Über Mittelwertsätze im Gebiete der Doppelsummen und Doppelintegrale. Leipzig. Ber. 55, 239-263 (1903).

Kullback, S. (1959). *Information theory and statistics*. John Wiley & Sons, Inc., New York; Chapman & Hall, Ltd., London.

Lax, P. D. and Terrell, M. S. (2017). *Multivariable calculus with applications*. Undergraduate Texts in Mathematics. Springer, Cham.

Lee, A. J. (1990). *U-statistics*, volume 110 of *Statistics: Textbooks and Monographs*. Marcel Dekker Inc., New York. Theory and practice.

Lee, S., Linton, O., and Whang, Y.-J. (2009). Testing for stochastic monotonicity. *Econometrica*, **77**(2), 585–602.

Lehmann, E. L. (1951). A general concept of unbiasedness. *Ann. Math. Statistics*, **22**, 587–592.

Leucht, A. (2012). Degenerate *U*- and *V*-statistics under weak dependence: asymptotic theory and bootstrap consistency. *Bernoulli*, **18**(2), 552–585.

Leucht, A. and Neumann, M. H. (2013). Degenerate *U*- and *V*-statistics under ergodicity: asymptotics, bootstrap and applications in statistics. *Ann. Inst. Statist. Math.*, **65**(2), 349–386.

Lim, F. and Stojanovic, V. M. (2013). On U-statistics and compressed sensing I: Non-asymptotic average-case analysis. *IEEE Trans. Signal Process.*, **61**(10), 2473–2485.

Litimein, O., Laksaci, A., Mechab, B., and Bouzebda, S. (2023). Local linear estimate of the functional expectile regression. *Statist. Probab. Lett.*, **192**, Paper No. 109682.

Liu, Q., Lee, J., and Jordan, M. (2016). A kernelized stein discrepancy for goodness-of-fit tests. In M. F. Balcan and K. Q. Weinberger, editors, *Proceedings of The 33rd International Conference on Machine Learning*, volume 48 of *Proceedings of Machine Learning Research*, pages 276–284, New York, New York, USA. PMLR.

Maillot, B. and Viallon, V. (2009). Uniform limit laws of the logarithm for nonparametric estimators of the regression function in presence of censored data. *Math. Methods Statist.*, **18**(2), 159–184.

McKay, I. (1993a). A note on bias reduction in variable-kernel density estimates. *Canad. J. Statist.*, **21**(4), 367–375.
McKay, I. J. (1993b). *Variable kernel methods in density estimation*. ProQuest LLC, Ann Arbor, MI. Thesis (Ph.D.)–Queen's University (Canada).

Mohammed, M., Bouzebda, S., and Laksaci, A. (2020). On the nonparametric estimation of the functional expectile regression. *C. R. Math. Acad. Sci. Paris*, **358**(3), 267–272.

Mohammed, M., Bouzebda, S., and Laksaci, A. (2021). The consistency and asymptotic normality of the kernel type expectile regression estimator for functional data. *J. Multivariate Anal.*, **181**, Paper No. 104673, 24.

Muller, H.-G. and Stadtmuller, U. (1987). Variable bandwidth kernel estimators of regression curves. *The Annals of Statistics*, pages 182–201.

Nadaraja, E. A. (1964). On a regression estimate. *Teor. Verojatnost. i Primenen.*, **9**, 157–159.

Nakarmi, J. and Sang, H. (2018). Central limit theorem for the variable bandwidth kernel density estimators. *J. Korean Statist. Soc.*, **47**(2), 201–215.

Nakarmi, J., Sang, H., and Ge, L. (2021). Variable bandwidth kernel regression estimation. *ESAIM Probab. Stat.*, **25**, 55–86.

Newey, W. K. and Powell, J. L. (1987). Asymmetric least squares estimation and testing. *Econometrica*, **55**(4), 819–847.

Niederreiter, H. (1992). *Random number generation and quasi-Monte Carlo methods*, volume 63 of CBMS-NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA.

Nolan, D. and Pollard, D. (1987). *U*-processes: rates of convergence. *Ann. Statist.*, **15**(2), 780–799.

Novak, S. Y. (1999). A generalized kernel density estimator. *Teor. Veroyatnost. i Primenen.*, **44**(3), 634–645.

Peng, W., Coleman, T., and Mentch, L. (2022). Rates of convergence for random forests via generalized U-statistics. *Electron. J. Stat.*, **16**(1), 232–292.

Piterbarg, V. I. (1996). *Asymptotic methods in the theory of Gaussian processes and fields*, volume 148 of *Translations of Mathematical Monographs*. American Mathematical Society, Providence, RI. Translated from the Russian by V. V. Piterbarg, Revised by the author.

Pollard, D. (1984). *Convergence of stochastic processes*. Springer Series in Statistics. Springer-Verlag, New York.

Prakasa Rao, B. L. S. and Sen, A. (1995). Limit distributions of conditional *U*-statistics. *J. Theoret. Probab.*, **8**(2), 261–301.

Rejchel, W. (2012). On ranking and generalization bounds. *J. Mach. Learn. Res.*, **13**, 1373–1392.

Rényi, A. (1959). On the dimension and entropy of probability distributions. *Acta Math. Acad. Sci. Hungar.*, **10**, 193–215 (unbound insert).

Rubin, H. and Vitale, R. A. (1980). Asymptotic distribution of symmetric statistics. *Ann. Statist.*, **8**(1), 165–170.

Schick, A., Wang, Y., and Wefelmeyer, W. (2011). Tests for normality based on density estimators of convolutions. *Statist. Probab. Lett.*, **81**(2), 337–343.

Sen, A. (1994). Uniform strong consistency rates for conditional *U*-statistics. *Sankhyā Ser. A*, **56**(2), 179–194.

Serfling, R. J. (1980). *Approximation theorems of mathematical statistics*. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, Inc., New York.

Shang, H. L. (2014). Bayesian bandwidth estimation for a functional nonparametric regression model with mixed types of regressors and unknown error density. *J. Nonparametr. Stat.*, **26**(3), 599–615.

Shannon, C. E. (1948). A mathematical theory of communication. *Bell System Tech. J.*, **27**, 379–423, 623–656.

Sherman, R. P. (1993). The limiting distribution of the maximum rank correlation estimator. *Econometrica*, **61**(1), 123–137.

Sherman, R. P. (1994). Maximal inequalities for degenerate *U*-processes with applications to optimization estimators. *Ann. Statist.*, **22**(1), 439–459.

Silverman, B. W. (1978). Distances on circles, toruses and spheres. *J. Appl. Probability*, **15**(1), 136–143.

Silverman, B. W. (1986). *Density estimation for statistics and data analysis*. Monographs on Statistics and Applied Probability. Chapman & Hall, London.

Solon, G. (1992). Intergenerational income mobility in the united states. *The American Economic Review*, **82**(3), 393–408.

Song, Y., Chen, X., and Kato, K. (2019). Approximating high-dimensional infinite-order *U*-statistics: statistical and computational guarantees. *Electron. J. Stat.*, **13**(2), 4794–4848.
Soukarieh, I. and Bouzebda, S. (2022). Exchangeably weighted bootstraps of general markov *U*-process. *Mathematics*, **10**(20), 1–42.

Soukarieh, I. and Bouzebda, S. (2023). Renewal type bootstrap for increasing degree *U*-process of a Markov chain. *J. Multivariate Anal.*, **195**, Paper No. 105143.

Stute, W. (1986). Conditional empirical processes. *Ann. Statist.*, **14**(2), 638–647.

Stute, W. (1991). Conditional *U*-statistics. *Ann. Probab.*, **19**(2), 812–825.

Stute, W. (1993). Almost sure representations of the product-limit estimator for truncated data. *Ann. Statist.*, **21**(1), 146–156.

Stute, W. (1994a). *L*^p-convergence of conditional *U*-statistics. *J. Multivariate Anal.*, **51**(1), 71–82.

Stute, W. (1994b). Universally consistent conditional *U*-statistics. *Ann. Statist.*, **22**(1), 460–473.

Stute, W. (1996). Symmetrized NN-conditional *U*-statistics. In *Research developments in probability and statistics*, pages 231–237. VSP, Utrecht.

Stute, W. and Wang, J.-L. (1993). Multi-sample *U*-statistics for censored data. *Scand. J. Statist.*, **20**(4), 369–374.

Suaray, K. (2011). Variable bandwidth kernel density estimation for censored data. *Journal of Statistical Theory and Practice*, **5**, 221–229.

Talagrand, M. (1994). Sharper bounds for Gaussian and empirical processes. *Ann. Probab.*, **22**(1), 28–76.

Terrell, G. R. and Scott, D. W. (1992). Variable kernel density estimation. *Ann. Statist.*, **20**(3), 1236–1265.

van der Vaart, A. W. (1998). *Asymptotic statistics*, volume 3 of *Cambridge Series in Statistical and Probabilistic Mathematics*. Cambridge University Press, Cambridge.

van der Vaart, A. W. and Wellner, J. A. (1996). *Weak convergence and empirical processes*. Springer Series in Statistics. Springer-Verlag, New York. With applications to statistics.

Vasicek, O. (1976). A test for normality based on sample entropy. *J. Roy. Statist. Soc. Ser. B*, **38**(1), 54–59.

Verdugo Lazo, A. C. G. and Rathie, P. N. (1978). On the entropy of continuous probability distributions. *IEEE Trans. Inform. Theory*, **IT-24**(1), 120–122.

Vitali, G. (1908). Sui gruppi di punti e sulle funzioni di variabili reali. *Torino Atti*, **43**, 229–246.

Vitu vskin, A. G. (1955). *O mnogomernyh variaciyah*. Gosudarstv. Izdat. Tehn.-Teor. Lit., Moscow.

von Mises, R. (1947). On the asymptotic distribution of differentiable statistical functions. *Ann. Math. Statistics*, **18**, 309–348.

Watson, G. S. (1964). Smooth regression analysis. *Sankhyā Ser. A*, **26**, 359–372.

Wu, T.-J., Chen, C.-F., and Chen, H.-Y. (2007). A variable bandwidth selector in multivariate kernel density estimation. *Statistics & probability letters*, **77**(4), 462–467.

Yuan, A., Giurcanu, M., Luta, G., and Tan, M. T. (2017). U-statistics with conditional kernels for incomplete data models. *Ann. Inst. Statist. Math.*, **69**(2), 271–302.