# On the variable bandwidth kernel estimation of conditional U -statistics at optimal rates in sup-norm

Salim Bouzebda, Nourelhouda Taachouche

► To cite this version:

Salim Bouzebda, Nourelhouda Taachouche. On the variable bandwidth kernel estimation of conditional U -statistics at optimal rates in sup-norm. Physica A: Statistical Mechanics and its Applications, 2023, 625, pp.129000. 10.1016/j.physa.2023.129000. hal-04152611

HAL Id: hal-04152611

https://inria.hal.science/hal-04152611v1

Submitted on 9 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# On the variable bandwidth kernel estimation of conditional U-statistics at optimal rates in sup-norm

Salim BOUZEBDA,\* and Nourelhouda TAACHOUCHE†

Université de technologie de Compiègne,
LMAC (Laboratory of Applied Mathematics of Compiègne),
CS 60 319 - 60 203 Compiègne Cedex

## Abstract

U-statistics represent a fundamental class of statistics from modelling quantities of interest defined by multi-subject responses. U-statistics generalize the empirical mean of a random variable X to sums over every k-tuple of distinct observations of X. W. Stute [Ann. Probab. 19 (1991) 812-825] introduced a class of so-called conditional U-statistics, which may be viewed as a generalization of the Nadaraya-Watson estimates of a regression function. Stute proved their strong pointwise consistency to:

$$r^{(k)}(\varphi, \mathbf{t}) = \mathbb{E}[\varphi(Y_1, \dots, Y_k) | (X_1, \dots, X_k) = \mathbf{t}], \text{ for } \mathbf{t} \in \mathbb{R}^{pk}.$$

In this paper we introduce a variable bandwidth kernel conditional U-statistics estimator for i.i.d. observations in $\mathbb{R}^{pq}$ to improve the classical Stute estimator. We apply in nontrivial way the methods developed in Dony and Mason [Bernoulli 14(4), 2008, 1108-1133] to establish uniform in $\mathbf{t}$ and in bandwidth consistency to $r^{(k)}(\varphi, \mathbf{t})$ of the proposed estimator. Our estimator includes density estimators proposed by McKay [Canad. J. Statist. 21 (1993) 367-375; Variable kernel methods in density estimation (1993)] as a particular case. In addition, uniform consistency is also established over $\varphi \in \mathcal{F}$ for a suitably restricted class $\mathcal{F}$, in both cases bounded and unbounded, satisfying some moment conditions. Our theorems allow data-driven local bandwidths for the considered estimators. Moreover, in the same context, we show the uniform bandwidth consistency for the nonparametric inverse probability of censoring weighted (I.P.C.W.) estimators of the regression function under random censorship, which is of its own interest. The theoretical uniform consistency results established in this paper are (or will be) key tools for many further developments in regression analysis. These results are proved under some standard structural conditions on the Vapnik-Chervonenkis classes of functions and some mild conditions on the model. The applications of our main results include, among many others, the discrimination problems, the metric learning, the Kendall rank correlation coefficient, the generalized U-Statistics and the set indexed conditional U-statistics. Finally, the finite-sample performance of the proposed method is investigated in a simulation study.

**Keywords:** Uniform in bandwidth; Nonparametric estimation; regression-type models; U-statistics; Empirical process methods; U-process; variable bandwidth; bias reduction; kernel estimation; machine learning problems.

**Mathematics Subject Classification:** 60F05; 60G15; 60G10; 62G05; 62G07; 62H12.

## 1 Introduction

Motivated by numerous applications, the theory of U-statistics (introduced in the seminal work by Hoeffding (1948)) and U-processes has received considerable attention in the past decades. U-statistics are useful for solving complex statistical problems. Examples are density estimation, non-parametric regression tests and goodness-of-fit tests. In particular, U-statistics play a role in the analysis of estimators (including function estimators) with varying degrees of smoothness. For example, Stute (1993) applies a.s. uniform bounds for $\mathbb{P}$-canonical U-processes to the analysis of the product limit estimator for truncated data. Arcones and Wang (2006) present two new tests for normality based on U-processes. Making use of the results of Giné and Mason (2007a,b), Schick et al. (2011) introduced new tests for normality which use as test statistics weighted $L_1$-distances between the standard normal density and local U-statistics based on standardized observations. Joly and Lugosi (2016) discussed the estimation of the mean of multivariate functions in case of possibly heavy-tailed distributions and introduce the median-of-means, that is based on U-statistics. U-processes are important tools for a broad range of statistical applications such as testing for qualitative features of functions in nonparametric statistics [Lee et al. (2009), Ghosal et al. (2000), Abrevaya and Jiang (2005a)], cross-validation for density estimation Nolan and Pollard (1987), and establishing limiting distributions of M-estimators [see, e.g., Arcones and Giné (1993), Sherman (1993), Sherman (1994), de la Peña and Giné (1999)]. U-statistics also naturally appear in other contexts, like in the theory of random graphs where they count occurrences of certain subgraphs, e.g., triangles, for instance, see Janson (1990). In a wide variety of problems that received a good deal of attention in the machine learning literature and ranging from clustering to image recognition through ranking or learning on graphs, natural estimates of the risk are not basic sample means but take the U-statistics form, refer to Clémençon et al. (2016a). The ranking problem can be considered as a pairwise classification, and the empirical ranking error of any given prediction rule is a U-statistic of order 2, refer to Clémençon et al. (2008), just like the within-cluster point scatter in cluster analysis, considered in Clémençon (2014), or empirical performance measures in metric learning, refer to Cao et al. (2016a) for instance. For

<footnote>\* Email: salim.bouzebda@utc.fr</footnote>

<footnote>† Email: nourelhouda.taachouche@utc.fr</footnote>
the U-statistics with random kernels of diverging orders we refer to Frees (1989), Heilig and Nolan (2001), Song et al. (2019) and Soukarieh and Bouzebda (2023). Infinite-order U-statistics are useful for constructing simultaneous prediction intervals that quantify the uncertainty of ensemble methods such as subbagging and random forests; for more details on the subject, refer to Peng et al. (2022). The MeanNN approach estimation for differential entropy introduced by Faivishevsky and Goldberger (2009) is a particular case of the U-statistic. Using U-statistics, Liu et al. (2016) proposed a new test statistic for goodness-of-fit tests. Cybis et al. (2018) have explored a model-free approach for clustering and classifying genetic data based on U-statistics that lead to alternative ways of looking at these problems. Cybis et al. (2018) were motivated by the fact that the U-statistics are versatile enough to be applied to various genetic problems and adaptable enough to consider the specificities of different datatypes. Lim and Stojanovic (2013) proposed using the U-statistics, in a natural way, for analyzing random compressed sensing matrices, in the non-asymptotic regime. Halmos (1946), Hoeffding (1948), who provided (amongst others) the first asymptotic results for the case that the underlying random variables are independent and identically distributed, refer also to Rubin and Vitale (1980), Serfling (1980), Dynkin and Mandelbaum (1983), Bretagnolle (1983). von Mises (1947) and Filippova (1959) obtained a similar result for V statistics. Under weak dependency assumptions asymptotic results are for instance shown in Borovkova et al. (2001), in Denker and Keller (1983) or more recently in Leucht (2012) and in more general setting in Leucht and Neumann (2013). For excellent resource of references on the U-statistics and U-processes the interested reader may refer to Borovskikh (1996), Lee (1990), and Arcones and Giné (1993). A very deep insight into the theory of U-processes is given by de la Peña and Giné (1999). For recent references on the subject, we refer to Soukarieh and Bouzebda (2022, 2023). There are two perspectives on U-processes: 1) they are infinite-dimensional versions of U-statistics (with one kernel); 2) they are stochastic processes that are nonlinear generalizations of empirical processes. Both views are useful in that: 1) statistically, it is of greater interest to consider a rich class of statistics rather than a single statistic; 2) mathematically, we can borrow the insights from empirical process theory to derive limit or approximation theorems for U-processes. Importantly, however, 1) extending U-statistics to U-processes requires substantial efforts and different techniques; and 2) generalization from empirical processes to U-processes is highly nontrivial especially when U-processes are not weakly convergent as processes. In this paper we consider the so-called conditional U-statistics introduced by Stute (1991). These statistics may be viewed as generalizations of the Nadaraya-Watson (Nadaraja (1964) and Watson (1964)) estimates of a regression function.

To be more precise, let us consider the sequence of independent and identically distributed random vectors $\{(\mathbf{X}_i, \mathbf{Y}_i) : i \in \mathbb{N}^*\}$ with $\mathbf{X}_i \in \mathbb{R}^p$ and $\mathbf{Y}_i \in \mathbb{R}^q$. Let $\varphi: \mathbb{R}^{qk} \to \mathbb{R}$ be a measurable function. In this paper, we are primarily concerned with the estimation of the conditional expectation, or regression function :

$$r^{(k)}(\varphi, \mathbf{t}) = \mathbb{E}(\varphi(\mathbf{Y}_1, \dots, \mathbf{Y}_k) | (\mathbf{X}_1, \dots, \mathbf{X}_k) = \mathbf{t}), \quad \text{for } \mathbf{t} \in \mathbb{R}^{pk}, \qquad (1.1)$$

whenever it exists, i.e., $\mathbb{E}(|\varphi(\mathbf{Y}_1, \dots, \mathbf{Y}_k)|) < \infty$. We now introduce a kernel function $K: \mathbb{R}^p \to \mathbb{R}$ with support contained in $[-B, B]^p$, $B > 0$, satisfying :

$$\sup_{\mathbf{t} \in \mathbb{R}^p} |K(\mathbf{t})| =: \kappa < \infty \quad \text{and} \quad \int K(\mathbf{t}) d\mathbf{t} = 1. \qquad (1.2)$$

Stute (1991) introduced a class of estimators for $r^{(k)}(\varphi, \mathbf{t})$, called conditional U-statistics, which is defined for each $\mathbf{t} \in \mathbb{R}^{pk}$ to be :

$$\hat{r}_n^{(k)}(\varphi, \mathbf{t}; h_n) = \frac{\sum_{(i_1, \dots, i_k) \in I(k,n)} \varphi(\mathbf{Y}_{i_1}, \dots, \mathbf{Y}_{i_k}) K\left(\frac{\mathbf{t}_1 - \mathbf{X}_{i_1}}{h_n}\right) \cdots K\left(\frac{\mathbf{t}_k - \mathbf{X}_{i_k}}{h_n}\right)}{\sum_{(i_1, \dots, i_k) \in I(k,n)} K\left(\frac{\mathbf{t}_1 - \mathbf{X}_{i_1}}{h_n}\right) \cdots K\left(\frac{\mathbf{t}_k - \mathbf{X}_{i_k}}{h_n}\right)}, \qquad (1.3)$$

where

$$I(k,n) = \{\mathbf{i} = (i_1, \dots, i_k) : 0 \le i_j \le n \text{ and } i_j \ne i_r \text{ if } j \ne r\},$$

is the set of all k-tuples of different integers between 1 and n and $\{h_n\}_{n \ge 1}$ is a sequence of positive constants converging to zero at the rate $nh_n^{pk} \to \infty$. The work of Sen (1994) was devoted to estimate the rate of the uniform convergence in $\mathbf{t}$ of $\hat{r}_n^{(k)}(\varphi, \mathbf{t}; h_n)$ to $r^{(k)}(\varphi, \mathbf{t})$.

In the paper of Prakasa Rao and Sen (1995), the limit distributions of $\hat{r}_n^{(k)}(\varphi, \mathbf{t}; h_n)$ are discussed and compared with those obtained by Stute. Harel and Puri (1996) extend the results of Stute (1991), under appropriate mixing conditions, to weakly dependent data and have applied their findings to verify the Bayes risk consistency of the corresponding discrimination rules. Stute (1996) proposed symmetrized nearest neighbor conditional U-statistics as alternatives to the usual kernel-type estimators. An important contribution is given in the paper Dony and Mason (2008) where a much stronger form of consistency holds, namely, uniform in $\mathbf{x}$ and in bandwidth consistency (i.e., $h_n, h_n \in [a_n, b_n]$ where $a_n < b_n \to 0$ at some specific rate) of $\hat{r}_n^{(k)}(\varphi, \mathbf{t}; h_n)$. In addition, uniform consistency is also established over $\varphi \in \mathcal{F}$ for a suitably restricted class $\mathcal{F}$, extended in Bouzebda and Nemouchi (2019, 2020, 2023), Bouzebda and Soukarieh (2022, 2023), Bouzebda et al. (2021, 2023a,b) and Bouzebda and Taachouche (2023b). In the particular case $k = 1$, the $r^{(k)}(\varphi, \mathbf{t})$ is reduced to $r^{(1)}(\varphi, \mathbf{t}) = \mathbb{E}(\varphi(\mathbf{Y})|\mathbf{X} = \mathbf{t})$ and Stute's estimator becomes the Nadaraya-Watson estimator of $r^{(1)}(\varphi, \mathbf{t})$ given by :

$$\hat{r}_n^{(1)}(\varphi, \mathbf{t}, h_n) = \frac{\sum_{i=1}^{n} \varphi(\mathbf{Y}_i) K\left(\frac{\mathbf{X}_i - \mathbf{t}}{h_n}\right)}{\sum_{i=1}^{n} K\left(\frac{\mathbf{X}_i - \mathbf{t}}{h_n}\right)}. $$

The uniform consistency of the Nadaraya-Watson estimator was shown in Bierens (1983) for the case of discrete X, for more recent references see Bouzebda and Taachouche (2023a) and Bouzebda (2023a).
In this paper, we study more general version of the ideal<fnref n="1" /> variable bandwidth kernel density estimator (VBKDE) introduced by McKay (1993a,b) and Jones et al. (1994) and the plug-in practical version of the variable bandwidth kernel estimator with two sequences of bandwidths as Giné and Sang (2013); Nakarmi and Sang (2018); Nakarmi et al. (2021). The model of variable kernel density estimator was first proposed by Breiman et al. (1977), who introduced the variable bandwidths for each data point in the given dataset and represented the bandwidth with distance from the data point to its k-th nearest neighbor. Giné and Sang (2010, 2013) studied plug-in variable bandwidth kernel density estimator for one and d-dimensional observations. They proved that the discrepancy between the plug-in estimator and the true value converges uniformly over a data-adaptive region at a rate of $O_{a.s.}(\log n/n)^{4/(8+d)}$ by applying empirical process techniques. Nakarmi et al. (2021) proposed a variable bandwidth kernel regression estimator for i.i.d. observations in $\mathbb{R}^2$ to improve the classical Nadaraya-Watson estimator. They established the central limit theorems for the proposed ideal and true variable kernel regression estimators. Motivated by the works on variable bandwidth kernel density estimation and the regression function, in this paper, to improve the accuracy of Stute's estimator (1.3) by introducing the variable bandwidth estimator (2.10), defined below. To our best knowledge, this problem was open up to present, and it gives the main motivation to our paper.

In the last decades, empirical process theory has provided very useful and powerful tools to analyze the large sample properties of several nonparametric estimators of functionals of the distribution, such as the regression function and the density function, refer to, van der Vaart and Wellner (1996) and Kosorok (2008). Nolan and Pollard (1987) were the first to introduce the notion of uniform in bandwidth consistency for kernel density estimators and they applied empirical process methods in their study. The present extends our previous works in several directions; the main one is that the present paper considers the conditional U-statistics, including the regression functions. More precisely, a uniform in bandwidth consistency results for some general kernel-type estimators are established. However, as will be seen later, the problem requires much more than “simply” combining ideas from the existing results. Delicate mathematical derivations will be required to cope with the empirical processes that we consider in this extended setting. In addition, we will consider the nonparametric Inverse Probability of Censoring Weighted (I.P.C.W.) estimators of the multivariate conditional U-statistic under random censorship and obtain uniform in bandwidth consistency results that are of independent interest.

The layout of present article is structured as follows. In section 2, we describe the framework of the present paper and provide the necessary notations that we need with technical assumptions. In section 3, we give the main results of the paper concerning the exponential inequalities and the limiting law. In Section 4, we provide an important application for the censored data setting that is of its own interest. In Section 5, we provide some applications including the discrimination in §5.1, the metric learning in §5.2, the multipartite ranking in §5.3, the Ranking problems in §5.4, Kendall rank correlation coefficient §5.5, generalized U-Statistics §5.6, set indexed conditional U-statistics §5.7 and testing stochastic monotonicity problems in §5.8. In Section 5.10, we collect some examples of a class of functions and examples of conditional U-statistics. In Section 6, we discussed the bandwidth selection criterion briefly. Section 7 summarizes extensive numerical experiments demonstrating the good finite-sample properties of the proposed method. Some concluding remarks and possible future developments are relegated to 8. All proofs are gathered in Section 9 to prevent interrupting the presentation flow. A few relevant technical results are given in the Appendix.

## 2 Preliminaries and estimation procedure

Let $(\mathbf{X}, \mathbf{Y})$ be a random vector, with $\mathbf{X} = (X_1, \dots, X_p) \in \mathbb{R}^p$ and $\mathbf{Y} = (Y_1, \dots, Y_q) \in \mathbb{R}^q$. We denote the joint distribution function [df] of $(\mathbf{X}, \mathbf{Y})$ by $F(\mathbf{x}, \mathbf{y}) = \mathbb{P}(\mathbf{X} \le \mathbf{x}, \mathbf{Y} \le \mathbf{y})$, for $\mathbf{x} \in \mathbb{R}^p$ and $\mathbf{y} \in \mathbb{R}^q$. Here, for $\mathbf{v}' = (v'_1, \dots, v'_r) \in \mathbb{R}^r$ and $\mathbf{v}'' = (v''_1, \dots, v''_r) \in \mathbb{R}^r$, we set $\mathbf{v}' \le \mathbf{v}''$ whenever $v'_j \le v''_j$ for $j = 1, \dots, r$. We assume that $(\mathbf{X}, \mathbf{Y})$ has a joint density $f_{\mathbf{X},\mathbf{Y}}(\mathbf{x}, \mathbf{y}) = \frac{\partial^{p+q}}{\partial x_1 \dots \partial x_p \partial y_1 \dots \partial y_q} F(\mathbf{x}, \mathbf{y})$ on $\mathbf{J} \times \mathbb{R}^q$, with respect to the Lebesgue measure $d\mathbf{x} \times d\mathbf{y}$, and denote by

$$f_{\mathbf{X}}(\mathbf{x}) = \int_{\mathbb{R}^q} f(\mathbf{x}, \mathbf{y}) d\mathbf{y} \quad \text{for} \quad \mathbf{x} \in \mathbf{J}$$

the density of $\mathbf{X}$ (which is only assumed to exist on $\mathbf{J}$). We next consider a sequence $\{(\mathbf{X}_i, \mathbf{Y}_i) : i \ge 1\}$ of independent and identically distributed [iid] random copies of the random vector [rv]$(\mathbf{X}, \mathbf{Y})$. For each $n \ge 1$, and for each choice of the bandwidth $h_{1,n} > 0$, we define the kernel estimator by

$$f_{\mathbf{X},n}(\mathbf{t}; h_{1,n}) = \frac{1}{nh_{1,n}^p} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{1,n}}\right). \qquad (2.1)$$

Let us recall the McKay (1993a,b)'s estimators

$$\bar{f}_{\mathbf{X};n}(\mathbf{t}; h_{2,n}) = \frac{1}{nh_{2,n}^p} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X}}(\mathbf{X}_i))\right) \alpha^p(f_{\mathbf{X}}(\mathbf{X}_i)), \qquad (2.2)$$

where $\alpha(s)$ is a smooth function of the form

$$\alpha(s) := cp^{1/2} (s/c^2). \qquad (2.3)$$

The function $p(\cdot)$ has at least fourth order derivative and satisfies $p(x) \ge 1$ for all $x$ and $p(x) = x$ for all $x \ge t_0$ for some $1 \le t_0 < \infty$, and a fixed number $c$, where $0 < c < \infty$. Eq. (2.2) is a variable bandwidth kernel density estimator since the bandwidth has form $h_n/\alpha(f(X_i))$ if we rewrite (2.2) in the form of the classical one. The plug-in estimator in Giné and Sang (2013) has the following representation

$$\hat{f}(\mathbf{t}; h_{1,n}, h_{2,n}) = \frac{1}{nh_{2,n}^p} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n})). \qquad (2.4)$$

<footnote n="1">¹The ideal estimator will only be a means to study the 'true' estimator, obtained from the ideal by replacement of $f_{\mathbf{X}}(\cdot)$ by a preliminary estimator.</footnote>
The research into variable bandwidth kernel density estimation dates back to Abramson (1982). Abramson (1982) introduced the following estimator

$$f_A(\mathbf{t}; h_n) = \frac{1}{nh_n^p} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_n} \gamma(\mathbf{t}, \mathbf{X}_i)\right) \gamma^p(\mathbf{t}, \mathbf{X}_i), \quad (2.5)$$

where $\gamma(\mathbf{t}, \mathbf{s}) = (f_X(\mathbf{s}) \vee f_X(\mathbf{t}))/10^{1/2}$. The bandwidth $h_n/\gamma(\mathbf{t}, \mathbf{X}_i)$ at each observation $\mathbf{X}_i$ is inversely proportional to $f^{1/2}(\mathbf{X}_i)$ if $f_X(\mathbf{X}_i) \ge f_X(\mathbf{t})/10$. Notice that (2.2) also has the square root law since $\alpha(f_X(\mathbf{X}_i)) = f_X^{1/2}(\mathbf{X}_i)$ if $f_X(\mathbf{X}_i) \ge t_0 c^2$ by the definition of the function $\mathbf{p}(\mathbf{x})$. The estimator (2.2) or (2.5) has clipping procedure in (2.3) or $\gamma(\mathbf{t}, \mathbf{s})$ since they make the true bandwidth $h_n/\alpha(f_X(\mathbf{X}_i)) \ge h_n/c$ or $h_n/\gamma(\mathbf{t}, \mathbf{X}_i) \ge 10^{1/2}h_n/f(\mathbf{t})^{1/2}$. The clipping procedures prevent too much contribution to the density estimation at $\mathbf{t}$ if the observation $\mathbf{X}_i$ is too far away from $\mathbf{t}$. Abramson showed that this square root law and the clipping procedure improve the bias from the order of $h_n^2$ to the order of $h_n^4$ for the estimator (2.5) while at the same time keep the variance at the order of $(nh_n^d)^{-1}$ if $f_X(\mathbf{t}) \ne 0$ and $f(\mathbf{x})$ has fourth order continuous derivatives at $\mathbf{t}$. So, one has a non-negative estimator of the density that performs asymptotically as a kernel estimator based on a fourth order (hence, partly negative) kernel. However, this variable bandwidth estimator (2.5) is not a density function of a true probability measure since the integral of $f_A(\mathbf{t}; h_n)$ over $\mathbf{t}$ is not 1. McKay (1993a) and Terrell and Scott (1992) proved that the following modification of the Abramson estimator without the 'clipping filter' $(f_X(\mathbf{t})/10)^{1/2}$ on $f_X^{1/2}(\mathbf{X}_i)$ studied in Hall and Marron (1988),

$$f_{\text{HM}}(\mathbf{t}; h_n) = \frac{1}{nh_n^p} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_n} h_n^{-1} f_X^{1/2}(\mathbf{X}_i)\right) f_X^{p/2}(\mathbf{X}_i), \quad (2.6)$$

which has integral 1 and thus is a true probability density, may have bias of order much larger than $h_n^4$. Therefore, the clipping is necessary for such bias reduction. In the case $d=1$, Hall et al. (1995) proposed the estimator

$$f_{\text{HHM}}(\mathbf{t}; h_n) = \frac{1}{nh_n} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_n} f_X^{1/2}(\mathbf{X}_i)\right) f_X^{1/2}(\mathbf{X}_i) I(|\mathbf{t} - \mathbf{X}_i| < h_n B) \quad (2.7)$$

where $B$ is a fixed constant; see also Novak (1999) for a similar estimator. This estimator is non-negative and achieves the desired bias reduction but, like Abramson's, it does not integrate to 1. In conclusion, it seems that the estimator (2.2) has all the advantages: it is a true density function with square root law and smooth clipping procedure. However, notice that this estimator and all the other variable bandwidth kernel density estimators are not applicable in practice since they all include the studied density function $f(\cdot)$. Therefore, we call them ideal estimators in the literature. Hall and Marron (1988) studied a true density estimator

$$\hat{f}_{\text{HM}}(\mathbf{t}; h_{1,n}, h_{2,n}) = \frac{1}{nh_{2,n}^p} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} f_{\mathbf{X};n}^{1/2}(\mathbf{t}; h_{1,n})\right) f_{\mathbf{X};n}^{p/2}(\mathbf{t}; h_{1,n}),$$

by plugging in a pilot estimator, the classical estimator (2.1), into the estimator (2.6). Here the bandwidth sequence $h_{2,n}$ is the $h_n$ as in (2.6) and the bandwidth sequence $h_{1,n}$ is applied in the classical kernel density estimator (2.1). Jones (1990) clarified the difference between VBKDE employing a different bandwidth for each data point and VBKDE with bandwidth as a function of estimation location. Wu et al. (2007) proposed a strategy to express the variable bandwidth in VBKDE as the product of a local bandwidth factor and a global smoothing parameter. Suaray (2011) proposed a VBKDE for the p.d.f. estimation of censored data. Klebanov (2018) showed an axiomatic approach to construct a VBKDE, which guaranteed the density estimation invariance under linear transformations of original density as well as under splitting of density into several well-separated parts. Muller and Stadtmuller (1987) studied kernel regression estimation with fixed design points $\{x_i\}_{i=1}^n$ and variable bandwidth depending on the estimation point $\mathbf{t}$.
First of all, define

$$\mathcal{D}_r := \{\mathbf{t} \in \mathbb{R}^p : f(\mathbf{t}) > r > t_0 c^2, \|\mathbf{t}\| < 1/r\}, \quad r > 0, \quad (2.8)$$

$$\text{and} \qquad \hat{\mathcal{D}}_r^n := \{\mathbf{t} \in \mathbb{R}^p : f_{\mathbf{X};n}(\mathbf{t}; h_{1,n}) > 2r > t_0 c^2, \|\mathbf{t}\| < 1/r\}, \quad r > 0. \quad (2.9)$$

Here, $c$ and $t_0$ are the constants in the clipping function $\alpha(\cdot)$ in (2.3). Our results will be uniform over the above regions $\mathcal{D}_r$ and $\hat{\mathcal{D}}_r^n$. We use the notation $\tilde{\mathbf{t}} = (\mathbf{t}_1, \dots, \mathbf{t}_k) \in \mathbb{I}^k$, where $\mathbb{I} \in \mathcal{D}_r$. We are mainly interested in studying the uniform in $\tilde{\mathbf{t}}$ and the uniform in bandwidth consistency of the following $U$-statistic

$$\check{\gamma}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) = \frac{\sum_{(i_1, \dots, i_k) \in I(k,n)} \varphi(\mathbf{Y}_{i_1}, \dots, \mathbf{Y}_{i_k}) \prod_{j=1}^k K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))}{\sum_{(i_1, \dots, i_k) \in I(k,n)} \prod_{j=1}^k K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))} \quad (2.10)$$

to the regression function $r^{(k)}(\varphi, \tilde{\mathbf{t}}) := \mathbb{E}(\varphi(\mathbf{Y}_1, \dots, \mathbf{Y}_k) | (\mathbf{X}_1, \dots, \mathbf{X}_k) = \tilde{\mathbf{t}})$, where $\varphi: \mathbb{R}^{qk} \to \mathbb{R}$ is a measurable function belonging to some class of functions $\mathcal{F}$, and $h_{1,n}$ and $h_{2,n}$ are sequences of positive real numbers that belong to the intervals $[a_n, b_n]$ and $[\tilde{a}_n, \tilde{b}_n]$ respectively with

$$\lim_{n \to \infty} a_n = \lim_{n \to \infty} \tilde{a}_n = \lim_{n \to \infty} b_n = \lim_{n \to \infty} \tilde{b}_n = 0.$$
This means that, with probability 1,

$$ \limsup_{n \to \infty} \sup_{a_n \le h_{1,n} \le b_n} \sup_{\tilde{a}_n \le h_{2,n} \le \tilde{b}_n} \sup_{\tilde{t} \in \mathbb{I}^k} \left| \tilde{r}_n^{(k)}(\varphi, \tilde{t}; h_{1,n}, h_{2,n}) - r^{(k)}(\varphi, \tilde{t}) \right| = 0. \quad (2.11) $$

Moreover, we shall show that (2.11) also holds uniformly in $\varphi \in \mathcal{F}$ for certain classes of functions $\mathcal{F}$.

**Remark 2.0.1.** Giné and Sang (2013) provided a five time differentiable clipping function $\mathfrak{p}(\cdot)$ with $t_0 = 2$.

$$ \mathfrak{p}(t) = \begin{cases} 1 + \frac{t^6}{64} \left(1 - 2(t-2) + \frac{9}{4}(t-2)^2 - \frac{7}{4}(t-2)^3 + \frac{7}{8}(t-2)^4\right), & \text{if } 0 \le t \le 2, \\ t, & \text{if } t \ge 2, \\ 1, & \text{if } t \le 0. \end{cases} $$

We will use this clipping function in the simulation study.

## 2.1 Notation and Assumptions

For the reader's convenience, we have gathered the essential notation and assumptions.

### 2.1.1 Notation

For $v = (v_1, \dots, v_d) \in (\mathbb{N} \cup \{0\})^d$ and vector $u = (u_1, \dots, u_d)^T$, set

$$ \begin{align} |v| &= \sum_{i=1}^{d} v_i, \quad v! = v_1! \cdots v_d!, \notag \\ D_v &= D_{u_1}^{v_1} \circ \cdots \circ D_{u_d}^{v_d}, \quad u^v = u_1^{v_1} \cdots u_d^{v_d}, \tag{2.12} \\ \tau_v &= \int_{\mathbb{R}^d} u^v K(u) du, \quad \mu_v = \int_{\mathbb{R}^d} u^v K^2(u) du, \notag \end{align} $$

where $D_v$ means that we take $v_1$ partial order derivatives on the first coordinate, $v_2$ partial order derivatives on the second coordinate, until we take $v_d$ partial order derivatives on the $d$-th coordinate. We shall use also, throughout the paper, the notation

$$ \begin{align*} \mathbf{X} &= (\mathbf{X}_1, \dots, \mathbf{X}_k) \in \mathbb{R}^{pk} && \text{and} && \mathbf{X}_i := (\mathbf{X}_{i1}, \dots, \mathbf{X}_{i\ell}) \in \mathbb{R}^{p\ell}, \quad \mathbf{i} \in \mathcal{I}_n^k, \\ \mathbf{Y} &= (\mathbf{Y}_1, \dots, \mathbf{Y}_k) \in \mathbb{R}^{kq} && \text{and} && \mathbf{Y}_i := (\mathbf{Y}_{i1}, \dots, \mathbf{Y}_{i\ell}) \in \mathbb{R}^{q\ell}, \quad \mathbf{i} \in \mathcal{I}_n^k. \end{align*} $$

### 2.1.2 Assumptions

**Assumption 1.** [Kernel assumptions] Let the kernel $K$ below the following assumptions :

* $K(\cdot)$ is a symmetric kernel function with support contained in $[-1/2, 1/2]$ satisfying

$$ \int K(\mathbf{z}) d\mathbf{z} = 1 \qquad (K.i) $$

and

$$ \sup_{\mathbf{x} \in \mathbb{R}^p} |K(\mathbf{x})| =: \kappa < \infty \qquad (K.ii) $$

* $K(\cdot)$ is of order $s$, that is,

$$ \begin{align} \int_{\mathbb{R}^p} u_1^{j_1} \dots u_p^{j_p} K(\mathbf{u}) d\mathbf{u} = 0, \quad j_1, \dots, j_p \ge 0, \quad j_1 + \dots + j_p = 1, \dots, s-1, \tag{K.iii} \\ \int_{\mathbb{R}^p} |u_1^{j_1} \dots u_p^{j_p}| K(\mathbf{u}) d\mathbf{u} < \infty, \quad j_1, \dots, j_p \ge 0, \quad j_1 + \dots + j_p = s. \notag \end{align} $$

**Assumption 2.** [Class of functions assumptions]

The classes of functions $\mathcal{F}$ is such that :

* $\mathcal{F}$ is a pointwise measurable class, i.e., there exists a countable subclass $\mathcal{F}_0$ of $\mathcal{F}$ such that we can find, for every function $g \in \mathcal{F}$, a sequence of functions $\{g_n\} \in \mathcal{F}_0$ for which

$$ g_n(\mathbf{y}_1, \dots, \mathbf{y}_k) \to g(\mathbf{y}_1, \dots, \mathbf{y}_k), \quad (\mathbf{y}_1, \dots, \mathbf{y}_k) \in \mathbb{R}^{kq}. \qquad (F.i) $$
• $\mathcal{F}$ has a (measurable) envelope function $F$, i.e.,

$$ \sup_{\phi \in \mathcal{F}} |\phi(\mathbf{y}_1, \dots, \mathbf{y}_k)| \le F(\mathbf{y}_1, \dots, \mathbf{y}_k), \quad (\mathbf{y}_1, \dots, \mathbf{y}_k) \in \mathbb{R}^{kq}, \qquad (F.ii) $$

• $\mathcal{F}$ is supposed to be of VC-type with envelope function previously defined. Hence, there are two finite constants $A$ and $b$ such that:

$$ \mathcal{N}(\mathcal{F}, L_2(\mathbb{Q}), \varepsilon) \le \left( \frac{A \|F\|_{L_2(\mathbb{Q})}}{\varepsilon} \right)^b, \quad \text{for } 0 < \varepsilon \le 2 \|F\|_{L_2(\mathbb{Q})}, \qquad (F.iii) $$

• $\mathcal{F}$ is bounded and its envelope function satisfy for some $0 < \mathbb{M} < \infty$:

$$ F(\mathbf{y}_1, \dots, \mathbf{y}_k) \le \mathbb{M}, \quad (\mathbf{y}_1, \dots, \mathbf{y}_k) \in \mathbb{R}^{kq}. \qquad (F.iv) $$

• $\mathcal{F}$ is unbounded and its envelope function satisfies for some $\mathbf{v} > 2$:

$$ \theta_{\mathbf{v}} := \sup_{(\mathbf{t}_1, \dots, \mathbf{t}_k) \in \mathbb{I}^k} \mathbb{E}(F^{\mathbf{v}}(\mathbf{Y}) | \mathbf{X} = (\mathbf{t}_1, \dots, \mathbf{t}_k)) < \infty. \qquad (F.v) $$

### 2.1.3 Comments

Some comments on the assumptions are in order. The conditions (K.i)-(K.ii) are classical in the non-parametric kernel estimations. Note that the fact that $K(\cdot)$ is a symmetric kernel function with support contained in $[-1/2, 1/2]$ and (K.ii), for $p = 1$, imply that for any $h > 0$,

$$ \frac{1}{h_n} \int |K|(u/h_n)du \le \kappa. $$

By imposing the condition (K.iii), the kernel function exploits the smoothness of the density function or the regression function. It is well known that the best available rate of convergence of the kernel estimator, in the AMISE sense, is of order $n^{-4/5}$, in the univariate case. The convergence rate could be faster if we lose the condition that the kernel function $K(\cdot)$ must be a density. Indeed, the convergence rate can be made arbitrarily close to the parametric $n^{-1}$ as the order increases. In fact, Chacón et al. (2007) showed that the parametric rate $n^{-1}$ can be attained by using super kernels, and those super kernel density estimators automatically adapt to the unknown degree of smoothness of the density. The main drawback of higher-order kernels in this situation is that the negative contributions of the kernel may make the estimated density, not the density itself. The interested reader may refer to, e.g., Jones et al. (1995), and Jones (1995). Our results remain valid when replacing the condition that the kernel function $K(\cdot)$ has compact support with another condition whose content is as follows:

• There exists a sequence of positive real numbers $a_n$ such that $a_n h_n^d$ tends to zero when $n$ tends to infinity, and

$$ \sqrt{n} \int_{\{\|\mathbf{v}\| > a_n\}} |K(\mathbf{v})| d\mathbf{v} \to 0. $$

Assumption (F.i) is made to avoid measurability difficulties. Our definition of “pointwise measurability” is borrowed from example 2.3.4 in van der Vaart and Wellner (1996), (Giné and Nickl, 2016, p.262) calls a pointwise measurable function class satisfying the pointwise countable approximation property. This condition is discussed in (van der Vaart and Wellner, 1996, Example 2.3.4. p 110) and (Kosorok, 2008, §8.2. p. 110) and it is satisfied whenever $g(\cdot)$ is right continuous. Assumption (F.iii) ensures that $\mathcal{F}$ is VC type with characteristics $A$ and $b$ for envelope $F$. Since $F \in L^2(P)$ by assumption (F.iii), it is seen from Dudley's criterion on sample continuity of Gaussian processes (see, e.g., [Giné and Nickl (2016), Theorem 2.3.7]) that the function class $\mathcal{F}$ is $\mathbb{P}$-pre-Gaussian, i.e., there exists a tight Gaussian random variable $W_P$ in $l^\infty(\mathcal{F})$ with mean zero and covariance function

$$ \mathbb{E}[W_P(g)W_P(g')] = \text{Cov}(g(X_1), g'(X_1)), g, g' \in \mathcal{F}. $$

For Assumption (F.iii), see (Pollard, 1984, Examples 26 and 38), (Nolan and Pollard, 1987, Lemma 22), (Dudley, 1999, §4.7.), (van der Vaart and Wellner, 1996, Theorem 2.6.7), (Kosorok, 2008, §9.1) provide a number of sufficient conditions under which (K.vi) holds, we may also refer to (Deheuvels, 2011, §3.2) for further discussions. For instance, it is satisfied, for general $d \ge 1$, whenever $g(\mathbf{x}) = \phi(p(\mathbf{x}))$, with $p(\mathbf{x})$ being a polynomial in $d$ variables and $\phi(\cdot)$ being a real-valued function of bounded variation, we refer the reader to (Einhorn and Mason, 2005, p. 1381). We also mention that Remark that condition (K.vi) is satisfied whenever $K(\cdot)$ is of bounded variation on $\mathbb{R}^d$ (in the sense of Hardy and Kauser (Hardy (1905), Krause (1903), Vitali (1908)), see, e.g., Clarkson and Adams (1933), Vitu vskin (1955), Hobson (1958) and Niederreiter (1992)).
# 3 Main results

## 3.1 Uniform in bandwidth consistency of the regression function

In this section, we establish the uniform consistency in the bandwidth of the estimator of regression in its general form, which is given by

$$ \check{r}_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n}) = \frac{\sum_{i=1}^{n} \varphi(\mathbf{Y}_i) K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))}{\sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))} \quad (3.1) $$

to the regression function $r^{(1)}(\varphi, \mathbf{t}) := \mathbb{E}(\varphi(\mathbf{Y}_1) | (\mathbf{X}_1) = \mathbf{t})$. The uniform consistency that we wish establish in terms of $\mathbf{t}$, in terms of $\varphi$ and in terms of $h_{1,n}$ and $h_{2,n}$ is considered when $\mathbf{t}$ belongs to some subset $\mathbf{I} \subset \mathcal{D}_r$, given in (2.8), $\varphi$ belongs to some classes of functions $\mathcal{F}$ and $h_{1,n}$ and $h_{2,n}$ to some intervals $[a_n, b_n]$ and $[\tilde{a}_n, \tilde{b}_n]$ respectively, which means that:

$$ \limsup_{n \to \infty} \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \left| \check{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - r^{(1)}(\varphi, \mathbf{t}) \right| = 0 \quad \text{a.s.} \quad (3.2) $$

Denote

$$ U_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n}) := \frac{1}{nh_{2,n}} \sum_{i=1}^{n} \varphi(\mathbf{Y}_i) K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n})). $$

So, we have

$$ \check{r}_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n}) = \frac{U_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n})}{U_n^{(1)}(1, \mathbf{t}; h_{1,n}, h_{2,n})}. $$

The main results of this section to be proved here can now be stated as follows. In the following theorem, we give the main result concerning the uniform consistency when the class of functions is bounded.

**Theorem 3.1.1.** Under Assumptions (K.i), (K.ii), (F.i), (F.ii), (F.iii) and (F.iv), we have

$$ \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \left| U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - \mathbb{E} U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) \right| = O\left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}} \right) \quad \text{a.s.} \quad (3.3) $$

In the following theorem, we give the main result concerning the uniform consistency when the class of functions is unbounded.

**Theorem 3.1.2.** Under Assumptions (K.i), (K.ii), (F.i), (F.ii), (F.iii) and (F.v), we have

$$ \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \left| U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - \mathbb{E} U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) \right| = O\left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}} \right) \quad \text{a.s.} \quad (3.4) $$

**Theorem 3.1.3.** Under Assumptions (K.i), (K.ii), (K.iii), (F.i), (F.ii), (F.iii), (F.iv) and (G.i), we get

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \left| \mathbb{E} U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - r^{(1)}(\varphi, \mathbf{t}) f_{\mathbf{X}}(\mathbf{t}) \right| = O(\tilde{b}_n^4 + b_n^2 \tilde{b}_n^2). \quad (3.5) $$

In the sequel, we need the following additional assumption for our analysis.

* There exists a constant $0 < \mathcal{C} < \infty$ such that the function $g(\varphi, \cdot) := r^{(1)}(\varphi, \cdot) f_{\mathbf{X}}(\cdot)$ satisfies

$$ \sup_{\mathbf{u} \in \mathbf{I}} \left| \frac{\partial^s g(\varphi, \mathbf{u})}{\partial^{j_1} u_1 \dots \partial^{j_p} u_p} \right| \le \mathcal{C}, \quad j_1 + \dots + j_p = s. \quad (G.i) $$

**Corollary 3.1.4.** Under Assumptions of Theorems 3.1.1, 3.1.2 and 3.1.3, we get

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \left| U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - r^{(1)}(\varphi, \mathbf{t}) f_{\mathbf{X}}(\mathbf{t}) \right| = O\left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}} \right) + O(\tilde{b}_n^4 + b_n^2 \tilde{b}_n^2) \quad \text{a.s.} \quad (3.6) $$

By setting $\varphi \equiv 1$ in the last corollary, we obtain the following result, which we think is new in the literature. In this case, we obtain a result for the density estimator treated in Giné and Sang (2013). The main difference is that our result is uniform in bandwidth.
**Corollary 3.1.5.** However, we have enriched these results with an additional uniformity in bandwidths $h_{1,n}$ and $h_{2,n}$ in their theorems. So, for $\mathbf{I} \subset \mathcal{D}_r$, we have

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} |\hat{f}(\mathbf{t}; h_1, h_2) - f_{\mathbf{X}}(\mathbf{t})| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) + O(\tilde{b}_n^4 + b_n^2 \tilde{b}_n^2) \quad \text{a.s.} $$

and, for $\hat{\mathbf{I}} \subset \hat{\mathcal{D}}_r^n$,

$$ \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \hat{\mathbf{I}}} |\hat{f}(\mathbf{t}; h_1, h_2) - f_{\mathbf{X}}(\mathbf{t})| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) + O(\tilde{b}_n^4) \quad \text{a.s.,} $$

where

$$ \hat{f}(\mathbf{t}; h_1, h_2) = \frac{1}{nh_2^p} \sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_2} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_1))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_1)). $$

Now, to prove the strong consistency of $\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)$, we shall consider another, but more appropriate and more computationally convenient, centering factor than the expectation $\mathbb{E}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)$, which is delicate to handle. This is given by

$$ \hat{\mathbb{E}}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) := \frac{\mathbb{E}\left(\varphi(\mathbf{Y})K\left(\frac{\mathbf{t}-\mathbf{X}}{h_2}\alpha(f_{\mathbf{X};n}(\mathbf{X};h_1))\right)\alpha^p(f_{\mathbf{X};n}(\mathbf{X};h_1))\right)}{\mathbb{E}\left(K\left(\frac{\mathbf{t}-\mathbf{X}}{h_2}\alpha(f_{\mathbf{X};n}(\mathbf{X};h_1))\right)\alpha^p(f_{\mathbf{X};n}(\mathbf{X};h_1))\right)} =: \frac{\mathbb{E}(r_n(\varphi, \mathbf{t}; h_1, h_2))}{\mathbb{E}(f_n(\mathbf{t}; h_1, h_2))}. \quad (3.7) $$

**Remark 3.1.6.** Note that $\hat{\mathbb{E}}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)$ does not coincide in general with $\mathbb{E}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)$. However, under mild regularity assumptions, the difference between these two quantities becomes asymptotically negligible.

The following result handles the uniform deviation of the estimate $\hat{\mathbb{E}}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)$ with respect to $\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)$.

**Corollary 3.1.7.** Assume that the conditions of Theorems 3.1.1 and 3.1.2 hold and the envelope function $F$ satisfies (F.iv) or (F.v) according to the class $\mathcal{F}$ is bounded or unbounded. Then, we have with probability 1

$$ \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} |\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - \hat{\mathbb{E}}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) \quad \text{a.s.} \quad (3.8) $$

The following result handles the uniform deviation of the estimate $\hat{\mathbb{E}}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2)$ with respect to $r^{(1)}(\varphi, \mathbf{t})$.

**Corollary 3.1.8.** Assume that the conditions of Theorem 3.1.3 hold. Then, we have with $b_n \to 0$, $\tilde{b}_n \to 0$ and with probability 1

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} |\hat{\mathbb{E}}\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - r^{(1)}(\varphi, \mathbf{t})| = O(\tilde{b}_n^4 + b_n^2 \tilde{b}_n^2). \quad (3.9) $$

**Corollary 3.1.9.** Assume that the conditions of Theorems 3.1.1, 3.1.2 and 3.1.3. Then, we have with probability 1

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} |\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - r^{(1)}(\varphi, \mathbf{t})| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) + O(\tilde{b}_n^4 + b_n^2 \tilde{b}_n^2) \quad \text{a.s.} \quad (3.10) $$

Making use of the fact that

$$ \|f_{\mathbf{X},n}(\cdot; h_{1,n}) - f_{\mathbf{X}}(\cdot)\|_{\infty} \le O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right), $$

we infer that $\hat{\mathcal{D}}_r^n \subset \mathcal{D}_r$, for $\hat{\mathbf{I}} \subset \hat{\mathcal{D}}_r^n$. We, therefore, obtain the following result.

**Corollary 3.1.10.** Under Assumptions of Theorems 3.1.1, 3.1.2 and 3.1.3, we get

$$ \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \hat{\mathbf{I}}} \sup_{\varphi \in \mathcal{F}} |\tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) - r^{(1)}(\varphi, \mathbf{t})| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) + O(\tilde{b}_n^4) \quad \text{a.s.} \quad (3.11) $$
**Remark 3.1.11.** If we set, for $n \in \mathbb{N}$,

$$h_{1,n} = ((\log n)/n)^{1/(4+p)} \qquad (3.12)$$

$$h_{2,n} = ((\log n)/n)^{1/(8+p)}. \qquad (3.13)$$

Then, the estimator $\tilde{r}_n^{(1)}(\varphi, t; h_{1,n}, h_{2,n})$, given by (3.1), satisfies, for $\mathbf{I} \subset \mathcal{D}_r$ and $\hat{\mathbf{I}} \subset \hat{\mathcal{D}}_r^n$,

$$\sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \left| \tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n}) - r^{(1)}(\varphi, \mathbf{t}) \right| = O_{\text{a.s.}} \left( \left( \frac{\log n}{n} \right)^{4/(8+p)} \right),$$

and

$$\sup_{\mathbf{t} \in \hat{\mathbf{I}}} \sup_{\varphi \in \mathcal{F}} \left| \tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n}) - r^{(1)}(\varphi, \mathbf{t}) \right| = O_{\text{a.s.}} \left( \left( \frac{\log n}{n} \right)^{4/(8+p)} \right).$$

It is known (Hall et al. (1995); Novak (1999)) that the bias reduction does hold for $f_X(\cdot)$ and $K(\cdot)$ four times differentiable and that then one has a bias of the order of $h_{2,n}^4$. This leads almost immediately in the case of the ideal estimator, and with some relatively hard work in the case of the real estimator, to an a.s. rate of convergence of $\hat{f}(t; h_{1,n}, h_{2,n}) - f(t)$ (fixed $t$) of the order of $n^{-4/9}$ if we take $h_{2,n} \simeq n^{-1/9}$ and $h_{1,n} \simeq n^{-2/9}$ or of a smaller order, and this is best possible ($n^{-4/9}$ is the minimax rate in the $L_2(\mathbb{P})$-norm for estimating $f(t)$ four times differentiable with continuity). In this case, the sup-norm's minimax rate is $(n/\log n)^{-4/9}$. Giné and Sang (2013) showed that this rate is achieved by the estimator (Hall et al. (1995)) uniformly in $D_r$ and in a similar data-dependent region (see, e.g., Efromovich (1999), for minimax rates.) In the particular case of $p=1$, Giné and Sang (2013) have a similar convergence rate for the variable kernel regression estimator.

### 3.1.1 Kernel estimator of the conditional distribution function

By setting $\varphi(\mathbf{Y}) = 1\{\mathbf{Y} \le \mathbf{t}\}$, for $\mathbf{t} \in \mathbb{R}^q$, into (3.1), we obtain the kernel estimator of the conditional distribution function $F(\mathbf{y} | \mathbf{X} = \mathbf{t})$

$$F_{n;h_{1,n},h_{2,n}}(\mathbf{y} | \mathbf{x}) = \frac{\sum_{i=1}^{n} 1\{\mathbf{Y}_i \le \mathbf{y}\} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))}{\sum_{i=1}^{n} K\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))}. \qquad (3.14)$$

Stute (1986) calls this the conditional empirical distribution function and was the first to establish uniform consistency results for it. An application of Corollary 3.1.10 gives the following immediate result.

**Corollary 3.1.12.** Under Assumptions of Corollary 3.1.10, we get

$$\sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\mathbf{y} \in \mathbb{R}^q} \left| F_{n;h_{1,n},h_{2,n}}(\mathbf{t} | \mathbf{x}) - F(\mathbf{y} | \mathbf{X} = \mathbf{t}) \right| = O \left( \sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}} \right) + O(\tilde{b}_n^4 + \tilde{b}_n^2 \tilde{b}_n^2) \quad \text{a.s.} \qquad (3.15)$$

**Remark 3.1.13.** (Expectile regression). For $p \in (0, 1)$, let $\psi(T - \boldsymbol{\theta}) = (p - \{T - \boldsymbol{\theta} \le 0\})|T - \boldsymbol{\theta}|$, then the zero of $m(\psi, \cdot)$ with respect to $\boldsymbol{\theta}$ leads to quantities named expectiles by Newey and Powell (1987). Expectiles, as represented by Newey and Powell (1987), may be introduced either as a generalization of the mean or as an alternative to quantiles. Clearly, classical regression provides us with a high sensitivity to extreme values, allowing for more reactive risk management. Quantile regression, on the other hand, provides the ability to obtain exhaustive information on the effect of the explanatory variable on the response variable by examining its conditional distribution; refer to Mohammedi et al. (2020, 2021); Almanjahie et al. (2022); Litimein et al. (2023) for further details on expectiles in functional data setting.

**Remark 3.1.14.** (Quantile regression). For $p \in (0, 1)$, let $\psi(T - \boldsymbol{\theta}) = p - \{T - \boldsymbol{\theta} \le 0\}$. Then then the zero of $m(\psi, \cdot)$ with respect to $\boldsymbol{\theta}$ is the conditional p-th quantile, initially introduced in Koenker and Bassett (1978) in the real and linear framework, for more general setting, refer to Bouzebda and Chaouch (2022).

**Remark 3.1.15.** (Conditional winsorized mean). As in Huber (1964), if we consider $\psi(T - \boldsymbol{\theta}) = -k, T - \boldsymbol{\theta}, k$ if $T - \boldsymbol{\theta} < -k, |T - \boldsymbol{\theta}| \le k, \text{ or } T - \boldsymbol{\theta} > k$, then the zero of $m(\psi, \cdot)$ with respect to $\boldsymbol{\theta}$ will be the conditional winsorized mean. Notably, this parameter was not considered in the literature on nonparametric functional data analysis involving wavelet estimators. Our paper offers asymptotic results for the conditional winsorized mean when the covariates are functions.

## 3.2 Kernel estimator of the Shannon's entropy

The differential (or Shannon) entropy of $f(\cdot)$ is defined to be

$$H(f) := - \int_{\mathbb{R}^d} f(\mathbf{x}) \log(f(\mathbf{x})) d\mathbf{x} \qquad (3.16)$$
$$ := - \int_{\mathbb{R}^d} \log(f(\mathbf{x})) d\mathbb{F}(\mathbf{x}), \qquad (3.17) $$

whenever this integral is meaningful, and where, for $\mathbf{x} = (x_1, \dots, x_d)$, $d\mathbf{x}$ denotes Lebesgue measure in $\mathbb{R}^d$. We will use the convention that $0 \log(0) = 0$ since $u \log(u) \to 0$ as $u \to 0$. The concept of differential entropy was originally introduced in Shannon's paper Shannon (1948). Since this early epoch, the notion of entropy has been the subject of great theoretical and applied interest. We refer to (Cover and Thomas, 2006, Chapter 8.) for a comprehensive overview of differential entropy and their mathematical properties. Entropy concepts and principles play a fundamental role in many applications, such as quantization theory Rényi (1959), statistical decision theory Kullback (1959), and contingency table analysis Gokhale and Kullback (1978). Csiszár (1962) introduced the concept of convergence in entropy and showed that the latter convergence concept implies convergence in $\mathcal{L}_1$. This property indicates that entropy is a useful concept to measure “closeness in distribution”, and also justifies heuristically the usage of sample entropy as test statistics when designing entropy-based tests of goodness-of-fit. This line of research has been pursued by Vasicek (1976); Dudewicz and van der Meulen (1981); Ebrahimi et al. (1992) [including the references therein]. The idea here is that many families of distributions are characterized by maximization of entropy subject to constraints (see, e.g., Jaynes (1957) and Verdugo Lazo and Rathie (1978)). Given $\hat{f}(\cdot; h_{1,n}, h_{2,n})$ in (2.4), we estimate $H(f)$ using the representation (3.16), by setting

$$ H_{n,h_{1,n},h_{2,n}}(f) := - \int_{A_n} \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \log (\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})) d\mathbf{x}, \qquad (3.18) $$

where

$$ \mathbf{A}_n := \{\mathbf{x} \in \mathbb{R}^d : \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \ge \gamma_n\}, $$

and $\gamma_n \downarrow 0$ is a sequence of positive constant. To prove the strong consistency of $H_{n,h_n}(f)$, we shall consider another, but more appropriate and more computationally convenient, centering factor than the expectation $\mathbb{E}H_{n,h_n}(f)$, which is delicate to handle. This is given by

$$ \hat{\mathbb{E}}H_{n,h_{1,n},h_{2,n}}(f) := - \int_{A_n} \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \log (\mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})) d\mathbf{x}. $$

We first decompose $H_{n,h_{1,n},h_{2,n}}(f) - \hat{\mathbb{E}}H_{n,h_{1,n},h_{2,n}}(f)$ into the sum of two components, by writing

$$ \begin{aligned} H_{n,h_{1,n},h_{2,n}}(f) - \hat{\mathbb{E}}H_{n,h_{1,n},h_{2,n}}(f) & = - \int_{A_n} \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \log (\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})) d\mathbf{x} \\ & \quad + \int_{A_n} \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \log (\mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})) d\mathbf{x} \\ & = - \int_{A_n} \left\{ \log \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) - \log \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \right\} \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) d\mathbf{x} \\ & \quad - \int_{A_n} \left\{ \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) - \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \right\} \log \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) d\mathbf{x} \\ & := \mathbf{\Delta}_{1,n,h_n} + \mathbf{\Delta}_{2,n,h_n}. \end{aligned} \qquad (3.19) $$

We observe that for all $z > 0$,

$$ |\log z| \le \left| \frac{1}{z} - 1 \right| + |z - 1|. $$

Therefore, for any $\mathbf{x} \in \mathbf{A}_n$, we get

$$ \begin{aligned} & |\log \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) - \log \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})| = \left| \log \frac{\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})}{\mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})} \right| \\ & \le \left| \frac{\mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})}{\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})} - 1 \right| + \left| \frac{\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})}{\mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})} - 1 \right| \\ & = \frac{\left| \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) - \hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \right|}{\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})} + \frac{\left| f_{\mathbf{x};n}(\mathbf{x}; h_n) - \mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n}) \right|}{\mathbb{E}\hat{f}(\mathbf{x}; h_{1,n}, h_{2,n})}. \end{aligned} $$

Under conditions of Corollary 3.1.5, in addition we assume, $cn^{-1}\gamma_n^{-4}(\log n) \le h_n$, then we can prove the following result, which is an extension of Theorem 2.1 of Bouzebda and Elhattab (2011), there exists a positive constant $\Upsilon$, such that

$$ \limsup_{n \to \infty} \sup_{h_n \le h_{2,n} \le 1} \frac{\sqrt{nh^d \gamma_n^4 |H_{n,h_{1,n},h_{2,n}}(f) - \hat{\mathbb{E}}H_{n,h_{1,n},h_{2,n}}(f)|}}{\sqrt{(\log(1/h) \vee \log \log n)}} \le \Upsilon \quad \text{a.s.} $$
## 3.3 Uniform in bandwidth consistency of the conditional U-statistics

The main objective of this section is to establish the uniform consistency and the uniform in bandwidth consistency of the conditional U-statistic given by (2.10). First of all, we introduce the ideal estimator

$$ \hat{\mathbf{r}}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{2,n}) = \frac{\sum_{(i_1, \dots, i_k) \in I(k,n)} \varphi(\mathbf{Y}_{i_1}, \dots, \mathbf{Y}_{i_k}) \prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_j}))\right) \alpha^p(f(\mathbf{X}_{i_j}))}{\sum_{(i_1, \dots, i_k) \in I(k,n)} \prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f(\mathbf{X}_{i_j}))\right) \alpha^p(f(\mathbf{X}_{i_j}))} \quad (3.20) $$

For such kernels, we consider the class of functions

$$ \mathcal{H} := \{h_2 K_{h_2}((\tilde{\mathbf{t}} - \cdot)\alpha(f(\cdot))\alpha^p(f(\cdot)) : h_2 > 0, \ \tilde{\mathbf{t}} \in \mathbb{R}^{pk}\}, $$

where, as usual, $K_{h_2}(\mathbf{z}) = h_2^{-1}K(\mathbf{z}/h_2)$, $\mathbf{z} \in \mathbb{R}$ and we assume that $\mathcal{H}$ is pointwise measurable and of VC-type. Let

$$ \tilde{K}(\tilde{\mathbf{t}}) := \prod_{j=1}^{k} K(\mathbf{t}_j), \quad \tilde{\mathbf{t}} = (\mathbf{t}_1, \dots, \mathbf{t}_k), $$

denote the product kernel. For a bandwidth $0 < h_2 < 1$ and $\varphi \in \mathcal{F}$, consider the U-kernel

$$ G_{\varphi, h_{2,n}, \tilde{\mathbf{t}}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) := \varphi(\tilde{\mathbf{y}}) \tilde{K}_{h_2}((\tilde{\mathbf{t}} - \tilde{\mathbf{x}})\alpha(f(\tilde{\mathbf{x}}))) \alpha(\tilde{\mathbf{x}}), \quad \tilde{\mathbf{x}}, \tilde{\mathbf{t}} \in \mathbb{R}^{pk}, \tilde{\mathbf{y}} \in \mathbb{R}^{kq}, $$

where

$$ \alpha(\tilde{\mathbf{x}}) = \prod_{i=1}^{k} \alpha^p(f(\mathbf{x}_i)). $$

For the sample $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$, define

$$ U_n(\varphi, h_2, \tilde{\mathbf{t}}) := U_n^{(k)}(G_{\varphi, h_2, \tilde{\mathbf{t}}}) = \frac{(n-k)!}{n!} \sum_{\mathbf{i} \in I_n^k} G_{\varphi, h_2, \tilde{\mathbf{t}}} (\mathbb{X}_{\mathbf{i}}, \mathbb{Y}_{\mathbf{i}}). $$

Now, introduce the U-statistic process

$$ u_n(\varphi, h_2, \tilde{\mathbf{t}}) := \sqrt{n} \{ U_n(\varphi, h_2, \tilde{\mathbf{t}}) - \mathbb{E} U_n(\varphi, h_2, \tilde{\mathbf{t}}) \}. \quad (3.21) $$

**Remark 3.3.1.** If the elements $\varphi$ of the class of functions $\mathcal{F}$ are not symmetric which implies that $G_{\varphi, h_2, \tilde{\mathbf{t}}}$ are not symmetric, we will need to symmetrize them. To do this we have:

$$ \bar{G}_{\varphi, h_2, \tilde{\mathbf{t}}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) := (k!)^{-1} \sum_{\sigma \in I_k^k} G_{\varphi, h_2, \tilde{\mathbf{t}}}(\tilde{\mathbf{x}}_{\sigma}, \tilde{\mathbf{y}}_{\sigma}) = (k!)^{-1} \sum_{\sigma \in I_k^k} \varphi(\tilde{\mathbf{y}}_{\sigma}) \tilde{K}_{h_2}((\tilde{\mathbf{t}} - \tilde{\mathbf{x}}_{\sigma})\alpha(f(\tilde{\mathbf{x}}_{\sigma}))) \alpha(\tilde{\mathbf{x}}_{\sigma}), $$

where $\tilde{\mathbf{x}}_{\sigma} := (\mathbf{x}_{\sigma_1}, \dots, \mathbf{x}_{\sigma_k})$ and $\tilde{\mathbf{y}}_{\sigma} := (\mathbf{y}_{\sigma_1}, \dots, \mathbf{y}_{\sigma_k})$. Obviously, the expectation of $G_{\varphi, h_2, \tilde{\mathbf{t}}}$ remains unchanged after symmetrization, i.e.,

$$ \mathbb{E}(\bar{G}_{\varphi, h_2, \tilde{\mathbf{t}}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}})) = \mathbb{E}(G_{\varphi, h_2, \tilde{\mathbf{t}}}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}})), $$

and the U-statistic

$$ U_n^{(k)}(G_{\varphi, h_2, \tilde{\mathbf{t}}}) = U_n^{(k)}(\bar{G}_{\varphi, h_2, \tilde{\mathbf{t}}}) := U_n(\varphi, h_2, \tilde{\mathbf{t}}) $$

do not change. So the U-statistic process in (3.21) may be redefined using the symmetrized kernels, hence we consider

$$ u_n^{(k)}(\varphi, h_2, \tilde{\mathbf{t}}) := \sqrt{n} \{ U_n^{(k)}(\bar{G}_{\varphi, h_2, \tilde{\mathbf{t}}}) - \mathbb{E} U_n^{(k)}(\bar{G}_{\varphi, h_2, \tilde{\mathbf{t}}}) \} $$

For more details, for instance, consult the book of de la Peña and Giné (1999).

The following result handles the uniform deviation of the estimate $U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}})$ with respect to $\mathbb{E}[U_n(\varphi, h_{2,n}, \tilde{\mathbf{t}})]$ when the class of functions is bounded.

**Theorem 3.3.2.** Suppose that the marginal density $f_{\mathcal{X}}(\cdot)$ of $\mathcal{X}$ is bounded and let $\tilde{a}'_n = c(\log n/n)^{1/kp}$ for $c > 0$. Suppose the conditions (K.i), (K.ii), (F.i), (F.ii), (F.iii) and (F.iv) are fulfilled, we infer that for all $0 < \tilde{b}_0 < 1$, there exists a constant $0 < \mathcal{C} < \infty$ such that

$$ \limsup_{n \to \infty} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^{pk}} |U_n(\varphi, h_2, \tilde{\mathbf{t}}) - \mathbb{E}U_n(\varphi, h_2, \tilde{\mathbf{t}})|}{\sqrt{|\log h_2| \vee \log \log n}} \le \mathcal{C} \quad \text{a.s.} \quad (3.22) $$
The following result handles the uniform deviation of the estimate $U_n(\varphi, h_2, \tilde{\mathbf{t}})$ with respect to $\mathbb{E}U_n(\varphi, h_2, \tilde{\mathbf{t}})$ when the class of functions is unbounded satisfying general moment condition.

**Theorem 3.3.3.** Suppose that the marginal density $f_{\mathcal{X}}(\cdot)$ of $\mathcal{X}$ is bounded and let $\tilde{a}'_n = c((\log n/n)^{1-2/v})^{1/kp}$ for $c > 0$. Suppose the conditions (K.i), (K.ii), (F.i), (F.ii), (F.iii) and (F.v) are fulfilled, we infer that for all $0 < \tilde{b}_0 < 1$, there exists a constant $0 < \mathcal{C}' < \infty$ such that

$$ \limsup_{n \to \infty} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^{pk}} |U_n(\varphi, h_2, \tilde{\mathbf{t}}) - \mathbb{E}U_n(\varphi, h_2, \tilde{\mathbf{t}})|}{\sqrt{|\log h_2| \vee \log \log n}} \le \mathcal{C}' \quad \text{a.s.} \quad (3.23) $$

Now, consider the U-kernel :

$$ \tilde{G}_{\varphi, \tilde{\mathbf{t}}, h_1, h_2}(\tilde{\mathbf{x}}, \tilde{\mathbf{y}}) := \varphi(\tilde{\mathbf{y}}) \tilde{K}_{h_2}((\tilde{\mathbf{t}} - \tilde{\mathbf{x}}) \alpha(f_{\mathbf{X};n}(\tilde{\mathbf{x}}; h_1))) \alpha(\tilde{\mathbf{x}}), \quad \tilde{\mathbf{x}}, \tilde{\mathbf{t}} \in \mathbb{R}^{pk}, \tilde{\mathbf{y}} \in \mathbb{R}^{kq}, $$

where

$$ f_{\mathbf{X};n}(\mathbf{x}; h_1) = \frac{1}{nh_1^p} \sum_{i=1}^{n} K \left( \frac{\mathbf{x} - \mathbf{X}_i}{h_1} \right) \quad (3.24) $$

and

$$ \alpha(\tilde{\mathbf{x}}) = \prod_{i=1}^{k} \alpha^{p}(f_{\mathbf{X};n}(\mathbf{x}_i; h_1)). \quad (3.25) $$

For the sample $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$, define

$$ \tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) := \tilde{\mathbf{U}}_{\mathbf{n}}^{(k)}(\tilde{G}_{\varphi, \tilde{\mathbf{t}}, h_1, h_2}) = \frac{(n-k)!}{n!} \sum_{\mathbf{i} \in I_{\mathbf{n}}^k} \tilde{G}_{\varphi, \tilde{\mathbf{t}}, h_1, h_2}(\mathbf{X}_{\mathbf{i}}, \mathbf{Y}_{\mathbf{i}}). \quad (3.26) $$

The following result handles the uniform deviation of the estimate $\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})$ with respect to $\mathbb{E}\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ when the class of functions is bounded.

**Theorem 3.3.4.** Let $\tilde{a}'_n = c((\log n/n))^{1/kp}$ for $c > 0$. Under Assumptions (K.i), (K.ii), (F.i), (F.ii), (F.iii) and (F.iv), we have that, for $0 < \tilde{C} < \infty$,

$$ \limsup_{n \to \infty} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^{pk}} |\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) - \mathbb{E}\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)|}{\sqrt{|\log h_2| \vee \log \log n}} \le \tilde{C} \quad \text{a.s.} \quad (3.27) $$

The following result handles the uniform deviation of the estimate $\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ with respect to $\mathbb{E}\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ when the class of functions is unbounded satisfying general moment condition.

**Theorem 3.3.5.** Let $\tilde{a}'_n = c((\log n/n)^{1-2/v})^{1/kp}$ for $c > 0$. Under Assumptions (K.i), (K.ii), (F.i), (F.ii), (F.iii) and (F.v), we get, for some constant $0 < \tilde{C}' < \infty$, that

$$ \limsup_{n \to \infty} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}'_n \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^{pk}} |\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) - \mathbb{E}\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)|}{\sqrt{|\log h_2| \vee \log \log n}} \le \tilde{C}' \quad \text{a.s.} \quad (3.28) $$

The proof of this theorem in the unbounded case is similar to the proof of the bounded case and therefore omitted.

From now, we shall write $\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ for the estimator of the regression function defined in (2.10). It is clear that $\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ can be rewritten, for all $\varphi \in \mathcal{F}$, as

$$ \tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) = \frac{\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)}{\tilde{\mathbf{U}}_{\mathbf{n}}(1, \tilde{\mathbf{t}}; h_1, h_2)}, \quad (3.29) $$

where we denote by $\tilde{\mathbf{U}}_{\mathbf{n}}(1, \tilde{\mathbf{t}}; h_1, h_2)$ the U-statistic $\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ with $\varphi \equiv 1$. Hence, the study of the uniform consistency and the uniform in bandwidth consistency of $\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ to $r^{(k)}(\varphi, \tilde{\mathbf{t}})$ returns to the study of the uniform consistency and the uniform in bandwidth consistency of $\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ to $\mathbb{E}[\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)]$ when $\varphi \in \mathcal{F}$ and when $\varphi \equiv 1$. Which means that we shall consider another appropriate centering factor, as in the case $m = 1$, than the expectation $\mathbb{E}[\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)]$, so define :

$$ \hat{\mathbb{E}}[\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)] = \frac{\mathbb{E}[\tilde{\mathbf{U}}_{\mathbf{n}}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)]}{\mathbb{E}[\tilde{\mathbf{U}}_{\mathbf{n}}(1, \tilde{\mathbf{t}}; h_1, h_2)]}. \quad (3.30) $$
For any subset $\mathbf{I} \in \mathcal{D}_r$, introduce the class of functions defined on the subset $\mathbf{J} = \mathbf{I}^\eta \times \dots \times \mathbf{I}^\eta$ of $\mathbb{R}^{pk}$,

$$ \mathcal{M} = \{ r^{(k)}(\cdot, \tilde{\mathbf{t}}) \tilde{f}(\cdot) : \varphi \in \mathcal{F} \}, \qquad (3.31) $$

where the function $\tilde{f}: \mathbb{R}^{pk} \to \mathbb{R}$ is defined as

$$ \tilde{f}(\tilde{\mathbf{t}}) := \int f_{\mathbb{X}\mathcal{Y}}(\mathbf{t}_1, \mathbf{y}_1) \cdots f_{\mathbb{X}\mathcal{Y}}(\mathbf{t}_k, \mathbf{y}_k) \, \mathrm{d}\mathbf{y}_1 \cdots \mathrm{d}\mathbf{y}_k = f_{\mathbb{X}}(\mathbf{t}_1) \cdots f_{\mathbb{X}}(\mathbf{t}_k). \qquad (3.32) $$

The following result handles the uniform deviation of the estimate $\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ with respect to $\hat{\mathbb{E}}(\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2))$ in both cases, when the class of functions is bounded or unbounded satisfying general moment condition.

**Theorem 3.3.6.** Suppose that the marginal density function $f_{\mathbb{X}}(\cdot)$ of $\mathbb{X}$ is continuous and strictly positive on the interval $\mathbf{J} = \mathbf{I}^\eta$, where $\mathbf{I}$ is a subset in $\mathcal{D}_r$ and $\eta > 0$. Let $\tilde{a}_n' = c((\log n/n)^{1-2/v})^{1/kp}$ for $c > 0$. It then follows under the abovementioned assumptions on $\mathcal{F}$ that for all sequences $0 < \tilde{b}_n < 1$ with $\tilde{a}_n \le \tilde{b}_n \to 0$, there exists a constant $0 < \tilde{C}'' < \infty$ such that:

$$ \limsup_{n \to \infty} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n' \le h_2 \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_2^{pk}} |\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) - \hat{\mathbb{E}}\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)|}{\sqrt{|\log h_2| \vee \log \log n}} \le \tilde{C}'' \quad \text{a.s.} \qquad (3.33) $$

**Theorem 3.3.7.** Besides being bounded, suppose that the marginal density function $f_{\mathbb{X}}(\cdot)$ of $\mathbb{X}$ is continuous and strictly positive on the interval $\mathbf{J} = \mathbf{I}^\eta$, where $\mathbf{I}$ is a subset in $\mathcal{D}_r$ and $\eta > 0$. Assume that the class of functions $\mathcal{M}$ is uniformly equicontinuous. It then follows that for all sequences $0 < \tilde{b}_n < 1$ with $\tilde{b}_n \to 0$,

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n' \le h_2 \le \tilde{b}_n} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} |\hat{\mathbb{E}}\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) - r^{(k)}(\varphi, \tilde{\mathbf{t}})| = o(1). \qquad (3.34) $$

**Proposition 3.3.8.** Besides being bounded, suppose that the marginal density function $f_{\mathbb{X}}(\cdot)$ of $\mathbb{X}$ is continuous and strictly positive on the interval $\mathbf{J}$. Assume that the class of functions $\mathcal{M}$ is uniformly equicontinuous. It then follows that for all sequences $0 < \tilde{b}_n < 1$ with $\tilde{b}_n \to 0$,

$$ \sup_{a_n \le h_1 \le b_n} \sup_{\tilde{a}_n' \le h_2 \le \tilde{b}_n} \sup_{\varphi \in \mathcal{F}} \sup_{\tilde{\mathbf{t}} \in \mathbf{I}^k} |\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - r^{(k)}(\varphi, \tilde{\mathbf{t}})| \to 0. \qquad (3.35) $$

**Corollary 3.3.9.** Under Assumptions of Theorems 3.3.6 and 3.3.7, for $\hat{\mathbf{I}} \subset \hat{\mathcal{D}}_r^n$, we get the following result,

$$ \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_n} \sup_{\mathbf{t} \in \hat{\mathbf{I}}^k} \sup_{\varphi \in \mathcal{F}} |\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2) - r^{(k)}(\varphi, \tilde{\mathbf{t}})| = O\left(\sqrt{\frac{\log(1/\tilde{a}_n) \vee \log \log n}{n\tilde{a}_n}}\right) + O(\tilde{b}_n^4) \quad \text{a.s.} \qquad (3.36) $$

**Remark 3.3.10.** If we take $h_{1,n}$ and $h_{2,n}$ as in (3.12) and (3.13). Then, the estimator $\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})$, given by (2.10), satisfies, for $\mathbf{I} \subset \mathcal{D}_r$ and $\hat{\mathbf{I}} \subset \hat{\mathcal{D}}_r^n$,

$$ \sup_{\mathbf{t} \in \hat{\mathbf{I}}^k} \sup_{\varphi \in \mathcal{F}} |\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - r^{(k)}(\varphi, \tilde{\mathbf{t}})| = O_{\text{a.s.}}\left(\left(\frac{\log n}{n}\right)^{(8+p(1-k))/2(8+p)}\right), $$

and

$$ \sup_{\mathbf{t} \in \hat{\mathbf{I}}^k} \sup_{\varphi \in \mathcal{F}} |\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - r^{(k)}(\varphi, \tilde{\mathbf{t}})| = O_{\text{a.s.}}\left(\left(\frac{\log n}{n}\right)^{(8+p(1-k))/2(8+p)}\right). $$

**Remark 3.3.11.** Let us define $\mathfrak{G}(\varphi, \mathbf{t}) := r^{(1)}(\varphi, \mathbf{t}) = \|\nabla \mathbb{E}(\varphi(\mathbf{Y})|\mathbf{X} = \mathbf{t})\|_{f_{\mathbb{X}}(\mathbf{t})}$. A natural estimator of $\mathfrak{G}(\varphi, \cdot)$ is defined by

$$ \mathfrak{G}_n(\varphi, \mathbf{t}, h_{1,n}) = \|\nabla \tilde{r}_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_{2,n})\|_{f_{\mathbb{X};n}(\mathbf{t}; h_{1,n})}. \qquad (3.37) $$

Alternative estimators to (2.10) are given by

$$ \tilde{\tilde{r}}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) = \frac{\sum_{(i_1, \dots, i_k) \in I(k,n)} \varphi(\mathbf{Y}_{i_1}, \dots, \mathbf{Y}_{i_k}) \prod_{j=1}^k K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(\mathfrak{G}_n(\varphi, \mathbf{t}, h_{1,n}))\right) \alpha^p(\mathfrak{G}_n(\varphi, \mathbf{t}, h_{1,n}))}{\sum_{(i_1, \dots, i_k) \in I(k,n)} \prod_{j=1}^k K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(\mathfrak{G}_n(\varphi, \mathbf{t}, h_{1,n}))\right) \alpha^p(\mathfrak{G}_n(\varphi, \mathbf{t}, h_{1,n}))}. \qquad (3.38) $$
By using the same lines of the proof of Theorem 3.3.6, one can show

$$ \limsup_{n \to \infty} \sup_{a_n \le h_{1,n} \le b_0} \sup_{\tilde{a}_n \le h_{2,n} \le \tilde{b}_0} \sup_{\varphi \in \mathcal{F} \tilde{\mathbf{t}} \in \mathbf{I}^k} \frac{\sqrt{nh_{2,n}^{pk}} |\tilde{\mathring{r}}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - \hat{\mathbb{E}}\tilde{\mathring{r}}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n})|}{\sqrt{|\log h_{2,n}| \vee \log \log n}} \le \tilde{C}'' \quad \text{a.s.} \quad (3.39) $$

In the same fashion of Proposition 3.3.8, we show that

$$ \sup_{a_n \le h_{1,n} \le b_n} \sup_{\tilde{a}_n \le h_{2,n} \le \tilde{b}_n} \sup_{\varphi \in \mathcal{F} \tilde{\mathbf{t}} \in \mathbf{I}^k} |\tilde{\mathring{r}}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - r^{(k)}(\varphi, \tilde{\mathbf{t}})| \to 0. \quad (3.40) $$

By combining (3.39) and (3.40), we obtain a similar result to Corollary 3.3.9, given by

$$ \sup_{a_n \le h_{1,n} \le b_n} \sup_{\tilde{a}_n \le h_{2,n} \le \tilde{b}_n} \sup_{\varphi \in \mathcal{F} \tilde{\mathbf{t}} \in \mathbf{I}^k} |\tilde{\mathring{r}}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) - r^{(k)}(\varphi, \tilde{\mathbf{t}})| \to 0. \quad (3.41) $$

**Remark 3.3.12.** We note that the main problem in using an estimator such as in (2.10) is properly choosing the smoothing parameter $h$. The uniform in bandwidth consistency results given in Corollary 3.3.9 shows that any choice of $h_1, h_2$ in some intervals ensures the consistency of $\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$. Namely, the fluctuation of the bandwidth in a small interval does not affect the consistency of the nonparametric estimator $\tilde{r}_n^{(k)}(\varphi, \tilde{\mathbf{t}}; h_1, h_2)$ of $r^{(k)}(\varphi, \tilde{\mathbf{t}})$.

**Remark 3.3.13.** For notational convenience, we have chosen the same bandwidth sequence for each margins. This assumption can be dropped easily. If one wants to make use of the vector bandwidths (see, in particular, Chapter 12 of Devroye and Lugosi (2001)). With obvious changes of notation, our results and their proofs remain true when $h$ is replaced by a vector bandwidth $\mathbf{h}_n = (h_n^{(1)}, \dots, h_n^{(d)})$, where $\min h_n^{(i)} > 0$. In this situation we set $h = \prod_{i=1}^d h^{(i)}$, and for any vector $\mathbf{v} = (v_1, \dots, v_d)$ we replace $\mathbf{v}/h$ by $(v_1/h^{(1)}, \dots, v_1/h^{(d)})$. For ease of presentation, we chose to use real-valued bandwidths throughout.

**Remark 3.3.14.** Observe that the standardizing factor is $(nh_n^{pk})^{1/2}$ with $h_n \to 0$, indicating a lower rate of convergence. This is the cost incurred when drawing conclusions regarding conditional (local) quantities.

**Remark 3.3.15.** It is straightforward to modify the proofs of our results to show that it remains true when entropy condition is substituted by the bracketing condition: For some $C_0 > 0$ and $v_0 > 0$,

$$ \mathcal{N}_{[1]}(\mathcal{F}, L_2(\mathbb{P}), \varepsilon) \le C_0 \varepsilon^{-v_0}, 0 < \varepsilon < 1. $$

Refer to p. 270 of van der Vaart (1998) for the definition of $\mathcal{N}_{[1]}(\mathcal{F}, L_2(\mathbb{P}), \varepsilon)$.

**Remark 3.3.16.** Note that the condition (F.v) may be replaced by more general hypotheses upon moments of $\mathbf{Y}$. That is

(M.1)' We denote by $\{\mathcal{M}(x) : x \ge 0\}$ a nonnegative continuous function, increasing on $[0, \infty)$, and such that, for some $s > 2$, ultimately as $x \uparrow \infty$,

$$ (i) \quad x^{-s} \mathcal{M}(x) \downarrow; \quad (ii) \quad x^{-1} \mathcal{M}(x) \uparrow. \quad (3.42) $$

For each $t \ge \mathcal{M}(0)$, we define $\mathcal{M}^{\text{inv}}(t) \ge 0$ by $\mathcal{M}(\mathcal{M}^{\text{inv}}(t)) = t$. We assume further that:

$$ \sup_{\mathbf{x} \in \mathbf{I}^d} \mathbb{E} \left( \mathcal{M} \left( |F(t'(\mathbf{Y}))| \right) \middle| \mathbf{X} = \mathbf{x} \right) < \infty. $$

The following choices of $\mathcal{M}(\cdot)$ are of particular interest:

(i) $\mathcal{M}(x) = x^p$ for some $p > 2$;

(ii) $\mathcal{M}(x) = \exp(sx)$ for some $s > 0$.

**Remark 3.3.17.** When the dimension of the explanatory variable $\mathbf{X}$ is high, it is a well-known fact that estimation issues of a regression function, also known as a conditional model, are extremely challenging. This is a well-established fact. It is important to take note of the fact that one of the repercussions of this is that the optimal minimax rate of convergence $n^{-2s/(2s+d)}$ for the estimation of a $s$ times differentiable regression function converges to zero rather slowly if the dimension $d$ of $\mathbf{X}$ is large in comparison to $s$. This is something that should be kept in mind. The only approach to avoid the so-called "curse of dimensionality" is to add additional assumptions to the regression functions, which is the only way to do it. The most straightforward approach is to think in terms of linear models; nevertheless, this somewhat restrictive parametric assumption can be stretched in a number of different ways. Additive models could be used to reduce the challenge of regression estimation by only fitting functions to the data that share the same additive structure. This is one way that the additive models could be used. In projection pursuit, one further generalizes this by assuming that the regression function is a sum of univariate functions that are applied to projections of $\mathbf{t}$ onto various directions. We note that this includes the single index models as particular cases, and the interested reader may refer to (Györfi et al., 2002, Chapter 22) for more rigorous developments of such techniques. Other approaches, such as the semi-parametric models, which are to be investigated, are to be considered intermediary models between linear and nonparametric ones. These models aim to combine the adaptability of nonparametric methods with the interpretability of parametric methods, and this is to be done in order to get specifics on these methods for functional data.
# 4 Conditional U-statistics for censored data

Consider a triple $(Y, C, \mathbf{X})$ of random variables defined in $\mathbb{R} \times \mathbb{R} \times \mathbb{R}^d$. Here $Y$ is the variable of interest, $C$ is a censoring variable, and $\mathbf{X}$ is a concomitant variable. Throughout, we will use Maillot and Viallon (2009) notation and we work with a sample $\{(Y_i, C_i, \mathbf{X}_i)_{1 \le i \le n}\}$ of independent and identically distributed replication of $(Y, C, \mathbf{X})$, $n \ge 1$. Actually, in the right censorship model, the pairs $(\tilde{Y}_i, C_i)$, $1 \le i \le n$, are not directly observed, and the corresponding information is given by $Z_i := \min\{Y_i, C_i\}$ and $\Delta_i := 1\{Y_i \le C_i\}$, $1 \le i \le n$. Accordingly, the observed sample is

$$ \mathcal{D}_n = \{(Z_i, \Delta_i, \mathbf{X}_i), i = 1, \dots, n\}. $$

Survival data in clinical trials or failure time data in reliability studies, for example, are often subject to such censoring. To be more specific, many statistical experiments result in incomplete samples, even under well-controlled conditions. For example, clinical data for surviving most types of disease are usually censored by other competing risks to life which result in death. In the sequel, we impose the following assumptions upon the distribution of $(\mathbf{X}, Y)$. Denote by $\mathcal{I}$ a given compact set in $\mathbb{R}^d$ with nonempty interior and set, for any $\alpha > 0$,

$$ \mathcal{I}_{\alpha} = \{\mathbf{x} : \inf_{\mathbf{u} \in \mathcal{I}} \|\mathbf{x} - \mathbf{u}\| \le \alpha\}. $$

We will assume that, for a given $\alpha > 0$, $(\mathbf{X}, Y)$ [resp. $\mathbf{X}$] has a density function $f_{\mathbf{X},Y}$ [resp. $f_{\mathbf{X}}$] with respect to the Lebesgue measure on $\mathcal{I}_{\alpha} \times \mathbb{R}$ [resp. $\mathcal{I}_{\alpha}$]. For $-\infty < t < \infty$, set

$$ F_Y(t) = \mathbb{P}(Y \le t), \quad G(t) = \mathbb{P}(C \le t), \quad \text{and} \quad H(t) = \mathbb{P}(Z \le t), $$

the right-continuous distribution functions of $Y$, $C$ and $Z$ respectively. For any right-continuous distribution function $L$ defined on $\mathbb{R}$, denote by

$$ T_L = \sup\{t \in \mathbb{R} : L(t) < 1\} $$

the upper point of the corresponding distribution. Now consider a pointwise measurable class $\mathcal{F}$ of real measurable functions defined on $\mathbb{R}$, and assume that $\mathcal{F}$ is of VC-type. We recall the regression function of $\psi(Y)$ evaluated at $\mathbf{X} = \mathbf{x}$, for $\psi \in \mathcal{F}$ and $\mathbf{x} \in \mathcal{I}_{\alpha}$, given by

$$ r^{(1)}(\psi, \mathbf{x}) = \mathbb{E}(\psi(Y) | \mathbf{X} = \mathbf{x}), $$

when $Y$ is right-censored. To estimate $r^{(1)}(\psi, \cdot)$, we make use of the Inverse Probability of Censoring Weighted (I.P.C.W.) estimators have recently gained popularity in the censored data literature (see Kohler et al. (2002), Carbonez et al. (1995)). The key idea of I.P.C.W. estimators are as follows. Introduce the real-valued function $\Phi_{\psi}(\cdot, \cdot)$ defined on $\mathbb{R}^2$ by

$$ \Phi_{\psi}(y, c) = \frac{1\{y \le c\} \psi(y \wedge c)}{1 - G(y \wedge c)}. \qquad (4.1) $$

Assuming the function $G(\cdot)$ to be known, first note that $\Phi_{\psi}(Y_i, C_i) = \Delta_i \psi(Z_i)/(1 - G(Z_i))$ is observed for every $1 \le i \le n$. Moreover, under the Assumption (I) below,

(I) $C$ and $(Y, \mathbf{X})$ are independent.

We have

$$ \begin{aligned} r^{(1)}(\Phi_{\psi}, \mathbf{x}) &:= \mathbb{E}(\Phi_{\psi}(Y, C) | \mathbf{X} = \mathbf{x}) \\ &= \mathbb{E}\left\{\frac{1\{Y \le C\} \psi(Z)}{1 - G(Z)} \middle| \mathbf{X} = \mathbf{x}\right\} \\ &= \mathbb{E}\left\{\frac{\psi(Y)}{1 - G(Y)} \mathbb{E}(1\{Y \le C\} | \mathbf{X}, Y) \middle| \mathbf{X} = \mathbf{x}\right\} \\ &= r^{(1)}(\psi, \mathbf{x}). \end{aligned} \qquad (4.2) $$

Therefore, any estimate of $r^{(1)}(\Phi_{\psi}, \cdot)$, which can be built on fully observed data, turns out to be an estimate for $r^{(1)}(\psi, \cdot)$ too. Thanks to this property, most statistical procedures known to provide estimates of the regression function in the uncensored case can be naturally extended to the censored case. For instance, kernel-type estimates are particularly easy to construct. Set, for $\mathbf{x} \in \mathcal{I}$, $h \ge 0$, $1 \le i \le n$,

$$ \bar{\omega}_{n,K,h,i}^{(1)}(\mathbf{x}) := K\left(\frac{\mathbf{x} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n})) / \sum_{j=1}^{n} K\left(\frac{\mathbf{x} - \mathbf{X}_j}{h}\right). \qquad (4.3) $$

In view of (4.1), (4.2), and (4.3), whenever $G(\cdot)$ is known, a kernel estimator of $r^{(1)}(\psi, \cdot)$ is given by

$$ \check{r}_n^{(1)}(\psi, \mathbf{x}; h_n) = \sum_{i=1}^{n} \bar{\omega}_{n,K,h,i}^{(1)}(\mathbf{x}) \frac{\Delta_i \psi(Z_i)}{1 - G(Z_i)}. \qquad (4.4) $$
The function $G(\cdot)$ is generally unknown and has to be estimated. We will denote by $G_n^*(\cdot)$ the Kaplan-Meier estimator of the function $G(\cdot)$ Kaplan and Meier (1958). Namely, adopting the conventions

$$ \prod_{i=1}^{k} = 1 $$

and $0^0 = 1$ and setting

$$ N_n(u) = \sum_{i=1}^{n} 1\{Z_i \ge u\}, $$

we have

$$ G_n^*(u) = 1 - \prod_{i:Z_i \le u} \left\{ \frac{N_n(Z_i) - 1}{N_n(Z_i)} \right\}^{(1-\Delta_i)}, \quad \text{for } u \in \mathbb{R}. $$

Given these notation, we will investigate the following estimator of $r^{(1)}(\psi, \cdot)$

$$ \check{r}_n^{(1)*}(\psi, \mathbf{x}; h_n) = \sum_{i=1}^{n} \bar{\omega}_{n,K,h,i}^{(1)}(\mathbf{x}) \frac{\Delta_i \psi(Z_i)}{1 - G_n^*(Z_i)}, \quad (4.5) $$

refer to Kohler et al. (2002) and Maillot and Viallon (2009). Adopting the convention $0/0 = 0$, this quantity is well defined, since $G_n^*(Z_i) = 1$ if and only if $Z_i = Z_{(n)}$ and $\Delta_{(n)} = 0$, where $Z_{(k)}$ is the kth ordered statistic associated with the sample $(Z_1, \dots, Z_n)$ for $k = 1, \dots, n$ and $\Delta_{(k)}$ is the $\Delta_i$ corresponding to $Z_k = Z_j$. A right-censored version of an unconditional U-statistic with a kernel of degree $m \ge 1$ is introduced by the principle of a mean preserving reweighting scheme in Datta et al. (2010). Stute and Wang (1993) have proved almost sure convergence of multi-sample U-statistics under random censorship and provided application by considering the consistency of a new class of tests designed for testing equality in distribution. To overcome potential biases arising from right-censoring of the outcomes and presence of confounding covariates, Chen and Datta (2019) proposed adjustments to the classical U-statistics. Yuan et al. (2017) proposed a different way in the estimation procedure of the U-statistic by using a substitution estimator of the conditional kernel given the observed data. We refer also to Bouzebda and Taachouche (2023a) and Bouzebda and El-hadjali (2020). To our best knowledge, the problem of the estimation of the conditional U-statistics in the censored setting with variable bandwidth was open up to present, and it gives and main motivation to the study of this section. A natural extension of the function defined in (4.1) is given by

$$ \Phi_{\psi}(y_1, \dots, y_k, c_1, \dots, c_k) = \frac{\prod_{i=1}^{k} \{1\{y_i \le c_i\} \psi(y_1 \wedge c_1, \dots, y_k \wedge c_m) \}}{\prod_{i=1}^{k} \{1 - G(y_i \wedge c_i)\}} $$

From this, we have an analogous relation to (4.2) given by

$$ \begin{aligned} & \mathbb{E}(\Phi_{\psi}(Y_1, \dots, Y_k, C_1, \dots, C_k) | (\mathbf{X}_1, \dots, \mathbf{X}_k) = \mathbf{x}) \\ &= \mathbb{E} \left( \frac{\prod_{i=1}^{k} \{1\{Y_i \le C_i\} \psi(Y_1 \wedge C_1, \dots, Y_k \wedge C_k) \}}{\prod_{i=1}^{k} \{1 - G(Y_i \wedge C_i)\}} \middle| (\mathbf{X}_1, \dots, \mathbf{X}_k) = \mathbf{x} \right) \\ &= \mathbb{E} \left( \frac{\psi(Y_1, \dots, Y_k)}{\prod_{i=1}^{k} \{1 - G(Y_i)\}} \mathbb{E} \left( \prod_{i=1}^{k} \{1\{Y_i \le C_i\} | (Y_1, X_1), \dots, (Y_k, X_k) \} \middle| (\mathbf{X}_1, \dots, \mathbf{X}_k) = \mathbf{x} \right) \right) \\ &= \mathbb{E}(\psi(Y_1, \dots, Y_k) | (\mathbf{X}_1, \dots, \mathbf{X}_k) = \mathbf{x}). \end{aligned} $$

An analogue estimator to (1.3) in the censored case is given by

$$ \check{r}_n^{(k)}(\psi, \mathbf{x}; m_n) = \sum_{(i_1, \dots, i_k) \in I(k,n)} \frac{\Delta_{i_1} \cdots \Delta_{i_k} \psi(Z_{i_1}, \dots, Z_{i_k})}{(1 - G(Z_{i_1}) \cdots (1 - G(Z_{i_k})))} \bar{\omega}_{n, \mathbf{i}}^{(k)}(\mathbf{x}), \quad (4.6) $$

where, for $\mathbf{i} = (i_1, \dots, i_k) \in I(k, n)$,

$$ \bar{\omega}_{n, \mathbf{i}}^{(k)}(\mathbf{x}) = \frac{\prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))}{\sum_{(i_1, \dots, i_k) \in I(k,n)} \prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))} \quad (4.7) $$
The estimator that we will investigate is given by

$$ \tilde{r}_n^{(k)*}(\psi, \mathbf{x}) = \sum_{(i_1, \dots, i_k) \in I(k,n)} \frac{\Delta_{i_1} \cdots \Delta_{i_k} \psi(Z_{i_1}, \dots, Z_{i_k})}{(1 - G_n^*(Z_{i_1}) \cdots (1 - G_n^*(Z_{i_k}))} \bar{\omega}_{n,i}^{(k)}(\mathbf{x}). \quad (4.8) $$

Recall the definition of $\mathcal{M}$ given in (3.31). To obtain our results, we will work under the following assumptions.

(A.1) $\mathcal{F} = \{\psi := \psi_1 1_{\{-\infty, \tau\}^k}, \psi_1 \in \mathcal{F}_1\}$, where $\tau < T_H$ and $\mathcal{F}_1$ is a pointwise measurable class of real measurable functions defined on $\mathbb{R}$ and of type VC.

(A.2) The class of functions $\mathcal{F}$ has a measurable and uniformly bounded envelope function $\Upsilon$ with,

$$ \Upsilon(y_1, \dots, y_k) \geq \sup_{\psi \in \mathcal{F}} |\psi(y_1, \dots, y_k)|, \quad y_i \leq T_H. $$

(A.3) The class of functions $\mathcal{M}$ is relatively compact concerning the sup-norm topology on $\mathcal{F}_\alpha^k$.

Let us assume the following conditions.

(H.1) $h \downarrow 0, 0 < h < 1$, and $nh^d \uparrow \infty$;

(H.2) $nh^d / \log n \to \infty$ as $n \to \infty$;

(H.3) $\log(1/h) / \log \log n \to \infty$ as $n \to \infty$.

Assumptions (H.1)-(H.3) are classical in the empirical process theory and are often referred to as the Csörgő-Révész-Stute [CRS] conditions [Csörgő and Révész (1981), Stute (1986)]. We now have all the ingredients to state the result corresponding to the censored case. Let $\{h'_n\}_{n \ge 1}$ and $\{h''_n\}_{n \ge 1}$ ($\{h'_n\}_{n \ge 1}$ and $\{h''_n\}_{n \ge 1}$, respectively) be two sequences of positive constants fulfilling Assumptions (H.1-H.3) with

$$ 0 < h'_n \le h''_n < 1, $$

(0 < \tilde{h}'_n \le \tilde{h}''_n < 1, respectively). The main result of this section is given in the following corollary.

**Corollary 4.0.1.** Assume that the conditions (C.1)-(C.4), (C.5'), (I) and (H.1)-(H.3) are satisfied. We have, as $n \to \infty$,

$$ \limsup_{n \to \infty} \sup_{h'_n \le h_1 \le h''_n} \sup_{\tilde{h}'_n \le h_2 \le \tilde{h}''_n} \sup_{\phi \in \mathcal{F}} \left| \tilde{r}_n^{(k)*}(\psi, \mathbf{x}) - \hat{\mathbb{E}} \tilde{r}_n^{(k)*}(\psi, \mathbf{x}) \right| = O \left( \frac{\sqrt{|\log h_2| \vee \log \log n}}{nh_2^{pk}} \right), \quad \text{a.s.} $$

This last result is a direct consequence of Theorem 3.3.6 and the law of iterated logarithm for $G_n^*(\cdot)$ established in Földes and Rejtő (1981) ensures that

$$ \sup_{t \le \tau} |G_n^* - G(t)| = O\left(\sqrt{\frac{\log \log n}{n}}\right) \quad \text{almost surely as } n \to \infty. $$

For more details refer to Bouzebda et al. (2023b,a), Bouzebda and Nezzal (2022, 2023), Bouzebda and Soukarieh (2023) and Bouzebda (2023b).

# 5 Applications

Although only the following examples will be given here, they stand as archetypes for a variety of problems that can be investigated in a similar way.

## 5.1 Discrimination problems

Now, we apply the results the problem of discrimination described in Section 3 of Stute (1994b), refer to also to Stute (1994a). We will use similar notation and setting. Let $\varphi(\cdot)$ be any function taking at most finitely many values, say $1, \dots, M$. The sets

$$ A_j = \{(\mathbf{y}_1, \dots, \mathbf{y}_k) : \varphi(\mathbf{y}_1, \dots, \mathbf{y}_k) = j\}, \quad 1 \le j \le M, $$

then yield a partition of the feature space. Predicting the value of $\varphi(\mathbf{Y}_1, \dots, \mathbf{Y}_k)$ is tantamount to predicting the set in the partition to which $(\mathbf{Y}_1, \dots, \mathbf{Y}_k)$ belongs. For any discrimination rule $g$, we have

$$ \mathbb{P}(g(\mathbf{X}) = \varphi(\mathbf{Y})) \le \sum_{j=1}^{M} \int_{\mathbf{x}:g(\mathbf{x})=j} \max \mathfrak{M}^j(\tilde{\mathbf{t}}) d\mathbb{P}(\tilde{\mathbf{t}}), $$

where

$$ \mathfrak{M}^j(\tilde{\mathbf{x}}) = \mathbb{P}(\varphi(\mathbf{Y}) = j | \mathbf{X} = \tilde{\mathbf{t}}), \quad \tilde{\mathbf{t}} \in \mathbb{R}^{pk}. $$
The above inequality becomes an equality if

$$g_0(\tilde{\mathbf{x}}) = \arg \max_{1 \le j \le M} \mathfrak{M}^j(\tilde{\mathbf{t}}).$$

$g_0(\cdot)$ is called the Bayes rule, and the pertaining probability of error

$$\mathbf{L}^* = 1 - \mathbb{P}(g_0(\mathbf{X}) = \varphi(\mathbf{Y})) = 1 - \mathbb{E} \left\{ \max_{1 \le j \le M} \mathfrak{M}^j(\tilde{\mathbf{t}}) \right\}$$

is called the Bayes risk. Each of the above unknown function $\mathfrak{M}^j$'s can be consistently estimated by one of the methods discussed in the preceding sections. Let, for $1 \le j \le M$,

$$\mathfrak{M}_n^j(\tilde{\mathbf{t}}) = \frac{\sum_{(i_1, \dots, i_k) \in I(k,n)} \mathbb{1}\{\varphi(\mathbf{Y}_{i_1}, \dots, \mathbf{Y}_{i_k}) = j\} \prod_{j=1}^k K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))}{\sum_{(i_1, \dots, i_k) \in I(k,n)} \prod_{j=1}^k K\left(\frac{\mathbf{t}_j - \mathbf{X}_{i_j}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{i_j}; h_{1,n}))}, \quad (5.1)$$

Set

$$g_{0,n}(\tilde{\mathbf{t}}) = \arg \max_{1 \le j \le M} \mathfrak{M}_n^j(\tilde{\mathbf{t}}).$$

Let us introduce

$$\mathbf{L}_n^* = \mathbb{P}(g_{0,n}(\mathbf{X}) \neq \varphi(\mathbf{Y})).$$

The discrimination rule $g_{0,n}(\cdot)$ is asymptotically Bayes' risk consistent

$$\mathbf{L}_n^* \to \mathbf{L}^*.$$

This follows from the obvious relation

$$|\mathbf{L}^* - \mathbf{L}_n^*| \le 2\mathbb{E} \left[ \max_{1 \le j \le M} \left| \mathfrak{M}_n^j(\mathbf{X}) - \mathfrak{M}^j(\mathbf{X}) \right| \right].$$

## 5.2 Metric learning

Metric learning aims to adapt the metric to the data and has garnered significant interest in recent years; for an overview of metric learning and its applications, see Bellet et al. (2013) and Clémençon et al. (2016b), for an account of metric learning and its applications. This is motivated by various applications ranging from computer vision to information retrieval through bioinformatics. To illustrate the usefulness of this concept, we present the metric learning problem for supervised classification as in Clémençon et al. (2016b). Let us consider independent copies $(\mathbf{Y}_{1,1}, \mathbf{Y}_{2,1}), \dots, (\mathbf{Y}_{1,n}, \mathbf{Y}_{2,n})$ of a $\mathcal{Y}_1 \times \mathcal{Y}_2$ valued random couple $(\mathbf{Y}_1, \mathbf{Y}_2)$, where $\mathcal{Y}_1$ is some feature space and $\mathcal{Y}_2 = \{1, \dots, C\}$, with $C \ge 2$ say, a finite set of labels. Let $\mathcal{D}$ be a set distance measures $D: \mathcal{Y}_1 \times \mathcal{Y}_2 \to \mathbb{R}_+$. Intuitively, metric learning aims in this setting to find a metric under which pairs of points with the same label are close to each other and those with different labels are far away. The natural way to define the risk of a metric $D$ is

$$R(D) = \mathbb{E} \left[ \phi \left( (1 - D(\mathbf{Y}_1, \mathbf{Y}_1')) \cdot (2\mathbb{1}\{\mathbf{Y}_2 = \mathbf{Y}_2'\} - 1) \right) \right], \quad (5.2)$$

where $\phi(u)$ is a convex loss function upper bounding the indicator function $\mathbb{1}\{u \ge 0\}$, for instance, the hinge loss $\phi(u) = \max(0, 1-u)$. To estimate $R(D)$, we consider the natural empirical estimator

$$R_n(D) = \frac{2}{n(n-1)} \sum_{1 \le i < j \le n} \phi \left( (D(\mathbf{Y}_{1,i}, \mathbf{Y}_{1,j}) - 1) \cdot (2\pi \{\mathbf{Y}_{2,i} = \mathbf{Y}_{2,j}\} - 1) \right), \quad (5.3)$$

which is one sample U-statistic of degree two with kernel given by:

$$\varphi_D((\mathbf{y}_1, \mathbf{y}_2), (\mathbf{y}'_1, \mathbf{y}'_2)) = \phi((D(\mathbf{y}_1, \mathbf{y}'_1) - 1) \cdot (2\mathbb{1}\{\mathbf{y}_2 = \mathbf{y}'_2\} - 1)).$$

The convergence to (5.2) of a minimizer of (5.3) has been studied in the frameworks of algorithmic stability (Jin et al. (2009)), algorithmic robustness (Bellet and Habrard (2015)) and based on the theory of U-processes under appropriate regularization (Cao et al. (2016b)).

## 5.3 Multipartite ranking

Let us recall the problem from Clémençon et al. (2016b). Let $\mathbf{Y}_1 \in \mathcal{Y}_1$ be a random vector of attributes/features and the (temporarily hidden) ordinal labels $\mathbf{Y}_2 \in \{1, \dots, K\}$ assigned to it. Based on a training set of labeled examples, multipartite ranking aims to rank the attributes/features in the same order as that induced by the labels. This statistical learning problem is applied in various fields (e.g., medicine, finance, search engines, e-commerce). Rankings are generally defined through a scoring function $s: \mathcal{Y}_1 \to \mathbb{R}$, transporting the natural order on the real line onto the feature space and the gold standard for evaluating the ranking performance
of $s(\mathbf{x})$ is the ROC manifold, or its usual summary the VUS criterion (VUS standing for volume Under the ROC Surface), see Clémençon and Robbiano (2015) and the references therein. Clémençon et al. (2013) characterized the optimal scoring functions as those that are optimal for all bipartite subproblems. More precisely, they are increasing transforms of the likelihood ratio $dF_{k+1}/dF_k$, where $F_k$ denotes the class-conditional distribution for the k-th class. The authors proved when the set of optimal scoring functions is non-empty, that it corresponds to the functions which maximize the volume under the ROC surface

$$VUS(s) = \mathbb{P}\{s(\mathbf{Y}_{1,1}) < \dots < s(\mathbf{Y}_{1,K}) | \mathbf{Y}_{2,1} = 1, \dots, \mathbf{Y}_{2,K} = K\}$$

Given K independent samples $(\mathbf{Y}_{1,1}^{(k)}, \dots, \mathbf{Y}_{1,n_k}^{(k)}) \stackrel{\text{i.i.d.}}{\sim} F_k(dx)$ for $k = 1, \dots, K$, the empirical counterpart of the VUS can be written in the following way:

$$\widehat{VUS}(s) = \frac{1}{\prod_{k=1}^{K} n_k} \sum_{i_1=1}^{n_1} \dots \sum_{i_k=1}^{n_K} \mathbb{1}\{s(\mathbf{Y}_{1,i_1}^{(1)}) < \dots < s(\mathbf{Y}_{1,i_K}^{(K)})\} \quad (5.4)$$

The empirical VUS (5.4) is a K-sample U-statistic of degree $(1, \dots, K)$ with kernel given by:

$$\varphi_s(\mathbf{y}_{1,1}, \dots, \mathbf{y}_{1,K}) = \mathbb{1}\{s(\mathbf{y}_{1,1}) < \dots < s(\mathbf{y}_{1,K})\}.$$

## 5.4 Ranking problems

For its great importance, the problem of ranking instances has received particular attention in machine learning. The ordering problems have many applications in different areas of banking (Data mining process for direct marketing data extraction), document type classification, and so on. In some specific ranking problems, it is necessary to compare two different observations based on their observed characteristics and decide which one is better instead of simply classifying them. The problems of ordering/ranking are frequent problems in which U-statistics come into play. In this kind of challenge, the aim is to establish a universal and consistent ordering method. Suppose that we want to establish an order between the first components of the two pairs $(\mathbf{X}, \mathbf{Y})$, $(\mathbf{X}', \mathbf{Y}')$ of independent and identically distributed observations in $\mathcal{X} \times \mathbb{R}$. The variables $\mathbf{Y}$ and $\mathbf{Y}'$ are respective labels of the variables $\mathbf{X}$ and $\mathbf{X}'$ that we want to order by observing them (and not their labels). Usually, we decide that $\mathbf{X}$ is better than $\mathbf{X}'$ if $\mathbf{Y} > \mathbf{Y}'$. To see things more clearly, we introduce the new variable:

$$\mathbf{Z} = \frac{\mathbf{Y} - \mathbf{Y}'}{2},$$

then $\mathbf{Y} > \mathbf{Y}'$ is equivalent to $\mathbf{Z} > 0$. As mentioned, the goal is to establish a classification rule between $\mathbf{X}$ and $\mathbf{X}'$ with minimal risk, i.e., the probability that the label of the highest ranked variable is the smallest is small. Mathematically speaking, the decision rule is given by the function:

$$r(\mathbf{x}, \mathbf{x}') = \begin{cases} 1 & \text{if } \mathbf{x} > \mathbf{x}', \\ -1 & \text{else.} \end{cases}$$

The following ranking risk gives the performance measure of $r$:

$$L(r) = \mathbb{P}(\mathbf{Z}.r(\mathbf{X}, \mathbf{X}')).$$

A natural estimate for $L(\cdot)$ according to Clémençon et al. (2008) is:

$$L_n(r) := \frac{1}{n(n-1)} \sum_{i \neq j} \mathbb{1}\{\mathbf{z}_{i,j}, r(\mathbf{x}_i, \mathbf{x}_j) < 0\},$$

where $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$ are n independent, identically distributed copies of $(\mathbf{X}, \mathbf{Y})$, and $\mathbf{Z}_{i,j} = \frac{\mathbf{Y}_i - \mathbf{Y}_j}{2}$. One can easily see that $L_n$ is a U-statistic with $k = 2$. For more details the reader is invited to consult Clémençon et al. (2008) and Rejchel (2012).

## 5.5 Kendall rank correlation coefficient

To test the independence of one-dimensional random variables $\mathbf{Y}_1$ and $\mathbf{Y}_2$ Kendall (1938) proposed a method based on the U-statistic $K_n$ with the kernel function:

$$\varphi((\mathbf{s}_1, \mathbf{t}_1), (\mathbf{s}_2, \mathbf{t}_2)) = \mathbb{1}\{(\mathbf{s}_2 - \mathbf{s}_1)(\mathbf{t}_2 - \mathbf{t}_1) > 0\} - \mathbb{1}\{(\mathbf{s}_2 - \mathbf{s}_1)(\mathbf{t}_2 - \mathbf{t}_1) \le 0\}. \quad (5.5)$$

Its rejection on the region is of the form $\{\sqrt{n}K_n > \gamma\}$. In this example, we consider a multivariate case. To test the conditional independence of $\xi, \eta: Y = (\xi, \eta)$ given $X$, we suggest a method based on the conditional U-statistic:

$$\hat{r}_n^{(2)}(\varphi, \tilde{\mathbf{t}}) = \frac{\sum_{i \neq j}^{n} \varphi(\mathbf{Y}_i, \mathbf{Y}_j) \prod_{j=1}^{2} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))}{\sum_{i \neq j}^{n} \prod_{j=1}^{2} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))},$$
where $\tilde{\mathbf{t}} = (\mathbf{t}_1, \mathbf{t}_2) \in \mathbb{I} \subset \mathbb{R}^2$ and $\varphi(\cdot)$ is Kendall's kernel (5.5). Suppose that $\xi$ and $\eta$ are $d_1$ and $d_2$-dimensional random vectors respectively and $d_1 + d_2 = d$. Furthermore, suppose that $\mathbf{Y}_1, \dots, \mathbf{Y}_n$ are observations of $(\xi, \eta)$, we are interested in testing:

$$H_0: \xi \text{ and } \eta \text{ are conditionally independent given } X. \text{ vs } H_a: H_0 \text{ is not true.} \tag{5.6}$$

Let $\mathbf{a} = (\mathbf{a}_1, \mathbf{a}_2) \in \mathbb{R}^d$ such as $\|\mathbf{a}\| = 1$ and $\mathbf{a}_1 \in \mathbb{R}^{d_1}$, $\mathbf{a}_2 \in \mathbb{R}^{d_2}$, and $F(\cdot), G(\cdot)$ be the distribution functions of $\xi$ and $\eta$ respectively. Suppose $F^{a_1}(\cdot)$ and $G^{a_2}(\cdot)$ to be continuous for any unit vector $\mathbf{a} = (\mathbf{a}_1, \mathbf{a}_2)$ where $F^{a_1}(t) = P(\mathbf{a}_1^\top \xi < t)$ and $G^{a_2}(t) = P(\mathbf{a}_2^\top \eta < t)$ and $\mathbf{a}_1^\top$ means the transpose of the vector $\mathbf{a}_i, 1 \le i \le 2$. For $n=2$, let $\mathbf{Y}^{(1)} = (\xi^{(1)}, \eta^{(1)})$ and $\mathbf{Y}^{(2)} = (\xi^{(2)}, \eta^{(2)})$ such as $\xi^{(i)} \in \mathbb{R}^{d_1}$ and $\eta^{(i)} \in \mathbb{R}^{d_2}$ for $i=1,2$, and:

$$\varphi^a(\mathbf{Y}^{(1)}, \mathbf{Y}^{(2)}) = \varphi\left(\left(\mathbf{a}_1^\top \xi^{(1)}, \mathbf{a}_2^\top \eta^{(1)}\right), \left(\mathbf{a}_1^\top \xi^{(2)}, \mathbf{a}_2^\top \eta^{(2)}\right)\right).$$

An application of Proposition 3.3.8 gives

$$|\hat{r}_n^{(2)}(\varphi^a, \mathbf{x}; m_n) - r^{(2)}(\varphi^a, \mathbf{x})| = o(1). \tag{5.7}$$

## 5.6 Generalized U-Statistics

The extension to the case of several samples is straightforward. Consider $k$ independent collections, of independent observations

$$\left\{ \left( \mathbf{X}_1^{(1)}, \mathbf{Y}_1^{(1)} \right), \left( \mathbf{X}_2^{(1)}, \mathbf{Y}_2^{(1)} \right), \dots \right\}, \dots, \left\{ \left( \mathbf{X}_1^{(k)}, \mathbf{Y}_1^{(k)} \right), \left( \mathbf{X}_2^{(k)}, \mathbf{Y}_1^{(k)} \right), \dots \right\}.$$

Let, for $\tilde{\mathbf{t}} \in \mathbf{I}^{(m_1+\dots+m_k)}$,

$$r^{(m,k)}(\varphi, \tilde{\mathbf{t}}) = r^{(m,k)}(\varphi, \mathbf{t}_1, \dots, \mathbf{t}_k) \\ = \mathbb{E} \left( \varphi \left( \mathbf{Y}_1^{(1)}, \dots, \mathbf{Y}_{m_1}^{(1)}; \dots; \mathbf{Y}_1^{(k)}, \dots, Y_{m_k}^{(k)} \right) \mid \left( \mathbf{X}_1^{(j)}, \dots, \mathbf{X}_{m_j}^{(j)} \right) = \mathbf{t}_j, j = 1, \dots, k \right),$$

where $\varphi$ is assumed, without loss of generality, to be symmetric within each of its $\ell$ blocks of arguments. Corresponding to the "kernel" $\varphi$ and assuming $n_1 \ge m_1, \dots, n_k \ge m_k$, the conditional U-statistic for estimation of $r^{(m,k)}(\varphi, \mathbf{t})$ is defined as

$$\hat{r}_n^{(m,k)}(\varphi, \mathbf{t}) = \frac{\sum_c \varphi(\mathbf{Y}_{i_{11}}^{(1)}, \dots, \mathbf{Y}_{i_{1m_1}}^{(1)}; \dots; \mathbf{Y}_{i_{k1}}^{(k)}, \dots, \mathbf{Y}_{i_{km_k}}^{(k)}) \mathbf{K}(\mathbf{X}_{i_{11}}^{(1)}, \dots, \mathbf{X}_{i_{1m_1}}^{(1)}; \dots; \mathbf{X}_{i_{k1}}^{(k)}, \dots, \mathbf{X}_{i_{km_k}}^{(k)})}{\sum_c \mathbf{K}(\mathbf{X}_{i_{11}}^{(1)}, \dots, \mathbf{X}_{i_{1m_1}}^{(1)}; \dots; \mathbf{X}_{i_{k1}}^{(k)}, \dots, \mathbf{X}_{i_{km_k}}^{(k)})},$$

where

$$\mathbf{K}(\mathbf{X}_{i_{11}}^{(1)}, \dots, \mathbf{X}_{i_{1m_1}}^{(1)}; \dots; \mathbf{X}_{i_{m_1}}^{(k)}, \dots, \mathbf{X}_{i_{km_k}}^{(k)}) \\ = \prod_{\ell=1}^{k} \prod_{j=1}^{m_\ell} K \left( \frac{\mathbf{t}_j - \mathbf{X}_{i_{\ell j}}^{(\ell)}}{h_{2,n}} \alpha \left( f_{\mathbf{X},n}(\mathbf{X}_{i_{\ell j}}^{(\ell)}; h_{1,n}) \right) \right) \alpha^p \left( f_{\mathbf{X},n}(\mathbf{X}_{i_{\ell j}}^{(\ell)}; h_{1,n}) \right).$$

Here $\{i_{j1}, \dots, i_{jm_j}\}$ represents a set of $k_j$ distinct elements of the set $\{1, 2, \dots, n_j\}$ $1 \le j \le \ell$, and $\Sigma_c$ denotes summation over all such combinations. The extension of Hoeffding (1948) treatment of one-sample U-statistics to the $\ell$ sample case is due to Lehmann (1951) and Dwass (1956). One can use Proposition 3.3.8 to infer that

$$|\hat{r}_n^{(m,k)}(\varphi, \tilde{\mathbf{t}}) - r^{(m,k)}(\varphi, \tilde{\mathbf{t}})| \to 0. \tag{5.8}$$

## 5.7 Set indexed conditional U-statistics

We aim to study the links between $\mathbf{X}$ and $\mathbf{Y}$, by estimating functional operators associated to the conditional distribution of $\mathbf{Y}$ given $\mathbf{X}$ such as the regression operator, for $C_1 \times \dots \times C_m := \tilde{\mathbb{C}} \in \mathbb{R}^{qk}$,

$$\mathbb{G}^{(k)}(C_1 \times \dots \times C_m | \tilde{\mathbf{t}}) = \mathbb{E} \left( \prod_{i=1}^{k} \mathbb{1}_{\{\mathbf{Y}_i \in C_i\}} \mid (\mathbf{X}_1, \dots, \mathbf{X}_k) = \tilde{\mathbf{t}} \right) \text{ for } \tilde{\mathbf{t}} \in \mathbb{R}^{pk}.$$
As a particular case of (2.10), we estimate $\mathbb{G}^{(k)}(C_1 \times \cdots \times C_m | \tilde{\mathbf{t}})$

$$ \hat{\mathbb{G}}_n^{(k)}(\tilde{\mathbb{C}}, \mathbf{t}) = \frac{\sum_{(i_1, \dots, i_k) \in I(k,n)} \mathbb{1}_{\{Y_{ij} \in C_j\}} \prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))}{\sum_{(i_1, \dots, i_k) \in I(k,n)} \prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))} \quad (5.9) $$

One can apply Proposition 3.3.8 to infer that

$$ \left| \hat{\mathbb{G}}_n^{(k)}(\tilde{\mathbb{C}}, \tilde{\mathbf{t}}) - \mathbb{G}^{(k)}(\tilde{\mathbb{C}} | \tilde{\mathbf{t}}) \right| \to 0. \quad (5.10) $$

## 5.8 Testing stochastic monotonicity

This application is given in Chen and Kato (2020). Let us recall their setting. Let $\mathbf{X}, \mathbf{Y}$ be real-valued random variables and denote by $F_{\mathbf{Y}|\mathbf{X}}(\mathbf{y} | \mathbf{x})$ the conditional distribution function of $\mathbf{Y}$ given $\mathbf{X}$. Consider the problem of testing the stochastic monotonicity

$$ H_0: F_{\mathbf{Y}|\mathbf{X}}(\mathbf{y} | \mathbf{x}) \le F_{\mathbf{Y}|\mathbf{X}}(\mathbf{y} | \mathbf{x}') \quad \forall \mathbf{y} \in \mathbb{R} \quad \text{whenever } \mathbf{x} \ge \mathbf{x}'. $$

Testing for the stochastic monotonicity is an important topic in a variety of applied fields such as economics Solon (1992); Blundell et al. (2007); Ellison and Ellison (2011). For this problem, Lee et al. (2009) consider a test for $H_0$ based on a local Kendall’s tau statistic, inspired by Ghosal et al. (2000). Let $(\mathbf{X}_i, \mathbf{Y}_i), i = 1, \dots, n$ be i.i.d. copies of $(\mathbf{X}, \mathbf{Y})$. Lee et al. (2009) consider the $U$-process

$$ U_n(\mathbf{x}, \mathbf{y}) = \frac{1}{n(n-1)} \sum_{1 \le i \ne j \le n} \{\mathbb{1}(\mathbf{Y}_i \le \mathbf{y}) - \mathbb{1}(\mathbf{Y}_j \le \mathbf{y})\} \mathrm{sign}(\mathbf{X}_i - \mathbf{X}_j) K\left(\frac{\mathbf{x} - \mathbf{X}_i}{h_n}\right) K\left(\frac{\mathbf{x} - \mathbf{X}_j}{h_n}\right), $$

where $\mathrm{sign}(\mathbf{x}) = \mathbb{1}(\mathbf{x} > 0) - \mathbb{1}(\mathbf{x} < 0)$ is the sign function. They propose to reject the null hypothesis if

$$ S_n = \sup_{(\mathbf{x}, \mathbf{y}) \in \mathcal{X} \times \mathcal{Y}} U_n(\mathbf{x}, \mathbf{y}) / c_n(\mathbf{x}) $$

is large, where $\mathcal{X}, \mathcal{Y}$ are subsets of the supports of $\mathbf{X}, \mathbf{Y}$, respectively and $c_n(\mathbf{x}) > 0$ is a suitable normalizing constant. Under independence between $\mathbf{X}$ and $\mathbf{Y}$, and under regularity conditions, they derive a Gumbel limiting distribution for a properly scaled version of $S_n$ using techniques from (Piterbarg, 1996). Using the procedure of the present paper, we can give the following generalisation

$$ U_n(\mathbf{x}, \mathbf{y}) = \frac{1}{n(n-1)} \sum_{1 \le i \ne j \le n} \{\mathbb{1}(\mathbf{Y}_i \le \mathbf{y}) - \mathbb{1}(\mathbf{Y}_j \le \mathbf{y})\} \mathrm{sign}(\mathbf{X}_i - \mathbf{X}_j) \\ \times \prod_{j=1}^{2} K\left(\frac{\mathbf{x} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n})). $$

## 5.9 Testing curvature and monotonicity of nonparametric regression

This application is given in Chen and Kato (2020). Let us remember their setting. Consider the nonparametric regression model

$$ \mathbf{Y} = f(\mathbf{X}) + \varepsilon \quad \text{with} \quad \mathbb{E}[\varepsilon | \mathbf{X}] = 0, $$

where $\mathbf{Y}$ is a scalar outcome variable, $\mathbf{X}$ is an $k$-dimensional vector of regressors, $\varepsilon$ is an error term, and $f$ is the conditional mean function $f(\mathbf{x}) = \mathbb{E}[\mathbf{Y} | \mathbf{X} = \mathbf{x}]$. We observe i.i.d. copies $\mathbf{V}_i = (\mathbf{X}_i, \mathbf{Y}_i), i = 1, \dots, n$ of $\mathbf{V} = (\mathbf{X}, \mathbf{Y})$. We are interested in testing for qualitative features (e.g., curvature, monotonicity) of the regression function $f(\cdot)$. Abrevaya and Jiang (2005b) consider a simplex statistic to test linearity, concavity, and convexity of $f$ under the assumption that the conditional distribution of $\varepsilon$ given $\mathbf{X}$ is symmetric. To define their test statistics, for $\mathbf{x}_1, \dots, \mathbf{x}_{k+1} \in \mathbb{R}^k$, let

$$ \Delta^{\circ}(\mathbf{x}_1, \dots, \mathbf{x}_{m+1}) = \left\{ \sum_{i=1}^{k+1} a_i \mathbf{x}_i : 0 < a_j < 1, j = 1, \dots, k+1, \sum_{i=1}^{k+1} a_i = 1 \right\} $$

denote the interior of the simplex spanned by $\mathbf{x}_1, \dots, \mathbf{x}_{k+1}$, and define

$$ \mathcal{D} = \bigcup_{j=1}^{k+2} \mathcal{D}_j, $$

where

$$ \mathcal{D}_j = \left\{ (\mathbf{x}_1, \dots, \mathbf{x}_{k+2}) \in \mathbb{R}^{k \times (k+2)} : \begin{array}{l} \mathbf{x}_1, \dots, \mathbf{x}_{k-1}, \mathbf{x}_{k+1}, \dots, \mathbf{x}_{k+2} \text{ are affinely independent} \\ \text{and } \mathbf{x}_j \in \Delta^{\circ}(\mathbf{x}_1, \dots, \mathbf{x}_{j-1}, \mathbf{x}_{j+1}, \dots, \mathbf{x}_{k+2}) \end{array} \right\}. $$
The sets $\mathcal{D}_1, \dots, \mathcal{D}_{k+2}$ are disjoint. For given $\mathbf{v}_i = (\mathbf{x}_i, y_i) \in \mathbb{R}^k \times \mathbb{R}$, $i = 1, \dots, k+2$, if $(\mathbf{x}_1, \dots, \mathbf{x}_{k+2}) \in \mathcal{D}$ then there exist a unique index $j = 1, \dots, k+2$ and a unique vector $(a_i)_{1 \le i \le k+2, i \ne j}$ such that $0 < a_i < 1$ for all $i \ne j$,

$$ \sum_{i \neq j} a_i = 1, \text{ and } \mathbf{x}_j = \sum_{i \neq j} a_i \mathbf{x}_i, $$

then, define

$$ w(v_1, \dots, v_{k+2}) = \sum_{i \neq j} a_i y_i - y_j. $$

The index $j$ and vector $(a_i)_{1 \le i \le k+2, i \ne j}$ are functions of $\mathbf{x}_i$'s. The set $\mathcal{D}$ is symmetric (i.e., its indicator function is symmetric) and $w(v_1, \dots, v_{k+2})$ is symmetric in its arguments. Under this notation, Abrevaya and Jiang (2005b) consider the following localized simplex statistic

$$ U_n(x) = \frac{(n-k-2)!}{n!} \sum_{(i_1, \dots, i_{k+2}) \in I_{n,k+2}} \varphi(\mathbf{V}_{i_1}, \dots, \mathbf{V}_{i_{k+2}}) \prod_{j=1}^{k+2} K \left( \frac{\mathbf{x} - \mathbf{X}_{ij}}{h_n} \right), \quad (5.11) $$

where

$$ \varphi(\mathbf{v}_1, \dots, \mathbf{v}_{k+2}) = 1\{ (\mathbf{x}_1, \dots, \mathbf{x}_{k+2}) \in \mathcal{D} \} \mathrm{sign}(w(\mathbf{v}_1, \dots, \mathbf{v}_{k+2})), $$

which is a U-statistic of order $(k+2)$. To test concavity and convexity of $f$, Abrevaya and Jiang (2005b) propose to reject the hypotheses if

$$ \bar{S}_n = \sup_{\mathbf{x} \in \mathcal{X}} U_n(\mathbf{x})/c_n(\mathbf{x}) \text{ and } S_n = \inf_{\mathbf{x} \in \mathcal{X}} U_n(\mathbf{x})/c_n(\mathbf{x}) $$

are large and small, respectively, where $\mathcal{X}$ is a subset of the support of $\mathbf{X}$ and $c_n(\mathbf{x}) > 0$ is a suitable normalizing constant. On the other hand, Ghosal et al. (2000) study testing monotonicity of $f(\cdot)$ when $k = 1$ and $\varepsilon$ is independent of $\mathbf{X}$. To be specific, they test whether $f(\cdot)$ is increasing, and propose to reject the hypothesis if

$$ S_n = \sup_{\mathbf{x} \in \mathcal{X}} \tilde{U}_n(\mathbf{x})/c_n(\mathbf{x}), $$

is large, where $\mathcal{X}$ is a subset of the support of $\mathbf{X}$,

$$ \tilde{U}_n(\mathbf{x}) = \frac{1}{n(n-1)} \sum_{1 \le i \ne j \le n} \mathrm{sign}(\mathbf{Y}_j - \mathbf{Y}_i) \mathrm{sign}(\mathbf{X}_i - \mathbf{X}_j) K \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha (f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n})) \right) K \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha (f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n})) \right), \quad (5.12) $$

and $c_n(\mathbf{t}) > 0$ is a suitable normalizing constant. Ghosal et al. (2000) argue that as far as the size control is concerned, it is enough to choose, as a critical value, the $(1 - \alpha)$-quantile of $S_n$ when $f \equiv 0$, under which $U_n(\mathbf{x})$ is centered. Under $f \equiv 0$ and regularity conditions, Ghosal et al. (2000) derive a Gumbel limiting distribution for a properly scaled version of $S_n$. Using the methods proposed in the present paper, alternative procedures to (5.11) and (5.12) are given by

$$ U_n(\mathbf{x}) = \frac{(n-k-2)!}{n!} \sum_{(i_1, \dots, i_{k+2}) \in I_{n,k+2}} \varphi(\mathbf{V}_{i_1}, \dots, \mathbf{V}_{i_{k+2}}) \prod_{j=1}^{k+2} K \left( \frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha (f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n})) \right) \alpha^p (f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n})), \quad (5.13) $$

and

$$ \tilde{U}_n(\mathbf{x}) = \frac{1}{n(n-1)} \sum_{1 \le i \ne j \le n} \mathrm{sign}(\mathbf{Y}_j - \mathbf{Y}_i) \mathrm{sign}(\mathbf{X}_i - \mathbf{X}_j) \prod_{j=1}^{2} K \left( \frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha (f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n})) \right) \alpha^p (f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n})). \quad (5.14) $$

## 5.10 Examples of U-statistics

**Example 5.10.1.** Let $\tilde{Y}_1\tilde{Y}_2$ denote the oriented angle between $Y_1, Y_2 \in T$, $T$ is the circle of radius 1 and center 0 in $\mathbb{R}^2$. Let:

$$ \varphi_t(Y_1, Y_2) = 1\{\tilde{Y}_1\tilde{Y}_2 \le t\} - t/\pi, \text{ for } t \in [0, \pi). $$

Silverman (1978) has used this kernel in order to propose a U-process to test uniformity on the circle.

**Example 5.10.2.** Hoeffding (1948) introduced the parameter

$$ \Delta = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} D^2(y_1, y_2) dF(y_1, y_2), $$

where

$$ D(y_1, y_2) = F(y_1, y_2) - F(y_1, \infty)F(\infty, y_2) $$
and $F(\cdot, \cdot)$ is the distribution function of $Y_1$ and $Y_2$. The parameter $\Delta$ has the property that $\Delta = 0$ if and only if $Y_1$ and $Y_2$ are independent. From Lee (1990), an alternative expression for $\Delta$ can be developed by introducing the functions

$$ \psi(y_1, y_2, y_3) = \begin{cases} 1 & \text{if } y_2 \le y_1 < y_3 \\ 0 & \text{if } y_1 < y_2, y_3 \text{ or } y_1 \ge y_2, y_3 \\ -1 & \text{if } y_3 \le y_1 < y_2 \end{cases} $$

and

$$ \varphi(y_{1,1}, y_{1,2}, \dots, y_{5,1}, y_{5,2}) = \frac{1}{4} \psi(y_{1,1}, y_{1,2}, y_{1,3}) \psi(y_{1,1}, y_{1,4}, y_{1,5}) \psi(y_{1,2}, y_{2,2}, y_{3,2}) \psi(y_{1,2}, y_{4,2}, y_{5,2}). $$

We have

$$ \Delta = \int \dots \int \varphi(y_{1,1}, y_{1,2}, \dots, y_{5,1}, y_{5,2}) dF(y_{1,1}, y_{1,2}) \dots dF(y_{1,5}, y_{2,5}). $$

We have

$$ \begin{aligned} r^{(5)}(\varphi, t_1, t_2, t_3, t_4, t_5) &= \mathbb{E}(\varphi((Y_{1,1}, Y_{1,2}), \dots, (Y_{5,1}, Y_{5,2}))) | X_1 = X_2 = X_3 = X_4 = X_5 = t). \end{aligned} $$

The corresponding U-statistics may be used to test the conditional independence.

**Example 5.10.3.** For $m = 3$, let $\varphi(Y_1, Y_2, Y_3) = 1\{Y_1 - Y_2 - Y_3 > 0\}$, the corresponding U-Statistic corresponds to the Hollander-Proschan test-statistic (Hollander and Proschan (1972)).

**Example 5.10.4.** (Hoeffding's D). From the symmetric kernel,

$$ \begin{aligned} h_D(z_1, \dots, z_5) &:= \frac{1}{16} \sum_{(i_1, \dots, i_5) \in \mathcal{P}_5} \left[ \left\{ 1(z_{i_1,1} \le z_{i_5,1}) - 1(z_{i_2,1} \le z_{i_5,1}) \right\} \left\{ 1(z_{i_3,1} \le z_{i_5,1}) - 1(z_{i_4,1} \le z_{i_5,1}) \right\} \right] \right. \\ & \qquad \left. \times \left[ \left\{ 1(z_{i_1,2} \le z_{i_5,2}) - 1(z_{i_2,2} \le z_{i_5,2}) \right\} \left\{ 1(z_{i_3,2} \le z_{i_5,2}) - 1(z_{i_4,2} \le z_{i_5,2}) \right\} \right] \right], \end{aligned} $$

we recover Hoeffding's D statistic, which is a rank-based U-statistic of order 5 and gives rise to the Hoeffding's D correlation measure $\mathbb{E}h_D$.

**Example 5.10.5.** (Blum-Kiefer-Rosenblatt's R). The symmetric kernel

$$ \begin{aligned} h_R(z_1, \dots, z_6) &:= \frac{1}{32} \sum_{(i_1, \dots, i_6) \in \mathcal{P}_6} \left[ \left\{ 1(z_{i_1,1} \le z_{i_5,1}) - 1(z_{i_2,1} \le z_{i_5,1}) \right\} \left\{ 1(z_{i_3,1} \le z_{i_5,1}) - 1(z_{i_4,1} \le z_{i_5,1}) \right\} \right] \right. \\ & \qquad \left. \times \left[ \left\{ 1(z_{i_1,2} \le z_{i_6,2}) - 1(z_{i_2,2} \le z_{i_6,2}) \right\} \left\{ 1(z_{i_3,2} \le z_{i_6,2}) - 1(z_{i_4,2} \le z_{i_6,2}) \right\} \right] \right] \end{aligned} $$

yields Blum-Kiefer-Rosenblatt's R statistic (Blum et al. (1961)), which is a rank-based U-statistic of order 6.

**Example 5.10.6.** (Bergsma-Dassios-Yanagimoto's $\tau^*$). Bergsma and Dassios (2014) introduced a rank correlation statistic as a U-statistic of order 4 with the symmetric kernel

$$ \begin{aligned} h_{\tau^*}(z_1, \dots, z_4) &:= \frac{1}{16} \sum_{(i_1, \dots, i_4) \in \mathcal{P}_4} \left\{ \begin{aligned} & 1(z_{i_1,1}, z_{i_3,1} < z_{i_2,1}, z_{i_4,1}) + 1(z_{i_2,1}, z_{i_4,1} < z_{i_1,1}, z_{i_3,1}) \\ & - 1(z_{i_1,1}, z_{i_4,1} < z_{i_2,1}, z_{i_3,1}) - 1(z_{i_2,1}, z_{i_3,1} < z_{i_1,1}, z_{i_4,1}) \\ & \times \begin{aligned}[t] \{ 1(z_{i_1,2}, z_{i_3,2} < z_{i_2,2}, z_{i_4,2}) + 1(z_{i_2,2}, z_{i_4,2} < z_{i_1,2}, z_{i_3,2}) \end{aligned} \\ & - 1(z_{i_1,2}, z_{i_4,2} < z_{i_2,2}, z_{i_3,2}) - 1(z_{i_2,2}, z_{i_3,2} < z_{i_1,2}, z_{i_4,2}) \end{aligned} \right\} \end{aligned} $$

Here, $1(y_1, y_2 < y_3, y_4) := 1(y_1 < y_3) 1(y_1 < y_4) 1(y_2 < y_3) 1(y_2 < y_4)$.

**Example 5.10.7.** The sample covariance matrix

$$ \hat{S}_n = (n-1)^{-1} \sum_{i=1}^{n} (X_i - \bar{X}) (X_i - \bar{X})^{\top} $$

is an unbiased estimator of the covariance matrix $\Sigma = \text{Cov}(X_1)$. Here, $\hat{S}_n$ is a matrix-valued U-statistic of form (1) with the quadratic kernel $h(x_1, x_2) = (x_1 - x_2)(x_1 - x_2)^{\top}/2$ for $x_1, x_2 \in \mathbb{R}^p$.
**Example 5.10.8.** Two generic vectors $y = (y_1, y_2)$ and $z = (z_1, z_2)$ in $\mathbb{R}^2$ are said to be concordant if

$$ (y_1 - z_1)(y_2 - z_2) > 0. $$

For $m, k = 1, \dots, p$, define

$$ \tau_{mk} = \frac{1}{n(n-1)} \sum_{1 \le i \ne j \le n} 1 \{ (X_{im} - X_{jm})(X_{ik} - X_{jk}) > 0 \}. $$

Then Kendall's tau rank correlation coefficient matrix $T = \{\tau_{mk}\}_{m,k=1}^p$ is a matrix valued U-statistic with a bounded kernel. It is clear that $\tau_{mk}$ quantifies the monotonic dependency between $(X_{1m}, X_{1k})$ and $(X_{2m}, X_{2k})$ and it is an unbiased estimator of $P((X_{1m} - X_{2m})(X_{1k} - X_{2k}) > 0)$, that is, the probability that $(X_{1m}, X_{1k})$ and $(X_{2m}, X_{2k})$ are concordant.

**Example 5.10.9.** The Gini mean difference. The Gini index provides another popular measure of dispersion. It corresponds to the case where $E \subset \mathbb{R}$ and $h(x, y) = |x - y|$:

$$ G_n = \frac{2}{n(n-1)} \sum_{1 \le i < j \le n} |X_i - X_j|. $$

**Example 5.10.10.** The Wilcoxon Statistic. Suppose that $E \subset \mathbb{R}$ is symmetric around zero. As an estimate of the quantity $\int_{(x,y) \in E^2} \{2\mathbb{I}_{\{x+y>0\}} - 1\}$ it is pertinent to consider the statistic

$$ W_n = \frac{2}{n(n-1)} \sum_{1 \le i < j \le n} \{2 \cdot 1_{\{X_i+X_j>0\}} - 1\}, $$

which is relevant for testing whether or not $\mu$ is located at zero.

**Example 5.10.11.** The Takens estimator. Suppose that $E \subset \mathbb{R}^d$, $d \ge 1$. Denote by $\| \cdot \|$ the usual Euclidean norm on $\mathbb{R}^d$. In Borovkova et al. (1999), the following estimate of the correlation integral,

$$ C_F(r) = \int \mathbb{I}_{\{\|x-x'\| \le r\}} dF(x) dF(x'), \quad r > 0, $$

is considered:

$$ C_n(r) = \frac{1}{n(n-1)} \sum_{1 \le i \ne j \le n} \mathbb{I}_{\{\|x_i - x_j\| \le r\}} $$

In the case where a scaling law holds for the correlation integral, i.e., when there exists $(\alpha, r_0, c) \in \mathbb{R}_+^3$ such that $C_F(r) = c \cdot r^{-\alpha}$ for $0 < r \le r_0$, the U-statistic

$$ T_n = \frac{1}{n(n-1)} \sum_{1 \le i \ne j \le n} \log \left( \frac{\|X_i - X_j\|}{r_0} \right), $$

is used in order to build the Takens estimator $\hat{\alpha}_n = -T_n^{-1}$ of the correlation dimension $\alpha$.

# 6 The bandwidth selection criterion

Following Dony and Mason (2008) and Bouzebda and Nemouchi (2023), the leave-one-out cross validation procedure allows to define, for any fixed $\mathbf{i} = (i_1, \dots, i_k) \in I(k, n)$:

$$ \check{r}_{n, \mathbf{i}}^{(k)}(\varphi, \tilde{\mathbf{t}}; h_{1,n}, h_{2,n}) = \frac{\sum_{(j_1, \dots, j_k) \in I(k,n)(\mathbf{i})} \varphi(\mathbf{Y}_{j_1}, \dots, \mathbf{Y}_{j_k}) \prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))}{\sum_{(i_1, \dots, i_k) \in I(k,n)} \prod_{j=1}^{k} K\left(\frac{\mathbf{t}_j - \mathbf{X}_{ij}}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_{ij}; h_{1,n}))}, \quad (6.1) $$

where

$$ I(k, n)(\mathbf{i}) := \{\mathbf{j} \in I(k, n) \text{ and } \mathbf{j} \neq \mathbf{i}\} = I(k, n) \setminus \{\mathbf{i}\}. $$

To minimize the quadratic loss function, we introduce the following criterion, we have for some (known) non-negative weight function $\mathcal{W}(\cdot)$:

$$ CV(\varphi, h) := \frac{(n-k)!}{n!} \sum_{\mathbf{i} \in I(k,n)} \left( \varphi(\mathbf{Y}_{\mathbf{i}}) - \check{r}_{n,\mathbf{i}}^{(k)}(\varphi, \mathbf{X}_{\mathbf{i}}; h_{1,n}, h_{2,n}) \right)^2 \tilde{\mathcal{W}}(\mathbf{X}_{\mathbf{i}}), \quad (6.2) $$

where

$$ \tilde{\mathcal{W}}(\tilde{\mathbf{i}}) := \prod_{i=1}^{k} \mathcal{W}(\mathbf{t}_i). $$
A natural way for choosing the bandwidth is to minimize the precedent criterion, so let's choose $\hat{m}_n$ minimizing

$$ \sup_{\varphi \in \mathcal{F}} \mathrm{CV}(\varphi, h). $$

One can replace (6.2) by

$$ \mathrm{CV}(\varphi, h) := \frac{(n-k)!}{n!} \sum_{\mathbf{i} \in I(k,n)} \left( \varphi(\mathbf{Y}_{\mathbf{i}}) - \tilde{r}_{n,\mathbf{i}}^{(k)}(\varphi, \mathbf{X}_{\mathbf{i}}; h_{1,n}, h_{2,n}) \right)^2 \widehat{\mathcal{W}}(\mathbf{X}_{\mathbf{i}}, \mathbf{t}), \quad (6.3) $$

where

$$ \widehat{\mathcal{W}}(\tilde{\mathbf{s}}, \tilde{\mathbf{t}}) := \prod_{i=1}^{k} \widehat{W}(\mathbf{s}_i, \mathbf{t}_i). $$

In practice, one takes for $\mathbf{i} \in I(k,n)$, the uniform global weights $\widehat{\mathcal{W}}(\mathbf{X}_{\mathbf{i}}) = 1$, and the local weights

$$ \widehat{W}(\mathbf{X}_{\mathbf{i}}, \mathbf{t}) = \begin{cases} 1 & \text{if } \|\mathbf{X}_{\mathbf{i}} - \mathbf{t}\| \le h, \\ 0 & \text{otherwise.} \end{cases} $$

For the sake of brevity, we have just considered the most popular method, that is, the cross-validated selected bandwidth. This may be extended to any other bandwidth selector, such the bandwidth based on Bayesian ideas Shang (2014).

**Remark 6.0.1.** We can utilize a different bandwidth criterion suggested by Silverman (1986), the rule of thumb. Strictly speaking, since the cross-validated bandwidth $m_n$ is random, the asymptotic theory can only be justified using a specific stochastic equicontinuity argument. For testing a parametric model for conditional mean function against a nonparametric alternative, Horowitz and Spokoiny (2001) proposed an adaptive-rate-optimal rule. Gao and Gijbels (2008) presents another method for selecting a proper bandwidth. Gao and Gijbels (2008) propose, using the Edgeworth expansion of the asymptotic distribution of the test, to choose the bandwidth such that the power function of the test problem is maximized while the size function is controlled. Although any choice of bandwidth $h_n$ that fulfills the assumption will produce the asymptotic distribution in Proposition 3.3.8, we need guidance on how to choose $h_n$ in practice.

**Remark 6.0.2.** Since the confidence bands depend upon $h$, an optimal choice of $h$ with respect to some criterion will improve them. Notice that the paper Deheuvels and Mason (2004) consider local plug-in type estimators $\hat{h}_n = \hat{h}_n(\mathbf{x})$ of $h$ that satisfy the condition

$$ \mathbb{P} \left( a_n \le \hat{h}_n(\mathbf{x}) \le b_n : \mathbf{x} \in \mathbf{I} \right) \to 1, $$

where $a_n = c_1h$ and $b_n = c_2h$, with $c_1 < c_2$, or fulfill, for any $\varepsilon > 0$

$$ \mathbb{P} \left( \sup_{\mathbf{x} \in \mathbf{I}} \left| \frac{\hat{h}_n(\mathbf{x})}{h} - \varphi(\mathbf{x}) \right| > \varepsilon \right) \to 0, \quad (6.4) $$

where $\varphi(\cdot)$ is an appropriate continuous function on $\mathbf{I}$.

# 7 Monte Carlo experiments

The computing program codes are implemented in $\mathbb{R}$. The setup of our simulation study closely follows that of Derumigny and Fermanian (2019) and Bouzebda et al. (2021). More precisely, we consider the case of drawing i.i.d. random samples $(X_i, Y_{i,1}, Y_{i,2})$, $i = 1, \dots, n$, with univariate explanatory variables. We assume that underlying distribution is continuous. Remark that the conditional Kendall's tau, see for instance Kendall (1938), can be defined as follows

$$ \begin{aligned} \tau_{1,2|X=t} &= 4\mathbb{P}(Y_{1,1} > Y_{2,1}, Y_{1,2} > Y_{2,2} | X_1 = X_2 = t) - 1 \\ &= 1 - 4\mathbb{P}(Y_{1,1} > Y_{2,1}, Y_{1,2} < Y_{2,2} | X_1 = X_2 = t) \\ &= \mathbb{P}((Y_{1,1} - Y_{2,1})(Y_{1,2} - Y_{2,2}) > 0 | X_1 = X_2 = t) \\ &\quad - \mathbb{P}((Y_{1,1} - Y_{2,1})(Y_{1,2} - Y_{2,2}) < 0 | X_1 = X_2 = t). \end{aligned} $$

Motivated by these expressions, Derumigny and Fermanian (2019) introduced several kernel-based estimators of $\tau_{1,2|X=t}$:

$$ \hat{\tau}_{1,2|X=t}^{(1)} := 4 \sum_{i=1}^{n} \sum_{j=1}^{n} w_{i,n}(t) w_{j,n}(t) \mathbb{1}\{Y_{i,1} < Y_{j,1}, Y_{i,2} < Y_{j,2}\} - 1, $$

$$ \hat{\tau}_{1,2|X=t}^{(2)} := \sum_{i=1}^{n} \sum_{j=1}^{n} w_{i,n}(t) w_{j,n}(t) \left( \mathbb{1}\{ (Y_{i,1} - Y_{j,1}) (Y_{i,2} - Y_{j,2}) > 0 \} \right) $$
$$-1\{(Y_{i,1}-Y_{j,1}).(Y_{i,2}-Y_{j,2})<0\}),$$

$$\hat{\tau}_{1,2|X=t}^{(3)} := 1 - 4 \sum_{i=1}^{n} \sum_{j=1}^{n} w_{i,n}(t)w_{j,n}(t) \mathbf{1}\{Y_{i,1} < Y_{j,1}, Y_{i,2} > Y_{j,2}\},$$

where $\mathbf{1}\{A\}$ denotes as usual the indicator function of the set $A$, $w_{i,n}(\cdot)$ is a sequence of Nadaraya-Watson weights defined by

$$w_{i,n}(t) = \frac{K_h(X_i - t)}{\sum_{j=1}^{n} K_h(X_j - t)}, \qquad (7.1)$$

with the notation $K_h(\cdot) := h^{-1}K(\cdot/h)$ for some kernel $K(\cdot)$ on $\mathbb{R}$, and $h = h(n)$ denotes a usual bandwidth sequence that tends to zero when $n \to \infty$. Let us introduce our weights

$$\tilde{w}_{i,n}(t) = \frac{K_{h_1}((X_i - t)\alpha(f_{n,h_2}(X_i)))\alpha(f_{n,h_2}(X_i))}{\sum_{j=1}^{n} K_{h_1}((X_j - t)\alpha(f_{n,h_2}(X_j)))\alpha(f_{n,h_2}(X_j))}, \qquad (7.2)$$

where

$$\alpha(w) = cp^{1/2} (w/c^2) \qquad (7.3)$$

and the five time differentiable clipping function $p(\cdot)$ with $t_0 = 2$, given in Remark 2.0.1. Instead, we take the bandwidths, $h_{1,n}$ and $h_{2,n}$, as $n^{-1/7}$ and $n^{-1/9}$ respectively for all following simulation study including the graphs and numerical comparison. In this simulation study we set $c$ in (7.3) to be 0.99. These weights are a bit different from those used before, which are adapted to the present setting. The variable kernel estimators are defined by

$$\tilde{\tau}_{1,2|X=t}^{(1)} := 4 \sum_{i=1}^{n} \sum_{j=1}^{n} \tilde{w}_{i,n}(t) \tilde{w}_{j,n}(t) \mathbf{1}\{Y_{i,1} < Y_{j,1}, Y_{i,2} < Y_{j,2}\} - 1,$$

$$\tilde{\tau}_{1,2|X=t}^{(2)} := \sum_{i=1}^{n} \sum_{j=1}^{n} \tilde{w}_{i,n}(t) \tilde{w}_{j,n}(t) \left( \mathbf{1}\{(Y_{i,1} - Y_{j,1}).(Y_{i,2} - Y_{j,2}) > 0\} - \mathbf{1}\{(Y_{i,1} - Y_{j,1}).(Y_{i,2} - Y_{j,2}) < 0\} \right),$$

$$\tilde{\tau}_{1,2|X=t}^{(3)} := 1 - 4 \sum_{i=1}^{n} \sum_{j=1}^{n} \tilde{w}_{i,n}(t) \tilde{w}_{j,n}(t) \mathbf{1}\{Y_{i,1} < Y_{j,1}, Y_{i,2} > Y_{j,2}\}.$$

In this finite sample study, we consider two simple cases: bounded and/or unbounded explanatory variables respectively, given below.

1. The variable $X$ is uniformly on $[0, 1]$. Conditionally on $X = t$, $Y_1|X = t$ and $Y_2|X = t$ both follow a Gaussian distribution $N(t, 1)$. Their associated conditional copula is Gaussian and their conditional Kendall's tau is given by

$$\tau_{1,2|X=t} = 2t - 1.$$

2. The variable $X$ is a standard Gaussian variable $N(0, 1)$. Conditionally on $X = t$, $Y_1|X = t$ and $Y_2|X = t$ both follow a Gaussian distribution $N(\Phi(t), 1)$, where $\Phi(\cdot)$ is the cdf of the $X$. Their associated conditional copula is Gaussian and their conditional Kendall's tau is given by

$$\tau_{1,2|X=t} = 2\Phi(t) - 1.$$

In our simulation study, we make use of the following kernels:

* The tricube kernel

$$K(u) = \frac{70}{81} (1 - |u|^3)^3 \mathbf{1}_{|u| \le 1},$$

* the Gaussian kernel:

$$K(x/h) = \frac{1}{\sqrt{2\pi}} e^{-x^2/2h^2},$$

* the Epanechnikov (1969) kernel:

$$K(x/h) = \frac{3}{4} (1 - (x/h)^2)^3 \mathbf{1}\{|x/h| \le 1\},$$

* the beta kernel:

$$K_{t/h+1,(1-t)/h+1}(x) = \frac{x^{t/h}(1-x)^{(1-t)/h} \mathbf{1}\{0 \le x \le 1\}}{B(t/h+1,(1-t)/h+1)},$$

where $B(\cdot)$ is the beta function.
For the classical estimator, we adopt the rule-of-thumb method to select the bandwidth, i.e., we chose h to be $\alpha_h \hat{\sigma}(X)n^{-1/5}$ where $\alpha_h$ is some positive constant and $\hat{\sigma}(X)$ is the standard deviation of X. These frameworks allow us to examine the finite sample properties of our estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$. To this end, we compute our estimators, for each of the for kernels presented above, and each $n \in \{500, 1000, 2000\}$. For each setting, we consider three local measures of goodness-of-fit: for a given t and for any Kendall's tau estimate (say $\tilde{\tau}_{1,2|X=t}$), let

* the (local) bias: $Bias(t) := E[\tilde{\tau}_{1,2|X=t} - \tilde{\tau}_{1,2|X=t}$,

* the (local) standard deviation: $Sd(t) := E[(\tilde{\tau}_{1,2|X=t} - E[\tilde{\tau}_{1,2|X=t}])^2]^{1/2}$,

* the (local) mean square-error: $MSE(t) := E[(\tilde{\tau}_{1,2|X=t} - \tau_{1,2|X=t})^2]$.

We also consider their integrated version w.r.t the usual Lebesgue measure on the whole support of t, respectively denoted by IBias, ISd and IMSE. For the sake of effective calculations of these measures, all the theoretical previous expectations are replaced by their empirical counterparts based on 1000 simulations. Notice that, as in any other inferential context, the greater the sample size is, the better the performance is. Simple inspection of the results reported in the preceding Tables 2 and 3 allows deducing that large values of the sample size n give smaller IBias, ISd and IMSE. In most all situations, $\tilde{\tau}_{1,2|X=t}^{(k)}$ is particularly of good behavior, see Figures 4 and 8. We mention that $\tilde{\tau}_{1,2|X=t}^{(1)}$ and $\tilde{\tau}_{1,2|X=t}^{(3)}$ have also good performance that are not reported in this simulation study. Notice that the estimators based on the beta have better performance than the other kernels, see 4. Remark that the new estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$, for $k = 1, 2, 3$ behave better in terms of variance, in the two settings, see Figures 2 and 6. In terms of MSE, the proposed estimators outperform the classical ones in particular for the tricube, Epanechnikov and beta kernels for the setting 1, see Figure 3, tricube and Epanechnikov kernels for the setting 2, see Figure 7. In terms of bias, the new estimators have similar behavior as in the classical case, see Figures 1 and 5. In the present study, the main interest is to investigate the tricube that better fits of theoretical study. In this last situation, the proposed estimators ameliorate the existing ones. In addition, we have studied the other kernels in the simulation to show the behavior of the proposed estimators in other situation. One can see that the proposed estimators outperform the exiting ones in the case of Epanechnikov kernel. For the beta kernel, the performances of the proposed estimators and the existing ones are very similar. In the case of the normal kernel, the performances of our estimators can be ameliorated by doing extensive simulation to calibrate the bandwidth and the constant c in (7.3). In order to extract methodological recommendations for the use of the proposed estimators in this work, it will be interesting to conduct extensive Monte Carlo experiments to compare our procedures with other alternatives presented in the literature, but this would go well beyond the scope of the present paper.

# 8 Concluding remarks

In this research, we enhance the standard Stute estimator by introducing a variable bandwidth kernel conditional U-statistics estimator for independent and identically distributed data. We employ the empirical processes approaches in a nontrivial manner to establish uniform in t and in bandwidth consistency the proposed estimator. Our estimator incorporates as a particular case density estimators introduced by McKay (1993a) and the regression estimators investigated by Nakarmi et al. (2021). In addition, uniform consistency is established over $\varphi \in \mathcal{F}$ for a properly restricted class $\mathcal{F}$ given particular moment conditions in both the bounded and unbounded situations. For the estimators under discussion, our theorems allow data-driven local bandwidths. We also demonstrate the uniform bandwidth consistency for the nonparametric inverse probability of censoring weighted (I.P.C.W.) estimators of the regression function under random censorship, which is interesting in its own right. Our findings are demonstrated under a number of conventional structural conditions on the Vapnik-Chervonenkis classes of functions and under a some mild conditions on the model. The finite-sample performance of the suggested approach is illustrated in a simulation analysis. It will be of interest to investigate the weak convergence of conditional U-processes in the present setting. The main is the equicontinuity that needs more complicated control and details. The proof of such a statement, however, should require a different methodology than that used in the present paper, and we leave this problem open for future research. A future research direction would be to study the proposed estimator in the setting of serially dependent observations. Notice that mixing is some kind of asymptotic independence assumption which is commonly used for seek of simplicity but which can be unrealistic in situations where there is strong dependence between the data. Extending non-parametric conditional U-statistics to general dependence structure is a rather underdeveloped field. Note that the ergodic framework avoids the widely used strong mixing condition and its variants to measure the dependency and the very involved probabilistic calculations that it implies. It would be interesting to extend our work to the case of the functional ergodic data, which requires non trivial mathematics, this would go well beyond the scope of the present paper. It will of interest to consider oracle inequalities and upper bounds for conditional U-statistic estimators on a very broad class of metric spaces. Precisely, we will consider the setting of a doubling measure metric space in the presence of a non-negative self-adjoint operator whose heat kernel enjoys Gaussian regularity.

# 9 Mathematical development

This section is devoted to the proofs of our results. The previously presented notation continues to be used in the following. The proofs being quite lengthy, we limit ourselves to the main arguments. Define $\delta(\mathbf{t}) = \delta(\mathbf{t}, n)$ by the equation

$$ \delta(\mathbf{t}) = \frac{\alpha(f_{X:n}(\mathbf{t}; h_{1,n})) - \alpha(f(\mathbf{t}))}{\alpha(f(\mathbf{t}))} $$
<caption>Table 1: Results of the simulation in Setting 1. Bold values indicate optimal choices for the chosen measure of performance. These results are integrated measures of performance over the (0, 1) interval.</caption>

<table><caption>Table 2: Tracking daily activities.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">n = 500</th><th colspan="3">n = 1000</th><th colspan="3">n = 2000</th></tr><tr><th>IBias</th><th>ISd</th><th>IMSE</th><th>IBias</th><th>ISd</th><th>IMSE</th><th>IBias</th><th>ISd</th><th>IMSE</th></tr></thead><tbody><tr><td rowspan="5">Gaussian</td><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>0,0026</td><td>0,0014</td><td>0,0091</td><td>0,0038</td><td>0,0007</td><td>0,0064</td><td>0,0044</td><td>0,0004</td><td>0,0046</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0084</td><td>0,0014</td><td>0,0092</td><td>0,0069</td><td>0,0007</td><td>0,0065</td><td>0,0061</td><td>0,0004</td><td>0,0046</td></tr><tr><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0142</td><td>0,0014</td><td>0,0093</td><td>0,01</td><td>0,0007</td><td>0,0065</td><td>0,0078</td><td>0,0004</td><td>0,0047</td></tr><tr><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0026</td><td>0,0018</td><td>0,0054</td><td>-0,0006</td><td>0,001</td><td>0,0033</td><td>0,0006</td><td>0,0005</td><td>0,0021</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0049</td><td>0,0018</td><td>0,0054</td><td>0,0036</td><td>0,001</td><td>0,0033</td><td>0,003</td><td>0,0005</td><td>0,0021</td></tr><tr><td></td><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0125</td><td>0,0018</td><td>0,0055</td><td>0,0079</td><td>0,001</td><td>0,0034</td><td>0,0054</td><td>0,0005</td><td>0,0021</td></tr><tr><td rowspan="5">Epanechnikov</td><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0095</td><td>0,0028</td><td>0,0042</td><td>-0,0048</td><td>0,0014</td><td>0,0024</td><td>-0,0021</td><td>0,0007</td><td>0,0014</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0018</td><td>0,0028</td><td>0,0041</td><td>0,0013</td><td>0,0014</td><td>0,0024</td><td>0,0012</td><td>0,0007</td><td>0,0014</td></tr><tr><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0132</td><td>0,0028</td><td>0,0043</td><td>0,0075</td><td>0,0014</td><td>0,0024</td><td>0,0046</td><td>0,0007</td><td>0,0014</td></tr><tr><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0143</td><td>0,0038</td><td>0,0049</td><td>-0,0082</td><td>0,002</td><td>0,0027</td><td>-0,0045</td><td>0,0011</td><td>0,0015</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0008</td><td>0,0038</td><td>0,0047</td><td>0,0004</td><td>0,002</td><td>0,0026</td><td>0,0003</td><td>0,0011</td><td>0,0015</td></tr><tr><td></td><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,016</td><td>0,0038</td><td>0,0049</td><td>0,009</td><td>0,002</td><td>0,0027</td><td>0,0052</td><td>0,0011</td><td>0,0015</td></tr><tr><td rowspan="5">Tricube</td><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0111</td><td>0,0031</td><td>0,0043</td><td>-0,0058</td><td>0,0016</td><td>0,0024</td><td>-0,0028</td><td>0,0008</td><td><strong>0,0014</strong></td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0014</td><td>0,0031</td><td>0,0042</td><td>0,001</td><td>0,0016</td><td>0,0024</td><td>0,0009</td><td>0,0008</td><td><strong>0,0014</strong></td></tr><tr><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,014</td><td>0,0031</td><td>0,0044</td><td>0,0078</td><td>0,0016</td><td>0,0024</td><td>0,0047</td><td>0,0008</td><td><strong>0,0014</strong></td></tr><tr><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0173</td><td>0,0044</td><td>0,0055</td><td>-0,01</td><td>0,0024</td><td>0,003</td><td>-0,0056</td><td>0,0013</td><td>0,0016</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0004</td><td>0,0044</td><td>0,0052</td><td>0,0001</td><td>0,0024</td><td>0,0029</td><td><strong>0,0001</strong></td><td>0,0013</td><td>0,0016</td></tr><tr><td></td><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0183</td><td>0,0044</td><td>0,0056</td><td>0,0102</td><td>0,0024</td><td>0,003</td><td>0,0059</td><td>0,0013</td><td>0,0017</td></tr><tr><td rowspan="5">Beta</td><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>0,02</td><td>0,0011</td><td>0,0045</td><td>0,0207</td><td>0,0006</td><td>0,0031</td><td>0,0206</td><td><strong>0,0003</strong></td><td>0,0023</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0264</td><td>0,0011</td><td>0,0046</td><td>0,024</td><td>0,0006</td><td>0,0032</td><td>0,0224</td><td><strong>0,0003</strong></td><td>0,0024</td></tr><tr><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0329</td><td>0,0011</td><td>0,005</td><td>0,0273</td><td>0,0006</td><td>0,0034</td><td>0,0241</td><td><strong>0,0003</strong></td><td>0,0025</td></tr><tr><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>0,005</td><td>0,0012</td><td>0,0042</td><td>0,0062</td><td>0,0006</td><td>0,0028</td><td>0,0068</td><td><strong>0,0003</strong></td><td>0,0019</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0115</td><td>0,0012</td><td>0,0043</td><td>0,0096</td><td>0,0006</td><td>0,0028</td><td>0,0087</td><td><strong>0,0003</strong></td><td>0,002</td></tr><tr><td></td><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0179</td><td>0,0012</td><td>0,0045</td><td>0,013</td><td>0,0006</td><td>0,0029</td><td>0,0105</td><td><strong>0,0003</strong></td><td>0,002</td></tr></tbody></table>
<table><caption>Table 3: Results of the simulation in Setting 2. Bold values indicate optimal choices for the chosen measure of performance. These results are integrated measures of performance over the (−2, 2) interval.</caption><thead><tr><th rowspan="2"></th><th colspan="3">n = 500</th><th colspan="3">n = 1000</th><th colspan="3">n = 2000</th></tr><tr><th>IBias</th><th>ISd</th><th>IMSE</th><th>IBias</th><th>ISd</th><th>IMSE</th><th>IBias</th><th>ISd</th><th>IMSE</th></tr></thead><tbody><tr><td rowspan="5">Gaussian</td><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>0,0092</td><td>0,0048</td><td>0,0603</td><td>0,0168</td><td>0,0024</td><td>0,0417</td><td>0,0189</td><td><b>0,0012</b></td><td>0,0293</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0388</td><td>0,0048</td><td>0,0605</td><td>0,0334</td><td>0,0024</td><td>0,0419</td><td>0,0281</td><td><b>0,0012</b></td><td>0,0294</td></tr><tr><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0685</td><td>0,0048</td><td>0,0613</td><td>0,05</td><td>0,0024</td><td>0,0422</td><td>0,0374</td><td><b>0,0012</b></td><td>0,0295</td></tr><tr><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0224</td><td>0,0057</td><td>0,0269</td><td>-0,0077</td><td>0,003</td><td>0,0158</td><td>-0,0009</td><td>0,0016</td><td>0,0096</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0218</td><td>0,0057</td><td>0,0266</td><td>0,0179</td><td>0,003</td><td>0,0158</td><td>0,0138</td><td>0,0016</td><td>0,0096</td></tr><tr><td></td><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0661</td><td>0,0057</td><td>0,0277</td><td>0,0436</td><td>0,003</td><td>0,0162</td><td>0,0287</td><td>0,0016</td><td>0,0097</td></tr><tr><td rowspan="5">Epanechnikov</td><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,061</td><td>0,0081</td><td>0,0181</td><td>-0,0317</td><td>0,0041</td><td>0,0097</td><td>-0,0152</td><td>0,0022</td><td>0,0056</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,008</td><td>0,008</td><td>0,0166</td><td>0,0062</td><td>0,0041</td><td>0,0093</td><td>0,0056</td><td>0,0022</td><td>0,0055</td></tr><tr><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0771</td><td>0,0081</td><td>0,0186</td><td>0,0443</td><td>0,0041</td><td>0,0099</td><td>0,0266</td><td>0,0022</td><td>0,0057</td></tr><tr><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0969</td><td>0,0115</td><td>0,0224</td><td>-0,0548</td><td>0,006</td><td>0,0107</td><td>-0,0304</td><td>0,0032</td><td>0,0056</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0038</td><td>0,0113</td><td>0,0186</td><td>0,0025</td><td>0,0059</td><td>0,0095</td><td><b>0,0023</b></td><td>0,0032</td><td>0,0052</td></tr><tr><td></td><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,1046</td><td>0,0113</td><td>0,0228</td><td>0,0599</td><td>0,0059</td><td>0,0109</td><td>0,0351</td><td>0,0032</td><td>0,0056</td></tr><tr><td rowspan="5">Tricube</td><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,0767</td><td>0,0094</td><td>0,0192</td><td>-0,0408</td><td>0,0048</td><td>0,0096</td><td>-0,0207</td><td>0,0025</td><td>0,0053</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0057</td><td>0,0093</td><td>0,0168</td><td>0,0043</td><td>0,0048</td><td>0,0089</td><td>0,0041</td><td>0,0025</td><td><b>0,0051</b></td></tr><tr><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,0881</td><td>0,0094</td><td>0,0196</td><td>0,0495</td><td>0,0048</td><td>0,0098</td><td>0,029</td><td>0,0025</td><td>0,0053</td></tr><tr><td>τ̃<sup>(1)</sup><sub>1,2|X=.</sub></td><td>-0,1174</td><td>0,0139</td><td>0,0272</td><td>-0,0664</td><td>0,0071</td><td>0,0124</td><td>-0,0371</td><td>0,0038</td><td>0,0062</td></tr><tr><td>τ̃<sup>(2)</sup><sub>1,2|X=.</sub></td><td>0,0027</td><td>0,0135</td><td>0,0216</td><td>0,0017</td><td>0,007</td><td>0,0107</td><td>0,0017</td><td>0,0038</td><td>0,0057</td></tr><tr><td></td><td>τ̃<sup>(3)</sup><sub>1,2|X=.</sub></td><td>0,1228</td><td>0,0135</td><td>0,0275</td><td>0,0698</td><td>0,007</td><td>0,0125</td><td>0,0406</td><td>0,0038</td><td>0,0063</td></tr></tbody></table>
<figure><img src="image_2.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(d) Beta kernel.</figcaption></figure>

<caption>Figure 1: The local bias for $n = 2000$ in Setting 1, for the estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ continuous lines, blue, green and red, and $\hat{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ dashed lines, blue, green and red.</caption>
<figure><img src="image_6.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(d) Beta kernel.</figcaption></figure>

<caption>Figure 2: The local variance for $n = 2000$ in Setting 1, for the estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ continuous lines, blue, green and red, and $\hat{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ dashed lines, blue, green and red.</caption>
<figure><img src="image_10.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(d) Beta kernel.</figcaption></figure>

<caption>Figure 3: The local MSE for $n = 2000$ in Setting 1, for the estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ continuous lines, blue, green and red, and $\hat{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ dashed lines, blue, green and red.</caption>
<figure><img src="image_14.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>(d) Beta kernel.</figcaption></figure>

<caption>Figure 4: Fitting of $\tilde{\tau}_{1,2|X=\cdot}^{(2)}$ curves based on 1000 simulations for $n = 2000$ in Setting 1, for the estimator $\tilde{\tau}_{1,2|X=t}^{(2)}$.</caption>
<figure><img src="image_18.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<caption>Figure 5: The local bias for $n = 2000$ in Setting 2, for the estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ continuous lines, blue, green and red, and $\hat{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ dashed lines, blue, green and red.</caption>
<figure><img src="image_21.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<caption>Figure 6: The local variance for $n = 2000$ in Setting 2, for the estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ continuous lines, blue, green and red, and $\hat{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ dashed lines, blue, green and red.</caption>
<figure><img src="image_24.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<caption>Figure 7: The local MSE for $n = 2000$ in Setting 2, for the estimators $\tilde{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ continuous lines, blue, green and red, and $\hat{\tau}_{1,2|X=t}^{(k)}$ for $k = 1, 2, 3$ dashed lines, blue, green and red.</caption>
<figure><img src="image_27.png" /><figcaption>(a) Normal kernel.</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>(b) Tricube kernel.</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>(c) Epanechnikov kernel.</figcaption></figure>

<caption>Figure 8: Fitting of $\tilde{\tau}_{1,2|X=\cdot}^{(2)}$ curves based on 1000 simulations for $n = 2000$ in Setting 2, for the estimator $\tilde{\tau}_{1,2|X=t}^{(2)}$.</caption>
So, we get

$$ \alpha(f_{\mathbf{X};n}(\mathbf{t};h_{1,n})) = \alpha(f(\mathbf{t}))(1 + \delta(\mathbf{t})). \quad (9.1) $$

We assume that $\mathbf{p}$ is a Lipschitz function and $\mathbf{p} \ge 1$. So, we have

$$ |\delta(\mathbf{t})| \le B c^{-2} |f_{\mathbf{X};n}(\mathbf{t};h_{1,n}) - f(\mathbf{t})| \quad (9.2) $$

for a constant $B$ that depends only on the function $\mathbf{p}$. We obtained the following expansion of $\alpha^d(f_{\mathbf{X};n})$ implied by (9.1)

$$ \alpha^p(f_{\mathbf{X};n}(\mathbf{t};h_{1,n})) = \alpha^p(f(\mathbf{t}))(1 + d\delta(\mathbf{t})) + \delta_1(\mathbf{t}). \quad (9.3) $$

We also have, for further use,

$$ \delta(\mathbf{t}) = \frac{\alpha'(f(\mathbf{t})) [f_{\mathbf{X};n}(\mathbf{t};h_{1,n}) - f(\mathbf{t})]}{f(\mathbf{t})} + \frac{\alpha''(\eta) [f_{\mathbf{X};n}(\mathbf{t};h_{1,n}) - f(\mathbf{t})]^2}{2f(\mathbf{t})}, \quad (9.4) $$

where $\eta = \eta(\mathbf{t}) \ge 0$ is between $f_{\mathbf{X};n}(\mathbf{t};h_{1,n})$ and $f(\mathbf{t})$. We then have the following Taylor series expansion, which plays a crucial role in our proofs

$$ \begin{aligned} K \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_{1,n})) \right) &= K \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i)) + \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \right) \\ &= K \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i)) \right) \\ &\quad + \sum_{j=1}^{p} K'_{j} \left( \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i)) \right) \frac{(\mathbf{t} - \mathbf{X}_i)_j}{h_{2,n}} \alpha(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) + \delta_2(\mathbf{t}; \mathbf{X}_i), \end{aligned} \quad (9.5) $$

where

$$ \delta_2(\mathbf{t}, \mathbf{X}_i) = \sum_{j, \ell' = 1}^{p} K''_{j, \ell'}(\xi) \frac{(\mathbf{t} - \mathbf{X}_i)_j (\mathbf{t} - \mathbf{X}_i)_{\ell'}}{2h_{2,n}^2} \alpha^2(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i), $$

$\xi$ being a (random) number between $\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))$ and $\frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i)) + \frac{\mathbf{t} - \mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i)) \delta(\mathbf{X}_i)$. Set

$$ L_1(\mathbf{t}) = \sum_{i=1}^{p} \mathbf{t}_i K'_i(\mathbf{t}) \quad \text{and} \quad L(\mathbf{t}) = pK(\mathbf{t}) + L_1(\mathbf{t}), \quad \mathbf{t} \in \mathbb{R}^p. $$

In order to obtain desired results, we need the following Lemma. First of all, we recall that $\mathcal{N}(\mathcal{H}, d, \varepsilon)$, the $\varepsilon$ covering number of the metric or pseudo-metric space $(\mathcal{H}, d)$, is defined as the smallest number of (open) $d$-balls of radius not exceeding $\varepsilon$ needed to cover $\mathcal{H}$. Also, a collection of measurable functions $\mathcal{H}$ on a measurable space $(S, \mathcal{S})$ is of VC type relative to an envelope $\kappa$, if there exist finite constants $A_1, b_1$ such that, for all probability measures $\mathbb{Q}$ on $(S, \mathcal{S})$,

$$ \mathcal{N}(\mathcal{H}, L_2(\mathbb{Q}), \varepsilon) \le \left( \frac{A_1 \|\mathbf{\kappa}\|_{L_2(\mathbb{Q})}}{\varepsilon} \right)^{b_1}, \quad 0 < \varepsilon \le 2 \sup_{f,g \in \mathcal{H}} \|f - g\|_{L_2(\mathbb{Q})}. \quad (9.6) $$

**Lemma 9.0.1.** Let $\tilde{\mathcal{G}}$ be a uniformly bounded VC type class of measurable functions on $\mathbb{R}^d$ for a constant envelope $G$ and admitting constants $A_1, b_1$ in equation (9.6). Let $\mathcal{H}$ be the class of functions

$$ \mathcal{H} = \left\{ K \left( \frac{\mathbf{t} - \cdot}{h} \alpha(\cdot) \right) g(\cdot) : \mathbf{t} \in \mathbf{I}, h > 0, g \in \tilde{\mathcal{G}} \right\} $$

Then, there exists a universal constant $R$ such that for every Borel probability measure $\mathbb{Q}$ on $\mathbb{R}^d$,

$$ \mathcal{N}(\mathcal{H}, L_2(\mathbb{Q}), \varepsilon) \le \left( \frac{(R \vee A_1) G \kappa}{\varepsilon} \right)^{b'} $$

That is, $\mathcal{H}$ is a bounded class of functions of VC type with envelope $\kappa := G\kappa$ and admitting characteristic constants $A' = R \vee A_1$ and $b'$.

This lemma is important for our proofs because it allows direct use of a version of Talagrand (1994) inequality, i.e., in the form given in Einmahl and Mason (2000).
## 9.1 Proof of Theorem 3.1.1

For the first point, we will use the Taylor expansion (9.5), which with the decompositions (9.1) and (9.3), give us

$$ 
\begin{aligned}
U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) = & \frac{1}{nh_2^p} \sum_{i=1}^n \varphi(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \\
& + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi(\mathbf{Y}_i) L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \\
& + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi(\mathbf{Y}_i) \left[ K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \\
& \left. \qquad + pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] \\
& + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi(\mathbf{Y}_i) \left[ L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta(\mathbf{X}_i) \delta_1(\mathbf{X}_i) + p\alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \\
& + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi(\mathbf{Y}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \delta_1(\mathbf{X}_i).
\end{aligned}
$$

Therefore, we obtain

$$ \left| U_n^{(1)}(\varphi, \mathbf{t}; h_{1,n}, h_2) - \mathbb{E} U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) \right| \le \frac{M}{nh_2^p} \left| \sum_{i=1}^n K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) - \mathbb{E}\left[\sum_{i=1}^n K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i))\right] \right| \quad (9.7) $$

$$ + \frac{M}{nh_2^p} \left| \sum_{i=1}^n L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) - \mathbb{E}\left[\sum_{i=1}^n L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i)\right] \right| \quad (9.8) $$

$$ + \frac{M}{nh_2^p} \left| \sum_{i=1}^n \left[ K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \right. \\
+ pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] \\
- \mathbb{E}\left[ \left. \sum_{i=1}^n \left( K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \right. \\
\left. \left. + pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right) \right] \right| \quad (9.9) $$

$$ + \frac{M}{nh_2^p} \left| \sum_{i=1}^n \left[ L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta(\mathbf{X}_i) \delta_1(\mathbf{X}_i) + p\alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \right. \\
- \mathbb{E}\left[ \left. \sum_{i=1}^n \left( L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta(\mathbf{X}_i) \delta_1(\mathbf{X}_i) + d\alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \right) \right] \right| \quad (9.10) $$

$$ + \frac{M}{nh_2^p} \left| \sum_{i=1}^n \delta_2(\mathbf{t}, \mathbf{X}_i) \delta_1(\mathbf{X}_i) - \mathbb{E}\left[\sum_{i=1}^n \delta_2(\mathbf{t}, \mathbf{X}_i) \delta_1(\mathbf{X}_i)\right] \right| \quad (9.11) $$

Let $\alpha_n$ be the empirical process based on the sample $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$, i.e., if $\mathbf{g}: \mathbb{R}^d \times \mathbb{R}^q \to \mathbb{R}$, we have

$$ \alpha_n(\mathbf{g}) = \sum_{i=1}^{n} \left( \mathbf{g}(\mathbf{X}_i, \mathbf{Y}_i) - \mathbb{E} \mathbf{g}(\mathbf{X}, \mathbf{Y}) \right) / \sqrt{n}. $$

We start with the first term (9.7):

$$ T_1 := \frac{M}{nh_2^p} \left| \sum_{i=1}^n K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) - \mathbb{E}\left[\sum_{i=1}^n K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i))\right] \right| $$

Denote

$$ \eta_{n,h_2}(\mathbf{t}) := K\left(\frac{\mathbf{t}-\mathbf{X}}{h_2} \alpha(f(\mathbf{X}))\right) \alpha^p(f(\mathbf{X})). $$
We consider the classes of functions

$$ \mathcal{H}_1 = \left\{ K \left( \frac{\mathbf{t} - \mathbf{\cdot}}{h_2} \alpha(f(\cdot)) \right) \alpha^p(f(\cdot)) : \mathbf{t} \in \mathbf{I}, h_2 > 0 \right\}. $$

These classes are all VC and admit the same constants $A'$ and $b'$ as $\mathcal{H}$. We first note that, by the usual change of variable $\mathbf{u} = (\mathbf{t} - \mathbf{s})/h_2$,

$$ \begin{aligned} \mathbb{E} [\eta_{n,h_2}^2(\mathbf{t})] &= \mathbb{E} \left( K^2 \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^{2p} (f(\mathbf{X})) \right) \\ &= h_2^p \int_{\mathbb{R}^p} h_2^{-p} K^2 \left( \frac{\mathbf{t} - \mathbf{s}}{h_2} \alpha(f(\mathbf{s})) \right) \alpha^{2p} (f(\mathbf{s})) f(\mathbf{s}) d\mathbf{s} \\ &\le h_2^p \|\mathbf{K}\|_{\infty}^2 \|\alpha\|_{\infty}^{2p} \|f\|_{\infty}, \end{aligned} $$

Set for $j, r \ge 0$ and $c > 0$, $n_r = 2^r$, $h_{2,j,r} = (2^j c \log n_r)/n_r$ and

$$ \mathcal{H}_{1,j,r} = \left\{ K \left( \frac{\mathbf{t} - \mathbf{\cdot}}{h_2} \alpha(f(\cdot)) \right) \alpha^p(f(\cdot)) : h_{2,j,r} \le h_2 \le h_{2,j+1,r}, \mathbf{t} \in \mathbf{I} \right\}. $$

Clearly, for $h_{2,j,r} \le h_2 < h_{2,j+1,r}$ and with $\kappa$ as in (K.ii), we have

$$ \begin{aligned} \mathbb{E} \left[ K^2 \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^{2p} (f(\mathbf{X})) \right] &\le \kappa^2 \|\alpha\|_{\infty}^{2p} \wedge 2^p h_{2,j,r}^p \|K\|_{\infty}^2 \|f\|_{\infty} \|\alpha\|_{\infty}^{2p} \\ &=: \kappa^2 \|\alpha\|_{\infty}^{2p} \wedge D_0 h_{2,j,r}^p \\ &=: \sigma_{1,j,r}^2. \end{aligned} $$

Now, making use of the corollary 10.0.3, we have to bound

$$ \mathbb{E} \left\| \sum_{i=1}^{n_r} \varepsilon_i g(\mathbf{X}_i) \right\|_{\mathcal{H}_{1,j,r}}. $$

We notice that each $\mathcal{H}_{1,j,r}$ satisfies (i) with $G = \beta = \kappa \|\alpha\|_{\infty}^p$. Additionally, since $\mathcal{H}_{1,j,r} \subset \mathcal{H}$ we see that each $\mathcal{H}_{1,j,r}$ also fulfills (ii). Notice that

$$ C_1 \beta / \sigma_0 \le (\beta^2 / \sigma_0^2) \vee C_1^2 $$

and the function $h \to h \log (h^{-1} \vee C_1^2)$ is increasing for $h \ge 0$ (recall that $C_1 \ge e$). By applying Corollary 10.0.3 with $U = \beta = \kappa \|\alpha\|_{\infty}^p$ and utilizing the bound $\sigma_0 \le \sigma_{1,j,r}$, we obtain after a small calculation that for suitable positive constants $D_2$ and $D_3$ and for $r \ge 1$ and $j \ge 0$,

$$ \begin{aligned} \mathbb{E} \left\| \sum_{i=1}^{n_r} \varepsilon_i g(\mathbf{X}_i) \right\|_{\mathcal{H}_{1,j,r}} &\le D_2 \sqrt{n_r h_{2,j,r}^p \log \left( \frac{1}{D_2 h_{2,j,r}^p} \vee C_1^2 \right)} \\ &\le D_3 a_{1,j,r}, \end{aligned} $$

where

$$ a_{1,j,r} = \sqrt{n_r h_{2,j,r}^p \log \left( \left( \frac{1}{D_2 h_{2,j,r}^p} \right) \vee \log \log n_r \right)}. $$

Now, applying the Inequality of Talagrand in Theorem 10.0.4 with

$$ \mathbf{M} = \kappa \|\alpha\|_{\infty}^{p} $$

and

$$ \sigma_g^2 = \sigma_{\mathcal{H}_{1,j,r}}^2 \le D_0 h_{2,j,r}, $$

we get for any $\mathbf{u} > 0$,

$$ \begin{aligned} \mathbb{P} \left\{ \max_{n_{r-1} \le n \le n_r} \|\sqrt{n} \alpha_n\|_{\mathcal{H}_{1,j,r}} \ge A_1 (D_3 a_{1,j,r} + \mathbf{u}) \right\} \\ \le 2 \left[ \exp \left( - \frac{A_2 \mathbf{u}^2}{D_0 n_r h_{2,j,r}^p} \right) + \exp \left( - \frac{A_2 \mathbf{u}}{\kappa \|\alpha\|_{\infty}^p} \right) \right]. \end{aligned} $$
Setting for any $\rho > 1, j \ge 0$ and $r \ge 1$,

$$p_{1,j,r}(\rho) = \mathbb{P} \left\{ \max_{n_{r-1} \le n \le n_r} \left\| \sqrt{n}\alpha_n \right\|_{\mathcal{H}_{1,j,r}} \ge A_1 (D_3 + \rho) a_{1,j,r} \right\},$$

and using the fact that $a_{1,j,r}^2/n_r h_{2,j,r} \ge \log \log n_r$, we can infer that for large $k$,

$$p_{1,j,r}(\rho) \le 2 \left[ \exp\left(-\frac{\rho^2 A_2}{D_0} \log \log n_r\right) + \exp\left(-\frac{A_2 \rho}{\kappa \|\alpha\|_{\infty}^p} \sqrt{n_r h_{2,j,r}^p \log \log n_r}\right) \right].$$

Recalling that $h_{2,j,r} \ge c \log n_r/n_r$, we readily obtain that for large $r$ and any $j \ge 0$,

$$p_{1,j,r}(\rho) \le 4 (\log n_r)^{-\tilde{\rho}}, \quad (9.12)$$

where $\tilde{\rho} = \frac{A_2}{D_0} \rho^2$. Set

$$l_k = \max \{j : h_{2,j,r} \le 2\}.$$

It is easy to see that for large $r$,

$$l_r \le 2 \log n_r. \quad (9.13)$$

Keep in mind (9.12) and (9.13), we have for large $r$ and $\rho \ge 1$,

$$P_{1,r}(\rho) := \sum_{j=0}^{l_r-1} p_{1,j,r}(\rho) \le 8 (\log n_r)^{1-\tilde{\rho}},$$

this, in turn, implies that

$$\sum_{r=1}^{\infty} P_r(\rho) < \infty. \quad (9.14)$$

By definition of $l_r$ for large $r$, notice that

$$2h_{2,l_r,r} = h_{2,l_r+1,k} \ge 2.$$

Therefore, we then have for $n_{r-1} \le n \le n_r$,

$$\left[ \frac{c \log n}{n}, 1 \right] \subset \left[ \frac{c \log n_r}{n_r}, h_{2,l_r,r} \right].$$

Thus for all large enough $r$ and $n_{r-1} \le n \le n_r$,

$$A_{1,r}(\rho) := \left\{ \begin{array}{ll} \displaystyle \max_{n_{r-1} \le n \le n_r} \sup_{c \log n/n \le h_{2,1,k} \le 1} \frac{\sqrt{nh_2^p} \|\eta_{n,h_2} - \mathbb{E}\eta_{n,h_2}\|_{\mathbf{I}}}{\sqrt{\log(1/h_2) \vee \log \log n}} & > 2A_1(D_3 + \rho) \\ \\ \subset \bigcup_{j=0}^{l_r-1} \left\{ \max_{n_{r-1} \le n \le n_r} \|\sqrt{n}\alpha_n\|_{\mathcal{H}_{1,j,r}} \ge A_1(D_3 + \rho) a_{1,j,r} \right\}, \end{array} \right.$$

and we see that

$$\mathbb{P}(A_{1,r}(\rho)) \le P_{1,r}(\rho).$$

So, the proof of the first term is achieved, making use (9.14), by applying the Borel-Cantelli lemma. Continue, now, with the second term (9.8):

$$T_2 := \frac{\mathbb{M}}{nh_2^p} \left| \sum_{i=1}^n L\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) - \mathbb{E}\left[\sum_{i=1}^n L\left(\frac{\mathbf{t} - \mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i)\right] \right|.$$

Denote

$$\bar{\eta}_{n,h_2}(\mathbf{t}) := L\left(\frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X}))\right) \alpha^p(f(\mathbf{X})) \delta(\mathbf{X}).$$

Consider the classes of functions

$$\mathcal{H}_2 = \left\{ L\left(\frac{\mathbf{t} - \cdot}{h_2} \alpha(f(\cdot)) \right) \alpha^p(f(\cdot)) \delta(\cdot) : \mathbf{t} \in \mathbf{I}, h_2 > 0 \right\}.$$
We see that the kernel $L$ satisfies the hypotheses of the kernel $K$. So, these classes conform to Lemma 9.0.1 with $\tilde{\mathcal{G}}$ the class consisting of the single function $\alpha^p f(\cdot)\delta(\cdot)$, which is uniformly bounded by $2\|\max \alpha\|_{\infty}^{p+1}$. We conclude by that Lemma that they are VC each with envelope $2\bar{M}(c, \mathfrak{p}, C)\|\max \alpha\|_{\infty}^{p+1}$, for some constant $\bar{M}$ depending only on $c, \mathfrak{p}$ and $C$, for some other constant depending on the stated object, and all with the same characteristics constants $A'$ and $b'$ as $\mathcal{K}$. Furthermore, the continuity hypotheses make these classes measurable. Consequently, we observe that, by using the change of variable $\mathbf{u} = (\mathbf{t} - \mathbf{x})/h_2$,

$$
\begin{align*}
\mathbb{E}[\bar{\eta}_{n,h_2}^2(\mathbf{t})] &= \mathbb{E}\left[L^2\left(\frac{\mathbf{t}-\mathbf{X}}{h_2}\alpha(f(\mathbf{X}))\right)\alpha^{2p}(f(\mathbf{X}))\delta^2(\mathbf{X})\right] \\
&= h_2^p \int_{\mathbb{R}^p} h_2^{-p} L^2\left(\frac{\mathbf{t}-\mathbf{s}}{h_2}\alpha(f(\mathbf{s}))\right) \alpha^{2p}(f(\mathbf{s}))\delta^2(\mathbf{s})f(\mathbf{s})d\mathbf{s} \\
&\le 2h_2^p \|\max \alpha\|_{\infty}^{2p+2} \|f\|_{\infty} \int_{\mathbb{R}^p} L^2(\mathbf{u}\alpha(f(\mathbf{t}-\mathbf{u}h_2)))d\mathbf{u}. \\
&\le 2h_2^p \mathcal{M}(c, \mathfrak{p}, C, K) \|\max \alpha\|_{\infty}^{2p+2} \|f\|_{\infty} =: h_2^p \bar{D}_0,
\end{align*}
$$

where $\mathcal{M}(c, \mathfrak{p}, C, K)$ is a finite constant depending only on $c, \mathfrak{p}, C$ and $K$ and $c$ and $\mathfrak{p}$ are applied in the definition of $\alpha(\cdot)$ in (2.3). As before, for $j, r \ge 0$ and $h_{2,j,r} = (2^j c \log n_r)/n_r$, we set

$$
\mathcal{H}_{2,j,r} = \left\{ L \left( \frac{\mathbf{t} - \mathbf{\alpha}(f(\cdot))}{h_2} \right) \alpha^p(f(\cdot)) \delta(\cdot) : h_{2,j,r} \le h_2 \le h_{2,j+1,r}, \mathbf{t} \in \mathbf{I} \right\}.
$$

Clearly, for $h_{2,j,r} \le h_2 \le h_{2,j+1,r}$

$$
\mathbb{E}[\bar{\eta}_{n,h_2}^2(\mathbf{t})] \leq h_2^p \bar{D}_0 \leq 2^p h_{2,j,r}^p \bar{D}_0.
$$

So, as we saw in the first term (9.7), we shall use Corollary 10.0.3 to bound

$$
\mathbb{E} \left\| \sum_{i=1}^{n_r} \varepsilon_i g(\mathbf{X}_i) \right\|_{\mathcal{H}_{2,j,r}} .
$$

We note that each $\mathcal{H}_{2,j,r}$ satisfies (i) with $G = \beta = 2\bar{M}(c, \mathfrak{p}, C)\|\max \alpha\|_{\infty}^{p+1}$. Further, since $\mathcal{H}_{2,j,r} \subset \mathcal{H}$ we see that each $\mathcal{H}_{2,j,r}$ also
fulfills (ii). By applying Corollary 10.0.3 with

$$
U = \beta = 2\bar{M}(c, \mathfrak{p}, C) \|\max \alpha\|_{\infty}^{p+1},
$$

we obtain after a small calculation that, for $j \ge 0$,

$$
\begin{align*}
\mathbb{E} \left\| \sum_{i=1}^{n_r} \varepsilon_i g(\mathbf{X}_i) \right\|_{\mathcal{H}_{2,j,r}} &\le \bar{D}_2 \sqrt{n_r h_{2,j,r}^p \log \left( \frac{1}{\bar{D}_2 h_{2,j,r}^p} \vee C_1^2 \right)} \\
&\le \bar{D}_3 a_{2,j,r},
\end{align*}
$$

where $\bar{D}_2$ and $\bar{D}_3$ are positive constants and

$$
a_{2,j,r} = \sqrt{n_k h_{2,j,r}^p \log \left( \left( \frac{1}{\bar{D}_2 h_{2,j,r}^p} \right) \vee \log \log n_r \right)}.
$$

Now, applying the Inequality of Talagrand in Theorem 10.0.4 with

$$
\mathbf{M} = 2\bar{M}(c, \mathfrak{p}, C) \|\max \alpha\|_{\infty}^{p+1},
$$

and

$$
\sigma_g^2 = \sigma_{\mathcal{H}_{2,j,r}}^2 \leq \bar{D}_0 h_{2,j,r}^p,
$$

we have, for any $\mathbf{u} > 0, \rho > 1, j \ge 0$ and $r \ge 1$,

$$
p_{2,j,r}(\rho) := \mathbb{P} \left\{ \max_{n_{r-1} \le n \le n_r} \| \sqrt{n} \alpha_n \|_{\mathcal{H}_{2,j,r}} \ge A_1 (\bar{D}_3 a_{2,j,r} + \mathbf{u}) \right\} \\
\le 2 \left[ \exp \left( - \frac{A_2 \mathbf{u}^2}{\bar{D}_0 n_r h_{2,j,r}^p} \right) + \exp \left( - \frac{A_2 \mathbf{u}}{2 \bar{M}(c, \mathfrak{p}, C) \| \max \alpha \|_{\infty}^{p+1}} \right) \right]
$$
$$ \leq 2 \left[ \exp\left(-\frac{\rho^2 A_2}{\bar{D}_0} \log \log n_r\right) + \exp\left(-\frac{A_2 \rho}{2\bar{M}(c, \mathbf{p}, C) \|\max \alpha\|_{\infty}^{p+1}} \sqrt{n_r h_{2,j,r}^p \log \log n_r}\right) \right], $$

where we use the fact that $a_{2,j,r}^2/n_r h_{2,j,r} \ge \log \log n_r$ in the last inequality. So, we easily get that for large $r$ and any $j \ge 0$,

$$ p_{2,j,r}(\rho) \le 4 (\log n_r)^{-\tilde{\rho}_1}, \quad (9.15) $$

where $\tilde{\rho}_1 = \frac{A_2}{\bar{D}_0} \rho^2$. In view of (9.13) and (9.15), we have for large $k$ and $\rho \ge 1$,

$$ P_{2,r}(\rho) := \sum_{j=0}^{l_r-1} p_{2,j,r}(\rho) \le 8 (\log n_r)^{1-\tilde{\rho}_1}. $$

By definition of $l_r$ for all large enough $r$ and $n_{r-1} \le n \le n_r$, we obtain

$$ A_{2,r}(\rho) := \left\{ \begin{array}{ll} \max_{n_{r-1} \le n \le n_r} \sup_{c \log n / n \le h_2 \le 1} \frac{\sqrt{nh_2^p} \|\bar{\eta}_{n,h_2} - \mathbb{E}\bar{\eta}_{n,h_2}\|_{\mathbf{I}}}{\sqrt{\log(1/h_2) \vee \log \log n}} & > 2A_1(\bar{D}_3 + \rho) \\ \\ \subset \bigcup_{j=0}^{l_r-1} \left\{ \max_{n_{r-1} \le n \le n_r} \|\sqrt{n}\alpha_n\|_{\mathcal{H}_{1,j,r}'} \ge A_1(\bar{D}_3 + \rho) a_{2,j,r} \right\}. \end{array} \right. $$

Therefore, by applying the Borel-Cantelli lemma, the second term is proved. We consider, now, the third term (9.9):

$$ \begin{align} T_3 &:= \frac{M}{nh_2^{\tilde{p}}} \left| \sum_{i=1}^{n} \left( K \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) + pL_1 \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right. \right. \nonumber \\ &\quad \left. - \mathbb{E} \left[ \sum_{i=1}^{n} \left( K \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) + pL_1 \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right) \right] \right| \nonumber \\ &\le \frac{M}{nh_2^{\tilde{p}}} \left| \sum_{i=1}^{n} \left( K \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \delta_1(\mathbf{X}_i) \right) - \mathbb{E} \left[ \sum_{i=1}^{n} \left( K \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \delta_1(\mathbf{X}_i) \right) \right| \nonumber \\ &\quad + \frac{M}{nh_2^{\tilde{p}}} \left| \sum_{i=1}^{n} \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) - \mathbb{E} \left[ \sum_{i=1}^{n} \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \right| \nonumber \\ &\quad + \frac{pM}{nh_2^{\tilde{p}}} \left| \sum_{i=1}^{n} L_1 \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) - \mathbb{E} \left[ \sum_{i=1}^{n} L_1 \left( \frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i)) \right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] \right| \tag{9.16} \end{align} $$

To prove (9.9), we begin with the second term in the inequality (9.16). Consider the classes of functions

$$ \mathcal{H}_3 := \{ \alpha^p(f(\cdot)) \delta_2(\mathbf{t}, \cdot) : \mathbf{t} \in \mathbf{I}, h_2 > 0 \}, $$

where

$$ \delta_2(\mathbf{t}, \mathbf{X}) := \sum_{j,l'=1}^{p} K_{j,l'}''(\xi) \frac{(\mathbf{t}-\mathbf{X}_i)_j (\mathbf{t}-\mathbf{X}_i)_{l'}}{2h_2^2} \alpha^2(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i). $$

According to Lemma 9.0.1, these classes are all VC and admit the same constants $A'$ and $b'$ as $\mathcal{H}$ with envelope $\tilde{M}(c, \mathbf{p}, C') \|\alpha\|_{\infty}^p$. Since the continuity hypotheses make these classes measurable, this will allow us to apply Talagrand's inequality as before. If we set

$$ Q_i(\mathbf{t}) = \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i), $$

it then follows, for all $\mathbf{t}$,

$$ \begin{align} \mathbb{E}[Q_i^2(\mathbf{t})] &= \mathbb{E}[\alpha^{2p}(f(\mathbf{X}_i))\delta^2(\mathbf{t}, \mathbf{X}_i)] \nonumber \\ &\le h_2^p p^4 \|K''\|_{\infty} (2/c)^4 \|\max \alpha\|_{\infty}^4 \|f\|_{\infty} I^2 (\|\mathbf{t}-\mathbf{x}\| \le 2c^{-1}h_2) \nonumber \\ &=: h_2^{\tilde{p}} \tilde{D}_1. \tag{9.17} \end{align} $$

We see that each $\mathcal{H}_3$ satisfies (i) in Corollary 10.0.3 with

$$ G = \beta = \tilde{M}(c, \mathbf{p}, C') \|\alpha\|_{\infty}^{p}. $$

Further, since $\mathcal{H}_3 \subset \mathcal{H}$, we can see also that each $\mathcal{H}_3$ also fulfills (ii). So, we can likewise follow the proof of terms (9.7) and (9.8) by applying Corollary 10.0.3 with

$$ U = \beta = \tilde{M}(c, \mathbf{p}, C') \|\alpha\|_{\infty}^{p}, $$

so we get a similar result for this term. The other terms in (9.16) are treated the same way as in the preceding proof. The proofs of (9.10) and (9.11) are done similarly as (9.9). Therefore the theorem is proved. $\square$
## 9.2 Proof of Theorem 3.1.2

In this Theorem, we suppose that the class of functions $\mathcal{F}$ is not necessarily bounded but satisfies a general condition. we proceed as follows, for each function $\varphi \in \mathcal{F}$

$$ 
\begin{aligned}
\varphi(\mathbf{y}) &= \varphi(\mathbf{y})\mathbb{1}_{\{F(\mathbf{y}) \le (n_r/r)^{1/v}\}} + \varphi(\mathbf{y})\mathbb{1}_{\{F(\mathbf{y}) > (n_r/r)^{1/v}\}} \\
&= \varphi_r^{(T)}(\mathbf{y}) + \varphi_r^{(R)}(\mathbf{y}).
\end{aligned}
$$

So, we write

$$ 
\begin{aligned}
U_n^{(1)}(\varphi, \mathbf{t}; h_1, h_2) &:= \frac{1}{nh_2^p} \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_1))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_1,n)) \\
&\quad + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi_r^{(R)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f_{\mathbf{X};n}(\mathbf{X}_i; h_1))\right) \alpha^p(f_{\mathbf{X};n}(\mathbf{X}_i; h_1)) \\
&= \vartheta_{\varphi, n, h_1, h_2, r}^{(T)}(\mathbf{t}) + \vartheta_{\varphi, n, h_1, h_2, r}^{(R)}(\mathbf{t}).
\end{aligned}
$$

The proof of the Theorem 3.1.2 in the unbounded case will be a consequence of two Lemmas. We will be first show

**Lemma 9.2.1.** Under the assumptions of Theorem 3.1.2, there exists a constant $0 < \mathcal{Q}_1 < 0$, such that with probability 1, we have

$$ \limsup_{k \to \infty} \max_{n-1 \le n \le n_r} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathcal{I}} \sup_{\varphi \in \mathcal{F}} \left| \vartheta_{\varphi, n, h_1, h_2, r}^{(T)} - \mathbb{E} \vartheta_{\varphi, n, h_1, h_2, r}^{(T)} \right| = \mathcal{Q}_1 \left( \sqrt{\frac{\log(1/h_2) \vee \log \log n}{nh_2^p}} \right) \quad (9.18) $$

### Proof of Lemma 9.2.1

As we saw before, by using the decompositions (9.1), (9.3) and (9.5), we get

$$ 
\begin{aligned}
& \left| \vartheta_{\varphi, n, h_1, h_2, r}^{(T)} - \mathbb{E} \vartheta_{\varphi, n, h_1, h_2, r}^{(T)} \right| \\
&= \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \right. \\
&\quad \left. - \mathbb{E}\left[\sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i))\right] \right| \quad (9.19)
\end{aligned}
$$

$$ + \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \right. \\
- \mathbb{E}\left[\sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i)\right] \quad (9.20) $$

$$ + \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \left[ K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \right. \\
+ pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] $$

$$ - \mathbb{E}\left[\sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \left( K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \right. \\
+ pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] \quad (9.21) $$

$$ + \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \left[ L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta(\mathbf{X}_i) \delta_1(\mathbf{X}_i) + p\alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \right. \\
- \mathbb{E}\left[\sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \left( L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta(\mathbf{X}_i) \delta_1(\mathbf{X}_i) + d\alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \right) \right] \quad (9.22) $$

$$ + \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \delta_1(\mathbf{X}_i) - \mathbb{E}\left[\sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \delta_1(\mathbf{X}_i)\right] \right| \quad (9.23) $$
Let us begin with the first term (9.19):

$$ T_1 := \frac{1}{nh_2^p} \left| \sum_{i=1}^{n} \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) - \mathbb{E}\left[\sum_{i=1}^{n} \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i))\right] \right| $$

Set, for $\mathbf{t} \in \mathbf{I}$ and $\varphi \in \mathcal{F}$,

$$ \eta_{\varphi,n,h_2,r}^{(T)}(\mathbf{t}) = \sum_{i=1}^{n} \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)), $$

$$ v_{\varphi,h_2,t}(\mathbf{u},\mathbf{v}) = \varphi(\mathbf{v}) K\left(\frac{\mathbf{t}-\mathbf{u}}{h_2} \alpha(f(\mathbf{u}))\right) \alpha^p(f(\mathbf{u})), $$

$$ v_{\varphi,h_2,r,t}^{(T)}(\mathbf{u},\mathbf{v}) = \varphi_r^{(T)}(\mathbf{v}) K\left(\frac{\mathbf{t}-\mathbf{u}}{h_2} \alpha(f(\mathbf{u}))\right) \alpha^p(f(\mathbf{u})). $$

We can see that

$$ \eta_{\varphi,n,h_2,r}^{(T)} - \mathbb{E}\eta_{\varphi,n,h_2,r}^{(T)} = \sqrt{n}\alpha_n(v_{\varphi,h_2,r,t}^{(T)}), $$

where $\alpha_n$ is the empirical process based on $(\mathbf{X}_1, \mathbf{Y}_1), \dots, (\mathbf{X}_n, \mathbf{Y}_n)$. Denote

$$ \mathcal{G}_r := \{ v_{\varphi,h_2,r,t}^{(T)} : \varphi \in \mathcal{F} \text{ and } \mathbf{t} \in \mathbf{I} \}. $$

Note that for each $v_{\varphi,h_2,r,t}^{(T)} \in \mathcal{G}_r$,

$$ \|v_{\varphi,h_2,t}^{(T)}\|_{\infty} \le \kappa \|\alpha\|_{\infty}^{p} (n_r/r)^{1/v} =: B_0(n_r/r)^{1/v}. \qquad (9.24) $$

We get that, making use of the condition (F.v),

$$ \begin{aligned} \mathbb{E} \left[ (v_{\varphi,h_2,t}^{(T)})^2 (\mathbf{X}, \mathbf{Y}) \right] &\le \mathbb{E} \left[ v_{\varphi,h_2,t}^2 (\mathbf{X}, \mathbf{Y}) \right] \le \mathbb{E} \left[ \varphi^2(\mathbf{Y}) K^2 \left( \frac{\mathbf{t}-\mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^{2p} (f(\mathbf{X})) \right] \\ &\le h_2^p \theta_v^{2/v} \|\alpha\|_{\infty}^{2p} \|f\|_{\infty} \|K\|_{\infty}^2 =: h_2^p B_1. \end{aligned} $$

Therefore, we infer that

$$ \sup_{\mathbf{v} \in \mathcal{G}_r} \mathbb{E} v^2(\mathbf{X}, \mathbf{Y}) \le h_2^p B_1. \qquad (9.25) $$

Remembering that for $j, r \ge 0$,

$$ h_{2,j,r} = 2^j a_r = 2^j c \left( \frac{\log n_r}{n_r} \right)^\gamma, \qquad (9.26) $$

and set

$$ \mathcal{G}_{j,r} := \{ v_{\varphi,h_2,r,t}^{(T)} : \varphi \in \mathcal{F} \text{ and } h_{2,j,r} \le h_2 \le h_{2,j+1,r} \}. $$

So, we have

$$ \sup_{\mathbf{v} \in \mathcal{G}_{j,r}} \mathbb{E} v^2(\mathbf{X}, \mathbf{Y}) \le h_2^p B_1 \le 2^p B_1 h_{2,j,r}^p =: \sigma_{j,r}^2. \qquad (9.27) $$

As the bounded case, we shall use Corollary 10.0.3 in Appendix to bound $\mathbb{E} \|\sum_{i=1}^{n_r} \varepsilon_i v(\mathbf{X}_i, \mathbf{Y}_i)\|_{\mathcal{G}_{j,r}}$. First of all, we remark that each $\mathcal{G}_{j,r} \in \mathcal{G}$, where $\mathcal{G}$ is pointwise measurable class of functions with envelope function

$$ G(\mathbf{y}) = \|\alpha\|_{\infty}^{p} \|K\|_{\infty} F(\mathbf{y}), $$

and satisfies the uniform entropy conditions. Further, we note that each $\mathcal{G}_{j,r}$ satisfies (i) and (v) of the corollary 10.0.3 with

$$ \beta^2 = \theta_p^{2/p}, \quad \sigma^2 = \sigma_{j,r}^2 \quad \text{and} \quad U = B_0(n_r/r)^{1/v}. $$
We obtain, after a small calculation, that, for a suitable positive constants $B_2$ and $B_3$ and for $k \ge 1$, $j \ge 0$,

$$
\begin{align}
\mathbb{E} \left\| \sum_{i=1}^{n_r} \epsilon_i v(\mathbf{X}_i, \mathbf{Y}_i) \right\|_{\mathcal{G}_{j,r}} &\le B_3 \sqrt{n_r h_{2,j,r}^p \log((B_2 h_{2,j,r}^p)^{-1} \lor C_1)} \nonumber \\
&\le B_3 a_{2,j,r}, \tag{9.28}
\end{align}
$$

where

$$
a_{2,j,r} = \sqrt{n_r h_{2,j,r}^p \log((B_2 h_{2,j,r}^p)^{-1}) \lor \log \log n_r}.
$$

Applying Talagrand's inequality in Theorem 10.0.4 with

$$
\mathbf{M} = B_0(n_r/r)^{1/v} \quad \text{and} \quad \sigma_g^2 = \sigma_{j,r}^2 \le 2^p B_1 h_{2,j,r}^p,
$$

we obtain for any $\mathbf{u} > 0$ and large enough $r$,

$$
\mathbb{P} \left\{ \max_{n_{r-1} \le n \le n_r} \left\| \sqrt{n} \alpha_n \right\|_{\mathcal{G}_{j,r}} \ge A_1 (B_3 a_{j,r} + \mathbf{u}) \right\} \\
\le 2 \left\{ \exp \left( -A_2 \mathbf{u}^2 / (2^p B_1 n_r h_{2,j,r}^p) \right) + \exp \left( -A_2 \mathbf{u} r^{1/v} / (B_0 n_r)^{1/v} \right) \right\}.
$$

Denote, for any $\lambda > 1$, $j \ge 0$ and $k \ge 1$,

$$
p_{2,j,r}(\lambda) = \mathbb{P} \left\{ \max_{n_{r-1} \le n \le n_r} \left\| \sqrt{n} \alpha_n \right\|_{\mathcal{G}_{j,r}} \ge A_1 (D_3 + \lambda) a_{2,j,r} \right\}.
$$

We get that

$$
p_{2,j,r}(\lambda) \le 2 \exp \left( - (\lambda^2 A_2 / B_1) \log \log n_r \right) + 2 \exp \left( \left( \sqrt{c\lambda} A_2 / B_0 \right) \sqrt{\log n_r \log \log n_r} \right), \quad (9.29)
$$

where we use the fact that $a_{2,j,r}^2 / \sqrt{n_r h_{2,j,r}} \ge \sqrt{\log \log n_r}$ and $h_{2,j,r} \ge c(\log n_r / n_r)^{1-2/v}$. For $\tilde{\lambda} = A_2 / B_1 \wedge \sqrt{c} A_2 / B_0$, (9.29) implies

$$
p_{2,j,r}(\rho) \le 4 \exp(-\lambda \tilde{\lambda} \log \log n_r).
$$

Let for large enough $r$

$$
l_r = \max\{j : h_{2,j,r} \le 2h_0\}.
$$

Then, we have $l_r \le r$ for all large $r$ and large enough $\lambda$ consequently,

$$
P_r(\lambda) := \sum_{j=0}^{l_r-1} p_{2,j,r}(\lambda) \le 4l_r (\log n_r)^{-\lambda\tilde{\lambda}} \le \frac{1}{r^2}.
$$

Further, keep in mind the definition of $l_r$, we have for large $k$,

$$
2h_{2,l_r,r} = h_{2,l_r+1,r} \ge 2h_0,
$$

from which we infer that, for $n_{r-1} \le n \le n_r$,

$$
\left[ \frac{c \log n}{n}, h_0 \right] \subset \left[ \frac{c \log n_r}{n_r}, h_{2,l_r,r} \right].
$$

Thus for all large enough $k$,

$$
A_r(\lambda) := \left\{
\begin{array}{ll}
\displaystyle \max_{n_{r-1} \le n \le n_{r-1}} \sup_{a_r \le h_2 \le h_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{n} h_2^p \| \eta_{\varphi,n,h_2,r}^{(T)} - \mathbb{E} \eta_{\varphi,n,h_2,r}^{(T)} \|_{\mathbf{I}}}{\sqrt{(\log(1/n_r h_2^p) \vee \log \log n)}} & > 2A_1(B_3 + \lambda) \\
\subset \bigcup_{j=0}^{l_r-1} \left\{ \max_{n_{r-1} \le n \le n_r} \| \sqrt{n} \alpha_n \|_{\mathcal{G}_{j,r}} \ge A_1(B_3 + \lambda) a_{2,j,r} \right\}.
\end{array}
\right.
$$

It, therefore, follows that with an appropriate choice of $\lambda$

$$
\mathbb{P}(A_r(\lambda)) \le P_r(\lambda) \le \frac{1}{r^2},
$$
which by Borel-Cantelli lemma implies the first term (9.19). Now, let's consider the second term (9.20) :

$$
\mathbb{T}_2 := \frac{1}{nh_{2,n}^{p+1}} \left| \sum_{i=1}^{n} \varphi_r^{(T)}(\mathbf{Y}_i) L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \right. \\
\left. - \mathbb{E}\left[\sum_{i=1}^{n} \varphi_r^{(T)}(\mathbf{Y}_i) L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i)\right] \right|.
$$

Set, for $\mathbf{t} \in \mathbf{I}$ and $\varphi \in \mathcal{F}$,

$$
\tilde{\eta}_{\phi,n,h_2,r}^{(T)}(\mathbf{t}) = \sum_{i=1}^{n} \varphi_r^{(T)}(\mathbf{Y}_i) L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i),
$$

$$
\tilde{v}_{\phi,h_2,t}(\mathbf{u},\mathbf{v}) = \varphi(\mathbf{v})L\left(\frac{\mathbf{t}-\mathbf{u}}{h_2}\alpha(f(\mathbf{u}))\right) \alpha^p(f(\mathbf{u}))\delta(\mathbf{X}_i),
$$

$$
\tilde{v}_{\phi,h_2,r,t}^{(T)}(\mathbf{u},\mathbf{v}) = \varphi_r^{(T)}(\mathbf{v}) K\left(\frac{\mathbf{t}-\mathbf{u}}{h_2}\alpha(f(\mathbf{u}))\right) \alpha^p(f(\mathbf{u}))\delta(\mathbf{X}_i).
$$

We observe that

$$
\tilde{\eta}_{\phi,n,h_2,r}^{(T)} - \mathbb{E} \tilde{\eta}_{\phi,n,h_2,r}^{(T)} = \sqrt{n} \alpha_n (\tilde{v}_{\phi,h_2,r,t}^{(T)}),
$$

where $\alpha_n$ is an empirical process indexed by the class of functions

$$
\tilde{\mathcal{G}}_r := \{ \tilde{v}_{\phi,h_2,r,t}^{(T)} : \varphi \in \mathcal{F} \text{ and } \mathbf{t} \in \mathbf{I} \}.
$$

We can see that for each $\tilde{v}_{\phi,h_2,r,t}^{(T)} \in \tilde{\mathcal{G}}_r$,

$$
\| \tilde{v}_{\phi,h_2,t}^{(T)} \|_{\infty} \le 2M(c,p,C) \| \max \alpha \|_{\infty}^{p+1} (n_r/r)^{1/v} =: \tilde{B}_0 (n_r/r)^{1/v}. \quad (9.30)
$$

Taking Advantage of the Condition (F.v), we obtain

$$
\begin{align*}
\mathbb{E} \left[ (\tilde{v}_{\phi,h_2,t}^{(T)})^2 (\mathbf{X}, \mathbf{Y}) \right] &\le \mathbb{E} \left[ \nu_{\phi,h_2,t}^2 (\mathbf{X}, \mathbf{Y}) \right] \\
&\le \mathbb{E} \left[ \varphi^2 (\mathbf{Y}) L^2 \left( \frac{\mathbf{t} - \mathbf{X}}{h_2} \alpha(f(\mathbf{X})) \right) \alpha^{2p} (f(\mathbf{X})) \delta^2 (\mathbf{X}) \right] \\
&\le 4h_2^p \tilde{M} (c, \mathbf{p}, C, K)^2 \theta_v^{2/p} \| \max \alpha \|_{\infty}^{2p+2} \| f \|_{\infty} \\
&=: h_2^p \tilde{B}_1.
\end{align*}
$$

Thus

$$
\sup_{\tilde{v} \in \mathcal{G}_r} \mathbb{E} \tilde{v}^2(\mathbf{X}, \mathbf{Y}) \leq h_2^p \tilde{B}_1. \quad (9.31)
$$

Set, keep in mind the relation (9.26),

$$
\tilde{\mathcal{G}}_{j,r} := \{ \tilde{v}_{\phi,h_2,r,t}^{(T)} : \varphi \in \mathcal{F} \text{ and } h_{2,j,r} \le h_2^p \le h_{2,j+1,r} \}.
$$

So, we have

$$
\sup_{\tilde{v} \in \tilde{\mathcal{G}}_{j,r}} \mathbb{E} \tilde{v}^2(\mathbf{X}, \mathbf{Y}) \le h_2^p \tilde{B}_1 \le 2^p \tilde{B}_1 h_{2,j,r}^p =: \tilde{\sigma}_{j,r}^2. \quad (9.32)
$$

As we proved in the first term, we shall use Corollary 10.0.3 in the Appendix to bound $\mathbb{E} \| \sum_{i=1}^{n_r} \varepsilon_i \tilde{v}(\mathbf{X}_i, \mathbf{Y}_i) \|_{\tilde{\mathcal{G}}_{j,r}}$. We can see that each $\tilde{\mathcal{G}}_{j,r} \in \tilde{\mathcal{G}}$, where $\tilde{\mathcal{G}}$ is pointwise measurable class of functions with envelope function

$$
\tilde{G}(\mathbf{x}, \mathbf{y}) = 2M(c, \mathbf{p}, C) \| \alpha \|_{\infty}^{p+1} F(\mathbf{y}).
$$

We note also that each $\tilde{\mathcal{G}}_{j,r}$ satisfies $(\mathbf{i})$ and $(\mathbf{v})$ of the corollary 10.0.3 with

$$
\beta^2 = \theta_p^{2/p}, \quad \sigma^2 = \tilde{\sigma}_{j,r}^2 \quad \text{and} \quad U = \tilde{B}_0 (n_r/r)^{1/v}.
$$
We obtain that for a positive constants $\tilde{B}_2$ and $\tilde{B}_3$ and for $r \ge 1, j \ge 0$

$$
\begin{align}
\mathbb{E} \left\| \sum_{i=1}^{n_r} \varepsilon_i \tilde{v}(\mathbf{X}_i, \mathbf{Y}_i) \right\|_{\tilde{\mathcal{G}}_{j,r}} &\le \tilde{B}_3 \sqrt{n_r h_{2,j,r}^p \log((\tilde{B}_2 h_{2,j,r}^p)^{-1} \lor C_1)} \nonumber \\
&\le \tilde{B}_3 \tilde{a}_{2,j,r}, \tag{9.33}
\end{align}
$$

where

$$
\tilde{a}_{2,j,r} = \sqrt{n_r h_{2,j,r}^p \log((\tilde{B}_2 h_{2,j,r}^p)^{-1}) \lor \log \log n_r}.
$$

In the same manner as the first term, by applying Talagrand’s inequality in Theorem 10.0.4 with

$$
\mathbf{M} = \tilde{B}_0(n_r/r)^{1/v} \quad \text{and} \quad \sigma_g^2 = \tilde{\sigma}_{j,r}^2 \le 2^p \tilde{B}_1 h_{2,j,r}^p.
$$

we get, for large enough r

$$
\tilde{A}_r(\lambda) := \left\{
\begin{array}{@{}l@{}}
  \displaystyle \max_{n_{r-1} \le n \le n_{r-1}} \sup_{a_r \le h_2 \le h_0} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^{p+1}} \|\tilde{\eta}_{\varphi,n,h_2,r}^{(T)} - \mathbb{E}\tilde{\eta}_{\varphi,n,h_2,r}^{(T)}\|_{\mathbf{I}}}{\sqrt{(\log(1/h_2^p)) \lor \log \log n}} > 2A_1(\tilde{B}_3 + \lambda) \\
  \subset \bigcup_{j=0}^{l_r-1} \left\{ \max_{n_{r-1} \le n \le n_r} \|\sqrt{n}\alpha_n\|_{\tilde{\mathcal{G}}_{j,r}} \ge A_1(\tilde{B}_3 + \lambda) \tilde{a}_{2,j,r} \right\}.
\end{array}
\right.
$$

So, it follows that

$$
\mathbb{P}(\tilde{A}_r(\lambda)) \le \frac{1}{r^2},
$$

which by Borel-Cantelli lemma implies the second term (9.20). Now, we continue with the third term (9.21) :

$$
\begin{align}
T_3 &:= \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \left[ K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \right. \nonumber \\
&\qquad \left. \left. + pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] \right. \nonumber \\
&\quad - \mathbb{E} \left[ \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \left( K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \right. \nonumber \\
&\qquad \left. \left. + pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right) \right] \right| \nonumber \\
&\le \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) - \mathbb{E} \left[ \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) \right] \right| \nonumber \\
&\quad + \frac{1}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) - \mathbb{E} \left[ \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \right| \nonumber \\
&\quad + \frac{p}{nh_2^p} \left| \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right. \nonumber \\
&\qquad \left. - \mathbb{E} \left[ \sum_{i=1}^n \varphi_r^{(T)}(\mathbf{Y}_i) L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] \right| \tag{9.34}
\end{align}
$$

We start by considering the second term in (9.34). Denote, for $\mathbf{t} \in \mathbf{I}$ and $\varphi \in \mathcal{F}$,

$$
\begin{align*}
\mu_{\varphi,n,h_2,r}^{(T)}(\mathbf{t}) &= \sum_{i=1}^{n} \varphi_r^{(T)}(\mathbf{Y}_i) \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \\
&= \sum_{i=1}^{n} \left[ \varphi_r^{(T)}(\mathbf{Y}_i) \alpha^p(f(\mathbf{X}_i)) \sum_{j,\ell=1}^{p} K_{j,\ell}^{\prime\prime}(\xi) \frac{(\mathbf{t}-\mathbf{X}_i)_j (\mathbf{t}-\mathbf{X}_i)_{\ell}}{2h_{2,n}^2} \alpha^2(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right],
\end{align*}
$$

$$
\vartheta_{\varphi,h_2,t}(\mathbf{u},\mathbf{v}) = \varphi(\mathbf{v})\alpha^p(f(\mathbf{u})) \sum_{j,\ell=1}^{p} K_{j,\ell}^{\prime\prime}(\xi) \frac{(\mathbf{t}-\mathbf{u})_j(\mathbf{t}-\mathbf{u})_{\ell}}{2h_{2,n}^2} \alpha^2(f(\mathbf{u}))\delta^2(\mathbf{u}),
$$
$$
\vartheta_{\phi,h_2,r,t}^{(T)}(\mathbf{u},\mathbf{v}) = \varphi_r^{(T)}(\mathbf{v})\alpha^p(f(\mathbf{u})) \sum_{j,l=1}^{p} K_{j,l}''(\xi) \frac{(\mathbf{t}-\mathbf{u})_j(\mathbf{t}-\mathbf{u})_l}{2h_{2,n}^2} \alpha^2(f(\mathbf{u}))\delta^2(\mathbf{u}).
$$

We can see that

$$
\mu_{\phi,n,h_2,r}^{(T)} - \mathbb{E}\mu_{\phi,n,h_2,r}^{(T)} = \sqrt{n}\alpha_n \left( \vartheta_{\phi,h_2,r,t}^{(T)} \right),
$$

where $\alpha_n$ is an empirical process indexed by the class of functions

$$
\mathcal{S}_r := \{ \vartheta_{\phi,h_2,r,t}^{(T)} : \varphi \in \mathcal{F} \text{ and } \mathbf{t} \in \mathbf{I} \}.
$$

Note that for each $\vartheta_{\phi,h_2,r,t}^{(T)} \in \mathcal{S}_r$,

$$
\| \vartheta_{\phi,h_2,r,t}^{(T)} \|_{\infty} \le p^2 \| K'' \|_{\infty} (2/c)^2 \| \max \alpha \|_{\infty}^{2+p} I(\|\mathbf{t}-\mathbf{x}\| \le 2c^{-1}h_2) (n_r/r)^{1/v} =: C_0 (n_r/r)^{1/v}. \quad (9.35)
$$

We obtain, making use of the condition (F.v), that

$$
\begin{align*}
\mathbb{E} \left[ \left( \vartheta_{\phi,h_2,r,t}^{(T)} \right)^2 (\mathbf{X}, \mathbf{Y}) \right] &\le \mathbb{E} \left[ \vartheta_{\phi,h_2,t}^2 (\mathbf{X}, \mathbf{Y}) \right] \le \mathbb{E} \left[ \varphi^2 (\mathbf{Y}) \alpha^{2p} (f(\mathbf{X})) \delta_2^2 (\mathbf{t}, \mathbf{X}) \right] \\
&\le 2^p h_2^p p^4 \theta_v^{2/v} \|K''\|_{\infty}^{2} (2/c)^4 \|\max \alpha\|_{\infty}^{4+2p} I^2 (\|\mathbf{t}-\mathbf{x}\| \le 2c^{-1}h_2) \|f\|_{\infty} =: h_2^p C_1,
\end{align*}
$$

Keep in mind the relation (9.26), we set

$$
\mathcal{S}_{j,r} := \{ \vartheta_{\phi,h_2,r,t}^{(T)} : \varphi \in \mathcal{F} \text{ and } h_{2,j,r} \le h_2 \le h_{2,j+1,r} \}.
$$

So, we have

$$
\sup_{\vartheta \in \mathcal{S}_{j,r}} \mathbb{E} \vartheta^2(\mathbf{X}, \mathbf{Y}) \le 2^p C_1 h_{2,j,r}^p =: \bar{\sigma}_{j,r}^2. \quad (9.36)
$$

We get the desired result by following the same procedures as in (9.19) and (9.20). The first and the third terms in (9.34) will be treated similarly to the second term. The proof of (9.22) and (9.23) are done in a similar manner as the other terms. Hence the proof of the lemma is complete. $\square$

**Lemma 9.2.2.** Suppose the conditions of Theorem 3.1.2 are fulfilled, we infer, with probability 1, that

$$
\limsup_{r \to \infty} \max_{n_{r-1} \le n \le n_r} \sup_{a_n \le h_1 \le b_0} \sup_{\tilde{a}_n \le h_2 \le \tilde{b}_0} \sup_{\mathbf{t} \in \mathbf{I}} \sup_{\varphi \in \mathcal{F}} \frac{\sqrt{nh_2^p} \left| \vartheta_{\phi,n,h_1,h_2,r}^{(R)} - \mathbb{E}\vartheta_{\phi,n,h_1,h_2,r}^{(R)} \right|}{\sqrt{(\log(1/h_2) \vee \log \log n)}} = 0. \quad (9.37)
$$

**Proof of Lemma 9.2.2**

We highlight that

$$
\begin{align*}
\vartheta_{\phi,n,h_1,h_2,r}^{(R)} &= \frac{1}{nh_2^p} \sum_{i=1}^n \varphi_r^{(R)}(\mathbf{Y}_i) K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \\
&\quad + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi_r^{(R)}(\mathbf{Y}_i) L\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \\
&\quad + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi_r^{(R)}(\mathbf{Y}_i) \left[ K\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta_1(\mathbf{X}_i) + \alpha^p(f(\mathbf{X}_i)) \delta_2(\mathbf{t}, \mathbf{X}_i) \right. \\
&\qquad \left. \qquad + pL_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_{2,n}} \alpha(f(\mathbf{X}_i))\right) \alpha^p(f(\mathbf{X}_i)) \delta^2(\mathbf{X}_i) \right] \\
&\quad + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi_r^{(R)}(\mathbf{Y}_i) \left[ L_1\left(\frac{\mathbf{t}-\mathbf{X}_i}{h_2} \alpha(f(\mathbf{X}_i))\right) \delta(\mathbf{X}_i) \delta_1(\mathbf{X}_i) + p\alpha^p(f(\mathbf{X}_i)) \delta(\mathbf{X}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \right] \\
&\quad + \frac{1}{nh_2^p} \sum_{i=1}^n \varphi_r^{(R)}(\mathbf{Y}_i) \delta_2(\mathbf{t}, \mathbf{X}_i) \delta_1(\mathbf{X}_i),
\end{align*}
$$

where

$$
\varphi_r^{(R)}(\mathbf{y}) := \varphi(\mathbf{y}) \mathbb{1}\{F(\mathbf{y}) > (n_r/r)^{1/v}\}. \quad (9.38)
$$