# Stochastic approximation algorithms for multivariate functionals estimation with medical and cognitive fields applications.

Sahar Slama

► To cite this version:

Sahar Slama. Stochastic approximation algorithms for multivariate functionals estimation with medical and cognitive fields applications. *Statistics [math.ST]*. University of Sousse (Tunisia); Université de Poitiers, 2022. English. NNT: tel-03857998

HAL Id: tel-03857998

https://hal.science/tel-03857998

Submitted on 17 Nov 2022

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Republic of Tunisia
Ministry of Higher Education and Scientific Research

# Thesis

Presented at

High School of Sciences and Technologie of Hammam Sousse
Mathematics Laboratory LAMMDA
University of Sousse

And in collaboration with the Laboratory of Mathematics and Applications
University of Poitiers

to obtain the degree of

## DOCTOR OF MATHEMATICS

Mention: Applied Mathematics

Specialty: Statistics

by

Sahar SLAMA

Stochastic approximation algorithms for
multivariate functionals estimation with
medical and cognitive fields applications.

Defended on 31 October 2022 front the committee composed of :

Pr. Adel DAOUAS

Pr. Sophie DABO

Pr. Sofiane GASMI

Pr. Ali GANNOUN

Pr. Moez KHENISSI

Pr. Cyril PERRET

Pr. Khalifa EL MABROUK

Pr. Yousri SLAOUI

Pr. Hamdi FATHALLAH

Sousse University

Lille University

Tunis University

Montpelier University

Sousse University

Poitiers University

Sousse University

Poitiers University

Sousse University

President

Referee

Referee

Member

Member

Member (Invited)

Supervisor

Supervisor

Co-Supervisor
<figure><img src="image_2.png" /></figure>
Acknowledgments

As a preamble to this thesis, I thank ALLAH who helped me and granted me patience and
courage over these years of study.

My highest acknowledgments go first of all to the members of the jury for having agreed to
asses this work and for all their pertinent remarks and outstanding reviews. I have the honor
to warmly thank the referees Pr. Sophie DABO and Pr. Sofiane GASMI, for having agreed to
participate in the assessment of this work.

I have the great pleasure to specifically thank the chairman Pr. Adel DAOUAS and the members
Pr. Moez KHENISSI, Pr. Ali GANNOUN and Pr. Cyril PERRET.

I am deeply indebted to my supervisor in Poitiers Pr. Yousri SLAOUI and my supervisors
in Sousse, Pr. Khalifa EL MABROUK and Pr. Hamdi FATHALLAH. I thank them infinitely
for having framed me and advised me.

I also wish to address my most sincere thanks to the professorial and administrative staff of
the Higher School of Sciences and Technology of Hammam Sousse, of the Laboratory LAMMDA
and of the Laboratory of Mathematics and Applications of Poitiers, for not only their helpful
assistance but also their active contribution to the development of this thesis as well as to the
success of these long years of preparation.

My special thanks go to my dearest parents, Salem and Sihem, who have always been there
for me, "You have sacrificed everything for your only daughter, sparing neither health nor efforts.
I am indebted to you for a life of which I am proud."

I am equally grateful to my fiancée Chokri. "You are the only one who knows the magic trick
to make me smile even when I am going through tough time. I am truly blessed to have you as
part of my life. Thank you for being my lover, my friend and confident.

I especially extend my honorable thanks to my colleague Carlos for his constant support and
his judicious advice.

Eventually, I would like to express my endless gratitude to my grandmother Habiba as well
as all my family members and friends who have provided me with moral and intellectual support
throughout my academic journey.

My sincere apology goes to those whom I didn't mention personally one by one.
# Résumé

L'estimation récursive multivariée est le point central de cette thèse. Notre objectif fondamental est de construire des estimateurs des fonctionnelles multivariées en utilisant des méthodes d'approximations stochastiques. Dans la section d'ouverture, nous fournissons une introduction générale du sujet de l'estimation non-paramétrique et de l'algorithme original d'approximation stochastique récursive. Pour le premier chapitre, nous introduisons un estimateur récursif multivarié pour la fonction de répartition. Nous étudions les propriétés asymptotiques de cet estimateur généralisé et nous le comparons avec l'estimateur multivarié non récursif de Nadaraya. Il s'avère qu'avec un choix adéquat de la taille des pas et un choix approprié de la fenêtre, l'erreur quadratique moyenne MSE (Mean Squared Error en anglais) de l'estimateur multivarié avec plug-in comme méthode de sélection de la fenêtre est plus petite que celle des deux autres estimateurs, à savoir l'estimateur multivarié récursif avec sélection par validation croisée et l'estimateur non récursif de Nadaraya. Le deuxième chapitre traite le problème de l'estimation non paramétrique d'une fonction de répartition cumulative conditionnelle $\pi : (y|x) \mapsto \mathbb{P}[Y \le y | X = x]$. En utilisant la même approche récursive, nous proposons un estimateur récursif multivarié défini par un algorithme d'approximation stochastique. Nous étudions les inférences statistiques de notre estimateur et les comparons avec celles de l'estimateur non récursif de Nadaraya-Watson. Étant donné l'idée d'estimation conditionnelle, et pour le troisième chapitre, nous construisons un estimateur semi-récursif généralisé de type noyau de la fonction de régression $r_{\varphi} : x \mapsto \mathbb{E}[\varphi(Y) | X = x]$, pour une fonction mesurable choisie $\varphi$ et $x \in \mathbb{R}^d$. Afin d'examiner ces propriétés asymptotiques, nous calculons d'abord le biais et la variance de notre estimateur proposé qui dépendent fortement du choix de trois paramètres qui sont les pas et la fenêtre. De plus, nous sommes intéressés par l'étude de la convergence forte de notre estimateur. Il s'avère que pour l'estimation par intervalles de confiance, l'estimateur proposé est meilleur que celui de l'estimateur de Nadaraya Watson. En ce qui concerne le quatrième chapitre, nous tentons d'explorer les processus cognitif et les représentations mentales mobilisées lorsqu'un être humain se prépare à écrire un mot selon l'idée développée dans Perret and Olive(2019). Dans cette perspective, nous proposons une estimation non paramétrique à noyau récursif d'une régression multivariée avec données manquantes pour décrire la production de mots d'écriture. Nous étudions les propriétés asymptotiques de notre estimateur récursif et ses performances par rapport à l'estimateur non récursif. L'estimateur proposé est ensuite appliqué aux données comportementales pour classer certains participants dans des groupes. Cette classification peut être un début pour comprendre les variations de comportement écrites. La dernière section est consacré à la partie conclusion ainsi qu'à quelques perspectives de future recherches.

**Mots-clés:** estimation multivariée, algorithme d'approximation stochastique, méthode d'estimation du noyau, estimation par plug-in, lissage et ajustement de courbe, champ manuscrit et psychologie.
Abstract

Multivariate recursive estimation is the central focus of this thesis. Our basic objective is to construct multivariate functional estimators using stochastic approximations methods. In the opening section, we provide a general introduction of non-parametric estimation topic and the original recursive stochastic approximation algorithm. In the first chapter, we introduce a multi-variate recursive estimator for the distribution function. We study the asymptotic properties of this generalized estimator and we compare it with non-recursive Nadaraya's multivariate distribution estimator. It turns out that, with an adequate choice of the stepsize and an appropriate choice of the bandwidth, the MSE (Mean Squared Error) of the multivariate estimator with plug-in bandwidth selection method can be smaller than the two other estimators, namely the multivariate recursive one with cross-validation selection and the non-recursive one of Nadaraya's estimator. The second chapter deals with non-parametric estimation of a conditional cumulative distribution function (CCDF) $\pi : (y|x) \mapsto \mathbb{P}[Y \le y | X = x]$. Using the same recursive approach, we suggest a multivariate recursive estimator defined by stochastic approximation algorithm. We investigate the statistical inferences of our estimator and compare them with those of non-recursive Nadaraya-Watson's estimator. Given the idea of conditional estimation, and for the third chapter, we construct a generalized semi-recursive kernel-type estimator of the regression function $r_{\phi} : x \mapsto \mathbb{E}[\phi(Y)|X = x]$, for a chosen measurable function $\phi$ and $x \in \mathbb{R}^d$. In order to examine the asymptotic properties of this estimator, we first calculate the bias and the variance of our proposed estimator which strongly depend on the choice of three parameters which are the stepsizes and the bandwidth. Moreover, we are interested in studying the strong pointwise convergence rate of our estimator. It turns out that under the estimation by confidence intervals, the proposed estimator is better than that of the non-recursive Nadaraya-Watson regression one. As far as the fourth chapter is concerned, we attempt to explore cognitive processes and mental representations enacted when a human being prepares to write a word according to the idea developed in Perret and Olive(2019). For this objective, we set forward a non-parametric multivariate recursive kernel regression estimation under missing data to describe writing word production. We examine the asymptotic properties of our recursive estimator and its performance against the non-recursive one. As application, the proposed estimator is then applied to the behavioral data to classify some participants in groups. This classification can be a departure point to understand written behavior variations. The closing section is devoted for the conclusion part as well as some perspectives for future researches.

**Keywords:** multivariate estimation, stochastic approximation algorithm, kernel method, plug-in estimate, smoothing and curve fitting, handwritten and psychology field.
# Contents

Abstract iv

Contents v

List of Figures viii

List of Tables x

Introduction 1

1 Multivariate distribution function estimation using stochastic approximation method 13

1.1 Introduction 13

1.1.1 Presentation of the method 13

1.1.2 Notations and assumptions 15

1.2 Main results 16

1.2.1 Bias and variance of $F_n$ 16

1.2.2 Weak pointwise convergence rate of $F_n$ 16

1.3 Optimal choice of the stepsizes 17

1.3.1 Asymptotic expressions of MWISE[$F_n$] 17

1.4 Asymptotic properties of $\tilde{F}_n$ 18

1.5 Bandwidth selection 19

1.5.1 Cross-Validation 20

1.5.2 Plug-in method 21

1.6 Numerical applications 23

1.6.1 Simulation studies 23

1.6.2 Real Datasets 28

1.7 Conclusion 30

1.8 Proofs 31

2 Statistical inferences for multivariate conditional cumulative distribution function estimation by stochastic approximation method 40

2.1 Introduction 40

2.1.1 Presentation of the method 41

2.1.2 Notations and assumptions 42

2.1.3 Bias and variance of $f_n$ 43

2.2 Main results 43

2.2.1 Bias and variance of $a_n$ 43

2.2.2 Bias and variance of $\pi_n$ 44

2.2.3 Weak pointwise convergence rate of $\pi_n$ 44

2.3 Optimal choice of the stepsizes 45

2.3.1 Asymptotic expressions of MWISE[$\pi_n$] 45
<table><tr><td>2.4</td><td>Asymptotic properties of π̃<sub>n</sub></td><td>47</td></tr><tr><td>2.4.1</td><td>Bias and variance of π̃<sub>n</sub></td><td>47</td></tr><tr><td>2.4.2</td><td>Asymptotic normality of π̃<sub>n</sub></td><td>47</td></tr><tr><td>2.4.3</td><td>Asymptotic expression of MWISE[π̃<sub>n</sub>]</td><td>47</td></tr><tr><td>2.5</td><td>Bandwidth selection</td><td>48</td></tr><tr><td>2.5.1</td><td>Plug-in bandwidth selection:</td><td>48</td></tr><tr><td>2.6</td><td>Numerical applications</td><td>51</td></tr><tr><td>2.6.1</td><td>Simulation studies</td><td>51</td></tr><tr><td>2.6.2</td><td>Simulation Algorithm</td><td>52</td></tr><tr><td>2.6.3</td><td>Real Datasets:</td><td>56</td></tr><tr><td>2.7</td><td>Conclusion</td><td>58</td></tr><tr><td>2.8</td><td>Proofs</td><td>59</td></tr></table>

<table><tr><td>3</td><td>The stochastic approximation method for semi-recursive multivariate kernel-type regression estimation</td><td>66</td></tr><tr><td>3.1</td><td>Introduction</td><td>66</td></tr><tr><td>3.1.1</td><td>Presentation of the method</td><td>67</td></tr><tr><td>3.1.2</td><td>Notations and assumptions</td><td>68</td></tr><tr><td>3.2</td><td>Main results</td><td>68</td></tr><tr><td>3.2.1</td><td>Bias and variance of a<sub>ϕn</sub></td><td>69</td></tr><tr><td>3.2.2</td><td>Bias and variance of r<sub>ϕn</sub></td><td>69</td></tr><tr><td>3.2.3</td><td>Weak pointwise convergence rate of r<sub>ϕn</sub></td><td>70</td></tr><tr><td>3.2.4</td><td>Strong pointwise convergence rate of r<sub>ϕn</sub></td><td>71</td></tr><tr><td>3.3</td><td>Optimal choice of the stepsizes</td><td>72</td></tr><tr><td>3.3.1</td><td>Asymptotic expressions of MWISE[r<sub>ϕn</sub>]</td><td>72</td></tr><tr><td>3.4</td><td>Asymptotic properties of r̃<sub>ϕn</sub></td><td>74</td></tr><tr><td>3.5</td><td>Bandwidth selection</td><td>75</td></tr><tr><td>3.5.1</td><td>Plug-in method</td><td>76</td></tr><tr><td>3.5.2</td><td>Wild Bootstrap approach</td><td>78</td></tr><tr><td>3.6</td><td>Confidence intervals</td><td>79</td></tr><tr><td>3.7</td><td>Numerical applications</td><td>79</td></tr><tr><td>3.7.1</td><td>Simulation studies</td><td>80</td></tr><tr><td>3.7.2</td><td>Real Datasets</td><td>84</td></tr><tr><td>3.8</td><td>Conclusion</td><td>88</td></tr><tr><td>3.9</td><td>Proofs</td><td>88</td></tr></table>

<table><tr><td>4</td><td>Non-parametric multivariate kernel regression estimation to describe cognitive processes and mental representations</td><td>99</td></tr><tr><td>4.1</td><td>Introduction</td><td>99</td></tr><tr><td>4.1.1</td><td>Presentation of the method</td><td>100</td></tr><tr><td>4.1.2</td><td>Notations and assumptions</td><td>101</td></tr><tr><td>4.2</td><td>Main results</td><td>101</td></tr><tr><td>4.2.1</td><td>Bias and variance of m<sub>n</sub></td><td>101</td></tr><tr><td>4.2.2</td><td>Bias and variance of p<sub>n</sub></td><td>102</td></tr><tr><td>4.2.3</td><td>Asymptotic normality of p<sub>n</sub></td><td>102</td></tr><tr><td>4.3</td><td>Optimal choice of the stepsizes</td><td>102</td></tr><tr><td>4.3.1</td><td>Asymptotic expressions of MWISE of p<sub>n</sub></td><td>103</td></tr><tr><td>4.4</td><td>Asymptotic properties of p̃<sub>n</sub></td><td>104</td></tr><tr><td>4.5</td><td>Bandwidth selection</td><td>104</td></tr><tr><td>4.5.1</td><td>Plug-in bandwidth selection method</td><td>104</td></tr><tr><td>4.6</td><td>Application to the handwritten word production</td><td>108</td></tr><tr><td>4.7</td><td>Conclusion</td><td>112</td></tr></table>
<table><tr><td>4.8 Proofs</td><td>112</td></tr><tr><td>Conclusion and perspectives</td><td>119</td></tr><tr><td>Bibliography</td><td>122</td></tr></table>
List of Figures

<table>
  <tbody>
    <tr>
      <td>1</td>
      <td>Representation of some classical kernels.</td>
      <td>3</td>
    </tr>
    <tr>
      <td>1.1</td>
      <td>Qualitative comparison between Nadaraya's distribution estimator and the generalized recursive estimator for Model 1 with n=50 and (γ<sub>n</sub>) = ((2/3 + 0.05)n<sup>-1</sup>).</td>
      <td>24</td>
    </tr>
    <tr>
      <td>1.2</td>
      <td>Qualitative comparison between Nadaraya's distribution estimator with the generalized recursive estimator using Model 1, n=100 and (γ<sub>n</sub>) = ((2/3 + 0.05)n<sup>-1</sup>).</td>
      <td>25</td>
    </tr>
    <tr>
      <td>1.3</td>
      <td>Qualitative comparison between Nadaraya's distribution estimator with the generalized recursive estimator using Model 4, n=50 and (γ<sub>n</sub>) = ((2/3 + 0.05)n<sup>-1</sup>).</td>
      <td>25</td>
    </tr>
    <tr>
      <td>1.4</td>
      <td>Qualitative comparison between Nadaraya's distribution estimator with the generalized recursive estimator using Model 4, n=100 and (γ<sub>n</sub>) = ((2/3 + 0.05)n<sup>-1</sup>).</td>
      <td>25</td>
    </tr>
    <tr>
      <td>1.5</td>
      <td>The reference distribution function F<sub>s</sub> using Model 1.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>1.6</td>
      <td>The reference distribution function F<sub>s</sub> using Model 3.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>1.7</td>
      <td>Nadaraya's estimator F&#x0303; using Model 1 with n=50.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>1.8</td>
      <td>Nadaraya's estimator F&#x0303; using Model 3 with n=50.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>1.9</td>
      <td>The recursive estimator F<sub>n</sub> using Model 1 with n=50.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>1.10</td>
      <td>The recursive estimator F<sub>n</sub> using Model 3 with n=50.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>1.11</td>
      <td>Qualitative comparison between Nadaraya's distribution estimator and the proposed distribution estimator with stepsize (γ<sub>n</sub>) = ([2/3 + 0.05]n<sup>-1</sup>) via lh data of the package datasets and through a plug-in method.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>1.12</td>
      <td>The empirical distribution function.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>1.13</td>
      <td>Nadaraya's estimator.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>1.14</td>
      <td>The recursive estimator.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.1</td>
      <td>The reference CCDF for Model 1 for one simple simulation.</td>
      <td>54</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>The recursive CCDF estimator for Model 1 for one simple simulation.</td>
      <td>54</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>The non-recursive CCDF estimator for Model 1 for one simple simulation.</td>
      <td>54</td>
    </tr>
    <tr>
      <td>2.4</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for Model 1 with n = 200, N = 500 and x = 0.</td>
      <td>55</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for Model 1 with n = 500, N = 500 and x = 0.</td>
      <td>55</td>
    </tr>
    <tr>
      <td>2.6</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for Model 2 with n = 100, N = 500 and x = (0, 0).</td>
      <td>55</td>
    </tr>
    <tr>
      <td>2.7</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for Model 2 with n = 500, N = 500 and x = (0, 0).</td>
      <td>55</td>
    </tr>
    <tr>
      <td>2.8</td>
      <td>The reference CCDF for Model 3 for one simple simulation with n = 500 and x = (1, 1, 1).</td>
      <td>55</td>
    </tr>
    <tr>
      <td>2.9</td>
      <td>The recursive CCDF estimator for Model 3 for one simple simulation with n = 500 and x = (1, 1, 1).</td>
      <td>55</td>
    </tr>
    <tr>
      <td>2.10</td>
      <td>The non-recursive CCDF estimator for Model 3 for one simple simulation with n = 500 and x = (1, 1, 1).</td>
      <td>55</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>2.11</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for the dataset Model 1 with x = 1.</td>
      <td>57</td>
    </tr>
    <tr>
      <td>2.12</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for the dataset Model 2 with x = (0, 0, 0, 0).</td>
      <td>57</td>
    </tr>
    <tr>
      <td>2.13</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for the COVID-19 epidemic dataset Model 1 with x = 17.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>2.14</td>
      <td>Qualitative comparison between the recursive estimator and the non-recursive one for the COVID-19 dataset Model 2 with x = (2, 0, 17).</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>Qualitative comparison between the Nadaraya-Watson estimator and the recursive estimator for Model 1 with n=50 and σε = 0.01.</td>
      <td>81</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Qualitative comparison between the Nadaraya-Watson estimator and the recursive estimator for Model 1 with n=500 and σε = 0.01.</td>
      <td>82</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>Qualitative comparison between the Nadaraya-Watson estimator and the recursive estimator for Model 2 with n=100 and σε = 0.1.</td>
      <td>82</td>
    </tr>
    <tr>
      <td>3.4</td>
      <td>Qualitative comparison between the Nadaraya-Watson estimator and the recursive estimator for Model 2 with n=200 and σε = 0.1.</td>
      <td>82</td>
    </tr>
    <tr>
      <td>3.5</td>
      <td>The reference regression function for Model 3 for one simple simulation with n = 500.</td>
      <td>83</td>
    </tr>
    <tr>
      <td>3.6</td>
      <td>The recursive regression estimator for Model 3 for one simple simulation with n = 500.</td>
      <td>83</td>
    </tr>
    <tr>
      <td>3.7</td>
      <td>The non-recursive regression estimator for Model 3 for one simple simulation with n = 500.</td>
      <td>83</td>
    </tr>
    <tr>
      <td>3.8</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the bivariate COVID-19 application Model 1.</td>
      <td>85</td>
    </tr>
    <tr>
      <td>3.9</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the bivariate COVID-19 application Model 1.</td>
      <td>85</td>
    </tr>
    <tr>
      <td>3.10</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the bivariate COVID-19 application Model 2.</td>
      <td>85</td>
    </tr>
    <tr>
      <td>3.11</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the bivariate COVID-19 application Model 2.</td>
      <td>86</td>
    </tr>
    <tr>
      <td>3.12</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the multivariate PL application Model 3.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>3.13</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the multivariate PL application Model 3.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>3.14</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the multivariate PL application Model 4.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>3.15</td>
      <td>Box-plot of the relative error estimation of the four considered estimators for the multivariate PL application Model 4.</td>
      <td>88</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>Participants' behavior representations, the regression between the reaction time variable Y = CRTs and each covariate (X<sub>1</sub> = H, X<sub>2</sub> = IA, X<sub>3</sub> = Ivar and X<sub>4</sub> = AoA) with the entire database (a total of 137 participants).</td>
      <td>110</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>Box-plot of the relative error estimation of both considered estimators, the recursive one on the left and the non-recursive one on the right.</td>
      <td>110</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>The elbow method of selecting the optimal number of clusters (k = 3) for K-means clustering on the MSE vector.</td>
      <td>111</td>
    </tr>
  </tbody>
</table>
List of Tables

<table>
  <tbody>
    <tr>
      <td>1.1</td>
      <td>Epanechnikov properties.</td>
      <td>24</td>
    </tr>
    <tr>
      <td>1.2</td>
      <td>Quantitative comparison between Nadaraya's distribution estimator and the generalized recursive distribution estimator with the stepsize (γ<sub>n</sub>) = ([2/3 + 0.05]n<sup>-1</sup>) through a plug-in method as well as a cross-validation one in the unidimensional case.</td>
      <td>26</td>
    </tr>
    <tr>
      <td>1.3</td>
      <td>Quantitative comparison between Nadaraya's distribution estimator and the proposed distribution estimator with stepsize (γ<sub>n</sub>) = ([2/3 + 0.05]n<sup>-1</sup>) through a plug-in method as well as a cross-validation one in the bidimensional case.</td>
      <td>28</td>
    </tr>
    <tr>
      <td>1.4</td>
      <td>Quantitative comparison between the I<sub>1</sub>, I<sub>2</sub>, V<sub>F</sub>, MWISE and PSE of Nadaraya's distribution estimator as well as the proposed distribution estimator with stepsize (γ<sub>n</sub>) = ([2/3 + 0.05]n<sup>-1</sup>) via 1h data of the package datasets and through a plug-in method.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>1.5</td>
      <td>Quantitative comparison between the I<sub>1</sub>, I<sub>2</sub>, V<sub>F</sub>, MWISE and PSE of Nadaraya's distribution estimator as well as the proposed distribution estimator with stepsize (γ<sub>n</sub>) = ([2/3 + 0.05]n<sup>-1</sup>) via nwip data of the package kerdiest and through a plug-in method.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>1.6</td>
      <td>Quantitative comparison between the I<sub>1</sub>, I<sub>2</sub>, V<sub>F</sub>, MWISE and PSE of Nadaraya's distribution estimator as well as the proposed distribution estimator with stepsize (γ<sub>n</sub>) = ([2/3 + 0.05]n<sup>-1</sup>) via iris data of the package datasets and through a plug-in method.</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.1</td>
      <td>Quantitative comparison between Nadaraya-Watson estimator and the proposed estimator with stepsizes (γ<sub>n</sub>) = (n<sup>-1</sup>) through a plug-in method for Model 1.</td>
      <td>53</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>Quantitative comparison between the recursive estimator and the non-recursive one with stepsizes (γ<sub>n</sub>) = (n<sup>-1</sup>) through a plug-in method for Model 2.</td>
      <td>53</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>Quantitative comparison between the recursive estimator and the non-recursive one with stepsizes (γ<sub>n</sub>) = (n<sup>-1</sup>) through a plug-in method for Model 3.</td>
      <td>53</td>
    </tr>
    <tr>
      <td>2.4</td>
      <td>Quantitative comparison between the recursive estimator and the non-recursive one with stepsizes (γ<sub>n</sub>) = (n<sup>-1</sup>) through a plug-in method for Model 4.</td>
      <td>53</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>Quantitative comparison between Nadaraya-Watson estimator and the proposed estimator with stepsizes (γ<sub>n</sub>) = (n<sup>-1</sup>) through a plug-in method for the Insurance Company Benchmark (COIL 2000) dataset case.</td>
      <td>56</td>
    </tr>
    <tr>
      <td>2.6</td>
      <td>Quantitative comparison between Nadaraya-Watson estimator and the proposed estimator with stepsizes (γ<sub>n</sub>) = (n<sup>-1</sup>) through a plug-in method for the COVID-19 epidemic dataset case.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>Quantitative comparison between Nadaraya-Watson estimator and the proposed estimator with stepsizes (β<sub>n</sub>, γ<sub>n</sub>) = (n<sup>-1</sup>, n<sup>-1</sup>) through a plug-in method and a bootstrap one in the unidimensional case.</td>
      <td>81</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Quantitative comparison between Nadaraya-Watson estimator and the proposed one with stepsizes (β<sub>n</sub>, γ<sub>n</sub>) = (n<sup>-1</sup>, n<sup>-1</sup>) through plug-in method and the bootstrap one.</td>
      <td>84</td>
    </tr>
  </tbody>
</table>
3.3 Quantitative comparison between Nadaraya-Watson estimator and the proposed one with stepsizes $(\beta_n, \gamma_n) = (n^{-1}, n^{-1})$ through plug-in method and the bootstrap one. 86

4.1 Quantitative comparison between the mean relative error of the multivariate non-recursive Nadaraya-Watson's regression estimator (Non-recursive) and the proposed multivariate recursive kernel regression estimator (Recursive) with stepsize $(\beta_n) = (n^{-1})$ through a plug-in method. 110
# Introduction en français

## Analyse Statistiques

De nos jours, les données sont partout. Elles sont à la disposition de tous. Les données elles-mêmes ne sont que des faits et des chiffres qui doivent être explorés pour obtenir des informations significatives. L'analyse des données est donc cruciale. Il s'agit du processus d'application de techniques statistiques et logiques pour visualiser, réduire, décrire et évaluer les données en informations utiles qui fournissent un meilleur contexte pour les données.

L'analyse des données joue un rôle-clé dans la recherche d'informations significatives qui aideront les entreprises à prendre de meilleures décisions sur la base des résultats.

La science des données est tout aussi importante que l'analyse des données. La science des données correspond à un domaine combinant de multiples méthodes de méthodologie scientifique, des processus, des algorithmes et des outils pour extraire des informations, en particulier d'énormes ensembles de données pour obtenir des informations sur des données structurées et non structurées. Différents termes liés à l'extraction, au nettoyage, à l'analyse et à l'interprétation des données sont souvent utilisés de manière interchangeable dans la science des données.

L'analyse des données consiste à examiner divers facteurs ou variables pour déterminer leur impact sur certaines situations et résultats. Cela nous aide à comprendre pourquoi certains résultats se produisent, ce qui nous permet de faire des prévisions et de prendre des décisions éclairées pour l'avenir. Lorsque nous élaborons des données impliquant plus de deux variables, nous utilisons l'analyse multivariée qui n'est pas une méthode spécifique, mais qui englobe plutôt l'ensemble des techniques statistiques utilisées pour analyser plus de deux variables à la fois. Ces approches nous permettent d'obtenir une vision plus approfondie des données en relation avec des scénarios spécifiques de l'entreprise ou du monde réel. En fait, si vous êtes un analyste ou un scientifique des données ambitieux, l'analyse multivariée est un concept intrinsèque à aborder. Cette technique sera le point central de cette thèse.

## Introduction à la théorie d'estimation

La statistique d'estimation, ou simplement l'estimation, correspond à un cadre d'analyse des données qui repose sur une combinaison de taille d'effet, d'intervalles de confiance, de planification de précision pour analyser les données et interpréter les résultats. Elle se distingue du test de signification de l'hypothèse nulle (TSN) qui est moins informatif.

En statistiques, un estimateur est une fonction permettant d'estimer un paramètre inconnu lié à une distribution de probabilité. Il peut être investi pour déterminer certaines caractéristiques d'une population totale à partir des données observées. Par exemple, dans une enquête, la moyenne de l'échantillon est l'estimateur le plus couramment utilisée de la moyenne de la population.

L'approche de l'estimation est l'une des branches les plus remarquables de la statistique, qui s'intéresse aux propriétés des estimateurs exprimées en terme de convergence, de biais et d'efficacité. De nombreuses approches peuvent être élaborées afin de proposer des estimateurs de qualités multiples pour une même quantité et sur une même base de données.
Par ailleurs, la théorie de l'estimation peut être divisée en deux composantes principales, l'estimation paramétrique et l'estimation non paramétrique.

La statistique paramétrique, qui remonte à Fisher en 1920, représente le cadre classique de la statistique. Le modèle statistique est caractérisé par un nombre fini de paramètres. On prend $M = \{\mathbb{P}_{\theta} : \theta \in \mathbb{R}^p\}$ comme modèle statistique typique qui décrit la distribution des variables aléatoires observées.

Cependant, la statistique non paramétrique étudie des problèmes statistiques où la paramétrisation n'est pas fixe, mais où il existe différents choix de paramètres afin de trouver ceux qui entraînent les procédures les plus efficaces. En fait, la loi est entièrement inconnue. Pour cette raison, nous avons recours à l'estimation des fonctionnelles décrivant le modèle.

Nous nous intéressons essentiellement à la question de l'estimation non-paramétrique d'une fonction de répartition $F$, d'une densité $f$ et d'une fonction de régression $r$.

Par conséquent, afin de résoudre un nombre de problèmes statistiques, il suffit d'avoir une estimation convenable.

## Estimation non paramétrique

L'estimation non paramétrique joue un rôle important dans l'exploration d'une infinité de phénomènes de nature aléatoire. Elle se trouve au cœur de la modélisation de problèmes réels relevant de multiples domaines scientifiques, à savoir les sciences de l'environnement, la sismologie et la psychologie, l'imagerie médicale et les neurosciences.

Un large éventail de méthodes d'estimation non paramétrique de la densité a été élaboré, comme l'estimateur par histogramme, l'estimateur simple et l'estimateur par noyau.

L'estimateur de l'histogramme, introduit par John Graunt (1662), est une fonction en escalier, et donc discontinue, ce qui constitue une lacune pour l'estimation d'une densité. Pour surmonter ce problème, la fonction indicatrice est remplacée par une fonction réelle, appelée estimateur à noyau. En effet, alors qu'un histogramme compte le nombre de points de données dans des intervalles assez arbitraires, l'estimation de la densité par noyau est une fonction définie comme la somme d'une fonction noyau sur chaque point de données. Par conséquent, l'estimation de la densité du noyau est une question essentielle de lissage des données qui permet de faire des inférences sur la population à partir d'un échantillon fini de données.

La technique d'estimation par noyau constitue le point central du paragraphe suivant et de l'ensemble de l'ouvrage.

### Estimation par noyau

En statistique, l'estimation par noyau désigne un processus non paramétrique d'estimation d'une fonction de probabilité inconnue d'une variable aléatoire à l'aide d'une fonction noyau.

Fondamentalement, on suppose que le noyau présente généralement les propriétés suivantes :

$$ \text{i)} \quad \int_{\mathbb{R}} \mathbf{K}(x)dx = 1. $$

$$ \text{ii)} \quad \int_{\mathbb{R}} \mathbf{K}^2(x)dx < +\infty. $$

$$ \text{iii)} \quad \int_{\mathbb{R}} x\mathbf{K}(x)dx = 0. $$

$$ \text{iv)} \quad \int_{\mathbb{R}} x^2 \mathbf{K}(x)dx < +\infty. $$
**Examples 0.1.** *Les noyaux les plus couramment utilisés sont :*

* Le noyau uniforme : $\mathbf{K}(x) = \frac{1}{2} \mathbf{1}_{[-1, 1]}(x)$.

* Le noyau triangulaire : $\mathbf{K}(x) = (1 - |x|) \mathbf{1}_{[-1, 1]}(x)$.

* Le noyau d'Epanechnikov : $\mathbf{K}(x) = \frac{3}{4} (1 - x^2) \mathbf{1}_{[-1, 1]}(x)$.

* Le noyau gaussien : $\mathbf{K}(x) = \frac{1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}}$.

<figure><img src="image_3.png" /><figcaption>Représentation de quelques noyaux classiques.</figcaption></figure>

Nous pouvons distinguer trois estimateurs de noyaux. Nous observons l'estimateur de Parzen-Rosenblatt pour la densité de probabilité, l'estimateur de Nadaraya pour la fonction de répartition et l'estimateur de Nadaraya-Watson pour la fonction de régression.

Soient $(X, Y) \in \mathbb{R} \times \mathbb{R}$ et $(X_1, Y_1), \dots, (X_n, Y_n)$ des vecteurs aléatoires indépendants identiquement distribués comme $(X, Y)$ de densité jointe $g(x, y)$ et soient $f$ et $F$ la densité de probabilité et la fonction de répartition de $X$, qui sont inconnues.

## L'estimateur de densité à noyau : Rosenblatt (1956) et Parzen (1962)

L'estimateur de densité à noyau étudié par Murray Rosenblatt (1956) et Emanuel Parzen (1962) et appelé estimateur de Parzen-Rosenblatt est indiqué par

$$ \tilde{f}_n(x) = \frac{1}{nh_n} \sum_{k=1}^{n} \mathbf{K} \left( \frac{x - X_k}{h_n} \right), $$

où, $\mathbf{K}$ désigne le noyau et $(h_n)$ représente le paramètre de lissage (la fenêtre) qui est une suite de nombres réels positifs qui tendent vers zéro.

## L'estimateur de la fonction de répartition à noyau : Nadaraya (1964)

L'estimateur de la fonction de répartition à noyau élaboré par Nadaraya (1964) et appelé estimateur de Nadaraya s'exprime comme suit

$$ \tilde{F}_n(x) = \frac{1}{n} \sum_{k=1}^{n} \mathcal{K} \left( \frac{x - X_k}{h_n} \right), $$

$$ \text{avec } \mathcal{K}(z) = \int_{-\infty}^{z} \mathbf{K}(u)du. $$
## L'estimateur de régression à noyau : Nadaraya-Watson (1964)

L'estimateur de régression à noyau identifié par Nadaraya (1964) et Watson (1964) et appelé estimateur de Nadaraya-Watson est noté par

$$ \tilde{r}_n(x) = \begin{cases} \frac{\tilde{a}_n(x)}{\tilde{f}_n(x)} & \text{si } \tilde{f}_n(x) \neq 0 \\ 0 & \text{sinon} \end{cases}, $$

$$ \text{avec } \tilde{a}_n(x) = \frac{1}{nh_n} \sum_{k=1}^{n} Y_k \mathbf{K} \left( \frac{x - X_k}{h_n} \right). $$

Pour la construction adéquate de ces estimateurs, deux choix pertinents doivent être faits : la fonction noyau K et le paramètre de lissage $h_n$. Comme le noyau est peu sensible à la forme de l'estimateur, le choix de la fonction noyau n'est pas critique et a moins d'importance que celui de la fenêtre.

Bien que l'étude théorique asymptotique permette d'obtenir la fenêtre optimale, le fait de ne pas connaître la fonction de densité rend son interprétation en pratique assez difficile. De ce point de vue, le lissage par noyau en statistique non paramétrique nécessite le choix d'un paramètre de fenêtre qui est crucial pour la performance effective des estimateurs. Une fenêtre appropriée peut aider à obtenir une fonction estimée proche de la vraie. Cependant, une fenêtre mal choisie peut sérieusement déformer les véritables caractéristiques sous-jacentes de la fonctionnelle. En fait, une petite fenêtre entraîne un sous-lissage et une grande fenêtre entraîne un sur-lissage. Ainsi, un choix judicieux de la fenêtre est fortement recommandé.

## Sélection du paramètre de lissage:

Afin de déterminer le choix optimal du paramètre de lissage, il faut trouver le paramètre qui minimise le risque. Les critères les plus utilisés pour le choix de $h_n$ sont l'Érreur quadratique moyenne MSE (Mean Squared Error), l'Érreur quadratique moyenne intégrée MISE (Mean Integrated Squared Error) et l'Érreur quadratique moyenne intégrée et pondérée MWISE (Mean Weighted Integrated Squared Error). Les MWISE et MISE correspondent au choix du paramètre de lissage constant et le MSE se réfère au paramètre de lissage variable. Tout au long de la thèse, la valeur optimale de la fenêtre $h_n$ est obtenue en minimisant la MWISE asymptotique.

Bien que l'étude théorique asymptotique permette d'obtenir la fenêtre optimale, le fait de ne pas connaître la fonction de densité rend son interprétation en pratique assez difficile. De ce point de vue, le lissage par noyau dans les statistiques non paramétriques nécessite le choix d'un paramètre de fenêtre qui est crucial pour la performance effective des estimateurs. Une fenêtre appropriée peut aider à obtenir une fonction estimée proche de la vraie fonction cible. Cependant, une fenêtre mal choisie peut sérieusement déformer les véritables caractéristiques sous-jacentes de la fonctionnelle. Ainsi, un choix judicieux de la fenêtre est fortement recommandé. Il existe une myriade de méthodes de sélection de la fenêtre basées sur les données recensées dans la littérature, que l'on peut diviser en trois grandes catégories : les techniques de validation croisée, les méthodes du plug-in de deuxième génération et l'approche bootstrap.

La méthode de sélection par validation croisée introduite par Sarda (1993) est une technique de ré-échantillonnage qui consiste à minimiser un estimateur approprié de l'erreur quadratique moyenne. Cependant, cette méthode présente certains inconvénients, comme l'a souligné Altman and Leger (1995) qui a développé une autre méthode efficace, une estimation plug-in qui minimise une estimation de l'erreur quadratique intégrée pondérée moyenne, en utilisant la fonction de densité comme fonction de poids. Une autre méthode est le bootstrap sauvage, introduit dans Hardle and Marron (1991) qui consiste à ré-échantillonner à partir des résidus estimés. La méthode plug-in et la méthode bootstrap sont toutes deux indiscernables et il a été largement
prouvé qu'elles se comportent de manière similaire. Une comparaison détaillée des trois techniques pratiques de sélection de la fenêtre est présentée dans Delaigle and Gijbels (2004).

Tout au long de la thèse, les approches non-paramétriques qui nous intéressent reposent sur la Méthode d'Approximation Stochastique notée SAM. Cette dernière est étudiée dans ce qui suit.

## Méthode d'approximation stochastique

Au cours de la dernière décennie, les flux de données sont devenus un domaine de recherche de plus en plus important. Certains des flux de données les plus courants incluent les données de paquets Internet, l'activité de Twitter, le fil d'actualité de Facebook, les transactions par carte de crédit et, plus récemment, les données relatives à l'épidémie COVID-19. Les algorithmes stochastiques ont été couramment utilisés dans de nombreuses applications de recherche, à savoir l'identification de systèmes, le contrôle adaptatif, les systèmes de transmission et la détection de changements séquentiels.

Dans ces situations, les données arrivent régulièrement de sorte qu'il est impossible de les stocker dans une base de données traditionnelle. Dans un tel contexte, il est très intéressant de construire un estimateur récursif qui n'a pas besoin de stocker toutes les données en mémoire et qui peut être facilement mis à jour pour gérer les données en ligne.

Le mérite fondamental des estimateurs récursifs réside dans le fait que l'estimation peut être mise à jour à chaque nouvelle observation. Par conséquent, au lieu de ré-exécuter les données à chaque fois, il est possible de réécrire notre estimateur considéré comme une combinaison de deux (ou plus) estimateurs, où chaque estimateur est basé sur des ensembles de données distincts, ce qui peut être très intéressant pour maintenir le coût de calcul raisonnablement bas. Il est à noter que tous les calculs et simulations ont été effectués à l'aide du logiciel statistique R.

## Cadre historique

Les algorithmes d'approximation stochastiques sont des versions stochastiques d'algorithmes déterministes comme l'algorithme de Newton ou l'algorithme du gradient. Nous nous intéressons au problème suivant : trouver le zéro $x^*$ d'une fonction à valeur réelle $S$. Nous supposons avoir une valeur très approximative $x_0$ de cette racine. L'idée naturelle de l'algorithme de Newton est de remplacer la courbe représentative de la fonction $S$ par sa tangente au point $x_n$. L'abscisse $x_{n+1}$ du point d'intersection de cette tangente avec l'axe des x est alors donnée pour $n \ge 1$ par

$$x_{n+1} = x_n - \frac{S(x_n)}{S'(x_n)}.$$

Lorsque la dérivée de $S$ n'est pas facilement calculable, nous pouvons considérer une version déterministe de l'algorithme de Robbins-Monro qui consiste à remplacer le calcul de la dérivée par une suite positive décroissante tendant vers 0 de pas ($\gamma_n$). Sous réserve que $S$ ait de bonnes propriétés de régularité, la suite définie par

$$x_{n+1} = x_n - \gamma_n S(x_n)$$

converge vers le zéro de la fonction $S$, noté $x^*$, pour toute valeur initiale $x_0$.

Dans de nombreuses situations, la fonction dont le zéro est recherché n'est connue que pour une perturbation proche du zéro. La recherche des zéros par des méthodes d'optimisation déterministes devient alors plus périlleuse et donc nous avons recours à des algorithmes stochastiques comme celui introduit par Robbins and Monro (1951). Les estimateurs récursifs que nous proposons ont été construits à partir de SAM. En effet, l'incorporation d'algorithmes d'approximation stochastique dans le contexte de la statistique non-paramétrique remonte aux papiers de Robbins and Monro (1951) et Kiefer and Wolfowitz (1952) pour un cadre unidimensionnel.
L'algorithme stochastique général, consacré essentiellement à l'approximation du mode d'une fonction de régression, a pour forme :

$$ \theta_n = \theta_{n-1} + \gamma_n \Phi(\theta_{n-1}, W_n) + \gamma_n^2 \mu_n(\theta_{n-1}, W_n), \quad (1) $$

où :

- $(\gamma_n)$ est une suite positive de nombres réels qui tend vers zéro,

- $(\theta_n)$ est la suite à mettre à jour récursivement,

- $(W_n)$ est une suite de variables aléatoires représentant les observations en ligne,

- $\Phi(\theta, W)$ est la fonction qui détermine essentiellement comment le paramètre est mis à jour en fonction d'une nouvelle observation,

- $\mu_n(\theta_{n-1}, W_n)$ définit une petite perturbation sur l'algorithme.

Par la suite, Blum (1954) a fourni une version multidimensionnelle de cet algorithme. Ces travaux de recherche ont été étendus dans plusieurs directions. En nous inspirant des plus importants, nous pouvons présenter l'algorithme suivant, analysé par Kushner and Clark (1978), Ruppert (1982).

$$ \theta_n = \theta_{n-1} + \gamma_n (\phi(\theta_{n-1}) - W_n + \beta_n), \quad (2) $$

où $(\beta_n)$ désigne une suite de variables aléatoires convergeant vers 0 presque sûrement et $\phi$ désigne une fonction mesurable inconnue. Ils ont démontré que (2) coïncide avec l'algorithme (1) et inclut les processus d'approximation stochastique de Robbins and Monro (1951) et Kiefer and Wolfowitz (1952), qui permettent la recherche du zéro $\theta^*$ de la fonction $\phi$.

Certaines modifications de base ont été incorporées par Nadaraya (1964), Polyak and Tsybakov (1990), Dippon and Renz (1997) et Dippon (2003). Plus tard, Duflo (1997) a corroboré que, sous des conditions standard sur la fonction $\phi$ et sur la suite $(\gamma_n)$, $(\theta_n)$ tend vers $\theta^*$ presque sûrement. Afin de construire un algorithme d'approximation stochastique, Révész (1973) a introduit une application de la procédure de Robbins-Monro et Mokaddem and Pelletier (2007b) ont développé l'application de la procédure Robbins-Monro-Blum. De plus, un algorithme d'estimation d'une fonction de régression a été élaboré par Révész (1977) et a ensuite été utilisé par Tsybakov (1990) pour approximer le mode d'une densité de probabilité. Slaoui (2006) a signalé un estimateur lisse de la fonction de densité dans un cas unidimensionnel en utilisant la méthode d'approximation stochastique. Ensuite, dans l'article de Mokkadem et al. (2009a), le cas multidimensionnel a été étudié afin d'estimer une densité de probabilité multivariée en utilisant l'estimation par intervalles de confiance. De plus, Slaoui (2014b) a réutilisé des méthodes d'approximation stochastique pour améliorer les qualités de l'estimateur de la fonction de répartition. Le principe de déviation large et modérée de cet estimateur a été prouvé dans Slaoui (2019).

L'estimateur classique de régression récursive a été abordé dans Mokaddem et al. (2009b) pour un cadre univarié et une extension multivariée de cet estimateur a été réalisée par Mokaddem and Pelletier (2016). Par la suite, Slaoui (2016) a établi le cas semi-récursif et a introduit un nouveau estimateur qui est la fraction d'une régression récursive par une fonction de densité récursive. De plus, Slaoui (2017) a adopté la technique de probabilité du score de propension et a construit un estimateur de la fonction de densité sous données manquantes. Récemment, un estimateur de la densité conditionnelle a été proposé dans Slaoui and Khardani (2020). L'objectif fondamental de ce chapitre est d'utiliser des algorithmes d'approximation stochastique pour définir des estimateurs d'une densité de probabilité en un point donné.

L'objectif principal de ce chapitre est d'utiliser des algorithmes d'approximation stochastique pour définir des estimateurs d'une densité de probabilité en un point donné. Rappelons que les algorithmes d'approximation stochastique utilisés pour la recherche du zéro d'une fonction inconnue $\phi: y \mapsto S(x) - y$ sont construits comme suit.
(i) On fixe $S_0(x) \in \mathbb{R}$ arbitrairement.

(ii) Pour tout $n \ge 1$, la suite $(S_n)$ est définie récursivement par

$$S_n(x) = S_{n-1}(x) + \gamma_n T_n(x),$$

où

• $(T_n)$ est une suite de fonctions $T_n : \mathbb{R} \to \mathbb{R}$ définie par

$$T_n(x) = \phi(S_{n-1}) - W_n + \beta_n. \quad (3)$$

En d'autres termes, $T_n(x)$ est une observation de la fonction $\phi$ au point $S_{n-1}(x)$.

• $(\gamma_n)$ est une suite de nombres réels positifs qui va jusqu'à zéro telle que

$$\sum \gamma_n = +\infty \quad \text{et} \quad \sum \gamma_n^2 < +\infty. \quad (4)$$

Il convient de noter que, sous la condition $\mathbb{E}[W_n|\mathcal{F}_{n-1}] = 0$ (où $\mathcal{F}_{n-1}$ représente la $\sigma$-algèbre des événements se produisant au temps $n-1$), nous avons

$$\mathbb{E}[T_n(x)] = \phi(S_{n-1}) + \beta_n = S(x) - S_{n-1}(x) + \beta_n. \quad (5)$$

## Estimation multivariée récursive:

Les statistiques multivariées sont axées sur l'exploration des relations entre les variables et leur adéquation au problème en question, ce qui implique plusieurs types d'analyses univariées et multivariées. Fondamentalement, cette dernière repose sur une procédure statistique comprenant les mesures et observations simultanées de données incluant plus d'un facteur de variables indépendantes qui ont un impact sur la variabilité des variables dépendantes. Par conséquent, le principal mérite de l'analyse multivariée réside dans le fait que, comme elle prend en compte plus d'une variable de résultat où diverses quantités différentes présentent un intérêt pour la même analyse, les conclusions obtenues sont plus précises et authentiques par rapport à la situation réelle. De plus, l'analyse multivariée récursive serait une procédure révolutionnaire pour résoudre de nombreux sujets non paramétriques reposant sur des ensembles de données complexes.

L'objectif principal de cette thèse est de proposer une large classe d'estimateurs récursifs à noyau de différentes fonctionnelles multivariées basés sur la méthode d'approximation stochastique. Tout d'abord, nous introduisons l'estimateur récursif de la densité de probabilité multivariée noté $f_n$ et défini dans Mokkadem et al. (2009a).

Soit $X \in \mathbb{R}^d$, $d \ge 1$ et soient $X_1, \dots, X_n$ des vecteurs aléatoires indépendants, identiquement distribués dans $\mathbb{R}^d$, et soient $f$ et $F$ la densité de probabilité et la fonction de distribution de $X$.

## L'estimateur récursif multivarié de la densité:

Pour construire un algorithme stochastique, qui approxime la fonction $f$ en un point donné $x$, nous devons définir un algorithme de recherche du zéro de la fonction

$$\phi : y \mapsto f(x) - y.$$

Nous procédons donc de la manière suivante :

(i) On fixe $f_0(x) \in \mathbb{R}$.

(ii) Pour tout $n \ge 1$, on pose $f_n(x) = f_{n-1}(x) + \gamma_n T_n(x)$,
où $T_n(x)$ est une observation de la fonction $\phi$ au point $f_{n-1}(x)$ vérifiant (17) et $(\gamma_n)$ est une suite positive satisfaisant (18).

Pour identifier $T_n(x)$, nous adoptons l'approche de Révész (1977) et de Tsybakov (1990), et insérons un noyau multivarié $\mathbf{K}$ (une fonction satisfaisant $\int_{\mathbb{R}^d} \mathbf{K}(t)dt = 1$), et une fenêtre $(h_n)$ (une suite de nombres réels positifs qui vont jusqu'à zéro). Par conséquent, sous certaines conditions de régularité sur la fonction de densité de $X$, on a $\mathbb{E}[h_n^{-d}K(\frac{x-X_n}{h_n})] = f(x) + \varepsilon_n(x)$, où $\varepsilon_n(x)$ devient nul lorsque $n$ va à l'infini. Puis suivant (19), on pose

$$T_n(x) = h_n^{-d} \mathbf{K} \left( \frac{x - X_n}{h_n} \right) - f_{n-1}(x).$$

L'algorithme d'approximation stochastique que nous intégrons pour estimer récursivement la densité $f$ au vecteur $x$ peut donc être exprimé comme suit

$$f_n(x) = (1 - \gamma_n)f_{n-1}(x) + \gamma_n h_n^{-d} \mathbf{K} \left( \frac{x - X_n}{h_n} \right). \quad (6)$$

Une forme générale utilisant la récurrence, et sous la condition que $f_0(x) = 0$, a été proposée par Slaoui (2006) dans une version univariée et Mokkadem et al. (2009a) pour une version multivariée. Par conséquent, l'estimateur récursif de la densité est donné par

$$f_n(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right) \quad \text{avec} \quad \Pi_n = \prod_{j=1}^{n} (1 - \gamma_j). \quad (7)$$

**Remarque 0.1.** La relation (21) définit une classe entière d'estimateurs récursifs à noyau d'une densité de probabilité. Il est intéressant de noter que cette classe inclut la sous-classe suivante introduite dans Hall et Patil (1994). Étant donné $(u_n)$ une suite positive non croissante telle que $\sum u_n = +\infty$ et lorsque le pas $(\gamma_n)$ est choisi égal à $(u_n(\sum_{i=1}^n u_i)^{-1})$, l'estimateur $f_n$ (21) est alors exprimé comme suit

$$f_n(x) = \frac{1}{\sum_{i=1}^{n} u_i} \sum_{k=1}^{n} u_k h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right). \quad (8)$$

Nous pouvons mentionner quelques choix particuliers de $(u_n)$. Plus tard, Amiri (2010) a exposé l'estimateur récursif de densité à noyau donné par le choix $(u_n) = (h_n^{d(1-l)})$, $l \in [0, 1]$, qui englobe les trois exemples ci-dessous.

(E1) Le choix $l = 1$, c'est-à-dire $(u_n) = 1$, qui correspond au cas $(\gamma_n) = (\frac{1}{n})$, produit l'estimateur proposé par Wolverton and Wagner (1969).

(E2) Le choix $l = 1/2$, c'est-à-dire $(u_n) = (h_n^{d/2})$ produit l'estimateur considéré par Wegman and Davies (1979).

(E3) Le choix $l = 0$, c'est-à-dire $(u_n) = (h_n^d)$ donne l'estimateur considéré par Deheuvels (1973) et Duflo (1997).

Comme premier résultat de recherche, nous présentons dans ce qui suit notre estimateur de fonction de répartition multivariée.
## L'estimateur récursif multivariée de la fonction de répartition:

Pour construire un algorithme stochastique qui approche la fonction $F$ d'un vecteur donné $x$, nous définissons un algorithme de recherche de la fonction zéro $\phi: y \mapsto F(x) - y$ et nous fixons :

$$ (i) F_0(x) \in [0, 1] $$

$$ (ii) \text{ pour tout } n \geq 1, F_n(x) = F_{n-1}(x) + \gamma_n T_n(x), $$

Afin de définir $T_n(x)$, nous adoptons l'approche de Slaoui (2014b) et nous introduisons un noyau multivarié modifié

$$ \mathcal{K}: \mathbb{R}^d \to [0, 1], x \mapsto \int_{\prod_{i=1}^{d} (-\infty, x_i)} \mathbf{K}(t) dt. $$

En fixant $T_n(x) = \mathcal{K}\left(\frac{x - X_n}{h_n}\right) - F_{n-1}(x)$, l'algorithme d'approximation stochastique que nous considérons pour estimer récursivement la fonction de distribution $F$ au vecteur $x$ peut être indiqué comme suit

$$ F_n(x) = \prod_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \mathcal{K} \left( \frac{x - X_k}{h_k} \right). \quad (9) $$

Pour notre deuxième sujet, reposant sur l'estimation de la fonction de distribution, nous présenterons notre version conditionnelle multivariée notée CCDF, l'estimateur récursif de la fonction de distribution cumulative conditionnelle multivariée.

## L'estimateur récursif de la fonction de distribution cumulative conditionnelle multivariée:

Soit $(X, Y)$ un vecteur aléatoire à valeurs dans $\mathbb{R}^d \times \mathbb{R}^q$, $q \geq 1$, avec densité jointe $f_{(X,Y)}$ et soit $f_X$ la densité de probabilité marginale de $X$. De plus, soit $(X_1, Y_1), \dots, (X_n, Y_n)$ des vecteurs aléatoires indépendants identiquement distribués suivant $(X, Y)$. L'algorithme d'approximation stochastique qui est consacré à l'estimation récursive de la fonction

$$ a: (x, y) \mapsto \int_{\mathbb{R}^q} \mathbb{1}_{\{u \le y\}} f_{(X,Y)}(x, u) du $$

à un couple de vecteurs $(x, y)$ peut être énoncé comme suit :

$$ a_n(x, y) = \prod_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \chi_k(y) h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right), \quad (10) $$

où $\chi$ est une fonction indicatrice multivariée identifiée par $\chi_k: \mathbb{R}^q \to \mathbb{R}$, $y \mapsto \mathbb{1}_{\{Y_k \le y\}}$. Nous nous concentrons sur le problème de l'estimation de la CCDF de $Y$ étant donné $X = x$, fournie par

$$ \pi: \mathbb{R}^q \times \mathbb{R}^d \to \mathbb{R} \\ (y|x) \mapsto \mathbb{P}[Y \le y | X = x] = \frac{a(x, y)}{f_X(x)}, $$

Un estimateur récursif de $\pi$ a été identifié dans Slama et al. (2021) et spécifié par

$$ \pi_n(y|x) = \begin{cases} \frac{a_n(x, y)}{f_n(x)} & \text{si } f_n(x) \neq 0. \\ 0 & \text{sinon} \end{cases}. \quad (11) $$

Dans une autre vision de l'estimation conditionnelle, nous étudions la régression multivariée dans un cas de type noyau.
## L'estimateur récursif de la régression multivariée de type noyau:

Soit $(X, Y)$ un vecteur aléatoire à valeurs dans $\mathbb{R}^d \times \mathbb{R}$ avec densité jointe $g(x, y)$ et soit $f$ la densité de probabilité de $X$. De plus, soit $(X_1, Y_1), \dots, (X_n, Y_n)$ des vecteurs aléatoires indépendants identiquement distribués suivant $(X, Y)$. L'algorithme d'approximation stochastique, qui estime récursivement la fonction de régression

$$a_{\varphi}: x \mapsto r_{\varphi}(x)f(x) = \int_{\mathbb{R}} \varphi(y)g(x, y)dy$$

à un vecteur donné $x$, pour une fonction mesurable $\varphi$ et $x \in \mathbb{R}^d$, peut être noté comme suit :

$$a_{\varphi_n}(x) = Q_n \sum_{k=1}^{n} Q_k^{-1} \beta_k \varphi(Y_k) h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right) \text{ avec } Q_n = \prod_{j=1}^{n} (1 - \beta_j),$$

où $(\beta_n)$ est une suite positive de nombres réels décroissant vers zéro satisfaisant (18).

Tout au long de ce travail, nous considérons l'estimateur général multivarié de type noyau pour la fonction de régression $r: x \mapsto \mathbb{E}[\varphi(Y)|X=x]$ au niveau du vecteur $x$

$$r_{\varphi_n}(x) = \begin{cases} \frac{a_{\varphi_n}(x)}{f_n(x)} & \text{si } f_n(x) \neq 0 \\ 0 & \text{sinon} \end{cases}.$$

**Exemples particuliers sur l'estimation de la régression multivariée:** Supposons que nous ayons une fonction mesurable $\varphi: \mathbb{R} \to \mathbb{R}$, on distingue les exemples particuliers suivants.

1. Pour $\varphi(y) := \mathbb{I}(y) = y$, nous avons la fonction de régression classique

$$r_{\mathbb{I}}(x) = \mathbb{E}[Y|X=x].$$

Un estimateur récursif de $r_{\mathbb{I}}$ a été traité dans Slaoui (2015).

2. Pour $\varphi(y) := \mathbb{I}(y) = y^m$, $m \in \mathbb{N}$, on a les moments conditionnels

$$r_{\mathbb{I}}(x) = \mathbb{E}[Y^m|X=x].$$

3. Pour $\varphi(y) := \chi_t(y) = \mathbb{1}_{\{y \le t\}}$, $t \in \mathbb{R}$, nous avons la fonction de distribution cumulative conditionnelle

$$r_{\chi_t}(y) = \pi(t|x) = \mathbb{P}[Y \le t | X = x].$$

Un estimateur récursif de $r_{\chi_t}$ a été identifié dans Slama et al. (2021).

En suivant l'esprit de l'estimation par régression, nous construisons une classe particulière d'estimateurs pour les régressions sous données manquantes. Cette dernière est donnée par l'approche du score de propension.

## Application : estimation récursive d'une fonction de régression multivariée avec données manquantes :

Soit $(X, T)$ un vecteur aléatoire à valeurs dans $\mathbb{R}^d \times \mathbb{R}$ avec une densité jointe $h(x, t)$ et soit $f$ la densité de probabilité de $X$. De plus, soit $(X_1, T_1), \dots, (X_n, T_n)$ des vecteurs aléatoires indépendants identiquement distribués suivant $(X, T)$. En supposant que $T_1, \dots, T_n$ sont sujets à des données manquantes, les variables aléatoires observées sont alors $Y_i$ et $\delta_i$, où

$$\delta_i = \mathbb{1}_{\{T_i \text{ est observé}\}} \text{ et } Y_i = T_i \times \delta_i, \text{ pour tout } i \in \{1, \dots, n\}.$$
En conséquence, lorsque certains $T_i$ sont manquants, nous introduisons le score de propension, une probabilité élaborée par Rosenbaum and Rubin (1983) et définie comme suit

$$ \psi_i := \mathbb{P}[\delta_i = 1 | T_i], \text{ pour tout } i \in \{1, \dots, n\}. $$

Notre tâche principal dans ce chapitre est de proposer un estimateur récursif permettant d'estimer récursivement la fonction de régression $p(x) = \mathbb{E}[T|X=x]$ sous censure des données. Notre objectif réside alors dans la construction d'un algorithme stochastique, qui approche la fonction de régression

$$ m : x \mapsto \mathbb{E}[T|X = x]f(x) = \int_{\mathbb{R}} th(x, t)dt $$

à un vecteur donné $x$. L'algorithmie d'approximation stochastique que nous considérons pour estimer récursivement la fonction de régression $m$ à un vecteur $x$ peut être exprimé comme suit

$$ m_n(x) = Q_n \sum_{k=1}^{n} Q_k^{-1} \beta_k Y_k \psi_k^{-1} h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right). \quad (12) $$

Un estimateur récursif de $p$ a été proposé dans Slama et al. (2021). Ce dernier est identifié par

$$ p_n(x) = \begin{cases} \frac{m_n(x)}{f_n(x)} & \text{si } f_n(x) \neq 0, \\ 0 & \text{sinon} \end{cases}, \quad (13) $$

où

$$ m_n(x) = Q_n \sum_{k=1}^{n} Q_k^{-1} \beta_k Y_k \psi_k^{-1} h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right). \quad (14) $$

A ce niveau de l'analyse, nous pouvons affirmer que le point central de cette thèse réside dans l'extension des approches existantes au cas multidimensionnel, ce qui est une tâche épineuse, que ce soit d'un point de vue théorique ou pratique, afin d'obtenir des compétences de programmation appropriées pour l'exécution d'algorithmes de simulation ainsi que pour l'analyse d'ensembles de données réelles. Dans cette optique, nous avons tenté d'élargir les domaines de recherche non seulement en construisant des estimateurs récursifs fonctionnels mais aussi en automatisant le choix de la fenêtre où nous avons utilisé les trois méthodes les plus efficaces de sélection de la fenêtre, à savoir la méthode plug-in, la validation croisée et la procédure bootstrap.

La première section est investi comme une partie introductive. En effet, les techniques d'estimation non paramétrique multivariée ont été élaborées. Ainsi, nous avons mis en avant la fameuse estimation par la méthode des noyaux qui est un outil de base pour l'approche d'estimation récursive basée sur des méthodes d'approximation stochastique.

Cette thèse est composée de quatre chapitres principaux qui s'organisent comme suit.

Dans le premier chapitre, qui est une extension de mon projet de master, nous essayons de nous appuyer sur le travail de Sloui (2014b) et de l'étendre au cas des données multivariées. Nous examinons les propriétés asymptotiques de cet estimateur généralisé et nous les comparons à celles de l'estimateur non récursif de la distribution multivariée de Nadaraya. Il s'avère que, avec un choix adéquat du pas ($\gamma_n$) et un choix approprié de la fenêtre ($h_n$), en investissant l'une des deux méthodes de sélection de la fenêtre, la procédure de validation croisée ainsi que la méthode du plug-in de deuxième génération, le MSE des estimateurs généralisés peut être plus petit que celui de l'estimateur de Nadaraya. Nous corroborons nos résultats théoriques par des études de simulation et en considérant quelques ensembles de données réelles telles qu'une application médicale unidimensionnelle des données de l'hormone lutéinisante dans des échantillons de sang féminin, une application bidimensionnelle des données des tremblements de terre
se produisant dans le nord-ouest de la péninsule ibérique *nwip* ainsi qu'une application multidimensionnelle des données de l'*iris* de Fisher ou d'Anderson.

Dans le même esprit que l'estimation d'une fonction de répartition multivariée mais dans un cadre conditionnel et en utilisant une approche récursive, notre deuxième chapitre aborde l'estimation non paramétrique d'une fonction de répartition cumulative conditionnelle (CCDF). En utilisant une approche récursive, nous présentons un estimateur récursif multivarié défini par un algorithme d'approximation stochastique. Notre objectif principal est d'étudier l'inférence statistique de notre estimateur et de la comparer à celle de l'estimateur non récursif de Nadaraya-Watson. Dans cette perspective, nous dérivons d'abord les propriétés asymptotiques de l'estimateur proposé qui dépendent fortement du choix de deux paramètres, le pas $(\gamma_n)$ ainsi que la fenêtre $(h_n)$. La méthode plug-in de deuxième génération implique le choix optimal de la fenêtre et maintient donc une sélection appropriée du paramètre pas. Fondamentalement, nous démontrons que, sous certaines conditions, le *MSE* de l'estimateur proposé peut être plus petit que celui de l'estimateur de Nadaraya Watson. Nous corroborons nos résultats théoriques par des études de simulation et deux applications de jeux de données réelles, à savoir le jeu de données de référence des compagnies d'assurance (COIL 2000) ainsi que les données hospitalières françaises de l'épidémie COVID-19.

En ce qui concerne le troisième chapitre, étant donné l'idée de l'estimation conditionnelle et en considérant un concept général, nous élaborons une extension de l'estimateur semi-récursif de la fonction de régression de type noyau. Nous étudions les propriétés asymptotiques de cet estimateur et nous les comparons à celles de l'estimateur de régression non récursif de Nadaraya Watson. Dans cette perspective, nous calculons d'abord le biais et la variance de l'estimateur proposé qui dépendent fortement du choix de trois paramètres, à savoir les pas $(\beta_n)$ et $(\gamma_n)$ ainsi que la fenêtre $(h_n)$ choisie en utilisant l'une des meilleures méthodes de sélection de fenêtre, l'approche bootstrap combinée avec la méthode plug-in. Un choix judicieux de ces paramètres permet de constater que, sous certaines conditions, l'*MSE* de l'estimateur proposé peut être inférieur à celui de l'estimateur de Nadaraya Watson. Nous confirmons nos résultats théoriques par des études de simulation et en considérant deux applications de jeux de données réelles, à savoir les données hospitalières françaises de l'épidémie COVID-19 ainsi que la charge parasitaire de Plasmodium Falciparum (PL).

Enfin, en ce qui concerne le quatrième chapitre, notre objectif central est d'explorer les processus cognitifs et les représentations mentales mobilisés lorsqu'un être humain se prépare à écrire un mot selon l'idée développée dans Perret and Olive(2019). Pour ce faire, nous mettons en avant un estimateur multivarié non paramétrique de régression récursive à noyau sous données manquantes utilisant l'approche du score de propension afin de caractériser la production de mots écrits. Nous examinons les propriétés asymptotiques de l'estimateur récursif proposé et les comparons à l'estimateur de régression bien connu de Nadaraya-Watson. Nous calculons le biais et la variance de l'estimateur proposé qui dépendent du choix de certains paramètres tels que le pas et la fenêtre. Nous utilisons des procédures basées sur les données pour sélectionner ces paramètres. Ainsi, nous démontrons que, sous certains choix optimaux de ces paramètres, le *MSE* de l'estimateur proposé peut être plus petit que celui obtenu en utilisant l'estimateur de régression de Nadaraya Watson. L'estimateur développé est ensuite appliqué aux données comportementales afin de classer certains participants en groupes. Cette classification peut constituer un point de départ pour aborder les variations de comportement écrites.

Enfin, la dernière section résume la conclusion, fournit des remarques finales et offre de nouvelles perspectives pour les futur travaux de recherche.
# Introduction

## Statistical Analysis

Nowadays, data are everywhere. They are available to everybody. Data themselves are just facts and figures that need to be explored to get meaningful information. Therefore, data analysis is crucial. It stands for the process of applying statistical and logical techniques to visualize, reduce, describe and assess data into useful information that provides a better context for the data. Data analysis plays a key role in finding meaningful information which will help business take better decision basis the output. Along with Data analysis, Data science is equally significant. Data science corresponds to an area combining multiple methods of scientific methodology, processes, algorithms, and tools to extract information from, particularly huge datasets for insights on structured and unstructured data. A different range of terms related to mining, cleaning, analyzing, and interpreting data are often used interchangeably in data science.

Data analytics is concerned with examining various factors or variables to trace how they might impact certain situations and outcomes. This helps us understand why certain outcomes occur, which in turn allows us to make informed predictions and decisions for the future. When elaborating data involving more than two variables, we shall use multivariate analysis which isn't just one specific method, it rather encompasses whole statistical techniques that are used to analyze more than two variables at once. These approaches allow us to gain a deeper insight of data in relation to specific business or real-world scenarios. As a matter of fact, if one is an ambitious data analyst or data scientist, multivariate analysis is an intrinsic concept to address. This technique would be the central focus of this thesis.

## Introduction to estimation principle

Estimation statistics, or simply estimation corresponds to a data analysis framework that rests upon a combination of effect sizes, confidence intervals, precision planning to analyze data and interpret results. It differs from null hypothesis significance testing (NHST) which is less informative.

In statistics, an estimator is a function for estimating an unknown parameter related to a probability distribution. It can be invested to determine certain characteristics of a total population from observed data. For instance, in a survey, the sample mean is the most commonly used estimator of the population mean. Estimation approach is one of the most outstanding branches of statistics, concerned with the properties of estimators expressed in terms of their convergence, bias and efficiency. Numerous approaches can be elaborated in order to set forward estimators of multiple qualities for the same quantity and on the same database.

Furthermore, estimation theory can be divided into two main components, parametric and non-parametric estimation. Parametric statistics, which dates back to Fisher in 1920, stands for the classical framework of statistics. The statistical model is characterized by a finite number of parameters. We take $M = \{P_{\theta} : \theta \in \mathbb{R}^p\}$ as a typical statistical model that describes the distribution of observed random variables.
However, non-parametric statistics investigates statistical problems where the parametrization is not fixed, but there are various choices of parameters in order to find those entailing to the most efficient procedures. In fact, the law is entirely unknown. For this reason, we resort to the estimation of the functionals describing the model. We are basically interested in the issue of the non-parametric estimation of a distribution function F, a density f and a regression function r. Therefore, in order to settle a number of statistical problems, it is sufficient to have a suitable estimate.

## Non-parametric estimation

Non-parametric estimation plays a significant role in exploring a myriad of phenomena with a random nature. It lies at the heart of modeling topic in real problems pertaining to multiple scientific areas, namely environmental sciences, seismology and psychology, medical imaging and neuroscience.

A wide range of methods for non parametric density estimation were elaborated, such as the Histogram estimator, the simple estimator and the kernel estimator. The histogram estimator, introduced by John Graunt (1662), is a stepped function, and therefore discontinuous, which is a deficiency for estimating a density. To overcome this issue, the indicator function is replaced by a real one, called the kernel estimator. Indeed, while a histogram counts the number of data points in quite arbitrary intervals, a kernel density estimate is a function defined as the sum of a kernel function on every data point. Therefore, Kernel density estimation is a critical data smoothing concern that allows inferences to be enacted about the population from a finite sample of data. The kernel estimation technique stands for the central focus of the following paragraph and the whole research work.

### Kernel estimation

In statistics, kernel estimation refers to a non-parametric process of estimating an unknown probability functional of a random variable using a kernel function. Basically, it is assumed that the kernel function typically exhibits the following properties:

$$i) \int_{\mathbb{R}} \mathbf{K}(x)dx = 1.$$

$$ii) \int_{\mathbb{R}} \mathbf{K}^2(x)dx < +\infty.$$

$$iii) \int_{\mathbb{R}} x\mathbf{K}(x)dx = 0.$$

$$iv) \int_{\mathbb{R}} x^2\mathbf{K}(x)dx < +\infty.$$

**Examples 0.2.** The most commonly used kernels are:

$$* \text{ The uniform kernel: } \mathbf{K}(x) = \frac{1}{2}\mathbb{1}_{[-1, 1]}(x).$$

$$* \text{ The triangular kernel: } \mathbf{K}(x) = (1 - |x|) \mathbb{1}_{[-1, 1]}(x).$$

$$* \text{ The Epanechnikov kernel: } \mathbf{K}(x) = \frac{3}{4} (1 - x^2) \mathbb{1}_{[-1, 1]}(x).$$

$$* \text{ The Gaussian kernel: } \mathbf{K}(x) = \frac{1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}}.$$
<figure><img src="image_4.png" /><figcaption>Figure 1: Representation of some classical kernels.</figcaption></figure>

We can distinguish three kernel estimators. We observe the Parzen-Rosenblatt estimator for the probability density, the Nadaraya estimator for the distribution function and the Nadaraya-Watson estimator for the regression function.

Let $(X, Y)$ be a random vector with values in $\mathbb{R} \times \mathbb{R}$ with a joint density function $g(x, y)$ and let $f$ and $F$ denote the probability density and the distribution function of $X$ which are unknown. Moreover, let $(X_1, Y_1), \dots, (X_n, Y_n)$ be independent random vectors identically distributed as $(X, Y)$.

### The kernel density estimator: Rosenblatt (1956) and Parzen (1962)

The kernel density estimator of the density function $f$ investigated by Murray Rosenblatt (1956) and Emanuel Parzen (1962) and called the Parzen-Rosenblatt's estimator is indicated by

$$ \tilde{f}_n(x) = \frac{1}{nh_n} \sum_{k=1}^{n} \mathbf{K} \left( \frac{x - X_k}{h_n} \right), $$

where $\mathbf{K}$ refers to the kernel function and $(h_n)$ denotes the smoothing parameter (the bandwidth), a positive deterministic sequence tending to zero.

### The kernel distribution function estimator: Nadaraya (1964)

The kernel distribution function estimator of the distribution function $F$ elaborated by Nadaraya (1964) and called Nadaraya's estimator is expressed by

$$ \tilde{F}_n(x) = \frac{1}{n} \sum_{k=1}^{n} \mathcal{K} \left( \frac{x - X_k}{h_n} \right), $$

$$ \text{with } \mathcal{K}(z) = \int_{-\infty}^{z} \mathbf{K}(u)du. $$
## The kernel regression estimator: Nadaraya-Watson (1964)

The kernel regression estimator of the regression function $r: x \mapsto \mathbb{E}[Y|X=x]$ identified by Nadaraya (1964) and Watson (1964) and called Nadaraya-Watson's estimator is denoted by

$$ \tilde{r}_n(x) = \begin{cases} \frac{\tilde{a}_n(x)}{\tilde{f}_n(x)} & \text{if } \tilde{f}_n(x) \neq 0, \\ 0 & \text{otherwise} \end{cases}, $$

with $\tilde{a}_n(x) = \frac{1}{nh_n} \sum_{k=1}^{n} Y_k \mathbf{K} \left( \frac{x - X_k}{h_n} \right)$.

For the adequate construction of these estimators, two pertinent choices shall be made: the kernel function $K$ and the smoothing parameter $h_n$. Since the kernel is barely sensitive to the shape of the estimator, the choice of the kernel function is not critical and of less importance than that of the bandwidth.

Although theoretical asymptotic study yields the optimal bandwidth, not knowing the density function makes it quite difficult to interpret it in practice. From this perspective, kernel smoothing in non-parametric statistics requires the choice of a bandwidth parameter which is crucial for the effective performance of the estimators. An appropriate bandwidth can help obtain an estimated functional close to the true one. However, a poorly chosen bandwidth can seriously distort the true underlying characteristics of the functional. In fact, a small bandwidth leads to undersmoothing and a large bandwidth leads to oversmoothing. Thus, judicious choice of bandwidth should be highly recommended.

## Bandwidth selection:

In order to determine the optimal choice of the smoothing parameter, one should find the parameter that minimises the risk. The most widely used criteria for the choice of $h_n$ are the MSE (Mean Squared Error), MISE (Mean Integrated Squared Error) and the MWISE (Mean Weighted Integrated Squared Error). The MWISE and MISE correspond to the choice of the constant smoothing parameter and the MSE refers to the variable smoothing parameter. Throughout the whole thesis work, the optimal value of the bandwidth $h_n$ is obtained through minimizing the asymptotic MWISE.

There are a myriad of data-driven bandwidth selection methods recorded in literature which can be divided into three broad classes: cross-validation techniques, second generation plug-in methods, and the bootstrap approach. The cross-validation selection method introduced by Sarda (1993) stands for a resampling technique that consists of minimising a suitable estimator of the mean squared error. Yet, this method has certain short comings as highlighted by Altman and Leger (1995) who developed another efficient method, a plug-in estimate which minimizes an estimate of the mean weighted integrated squared error, using the density function as a weight function. An alternative method is the wild bootstrap, introduced in Hardle and Marron (1991) which lies in resampling from the estimated residuals. Both, the plug-in method and the bootstrap one are indistinguishable and it has been widely proven that they behave similarly. A detailed comparison of the three practical bandwidth selection techniques is exhibited in Deлаigle and Gijbels (2004).

Throughout the entire thesis, the non-parametric approaches we are concerned with rest upon the Stochastic Approximation Method denoted SAM. The latter is investigated in what follows.
## Stochastic approximation method

Over the past decade, data streams have become an increasingly important area of research. Some of the most common data streams include Internet packet data, Twitter activity, Facebook newsfeed, credit card transactions and more recently COVID-19 epidemic data. Stochastic algorithms have been commonly used in many research applications, namely system identification, adaptive control, transmission systems and sequential change detection.

Recursivity can be crucial when one seeks to infer these kind of phenomena that evolve over time and that require constant updating of the estimates made. In such situations, the data arrives regularly so that it is impossible to store it in a traditional database. In such a context, it is very interesting to build a recursive estimator that does not need to store all the data in memory and that can be easily updated to handle the online data.

Suppose we have a big number of data, we need to store a lot of data in order to recalculate them and then the time of execution would be enormous, since adding a new observation means that non-recursive estimators have to be completely recalculated. For reasons of time calculation optimization and the nature of the data studied, we have chosen to study recursive estimators which can be updated with each new observation added to the database. Therefore, instead of re-running the data each time, it is possible to rewrite our considered estimator as a combination of two (or more) estimators, where each estimator is based on separate datasets, which can be very interesting to keep the computational cost reasonably low. Moreover, recursive estimators can be preferable to non-recursive versions because of their lower asymptotic variance.

It is noteworthy that all computation and simulation have been done using the R statistical software.

## Historical framework

Stochastic approximation algorithms correspond to stochastic versions of deterministic algorithms like Newton's algorithm or the gradient algorithm. We are basically interested in the following problem: finding the zero $x^*$ of a real-valued function $S$. We suppose to have a very approximate value $x_0$ of this root. The natural idea of Newton's algorithm is to replace the representative curve of the function $S$ by its tangent at the point $x_n$. The abscissa $x_{n+1}$ of the intersection point of this tangent with the x-axis is then provided for $n \ge 1$ by

$$x_{n+1} = x_n - \frac{S(x_n)}{S'(x_n)}.$$

When the derivative of $S$ is not easily computable, one can consider a deterministic version of the Robbins-Monro algorithm consisting of replacing the computation of the derivative with a decreasing positive sequence tending to 0 of steps $(\gamma_n)$. Provided that $S$ has good regularity properties, the sequence expressed by

$$x_{n+1} = x_n - \gamma_n S(x_n)$$

converges to the zero of the function $S$, denoted $x^*$, for any initial value $x_0$. In many situations the function whose zero is sought is known only at a perturbation close to zero. The search for zeros by deterministic optimization methods therefore becomes more perilous and one needs to resort to stochastic algorithms like the one introduced by Robbins and Monro (1951).

The recursive estimators that we propose were constructed based on dint of SAM. Indeed, incorporating stochastic approximation algorithms in the context of non-parametric statistics dates back to the papers Robbins and Monro (1951) and Kiefer and Wolfowitz (1952) for a unidimensional framework.

The general stochastic algorithm, devoted essentially to the approximation of the mode of a regression function, has the form:

$$\theta_n = \theta_{n-1} + \gamma_n \Phi(\theta_{n-1}, W_n) + \gamma_n^2 \mu_n(\theta_{n-1}, W_n), \quad (15)$$
where:

`. ($\gamma_n$) is a positive sequence of real numbers decreasing towards zero.`

`. ($\theta_n$) is the sequence to be updated recursively.`

`. ($W_n$) is a sequence of random variables representing the online observations.`

`. $\Phi(\theta, W)$ is the function which essentially determines how the parameter is updated according to a new observation.`

`. $\mu_n(\theta_{n-1}, W_n)$ defines a small perturbation on the algorithm.`

Subsequently, Blum (1954) provided a multidimensional version of this algorithm. These research works have been extended in several directions. Inspired by the most prominent ones, we can introduce the following algorithm analyzed by Kushner and Clark (1978), Ruppert (1982).

$$ \theta_n = \theta_{n-1} + \gamma_n (\phi(\theta_{n-1}) - W_n + \beta_n), \quad (16) $$

where ($\beta_n$) refers to a random variable converging to 0 almost surely and $\phi$ denotes an unknown measurable function. They demonstrated that (16) coincides with the (15) algorithm and includes the stochastic approximation processes of Robbins and Monro (1951) and Kiefer and Wolfowitz (1952), which allow the search for the zero $\theta^*$ of the $\phi$ function.

Some basic modifications were incorporated by Nadaraya (1964), Polyak and Tsybakov (1990), Dippon and Renz (1997) and Dippon (2003). Later, Duflo (1997) corroborated that, under standard conditions on the function $\phi$ and on the sequence ($\gamma_n$), ($\theta_n$) tends to $\theta^*$ almost surely. In order to build up a stochastic approximation algorithm, Révész (1973) introduced an application of the Robbins-Monro procedure and Mokaddem and Pelletier (2007b) developed the application of the Robbins-Monro-Blum procedure. In addition, an algorithm for estimating a regression function was elaborated by Révész (1977) and was afterwards used by Tsybakov (1990) to approximate the mode of a probability density.

Latterly, Slaoui (2006) reported a smooth estimator of the density function in a unidimensional case using the SAM. Next, in the paper of Mokkadem et al. (2009a), the multidimensional case was investigated in order to estimate a multivariate probability density using the estimation by confidence intervals. Additionally, Slaoui (2014b) reused SAM to enhance the qualities of the distribution function estimator. The large and moderate deviation principle of this estimator was proven in Slaoui (2019).

The classical recursive regression estimator was addressed in Mokaddem et al. (2009b) for univariate framework and a multivariate extension of this estimator was carried out by Mokaddem and Pelletier (2016). Subsequently, Slaoui (2016) established the semi-recursive case and introduced a new estimator which is the fraction of a recursive regression by a recursive density function. Afterward, Slaoui (2017) adopted the propensity score probability technique and constructed an estimator of the density function under missing data. More recently, a conditional density estimator was set forward in Slaoui and Khardani (2020).

The basic target of this chapter is to use stochastic approximation algorithms to define estimators of a probability density at a given point.

Let us recall that stochastic approximation algorithms used for the search of the zero of an unknown function $\phi: y \mapsto S(x) - y$ are built up as follows.

(i) We set $S_0(x) \in \mathbb{R}$ arbitrarily.

(ii) For all $n \ge 1$, the sequence ($S_n$) is recursively defined by

$$ S_n(x) = S_{n-1}(x) + \gamma_n T_n(x), $$
where

• $(T_n)$ is a sequence of functions $T_n : \mathbb{R} \to \mathbb{R}$ defined by

$$T_n(x) = \phi(S_{n-1}) - W_n + \beta_n. \qquad (17)$$

In other words, $T_n(x)$ is an observation of the function $\phi$ at the point $S_{n-1}(x)$.

• $(\gamma_n)$ is a sequence of positive real numbers that goes to zero such that

$$\sum \gamma_n = +\infty \quad \text{and} \quad \sum \gamma_n^2 < +\infty. \qquad (18)$$

It's noteworthy that, under the condition $\mathbb{E}[W_n|\mathcal{F}_{n-1}] = 0$ (where $\mathcal{F}_{n-1}$ stands for the $\sigma$-algebra of the events occurring at the time $n-1$), we have

$$\mathbb{E}[T_n(x)] = \phi(S_{n-1}) + \beta_n = S(x) - S_{n-1}(x) + \beta_n. \qquad (19)$$

## Recursive multivariate estimation:

Multivariate statistics centers around exploring the relationships between variables and their goodness of fit to the problem in question involving several types of univariate and multivariate analyses. Basically, the latter relies upon statistical procedure comprising the simultaneous measurements and observations of data including more than one factor of independent variables that impact the variability of dependent variables. Therefore, the major merit of multivariate analysis lies in the fact that as it takes into account more than one outcome variable where various different quantities are of interest to the same analysis, the obtained conclusions are more accurate and authentic to the real-life situation. Further more, recursive multivariate analysis would be a revolutionary procedure in terms of fixing many non-parametric topics relying on complex sets of data.

The major objective of this thesis is to set forward a large class of recursive kernel estimators of different multivariate functionals based on the stochastic approximation method.

First of all, we introduce the recursive multivariate probability density estimator noted $f_n$ which is defined in Mokkadem et al. (2009a).

Let $X \in \mathbb{R}^d$, $d \ge 1$ and let $X_1, \dots, X_n$ be independent, identically distributed $\mathbb{R}^d$-valued random vectors, and let $f$ and $F$ denote the probability density and the distribution function of $X$.

### The multivariate recursive density estimator:

To construct a stochastic algorithm, which approximates the function $f$ at a given point $x$, we need to define an algorithm of search of the zero of the function

$$\phi : y \mapsto f(x) - y.$$

We thus proceed in the following way :

(i) We set $f_0(x) \in \mathbb{R}$.

(ii) For all $n \ge 1$, we set $f_n(x) = f_{n-1}(x) + \gamma_n T_n(x)$,

where $T_n(x)$ is an observation of the function $\phi$ at the point $f_{n-1}(x)$ verifying (17) and $(\gamma_n)$ is a positive sequence satisfying (18).

To identify $T_n(x)$, we adopt the approach of Révész (1977) and of Tsybakov (1990), and insert a
multivariate kernel $\mathbf{K}$ (a function satisfying $\int_{\mathbb{R}^d} \mathbf{K}(t)dt = 1$), and a bandwidth $(h_n)$ (a sequence of positive real numbers that goes to zero). Therefore, under some regularity conditions on the density function of $X$, we have $\mathbb{E}\left[h_n^{-d}K\left(\frac{x-X_n}{h_n}\right)\right] = f(x) + \varepsilon_n(x)$, where $\varepsilon_n(x)$ goes to zero as $n$ goes to infinity. Then following (19), we set

$$T_n(x) = h_n^{-d}\mathbf{K}\left(\frac{x-X_n}{h_n}\right) - f_{n-1}(x).$$

The stochastic approximation algorithm we integrate to recursively estimate the density $f$ at the vector $x$ can thus be expressed as

$$f_n(x) = (1 - \gamma_n)f_{n-1}(x) + \gamma_n h_n^{-d} \mathbf{K} \left( \frac{x - X_n}{h_n} \right). \quad (20)$$

A general form using recurrence, and under the condition that $f_0(x) = 0$, was proposed by Slaoui (2006) in a univariate version and Mokkadem et al. (2009a) for a multivariate one. Therefore, the recursive density estimator is given by

$$f_n(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right) \quad \text{with} \quad \Pi_n = \prod_{j=1}^{n} (1 - \gamma_j). \quad (21)$$

**Remark 0.3.** The relation (21) defines a whole class of recursive kernel estimators of a probability density. It's noteworthy that this class includes the following subclass introduced in Hall and Patil (1994). Given $(u_n)$ a nonincreasing positive sequence such that $\sum u_n = +\infty$ and when the stepsize $(\gamma_n)$ is chosen equal to $(u_n(\sum_{i=1}^n u_i)^{-1})$, the $f_n$ estimator (21) is then expressed as

$$f_n(x) = \frac{1}{\sum_{i=1}^{n} u_i} \sum_{k=1}^{n} u_k h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right). \quad (22)$$

Later Amiri (2010) exhibited the recursive kernel density estimator given by the choice $(u_n) = (h_n^{d(1-l)})$, $l \in [0, 1]$, which englobes the three examples below.

(E1) The choice $l = 1$, i.e $(u_n) = 1$, which corresponds to the case $(\gamma_n) = (\frac{1}{n})$, produces the estimator proposed by Wolverton and Wagner (1969).

(E2) The choice $l = 1/2$, i.e $(u_n) = (h_n^{d/2})$ yields the estimator considered by Wegman and Davies (1979).

(E3) The choice $l = 0$, i.e $(u_n) = (h_n^d)$ gives the estimator considered by Deheuvels (1973) and Duflo (1997).

As a first research result, we introduce in the following our proposed multivariate distribution function estimator.

**The multivariate recursive distribution function estimator:**

To build up a stochastic algorithm which approaches the function $F$ to a given vector $x$, we define an algorithm of search of the zero function $\phi: y \mapsto F(x) - y$ and we set:

(i) $F_0(x) \in [0, 1]$

(ii) for all $n \ge 1$, $F_n(x) = F_{n-1}(x) + \gamma_n T_n(x)$,
In order to define $T_n(x)$, we adopt the approach of Slaoui (2014b) and we introduce a modified multivariate kernel

$$ \mathcal{K}: \mathbb{R}^d \to [0,1], x \mapsto \int_{\prod_{i=1}^{d} (-\infty, x_i)} \mathbf{K}(t)dt. $$

By setting $T_n(x) = \mathcal{K} \left( \frac{x - X_n}{h_n} \right) - F_{n-1}(x)$, the stochastic approximation algorithm that we consider to estimate recursively the distribution function $F$ at the vector $x$ can be indicated as follows

$$ F_n(x) = \prod_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \mathcal{K} \left( \frac{x - X_k}{h_k} \right). \quad (23) $$

For our second topic, resting upon the distribution function estimation, we shall present our multivariate conditional version denoted CCDF, the multivariate recursive conditional cumulative distribution function estimator.

## The multivariate recursive conditional cumulative distribution function estimator:

Let $(X, Y)$ be a random vector with values in $\mathbb{R}^d \times \mathbb{R}^q$ with a joint density function $f_{(X,Y)}$ and let $f_X$ denote the marginal probability density of $X$. Moreover, let $(X_1, Y_1), \dots, (X_n, Y_n)$ be independent random vectors identically distributed as $(X, Y)$.

The stochastic approximation algorithm that is devoted to estimate recursively the function

$$ a : (x, y) \mapsto \int_{\mathbb{R}^q} \mathbb{1}_{\{u \le y\}} f_{(X,Y)}(x, u) du $$

at a couple of vectors $(x, y)$ can be stated as follows

$$ a_n(x, y) = \prod_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \chi_k(y) h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right), \quad (24) $$

where $\chi$ is a multivariate indicator function identified by $\chi_k: \mathbb{R}^q \to \mathbb{R}$, $y \mapsto \mathbb{1}_{\{Y_k \le y\}}$.

Our central focus is upon the problem of estimating the CCDF of $Y$ given $X = x$ provided by

$$ \pi: \mathbb{R}^q \times \mathbb{R}^d \to \mathbb{R} \\ (y|x) \mapsto \mathbb{P}[Y \le y | X = x] = \frac{a(x, y)}{f_X(x)}, $$

A recursive estimator of $\pi$ was identified in Slama et al. (2021) and specified by

$$ \pi_n(y|x) = \begin{cases} \frac{a_n(x, y)}{f_n(x)} & \text{if } f_n(x) \neq 0 \\ 0 & \text{otherwise} \end{cases}. \quad (25) $$

In an other vision of conditional estimation, we investigate the multivariate regression in a kernel-type case.

## The multivariate recursive kernel-type regression estimator:

Let $(X, Y)$ be a random vector with values in $\mathbb{R}^d \times \mathbb{R}$ with a joint density function $g(x, y)$ and let $f$ denote the probability density of $X$. Moreover, let $(X_1, Y_1), \dots, (X_n, Y_n)$ be independent random vectors identically distributed as $(X, Y)$.

The stochastic approximation algorithm, which estimates recursively the regression function

$$ a_{\varphi}: x \mapsto r_{\varphi}(x)f(x) = \int_{\mathbb{R}} \varphi(y)g(x, y)dy $$
at a given vector $x$, for a chosen measurable function $\varphi$ and $x \in \mathbb{R}^d$, can be denoted as follows

$$a_{\varphi_n}(x) = Q_n \sum_{k=1}^{n} Q_k^{-1} \beta_k \varphi(Y_k) h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right) \quad \text{with} \quad Q_n = \prod_{j=1}^{n} (1 - \beta_j),$$

where $(\beta_n)$ is a positive sequence of real numbers decreasing towards zero satisfying (18).

Throughout this work, we consider the general multivariate kernel-type estimator for the regression function $r_{\varphi}: x \mapsto \mathbb{E}[\varphi(Y)|X=x]$ at the vector $x$

$$r_{\varphi_n}(x) = \begin{cases} \frac{a_{\varphi_n}(x)}{f_n(x)} & \text{if } f_n(x) \neq 0 \\ 0 & \text{if } f_n(x) = 0 \end{cases}. \qquad (26)$$

**Particular regression estimation cases:** Suppose we have a measurable function $\varphi: \mathbb{R} \to \mathbb{R}$, we distinguish the following particular examples.

1. For $\varphi(y) := \mathbb{I}(y) = y$, we have the classical regression function

$$r_{\mathbb{I}}(x) = \mathbb{E}[Y|X=x].$$

A recursive estimator of $r_{\mathbb{I}}$ was reported in Slaoui (2015).

2. For $\varphi(y) := \mathbb{I}(y) = y^m$, $m \in \mathbb{N}$, we have the conditional moments

$$r_{\mathbb{I}}(x) = \mathbb{E}[Y^m|X=x].$$

3. For $\varphi(y) := \chi_t(y) = 1_{\{y \le t\}}$, $t \in \mathbb{R}$, we have the conditional cumulative distribution function

$$r_{\chi_t}(y) = \pi(t|x) = \mathbb{P}[Y \le t | X = x].$$

A recursive estimator of $r_{\chi_t}$ was identified in Slama et al. (2021).

Following the spirit of regression estimation, we construct a particular class of estimators for regressions under missing data. This latter is given by the propensity score approach.

**Application: recursive estimation of multivariate regression function under missing data:**

Let $(X, T)$ be a random vector with values in $\mathbb{R}^d \times \mathbb{R}$ with a joint density function $h(x, t)$ and let $f$ denote the probability density of $X$. Moreover, let $(X_1, T_1), \dots, (X_n, T_n)$ be independent random vectors identically distributed as $(X, T)$. Assuming that $T_1, \dots, T_n$ are subjects to missing data, the observed random variables are then $Y_i$ and $\delta_i$, where

$$\delta_i = 1_{\{T_i \text{ is observed}\}} \text{ and } Y_i = T_i \times \delta_i, \text{ for all } i \in \{1, \dots, n\}.$$

Accordingly, when some $T_i$ are missing, we introduce the propensity score, a probability elabo- rated by Rosenbaum and Rubin (1983) and defined as followed

$$\psi_i := \mathbb{P}[\delta_i = 1 | T_i], \text{ for all } i \in \{1, \dots, n\}.$$

Our basic purpose in this chapter is to propose a recursive estimator to estimate recursively the regression function $p(x) = \mathbb{E}[T|X=x]$ under censoring data. Our aim then resides in building up a stochastic algorithm, which approaches the regression function $m: x \mapsto \mathbb{E}[T|X=x]f(x) =$
$\int_{\mathbb{R}} th(x,t)dt$ at a given vector $x$. The stochastic approximation algorithm that we consider to estimate recursively the regression function $m$ at a vector $x$ can be expressed as follows

$$m_n(x) = Q_n \sum_{k=1}^{n} Q_k^{-1} \beta_k Y_k \psi_k^{-1} h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right). \quad (27)$$

A recursive estimator of $p$ was recorded in Slama et al. (2021) and identified by

$$p_n(x) = \begin{cases} \frac{m_n(x)}{f_n(x)} & \text{if } f_n(x) \neq 0, \\ 0 & \text{if } f_n(x) = 0 \end{cases}, \quad (28)$$

## Thesis contributions

At this stage of analysis, we would assert that the central focus in this thesis resides in extending existing approaches to the multidimensional case, which is a thorny task, either from a theoretical or a practical point of view, in order to achieve fitting programming skills for running simulation algorithms as well as real datasets analysis. From this perspective, we attempted to broaden the research areas by not only building functional recursive estimators but also by automating the choice of the bandwidth where we used the three most effective methods of bandwidth selection, namely the plug-in method, the cross validation and the bootstrap procedure.

This section was invested as an introductory part. Indeed, the techniques for multivariate non-parametric estimation were elaborated. Therefore, we set forward the famous kernel method estimate which is a basic tool for the recursive estimation approach based on SAM.

This thesis is composed of four main chapters which are laid out as follows.

In the first chapter, being an extension to my master project, we attempt to build upon the work of Slaoui (2014b) and extend it to the case of multivariate data. We examine the asymptotic properties of this generalized estimator and we compare them with those of the non-recursive Nadaraya's multivariate distribution estimator. It turns out that, with an adequate choice of the stepsize ($\gamma_n$) and an appropriate choice of the bandwidth ($h_n$), investing one of both methods of bandwidth selection, the cross-validation procedure as well as the second generation plug-in method, the MSE of the generalized estimators can be smaller than the one of Nadaraya's estimator. We corroborate our theoretical results through simulation studies and by considering some real datasets such that a medical unidimensional application of the luteinizing hormone in female blood samples data, a bidimensional application of the earthquakes occurring in the Northwest of the Iberian Peninsula nwip data as well as a multidimensional application of Fisher's or Anderson's iris data.

Proceeding with the same spirit in terms of estimating a multivariate distribution function but in a conditional framework, our second chapter tackles non-parametric estimation of a conditional cumulative distribution function (CCDF). Using a recursive approach, we set forward a multivariate recursive estimator defined by stochastic approximation algorithm. Our basic objective is to investigate the statistical inference of our estimator and compare it with that of non-recursive Nadaraya-Watson's estimator. From this perspective, we first derive the asymptotic properties of the proposed estimator which highly depend on the choice of two parameters, the stepsize ($\gamma_n$) as well as the bandwidth ($h_n$). The second generation plug-in method entails the optimal choice of the bandwidth and therefore maintains an appropriate selection of the stepsize parameter.
Basically, we demonstrate that, under some conditions, the *MSE* of the proposed estimator can be smaller than the one of Nadaraya Watson's estimator. We corroborate our theoretical results through simulation studies and two real dataset applications, namely the Insurance Company Benchmark (COIL 2000) dataset as well as the French Hospital Data of COVID-19 epidemic.

As far as the third chapter is concerned, given the idea of conditional estimation and considering a general concept, we elaborate an extension of the semi-recursive kernel-type regression function estimator. We investigate the asymptotic properties of this estimator and compare them with non-recursive Nadaraya Watson regression estimator ones. From this perspective, we first calculate the bias and the variance of the proposed estimator which strongly depend on the choice of three parameters, namely the stepsizes ($\beta_n$) and ($\gamma_n$) as well as the bandwidth ($h_n$) chosen using one of the best methods of bandwidth selection, the bootstrap approach combined with the plug-in method. A convenient choice of those parameters yields that, under some conditions, the *MSE* of the proposed estimator can be smaller than that of Nadaraya Watson's estimator. We confirm our theoretical results through simulation studies and by considering two real dataset applications, namely the French Hospital Data of COVID-19 epidemic as well as the Plasmodium Falciparum Parasite Load (PL).

Last but not least, in terms of the fourth chapter, our central objective is to explore cognitive processes and mental representations mobilized when a human being prepares to write a word according to the idea developed in Perret and Olive(2019). For this purpose, we foreground a non-parametric multivariate recursive kernel regression estimator under missing data using the propensity score approach so as to characterize writing word production. We examine the asymptotic properties of the proposed recursive estimator and compare them to the well known Nadaraya-Watson's regression estimator. We calculate the bias and the variance of the proposed estimator which depend on the choice of some parameters such as the stepsize and the bandwidth. We handle some data-driven procedures to select these parameters. Thus, we demonstrate that, under some optimal choices of these parameters, the *MSE* of the proposed estimator can be smaller than the one obtained by using Nadaraya Watson's regression estimator. The developed estimator is then applied to the behavioral data so as to classify some participants in groups. This classification may stand for a departure point to tackle written behavior variations.

Eventually, the closing section wraps up the conclusion, provides outstanding concluding remarks and offers new perspectives for future research works.
# Chapter 1

## Multivariate distribution function estimation using stochastic approximation method

### 1.1 Introduction

The estimation of a distribution function stands for an intrinsic tool in the study of multiple random phenomena. In fact, it's at the heart of modeling questions in real problems of such scientific fields, such as environmental sciences, seismology and particularly in biology, medical imaging and neuroscience. Numerous methods were set forward and explored in order to elaborate an effective estimator compared to the empirical distribution function. Our basic method applied in this study will be the non-parametric adaptive kernel density estimator for recursive multivariate distribution function estimation.

Inspired by the most prominent expansions, Slaoui (2014b) reused stochastic approximation methods to set forward an univariate recursive estimator of the distribution function. Thereafter, he provided a special stepsize and an adequate plug-in bandwidth selection to achieve a better estimation compared to Nadaraya's unidimensional data estimation. The large and moderate deviation principle of this estimator was proven in Slaoui (2019). The central objective of this chapter lies in extending this latter estimator to a multidimensional case. We demonstrated that, under some suitable conditions, essentially on the stepsize and on the bandwidth, the generalized estimators are closer to the true distribution function compared to the non-recursive Nadaraya's multidimensional distribution estimator.

#### 1.1.1 Presentation of the method

Let $X, X_1, \dots, X_n$ be independent, identically distributed $\mathbb{R}^d$-valued random vectors, $d \ge 1$, and let $f$ and $F$ denote the probability density and the distribution function of $X$.

To build up a stochastic algorithm which approaches the function $F$ to a given vector $x \in \mathbb{R}^d$, we define an algorithm of search of the zero function $\phi: y \mapsto F(x) - y$ and we set:

$$ (i) F_0(x) \in [0, 1] $$

$$ (ii) \text{ for all } n \ge 1, F_n(x) = F_{n-1}(x) + \gamma_n T_n(x), $$

where $(\gamma_n)$ is a positive sequence of real numbers decreasing towards zero satisfying (18) and $T_n(x)$ is an observation of the function $\phi$ at the point $F_{n-1}(x)$ verifying (17).

In order to define $T_n(x)$, we first introduce a modified multivariate kernel

$$ \mathcal{K}: \mathbb{R}^d \to [0, 1], x \mapsto \int \prod_{i=1}^{d} (-\infty, x_i) \mathbf{K}(t)dt. $$
By setting $T_n(x) = K(\frac{x-X_n}{h_n}) - F_{n-1}(x)$, the stochastic approximation algorithm that we consider to estimate recursively the distribution function $F$ at the vector $x$ can be expressed as follows:

$$F_n(x) = (1 - \gamma_n)F_{n-1}(x) + \gamma_n K\left(\frac{x - X_n}{h_n}\right). \qquad (1.1)$$

Throughout this chapter, we consider that $F_0(x) = 0$. Then, our estimator (4.1) can be rewritten

$$F_n(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k K\left(\frac{x - X_k}{h_k}\right), \quad \text{with} \quad \Pi_n = \prod_{j=1}^{n} (1 - \gamma_j). \qquad (1.2)$$

Our major aim in this part is to explore the asymptotic properties of our generalized recursive kernel estimator of a distribution in the case of multivariate data and subsequently confirm its performances. We are basically interested also in comparing our generalized estimator to the generalized non-recursive multivariate distribution estimator of Nadaraya $\tilde{F}_n$ indicated by:

$$\tilde{F}_n(x) = \frac{1}{n} \sum_{i=1}^{n} K\left(\frac{x - X_i}{h_n}\right). \qquad (1.3)$$

For this reason, we attempt first to calculate the bias and the variance of the generalized estimator $F_n$ which largely depends on the choice of two parameters, namely the stepsize $(\gamma_n)$ and the bandwidth $(h_n)$. Moreover, we introduce the asymptotic properties of multivariate Nadaraya's estimator $F_n$. It turns out that, by using an adequate choice of the stepsize of the proposed algorithm, the Mean Weighted Integrated Squared Error of the generalized estimators is smaller than the one of Nadaraya's estimator. Furthermore, we depict that with an appropriate choice of the bandwidth $(h_n)$, using one of two data-driven bandwidth selections; the cross-validation procedure as well as the second generation plug-in method, the MSE of the generalized estimators can be smaller than the one of Nadaraya's multidimensional estimator. Likewise, we confirm that in our context the plug-in method can be more efficient than the cross-validation. We corroborate these theoretical results with some simulations as well as real datasets. We consider as a first application, a real dataset in a unidimensional case, the lh (Luteinizing Hormone in Blood Samples data) available on the datasets package for a biological concept. As a bivariate example, we consider the nwip data available on the kerdiest package for a seismology concept. Moreover, for a multidimensional application, we consider the iris data available on the datasets package for an environmental sciences concept.

First of all, let us recall the following definition of the class of regularly varying sequences.

**Definition 1.1.** Let $(v_n)_{n \ge 1}$ be a nonrandom positive sequence and $\gamma \in \mathbb{R}$. We state that

$$(v_n)_{n \ge 1} \in \mathcal{GS}(\gamma) \quad \text{if} \quad \lim_{n \to +\infty} n \left[ 1 - \frac{v_{n-1}}{v_n} \right] = \gamma.$$

This definition was introduced by Galambos and Seneta (1973) to characterize regularly varying sequences and by Mokaddem and Pelletier (2007a) in the context of stochastic approximation algorithms. Note that the acronym GS stands for (Galambos and Seneta). Typical sequences in $\mathcal{GS}(\gamma)$ are, for $b \in \mathbb{R}$, $n^\gamma, n^\gamma(\log n)^b, n^\gamma(\log \log n)^b$, and so on.

At this stage of analysis, it is worth mentioning that the following assumptions will be useful for the whole thesis investigation.

(A1) $\mathbf{K}: \mathbb{R}^d \to \mathbb{R}_+$ is a continuous bounded function satisfying:

$$\int_{\mathbb{R}^d} \mathbf{K}(u)du = 1, \forall j \in \{1, \dots, d\}, \int_{\mathbb{R}} u_j \mathbf{K}(u)du_j = 0 \text{ and } \int_{\mathbb{R}^d} u_j^2 \mathbf{K}(u)du < +\infty.$$
(A2) (i) $(\gamma_n)_{n \ge 1} \in \mathcal{GS}(-\alpha)$, with $\alpha \in (\frac{1}{2}, 1]$.

(ii) $(h_n)_{n \ge 1} \in \mathcal{GS}(-a)$, with $a \in (0, 1)$.

(iii) $\lim_{n \to +\infty} n\gamma_n \in (\min\{2a, \frac{a+a}{2}\}, +\infty]$.

**Discussion of the assumptions:**

All these assumptions are standard and are generally assumed within the context of non-parametric estimation. Classical assumption (A1) provides regularity conditions on the kernel density estimator introduced by Rosenblat (1956) and Parzen (1962). It is widely used in the non-parametric framework for the functional estimation. Assumption (A2) on the stepsize and the bandwidth was used in the recursive framework for the estimation of the density function in Mokkadem et al. (2009a), Slaoui (2014a) and for the distribution function estimation in Slaoui (2014b). Furthermore, it is notable that the assumption (A2)(iii) regarding the limit of $(n\gamma_n)$ as n goes to infinity is quite common within the context of stochastic approximation algorithms. More specifically, the limit $\xi := \lim_{n \to +\infty} (n\gamma_n)^{-1}$ is implied to be finite.

In what follows, we introduce two lemmas that will be widely invested throughout the theoretical studies of our recursive estimators. The proof of the first lemma was provided in Mokkadem et al. (2009a).

**Lemma 1.2.** Let $(v_n)_{n \ge 1} \in \mathcal{GS}(v^*), v^* \in \mathbb{R}$, $(\gamma_n)_{n \ge 1} \in \mathcal{GS}(-\alpha)$ and let $m > 0$ such that $m - v^*\xi > 0$. Therefore,

$$ \lim_{n \to +\infty} v_n \Pi_n^m \sum_{k=1}^n \Pi_k^{-m} \frac{\gamma_k}{v_k} = \frac{1}{m - v^* \xi}. $$

Moreover, for any positive sequence $(\alpha_n)_{n \ge 1}$ such that $\lim_{n \to +\infty} \alpha_n = 0$ and all $C \in \mathbb{R}$,

$$ \lim_{n \to +\infty} v_n \Pi_n^m \left[ \sum_{k=1}^n \Pi_k^{-m} \frac{\gamma_k}{v_k} \alpha_k + C \right] = 0. $$

The second lemma is a direct consequence of the definition 1.1 and is exhibited so as to help the reader simplify the assessment of proofs details.

**Lemma 1.3.** Let $(a_n)_{n \ge 1} \in \mathcal{GS}(a)$ and $(b_n)_{n \ge 1} \in \mathcal{GS}(b)$, $a, b \in \mathbb{R}$. Hence,

$$ \left( \frac{a_n^k}{b_n^z} \right)_{n \ge 1} \in \mathcal{GS}(ka - zb), \text{ with } k, z \in \mathbb{R}. $$

In particular, we get $(a_n^{-1})_{n \ge 1} \in \mathcal{GS}(-a)$.

### 1.1.2 Notations and assumptions

For our theoretical main results, we need the following assumptions:

(A3) (i) $F: \mathbb{R}^d \to [0, 1]$ is twice continuously differentiable.

(ii) For all $i, j \in \{1, \dots, d\}$, $F_i^{(1)} := \frac{\partial F}{\partial y_i}$ and $F_{ij}^{(2)} := \frac{\partial^2 F}{\partial y_i \partial y_j}$ are bounded.

(iii) For all $k \in \{1, \dots, d\}$, $\frac{\partial^{d-k+1} F}{\partial y_k \dots \partial y_d}$ exists and are continuous.

(iv) For all $i \in \{1, \dots, d\}$, there exists $f_i: \mathbb{R} \to \mathbb{R}_+$ bounded and integrable functions such that

$$ f(x_1, \dots, x_d) \le \prod_{i=1}^{d} f_i(x_i). $$
## 1.2 Main results

Our first result rests upon the following propositions highlighting respectively the bias and the variance of $F_n$. For the following, we note

$$ \text{For all } i, j \in \{1, \dots, d\}, \quad \mu_j(\mathbf{K}) := \int_{\mathbb{R}^d} z_j^2 \mathbf{K}(z) dz, \quad \phi_j(\mathbf{K}) := 2^d \int_{\mathbb{R}^d} z_j \mathbf{K}(z) \mathcal{K}(z) dz. $$

### 1.2.1 Bias and variance of $F_n$

**Proposition 1.4.** Under the assumptions $(A_1) - (A_3)$, we obtain

$$ \text{1. If } a \in (0, \frac{\alpha}{3}], \text{ then} \qquad \mathbb{E}[F_n(x)] - F(x) = \frac{h_n^2}{2(1-2a\xi)} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) + o(h_n^2). \quad (1.4) $$

$$ \text{If } a \in (\frac{\alpha}{3}, 1), \text{ then} \qquad \mathbb{E}[F_n(x)] - F(x) = o(\sqrt{\gamma_n h_n}). \quad (1.5) $$

$$ \text{2. If } a \in (0, \frac{\alpha}{4}), \text{ then} \qquad \mathrm{Var}[F_n(x)] = o(h_n^4). \quad (1.6) $$

$$ \text{If } a \in [\frac{\alpha}{4}, \frac{\alpha}{3}), \text{ then} \qquad \mathrm{Var}[F_n(x)] = \frac{\gamma_n}{2-\alpha\xi} F(x)(1-F(x)) + o(\gamma_n). \quad (1.7) $$

$$ \text{If } a \in [\frac{\alpha}{3}, 1), \text{ then} \qquad \mathrm{Var}[F_n(x)] = \frac{\gamma_n}{2-\alpha\xi} F(x)(1-F(x)) - \frac{\gamma_n h_n}{2-(a+\alpha)\xi} \sum_{i=1}^{d} \phi_i(\mathbf{K}) F_i^{(1)}(x) + o(\gamma_n h_n). \quad (1.8) $$

The bias and the variance of the estimator $F_n$ defined by the stochastic approximation algorithm (1.2) then largely depend on the choice of the stepsize $(\gamma_n)$.

At this level, let us display the following theorem which provides the weak pointwise convergence rate of the generalized recursive estimator $F_n$.

It is noteworthy that $\xrightarrow{n \to +\infty}^D$ denotes the convergence in distribution, $\mathcal{N}$ corresponds to the Gaussian distribution and $\xrightarrow{n \to +\infty}^P$ stands for the convergence in probability.

### 1.2.2 Weak pointwise convergence rate of $F_n$

**Theorem 1.5.** Under the assumptions $(A_1) - (A_3)$, we have:

$$ \text{1. If there exists } c \ge 0 \text{ such that } \gamma_n^{-1} h_n^4 \xrightarrow{n \to +\infty} c, \text{ then} \qquad \sqrt{\gamma_n^{-1}} (F_n(x) - F(x)) \xrightarrow{n \to +\infty}^D \mathcal{N} \left( \frac{\sqrt{c}}{2(1-2a\xi)} \sum_{j=1}^d \mu_j(\mathbf{K}) F_{jj}^{(2)}(x), \frac{1}{2-\alpha\xi} F(x)(1-F(x)) \right). $$

$$ \text{2. If } \gamma_n^{-1} h_n^4 \xrightarrow{n \to +\infty} +\infty, \text{ then} \qquad \frac{1}{h_n^2} (F_n(x) - F(x)) \xrightarrow{n \to +\infty}^P \frac{1}{2(1-2a\xi)} \sum_{j=1}^d \mu_j(\mathbf{K}) F_{jj}^{(2)}(x). $$
## 1.3 Optimal choice of the stepsizes

In order to measure the asymptotic quality of the recursive estimator $F_n$, we need to use the Mean Weighted Integrated Squared Error (MWISE).

### 1.3.1 Asymptotic expressions of MWISE[$F_n$]

The MWISE of the estimator $F_n$ is determined by

$$MWISE[F_n] = \int_{\mathbb{R}^d} (\mathbb{E}[F_n(x)] - F(x))^2 f(x)dx + \int_{\mathbb{R}^d} \mathrm{Var}[F_n(x)]f(x)dx. \quad (1.9)$$

We first note,

$$J_1 := \int_{\mathbb{R}^d} \sum_{i=1}^d F_i^{(1)}(x) \phi_i(\mathbf{K}) f(x) dx, \quad J_2 := \int_{\mathbb{R}^d} \left( \sum_{j=1}^d \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) \right)^2 f(x) dx, \\ V_F := \int_{\mathbb{R}^d} F(x)(1-F(x))f(x)dx.$$

**Proposition 1.6.** Under the assumptions $(A_1)-(A_3)$, we have

$$MWISE[F_n] = \begin{cases} \frac{J_2}{4(1-2a\xi)^2} h_n^4 + o(h_n^4) & \text{if } a \in (0, \frac{\alpha}{4}) \\ \frac{V_F}{2-\alpha\xi}\gamma_n + \frac{J_2}{4(1-2a\xi)^2}h_n^4 + o(h_n^4) & \text{if } a \in [\frac{\alpha}{4}, \frac{\alpha}{3}) \\ \frac{V_F}{2-\alpha\xi}\gamma_n - \frac{J_1}{2-(\alpha+a)\xi}\gamma_n h_n + \frac{J_2}{4(1-2a\xi)^2}h_n^4 + o(h_n^4) & \text{if } a = \frac{\alpha}{3} \\ \frac{V_F}{2-\alpha\xi}\gamma_n - \frac{J_1}{2-(\alpha+a)\xi}\gamma_n h_n + o(\gamma_n h_n) & \text{if } a \in (\frac{\alpha}{3}, 1). \end{cases}$$

*Proof.* By distinguishing the different possible cases according to the expressions of the Bias and Variance, one can prove this proposition and find the required result. □

The following corollary ensures that, for a special choice of the stepsize $(\gamma_n) = (\gamma_0 n^{-1})$, the optimal value for the bandwidth $(h_n)$ depends on $\gamma_0$ and as a matter of fact the corresponding MWISE depends also on $\gamma_0$.

**Corollary 1.7.** Let assumptions $(A_1)-(A_3)$ hold. To minimize the MWISE of $F_n$, we need to choose the stepsize $(\gamma_n)$ in $GS(-1)$ such that $\lim_{n \to +\infty} n\gamma_n = \gamma_0$, and the bandwidth $(h_n)$ needs to be equal to

$$\left(2^{-\frac{1}{3}}\left(\gamma_0 - \frac{2}{3}\right)^{\frac{1}{3}}\left(\frac{J_1}{J_2}\right)^{\frac{1}{3}}n^{-\frac{1}{3}}\right). \quad (1.10)$$

Therefore, the corresponding MWISE is specified by

$$MWISE[F_n] = \frac{\gamma_0^2}{2\gamma_0 - 1} n^{-1} V_F - \frac{3}{4} \frac{1}{2^{4/3}} \frac{\gamma_0^2}{(\gamma_0 - 2/3)^{2/3}} J_1^{\frac{4}{3}} J_2^{-\frac{1}{3}} n^{-\frac{4}{3}} + o\left(n^{-\frac{4}{3}}\right). \quad (1.11)$$

The following corollary follows immediately from theorem 1.5 and exhibits the asymptotic normality of $F_n$.
**Corollary 1.8.** Under the assumptions $(A_1)-(A_3)$, we have:

If there exists $s \ge 0$ such that $\gamma_n^{-1}h_n^3 \xrightarrow{n \to +\infty} s$, so that $\gamma_n^{-1}h_n^4 \xrightarrow{n \to +\infty} 0$, then

$$ \sqrt{\gamma_n^{-1}(F_n(x) - F(x))} \xrightarrow{n \to +\infty} \mathcal{N}\left(0, \frac{1}{2-\alpha\xi} F(x)(1-F(x))\right). $$

Now, let us examine the asymptotic properties of the generalized Nadaraya's non-recursive distribution estimator $\tilde{F}_n$.

## 1.4 Asymptotic properties of $\tilde{F}_n$

The main properties of the generalized non-recursive estimator $\tilde{F}_n$ are identified in the following proposition. (see Nadaraya (1964), Reiss (1981) and Hill (1985) for further details in the univariate case.)

**Proposition 1.9.** Let assumptions $(A_1)$ and $(A_3)$ hold. Hence, the asymptotic properties of Nadaraya's estimator are displayed as follows

* The bias of $\tilde{F}_n$:

$$ \mathbb{E}[\tilde{F}_n(x)] - F(x) = \frac{h_n^2}{2} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) + o(h_n^2). $$

* The variance of $\tilde{F}_n$:

$$ \mathrm{Var}[\tilde{F}_n(x)] = \frac{1}{n} F(x)(1-F(x)) - \frac{h_n}{n} \sum_{i=1}^{d} F_i^{(1)}(x) \phi_i(\mathbf{K}) + o(h_n n^{-1}). $$

* The asymptotic normality of $\tilde{F}_n$:

If we have $nh_n^4 \xrightarrow{n \to +\infty} 0$, then

$$ \sqrt{n}(\tilde{F}_n(x) - F(x)) \xrightarrow{n \to +\infty} \mathcal{N}\left(0, F(x)(1-F(x))\right). $$

It follows from this proposition that

$$ \mathrm{MWISE}[\tilde{F}_n] = \frac{h_n^4}{4} J_2 + o(h_n^4) + \frac{1}{n} V_F - \frac{h_n}{n} J_1 + o(h_n^4). $$

**Corollary 1.10.** Let assumptions $(A_1)$ and $(A_3)$ hold. To minimize the MWISE of $\tilde{F}_n$, the bandwidth $(h_n)$ must be equal to

$$ \left( \left( \frac{J_1}{J_2} \right)^{\frac{1}{3}} n^{-\frac{1}{3}} \right). \qquad (1.12) $$

Thus, the corresponding MWISE is expressed in terms of

$$ \mathrm{MWISE}[\tilde{F}_n] = n^{-1}V_F - \frac{3}{4}J_1^{\frac{4}{3}}J_2^{-\frac{1}{3}}n^{-\frac{4}{3}} + o\left(n^{-\frac{4}{3}}\right). \qquad (1.13) $$
**Remark 1.11.** In practice, the Robbins-Monro algorithm is very sensitive to parameters like the starting point or the calibration of the step sequence $(\gamma_n)$. The optimal speed of the algorithm is reached for a step of the form $\gamma_n = cn^{-\alpha}$ but the choice of the constants $c$ and $\alpha$ is extremely significant and determining it in practice proves to be very tricky.

The following theorem establishes that, for a special choice of the stepsize $(\gamma_n)$, the proposed recursive distribution estimator $F_n$ can dominate the generalized non-recursive Nadaraya's estimator $\tilde{F}_n$ in terms of the MWISE.

**Theorem 1.12.** Under assumptions $(A_1)-(A_3)$, and assuming that $(\gamma_n) = (\gamma_0 n^{-1})$ with $\gamma_0 = \frac{2}{3} + \varepsilon$ where $\varepsilon > 0$ (very close to zero), we consider the generalized estimators (1.2) with the bandwidth

$$ (h_n) = \left( 2^{-\frac{1}{3}} \left( \gamma_0 - \frac{2}{3} \right)^{\frac{1}{3}} \left( \frac{J_1}{J_2} \right)^{\frac{1}{3}} n^{-\frac{1}{3}} \right) $$

and the generalized Nadaraya's estimator (1.3) with the bandwidth

$$ (h_n) = \left( \left( \frac{J_1}{J_2} \right)^{\frac{1}{3}} n^{-\frac{1}{3}} \right). $$

For $n \in (n_{\varepsilon}^{\text{Lower}}, n_{\varepsilon}^{\text{Upper}})$, where

$$ n_{\varepsilon}^{\text{Lower}} = \frac{3^3 J_1^4}{2^{10} J_2 V_F^3} \frac{(\frac{1}{3} + 2\varepsilon)^3}{\varepsilon^2}, $$

and

$$ n_{\varepsilon}^{\text{Upper}} = \frac{3^3 J_1^4}{2^{10} J_2 V_F^3} \frac{(\frac{1}{3} + 2\varepsilon)^3 (\frac{2}{3} + \varepsilon - 2\frac{2}{3}\varepsilon^{\frac{1}{3}})^3 (\frac{2}{3} + \varepsilon + 2\frac{2}{3}\varepsilon^{\frac{1}{3}})^3}{(\varepsilon - \frac{1}{3})^6 \varepsilon^2}, $$

we have

$$ \text{MWISE}[F_n] < \text{MWISE}[\tilde{F}_n]. $$

**Remark 1.13.** Note that since the optimal MWISE formulas (1.11) and (1.13) must be positive, we need to opt for $n$ such that $n > n_{\varepsilon}^{\text{Lower}}$. In addition, in order to get a smaller MWISE of the proposed recursive estimator compared to non-recursive estimator, we need to choose $n < n_{\varepsilon}^{\text{Upper}}$; for supplementary details one can consult the proof given in Slaoui (2014b) in the case of unidimensionnel case, see also Jmaei et al. (2017) in the case of using the Bernstein polynomials rather than kernels. We can determine for each selected law and for a fixed sample size $n$ the upper and the lower bounds of the interval, in which the proposed recursive estimator can be better than the generalized classic Nadaraya's estimator.

## 1.5 Bandwidth selection

Kernel smoothing in non-parametric statistics requires the choice of a bandwidth parameter. This choice is critical and can substantially reduce precision. An appropriate bandwidth can help obtain an estimated distribution function close to the true distribution function. However, a poorly selected bandwidth can seriously distort the true underlying characteristics of the distribution function. Thus, wise choice of bandwidth is highly recommended. There are a myriad of methods for bandwidth selection. As far as our research is concerned, we are mainly interested in
two of them, namely the cross-validation procedure and the second generation plug-in approach. The cross-validation selection method was proposed by Sarda (1993), but this method has deficiencies as revealed by Altman and Leger (1995) who elaborated another efficient method, a plug-in approach which minimizes the estimate of the MWISE, using the density function as a weight function.

First of all, for the sake of simplicity, the kernel $\mathbf{K}$ is considered as a product of univariate kernels $\mathbf{K}$ satisfying

$$ \mathbf{K} = \otimes_{1}^{d} \mathbf{K} \quad \text{and} \quad \int_{\mathbb{R}} K(x)dx = 1. \qquad (1.14) $$

Hence, we introduce a function $\mathcal{K}$ defined by

$$ \mathcal{K}(x) = \int_{-\infty}^{x} K(t)dt \quad \text{such that} \quad \mathcal{K} = \otimes_{1}^{d} \mathcal{K}. $$

Beside, it is noteworthy that using the fact that $X$ is an $n \times d$ matrix, we will adopt the following notation in the whole thesis

$X_{ki} := X_{k,i}$, the entry in the k-th row and i-th column of the matrix $X$.

Hence, we infer that

$$ F_n(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \mathcal{K} \left( \frac{x - X_k}{h_k} \right) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \prod_{i=1}^{d} \mathcal{K} \left( \frac{x_i - X_{ki}}{h_k} \right) $$

and

$$ \tilde{F}_n(x) = \frac{1}{n} \sum_{k=1}^{n} \mathcal{K} \left( \frac{x - X_k}{h_n} \right) = \frac{1}{n} \sum_{k=1}^{n} \prod_{i=1}^{d} \mathcal{K} \left( \frac{x_i - X_{ki}}{h_n} \right). $$

Let us start by introducing the first bandwidth selection method.

### 1.5.1 Cross-Validation

The cross-validation procedure corresponds to a method of selecting the smoothing parameter. It's noteworthy that (see Sarda (1993)):

$$ CV(h) = \frac{1}{n} \sum_{i=1}^{n} (F_e - F_{-i})^2, $$

with $F_e$ is the empirical distribution function and $F_{-i}$ is the leave-one-out version of the considered estimator.

Thus, an estimator of the cross-validation criterion based on the generalized recursive estimator is identified by

$$ \widehat{CV}(h) = \frac{1}{n} \sum_{i=1}^{n} \left( F_e - \Pi_{n-1} \sum_{k=1, i \neq k}^{n} \Pi_k^{-1} \gamma_k \prod_{t=1}^{d} \mathcal{K} \left( \frac{X_{it} - X_{kt}}{h_k} \right) \right)^2, $$

and the considered cross-validation bandwidth selection adapted to our generalized recursive estimator is indicated by

$$ \hat{h}_{\text{opt}} = \arg \min_{h \in H} \widehat{CV}(h). $$
Likewise, an estimator of the cross-validation criterium based on the generalized Nadaraya's estimator is determined by

$$ \tilde{CV}(h) = \frac{1}{n} \sum_{i=1}^{n} \left( F_e - \frac{1}{n-1} \sum_{\substack{k=1 \\ i \neq k}}^{n} \prod_{t=1}^{d} \mathcal{H} \left( \frac{X_{it} - X_{kt}}{h_n} \right) \right)^2, $$

and the considered cross-validation bandwidth selection adapted to the generalized Nadaraya's estimator is defined by

$$ \tilde{h}_{\text{opt}} = \arg \min_{h \in H} \tilde{CV}(h). $$

In the next subsection, we tackle the second considered bandwidth selection method, which is called the second generation plug-in approach.

### 1.5.2 Plug-in method

In statistics, the plug-in principle stands for approximating a functional of a given population distribution by the same functional at the empirical distribution. Plug-in bandwidth selectors are a major class of bandwidth selectors which are derived from the MWISE expansion. Since the MWISE depends on the unknown quantities $J_1$ and $J_2$, we suggest constructing asymptotic unbiased estimators of $J_1$ and $J_2$.

Here, we notice $J_1 = \phi(K)I_1$ and $J_2 = \mu^2(K)I_2$, where

$$ \mu(K) = \int_{\mathbb{R}} z^2 K(z) dz, $$

$$ \phi(K) = 2 \int_{\mathbb{R}} z K(z) \mathcal{H}(z) dz, $$

$$ I_1 = \int_{\mathbb{R}^d} \sum_{i=1}^{d} F_i^{(1)}(x) f(x) dx \quad \text{and} \quad I_2 = \int_{\mathbb{R}^d} \left( \sum_{j=1}^{d} F_{jj}^{(2)}(x) \right)^2 f(x) dx. $$

In order to estimate the optimal bandwidth (1.10), we need to estimate $I_1$ and $I_2$, by using the Plug-in estimate approach of Altman and Leger (1995). From this perspective, we introduce $(b_n)_{n \ge 1} \in \mathcal{GS}(-\delta)$, $\delta \in (0, 1)$. In practice, we set

$$ b_n = n^{-\delta} \min \left\{ \hat{s}, \frac{Q_3 - Q_1}{1.349} \right\}, \quad (1.15) $$

with $\hat{s}$ being the sample standard deviation, and $Q_1, Q_3$ denoting the first and third quartiles of the sample $X$, respectively.

Moreover, we assume that $K_b$ stands for a kernel and $b_n$ is the associated bandwidth selected to be equal to (1.15) such that $\delta = \frac{2}{5}$, $K_{b'}^{(1)}$ denotes the first derivative of a kernel $K_{b'}$ with the associated bandwidth $b'_n$ such that $\delta = \frac{3}{10}$ and $\mathcal{H}_{b''}$ corresponds to the distribution function of a kernel $K_{b''}$ with the associated bandwidth $b''_n$ such that $\delta = \frac{1}{3}$.

For additional details concerning our choice for the parameter $\delta$, we recommend the reader to consult the work of Slaoui (2014a) for recursive kernel density estimation and Slaoui (2014b) for the distribution function estimation. In order to clarify this notion for the reader, we develop in the next subsection the proposed bandwidth selection based on the generalized recursive distribution estimator.
**Recursive estimator $F_n$:**

To estimate the optimal bandwidth (1.10), we need to estimate $I_1$ and $I_2$.

**Estimation of $I_1$:**

$$ \hat{I}_1 = \frac{\Pi_n}{n} \sum_{\substack{i,k=1 \\ i \neq j}}^n \Pi_k^{-1} \gamma_k b_k^{-1} \left[ \sum_{t=1}^d K_b \left( \frac{X_{it} - X_{kt}}{b_k} \right) \prod_{\substack{l=1 \\ l \neq t}}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_k''} \right) \right]. $$

**Estimation of $I_2$:**

$$ \hat{I}_2 = \frac{\Pi_n^2}{n} \sum_{\substack{i,j,k=1 \\ i \neq j \neq k}}^n \Pi_j^{-1} \Pi_k^{-1} \gamma_j \gamma_k b_j'^{-2} b_k'^{-2} \left[ \sum_{t=1}^d K_{b'}^{(1)} \left( \frac{X_{it} - X_{jt}}{b_j'} \right) \prod_{\substack{l=1 \\ l \neq t}}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{jl}}{b_j''} \right) \right] \\ \times \left[ \sum_{t=1}^d K_{b'}^{(1)} \left( \frac{X_{it} - X_{kt}}{b_k'} \right) \prod_{\substack{l=1 \\ l \neq t}}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_k''} \right) \right], $$

At this stage, we obtain

$$ \hat{J}_1 = \phi(K) \hat{I}_1 \text{ and } \hat{J}_2 = \mu^2(K) \hat{I}_2. $$

As a matter of fact, our considered plug-in bandwidth selection procedure to estimate (1.10) is specified by

$$ \left( 2^{-\frac{1}{3}} \left( \gamma_0 - \frac{2}{3} \right)^{\frac{1}{3}} \left\{ \frac{\hat{J}_1}{\hat{J}_2} \right\}^{\frac{1}{3}} n^{-\frac{1}{3}} \right), \quad (1.16) $$

**Estimation of $V_F$:**

Furthermore, in order to compute the MWISE of $F_n$, we need to estimate $V_F$ as follows

$$ \hat{V}_F = \frac{\Pi_n}{n} \sum_{\substack{i,k=1 \\ i \neq k}}^n \Pi_k^{-1} \gamma_k \left[ \prod_{l=1}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_k''} \right) \right] \\ - \frac{\Pi_n^2}{n} \sum_{\substack{i,j,k=1 \\ i \neq j \neq k}}^n \Pi_j^{-1} \Pi_k^{-1} \gamma_j \gamma_k \left[ \prod_{l=1}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{jl}}{b_j''} \right) \right] \times \left[ \prod_{l=1}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_k''} \right) \right]. $$

Therefore, the plug-in estimator of MWISE[$F_n$] is provided by

$$ \widehat{MWISE}[F_n] = \frac{\gamma_0^2}{2\gamma_0 - 1} n^{-1} \hat{V}_F - \frac{3}{4} \frac{1}{2^{4/3}} \frac{\gamma_0^2}{(\gamma_0 - 2/3)^{2/3}} \hat{J}_1^{\frac{4}{3}} \hat{J}_2^{-\frac{1}{3}} n^{-\frac{4}{3}} + o\left(n^{-\frac{4}{3}}\right). \quad (1.17) $$

In the next subsection, we handle the generalized Nadaraya's distribution estimator in order to be able to compare our generalized recursive distribution estimator to the non-recursive one.

**Non-recursive estimator $\tilde{F}_n$:**

To estimate the optimal bandwidth (1.12), we need to estimate $I_1$ and $I_2$.

**Estimation of $I_1$:**

$$ \tilde{I}_1 = \frac{1}{n(n-1)b_n} \sum_{\substack{i,k=1 \\ i \neq k}}^n \left[ \sum_{t=1}^d K_b \left( \frac{X_{it} - X_{kt}}{b_n} \right) \prod_{\substack{l=1 \\ l \neq t}}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_n''} \right) \right], $$
**Estimation of $I_2$:**

$$ \tilde{I}_2 = \frac{1}{n^3 b_n^4} \sum_{\substack{i,j,k=1 \\ i \neq j \neq k}}^n \left[ \sum_{t=1}^d K_{b'}^{(1)} \left( \frac{X_{it} - X_{jt}}{b_n'} \right) \prod_{\substack{l=1 \\ l \neq t}}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{jl}}{b_n''} \right) \right] \times \left[ \sum_{t=1}^d K_{b'}^{(1)} \left( \frac{X_{it} - X_{kt}}{b_n'} \right) \prod_{\substack{l=1 \\ l \neq t}}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_n''} \right) \right], $$

At this stage, we obtain

$$ \tilde{J}_1 = \phi(K) \tilde{I}_1 \text{ and } \tilde{J}_2 = \mu^2(K) \tilde{I}_2. $$

Hence, the considered plug-in bandwidth selection estimator of (1.12) is denoted by

$$ \left( \left\{ \frac{\tilde{J}_1}{\tilde{J}_2} \right\}^{\frac{1}{3}} n^{-\frac{1}{3}} \right), \qquad (1.18) $$

**Estimation of $V_F$:**

Furthermore, in order to compute the associated MWISE of $\tilde{F}_n$ using the previous plug-in bandwidth estimator of (1.18), we need to estimate $V_F$ as follows

$$ \tilde{V}_F = \frac{1}{n(n-1)} \sum_{\substack{i,k=1 \\ i \neq k}}^n \left[ \prod_{l=1}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_n''} \right) \right] \\ - \frac{1}{n(n-1)^2} \sum_{\substack{i,j,k=1 \\ i \neq j \neq k}}^n \left[ \prod_{l=1}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{jl}}{b_n''} \right) \right] \times \left[ \prod_{l=1}^d \mathcal{H}_{b''} \left( \frac{X_{il} - X_{kl}}{b_n''} \right) \right]. $$

Thus, the associated plug-in estimator of MWISE[$\tilde{F}_n$] is expressed by

$$ \widetilde{MWISE}[\tilde{F}_n] = n^{-1} \tilde{V}_F - \frac{3}{4} \tilde{J}_1^{\frac{4}{3}} \tilde{J}_2^{-\frac{1}{3}} n^{-\frac{4}{3}} + o(n^{-\frac{4}{3}}). \qquad (1.19) $$

The equations incorporated in (1.17) and (1.19) will be very useful in the next section at the level of comparing the performance of both estimators in terms of estimation error.

# 1.6 Numerical applications

The target underlying our numerical studies lies in comparing the performance of our generalized recursive estimator (1.2) with that of Nadaraya (1.3).

## 1.6.1 Simulation studies

When applying our generalized recursive estimator $F_n$, we need to choose three quantities:

* The function $K$, we use the Epanechnikov kernel.

* The stepsize ($\gamma_n$) = ($\gamma_0 n^{-1}$), where $\gamma_0 = 2/3 + \varepsilon$.

* The bandwidth ($h_n$) is chosen to be equal to (1.16).
When applying the non-recursive estimator $\tilde{F}_n$, we need to choose two quantities:

- The function $K$, we use the Epanechnikov kernel.

- The bandwidth ($h_n$) is chosen to be equal to (1.18).

We denote by $F_i^*$ the reference distribution and by $F_i$ the test distribution. Therefore, we calculate the following measures:

- Mean squared error: $MSE = \frac{1}{n} \sum_{i=1}^{n} (F_i - F_i^*)^2$.

- The linear correlation: $Cor = Cov(F_i, F_i^*)\sigma(F_i)^{-1}\sigma(F_i^*)^{-1}$.

Here are certain properties of the used Epanechnikov kernel:

<table><caption>Table 1.1: Epanechnikov properties.</caption><thead><tr><th>Kernel</th><th>K(x)</th><th>Support</th><th>φ(K)</th><th>μ(K)</th></tr></thead><tbody><tr><td>Epanechnikov</td><td><sup>3</sup>⁄<sub>4</sub>(1 - x<sup>2</sup>)</td><td>[-1, 1]</td><td><sup>9</sup>⁄<sub>35</sub></td><td><sup>1</sup>⁄<sub>5</sub></td></tr></tbody></table>

### The unidimensional case: d=1

In order to compare the considered generalized recursive estimator and the Nadaraya's generalized distribution one, we consider two sample sizes; $n = 50$ and $100$ using $N = 500$ trials of sample $n$ and the following four models:

- Model 1: The standard normal distribution $N(0, 1)$.

- Model 2: The normal distribution $N(\frac{1}{2}, 1)$.

- Model 3: The mixture normal distribution $\frac{1}{2}N(\frac{1}{2}, 1) + \frac{1}{2}N(-\frac{1}{2}, 1)$.

- Model 4: The mixture normal distribution $\frac{2}{3}N(\frac{1}{3}, 1) + \frac{1}{3}N(-\frac{1}{3}, 1)$.

<figure><img src="image_5.png" /><figcaption>Figure 1.1: Qualitative comparison between Nadaraya's distribution estimator and the generalized recursive estimator for Model 1 with $n=50$ and $(\gamma_n) = ((2/3 + 0.05)n^{-1})$.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Figure 1.2: Qualitative comparison between Nadaraya’s distribution estimator with the generalized recursive estimator using Model 1, n=100 and $(\gamma_n) = ((2/3 + 0.05)n^{-1})$.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 1.3: Qualitative comparison between Nadaraya’s distribution estimator with the generalized recursive estimator using Model 4, n=50 and $(\gamma_n) = ((2/3 + 0.05)n^{-1})$.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 1.4: Qualitative comparison between Nadaraya’s distribution estimator with the generalized recursive estimator using Model 4, n=100 and $(\gamma_n) = ((2/3 + 0.05)n^{-1})$.</figcaption></figure>
<table><caption>Table 1.2: Quantitative comparison between Nadaraya's distribution estimator and the generalized recursive distribution estimator with the stepsize ($\gamma_n$) = ([2/3+0.05]$n^{-1}$) through a plug-in method as well as a cross-validation one in the unidimensional case.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">MSE/Cor</th><th rowspan="2">n</th><th colspan="2">Plug-in</th><th colspan="2">Cross-Validation</th></tr><tr><th>Nadaraya's estimator</th><th>Recursive estimator<br>γ<sub>n</sub> = (2/3 + 0.05)n<sup>-1</sup></th><th>Nadaraya's estimator</th><th>Recursive estimator<br>γ<sub>n</sub> = (2/3 + 0.05)n<sup>-1</sup></th></tr></thead><tbody><tr><td rowspan="4">Model 1</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0008760976</td><td><b>0.0006684492</b></td><td>0.0008719240</td><td>0.0007052983</td></tr><tr><td>100</td><td>0.0003355597</td><td><b>0.0002681363</b></td><td>0.0003507760</td><td>0.0002798545</td></tr><tr><td>50</td><td>0.9991972854</td><td><b>0.9993061033</b></td><td>0.9985833690</td><td>0.9992424589</td></tr><tr><td>100</td><td>0.9996993596</td><td><b>0.9997911018</b></td><td>0.9996216460</td><td>0.9997260820</td></tr><tr><td rowspan="4">Model 2</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0003243463</td><td><b>0.0002961442</b></td><td>0.0003424373</td><td>0.0003047189</td></tr><tr><td>100</td><td>0.0002010728</td><td><b>0.0001532752</b></td><td>0.0001743897</td><td>0.0001540263</td></tr><tr><td>50</td><td>0.9997654430</td><td><b>0.9996854602</b></td><td>0.9996520644</td><td>0.9996463968</td></tr><tr><td>100</td><td>0.9998041657</td><td><b>0.9998307559</b></td><td>0.9997582078</td><td>0.9997787056</td></tr><tr><td rowspan="4">Model 3</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0006376542</td><td><b>0.0005857468</b></td><td>0.0006901210</td><td>0.0006039373</td></tr><tr><td>100</td><td>0.0005868853</td><td><b>0.0004730782</b></td><td>0.0006321977</td><td>0.0004962125</td></tr><tr><td>50</td><td>0.9989832200</td><td><b>0.9988033588</b></td><td>0.9987992520</td><td>0.9987239326</td></tr><tr><td>100</td><td>0.9990871155</td><td><b>0.9990431684</b></td><td>0.9989331049</td><td>0.9989420103</td></tr><tr><td rowspan="4">Model 4</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0005038682</td><td><b>0.0004442070</b></td><td>0.0006349498</td><td>0.0004793215</td></tr><tr><td>100</td><td>0.0000701021</td><td><b>0.0000310626</b></td><td>0.0001136351</td><td>0.0000382802</td></tr><tr><td>50</td><td>0.9993125926</td><td><b>0.9989769800</b></td><td>0.9988661040</td><td>0.9988299239</td></tr><tr><td>100</td><td>0.9999081901</td><td><b>0.9999493515</b></td><td>0.9998286569</td><td>0.9999350136</td></tr></tbody></table>

For the following, it is worthy to denote that $I_d$ stands for the identity matrix of size $d$.

### The bidimensional case: $d=2$

In order to compare the generalized recursive distribution estimator with Nadaraya's distribution estimator, we consider two sample sizes, $n = 50$ and $100$ using $N = 500$ trials of sample $n$ and four models:

* Model 1: The standard normal distribution $N((0), I_2)$.

* Model 2: The normal distribution $N((1/2), I_2)$.

* Model 3: The mixture normal distribution $\frac{1}{2}N((1/2), I_2) + \frac{1}{2}N((-1/2), I_2)$.

* Model 4: The mixture normal distribution $\frac{2}{3}N((1/3), I_2) + \frac{1}{3}N((-1/3), I_2)$.
<figure><img src="image_9.png" /><figcaption>Figure 1.5: The reference distribution function $F_s$ using Model 1.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 1.7: Nadaraya's estimator $\tilde{F}$ using Model 1 with $n=50$.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 1.9: The recursive estimator $F_n$ using Model 1 with $n=50$.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Figure 1.6: The reference distribution function $F_s$ using Model 3.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 1.8: Nadaraya's estimator $\tilde{F}$ using Model 3 with $n=50$.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 1.10: The recursive estimator $F_n$ using Model 3 with $n=50$.</figcaption></figure>
<table><caption>Table 1.3: Quantitative comparison between Nadaraya's distribution estimator and the proposed distribution estimator with stepsize ($\gamma_n$) = ([2/3 + 0.05]n⁻¹) through a plug-in method as well as a cross-validation one in the bidimensional case.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2">MSE/Cor</th><th rowspan="2">n</th><th colspan="2">Plug-in</th><th colspan="2">Cross-Validation</th></tr><tr><th>Nadaraya's estimator</th><th>Recursive estimator<br>γ<sub>n</sub> = (2/3 + 0.05)n<sup>-1</sup></th><th>Nadaraya's estimator</th><th>Recursive estimator<br>γ<sub>n</sub> = (2/3 + 0.05)n<sup>-1</sup></th></tr></thead><tbody><tr><td rowspan="4">Model 1</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0003157969</td><td><strong>0.0002118529</strong></td><td>0.0003309580</td><td>0.0003047189</td></tr><tr><td>100</td><td>0.0000876097</td><td><strong>0.0000668449</strong></td><td>0.0000871924</td><td>0.0000705320</td></tr><tr><td>50</td><td>0.9994881093</td><td><strong>0.9992621212</strong></td><td>0.9989031200</td><td>0.9989383580</td></tr><tr><td>100</td><td>0.9999197285</td><td><strong>0.9999306103</strong></td><td>0.9998583369</td><td>0.9999242405</td></tr><tr><td rowspan="4">Model 2</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0008261247</td><td><strong>0.0006548646</strong></td><td>0.0008385428</td><td>0.0006632885</td></tr><tr><td>100</td><td>0.0000908300</td><td><strong>0.0000600300</strong></td><td>0.0000977200</td><td>0.0000713295</td></tr><tr><td>50</td><td>0.9993041754</td><td><strong>0.9996147794</strong></td><td>0.9993122882</td><td>0.9996248268</td></tr><tr><td>100</td><td>0.9998238000</td><td><strong>0.9998288000</strong></td><td>0.9998037000</td><td>0.9998278900</td></tr><tr><td rowspan="4">Model 3</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0008668519</td><td><strong>0.0008134314</strong></td><td>0.0010215940</td><td>0.0009445246</td></tr><tr><td>100</td><td>0.0005842987</td><td><strong>0.0005468488</strong></td><td>0.0007121525</td><td>0.0006637816</td></tr><tr><td>50</td><td>0.9976693040</td><td><strong>0.9977774985</strong></td><td>0.9967051420</td><td>0.9974661303</td></tr><tr><td>100</td><td>0.9986429796</td><td><strong>0.9987608168</strong></td><td>0.9982977372</td><td>0.9983191907</td></tr><tr><td rowspan="4">Model 4</td><td rowspan="4">MSE<br>Cor</td><td>50</td><td>0.0006459259</td><td><strong>0.0004847252</strong></td><td>0.0006801415</td><td>0.0005460630</td></tr><tr><td>100</td><td>0.0002643474</td><td><strong>0.0001391675</strong></td><td>0.0002736224</td><td>0.0001653388</td></tr><tr><td>50</td><td>0.9986615779</td><td><strong>0.9989035867</strong></td><td>0.9985514195</td><td>0.9986705950</td></tr><tr><td>100</td><td>0.9997714558</td><td><strong>0.9998242084</strong></td><td>0.9997033162</td><td>0.9998010731</td></tr></tbody></table>

From Tables 1.2 and 1.3, we conclude that:

1. The MSE of the generalized recursive estimator with stepsize ($\gamma_n$) = ([2/3 + 0.05]n⁻¹) through a plug-in method is smaller than that of the generalized Nadaraya's estimator with the second generation considered plug-in method as well as the generalized recursive estimator and the generalized Nadaraya's estimator with the cross-validation approach.

2. The MSE decreases as the simple size increases.

3. The Cor increases as the sample size increases.

## 1.6.2 Real Datasets

In this section, we exhibit three real data applications. The first one is in an unidimensional case, the second one is in a bidimensional case while the third one is in a multidimensional case. For all of the considered cases, we report the values of the plug-in $I_1$, $I_2$, $V_F$ and MWISE using the generalized Nadaraya's estimator as well as the generalized recursive estimator using a specific choice of the stepsize. Moreover, we calculate the error defined by

$$PSE = \frac{1}{n} \sum_{i=1}^{n} (F_i - F_i^e)^2,$$

where $F_i^e$ is the empirical distribution function and $F_i$ is the test distribution function.

### An unidimensional application

At this level, we use lh data which appears in the R package datasets. These data are a sort of a regular time series giving the luteinizing hormone in blood samples at 10 mins intervals from a human female. Totally, we got 48 samples.
<table><caption>Table 1.4: Quantitative comparison between the $I_1$, $I_2$, $V_F$, MWISE and PSE of Nadaraya's distribution estimator as well as the proposed distribution estimator with stepsize ($\gamma_n$) = ([2/3 + 0.05]n⁻¹) via lh data of the package datasets and through a plug-in method.</caption><thead><tr><th></th><th><i>I</i><sub>1</sub></th><th><i>I</i><sub>2</sub></th><th><i>V</i><sub>F</sub></th><th><i>MWISE</i></th><th><i>PSE</i></th></tr></thead><tbody><tr><td>Nadaraya's estimator</td><td>0.42214350</td><td>0.64455170</td><td>0.16382990</td><td>0.00265951</td><td>0.00048493</td></tr><tr><td>Recursive estimator</td><td><b>0.45531620</b></td><td><b>0.33022020</b></td><td><b>0.16532920</b></td><td><b>0.00251792</b></td><td><b>0.00043248</b></td></tr></tbody></table>

<figure><img src="image_15.png" /><figcaption>Figure 1.11: Qualitative comparison between Nadaraya's distribution estimator and the proposed distribution estimator with stepsize ($\gamma_n$) = ([2/3 + 0.05]n⁻¹) via lh data of the package datasets and through a plug-in method.</figcaption></figure>

### A bidimensional application

Within this framework, we use the `nwip` data which appears in the R package `kerdiest`. This data set corresponds to the earthquakes occurring in the Northwest of the Iberian Peninsula, from 25/November/1924 to 31/July/2010. The area is limited by the coordinates 41 N - 44 N and 6 W - 10 W, involving the autonomic region of Galicia (Spain) and northern Portugal. The data catalog was obtained from the National Geographic Institute (IGN) of Spain. These data are available online at the web page http://www.ign.es. Within this data frame, we have 3491 observations on 10 variables, corresponding to the earthquake epicenters and time of occurrence. Within this application, we are interested in two variables, namely the longitude in degrees as well as the magnitude in Richter Scale. For the latter, we consider the date with the magnitude values > 3.1, to end up with a sample of 326 observations.

<table><caption>Table 1.5: Quantitative comparison between the $I_1$, $I_2$, $V_F$, MWISE and PSE of Nadaraya's distribution estimator as well as the proposed distribution estimator with stepsize ($\gamma_n$) = ([2/3 + 0.05]n⁻¹) via nwip data of the package `kerdiest` and through a plug-in method.</caption><thead><tr><th></th><th><i>I</i><sub>1</sub></th><th><i>I</i><sub>2</sub></th><th><i>V</i><sub>F</sub></th><th><i>MWISE</i></th><th><i>PSE</i></th></tr></thead><tbody><tr><td>Nadaraya's estimator</td><td>0.44454490</td><td>0.09554306</td><td>0.16091480</td><td>0.00037498</td><td>0.00170011</td></tr><tr><td>Recursive estimator</td><td><b>0.33153120</b></td><td><b>0.01013669</b></td><td><b>0.16707990</b></td><td><b>0.00035297</b></td><td><b>0.00089828</b></td></tr></tbody></table>
<figure><img src="image_16.png" /><figcaption>Figure 1.12: The empirical distribution function.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 1.13: Nadaraya’s estimator.</figcaption></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /><figcaption>Figure 1.14: The recursive estimator.</figcaption></figure>

<figure><img src="image_20.png" /></figure>

## A multidimensional application

In this application, we use the *iris* data which appears in the R package *datasets*. This famous (Fisher’s or Anderson’s) iris data set provides the measurements in centimeters of the variables sepal length and width and petal length and width, respectively, for 50 flowers from each of 3 species of iris. These species are *Iris setosa*, *versicolor*, and *virginica*. As far as this application is concerned, we consider data with the petal width values < 2 to end up with a sample of 121 observations. Therefore, the following results are obtained.

<table><caption>Table 1.6: Quantitative comparison between the $I_1$, $I_2$, $V_F$, MWISE and PSE of Nadaraya’s distribution estimator as well as the proposed distribution estimator with stepsize ($\gamma_n$) = ([2/3 + 0.05]$n^{-1}$) via iris data of the package datasets and through a plug-in method.</caption><thead><tr><th></th><th><i>I</i><sub>1</sub></th><th><i>I</i><sub>2</sub></th><th><i>V</i><sub>F</sub></th><th><i>MWISE</i></th><th><i>PSE</i></th></tr></thead><tbody><tr><td>Nadaraya's estimator</td><td>0.44976830</td><td>0.22750983</td><td>0.08894939</td><td>0.00039688</td><td>0.00086480</td></tr><tr><td>Recursive estimator</td><td><b>0.39024930</b></td><td><b>0.07616996</b></td><td><b>0.09344776</b></td><td><b>0.00030997</b></td><td><b>0.00075062</b></td></tr></tbody></table>

## 1.7 Conclusion

In this research work, we set forward a smooth estimator of the multivariate distribution function. We first studied the asymptotic properties of the generalized estimator. We computed the bias as well as the variance in order to demonstrate that our estimator asymptotically follows a normal distribution. Afterwards, we compared our generalized recursive estimator to non-recursive Nadaraya’s multivariate distribution estimator using two bandwidth selection approaches, namely the cross-validation method as well as the plug-in technique. In the simulation studies, and for all the cases, the generalized recursive estimator (1.2) with stepsize
$(\gamma_n) = ([2/3 + 0.05]n^{-1})$ ensures a better performance in terms of estimation error compared to the generalized non-recursive Nadaraya's estimator.

To sum up, the use of the generalized recursive distribution estimators enables us to obtain better results compared to the non-recursive estimator. With an appropriate choice of the bandwidth, we can demonstrate that our generalized recursive estimator is closer to the true distribution function than the generalized non-recursive Nadaraya's estimator.

## 1.8 Proofs

Throughout this section, devoted to the proofs of our main results, we use the following notation:

$$ \mathcal{Z}_n(x) = \mathcal{K} \left( \frac{x - X_n}{h_n} \right). \qquad (1.20) $$

*Proof of Proposition 1.4.*

Let $x$ be in $\mathbb{R}^d$.

In the general framework, using the relation (1.1) and without assuming that $F_0(x) = 0$, we have:

$$ \begin{aligned} F_n(x) - F(x) &= (1 - \gamma_n)F_{n-1}(x) + \gamma_n \mathcal{Z}_n(x) - F(x) \\ &= (1 - \gamma_n)[F_{n-1}(x) - F(x)] + \gamma_n[\mathcal{Z}_n(x) - F(x)]. \end{aligned} $$

By a simple recurrence, we obtain

$$ \begin{aligned} F_n(x) - F(x) &= \prod_{i=1}^{n} (1 - \gamma_i) [F_0(x) - F(x)] + \sum_{k=1}^{n-1} \sum_{i=k+1}^{n} (1 - \gamma_i) \gamma_k (\mathcal{Z}_k(x) - F(x)) + \gamma_n (\mathcal{Z}_n(x) - F(x)) \\ &= \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k (\mathcal{Z}_k(x) - F(x)) + \Pi_n [F_0(x) - F(x)]. \end{aligned} $$

Thus,

$$ \mathbb{E}[F_n(x)] - F(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k (\mathbb{E}[\mathcal{Z}_k(x)] - F(x)) + \Pi_n [F_0(x) - F(x)]. \quad (1.21) $$

### 1. Bias of $F_n$:

Our intrinsic goal at this level is to calculate the quantity:

$$ \mathbb{E}[\mathcal{Z}_k(x)] = \int_{\mathbb{R}^d} \mathcal{K} \left( \frac{x - y}{h_k} \right) f(y) dy. $$

Notably, under the conditions in (A$_1$), we assume that

$$ f(y) = \frac{\partial^d F}{\partial y_1 \dots \partial y_d}(y), \quad y = (y_1, \dots, y_d). $$

Correspondingly, under (A$_1$) and (1.14), we infer that

$$ \mathbf{K}(y) = \frac{\partial^d \mathcal{K}}{\partial y_1 \dots \partial y_d}(y) = \frac{\partial^d \mathcal{K}}{\partial y_d \dots \partial y_1}(y). $$

At this level, we aim to apply the following result, an extension of the integration by parts formula.
Under the assumptions (A₁), (A₃) and (1.14), we infer that

$$
\begin{align*}
\int_{\mathbb{R}^d} \mathcal{K} \left( \frac{x-y}{h_k} \right) f(y) dy &= \int_{\mathbb{R}^d} \mathcal{K} \left( \frac{x-y}{h_k} \right) \frac{\partial^d F}{\partial y_1 \dots \partial y_d} (y) dy \\
&= (-1)^d \int_{\mathbb{R}^d} \left( \frac{-1}{h_k} \right)^d \frac{\partial^d \mathcal{K}}{\partial y_1 \dots \partial y_d} \left( \frac{x-y}{h_k} \right) F(y) dy \\
&= \int_{\mathbb{R}^d} h_k^{-d} \mathbf{K} \left( \frac{x-y}{h_k} \right) F(y) dy.
\end{align*}
$$

Hence, a change of variables ensures

$$
\mathbb{E}[\mathcal{Z}_k(x)] = \int_{\mathbb{R}^d} \mathbf{K}(z) F(x - zh_k) dz, \quad z = (z_1, \dots, z_d).
$$

Moreover, using (A₃) and by applying the well known Taylor's development with integral remainder formula for F, we obtain

$$
F(x - zh_k) = F(x) - \sum_{i=1}^{d} F_i^{(1)}(x)z_i h_k + \int_{0}^{1} (1-t) \sum_{i,j=1}^{d} F_{ij}^{(2)}(x - tzh_k)z_i z_j h_k^2 dt.
$$

Then, it will be obvious that

$$
\mathbb{E}[\mathcal{Z}_k(x)] - F(x) = \int_{\mathbb{R}^d} \mathbf{K}(z) [F(x - zh_k) - F(x)] dz.
$$

Hence,

$$
\begin{align*}
\mathbb{E}[\mathcal{Z}_k(x)] - F(x) &= \int_{\mathbb{R}^d} \mathbf{K}(z) \left[ -\sum_{i=1}^d F_i^{(1)}(x)z_i h_k + \int_0^1 (1-t) \sum_{i,j=1}^d F_{ij}^{(2)}(x-tzh_k)z_i z_j h_k^2 dt \right] dz \\
&= -h_k \sum_{i=1}^d F_i^{(1)}(x) \int_{\mathbb{R}^d} \mathbf{K}(z) z_i dz + \frac{h_k^2}{2} \sum_{i,j=1}^d F_{ij}^{(2)}(x) \int_{\mathbb{R}^d} \mathbf{K}(z) z_i z_j dz \\
&\quad + \int_{\mathbb{R}^d} \mathbf{K}(z) \int_0^1 (1-t) \sum_{i,j=1}^d \left[ F_{ij}^{(2)}(x-tzh_k) - F_{ij}^{(2)}(x) \right] z_i z_j h_k^2 dt dz \\
&= \frac{h_k^2}{2} \sum_{j=1}^d \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) + h_k^2 \eta_k(x),
\end{align*}
$$

where

$$
\mu_j(\mathbf{K}) = \int_{\mathbb{R}^d} z_j^2 \mathbf{K}(z) dz \quad \text{and} \quad \eta_k(x) = \sum_{i,j=1}^{d} \int_{\mathbb{R}^d} \int_0^1 (1-t) [F_{ij}^{(2)}(x - tzh_k) - F_{ij}^{(2)}(x)] z_i z_j \mathbf{K}(z) dt dz.
$$

Therefore, thanks to the relation (1.21), we have

$$
\begin{align*}
\mathbb{E}[F_n(x)] - F(x) &= \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \left( \frac{h_k^2}{2} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) + h_k^2 \eta_k(x) \right) + \Pi_n [F_0(x) - F(x)] \\
&= \frac{1}{2} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 + \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 \eta_k(x) + \Pi_n [F_0(x) - F(x)] \\
&= S_{1,n}(x) + S_{2,n}(x) + \Pi_n [F_0(x) - F(x)]
\end{align*}
$$

where,

$$
S_{1,n}(x) = \frac{1}{2} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 \quad \text{and} \quad S_{2,n}(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 \eta_k(x).
$$
Here, we distinguish two cases:

1. For the case $a \le \alpha/3$, we have $\lim_{n \to +\infty} n\gamma_n > \min\left\{\frac{a+\alpha}{2}, 2a\right\} = 2a$.

**Asymptotic behaviour of $S_{1,n}(x)$:**

Here we opt to verify the conditions of lemma 1.2. We have $(v_n) := (h_n^{-2}) \in \mathcal{GS}(2a)$ and $m=1$, since $\xi^{-1} > 2a$, then $1-2a\xi > 0$. The application of lemma 1.2 ensures that

$$ \lim_{n \to +\infty} h_n^{-2} \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 = \frac{1}{1-2a\xi}. $$

Hence,

$$ S_{1,n}(x) = \frac{h_n^2}{2(1-2a\xi)} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) + o(h_n^2). $$

**Asymptotic behaviour of $S_{2,n}(x)$:**

Owing to the fact that $F_{ij}^{(2)}$ is bounded and continuous at $x$ for all $i, j \in \{1, \dots, d\}$, Lebesgue's convergence theorem ensures that $\lim_{k \to +\infty} \eta_k(x) = 0$, which ensures that, $\eta_k(x) = o(1)$.

Therefore, the second part of lemma 1.2 ensures that

$$ \lim_{n \to +\infty} h_n^{-2} \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 \eta_k = 0. $$

Thus,

$$ S_{2,n}(x) = o(h_n^2). $$

As a matter of fact, we infer that,

$$ \mathbb{E}[F_n(x)] - F(x) = \frac{h_n^2}{2(1-2a\xi)} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) + o(h_n^2). $$

2. For the case $a > \alpha/3$, we have $h_n^2 = o(\sqrt{\gamma_n h_n})$. Consequently, we obtain

$$ \begin{aligned} \mathbb{E}[F_n(x)] - F(x) &= \frac{1}{2} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 + \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^2 o(1) + \Pi_n [F_0(x) - F(x)] \\ &= \frac{1}{2} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k o(\sqrt{\gamma_k h_k}) + \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k o(\sqrt{\gamma_k h_k}) + O(\Pi_n) \\ &= o(\sqrt{\gamma_n h_n}). \end{aligned} $$

As a result, we get

$$ \mathbb{E}[F_n(x)] - F(x) = o(\sqrt{\gamma_n h_n}). $$

**Variance of $F_n$:**

In view of the independence of $X_i$'s, for $i = 1, \dots, n$, we deduce that

$$ \begin{aligned} \mathrm{Var}[F_n(x)] &= \mathrm{Var}[\Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \mathcal{Z}_k(x)] \\ &= \Pi_n^2 \sum_{k=1}^{n} \Pi_k^{-2} \gamma_k^2 \mathrm{Var}[\mathcal{Z}_k(x)] \\ &= \Pi_n^2 \sum_{k=1}^{n} \Pi_k^{-2} \gamma_k^2 (\mathbb{E}[\mathcal{Z}_k^2(x)] - \mathbb{E}[\mathcal{Z}_k(x)]^2). \end{aligned} $$
Hence, following (1.14) and owing to the fact that $2^d \int_{\mathbb{R}^d} \mathbf{K}(z)\mathcal{K}(z)dz = 1$, we infer that

$$ \mathcal{K}^2(y_1, \dots, y_d) = \prod_{i=1}^{d} \mathcal{K}^2(y_i) \quad \text{and} \quad \frac{\partial^d}{\partial y_1 \dots \partial y_d} (\mathcal{K}^2(y)) = \frac{\partial^d}{\partial y_d \dots \partial y_1} (\mathcal{K}^2(y)) = 2^d \mathbf{K}(y)\mathcal{K}(y). $$

Then, we get

$$ \begin{aligned} \mathbb{E}[\mathcal{Z}_k^2(x)] &= \int_{\mathbb{R}^d} \mathcal{K}^2 \left( \frac{x-y}{h_k} \right) f(y) dy \\ &= \int_{\mathbb{R}^d} 2^d \mathcal{K} \left( \frac{x-y}{h_k} \right) \mathbf{K} \left( \frac{x-y}{h_k} \right) h_k^{-d} F(y) dy \\ &= 2^d \int_{\mathbb{R}^d} \mathbf{K}(z) \mathcal{K}(z) F(x-zh_k) dz. \end{aligned} $$

From the application of taylor's theorem with integral remainder, we obtain

$$ F(x - zh_k) = F(x) - \int_{0}^{1} \sum_{i=1}^{d} F_{i}^{(1)}(x - tzh_k)z_i h_k dt. $$

Consequently, we find

$$ \begin{aligned} \mathbb{E}[\mathcal{Z}_k^2(x)] &= 2^d \int_{\mathbb{R}^d} \mathbf{K}(z)\mathcal{K}(z) \left[ F(x) - \int_0^1 \sum_{i=1}^d F_i^{(1)}(x - tzh_k)z_i h_k dt \right] dz \\ &= F(x) - h_k \sum_{i=1}^d \phi_i(\mathbf{K}) F_i^{(1)}(x) - \nu_k(x), \end{aligned} $$

with

$$ \phi_i(\mathbf{K}) = 2^d \int_{\mathbb{R}^d} z_i \mathbf{K}(z) \mathcal{K}(z) dz \quad \text{and} \quad \nu_k(x) = 2^d \sum_{i=1}^d \int_{\mathbb{R}^d} \int_0^1 \mathbf{K}(z) \mathcal{K}(z) \left[ F_i^{(1)}(x - tzh_k) - F_i^{(1)}(x) \right] z_i h_k dt dz. $$

Moreover, we have

$$ \mathbb{E}[\mathcal{Z}_k(x)] = F(x) + \tilde{\nu}_k(x), $$

$$ \text{where } \tilde{\nu}_k(x) = \int_{\mathbb{R}^d} \mathbf{K}(z) [F(x - zh_k) - F(x)] dz. $$

Then, combining the previous results we state

$$ \begin{aligned} \mathrm{Var}[F_n(x)] &= \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 (\mathbb{E}[\mathcal{Z}_k^2(x)] - \mathbb{E}[\mathcal{Z}_k(x)]^2) \\ &= \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 \left[ F(x) - h_k \sum_{i=1}^d \phi_i(\mathbf{K}) F_i^{(1)}(x) - \nu_k(x) - (F(x) + \tilde{\nu}_k(x))^2 \right] \\ &= [F(x)(1-F(x))] \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 - \sum_{i=1}^d \phi_i(\mathbf{K}) F_i^{(1)}(x) \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 h_k \\ &\quad - \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 [\nu_k(x) + \tilde{\nu}_k(x)^2 + 2F(x)\tilde{\nu}_k(x)]. \end{aligned} $$

Likewise,

$$ \mathrm{Var}[F_n(x)] = S_{3,n}(x) + S_{4,n}(x) - S_{5,n}(x). \qquad (1.22) $$
$$
S_{3,n}(x) = [F(x)(1-F(x))]\Pi_n^2 \sum_{k=1}^n \Pi_k^{-2}\gamma_k^2, \quad S_{4,n}(x) = \sum_{i=1}^d \phi_i(\mathbf{K}) F_i^{(1)}(x) \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2}\gamma_k^2 h_k \\
\text{and } S_{5,n}(x) = \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2}\gamma_k^2 [\nu_k(x) + \tilde{\nu}_k(x)^2 + 2F(x)\tilde{\nu}_k(x)].
$$

1. For the case where $a \in [\frac{\alpha}{3}, 1)$, we have $\lim_{n \to +\infty} n\gamma_n > \min\left\{\frac{a+\alpha}{2}, 2a\right\} = \frac{a+\alpha}{2}$.

**Asymptotic behavior of S<sub>3,n</sub>(x):**

The application of lemma 1.2 ensures that

$$
\lim_{n \to +\infty} \frac{1}{\gamma_n} \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 = \frac{1}{2 - \alpha \xi}.
$$

Consequently, we get

$$
S_{3,n}(x) = \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 [F(x)(1-F(x))] = F(x)(1-F(x)) \frac{\gamma_n}{2-\alpha\xi} + o(\gamma_n).
$$

**Asymptotic behavior of S<sub>4,n</sub>(x):**

Now, since $(\gamma_n h_n^{-1})_{n \ge 1} \in \mathcal{GS}(a+\alpha)$, the application of lemma 1.2 ensures that

$$
\lim_{n \to +\infty} \frac{1}{\gamma_n h_n} \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k \gamma_k h_k = \frac{1}{2 - (a + \alpha)\xi}.
$$

Therefore, we get

$$
S_{4,n}(x) = \frac{\gamma_n h_n}{2 - (a + \alpha)\xi} \sum_{i=1}^{d} \phi_i(\mathbf{K}) F_i^{(1)}(x) + o(\gamma_n h_n).
$$

**Asymptotic behaviour of S<sub>5,n</sub>(x):**

Owing to the fact that $F$ and $F_i^{(1)}$ are bounded and continuous at $x$ for all $i \in \{1, \dots, d\}$, we have $\lim_{k \to +\infty} \nu_k(x) = 0$ and $\lim_{k \to +\infty} \tilde{\nu}_k(x) = 0$. Thus, $\lim_{k \to +\infty} (\nu_k(x) + \tilde{\nu}_k(x)^2 + 2F(x)\tilde{\nu}_k(x)) = 0$.

The application of lemma 1.2 ensures that

$$
S_{5,n}(x) = \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k^2 [\nu_k(x) + \tilde{\nu}_k(x)^2 + 2F(x)\tilde{\nu}_k(x)] = o(\gamma_n).
$$

Therefore,

$$
\mathrm{Var}[F_n(x)] = \frac{\gamma_n}{2 - \alpha\xi} F(x)(1 - F(x)) - \frac{\gamma_n h_n}{2 - (a + \alpha)\xi} \sum_{i=1}^{d} \phi_i(\mathbf{K}) F_i^{(1)}(x) + o(\gamma_n h_n).
$$

2. In the case where $a \in [\frac{\alpha}{4}, \frac{\alpha}{3})$, we have $\gamma_n h_n = o(h_n^4)$ thus $\gamma_n h_n^{-3} \xrightarrow{n \to +\infty} 0$. Then, by applying the second part of lemma 1.2 along with the fact that $\alpha_k = o(h_k^4) \xrightarrow{n \to +\infty} 0$, it follows that

$$
\Pi_n^2 \sum_{k=1}^{n} \Pi_k^{-2} \gamma_k o(h_k^4) = o(1).
$$

Hence,

$$
\begin{align*}
\mathrm{Var}[F_n(x)] &= \frac{\gamma_n}{2 - \alpha\xi} F(x)(1 - F(x)) + o(\gamma_n) - \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2} \gamma_k o(h_k^4) + o(\gamma_n) \\
&= \frac{\gamma_n}{2 - \alpha\xi} F(x)(1 - F(x)) + o(\gamma_n).
\end{align*}
$$
3. For the case where $a \in (0, \frac{\alpha}{4})$, we have $\gamma_n = o(h_n^4)$ thus $\gamma_n h_n^{-4} \xrightarrow{n \to +\infty} 0$. The application of lemma 1.2 ensures that

$$
\begin{align*}
\mathrm{Var}[F_n(x)] &= [F(x)(1-F(x))]\Pi_n^2 \sum_{k=1}^n \Pi_k^{-2}\gamma_k o(h_k^4) - \sum_{i=1}^d \phi_i(\mathbf{K}) F_i^{(1)}(x) \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2}\gamma_k o(h_k^4) + o(\gamma_n) \\
&= \Pi_n^2 \sum_{k=1}^n \Pi_k^{-2}\gamma_k o(h_k^4) + o(\gamma_n) \\
&= o(h_n^4).
\end{align*}
$$

□

Now, let's recall the precise statement of Lyapunov's theorem, which shall be invested in the
next proof.

**Theorem 1.14.** Let $(X_n)$ be a sequence of independent random variables, centered all with a finite moment of order $2+p, p>0$. We note, $u_n^2 = \mathrm{Var}\left[\sum_{i=1}^n X_i\right] > 0$.
Under the following Lyapunov condition,

$$
\lim_{n \to +\infty} \frac{1}{u_n^{2+p}} \sum_{i=1}^{n} \mathbb{E} [|X_i|^{2+p}] = 0,
$$

we obtain

$$
\frac{1}{u_n} \sum_{i=1}^{n} X_i \xrightarrow{n \to +\infty} \mathcal{N}(0, 1).
$$

Proof of Theorem 1.5.

First, by the relation (1.1), we have

$$
F_n(x) = (1 - \gamma_n) F_{n-1}(x) + \gamma_n \mathcal{Z}_n(x),
$$

It follows that

$$
F_n(x) - \mathbb{E}[F_n(x)] = (1 - \gamma_n) (F_{n-1}(x) - \mathbb{E}[F_{n-1}(x)]) + \gamma_n (\mathcal{Z}_n(x) - \mathbb{E}[\mathcal{Z}_n(x)]).
$$

Using a simple recurrence, we obtain

$$
\begin{align*}
F_n(x) - \mathbb{E}[F_n(x)] &= \prod_{i=1}^{n} (1 - \gamma_i) (F_0(x) - \mathbb{E}[F_0(x)]) + \sum_{k=1}^{n} \prod_{j=k+1}^{n} (1 - \gamma_j) \gamma_k (\mathcal{Z}_k(x) - \mathbb{E}[\mathcal{Z}_k(x)]) \\
&= \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k (\mathcal{Z}_k(x) - \mathbb{E}[\mathcal{Z}_k(x)]).
\end{align*}
$$

Thus,

$$
F_n(x) - \mathbb{E}[F_n(x)] = \Pi_n \sum_{k=1}^{n} Y_k(x),
$$

where,

$$
Y_k(x) = \Pi_k^{-1} \gamma_k (\mathcal{Z}_k(x) - \mathbb{E}[\mathcal{Z}_k(x)]). \quad (1.23)
$$
Now, in order to apply Lyapunov's theorem for $Y_k(x)$, we mention that $\mathbb{E}[Y_k(x)] = 0$ and we state

$$
\begin{align*}
\nu_n^2 &= \sum_{k=1}^{n} \mathrm{Var}[Y_k(x)] \\
&= \sum_{k=1}^{n} \Pi_k^{-2} \gamma_k^2 \mathrm{Var}[\mathcal{Z}_k(x)] \\
&= \sum_{k=1}^{n} \Pi_k^{-2} \gamma_k^2 (\mathbb{E}[\mathcal{Z}_k^2(x)] - \mathbb{E}[\mathcal{Z}_k(x)]^2) \\
&= \sum_{k=1}^{n} \Pi_k^{-2} \gamma_k^2 \left( F(x) + \nu'_k(x) - F^2(x) - \tilde{\nu}_k(x)^2 - 2F(x)\tilde{\nu}_k(x) \right),
\end{align*}
$$

where $\nu'_k(x) = 2^d \int_{\mathbb{R}^d} \mathbf{K}(z)\mathcal{K}(z) [F(x - zh_k) - F(x)] dz$ and $\tilde{\nu}_k(x) = \int_{\mathbb{R}^d} \mathbf{K}(z) [F(x - zh_k) - F(x)] dz$.

It is obvious that $\lim_{k \to +\infty} \nu'_k(x) = 0$ and $\lim_{k \to +\infty} \tilde{\nu}_k(x) = 0$. Thus, $\lim_{k \to +\infty} (\nu'_k(x) - \tilde{\nu}_k(x)^2 - 2F(x)\tilde{\nu}_k(x)) = 0$. Hence, we get

$$
\nu_n^2 = \sum_{k=1}^{n} \Pi_k^{-2} \gamma_k^2 [F(x)(1-F(x)) + o(1)].
$$

The application of lemma 1.2 ensures that

$$
\nu_n^2 = \frac{\gamma_n}{\Pi_n^2} \left( \frac{1}{2 - \alpha\xi} F(x)(1 - F(x)) + o(1) \right).
$$

Additionally, using (1.23) we have

$$
\begin{align*}
\mathbb{E} [|Y_k(x)|^{2+p}] &= \mathbb{E} [|\Pi_k^{-1} \gamma_k (\mathcal{Z}_k(x) - \mathbb{E}[\mathcal{Z}_k(x)])|^{2+p}] \\
&= \Pi_k^{-2-p} \gamma_k^{2+p} \mathbb{E} [|\mathcal{Z}_k(x) - \mathbb{E}[\mathcal{Z}_k(x)]|^{2+p}].
\end{align*}
$$

Thus, we get

$$
\begin{align*}
\mathbb{E} [|Y_k(x)|^{2+p}] &\le 2\Pi_k^{-2-p} \gamma_k^{2+p} \mathbb{E} [|\mathcal{Z}_k(x)|^{2+p}] . \\
&= O \left( \Pi_k^{-2-p} \gamma_k^{2+p} \mathbb{E} [|\mathcal{Z}_k(x)|^{2+p}] \right).
\end{align*}
$$

Moreover, since we have $\mathcal{Z}_k(x) \le 1$, which gives,

$$
\forall p > 0, \mathbb{E}[|\mathcal{Z}_k(x)|^{2+p}] = O(1),
$$

we infer that

$$
\begin{align*}
\sum_{k=1}^{n} \mathbb{E}[|Y_k(x)|^{2+p}] &= O\left(\sum_{k=1}^{n} \Pi_k^{-2-p} \gamma_k^{2+p} \mathbb{E}[|\mathcal{Z}_k(x)|^{2+p}]\right) \\
&= O\left(\sum_{k=1}^{n} \Pi_k^{-2-p} \gamma_k^{2+p}\right).
\end{align*}
$$

Here we suppose that $a \ge \alpha/4$. Departing from the fact that $\lim_{n \to +\infty} (n\gamma_n) > \alpha/2$, which implies that there exists $p > 0$ such that

$$
\xi^{-1} = \lim_{n \to +\infty} n\gamma_n > \frac{1+p}{2+p}\alpha.
$$
Thus $(2+p)-(1+p)\alpha\xi > 0$ and the application of lemma 1.2 yields

$$ \sum_{k=1}^{n} \mathbb{E}[|Y_k(x)|^{2+p}] = O\left(\frac{\gamma_n^{1+p}}{\Pi_n^{2+p}}\right). $$

Thus,

$$ \frac{1}{v_n^{2+p}} \sum_{k=1}^{n} \mathbb{E}[|Y_k(x)|^{2+p}] = O\left(\gamma_n^{\frac{p}{2}}\right) = o(1). $$

Furthermore, since we have

$$ \lim_{n \to +\infty} \frac{1}{v_n^{2+p}} \sum_{k=1}^{n} \mathbb{E} \left[ |Y_k(x) - \mathbb{E}[Y_k(x)]|^{2+p} \right] = \lim_{n \to +\infty} \frac{1}{v_n^{2+p}} \sum_{k=1}^{n} \mathbb{E} [ |Y_k(x)|^{2+p} ] = 0. $$

Then, the application of the Lyapunov's theorem ensures that

$$ \sqrt{\gamma_n^{-1}} (F_n(x) - \mathbb{E}[F_n(x)]) \xrightarrow{D} \mathcal{N} \left( 0, \frac{1}{2-\alpha\xi} F(x)(1-F(x)) \right). \quad (1.24) $$

In what follows, we distinguish the two cases displayed below:

**First case: $a > \alpha/3$:**

We have

$$ \mathbb{E}[F_n(x)] - F(x) = o\left(\sqrt{\gamma_n h_n}\right). $$

By replacing the latter in (1.24), we obtain

$$ \sqrt{\gamma_n^{-1}} (F_n(x) - F(x)) \xrightarrow{D} \mathcal{N} \left( 0, \frac{1}{2 - \alpha\xi} F(x)(1 - F(x)) \right). $$

**Second case: $\alpha/4 \le a \le \alpha/3$:**

We have

$$ \mathbb{E}[F_n(x)] - F(x) = \frac{1}{2(1-2a\xi)} h_n^2 \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x) + o(h_n^2). $$

As $\gamma_n^{-1/2} h_n^2 \xrightarrow{n \to +\infty} c_n^{1/2}$, we obtain

$$ \sqrt{\gamma_n^{-1}} (F_n(x) - F(x)) \xrightarrow{D} \mathcal{N} \left( \frac{c_n^{1/2}}{2(1-2a\xi)} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x), \frac{1}{2-\alpha\xi} F(x)(1-F(x)) \right). $$

The case where $a < \alpha/4$ implies the convergence in probability in the second part of the theorem.

By applying the Bienaymé-Chebyshev inequality, we get

$$ \mathbb{P} \left[ \left| \frac{F_n(x) - F(x)}{h_n^2} - \mathbb{E} \left[ \frac{F_n(x) - F(x)}{h_n^2} \right] \right| \ge \epsilon \right] \le \frac{\mathrm{Var}[F_n(x)]}{h_n^4 \epsilon^2}. $$

Since we have $\gamma_n^{-1} h_n^4 \xrightarrow{n \to +\infty} +\infty$, then $\frac{\mathrm{Var}[F_n(x)]}{h_n^4 \epsilon^2} \xrightarrow{n \to +\infty} 0$. Hence, we deduce that

$$ \frac{1}{h_n^2} (F_n(x) - F(x)) \xrightarrow{P} \frac{1}{2(1-2a\xi)} \sum_{j=1}^{d} \mu_j(\mathbf{K}) F_{jj}^{(2)}(x). $$

□
Proof of Theorem 1.12.

Following similar steps as Slaoui (2014b), and by using (1.11) and (1.13), we obtain

$$ MWISE[F_n] - MWISE[\tilde{F}_n] = \frac{n^{-1}V_F}{(2\gamma_0 - 1)(\gamma_0 - \frac{2}{3})^{\frac{2}{3}}} \left\{ (\gamma_0 - 1)^2 \left(\gamma_0 - \frac{2}{3}\right)^{\frac{2}{3}} - 2^{-\frac{4}{3}}\frac{3}{4}(2\gamma_0 - 1) \left(\gamma_0^2 - 2^{\frac{4}{3}}\left(\gamma_0 - \frac{2}{3}\right)^{\frac{2}{3}}\right) V_F^{-1} J_1^{\frac{4}{3}} J_2^{-\frac{1}{3}} n^{-\frac{1}{3}} + o(n^{-\frac{1}{3}}) \right\}. $$

Which is no other than

$$ n < \left( \frac{2^{-\frac{4}{3}} \frac{3}{4} (2\gamma_0 - 1) \left( \gamma_0^2 - 2^{\frac{4}{3}} \left( \gamma_0 - \frac{2}{3} \right)^{\frac{2}{3}} \right) V_F^{-1} J_1^{\frac{4}{3}} J_2^{-\frac{1}{3}}}{(\gamma_0 - 1)^2 \left( \gamma_0 - \frac{2}{3} \right)^{\frac{2}{3}}} \right)^3. $$

The fact that $\gamma_0 = \frac{2}{3} + \varepsilon$ ensures that

$$ n < \frac{\frac{3}{2^{10}} \frac{J_1^4}{J_2 V_F^3} \frac{(\frac{1}{3} + 2\varepsilon)^3}{(\varepsilon - \frac{1}{3})^6} \left(\frac{2}{3} + \varepsilon - 2^{\frac{2}{3}}\varepsilon^{\frac{1}{3}}\right)^3 \left(\frac{2}{3} + \varepsilon + 2^{\frac{2}{3}}\varepsilon^{\frac{1}{3}}\right)^3}{\varepsilon^2}. $$

Thus, the result of the theorem is derived by taking into account the last inequality. $\square$

<figure><img src="image_21.png" /></figure>
# Chapter 2

## Statistical inferences for multivariate conditional cumulative distribution function estimation by stochastic approximation method

### 2.1 Introduction

Assume that we observe independent identically distributed vectors $(X_1, Y_1), \dots, (X_n, Y_n)$ of a bivariate random variable $(X, Y) \in \mathbb{R} \times \mathbb{R}$ with common cumulative distribution function $\pi(x, y)$ where one is interested in modeling the functional dependence of the observation $Y$ on the covariable $X$ by the conditional cumulative distribution function (CCDF) of $Y$ given $X = x$, denoted by, for all real $y$ and $x$,

$$ \pi(y|x) := \mathbb{P}[Y \le y | X = x]. $$

We shall also assume that the bivariate random variable $(X, Y)$ (resp. the random variable $X$) has a density function $f_{(X,Y)}$ (resp. $f_X$) with respect to the Lebesgue measure. Recall that for all real $y$ and $x$ such that $f_X(x) \ne 0$, the CCDF of $Y$ given $X = x$ is expressed by

$$ \pi(y|x) = \frac{\int_{\mathbb{R}} \mathbb{1}_{\{u \le y\}} f_{(X,Y)}(x, u) du}{f_X(x)}. $$

In a variety of non-parametric statistical problems, the estimation of a CCDF is a key aspect of inference. Remember that the CCDF has the merit of characterizing the conditional law of the considered bivariate random variables. Notably, the CCDF is often useful in reliability or survival analysis.

More specifically, the conditional survival function $S(y|x)$ defined by, for all real $y$ and $x$, $S(y,x) := 1 - \pi(y|x)$ is of extreme interest, either by itself, or by its independence with the conditional hazard function $H(y,x)$ indicated by, for all real $y$ and $x$, $H(y,x) := \frac{f(y,x)}{S(y,x)}$ where $f(y,x)$ denotes the conditional density of $Y$ given $X = x$. Furthermore, conditional quantiles can also be deduced from the CCDF $\pi$ by (pseudo)-inversion given $x$ of the function $y \to \pi(y|x)$ and the same procedure may be applied to the estimator of CCDF to find conditional quantile estimators.

Several non-parametric estimators have been elaborated to estimate the CCDF. Many of them rely initially on estimating the $\int_{\mathbb{R}} \mathbb{1}_{\{u \le y\}} f_{(X,Y)}(x,u) du$. The conditional cumulative distribution function was first extensively explored by Stute (1986) using a nearest-neighbor-type conditional empirical process. Subsequently, Hall et al. (1999), motivated by the problem of setting prediction intervals in time series analysis, developed a new non-parametric method for CCDF
estimation resting on an adjusted form of Nadaraya-Watson estimator. Afterwards, Ferrigno et al. (2014) established uniform asymptotic certainty bands for the CCDF using the same strategy.

For a general non-parametric regression model, Kiwitt and Neumeyer (2012) set up two estimators using a kernel approach, where the distribution of the error given the covariate is modeled by a CCDF provided by $P(\epsilon \le y | X = x)$.

On a given compact set, Brunel et al. (2010) constructed a minimax estimator of the CCDF. Thereafter, Veraverbeke et al. (2014) built up a new estimator of CCDF investing a method of pre-adjusting the original observations non-parametrically. Recently, Bouanani et al. (2020) introduced a new method to settle CCDF estimation problem based on local polynomial technique.

Many functional estimations are grounded on estimating the CCDF see e.g Laksaci and Maref (2009), Laksaci and Hachemi (2012) and Almanjahie et al. (2018). The CCDF is involved in a wide range of applications, for instance, in medicine see Gannoun et al. (2002), econometrics see Li et al. (2013) or machine learning domain see the recent work of Chilinski and Silva (2020). In a broader context, extensive state of art works including various non-parametric approaches tackled the conditional estimation. We can state for example Yu and Jones (1998), Fan et al. (1996), Berlinet et al. (1998b), Berlinet et al. (1998a), Honda (2000b), Honda (2000a) and Plancade (2013). For recent references see Benziadi et al. (2016), Choudhury et al. (2018), Al-Awadhi et al. (2019a), Chikr-Elmezouar et al. (2019b) and Slaoui and Khardani (2020).

Over the past decade, data streams have become an increasingly important area of research. Some of the most common data streams include Internet packet data, Twitter activity, Facebook newsfeed, credit card transactions and more recently COVID-19 epidemic data. In these situations, the data arrives regularly so that it is impossible to store it in a traditional database. In such a context, it is very interesting to build a recursive multivariate conditional cumulative distribution estimator that does not need to store all the data in memory and that can be easily updated to handle the online data. The basic target of this chapter is to provide a non-parametric strategy to recursively estimate the CCDF.

### 2.1.1 Presentation of the method

Let $(X, Y)$ be a random vector with values in $\mathbb{R}^d \times \mathbb{R}^q$, $q \ge 1$, with a joint density function $f_{(X,Y)}$ and let $f_X$ denote the marginal probability density of $X$ given by $f_X(x) = \int_{\mathbb{R}^q} f_{(X,Y)}(x, u) du$. Moreover, let $(X_1, Y_1), \dots, (X_n, Y_n)$ be independent random vectors identically distributed as $(X, Y)$. In this work, our central focus is upon the problem of estimating the CCDF of $Y$ given $X = x$ provided by

$$ \pi: \mathbb{R}^q \times \mathbb{R}^d \to \mathbb{R} \\ (y|x) \mapsto \mathbb{P}[Y \le y | X = x] = \frac{a(x, y)}{f_X(x)}, $$

where

$$ a(x, y) = \int_{\mathbb{R}^q} \mathbb{1}_{\{u \le y\}} f_{(X,Y)}(x, u) du \quad \text{and} \quad f_X(x) = \int_{\mathbb{R}^q} f_{(X,Y)}(x, u) du. $$

The recursive estimator was constructed based on a direct of stochastic approximation method. Since, Slaoui (2014b) reused stochastic approximation methods to enhance the qualities of the univariate distribution function estimator and the previous chapter elaborated the multivariate one and following the same recursive approach, we intend to establish a multivariate conditional cumulative distribution function estimator.

To build up a stochastic algorithm, which approaches the function $a$ at a given couple of vectors $(x, y)$, we define an algorithm of search of the zero function $\phi: z \mapsto a(x, y) - z$ and we set:

$$ (i) a_0(x, y) \in \mathbb{R} $$

$$ (ii) \text{ for all } n \ge 1, a_n(x, y) = a_{n-1}(x, y) + \gamma_n U_n(x, y), $$
where $U_n(x, y)$ corresponds to an observation of the function $\phi$ at the point $a_{n-1}(x, y)$. In the following, we set a multivariate indicator function denoted $\chi$ and identified by

$$ \chi_k : \mathbb{R}^q \to \mathbb{R}, y \mapsto \mathbb{1}_{\{Y_k \le y\}}. $$

By considering $U_n(x, y) = \chi_n(y) h_n^{-d} \mathbf{K} \left( \frac{x - X_n}{h_n} \right) - a_{n-1}(x, y)$, the stochastic approximation algorithm that is devoted to estimate recursively the function $a$ at a couple of vectors $(x, y)$ can be stated as follows:

$$ a_n(x, y) = (1 - \gamma_n)a_{n-1}(x, y) + \gamma_n \chi_n(y) h_n^{-d} \mathbf{K} \left( \frac{x - X_n}{h_n} \right). \quad (2.1) $$

Throughout this section, we consider that $a_0(x, y) = 0$. Therefore, by recurrence, we get

$$ a_n(x, y) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k \chi_k(y) h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right). \quad (2.2) $$

Within this framework, we recall the recursive multivariate probability density estimator of the density function noted $f_n$ and defined in Mokkadem et al. (2009a). It was constructed with the same tools of stochastic approximation algorithm and under the condition that $f_0(x) = 0$, we have:

$$ f_n(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right). \quad (2.3) $$

Therefore, we introduce our recursive estimator $\pi_n$ specified by

$$ \pi_n(y|x) = \begin{cases} \frac{a_n(x, y)}{f_n(x)} & \text{if } f_n(x) \neq 0 \\ 0 & \text{otherwise} \end{cases}. \quad (2.4) $$

Our main purpose is to examine the asymptotic properties of the proposed multivariate estimator of the CCDF and to corroborate its performances.

Moreover, we set forward the non-recursive estimator of the function $a$ given by

$$ \tilde{a}_n(x, y) = \frac{1}{nh_n^d} \sum_{k=1}^{n} \chi_k(y) \mathbf{K} \left( \frac{x - X_k}{h_n} \right) $$

and the non-recursive estimator of the multivariate density function $f$ defined by

$$ \tilde{f}_n(x) = \frac{1}{nh_n^d} \sum_{k=1}^{n} \mathbf{K} \left( \frac{x - X_k}{h_n} \right). \quad (2.5) $$

Hence, we shall compare our estimator to the generalized kernel CCDF estimator of Nadaraya-Watson Nadaraya (1964) and Watson (1964) $\tilde{\pi}_n$ expressed by

$$ \tilde{\pi}_n(y|x) = \begin{cases} \frac{\tilde{a}_n(x, y)}{\tilde{f}_n(x)} & \text{if } \tilde{f}_n(x) \neq 0 \\ 0 & \text{otherwise} \end{cases}. \quad (2.6) $$

### 2.1.2 Notations and assumptions

For this section and under ($A_1$) and ($A_2$) given in 1.1.1, we provide the following notations and assumptions which will be intensively used for our theoretical main results.
(A4) (i) The functions $f_X$ and $a$ are bounded and twice differentiable.

(ii) For all $i, j \in \{1, \dots, d\}$, $f_{X_{ij}}^{(2)} := \frac{\partial^2 f_X}{\partial x_i \partial x_j}$ and $a_{ij}^{(2)} := \frac{\partial^2 a}{\partial x_i \partial x_j}$ are bounded and continuous at $x$.

First of all, we need to recall the following proposition which introduces the bias and the variance of $f_n$. The proof of this result was depicted in Mokkadem et al. (2009a).

### 2.1.3 Bias and variance of $f_n$

**Proposition 2.1.** Under assumptions (A1), (A2) and (A4), we obtain

1. If $a \in (0, \frac{\alpha}{d+4}]$, then

$$ \mathbb{E}[f_n(x)] - f_X(x) = \frac{1}{2(1 - 2a\xi)} \left( \sum_{j=1}^{d} \mu_j(\mathbf{K}) f_{X_{jj}}^{(2)}(x) \right) h_n^2 + \mathbf{o}(h_n^2). \quad (2.7) $$

If $a \in (\frac{\alpha}{d+4}, 1)$, then

$$ \mathbb{E}[f_n(x)] - f_X(x) = \mathbf{o}\left(\sqrt{\gamma_n h_n^{-d}}\right). \quad (2.8) $$

2. If $a \in (0, \frac{\alpha}{d+4})$, then

$$ \mathrm{Var}[f_n(x)] = \mathbf{o}(h_n^4). \quad (2.9) $$

If $a \in (\frac{\alpha}{d+4}, 1)$, then

$$ \mathrm{Var}[f_n(x)] = \frac{1}{2 - (\alpha - ad)\xi} f_X(x) R(\mathbf{K}) \frac{\gamma_n}{h_n^d} + \mathbf{o}\left(\gamma_n h_n^{-d}\right). \quad (2.10) $$

## 2.2 Main results

In order to explore the asymptotic properties of our estimator $\pi_n$, we need first to introduce the following proposition which provides the bias and the variance of $a_n$.

### 2.2.1 Bias and variance of $a_n$

**Proposition 2.2.** Under assumptions (A1), (A2) and (A4), we obtain

1. If $a \in (0, \frac{\alpha}{d+4}]$, then

$$ \mathbb{E}[a_n(x, y)] - a(x, y) = \frac{1}{2(1 - 2a\xi)} \left( \sum_{j=1}^{d} \mu_j(\mathbf{K}) a_{jj}^{(2)}(x, y) \right) h_n^2 + \mathbf{o}(h_n^2). \quad (2.11) $$

If $a \in (\frac{\alpha}{d+4}, 1)$, then

$$ \mathbb{E}[a_n(x, y)] - a(x, y) = \mathbf{o}\left(\sqrt{\gamma_n h_n^{-d}}\right). \quad (2.12) $$
2. If $a \in (0, \frac{\alpha}{d+4})$, then

$$ \mathrm{Var}[a_n(x, y)] = \mathbf{o}(h_n^4). \qquad (2.13) $$

If $a \in [\frac{\alpha}{d+4}, 1)$, then

$$ \mathrm{Var}[a_n(x, y)] = \frac{1}{2 - (\alpha - ad)\xi} a(x, y) R(\mathbf{K}) \frac{\gamma_n}{h_n^d} + \mathbf{o}\left(\gamma_n h_n^{-d}\right). \qquad (2.14) $$

In the following theorem, we introduce our main result which provides the bias and the variance of our CCDF multivariate estimator $\pi_n$. The following notations are highly useful as they are invested throughout the whole chapter.

$$
\begin{align*}
\text{For all } i, j \in \{1, \dots, d\}, \quad & \pi_i^{(1)}(y, \cdot) := \frac{\partial \pi}{\partial x_i}(y, \cdot) \quad \pi_{ij}^{(2)}(y, \cdot) := \frac{\partial^2 \pi}{\partial x_i \partial x_j}(y, \cdot), \\
& a_i^{(1)}(\cdot, y) := \frac{\partial a}{\partial x_i}(\cdot, y), \quad f_{X_i}^{(1)}(\cdot) := \frac{\partial f_X}{\partial x_i}(\cdot).
\end{align*}
$$

### 2.2.2 Bias and variance of $\pi_n$

**Theorem 2.3.** Let assumptions (A$_1$), (A$_2$) and (A$_4$) hold and note $R(\mathbf{K}) := \int_{\mathbb{R}^d} \mathbf{K}^2(z) dz$, we obtain

1. If $a \in (0, \frac{\alpha}{d+4}]$, then

$$ \mathbb{E}[\pi_n(y|x)] - \pi(y|x) = \frac{1}{2(1 - 2a\xi)} \frac{1}{f_X(x)} \sum_{j=1}^{d} \mu_j(\mathbf{K}) \left[ \pi_{jj}^{(2)}(y|x) f_X(x) + 2\pi_j^{(1)}(y|x) f_{X_j}^{(1)}(x) \right] h_n^2 + \mathbf{o}(h_n^2). \qquad (2.15) $$

If $a \in (\frac{\alpha}{d+4}, 1)$, then

$$ \mathbb{E}[\pi_n(y|x)] - \pi(y|x) = \mathbf{o}\left(\sqrt{\gamma_n h_n^{-d}}\right). \qquad (2.16) $$

2. If $a \in (0, \frac{\alpha}{d+4})$, then

$$ \mathrm{Var}[\pi_n(y|x)] = \mathbf{o}(h_n^4). \qquad (2.17) $$

If $a \in [\frac{\alpha}{d+4}, 1)$, then

$$ \mathrm{Var}[\pi_n(y|x)] = \frac{R(\mathbf{K})}{2 - (\alpha - ad)\xi} \frac{\pi(y|x)(1 - \pi(y|x))}{f_X(x)} \frac{\gamma_n}{h_n^d} + \mathbf{o}\left(\frac{\gamma_n}{h_n^d}\right). \qquad (2.18) $$

In the sequel, let us present the following theorem which identifies the asymptotic normality of our recursive estimator $\pi_n$.

### 2.2.3 Weak pointwise convergence rate of $\pi_n$

**Theorem 2.4.** Let assumptions (A$_1$), (A$_2$) and (A$_4$) hold.
1. If there exists a non-negative real c such that $\gamma_n^{-1}h_n^{d+4} \xrightarrow{n \to +\infty} c$, then

$$ \sqrt{\gamma_n^{-1} h_n^d} (\pi_n(y|x) - \pi(y|x)) \xrightarrow{n \to +\infty} \mathcal{N}\left(\sqrt{c} M(x, y), \sigma^2(x, y)\right). \quad (2.19) $$

with

$$ M(x, y) = \frac{1}{2(1 - 2a\xi)} \frac{1}{f_X(x)} \sum_{j=1}^{d} \mu_j(\mathbf{K}) \left[ \pi_{jj}^{(2)}(y|x) f_X(x) + 2\pi_j^{(1)}(y|x) f_{X_j}^{(1)}(x) \right] $$

and

$$ \sigma^2(x, y) = \frac{R(\mathbf{K})}{2 - (\alpha - ad)\xi} \frac{\pi(y|x)(1 - \pi(y|x))}{f_X(x)}. $$

2. If $\gamma_n^{-1}h_n^{d+4} \xrightarrow{n \to +\infty} +\infty$, then

$$ \frac{1}{h_n^2} (\pi_n(y|x) - \pi(y|x)) \xrightarrow{n \to +\infty} M(x, y). $$

## 2.3 Optimal choice of the stepsizes

In order to assess the asymptotic quality of the CCDF recursive estimator $\pi_n$, we set up the Mean Weighted Integrated Squared Error (MWISE). We first introduce the MWISE expression:

$$ MWISE[\pi_n] = \int_{\mathbb{R}^{d+q}} \left[ (\mathbb{E}[\pi_n(y|x)] - \pi(y|x))^2 + \mathrm{Var}[\pi_n(y|x)] \right] f_X^2(x) f_{X,Y}(x, y) dx dy. \quad (2.20) $$

### 2.3.1 Asymptotic expressions of MWISE[$\pi_n$]

First of all, let us set the following notations

$$ I_1 := \int_{\mathbb{R}^{d+q}} \left( \sum_{j=1}^{d} \mu_j(\mathbf{K}) \left[ \pi_{jj}^{(2)}(y|x) f_X(x) + 2\pi_j^{(1)}(y|x) f_{X_j}^{(1)}(x) \right] \right)^2 f_{X,Y}(x, y) dx dy $$

and

$$ I_2 := \int_{\mathbb{R}^{d+q}} \pi(y|x) (1 - \pi(y|x)) f_X(x) f_{X,Y}(x, y) dx dy. $$

**Proposition 2.5.** The MWISE of the estimator $\pi_n$ is maintained as follows.

If $a \in (0, \frac{\alpha}{d+4})$, then

$$ MWISE[\pi_n] = \frac{I_1}{4(1 - 2a\xi)^2} h_n^4 + \mathbf{o}(h_n^4). $$

If $a = \frac{\alpha}{d+4}$, then

$$ MWISE[\pi_n] = \frac{I_2}{2 - (\alpha - ad)\xi} R(\mathbf{K}) \gamma_n h_n^{-d} + \frac{I_1}{4(1 - 2a\xi)^2} h_n^4 + \mathbf{o}(h_n^4). $$

If $a \in (\frac{\alpha}{d+4}, 1)$, then

$$ MWISE[\pi_n] = \frac{I_2}{2 - (\alpha - ad)\xi} R(\mathbf{K}) \gamma_n h_n^{-d} + \mathbf{o}(\gamma_n h_n^{-d}). $$
The following corollary ensures that the bandwidth which minimizes the MWISE of $\pi_n$ depends on the choice of the stepsize ($\gamma_n$). As a matter of fact, the corresponding MWISE depends also on ($\gamma_n$).

**Corollary 2.6.** Let assumptions (A₁), (A₂) and (A₄) hold. To minimize the MWISE of $\pi_n$, the bandwidth ($h_n$) must be equal to

$$ \left( \left( \frac{d(1-2a\xi)^2 I_2}{2-(\alpha-ad)\xi I_1} R(\mathbf{K}) \right)^{\frac{1}{d+4}} \gamma_n^{\frac{1}{d+4}} \right). $$

Hence, the corresponding MWISE is determined by

$$ \text{MWISE}[\pi_n] = \frac{d+4}{4d^{\frac{d}{d+4}}} \left( \frac{I_1}{(1-2a\xi)^2} \right)^{\frac{d}{d+4}} \left( \frac{I_2}{2-(\alpha-ad)\xi} \right)^{\frac{4}{d+4}} R(\mathbf{K})^{\frac{4}{d+4}} \gamma_n^{\frac{4}{d+4}} + \mathbf{o}\left( \gamma_n^{\frac{4}{d+4}} \right). $$

The following corollary holds in the special case where ($\gamma_n$) is chosen as ($\gamma_n$) = ($\gamma_0 n^{-1}$) in order to minimize the MWISE[$\pi_n$].

**Corollary 2.7.** Let assumptions (A₁), (A₂) and (A₄) hold. To minimize the MWISE of $\pi_n$, we need to opt for the stepsize ($\gamma_n$) in $\mathcal{G}S(-1)$ such that $\lim_{n \to +\infty} n\gamma_n = \gamma_0$. Then the bandwidth ($h_n$) must be equal to

$$ \left( \left( \frac{d(\gamma_0(d+4) - 2) I_2}{2(d+4)} I_1 R(\mathbf{K}) \right)^{\frac{1}{d+4}} n^{-\frac{1}{d+4}} \right). $$

Consequently, the corresponding MWISE is identified by

$$ \text{MWISE}[\pi_n] = \frac{(d+4)^{\frac{3d+8}{d+4}}}{4^{\frac{d+6}{d+4}} d^{\frac{d}{d+4}}} \gamma_0^2 (\gamma_0(d+4) - 2)^{-\frac{(2d+4)}{d+4}} I_1^{\frac{d}{d+4}} I_2^{\frac{4}{d+4}} R(\mathbf{K})^{\frac{4}{d+4}} n^{-\frac{4}{d+4}} + \mathbf{o}\left( n^{-\frac{4}{d+4}} \right). $$

In order to get the optimal choice of ($\gamma_n$), we deduce that the minimum of MWISE[$\pi_n$] is achieved at $\gamma_0 = 1$. Hence, we introduce the following corollary.

**Corollary 2.8.** Let assumptions (A₁), (A₂) and (A₄) hold. To minimize the MWISE of $\pi_n$, we must select the stepsize ($\gamma_n$) in $\mathcal{G}S(-1)$ such that $\lim_{n \to +\infty} n\gamma_n = 1$. Therefore, the optimal bandwidth ($h_n$) must equal

$$ \left( \left( \frac{d(d+2)}{2(d+4)} \frac{I_2}{I_1} R(\mathbf{K}) \right)^{\frac{1}{d+4}} n^{-\frac{1}{d+4}} \right). \qquad (2.21) $$

As a result, the corresponding MWISE is expressed by

$$ \text{MWISE}[\pi_n] = \frac{(d+4)^{\frac{3d+8}{d+4}}}{4^{\frac{d+6}{d+4}} d^{\frac{d}{d+4}} (d+2)^{\frac{2d+4}{d+4}}} I_1^{\frac{d}{d+4}} I_2^{\frac{4}{d+4}} R(\mathbf{K})^{\frac{4}{d+4}} n^{-\frac{4}{d+4}} + \mathbf{o}\left( n^{-\frac{4}{d+4}} \right). $$

**Remark 2.9.** Note that, for the particular case where the stepsize ($\gamma_n$) is in $\mathcal{G}S(-1)$ such that $\lim_{n \to +\infty} n\gamma_n = 1$ and the bandwidth ($h_n$) is chosen such that $\lim_{n \to +\infty} nh_n^{d+4} = 0$ (which corresponds to undersmoothing), the asymptotic normality of our proposed estimator is indicated as follows

$$ \sqrt{nh_n^d} (\pi_n(y|x) - \pi(y|x)) \xrightarrow[n \to +\infty]{D} \mathcal{N} \left( 0, \frac{d+4}{2(d+2)} R(\mathbf{K}) \frac{\pi(y|x)(1-\pi(y|x))}{f_X(x)} \right). \qquad (2.22) $$
The statistical inference of the CCDF multivariate non-recursive estimator $\tilde{\pi}_n$ is addressed in our next section.

The following results can be handled in nearly the same way as $\pi_n$. The unique difference lies in the fact that it pertains to a non-recursive case. (See Hall et al. (1999) for more details of the univariate case.)

## 2.4 Asymptotic properties of $\tilde{\pi}_n$

In order to tackle the asymptotic properties of our estimator $\pi_n$, we need first to introduce the following proposition which provides the bias and the variance of $\tilde{\pi}_n$.

### 2.4.1 Bias and variance of $\tilde{\pi}_n$

**Proposition 2.10.** Let assumptions (A₁) and (A₄) hold. Then the bias and variance of Nadaraya-Watson's estimator are displayed as follows.

1. The bias of $\tilde{\pi}_n$:

$$ \mathbb{E}[\tilde{\pi}_n(y|x)] - \pi(y|x) = \frac{1}{2f_X(x)} \left( \sum_{j=1}^{d} \mu_j(\mathbf{K}) \left[ \pi_{jj}^{(2)}(y|x)f_X(x) + 2\pi_j^{(1)}(y|x)f_X_j^{(1)}(x) \right] \right) h_n^2 + \mathbf{o}(h_n^2). $$

2. The variance of $\tilde{\pi}_n$:

$$ \mathrm{Var}[\tilde{\pi}_n(x)] = R(\mathbf{K}) \frac{\pi(y|x)(1 - \pi(y|x))}{f_X(x)} \frac{1}{nh_n^d} + \mathbf{o}\left(\frac{1}{nh_n^d}\right). $$

The following proposition yields the distribution convergence rate of the non-recursive estimator.

### 2.4.2 Asymptotic normality of $\tilde{\pi}_n$

**Theorem 2.11.** Let assumptions (A₁) and (A₄) hold and suppose that $nh_n^{d+4} \xrightarrow{n \to +\infty} 0$. Then,

$$ \sqrt{nh_n^d} (\tilde{\pi}_n(y|x) - \pi(y|x)) \xrightarrow{n \to +\infty} \mathcal{N} \left( 0, R(\mathbf{K}) \frac{\pi(y|x)(1 - \pi(y|x))}{f_X(x)} \right). \quad (2.23) $$

**Remark 2.12.** It is obvious to infer from the expressions (2.22) and (2.23) that our CCDF proposed estimator is better than non-recursive one in terms of variance.

In the next subsection, we exhibit the expression of the Mean Weighted Integrated Squared Error of Nadaraya-Watson's estimator.

### 2.4.3 Asymptotic expression of MWISE[$\tilde{\pi}_n$]

**Corollary 2.13.** The MWISE expression of the non-recursive CCDF estimator is given by

$$ \mathrm{MWISE}[\tilde{\pi}_n] = \frac{1}{4} I_1 h_n^4 + I_2 R(\mathbf{K}) \frac{1}{nh_n^d} + \mathbf{o} \left( h_n^4 + \frac{1}{nh_n^d} \right). $$

**Proposition 2.14.** Let assumptions (A₁) and (A₄) hold. To minimize the MWISE of $\tilde{\pi}_n$, the bandwidth ($h_n$) must be equal to

$$ \left( \left( d \frac{I_2}{I_1} R(\mathbf{K}) \right)^{\frac{1}{d+4}} n^{-\frac{1}{d+4}} \right). \quad (2.24) $$
Hence, the corresponding MWISE is determined by

$$MWISE[\tilde{\pi}_n] = \frac{d+4}{4d^{\frac{d}{d+4}}} I_2^{\frac{4}{d+4}} I_1^{\frac{d}{d+4}} R(\mathbf{K})^{\frac{4}{d+4}} n^{-\frac{4}{d+4}} + \mathbf{o}\left(n^{-\frac{4}{d+4}}\right).$$

## 2.5 Bandwidth selection

Although theoretical asymptotic study yields the optimal bandwidth, the fact that we do not know the density function makes it hard to interpret it in practice. Hence, kernel smoothing in non-parametric statistics requires the choice of a bandwidth parameter. This choice is crucial to obtain a good rate of consistency of the kernel estimators. It has a significant influence on the size of the bias. One has to find an appropriate bandwidth that produces an estimator which has a good balance between the bias and the variance of the estimator of the function $a(\cdot, \cdot)$ as well as $f(\cdot)$. It is worth noticing that the bandwidth selection methods reported in the literature can be divided into three broad classes: the cross-validation techniques, the plug-in ideas and the bootstrap procedure. In this investigation, we are basically interested in the plug-in method. Altman and Leger (1995) developed an efficient method of bandwidth selection, which minimizes an estimate of the mean weighted integrated squared error, using the density function as a weight function. For this reason, we followed the work of Slaoui (2014a).

### 2.5.1 Plug-in bandwidth selection:

As a result to the plug-in procedure, based on the expression of the MWISE, we estimate the unknown quantities $I_1$ and $I_2$ by elaborating asymptotic unbiased estimators. Basically, we recall $(b_n) \in \mathcal{G}\mathcal{S}(-\delta)$, as introduced in 1.15. In the following and for the sake of simplicity, the kernel $\mathbf{K}$ we shall use is considered as a product of univariate kernels given in (1.14).

Moreover, we assume that $K_b$ stands for a kernel with bandwidth $b_n$ such that $\delta = \frac{2}{5}$, and $K_{b'}^{(2)}$ corresponds to the second derivative of a kernel $K_{b'}$ with the associated bandwidth $b_n$ such that $\delta = \frac{3}{14}$. Note that our choice of the parameter $\delta$ is based on the work of Slaoui (2014a).

In addition, we note:

$$I_1 = \mu^2(K) (J_1 - 2J_2 + J_3),$$

where

$$J_1 = \int_{\mathbb{R}^{d+q}} \left( \sum_{j=1}^{d} a_{jj}^{(2)}(x,y) \right)^2 f_{X,Y}(x,y)dx dy, \quad J_3 = \int_{\mathbb{R}^{d+q}} \left( \sum_{j=1}^{d} f_{Xjj}^{(2)}(x) \right)^2 \pi^2(y|x) f_{X,Y}(x,y)dx dy, \\ J_2 = \int_{\mathbb{R}^{d+q}} \left( \sum_{j=1}^{d} a_{jj}^{(2)}(x,y) \right) \left( \sum_{j=1}^{d} f_{Xjj}^{(2)}(x) \right) \pi(y|x) f_{X,Y}(x,y)dx dy, \quad \mu(K) = \int_{\mathbb{R}} z^2 K(z)dz.$$

#### Recursive estimator $\pi_n$:

To estimate the optimal bandwidth (2.25), we need to estimate $I_1$ and $I_2$. Here we can write

$$a_n(x, y) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^{-d} \chi_k(y) \mathbf{K} \left( \frac{x - X_k}{h_k} \right) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^{-d} \prod_{i=1}^{d} K \left( \frac{x - X_{ki}}{h_k} \right) \chi_{ki}(y)$$

and

$$f_n(x) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^{-d} \mathbf{K} \left( \frac{x - X_k}{h_k} \right) = \Pi_n \sum_{k=1}^{n} \Pi_k^{-1} \gamma_k h_k^{-d} \prod_{i=1}^{d} K \left( \frac{x_i - X_{ki}}{h_k} \right).$$
Estimation of $I_1$:

$$ \hat{J}_1 = \frac{\Pi_n^2}{n} \sum_{\substack{i,j,k=1 \\ i \neq j \neq k}}^n \Pi_j^{-1} \Pi_k^{-1} \gamma_j \gamma_k b_j^{-(d+2)} b_k^{-(d+2)} \left[ \sum_{v=1}^d K_{b'}^{(2)} \left( \frac{X_{iv} - X_{jv}}{b_j'} \right) \prod_{\substack{l=1 \\ l \neq v}}^d K_b \left( \frac{X_{il} - X_{jl}}{b_j} \right) \right] \times \left[ \sum_{v=1}^d K_{b'}^{(2)} \left( \frac{X_{iv} - X_{kv}}{b_k'} \right) \prod_{\substack{l=1 \\ l \neq v}}^d K_b \left( \frac{X_{il} - X_{kl}}{b_k} \right) \right] \prod_{s=1}^q \chi_{js}(Y_{is}) \chi_{ks}(Y_{is}), $$

$$ \hat{J}_2 = \frac{\Pi_n^2}{n} \sum_{\substack{i,j,k,u=1 \\ i \neq j \neq k \neq u}}^n \Pi_j^{-1} \Pi_k^{-1} \gamma_j \gamma_k b_j^{-(d+2)} b_k^{-(d+2)} \left[ \sum_{v=1}^d K_{b'}^{(2)} \left( \frac{X_{iv} - X_{jv}}{b_j'} \right) \prod_{\substack{l=1 \\ l \neq v}}^d K_b \left( \frac{X_{il} - X_{jl}}{b_j} \right) \right] \times \left[ \sum_{v=1}^d K_{b'}^{(2)} \left( \frac{X_{iv} - X_{kv}}{b_k'} \right) \prod_{\substack{l=1 \\ l \neq v}}^d K_b \left( \frac{X_{il} - X_{kl}}{b_k} \right) \right] \prod_{s=1}^q \chi_{us}(Y_{is}) \chi_{js}(Y_{is}), $$

$$ \hat{J}_3 = \frac{\Pi_n^2}{n} \sum_{\substack{i,j,k,u,m=1 \\ i \neq j \neq k \neq u \neq m}}^n \Pi_j^{-1} \Pi_k^{-1} \gamma_j \gamma_k b_j^{-(d+2)} b_k^{-(d+2)} \left[ \sum_{v=1}^d K_{b'}^{(2)} \left( \frac{X_{iv} - X_{jv}}{b_j'} \right) \prod_{\substack{l=1 \\ l \neq v}}^d K_b \left( \frac{X_{il} - X_{jl}}{b_j} \right) \right] \times \left[ \sum_{v=1}^d K_{b'}^{(2)} \left( \frac{X_{iv} - X_{kv}}{b_k'} \right) \prod_{\substack{l=1 \\ l \neq v}}^d K_b \left( \frac{X_{il} - X_{kl}}{b_k} \right) \right] \prod_{s=1}^q \chi_{us}(Y_{is}) \chi_{ms}(Y_{is}), $$

At this stage, we obtain

$$ \hat{I}_1 = \mu^2(K) (\hat{J}_1 - 2\hat{J}_2 + \hat{J}_3). $$

Estimation of $I_2$:

$$ \hat{I}_2 = \frac{\Pi_n}{n} \sum_{\substack{i,k,u=1 \\ i \neq k \neq u}}^n \Pi_k^{-1} \gamma_k b_k^{-1} \prod_{l=1}^d K_b \left( \frac{X_{il} - X_{kl}}{b_k} \right) \prod_{s=1}^q \chi_{us}(Y_{is}) (1 - \chi_{ks}(Y_{is})), $$

As a result, the plug-in estimator of (2.25) is determined by

$$ (h_n) = \left( \left( \frac{d(d+2)}{2(d+4)} \right)^{\frac{1}{d+4}} \left( \frac{\hat{I}_2}{\hat{I}_1} \right)^{\frac{1}{d+4}} R(\mathbf{K})^{\frac{1}{d+4}} n^{-\frac{1}{d+4}} \right), \quad (2.25) $$

Eventually, an estimator of MWISE[$\pi_n$] is specified by

$$ \widehat{MWISE}[\pi_n] = \frac{(d+4)^{\frac{3d+8}{d+4}}}{4^{\frac{d+6}{d+4}} d^{\frac{d}{d+4}} (d+2)^{\frac{d+6}{d+4}}} (\hat{I}_1)^{\frac{d}{d+4}} (\hat{I}_2)^{\frac{4}{d+4}} R(\mathbf{K})^{\frac{1}{d+4}} n^{-\frac{4}{d+4}} + \mathbf{o}(n^{-\frac{4}{d+4}}). $$