# Bandwidth Selection for Recursive Kernel Density Estimators Defined by Stochastic Approximation Method

Authors: Yousri Slaoui
Journal: Journal of Probability and Statistics
 (2014)

---

## Abstract

We propose an automatic selection of the bandwidth of the recursive kernel estimators of a probability density function defined by the stochastic approximation algorithm introduced by Mokkadem et al. (2009a). We showed that, using the selected bandwidth and the stepsize which minimize the MISE (mean integrated squared error) of the class of the recursive estimators defined in Mokkadem et al. (2009a), the recursive estimator will be better than the nonrecursive one for small sample setting in terms of estimation error and computational costs. We corroborated these theoretical results through simulation study.


---

## Body

## 1. Introduction

The problem of automatic choice of smoothing parameters has been widely studied. There are many reasons to use an automatic choice of smoothing. One is in many situations the smoothing which is used by nonexperts. In this paper we focus only on one-dimensional kernel density estimation. The main ideas are useful in all types of nonparametric curve estimation, including regression, distribution, and time series. The bandwidth selection methods studied in the literature can be divided into two broad classes: the cross-validation techniques and the plug-in ideas.There are many varieties of the technique cross-validation: pseudolikelihood cross-validation [1], least squares cross-validation [2], and biased cross-validation [3]. Reviews of all these bandwidth selection methods can be found in Marron [4].Plug-in methods [5], also called “second generation methods” [6], need to use a pilot bandwidth to estimate the unknown quantities. For a choice of pilot bandwidth, a number of approaches have been proposed; see Jones et al. [7] for details and references. An interesting approach to choose the pilot bandwidth is the smoothed bootstrap [8]. In this paper, we developed a specific second generation bandwidth selection method of the recursive kernel estimators of a probability density function defined by the stochastic approximation algorithm introduced by Mokkadem et al. [9].LetX1,…,Xn be independent, identically distributed random variables and let f denote the probability density of X1. To construct a stochastic algorithm, which approximates the function f at a given point x, Mokkadem et al. [9] define an algorithm of search of the zero of the function h:y→f(x)-y. Following Robbins-Monro's procedure, this algorithm is defined by setting f0(x)∈R, and, for all n≥1,
(1)fn(x)=fn-1(x)+γnWn,
where Wn(x) is an “observation” of the function h at the point fn-1(x) and the stepsize (γn) is a sequence of positive real numbers that goes to zero. To define Wn(x), Mokkadem et al. [9] follow the approach of Révész [10, 11] and of Tsybakov [12] and introduce a kernel K (i.e., a function satisfying ∫R‍K(x)dx=1), a bandwidth (hn) (i.e., a sequence of positive real numbers that goes to zero), and set Wn(x)=hn-1K(hn-1(x-Xn))-fn-1(x). Then, the estimator fn to recursively estimate the density function f at the point x can be written as
(2)fn(x)=(1-γn)fn-1(x)+γnhn-1K(hn-1[x-Xn]).
This estimator was introduced by Mokkadem et al. [9] whose large and moderate deviation principles were established by Slaoui [13].Throughout this paper, we suppose thatf0(x)=0 and we let Πn=∏j=1n‍(1-γj); then it follows from (2) that one can estimate f recursively at the point x by
(3)fn(x)=Πn∑k=1n‍Πk-1γkhk-1K(x-Xkhk).
Moreover, it was shown in Mokkadem et al. [9] that the bandwidth which minimizes the MISE of fn depends on the choice of the stepsize (γn); they show in particular that the sequence (γn)=(n-1) belongs to this set, under some conditions of regularity of f, and they show that the bandwidth (hn) must equal
(4)((310)1/5{∫R‍K2(z)dz(∫R‍z2K(z)dz)2∫R‍(f(2)(x))2dx}1/5n-1/5).
The first aim of this paper is to propose an automatic selection of such bandwidth through a plug-in method, and the second aim is to give the conditions under which the recursive estimator fn will be better than the nonrecursive kernel density estimator introduced by Rosenblatt [14] (see also Parzen [15]) and defined as
(5)f~n(x)=1nhn∑k=1n‍K(x-Xkhn).
The simulation results given in Section 3 are corroborating these theoretical results. The remainder of the paper is organized as follows. In Section 2, we state our main results. Section 3 is devoted to our simulation results. We conclude the paper in Section 4.


## 2. Assumptions and Main Results

We define the following class of regularly varying sequences.Definition 1.
Letγ∈R and let (vn)n≥1 be a nonrandom positive sequence. One says that (vn)∈GS(γ) if
(6)lim⁡n→+∞n[1-vn-1vn]=γ.Condition (6) was introduced by Galambos and Seneta [16] to define regularly varying sequences (see also Bojanic and Seneta [17]) and by Mokkadem and Pelletier [18] in the context of stochastic approximation algorithms. Note that the acronym GS stands for [16]. Typical sequences in GS(γ) are, for b∈R, nγ(log⁡n)b, nγ(log⁡log⁡n)b, and so on.The assumptions to which we will refer are as follows.(A1)
K:R→R is a continuous, bounded function satisfying ∫R‍K(z)dz=1, ∫R‍zK(z)=0, and ∫R‍z2K(z)<∞.(A2)
(i)
(γn)∈GS(-α) with α∈]1/2,1].(ii)
(hn)∈GS(-a) with a∈]0,1[.(iii)
lim⁡n→∞(nγn)∈]min⁡{2a,(α-a)/2},∞].(A3)
f is bounded and differentiable and f(2) is bounded.Assumption(A2)(iii) on the limit of (nγn) as n goes to infinity is usual in the framework of stochastic approximation algorithms. It implies in particular that the limit of ([nγn]-1) is finite. Throughout this paper we will use the following notations:
(7)ξ=lim⁡n→∞(nγn)-1,R(K)=∫R‍K2(z)dz,μj(K)=∫R‍zjK(z)dz,Θ(K)=R(K)4/5μ2(K)2/5,Γ(K)=R(K(2))8/7μ2(K)10/7,ζ(K)=R(K(2))8/7μ2(K)-4/7μ4(K),I1=∫R‍f2(x)dx,I2=∫R‍(f(2)(x))2f(x)dx.
In order to measure the quality of our recursive estimator (3), we use the following quantity:
(8)MISE*[fn]=E∫R‍[fn(x)-f(x)]2f(x)dx=∫R‍(E(fn(x))-f(x))2f(x)dx+∫R‍Var⁡(fn(x))f(x)dx.
Moreover, in the case a=α/5, Proposition 1 in Mokkadem et al. [9] shows that
(9)E[fn(x)]-f(x)=12(1-2aξ)hn2f(2)(x)μ2(K)+o(hn2)
and that
(10)Var⁡[fn(x)]=12-(α-a)ξγnhn-1f(x)R(K)+o(γnhn-1).
Then
(11)MISE*[fn]=12-(α-a)ξγnhn-1R(K)I1+14(1-2aξ)2hn4μ22(K)I2+o(γnhn-1+hn4).
The following corollary ensures that the bandwidth which minimizes the MISE* depends on the stepsize (γn) and then the corresponding MISE* depends also on the stepsize (γn).Corollary 2.
Let assumptions(A1)–(A3) hold. To minimize the MISE* of fn, the stepsize (γn) must be chosen in GS(-1) and the bandwidth (hn) must equal
(12)((1-2aξ)2/5(2-(α-a)ξ)1/5{R(K)μ22(K)I1I2}1/5γn1/5).
Then, one has
(13)MISE*[fn]=54(1-2aξ)-2/5(2-(α-a)ξ)-4/5×Θ(K)I14/5I21/5γn4/5+o(γn4/5).The following corollary shows that, for a special choice of the stepsize(γn)=(γ0n-1), which fulfilled that lim⁡n→∞nγn=γ0 and that (γn)∈GS(-1), the optimal value for hn depends on γ0 and then the corresponding MISE* depends on γ0.Corollary 3.
Let assumptions(A1)–(A3) hold. To minimize the MISE* of fn, the stepsize (γn) must be chosen in GS(-1), lim⁡n→∞nγn=γ0, and the bandwidth (hn) must equal
(14)(2-1/5(γ0-25)1/5{R(K)μ22(K)I1I2}1/5n-1/5),
and one then has
(15)MISE*[fn]=54124/5γ02(γ0-2/5)6/5Θ(K)I14/5I21/5n-4/5+o(n-4/5).
Moreover, the minimum of γ02(γ0-2/5)-6/5 is reached at γ0=1; then the bandwidth (hn) must equal
(16)((310)1/5{R(K)μ22(K)I1I2}1/5n-1/5),
and one then has
(17)MISE*[fn]=5422/5(56)6/5Θ(K)I14/5I21/5n-4/5+o(n-4/5).In order to estimate the optimal bandwidth (16), we must estimate I1 and I2. We followed the approach of Altman and Léger [19], which is called the plug-in estimate, and we use the following kernel estimator of I1:
(18)I^1=Πnn∑i,k=1n‍Πk-1γkbk-1Kb(Xi-Xkbk),
where Kb is a kernel and b is the associated bandwidth.In practice, we take(19)bn=n-βmin⁡{s^,Q3-Q11.349},β∈]0,1[
(see Silverman [20]) with s^ the sample standard deviation and Q1, Q3 denoting the first and third quartiles, respectively.The following theorem gives the bias and variance ofI^1.Theorem 4.
Let assumptions(A2)-(A3) hold and suppose that the kernel Kb satisfies assumption (A1) and (bn)∈GS(-β), with β∈]0,1[; one has
(20)Bias[I^1]=12(1-2aξ)bn2μ2(Kb)×∫R‍f(2)(x)f(x)dx+o(bn2),Var⁡[I^1]=12-(α-a)ξγnnbnR(Kb)I1+1n(∫R‍f3(x)dx-I12)+o(1n+γnnbn).The following corollary shows that the bandwidth which minimizes theMISE of I^1 depends on the stepsize (γn) and then the corresponding MISE depends also on the stepsize (γn).Corollary 5.
Let the assumptions of Theorem4 hold. To minimize the MISE of I^1, the stepsize (γn) must be chosen in GS(-1) and the bandwidth (bn) must equal
(21)((1-2aξ)2/5(2-(1-a)ξ)1/5{R(Kb)μ22(Kb)I1(∫R‍f(2)(x)f(x)dx)2}1/5hh×{R(Kb)μ22(Kb)I1(∫R‍f(2)(x)f(x)dx)2}1/5(γnn)1/5).
Then, one has
(22)MISE[I^1]=54(1-2ξ)-2/5(2-(1-a)ξ)-4/5Θ(Kb)I14/5×(∫R‍f(2)(x)f(x)dx)2/5(γnn)4/5+1n(∫R‍f3(x)dx-I12)+o((γnn)4/5).The following corollary shows that, for a special choice of the stepsize(γn)=(γ0n-1), which fulfilled that lim⁡n→∞nγn=γ0 and that (γn)∈GS(-1), the optimal value for bn depends on γ0 and the corresponding MISE depends on γ0.Corollary 6.
Let the assumptions of Theorem4 hold. To minimize the MISE of I^1, the stepsize (γn) must be chosen in GS(-1), lim⁡n→∞nγn=γ0, and the bandwidth (bn) must equal
(23)((γ0-4/5)2/5(2γ0-3/5)1/5{R(Kb)μ22(Kb)I1(∫R‍f(2)(x)f(x)dx)2}1/5n-2/5hh×{R(Kb)μ22(Kb)I1(∫R‍f(2)(x)f(x)dx)2}1/5n-2/5).
Then, one has
(24)MISE[I^1]=1n(∫R‍f3(x)dx-I12)+54γ02×(γ0-45)-2/5(2γ0-35)-4/5Θ(Kb)I14/5×(∫R‍f(2)(x)f(x)dx)2/5n-8/5+o(n-8/5).Moreover, the minimum ofγ02(γ0-4/5)-2/5(2γ0-3/5)-4/5 is reached at γ0=(9+21)/10; then the bandwidth (bn) must equal
(25)(0.6817256{R(Kb)μ22(Kb)I1(∫R‍f(2)(x)f(x)dx)2}1/5n-2/5hh×{R(Kb)μ22(Kb)I1(∫R‍f(2)(x)f(x)dx)2}1/5n-2/5).
Then
(26)Bias[I^1]≃0.5653751×Θ(Kb)1/2I12/5×(∫R‍f(2)(x)f(x)dx)1/5n-4/5+o(n-4/5),Var⁡[I^1]≃1n(∫R‍f3(x)dx-I12)+1.278596×Θ(Kb)I14/5(∫R‍f(2)(x)f(x)dx)2/5×n-8/5+o(n-8/5).
It follows that
(27)MISE[I^1]≃1n(∫R‍f3(x)dx-I12)+1.598245×Θ(Kb)I14/5(∫R‍f(2)(x)f(x)dx)2/5n-8/5+o(n-8/5).
Furthermore, to estimate I2, we introduce the following kernel estimator:
(28)I^2=Πn2n∑i,j,k=1j≠kn‍Πj-1Πk-1γjγkbj′-3bk′-3hhhhhhhh×Kb′(2)(Xi-Xjbj′)Kb′(2)(Xi-Xkbk′),
where Kb′(2) is the second order derivative of a kernel Kb′. The bias and variance of I^2 are computed in the following theorem.Theorem 7.
Let assumptions(A2)-(A3) hold, and suppose that the kernel Kb′ satisfies assumption (A1) and (bn′)∈GS(-β′), with β′∈]0,1[; one has
(29)Bias[I^2]=1(1-2aξ)bn′2μ2(Kb′)×∫R‍f(4)(x)f(2)(x)f(x)dx+14(1-2aξ)2bn′4μ22(Kb′)×∫R‍(f(4)(x))2f(x)dx+112(1-4aξ)bn′4μ4(Kb′)×∫R‍f(6)(x)f(4)(x)f(x)dx-12-ξI2γn+o(bn′4+γn),Var⁡[I^2]=(12-(α-5a)ξ)2γn2nbn′10R(Kb′(2))4×∫R‍f3(x)dx+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(1n+γn2nbn′10).The following corollary ensures that the bandwidth which minimizes theMISE of I^2 depends on the stepsize (γn) and then the corresponding MISE depends also on the stepsize (γn).Corollary 8.
Let the assumptions of Theorem7 hold. To minimize the MISE of I^2, the stepsize (γn) must be chosen in GS(-1) and the bandwidth (bn′) must equal
(30)((52)1/14(1-2aξ)1/7(2-(1-5a)ξ)1/7{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14(γn2n)1/14hh×{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14hh{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14×(γn2n)1/14);
then one has
(31)MISE[I^2]=75(52)2/7(1-2aξ)-10/7×(2-(1-5a)ξ)-4/7Γ(Kb′)×(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)10/7(γn2n)2/7+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(n-1).The following corollary shows that, for a special choice of the stepsize(γn)=(γ0n-1), which fulfilled that lim⁡n→∞nγn=γ0 and that (γn)∈GS(-1), the optimal value for bn′ depends on γ0 and the corresponding MISE depends on γ0.Corollary 9.
Let the assumptions of Theorem7 hold. To minimize the MISE of I^2, the stepsize (γn) must be chosen in GS(-1), lim⁡n→∞nγn=γ0, and the bandwidth (bn′) must equal
(32)((52)1/14γ01/7(γ0-3/7)1/7(2γ0+1/14)1/7{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14n-3/14hh×{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14hh{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14×n-3/14);
then one has
(33)MISE[I^2]=75(52)2/7γ018/7(γ0-3/7)10/7(2γ0+1/14)4/7×Γ(Kb′)(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)10/7n-6/7+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(n-1).Moreover, the minimum ofγ018/7(γ0-3/7)-10/7(2γ0+1/14)-4/7 is reached at γ0=(10+227/2)/14; then the bandwidth (bn′) must equal
(34)(0.9699894{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14n-3/14hh×{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14hh{R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14×n-3/14);
then we have
(35)Bias[I^2]≃1.326127×Γ(Kb′)1/2(∫R‍f3(x)dx)1/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)5/7n-3/7+{0.4396531×Γ(Kb′)∫R‍(f(4)(x))2f(x)dxhhh+0.1760697×ζ(Kb′)hhh×∫R‍f(6)(x)f(2)(x)f(x)dx}×(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)-4/7n-6/7-1.115801×I2n-1+o(n-1),Var⁡[I^2]≃0.703445×Γ(Kb′)(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)10/7n-6/7+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(n-1).
It follows that
(36)MISE[I^2]≃2.462058×Γ(Kb′)(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)10/7n-6/7+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(n-1).
Finally, the plug-in estimator of the bandwidth (hn) using the recursive algorithm (3) must equal
(37)((310)1/5(R(K)μ22(K))1/5(I^1I^2)1/5n-1/5),(38)MISE*[fn]=5422/5(56)6/5Θ(K)I^14/5I^21/5n-4/5+o(n-4/5).
Now, let us recall that the bias and variance of Rosenblatt's estimator f~n are given by
(39)E[f~n(x)]-f(x)=12hn2f(2)(x)μ2(K)+o(hn2),Var⁡[f~n(x)]=1nhnf(x)R(K)+o(1nhn).
It follows that
(40)MISE*[f~n]=1nhnI1R(K)+14hn4I2μ22(K)+o(hn4+1nhn).
To minimize the MISE* of f~n, the bandwidth (hn) must equal
(41)({R(K)μ22(K)I1I2}1/5n-1/5),
and then we have
(42)MISE*[f~n]=54Θ(K)I14/5I21/5n-4/5+o(n-4/5).
To estimate the optimal bandwidth (41), we must estimate I1 and I2. As suggested by Hall and Marron [21], we use the following kernel estimator of I1:
(43)I~1=1n(n-1)bn∑i,j=1i≠jn‍Kb(Xi-Xjbn).
The following lemma gives the bias and variance of I~1.Lemma 10.
Suppose that the kernelKb satisfies assumption (A1), (bn), and assumption (A2)(ii) and f satisfies assumption (A3)(44)Bias[I~1]=12bn2μ2(Kb)∫R‍f(2)(x)f(x)dx+o(bn2),Var⁡[I~1]=1n(n-1)bnR(Kb)I1+1n-1(∫R‍f3(x)dx-I12)+o(1n+1n2bn).
To minimize the MISE of I~1, the bandwidth (bn) must equal
(45)({R(Kb)I1μ22(Kb)(∫R‍f(2)(x)f(x)dx)2}1/5n-2/5);
then one has
(46)Bias[I~1]=12Θ(Kb)1/2I12/5(∫R‍f(2)(x)f(x)dx)1/5×n-4/5+o(n-4/5),Var⁡[I~1]=1n(∫R‍f3(x)dx-I12)+Θ(Kb)I14/5(∫R‍f(2)(x)f(x)dx)2/5×n-8/5+o(n-8/5),MISE[I~1]=1n(∫R‍f3(x)dx-I12)+54Θ(Kb)I14/5(∫R‍f(2)(x)f(x)dx)2/5×n-8/5+o(n-8/5).Furthermore, to estimateI2, we use the following kernel estimator introduced in Hall and Marron [21]:
(47)I~2=1n3bn′6∑i,j,k=1j≠kn‍Kb′(2)(Xi-Xjbn′)Kb′(2)(Xi-Xkbn′).
The following lemma gives the bias and variance of I~2.Lemma 11.
Suppose that the kernelKb′ satisfies assumption (A1), (bn′), and assumption (A2)(ii) and f satisfies assumption (A3)(48)Bias[I~2]=bn′2μ2(Kb′)∫R‍f(4)(x)f(2)(x)f(x)dx+14bn′4μ22(Kb′)∫R‍(f(4)(x))2f(x)dx+112bn′4μ4(Kb′)∫R‍f(6)(x)f(2)(x)f(x)dx+o(bn′4),Var⁡[I~2]=1n3bn′10R(Kb′(2))4∫R‍f3(x)dx+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(1n+1n3bn′10).
To minimize the MISE of I~2, the bandwidth (bn′) must equal
(49)({52R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14hh{52R(Kb′(2))4∫R‍f3(x)dxμ22(Kb′)(∫R‍f(4)(x)f(2)(x)f(x)dx)2}1/14×n-3/14);
then one has
(50)Bias[I~2]≃1.139852×Γ(Kb′)1/2(∫R‍f3(x)dx)1/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)5/7n-3/7+{0.3248158×Γ(Kb′)∫R‍(f(4)(x))2f(x)dxhhh+0.1082719×ζ(Kb′)hhh×∫R‍f(6)(x)f(2)(x)f(x)dx}×(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)-4/7n-6/7+o(n-6/7),Var⁡[I~2]≃0.5197053×Γ(Kb′)(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)10/7n-6/7+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(n-1).
It follows then that
(51)MISE[I~2]≃1.81189×Γ(Kb′)(∫R‍f3(x)dx)2/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)10/7n-6/7+1n(∫R‍(f(2)(x))4f(x)dx-I22)+o(n-1).Then the plug-in estimator of the bandwidth(hn) using the nonrecursive estimator (5) must equal
(52)((R(K)μ22(K))1/5(I~1I~2)1/5n-1/5),(53)MISE*[f~n]=54Θ(K)I~14/5I~21/5n-4/5+o(n-4/5).
The following corollary gives the expected MISE* of the recursive estimator fn and the nonrecursive estimator f~n.Corollary 12.
Let the assumptions of Theorem7 hold. Then
(54)E[MISE*[fn]]=5422/5(56)6/5Θ(K)I14/5I21/5×[(1+0.26522×C1I2-1n-3/7hhh.+(0.08793×C2+0.03521×C3)hhh.×I2-1n-6/7-0.22316×n-1)+o(n-1)]×n-4/5(1+o(1)),E[MISE*[f~n]]=54Θ(K)I14/5I21/5×[(1+0.22797×C1I2-1n-3/7hhh.+(0.06496×C2+0.02165×C3)hhh.×I2-1n-6/7)]n-4/5(1+o(1)),
where
(55)C1=Γ(Kb′)1/2(∫R‍f3(x)dx)1/7×(∫R‍f(4)(x)f(2)(x)f(x)dx)5/7C2=Γ(Kb′)∫R‍(f(4)(x))2f(x)dxhhhhhhh×(∫R‍f3(x)dx)2/7hhhhhhh×(∫R‍f(4)(x)f(2)(x)f(x)dx)-4/7C3=ζ(Kb′)∫R‍f(6)(x)f(2)(x)f(x)dxhhhhhhh×(∫R‍f3(x)dx)2/7hhhhhhh×(∫R‍f(4)(x)f(2)(x)f(x)dx)-4/7.The following theorem gives the conditions under which the expectedMISE* of the recursive estimator fn will be smaller than the expected MISE* of the nonrecursive estimator f~n.Theorem 13.
Let the assumptions of Theorem7 hold and let the bandwidth (hn) equal (37) and the stepsize (γn)=(n-1). One has
(56)E[MISE*[fn]]E[MISE*[f~n]]<1forsmallsamplesetting.Then, the expectedMISE* of the recursive estimator defined by (3) is smaller than the expected MISE* of the nonrecursive estimator defined by (5) for small sample setting.


## 3. Simulation

The aim of our simulation study is to compare the performance of the nonrecursive Rosenblatt's estimator defined in (5) with that of the recursive estimators defined in (3).When applyingfn one needs to choose three quantities.(i)
For the functionK, we choose the normal kernel.(ii)
The stepsize(γn)=(n-1).(iii)
The bandwidth(hn) is chosen to be equal to (37).When applyingf~n one needs to choose two quantities. (i)
For the functionK, as in the recursive framework, we use the normal kernel.(ii)
The bandwidth(hn) is chosen to be equal to (52).In order to investigate the comparison between the two estimators, we consider two densities off: the standard normal N(0,1) distribution (see Table 1) and the exponential E(1/2) distribution (see Table 2). For each of these two cases, 500 samples of sizes n=50 and 100 were generated. For each fixed bandwidth h, we computed the mean and the standard deviation (over the 500 samples) of I1, I2, hn, and MISE*. The plug-in estimators (37) and (52) require two kernels to estimate I1 and I2. In both cases we use the normal kernel with bn and bn′ given in (19), with β equal, respectively, to 2/5 and 3/14. Both tables show that (1) the bias, respectively, and the standard deviation of I1 using the recursive algorithm (3) are very similar to the bias, respectively, and the standard deviation of I1 using the nonrecursive estimator (5), (2) the bias, respectively, and the standard deviation of I2 using the recursive algorithm (3) are always smaller than the bias, respectively, and the standard deviation of I2 using the nonrecursive estimator (5), (3) the mean, respectively, and the standard deviation of the bandwidths selected by the recursive estimator (3) are always smaller than the bias, respectively, and the standard deviation of the bandwidths selected by the nonrecursive estimator (5), and (4) the mean, respectively, and the standard deviation of the MISE* of the bandwidths selected by the recursive estimator (3) are always smaller than the bias, respectively, and the standard deviation of MISE* of the bandwidths selected by the nonrecursive estimator (5). In Tables 1 and 2 the Ref. column could be used as a reference for the mean of I1 and I2.Table 1
X
⇝
N
(
0,1
).
n
=
50
n
=
100
Rosenblatt
Recursive
Rosenblatt
Recursive
Ref.
Mean
sd
Mean
sd
Mean
sd
Mean
sd
I
1
0.2821
0.2780
0.0357
0.2864
0.0358
0.2788
0.0255
0.2866
0.0257
I
2
0.0612
0.0994
0.2744
0.0690
0.1697
0.0749
0.1908
0.0575
0.1078
h
n
0.4801
0.1435
0.4408
0.1399
0.4347
0.1169
0.3894
0.1197
MISE
*
0.0801
0.0288
0.0746
0.0280
0.0649
0.0203
0.0621
0.0195Table 2
X
⇝
E
(
1
/
2
).
n
=
50
n
=
100
Rosenblatt
Recursive
Rosenblatt
Recursive
Ref.
Mean
sd
Mean
sd
Mean
sd
Mean
sd
I
1
0.25
0.1930
0.0331
0.1768
0.0303
0.1957
0.0232
0.1839
0.0221
I
2
0.0078
0.0353
0.0598
0.0196
0.0327
0.0369
0.0336
0.0206
0.0199
h
n
0.5825
0.1203
0.5049
0.0989
0.4611
0.0598
0.4032
0.0525
MISE
*
0.0435
0.0166
0.0381
0.0144
0.0407
0.0100
0.0363
0.0091Figures1 and 2 show boxplots of the selected bandwidths by the two algorithms (3) and (5), respectively. For samples of size 50 and 100, the bandwidths selected by the recursive estimator (3) are always smaller than the bandwidths selected by the nonrecursive estimator (5).The bandwidths minimizing theMISE* of the nonrecursive estimator (5) and the recursive estimator (3) for 500 samples, respectively, of size n=50 (a) and of size n=100 (b) from a normal distribution.
(a)
(b)The bandwidths minimizing theMISE* of the nonrecursive estimator (5) and the recursive estimator (3) for 500 samples, respectively, of size n=50 (a) and of size n=100 (b) from an exponential distribution of rate 1/2.
(a)
(b)Figures3 and 4 show boxplots of the expected MISE* by the two algorithms (3) and (5), respectively. For samples of size 50 and 100, the expected MISE* of the selected bandwidths by the recursive estimator (3) are always smaller than the expected MISE* of the selected bandwidths by the nonrecursive estimator (5).TheMISE* of the optimal bandwidth of the nonrecursive estimator (5) and the recursive estimator (3) for 500 samples, respectively, of size n=50 (a) and of size n=100 (b) from a normal distribution.
(a)
(b)TheMISE* of the optimal bandwidth of the nonrecursive estimator (5) and the recursive estimator (3) for 500 samples, respectively, of size n=50 (a) and of size n=100 (b) from an exponential distribution of rate 1/2.
(a)
(b)In order to give some comparative elements with nonrecursive estimator (5), including computational costs, we consider 500 samples of size 50 generated from a standard normal distribution N(0,1); moreover, we suppose that we receive an additional 500 samples of size 50 generated also from a standard normal distribution N(0,1). Performing the two methods, the running time using the recursive estimator defined by algorithm (2) with stepsize (γn)=(n-1) and the bandwidth (hn) given in (37) was roughly 6880 s on the author's workstation, while the running time using the nonrecursive estimator defined by algorithm (5) with the bandwidth (hn) given in (52) was roughly 14080 s on the author's workstation.This simulation study shows the good performance of the recursive estimator defined by algorithm (3) with stepsize (γn)=(n-1) and the bandwidth (hn) given in (37) for small sample setting.


## 4. Conclusion

In this paper we proposed an automatic selection of the bandwidth of the recursive kernel estimators of a probability density function defined by the stochastic approximation algorithm (2). We showed that, using the selected bandwidth and the stepsize (γn)=(n-1) (the stepsize which minimizes the MISE of the class of the recursive estimators defined in Mokkadem et al. [9]), the recursive estimator will be better than the nonrecursive one for small sample setting. The simulation study corroborated these theoretical results. Moreover, the simulation results indicate that the proposed recursive estimator has more computing efficiency than the nonrecursive estimator.In conclusion, the proposed method allowed us to obtain better results than the nonrecursive estimator proposed by Rosenblatt [14] for small sample setting. Moreover, we plan to make an extension of our method in the future and to consider the case of a regression function as in Härdle and Marron [22] in recursive way (see Mokkadem et al. [23]) and the case of time series as in Hart and Vieu [24] in recursive way.



---

*Source: 739640-2014-06-02.xml*