# A Least Squares Method for Variance Estimation in Heteroscedastic Nonparametric Regression

Authors: Yuejin Zhou; Yebin Cheng; Tiejun Tong
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

Interest in variance estimation in nonparametric regression has grown greatly in the past several decades. Among the existing methods, the least squares estimator in Tong and Wang (2005) is shown to have nice statistical properties and is also easy to implement. Nevertheless, their method only applies to regression models with homoscedastic errors. In this paper, we propose two least squares estimators for the error variance in heteroscedastic nonparametric regression: the intercept estimator and the slope estimator. Both estimators are shown to be consistent and their asymptotic properties are investigated. Finally, we demonstrate through simulation studies that the proposed estimators perform better than the existing competitor in various settings.


---

## Body

## 1. Introduction

Consider the nonparametric regression model(1)
y
i
=
f
(
x
i
)
+
ε
i
,
i
=
1
,
…
,
n
,
where y
i are observations, x
i are design points with 0
≤
x
1
≤
⋯
≤
x
n
≤
1, f
(
·
) is an unknown mean function, and ε
i are independent random errors with mean zero and variance c
i
σ
2, respectively. In the special case when c
i are all the same, model (1) reduces to a homoscedastic nonparametric regression. In this paper, we are interested in estimating the variance σ
2 in the situation when c
i are not all the same but known constants. Note that such a setting can arise in various situations. As an illustration, we consider a regression model with r
i repeated observations on design points x
i, respectively, where the measurement errors are normal. If in practice we only report the average values on each design point, we have the new model as y
-
i
=
f
(
x
i
)
+
ε
-
i, where var
(
ε
-
i
)
=
c
i
σ
2 with c
i
=
1
/
r
i.Needless to say, an accurate estimate of variance is important in nonparametric regression. For instance, it is required in constructing confidence bands, in choosing the amount of smoothing, in testing the goodness of fit, and in estimating the detection limits of immunoassay [1–8]. In the past several decades, researchers have proposed many methods for estimating σ
2, especially when the regression model is homoscedastic. Among the existing methods, one popular class is referred to as difference-based estimators. The first-order difference-based estimator was proposed in Rice [9],
(2)
σ
^
R
2
=
1
2
(
n
-
1
)
∑
i
=
2
n
‍
(
y
i
-
y
i
-
1
)
2
.
Assume that f
(
·
) is a Lipschitz continuous function and max
⁡
2
≤
i
≤
n
{
x
i
-
x
i
-
1
}
=
O
(
1
/
n
). Note that y
i
-
y
i
-
1
=
f
(
x
i
)
-
f
(
x
i
-
1
)
+
ε
i
-
ε
i
-
1
≈
ε
i
-
ε
i
-
1 as n
→
∞. Therefore, σ
^
R
2 is an asymptotically unbiased estimator of σ
2. Since then, many difference-based estimators have been proposed in the literature. For instance, Gasser et al. [10] proposed a second-order difference-based estimator. Hall et al. [11] proposed an mth-order difference-based estimator with m
≥
2 a finite number. Other significant works include Dette et al. [12], Müller et al. [13], Tong et al. [14], Du and Schick [15], and Wang et al. [16], among others. Furthermore, Brown and Levine [17], Wang et al. [18], and Cai and Wang [19] considered the difference-based kernel and wavelet estimators for the variance function in nonparametric regression. Note that the difference-based estimators do not require an estimate of the mean function and so are popular in practice.As a variation of the difference-based estimation, Tong and wang [20] proposed a least squares estimator of σ
2. Let the lag-k Rice estimator be
(3)
σ
^
R
2
(
k
)
=
1
2
(
n
-
k
)
∑
i
=
k
+
1
n
‍
(
y
i
-
y
i
-
k
)
2
.
For the equally spaced design with x
i
=
i
/
n, it can be shown that σ
^
R
2
(
k
)
=
σ
2
+
J
d
k
+
o
(
d
k
) for any k
=
o
(
n
), where J
=
∫
0
1
‍
(
f
′
(
x
)
)
2
d
x
/
2 and d
k
=
k
2
/
n
2. This indicates that the lag-k Rice estimators are always positively biased estimators of σ
2, especially when the sample size n is small. To reduce bias, Tong and Wang regressed σ
^
R
2
(
k
) on d
k using a simple linear regression and then estimate σ
2 as the intercept. The least squares estimator achieves the asymptotically optimal rate that is usually possessed by residual-based estimators only. In addition, Tong et al. [21] established the asymptotic normality and also demonstrated the efficiency of the least squares estimator. We also note that Park et al. [22] investigated the least squares method in small sample nonparametric regression via a local quadratic approximation to determine the regressor and weights.The aforementioned methods have significantly advanced our understanding on the difference-based estimation of the error variance. Nevertheless, most of the above methods, including the least squares method, only applied to nonparametric regression models with homoscedastic errors. In practice, it is not uncommon that the errors may have different variances. In such situations, we note that the bias term of the least squares estimator in Tong and Wang [20] will be significantly enlarged; for more details, see Sections 2 and 3. Inspired by this, we propose two adaptive least squares estimators for the residual variance in heteroscedastic nonparametric regression.The remainder of this paper is organized as follows. In Section2, we propose two least squares estimators for the error variance: the intercept estimator and the slope estimator. In Section 3, we investigate the asymptotic properties of the proposed estimators and present some theoretical results including the asymptotic normalities of the estimators. In Section 4, we conduct simulation studies to evaluate the proposed estimators and compare them with the existing competitor in the literature. We then conclude the paper in Section 5 with a brief discussion and provide the technical proofs in Section 6.


## 2. Methodology

For model (1), without loss of generality, we assume that ∑
i
=
1
n
‍
c
i
=
n. In matrix notation, the model is written as
(4)
y
=
f
+
ε
,
where y
=
(
y
1
,
y
2
,
…
,
y
n
)
′, f
=
(
f
(
x
1
)
,
f
(
x
2
)
,
…
,
f
(
x
n
)
)
′, and ε
=
(
ε
1
,
ε
2
,
…
,
ε
n
)
′. The covariance matrix of ε is σ
2
Σ, where
(5)
Σ
=
(
c
1
0
⋯
0
0
c
2
⋯
0
⋯
⋯
⋯
⋯
0
0
⋯
c
n
)
.
When Σ
=
I, namely, c
i
=
1 for all i, it reduced to the homoscedastic setting in Tong and Wang [20]. In this paper, we assume that the c
i values are not all the same.For this setting, one naive approach is to apply the transformationΣ
-
1
/
2
y
=
Σ
-
1
/
2
f
+
Σ
-
1
/
2
ε. Through this transformation the errors become homogeneous. Nevertheless, meanwhile, it makes the transformed mean function Σ
-
1
/
2
f no longer a Lipschitz continuous function. Specifically, if c
i
≠
c
i
-
1 and f
(
x
i
)
≠
0, the difference f
(
x
i
)
/
c
i
-
f
(
x
i
-
1
)
/
c
i
-
1 will not be negligible when x
i
-
x
i
-
1 tends to be zero. As a consequence, the difference-based methods fail to apply in such situations.To advance the research project, we reconsider the lag-k Rice estimator defined in Tong and Wang [20]. Suppose that f has a bounded first derivative. For model (4), the expectation of the lag-k Rice estimator is
(6)
E
(
σ
^
R
2
(
k
)
)
=
a
k
σ
2
+
b
k
J
+
O
(
k
3
n
2
(
n
-
k
)
)
+
o
(
1
n
2
)
,
where a
k
=
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
/
2
(
n
-
k
), b
k
=
k
2
/
n
2, and J
=
∫
0
1
‍
(
f
′
(
x
)
)
2
d
x
/
2. Note that a
k
=
(
n
+
c
k
+
1
+
⋯
+
c
n
-
k
)
/
(
n
+
n
-
2
k
)
≠
1 when c
k
+
1
+
⋯
+
c
n
-
k
≠
n
-
2
k. Therefore, for model (4) with heteroscedastic errors, it is not guaranteed that σ
^
R
2
(
k
) is an asymptotically unbiased estimator of σ
2.In what follows, we develop two new estimators forσ
2: (
i
) the first method estimates σ
2 as the intercept and (
ii
) the second method estimates σ
2 as the slope. For the first method, we let z
k
=
σ
^
R
2
(
k
)
/
a
k and t
k
=
b
k
/
a
k. Then, for any k
=
o
(
n
), we have
(7)
E
(
z
k
)
=
σ
2
+
J
t
k
+
o
(
t
k
)
.
Now treating z
k as the response variable and t
k as the independent variable, we fit the following simple linear regression and estimate σ
2 as the fitted intercept,
(8)
z
k
=
α
+
β
t
k
+
e
k
,
k
=
1,2
,
…
,
m
,
where e
k are the random errors and m is the total number of pairs used for the fit. Note that z
k involves (
n
-
k
) pairs of difference; we assign weights w
k
=
(
n
-
k
)
/
N, where N
=
(
n
-
1
)
+
⋯
+
(
n
-
m
)
=
m
n
-
m
(
m
+
1
)
/
2, to the response variable z
k. We then fit the linear model (8) using the weight least squares that minimizes the weighted sum of squares ∑
k
=
1
m
‍
w
k
(
z
k
-
α
-
β
t
k
)
2. Specifically, the estimated error variance is
(9)
σ
^
1
2
=
α
^
=
z
-
w
-
β
^
t
-
w
,
where z
-
w
=
∑
k
=
1
m
‍
w
k
z
k, t
-
w
=
∑
k
=
1
m
‍
w
k
t
k, and β
^
=
∑
k
=
1
m
‍
w
k
z
k
(
t
k
-
t
-
w
)
/
∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2. Let d
0
=
0 and
(10)
d
k
=
1
a
k
(
1
-
(
t
k
-
t
-
w
)
t
-
w
∑
k
=
1
m
w
k
(
t
k
-
t
-
w
)
2
)
,
k
=
1
,
…
,
m
.
The quadratic form of σ
^
1
2 can be represented as σ
^
1
2
=
y
′
D
y
/
tr
⁡
(
D
Σ
), where D
=
(
d
i
j
)
n
×
n is a symmetric matrix with d
i
j
=
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
{
i
-
1
,
n
-
i
,
m
}
‍
d
k for i
=
j, d
i
j
=
-
d
|
i
-
j
| for 0
<
|
i
-
j
|
≤
m, and d
i
j
=
0 otherwise.For the second method, we fit the linear regression with two independent variablesa
k and b
k and with no intercept term. Specifically, we fit
(11)
σ
^
R
2
(
k
)
=
β
1
a
k
+
β
2
b
k
+
ξ
k
,
k
=
1,2
,
…
,
m
,
where ξ
k are the random errors associated with the linear regression. We then estimate σ
2 as the fitted slope β
^
1. For ease of notation, let s
k
=
σ
^
R
2
(
k
). By minimizing the weighted sum of squares ∑
k
=
1
m
‍
w
k
(
s
k
-
β
1
a
k
-
β
2
b
k
)
2, we have the second estimator of σ
2 as
(12)
σ
^
2
2
=
β
^
1
=
∑
k
=
1
m
w
k
a
k
b
k
∑
k
=
1
m
w
k
b
k
s
k
-
∑
k
=
1
m
w
k
b
k
2
∑
k
=
1
m
w
k
a
k
s
k
(
∑
k
=
1
m
w
k
a
k
b
k
)
2
-
∑
k
=
1
m
w
k
a
k
2
∑
k
=
1
m
w
k
b
k
2
.
Let v
0
=
0 and
(13)
v
k
=
b
k
∑
k
=
1
m
w
k
a
k
b
k
-
a
k
∑
k
=
1
m
w
k
b
k
2
(
∑
k
=
1
m
w
k
a
k
b
k
)
2
-
∑
k
=
1
m
w
k
a
k
2
∑
k
=
1
m
w
k
b
k
2
,
111111111111111111111
11111
k
=
1
,
…
,
m
.
It is easy to verify that σ
^
2
2 has the quadratic form σ
^
2
2
=
y
′
H
y
/
tr
⁡
(
H
Σ
), where H
=
(
h
i
j
)
n
×
n is a symmetric matrix with h
i
j
=
∑
k
=
1
m
‍
v
k
+
∑
k
=
0
min
⁡
{
i
-
1
,
n
-
i
,
m
}
‍
v
k for i
=
j, h
i
j
=
-
v
|
i
-
j
| for 0
<
|
i
-
j
|
≤
m, and h
i
j
=
0 otherwise.


## 3. Main Results

This section investigates the statistical properties of the proposed least squares estimators. Note thatσ
^
1
2 in (9) and σ
^
2
2 in (12) are two similar estimators, except that (9) treats e
k as i.i.d. random errors and (12) treats ξ
k
=
a
k
e
k as i.i.d. random errors. For simplicity, in what follows, we present the asymptotic results for σ
^
1
2 only. To evaluate the achievement of the proposed estimators, we will also investigate the behavior of σ
^
TW
2 in Tong and Wang [20] under the new model (4). Recall that for σ
^
TW
2, we have
(14)
σ
^
TW
2
=
y
′
D
~
y
2
N
,
where d
~
k
=
1
-
b
-
w
(
b
k
-
b
-
w
)
/
∑
k
=
1
m
‍
w
k
(
b
k
-
b
-
w
)
2, b
-
w
=
∑
k
=
1
m
‍
w
k
b
k, and D
~
=
(
d
~
i
j
)
n
×
n is a symmetric matrix with d
~
i
j
=
∑
k
=
1
m
‍
d
~
k
+
∑
k
=
0
min
⁡
{
i
-
1
,
n
-
i
,
m
}
‍
d
~
k for 1
≤
i
=
j
≤
n, d
~
i
j
=
-
d
~
|
i
-
j
| for 0
<
|
i
-
j
|
≤
m, and d
~
i
j
=
0 otherwise.Theorem 1.
For the equally spaced design, the estimatorσ
^
1
2 in (9) is an unbiased estimator of σ
2 when f is a linear function, regardless of the choice of m and c
i. Under the same setting, however, the estimator σ
^
T
W
2 in (14) does not preserve the unbiasedness property. More specifically, the bias term of σ
^
T
W
2 has the expression
(15)
Bias
(
σ
^
T
W
2
)
=
{
1
2
N
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
-
1
}
σ
2
.Theorem 2.
Assume thatf has a bounded second derivative and E
(
ε
4
)
<
∞ with ε
=
ε
1
/
c
1. When max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
1
), for any m
=
n
r with 0
<
r
<
1 and the equally spaced design, then
(16)
Bias
(
σ
^
1
2
)
=
O
(
m
2
n
2
)
,
Var
⁡
(
σ
^
1
2
)
=
C
1
n
Var
⁡
(
ε
2
)
+
o
(
1
n
)
,
where
(17)
C
1
=
n
4
N
2
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
j
)
2
.
As a comparison, the bias and variance of σ
^
T
W
2 are
(18)
Bias

(
σ
^
T
W
2
)
=
{
1
2
N
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
-
1
}
σ
2
+
o
(
m
2
n
2
)
,
Var
⁡
(
σ
^
T
W
2
)
=
C
2
n
Var
⁡
(
ε
2
)
+
o
(
1
n
)
,
where
(19)
C
2
=
n
4
N
2
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
~
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
~
j
)
2
.Theorem 3.
Assume thatf has a bounded second derivative and E
(
ε
4
)
<
∞ with ε
=
ε
1
/
c
1. When max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
n
s
) with 0
<
s
<
2
/
5, for any m
=
n
r with 0
<
r
<
1 and the equally spaced design, then
(20)
Bias

(
σ
^
1
2
)
=
O
(
m
2
n
2
)
,
Var
⁡
(
σ
^
1
2
)
=
O
(
1
n
1
-
s
)
.
As a comparison,
(21)
Bias

(
σ
^
T
W
2
)
=
O
(
m
n
1
-
s
)
,
Var
⁡
(
σ
^
T
W
2
)
=
O
(
1
n
1
-
s
)
.Theorem 4.
Assume thatf has a bounded second derivative and E
(
ε
4
)
<
∞ with ε
=
ε
1
/
c
1. For max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
n
s
) with 0
<
s
<
1
/
4 and any m
=
n
r with 0
<
r
<
1
/
2, then
(22)
n
δ
(
σ
^
1
2
-
σ
2
)
⟶
d
N
(
0
,
(
γ
4
-
1
)
σ
4
)
,
as
n
⟶
∞
,
where δ
=
∑
i
=
1
n
‍
c
i
2
/
n, γ
4
=
E
(
ε
1
4
)
/
(
c
1
2
σ
4
), and →
d denotes convergence in distribution.The proofs of the theorems are given in Section6, respectively. Theorems 1 and 2 indicate that σ
^
1
2 is an unbiased or asymptotically unbiased estimator of σ
2 whereas σ
^
TW
2 is not. The comparison on the asymptotic variances, or equivalently on C
1 and C
2, will be presented in Section 4. Furthermore, when the heteroscedasticity level is high, Theorem 3 shows that the bias term of σ
^
TW
2 is getting more severe so that it does not remain to be a consistent estimator. The asymptotic normality in Theorem 4 can be used to construct confidence intervals for σ
2. When n
>
δ
(
γ
4
-
1
)
z
α
/
2
2, an approximate 1
-
α confidence interval for σ
2 is
(23)
(
σ
^
1
2
{
1
+
z
α
/
2
δ
(
γ
4
-
1
)
/
n
}
,
σ
^
1
2
{
1
-
z
α
/
2
δ
(
γ
4
-
1
)
/
n
}
)
,
where z
α is the upper α-th percentile of the standard normal distribution. When ε
i are from normal distribution with variance c
i
σ
2, we have γ
4
=
3 so that the confidence interval is fully specified. In general, we need to give an estimate for the unknown γ
4.


## 4.  Simulation Studies

In this section, we conduct simulation studies to evaluate the finite sample performance of the proposed estimators,σ
^
1
2 and σ
^
2
2. Their performance will also be compared with the estimator σ
^
TW
2. Let x
i
=
i
/
n for i
=
1
,
…
,
n. Throughout the simulations, we choose the bandwidth m
=
n
1
/
3, as suggested in Tong and Wang [20].Our first simulation study considers only onec
i value being different from the others. Specifically, for a given location j, we let c
j
=
n
c
/
(
c
+
n
-
1
) and c
i
=
n
/
(
c
+
n
-
1
) for any i
≠
j, where c is a constant. Note that ∑
i
=
1
n
‍
c
i
=
n is satisfied. In this study, we let c
=
30. To investigate the behavior of the estimators along with the variance pattern, we consider the mean function f
=
5
x and f
=
5
sin
(
2
π
x
) and σ
=
0.5 and σ
=
2, respectively. Given the c
i and σ values, we then simulate ε
i independently from N
(
0
,
c
i
σ
2
). With 1000 repetitions, we plot the relative mean squared errors, MSE
/
(
2
σ
4
/
n
), along with the location j for n
=
30 in Figure 1. It is evident that our estimators σ
^
1
2 and σ
^
2
2 perform better than σ
^
TW
2 in most locations. To check the behavior near the boundary, we also plot the values of C
1 and C
2 along with the location j for n
=
30,50,100 (chosen c
=
30) and n
=
500 (chosen c
=
100) in Figure 2. Combining Figures 1 and 2, we recommend the use of the new estimators when no significant different variance appears in the boundaries.Plots of the relative MSEs of the estimators versus the locationj for n
=
30. The dotted, dashed, and solid lines correspond to σ
^
1
2, σ
^
2
2, and σ
^
TW
2, respectively.
(a)
(b)
(c)
(d)Plots ofC
1 in Var
⁡
(
σ
^
1
2
) and C
2 in Var
⁡
(
σ
^
TW
2
) versus the location j, under various n. The dashed and solid lines correspond to C
1 and C
2, respectively.
(a)
(b)
(c)
(d)Our second simulation study is to investigate the average improvement ofσ
^
1
2 and σ
^
2
2 over σ
^
TW
2 when one or more variances are different from the others. To proceed, we consider three mean functions,
(24)
f
1
(
x
)
=
5
sin
(
π
x
)
,
f
2
(
x
)
=
5
sin
(
2
π
x
)
,
f
3
(
x
)
=
5
sin
(
4
π
x
)
,
two standard deviations, σ
=
0.5 and 2, and three sample sizes, n
=
30, 100, and 500, respectively. In total, there are 18 combinations. The c values corresponding to n
=
30,100 and 500 are c
=
30, 100, and 200, respectively. We then randomly sample (
i
) one location or (
ii
) five locations from the set {
m
,
…
,
n
-
m
} without replacement. For (
i
), the choice of the c
i values follows the previous study. For (
ii
) with the five locations L
=
{
j
1
,
…
,
j
5
}, we let c
j
=
n
c
/
(
5
c
+
n
-
5
) for j
∈
L and c
i
=
n
/
(
c
+
n
-
1
) for i
∉
L. This results in ∑
i
=
1
n
‍
c
i
=
n. For each combination setting, we repeat the simulation 1000 times and report the relative MSEs in Table 1 for (
i
) and in Table 2 for (
ii
). From the simulation results, we observe that σ
^
1
2 and σ
^
2
2 have smaller relative MSEs than σ
^
TW
2 in all the settings. In addition, we note that the performances of σ
^
1
2 and σ
^
2
2 are almost identical.Table 1
The relative MSEs ofσ
^
1
2, σ
^
2
2, and σ
^
TW
2 for scenario (i) under various simulation settings.
n
σ
f
σ
^
1
2
σ
^
2
2
σ
^
TW
2
30
0.5
f
1
8.592259
8.588712
8.971043
f
2
9.105655
9.109716
9.59135
f
3
27.52861
27.93159
29.50932
2
f
1
8.568082
8.564596
8.922776
f
2
8.577039
8.57355
8.937509
f
3
8.655637
8.653321
9.055993
100
0.5
f
1
24.68565
24.67358
25.16409
f
2
24.63531
24.63209
25.13371
f
3
25.1026
25.10747
25.67267
2
f
1
24.64412
24.63515
25.1233
f
2
24.62441
24.61774
25.1069
f
3
24.62372
24.61822
25.11145
500
0.5
f
1
47.85348
47.85206
48.00883
f
2
47.86474
47.8618
48.01908
f
3
47.88284
47.87881
48.03735
2
f
1
47.85314
47.85136
48.00809
f
2
47.85561
47.85345
48.01027
f
3
47.85829
47.85587
48.01283Table 2
The relative MSEs ofσ
^
1
2, σ
^
2
2, and σ
^
TW
2 for scenario (ii) under various simulation settings.
n
σ
f
σ
^
1
2
σ
^
2
2
σ
^
TW
2
30
0.5
f
1
5.356385
5.358625
5.76456
f
2
5.825295
5.844773
6.410941
f
3
24.08339
24.74582
27.14087
2
f
1
5.291982
5.292094
5.659067
f
2
5.284907
5.285037
5.659967
f
3
5.367277
5.370764
5.809319
100
0.5
f
1
14.70915
14.70799
15.1483
f
2
14.6843
14.68265
15.1374
f
3
15.01971
15.02443
15.58368
2
f
1
14.71244
14.71141
15.14792
f
2
14.70282
14.70165
15.13869
f
3
14.69388
14.6925
15.13526
500
0.5
f
1
42.08678
42.08626
42.35964
f
2
42.09082
42.0904
42.36408
f
3
42.10065
42.10021
42.37591
2
f
1
42.0839
42.08347
42.3567
f
2
42.08482
42.08442
42.35764
f
3
42.08672
42.08632
42.35965


## 5. Conclusion

In this paper, we have proposed two least squares estimators for the error variance in heteroscedastic nonparametric regression: the intercept estimator and the slope estimator. Both estimators are shown to be consistent and their asymptotic properties are investigated, including the consistency and the asymptotic normalities. Simulation studies indicate that the proposed estimators perform better than the existing competitor in most settings. In the boundaries, however, we note that the proposed estimators behave not as well as expected when significantly different variances appear in the boundaries of design points. As a practical rule, we have suggested adopting the boundaries as[
1
,
m
) and (
n
-
m
,
n
]. Further research may be necessary in this direction.


## 6. Proofs

This section provides the technical proofs of the theorems in Section3. To prove the theorems, we first establish two lemmas. For ease of notation, let f
i
=
f
(
x
i
) and f
i
′
=
f
′
(
x
i
).Lemma 5.
Assume thatf has a bounded second derivative. When max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
1
), for any m
=
n
r with 0
<
r
<
1 and the equally spaced design, then (a)
∑
k
=
1
m
‍
k
t
d
k
=
O
(
m
t
+
1
), t
=
0,1
,
2,3;
(b)
f
′
D
f
=
O
(
m
3
/
n
);
(c)
f
′
D
Σ
D
′
f
=
o
(
m
4
/
n
);
(d)
f
′
D
~
f
=
o
(
m
3
/
n
);
(e)
f
′
D
~
Σ
D
~
′
f
=
o
(
m
4
/
n
).Proof.
(
a
) For simplicity, we prove only for t
=
2. Let u
k
=
1
-
(
(
t
k
-
t
-
w
)
t
-
w
/
∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2
) and υ
=
t
-
w
/
∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2. By the definition of a
k and ∑
i
=
1
n
‍
c
i
=
n, we have
(25)
n
-
k
n
<
1
a
k
<
2
(
n
-
k
)
n
,
k
=
1,2
,
…
,
m
.
First, we consider the upper bound of∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2. We know
(26)
∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2
=
∑
k
=
1
m
‍
w
k
t
k
2
-
t
-
w
2
<
∑
k
=
1
m
‍
w
k
t
k
2
.
Thus, we have
(27)
∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2
<
4
m
4
5
n
4
+
o
(
m
4
n
4
)
.
Next, we consider the lower bound of∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2. By the definition, we can know
(28)
k
2
(
n
-
k
)
n
3
<
t
k
<
2
k
2
(
n
-
k
)
n
3
,
m
2
3
n
2
+
o
(
m
2
n
2
)
<
t
-
w
<
2
m
2
3
n
2
+
o
(
m
2
n
2
)
.
Let α
k
=
k
2
(
n
-
k
)
/
n
3
-
2
m
2
/
3
n
2 and β
k
=
2
k
2
(
n
-
k
)
/
n
3
-
m
2
/
3
n
2. Then, α
k and β
k are monotonically increasing about k, and
(29)
α
k
+
o
(
m
2
n
2
)
<
t
k
-
t
-
w
<
β
k
+
o
(
m
2
n
2
)
,
k
=
1,2
,
…
,
m
.
Note that β
k is a monotonically increasing function of k for 1
≤
k
≤
m
=
o
(
n
) with β
1
<
0 and β
m
>
0. Let m
1 be the unique integer such that β
m
1
≤
0 and β
m
1
+
1
>
0. Therefore, we have
(30)
∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2
>
∑
k
=
1
m
1
‍
w
k
(
t
k
-
t
-
w
)
2
>
∑
k
=
1
m
1
‍
w
k
β
k
2
+
o
(
m
4
n
4
)
=
4
m
1
4
5
n
4
-
4
m
m
1
3
9
n
4
+
m
1
m
3
9
n
4
+
o
(
m
4
n
4
)
.
Letm
1
=
c
m with 0
<
c
<
1. It is easy to verify that c
>
1
/
3. Then, 4
m
1
4
/
5
n
4
-
4
m
m
1
3
/
9
n
4
+
m
1
m
3
/
9
n
4
=
(
m
4
/
n
4
)
(
(
4
/
5
)
c
4
-
(
4
/
9
)
c
3
+
(
1
/
9
)
c
). Let g
(
c
)
=
(
4
/
5
)
c
4
-
(
4
/
9
)
c
3
+
(
1
/
9
)
c. Since g
′
(
c
)
=
(
16
/
5
)
c
3
-
(
4
/
3
)
c
2
+
(
1
/
9
)
>
0 for 1
/
3
<
c
<
1, then g
(
c
)
>
0 for 1
/
3
<
c
<
1. Then,
(31)
g
(
c
)
m
4
n
4
+
o
(
m
4
n
4
)
<
∑
k
=
1
m
‍
w
k
(
t
k
-
t
-
w
)
2
<
4
m
4
5
n
4
+
o
(
m
4
n
4
)
.
Consequently, we obtain t
-
w
=
O
(
m
2
/
n
2
) and υ
=
O
(
n
2
/
m
2
). Note that
(32)
|
∑
k
=
1
m
‍
k
2
d
k
|
≤
∑
k
=
1
m
‍
k
2
a
k
|
u
k
|
≤
|
1
+
υ
t
-
w
|
∑
k
=
1
m
‍
k
2
a
k
+
υ
∑
k
=
1
m
‍
k
2
t
k
a
k
.
So, we can get ∑
k
=
1
m
‍
k
2
d
k
=
O
(
m
3
).
(
b
) Note that
(33)
f
′
D
f
=
∑
k
=
1
m
‍
{
d
k
∑
i
=
k
+
1
n
‍
(
f
i
-
f
i
-
k
)
2
}
=
∑
k
=
1
m
‍
{
d
k
∑
i
=
k
+
1
n
‍
(
f
i
′
k
n
+
O
(
k
2
n
2
)
)
2
}
=
∑
k
=
1
m
‍
[
d
k
{
k
2
n
2
∑
i
=
k
+
1
n
‍
f
i
′
2
+
O
(
(
n
-
k
)
k
3
n
3
)
}
]
=
J
n
∑
k
=
1
m
‍
k
2
d
k
+
O
(
1
n
2
)
∑
k
=
1
m
‍
k
3
d
k
,
where J
=
∫
0
1
‍
f
′
(
x
)
2
d
x. By (
a
), for any m
=
o
(
n
), we have
(34)
f
′
D
f
=
O
(
m
3
n
)
+
O
(
m
4
n
2
)
=
O
(
m
3
n
)
.
(
c
) Let d
0
=
0, f
0
=
0. We know
(35)
f
′
D
Σ
D
′
f
=
∑
i
=
1
m
‍
c
i
{
-
∑
j
=
0
i
-
1
‍
d
j
f
i
-
j
+
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
i
-
1
‍
d
j
)
f
i
-
∑
j
=
1
m
‍
d
j
f
i
+
j
}
2
+
∑
i
=
m
+
1
n
-
m
‍
c
i
{
-
∑
j
=
1
m
‍
d
j
f
i
-
j
+
2
∑
k
=
1
m
‍
d
k
f
i
-
∑
j
=
1
m
‍
d
j
f
i
+
j
}
2
+
∑
i
=
n
-
m
+
1
n
‍
c
i
{
-
∑
j
=
1
m
‍
d
j
f
i
-
j
+
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
1
n
-
i
‍
d
j
)
f
i
H
H
H
H
H
H
H
H
H
H
H
H
-
∑
j
=
1
n
-
i
‍
d
j
f
i
+
j
}
2
=
∑
i
=
1
m
‍
c
i
(
∑
j
=
i
m
‍
j
d
j
)
2
(
f
i
′
n
+
o
(
1
n
)
)
2
+
4
∑
i
=
m
+
1
n
-
m
‍
c
i
(
o
(
1
n
)
∑
j
=
1
m
‍
j
d
j
)
2
+
∑
i
=
n
-
m
+
1
n
‍
c
i
(
∑
j
=
n
-
i
+
1
m
‍
j
d
j
)
2
(
f
i
′
n
+
o
(
1
n
)
)
2
=
O
(
m
5
n
2
)
+
o
(
m
4
n
)
.
For any m
=
o
(
n
), we have
(36)
f
′
D
Σ
D
′
f
=
o
(
m
4
n
)
.
(
d
) We know
(37)
f
′
D
~
f
=
∑
k
=
1
m
‍
{
d
~
k
∑
i
=
k
+
1
n
‍
(
f
i
-
f
i
-
k
)
2
}
=
J
n
∑
k
=
1
m
‍
k
2
d
~
k
+
O
(
1
n
2
)
∑
k
=
1
m
‍
k
3
d
~
k
.
Note that
(38)
∑
k
=
1
m
‍
k
2
d
~
k
=
o
(
m
3
)
,
∑
k
=
1
m
‍
k
3
d
~
k
=
O
(
m
4
)
.
Thus,
(39)
f
′
D
~
f
=
o
(
m
3
n
)
.
(
e
) We know
(40)
f
′
D
~
Σ
D
~
′
f
=
1
n
2
∑
i
=
1
m
‍
c
i
(
f
i
′
+
o
(
1
)
)
2
(
∑
k
=
i
m
‍
k
d
~
k
)
2
+
4
∑
i
=
m
+
1
n
-
m
‍
c
i
(
∑
k
=
1
m
‍
k
d
~
k
)
2
o
(
1
n
2
)
+
1
n
2
∑
i
=
n
-
m
+
1
n
‍
c
i
(
f
i
′
+
o
(
1
)
)
2
(
∑
k
=
n
-
i
+
1
m
‍
k
d
~
k
)
2
.
Note that
(41)
∑
k
=
1
m
‍
k
d
~
k
=
O
(
m
2
)
,
∑
k
=
i
m
‍
k
d
~
k
=
O
(
m
2
)
,
1
≤
i
≤
m
.
Therefore, we have
(42)
f
′
D
~
Σ
D
~
′
f
=
o
(
m
4
n
)
.Lemma 6.
Assume thatf has a bounded second derivative. When max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
n
s
) with 0
<
s
<
1, for any m
=
n
r with 0
<
r
<
1 and the equally spaced design, then (i)
∑
k
=
1
m
‍
k
t
d
k
=
O
(
m
t
+
1
), t
=
0,1
,
2,3;
(ii)
f
′
D
f
=
O
(
m
3
/
n
);
(iii)
f
′
D
Σ
D
′
f
=
o
(
m
4
/
n
1
-
s
);
(iv)
f
′
D
~
f
=
o
(
m
3
/
n
);
(v)
f
′
D
~
Σ
D
~
′
f
=
o
(
m
4
/
n
1
-
s
).Proof.
(i) Here we only consider the proof oft
=
2. For max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
n
s
), it is similar with (
a
) in Lemma 5 to verify that t
-
w
=
O
(
m
2
/
n
2
) and υ
=
O
(
n
2
/
m
2
). Thus, we can get
(43)
∑
k
=
1
m
‍
k
2
d
k
=
O
(
m
3
)
.
(ii) According to(
i
), we have
(44)
f
′
D
f
=
J
n
∑
k
=
1
m
‍
k
2
d
k
+
O
(
1
n
2
)
∑
k
=
1
m
‍
k
3
d
k
=
O
(
m
3
n
)
.
(iii) By part(
c
) in Lemma 5, we know
(45)
f
′
D
Σ
D
′
f
=
1
n
2
∑
i
=
1
m
‍
c
i
(
f
i
′
+
o
(
1
)
)
2
(
∑
k
=
i
m
‍
k
d
k
)
2
+
4
∑
i
=
m
+
1
n
-
m
‍
c
i
(
∑
k
=
1
m
‍
k
d
k
)
2
o
(
1
n
2
)
+
1
n
2
∑
i
=
n
-
m
+
1
n
‍
c
i
(
f
i
′
+
o
(
1
)
)
2
(
∑
k
=
n
-
i
+
1
m
‍
k
d
k
)
2
.
For any m
=
o
(
n
), we have
(46)
f
′
D
Σ
D
′
f
=
o
(
m
4
n
1
-
s
)
.
(iv) By(
d
) in Lemma 5, we have
(47)
f
′
D
~
f
=
J
n
∑
k
=
1
m
‍
k
2
d
~
k
+
O
(
1
n
2
)
∑
k
=
1
m
‍
k
3
d
~
k
=
o
(
m
3
n
)
.
(v) Similar to(
e
) in Lemma 5, it is easy to get
(48)
f
′
D
~
Σ
D
~
′
f
=
1
n
2
∑
i
=
1
m
‍
c
i
(
f
i
′
+
o
(
1
)
)
2
(
∑
k
=
i
m
‍
k
d
~
k
)
2
+
4
∑
i
=
m
+
1
n
-
m
‍
c
i
(
∑
k
=
1
m
‍
k
d
~
k
)
2
o
(
1
n
2
)
+
1
n
2
∑
i
=
n
-
m
+
1
n
‍
c
i
(
f
i
′
+
o
(
1
)
)
2
(
∑
k
=
n
-
i
+
1
m
‍
k
d
~
k
)
2
=
O
(
m
5
n
2
-
s
)
+
o
(
m
4
n
)
=
o
(
m
4
n
1
-
s
)
.Lemma 7 (see [23]).
Leta
n
;
t
k be entries of a real symmetric matrix A
n
=
(
a
n
;
t
k
), let {
Z
t
,
t
=
1
,
…
,
n
} be i.i.d. random variables, and T
n
=
∑
t
,
k
=
1
n
‍
a
n
;
t
k
Z
t
Z
k. Assume that ∥
A
n
∥
s
p
/
∥
A
n
∥
→
0 and E
Z
t
4
<
∞; then
(49)
(
Var
⁡
(
T
n
)
)
-
1
/
2
(
T
n
-
E
T
n
)
⟶
d
N
(
0,1
)
,
where ∥
A
n
∥
=
(
∑
t
,
k
=
1
n
‍
a
n
;
t
k
2
)
1
/
2 and ∥
A
n
∥
s
p
=
max
⁡
∥
x
∥
=
1
∥
A
n
x
∥ are the Euclidean norm and the spectral norm of the matrix A
n, respectively.


### 6.1. Proof of Theorem1

Letf
(
x
)
=
μ
+
δ
x. It is easy to verify that
(50)
E
z
k
=
σ
2
+
k
2
2
n
2
a
k
δ
2
=
σ
2
+
1
2
t
k
δ
2
,
(51)
E
z
-
w
=
E
(
∑
k
=
1
m
‍
w
k
z
k
)
=
σ
2
+
1
2
t
-
w
δ
2
.
By combining (50) and (51), we have
(52)
E
(
σ
^
1
2
)
=
E
(
z
-
w
-
t
-
w
β
^
)
=
E
(
z
-
w
)
-
t
-
w
∑
k
=
1
m
w
k
(
t
k
-
t
-
w
)
2
×
{
∑
k
=
1
m
‍
w
k
t
k
E
z
k
-
t
-
w
E
(
z
-
w
)
}
=
σ
2
+
1
2
δ
2
t
-
w
-
t
-
w
∑
k
=
1
m
w
k
(
t
k
-
t
-
w
)
2
×
(
∑
k
=
1
m
‍
w
k
t
k
2
-
t
-
w
2
)
1
2
δ
2
=
σ
2
.
This shows that σ
^
1
2 is an unbiased estimator of σ
2. In what follows we consider σ
^
TW
2. Therefore,
(53)
E
(
σ
^
TW
2
)
=
tr
⁡
(
D
~
Σ
)
2
N
σ
2
+
1
2
N
f
′
D
~
f
.
For f
(
x
)
=
μ
+
δ
x, we have
(54)
1
2
N
f
′
D
~
f
=
1
2
N
∑
k
=
1
m
‍
{
d
~
k
∑
i
=
k
+
1
n
‍
(
f
i
-
f
i
-
k
)
2
}
=
1
2
N
∑
k
=
1
m
‍
(
d
~
k
∑
i
=
k
+
1
n
‍
δ
2
k
2
n
2
)
=
δ
2
2
N
∑
k
=
1
m
‍
(
n
-
k
)
b
k
d
~
k
=
δ
2
2
∑
k
=
1
m
‍
w
k
b
k
(
1
-
(
b
k
-
b
-
w
)
b
-
w
∑
k
=
1
m
w
k
(
b
k
-
b
-
w
)
2
)
=
0
,
(55)
tr
⁡
(
D
~
Σ
)
=
∑
i
=
1
m
‍
c
i
(
∑
k
=
1
m
‍
d
~
k
+
∑
k
=
0
i
-
1
‍
d
~
k
)
+
2
∑
i
=
m
+
1
n
-
m
‍
c
i
∑
k
=
1
m
‍
d
~
k
+
∑
i
=
n
-
m
+
1
n
‍
c
i
(
∑
k
=
1
m
‍
d
~
k
+
∑
k
=
1
n
-
i
‍
d
~
k
)
=
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
.
By (54) and (55), we get
(56)
Bias
(
σ
^
TW
2
)
=
{
∑
k
=
1
m
d
~
k
∑
i
=
k
+
1
n
(
c
i
+
c
i
-
k
)
2
N
-
1
}
σ
2
.


### 6.2. Proof of Theorem2

It is easy to verify thattr
⁡
(
D
Σ
)
=
2
N. This leads to
(57)
E
(
σ
^
1
2
)
=
E
(
1
2
N
y
′
D
y
)
=
1
2
N
E
(
f
′
D
f
+
2
f
′
D
ε
+
ε
′
D
ε
)
=
1
2
N
{
f
′
D
f
+
E
(
ε
′
D
ε
)
}
=
σ
2
+
1
2
N
f
′
D
f
.
Thus,
(58)
Bias
(
σ
^
1
2
)
=
E
(
σ
^
1
2
)
-
σ
2
=
1
2
N
f
′
D
f
.
By Lemma 5, for any m
=
o
(
n
), we have
(59)
Bias
(
σ
^
1
2
)
=
O
(
m
2
n
2
)
.
In what follows, we calculate Var
⁡
(
σ
^
1
2
). Note that
(60)
Var
⁡
(
σ
^
1
2
)
=
Var
⁡
(
1
2
N
y
′
D
y
)
=
1
4
N
2
Var
⁡
(
f
′
D
f
+
ε
′
D
ε
+
2
f
′
D
ε
)
=
1
4
N
2
{
Var
⁡
(
ε
′
D
ε
)
+
4
f
′
D
Var
⁡
(
ε
)
D
′
f
+
2
Cov
(
ε
′
D
ε
,
2
f
′
D
ε
)
}
=
1
4
N
2
(
I
1
+
I
2
+
I
3
)
.
For I
1, we have
(61)
I
1
=
∑
i
=
1
n
‍
d
i
i
2
Var
⁡
(
ε
i
2
)
+
4
∑
1
≤
i
<
j
≤
n
‍
d
i
j
2
E
(
ε
i
2
ε
j
2
)
=
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
j
)
2
Var
⁡
(
ε
2
)
+
4
σ
4
∑
k
=
1
m
‍
d
k
2
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
,
where d
0
=
0
,
ε
=
ε
i
/
c
i. For I
2, by Lemma 5, for any m
=
o
(
n
), we have
(62)
I
2
=
4
f
′
D
Var
⁡
(
ε
)
D
′
f
=
4
f
′
D
Σ
D
′
f
σ
2
=
o
(
m
4
n
)
.
Finally, we consider I
3,
(63)
I
3
=
4
E
{
f
′
D
ε
ε
′
D
ε
}
=
4
[
∑
s
=
1
m
‍
{
f
s
′
n
∑
k
=
s
m
‍
k
d
k
+
o
(
1
n
)
∑
k
=
s
m
‍
k
d
k
}
×
E
{
ε
s
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
+
∑
s
=
m
+
1
n
-
m
‍
E
{
ε
s
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
o
(
m
2
n
)
+
∑
s
=
n
-
m
+
1
n
‍
{
f
s
′
n
∑
k
=
n
-
s
+
1
m
‍
k
d
k
+
o
(
1
n
)
∑
k
=
n
-
s
+
1
m
‍
k
d
k
}
×
E
{
ε
s
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
]
=
O
(
m
4
n
)
+
o
(
m
3
)
.
Combining (61), (62), and (63), we know
(64)
Var
⁡
(
σ
^
1
2
)
=
Var
⁡
(
ε
2
)
4
N
2
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
j
)
2
+
σ
4
N
2
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
+
O
(
m
2
n
3
)
+
o
(
m
n
2
)
.
Note that, for max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
1
) and any m
=
n
r with 0
<
r
<
1, we have
(65)
σ
4
N
2
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
=
o
(
1
n
)
.
Therefore, we get
(66)
Var
⁡
(
σ
^
1
2
)
=
C
1
n
Var
⁡
(
ε
2
)
+
o
(
1
n
)
,
where
(67)
C
1
=
n
4
N
2
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
j
)
2
.LetU
1
=
Var
⁡
(
ε
′
D
~
ε
), U
2
=
Cov
(
ε
′
D
~
ε
,
f
′
D
~
ε
). Then, we have
(68)
U
1
=
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
~
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
~
j
)
2
Var
⁡
(
ε
2
)
+
4
σ
4
∑
k
=
1
m
‍
d
~
k
2
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
,
U
2
=
∑
s
=
1
m
‍
∑
j
=
s
m
‍
j
d
~
j
{
f
s
′
n
+
o
(
1
n
)
}
E
{
ε
s
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
+
∑
s
=
m
+
1
n
-
m
‍
E
{
ε
s
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
o
(
m
2
n
)
+
∑
s
=
n
-
m
+
1
n
‍
∑
j
=
n
-
s
m
‍
j
d
~
j
{
f
s
′
n
+
o
(
1
n
)
}
hhhhhhhhhhhhh
×
E
{
ε
s
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
.By (68) and (
d
) and (
e
) in Lemma 5, we can get
(69)
Bias
(
σ
^
TW
2
)
=
E
(
σ
^
TW
2
)
-
σ
2
=
tr
⁡
(
D
~
Σ
)
2
N
σ
2
+
1
2
N
f
′
D
~
f
-
σ
2
=
{
∑
k
=
1
m
d
~
k
∑
i
=
k
+
1
n
(
c
i
+
c
i
-
k
)
2
N
-
1
}
σ
2
+
o
(
m
2
n
2
)
,
Var
⁡
(
σ
^
TW
2
)
=
1
4
N
2
{
U
1
+
4
U
2
+
4
f
′
D
~
Σ
D
~
′
f
σ
2
}
hhhhhhhh
=
C
2
n
Var
⁡
(
ε
2
)
+
o
(
1
n
)
,
where C
2
=
(
n
/
4
N
2
)
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
~
k
+
∑
j
=
0
min
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
~
j
)
2 and d
~
0
=
0. This completes the proof of the theorem.


### 6.3. Proof of Theorem3

By lemma 2, we know(70)
Bias
(
σ
^
1
2
)
=
E
(
σ
^
1
2
)
-
σ
2
=
1
2
N
f
′
D
f
=
O
(
m
2
n
2
)
.
According to (63), for max
⁡
1
≤
i
≤
n
c
i
=
O
(
n
s
), we have
(71)
Cov
(
ε
′
D
ε
,
f
′
D
ε
)
=
O
(
m
4
n
1
-
(
3
/
2
)
s
)
+
o
(
m
3
n
(
3
/
2
)
s
)
.
Note that, under the condition ∑
i
=
1
n
‍
c
i
=
n and max
⁡
1
≤
i
≤
n
c
i
=
O
(
n
s
), it can be shown that
(72)
∑
i
=
1
n
‍
c
i
2
=
O
(
n
1
+
s
)
.
In addition, by Cauchy-Schwarz inequality, we know
(73)
∑
k
=
1
m
‍
d
k
2
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
=
O
(
m
n
1
+
s
)
.
Thus,
(74)
Var
⁡
(
σ
^
1
2
)
=
1
4
N
2
Var
⁡
(
ε
′
D
ε
)
+
1
N
2
f
′
D
Var
⁡
(
ε
)
D
′
f
+
1
N
2
Cov
(
ε
′
D
ε
,
f
′
D
ε
)
=
O
(
1
n
1
-
s
)
+
O
(
1
m
n
1
-
s
)
+
o
(
m
2
n
3
-
s
)
+
o
(
m
n
2
-
(
3
/
2
)
s
)
.
When r and s satisfy r
-
s
>
0 and r
+
(
3
/
2
)
s
≤
1, namely, 0
<
s
<
2
/
5 and 0
<
r
<
1, then
(75)
Var
⁡
(
σ
^
1
2
)
=
O
(
1
n
1
-
s
)
.Next, we consider the order of the bias and variance ofσ
^
TW
2. By (55), we have
(76)
tr
⁡
(
D
~
Σ
)
=
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
.
Note that
(77)
∑
k
=
1
m
‍
d
~
k
=
m
-
5
m
2
16
n
+
o
(
m
2
n
)
,
∑
i
=
1
n
‍
c
i
=
n
.
By (
iv
) in Lemma 6, then
(78)
Bias
(
σ
^
TW
2
)
=
{
(
(
2
n
∑
k
=
1
m
d
~
k
-
∑
k
=
1
m
d
~
k
∑
i
=
1
k
c
i
-
∑
k
=
1
m
d
~
k
∑
i
=
n
-
k
+
1
n
c
i
)
(
2
N
)
-
1
)
-
1
}
σ
2
+
o
(
m
2
n
2
)
=
{
-
5
m
16
n
-
(
∑
k
=
1
m
d
~
k
∑
i
=
1
k
c
i
+
∑
k
=
1
m
d
~
k
∑
i
=
n
-
k
+
1
n
c
i
)
(
2
N
)
-
1
}
σ
2
+
o
(
m
n
)
.
Consequently, it shows that
(79)
|
Bias
(
σ
^
TW
2
)
|
≤
{
5
m
16
n
+
∑
k
=
1
m
‍
k
d
~
k
O
(
n
s
)
N
}
σ
2
+
o
(
m
n
)
.
Thus, we get
(80)
Bias
(
σ
^
TW
2
)
=
O
(
m
n
1
-
s
)
.
By (68) and (
v
) in Lemma 6, for 0
<
s
<
2
/
5 and 0
<
r
<
1, it is similar with (74) to get
(81)
Var
⁡
(
σ
^
TW
2
)
=
1
4
N
2
Var
⁡
(
ε
′
D
~
ε
)
+
1
N
2
f
′
D
~
Var
⁡
(
ε
)
D
~
′
f
+
1
N
2
Cov
(
ε
′
D
~
ε
,
f
′
D
~
ε
)
=
O
(
1
n
1
-
s
)
.This completes the proof of the theorem.


### 6.4. Proof of Theorem4

By Theorem1, we know
(82)
σ
^
1
2
=
1
2
N
y
′
D
y
=
1
2
N
f
′
D
f
+
1
N
f
′
D
ε
+
1
2
N
ε
′
D
ε
.
Note that the first term corresponds to the bias term. By Lemma 6, we know (
1
/
2
N
)
f
′
D
f
=
O
(
m
2
/
n
2
). Thus, for any m
=
n
r with 0
<
r
<
3
/
4,
(83)
1
2
N
f
′
D
f
=
o
(
n
-
1
/
2
)
.
For the second term, by Lemma 6, we have
(84)
E
(
f
′
D
ε
N
)
2
=
f
′
D
Σ
D
′
f
σ
2
N
2
=
o
(
m
2
n
3
-
s
)
=
o
(
m
2
n
2
)
.
Thus, for any m
=
n
r with 0
<
r
<
1
/
2, we have
(85)
f
′
D
ε
N
=
o
p
(
n
-
1
/
2
)
.Now we consider the third term. Letε
~
i
=
ε
i
/
c
i, ε
~
=
(
ε
~
1
,
ε
~
2
,
…
,
ε
~
n
)
T, and C
=
diag
⁡
(
c
1
,
c
2
,
…
,
c
n
), then E
(
ε
~
i
)
=
0
,
Var
⁡
(
ε
~
i
)
=
σ
2, and ε
~
i are i.i.d. random variables, ε
=
C
ε
~. So we have
(86)
ε
T
D
ε
=
ε
~
T
C
T
D
C
ε
~
=
ε
~
T
T
ε
~
=
∑
i
,
j
=
1
n
‍
t
i
j
ε
~
i
ε
~
j
,
where T
=
C
T
D
C
=
(
t
i
j
)
n
×
n is a real symmetric matrix with t
i
j
=
c
i
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
{
i
-
1
,
n
-
i
,
m
}
‍
d
k
) for i
=
j, t
i
j
=
-
c
i
c
i
+
|
i
-
j
|
d
|
i
-
j
| for 0
<
|
i
-
j
|
≤
m, and t
i
j
=
0 otherwise.We know that the Euclidean norm of the matrixT can be denoted as
(87)
∥
T
∥
=
(
∑
i
,
j
=
1
n
‍
t
i
j
2
)
1
/
2
=
{
∑
i
=
1
m
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
i
-
1
‍
d
j
)
2
+
∑
i
=
m
+
1
n
-
m
‍
c
i
2
(
2
∑
k
=
1
m
‍
d
k
)
2
+
∑
i
=
n
-
m
+
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
)
2
+
2
∑
j
=
1
m
‍
(
∑
i
=
1
n
-
j
‍
c
i
c
i
+
j
d
j
2
)
}
1
/
2
,
where d
0
=
0. By the definition of spectral norm, we have
(88)
∥
T
∥
sp
=
max
⁡
∥
x
∥
=
1
∥
T
x
∥
=
(
maximum
eigenvalue
of
T
C
T
)
1
/
2
,
where T
C is the conjugate transpose of the matrix T. Since T is a real symmetric matrix, T
C
=
T. Then, T
C
T
=
T
2.Letλ
1
,
λ
2
,
…
,
λ
n be eigenvalues of matrix T and ordered to be nonincreasing in absolute value. Then, λ
i
2
,
i
=
1
,
…
,
n are eigenvalues of the matrix T
2. Thus, |
λ
1
| is the spectral norm of matrix T. Namely, ∥
T
∥
sp
=
|
λ
1
|. Let
(89)
L
n
2
=
max
⁡
1
≤
i
≤
n
∑
j
=
1
n
‍
t
i
j
2
,
Γ
n
=
max
⁡
1
≤
i
≤
n
∑
j
=
1
n
‍
|
t
i
j
|
.
It is well known that
(90)
L
n
≤
|
λ
1
|
≤
Γ
n
.Letmax
⁡
1
≤
i
≤
n
c
i
=
O
(
n
s
) and ω
=
min
⁡
1
≤
i
≤
m
,
n
-
m
+
1
≤
i
≤
n
{
|
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
i
-
1
‍
d
j
|
,
2
|
∑
k
=
1
m
‍
d
k
|
,
|
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
|
}. Then, combining (87) and (90), we have
(91)
∥
T
∥
sp
∥
T
∥
≤
max
⁡
1
≤
i
≤
n
∑
j
=
1
n
|
t
i
j
|
∥
T
∥
≤
max
⁡
1
≤
i
≤
n
(
2
c
i
|
∑
k
=
1
m
d
k
|
+
2
∑
k
=
1
m
c
i
c
i
+
k
|
d
k
|
)
∥
T
∥
≤
(
2
(
|
∑
k
=
1
m
‍
d
k
|
+
∑
k
=
1
m
‍
|
d
k
|
)
O
(
n
s
)
)
×
(
+
∑
i
=
n
-
m
+
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
)
2
}
1
/
2
{
∑
i
=
1
m
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
i
-
1
‍
d
j
)
2
+
∑
i
=
m
+
1
n
-
m
‍
c
i
2
(
2
∑
k
=
1
m
‍
d
k
)
2
+
∑
i
=
n
-
m
+
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
)
2
}
1
/
2
)
≤
2
(
|
∑
k
=
1
m
d
k
|
+
∑
k
=
1
m
|
d
k
|
)
ω
O
(
n
s
)
∑
i
=
1
n
c
i
2
.
Note that ∑
i
=
1
n
‍
c
i
=
n. By Cauchy-Schwarz inequality, we have
(92)
∑
i
=
1
n
‍
c
i
2
≥
n
.
Note also that
(93)
2
(
|
∑
k
=
1
m
d
k
|
+
∑
k
=
1
m
|
d
k
|
)
ω
=
O
(
1
)
.
Consequently, for 0
<
s
<
1
/
2, we have
(94)
∥
T
∥
sp
∥
T
∥
≤
2
(
|
∑
k
=
1
m
d
k
|
+
∑
k
=
1
m
|
d
k
|
)
ω
O
(
n
s
)
n
1
/
2
⟶
0
,
as
n
⟶
∞
.Note thatE
ε
~
4
<
∞. Then, by Lemma 7, we obtain
(95)
Var
⁡
(
ε
~
T
T
ε
~
)
-
1
/
2
(
ε
~
T
T
ε
~
-
E
(
ε
~
T
T
ε
~
)
)
⟶
d
N
(
0,1
)
.
That is,
(96)
Var
⁡
(
ε
T
D
ε
)
-
1
/
2
(
ε
T
D
ε
-
E
(
ε
T
D
ε
)
)
⟶
d
N
(
0,1
)
,
where
(97)
E
(
ε
T
D
ε
)
=
tr
⁡
(
D
Σ
)
σ
2
=
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
σ
2
,
Var
⁡
(
ε
T
D
ε
)
=
(
γ
4
-
1
)
σ
4
∑
i
=
1
n
‍
d
i
i
2
c
i
2
+
4
∑
1
≤
i
<
j
≤
n
‍
d
i
j
2
c
i
c
j
σ
4
=
(
γ
4
-
1
)
σ
4
∑
i
=
1
n
‍
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
k
)
2
c
i
2
+
4
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
σ
4
.
Hence, we get
(98)
n
(
(
1
/
2
N
)
ε
T
D
ε
-
μ
0
σ
2
)
σ
0
⟶
d
N
(
0,1
)
,
where μ
0
=
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
/
(
2
N
), σ
0
2
=
n
(
γ
4
-
1
)
σ
4
/
4
N
2
∑
i
=
1
n
‍
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
k
)
2
c
i
2
+
(
n
σ
4
/
N
2
)
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
). Combining (83), (85), and (98), by Slutsky’s theorem, we have
(99)
n
(
σ
^
1
2
-
μ
0
σ
2
)
σ
0
2
⟶
d
N
(
0,1
)
.Note thatμ
0
=
1
+
O
(
m
/
n
1
-
s
) and (
n
σ
4
/
N
2
)
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
=
O
(
n
s
/
m
). Then, when (
1
/
2
)
-
r
-
s
>
0 and r
>
s, namely, 0
<
s
<
1
/
4 and 0
<
r
<
1
/
2, we have n
(
μ
0
-
1
)
=
o
(
1
) and
(100)
σ
0
2
=
n
(
γ
4
-
1
)
σ
4
4
N
2
∑
i
=
1
n
‍
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
k
)
2
c
i
2
+
o
(
1
)
=
δ
(
γ
4
-
1
)
σ
4
+
o
(
1
)
,
where δ
=
∑
i
=
1
n
‍
c
i
2
/
n. By Slutsky’s theorem, we obtain
(101)
n
(
σ
^
1
2
-
σ
2
)
δ
(
γ
4
-
1
)
σ
4
=
σ
0
δ
(
γ
4
-
1
)
σ
4
×
{
n
(
σ
^
1
2
-
μ
0
σ
2
)
σ
0
+
n
(
μ
0
-
1
)
σ
2
σ
0
}
⟶
d
N
(
0,1
)
,
as
n
⟶
∞
.This proves the theorem.


## 6.1. Proof of Theorem1

Letf
(
x
)
=
μ
+
δ
x. It is easy to verify that
(50)
E
z
k
=
σ
2
+
k
2
2
n
2
a
k
δ
2
=
σ
2
+
1
2
t
k
δ
2
,
(51)
E
z
-
w
=
E
(
∑
k
=
1
m
‍
w
k
z
k
)
=
σ
2
+
1
2
t
-
w
δ
2
.
By combining (50) and (51), we have
(52)
E
(
σ
^
1
2
)
=
E
(
z
-
w
-
t
-
w
β
^
)
=
E
(
z
-
w
)
-
t
-
w
∑
k
=
1
m
w
k
(
t
k
-
t
-
w
)
2
×
{
∑
k
=
1
m
‍
w
k
t
k
E
z
k
-
t
-
w
E
(
z
-
w
)
}
=
σ
2
+
1
2
δ
2
t
-
w
-
t
-
w
∑
k
=
1
m
w
k
(
t
k
-
t
-
w
)
2
×
(
∑
k
=
1
m
‍
w
k
t
k
2
-
t
-
w
2
)
1
2
δ
2
=
σ
2
.
This shows that σ
^
1
2 is an unbiased estimator of σ
2. In what follows we consider σ
^
TW
2. Therefore,
(53)
E
(
σ
^
TW
2
)
=
tr
⁡
(
D
~
Σ
)
2
N
σ
2
+
1
2
N
f
′
D
~
f
.
For f
(
x
)
=
μ
+
δ
x, we have
(54)
1
2
N
f
′
D
~
f
=
1
2
N
∑
k
=
1
m
‍
{
d
~
k
∑
i
=
k
+
1
n
‍
(
f
i
-
f
i
-
k
)
2
}
=
1
2
N
∑
k
=
1
m
‍
(
d
~
k
∑
i
=
k
+
1
n
‍
δ
2
k
2
n
2
)
=
δ
2
2
N
∑
k
=
1
m
‍
(
n
-
k
)
b
k
d
~
k
=
δ
2
2
∑
k
=
1
m
‍
w
k
b
k
(
1
-
(
b
k
-
b
-
w
)
b
-
w
∑
k
=
1
m
w
k
(
b
k
-
b
-
w
)
2
)
=
0
,
(55)
tr
⁡
(
D
~
Σ
)
=
∑
i
=
1
m
‍
c
i
(
∑
k
=
1
m
‍
d
~
k
+
∑
k
=
0
i
-
1
‍
d
~
k
)
+
2
∑
i
=
m
+
1
n
-
m
‍
c
i
∑
k
=
1
m
‍
d
~
k
+
∑
i
=
n
-
m
+
1
n
‍
c
i
(
∑
k
=
1
m
‍
d
~
k
+
∑
k
=
1
n
-
i
‍
d
~
k
)
=
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
.
By (54) and (55), we get
(56)
Bias
(
σ
^
TW
2
)
=
{
∑
k
=
1
m
d
~
k
∑
i
=
k
+
1
n
(
c
i
+
c
i
-
k
)
2
N
-
1
}
σ
2
.


## 6.2. Proof of Theorem2

It is easy to verify thattr
⁡
(
D
Σ
)
=
2
N. This leads to
(57)
E
(
σ
^
1
2
)
=
E
(
1
2
N
y
′
D
y
)
=
1
2
N
E
(
f
′
D
f
+
2
f
′
D
ε
+
ε
′
D
ε
)
=
1
2
N
{
f
′
D
f
+
E
(
ε
′
D
ε
)
}
=
σ
2
+
1
2
N
f
′
D
f
.
Thus,
(58)
Bias
(
σ
^
1
2
)
=
E
(
σ
^
1
2
)
-
σ
2
=
1
2
N
f
′
D
f
.
By Lemma 5, for any m
=
o
(
n
), we have
(59)
Bias
(
σ
^
1
2
)
=
O
(
m
2
n
2
)
.
In what follows, we calculate Var
⁡
(
σ
^
1
2
). Note that
(60)
Var
⁡
(
σ
^
1
2
)
=
Var
⁡
(
1
2
N
y
′
D
y
)
=
1
4
N
2
Var
⁡
(
f
′
D
f
+
ε
′
D
ε
+
2
f
′
D
ε
)
=
1
4
N
2
{
Var
⁡
(
ε
′
D
ε
)
+
4
f
′
D
Var
⁡
(
ε
)
D
′
f
+
2
Cov
(
ε
′
D
ε
,
2
f
′
D
ε
)
}
=
1
4
N
2
(
I
1
+
I
2
+
I
3
)
.
For I
1, we have
(61)
I
1
=
∑
i
=
1
n
‍
d
i
i
2
Var
⁡
(
ε
i
2
)
+
4
∑
1
≤
i
<
j
≤
n
‍
d
i
j
2
E
(
ε
i
2
ε
j
2
)
=
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
j
)
2
Var
⁡
(
ε
2
)
+
4
σ
4
∑
k
=
1
m
‍
d
k
2
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
,
where d
0
=
0
,
ε
=
ε
i
/
c
i. For I
2, by Lemma 5, for any m
=
o
(
n
), we have
(62)
I
2
=
4
f
′
D
Var
⁡
(
ε
)
D
′
f
=
4
f
′
D
Σ
D
′
f
σ
2
=
o
(
m
4
n
)
.
Finally, we consider I
3,
(63)
I
3
=
4
E
{
f
′
D
ε
ε
′
D
ε
}
=
4
[
∑
s
=
1
m
‍
{
f
s
′
n
∑
k
=
s
m
‍
k
d
k
+
o
(
1
n
)
∑
k
=
s
m
‍
k
d
k
}
×
E
{
ε
s
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
+
∑
s
=
m
+
1
n
-
m
‍
E
{
ε
s
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
o
(
m
2
n
)
+
∑
s
=
n
-
m
+
1
n
‍
{
f
s
′
n
∑
k
=
n
-
s
+
1
m
‍
k
d
k
+
o
(
1
n
)
∑
k
=
n
-
s
+
1
m
‍
k
d
k
}
×
E
{
ε
s
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
]
=
O
(
m
4
n
)
+
o
(
m
3
)
.
Combining (61), (62), and (63), we know
(64)
Var
⁡
(
σ
^
1
2
)
=
Var
⁡
(
ε
2
)
4
N
2
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
j
)
2
+
σ
4
N
2
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
+
O
(
m
2
n
3
)
+
o
(
m
n
2
)
.
Note that, for max
⁡
1
≤
i
≤
n
⁡
c
i
=
O
(
1
) and any m
=
n
r with 0
<
r
<
1, we have
(65)
σ
4
N
2
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
=
o
(
1
n
)
.
Therefore, we get
(66)
Var
⁡
(
σ
^
1
2
)
=
C
1
n
Var
⁡
(
ε
2
)
+
o
(
1
n
)
,
where
(67)
C
1
=
n
4
N
2
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
j
)
2
.LetU
1
=
Var
⁡
(
ε
′
D
~
ε
), U
2
=
Cov
(
ε
′
D
~
ε
,
f
′
D
~
ε
). Then, we have
(68)
U
1
=
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
~
k
+
∑
j
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
~
j
)
2
Var
⁡
(
ε
2
)
+
4
σ
4
∑
k
=
1
m
‍
d
~
k
2
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
,
U
2
=
∑
s
=
1
m
‍
∑
j
=
s
m
‍
j
d
~
j
{
f
s
′
n
+
o
(
1
n
)
}
E
{
ε
s
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
+
∑
s
=
m
+
1
n
-
m
‍
E
{
ε
s
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
o
(
m
2
n
)
+
∑
s
=
n
-
m
+
1
n
‍
∑
j
=
n
-
s
m
‍
j
d
~
j
{
f
s
′
n
+
o
(
1
n
)
}
hhhhhhhhhhhhh
×
E
{
ε
s
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
ε
i
-
ε
i
-
k
)
2
}
.By (68) and (
d
) and (
e
) in Lemma 5, we can get
(69)
Bias
(
σ
^
TW
2
)
=
E
(
σ
^
TW
2
)
-
σ
2
=
tr
⁡
(
D
~
Σ
)
2
N
σ
2
+
1
2
N
f
′
D
~
f
-
σ
2
=
{
∑
k
=
1
m
d
~
k
∑
i
=
k
+
1
n
(
c
i
+
c
i
-
k
)
2
N
-
1
}
σ
2
+
o
(
m
2
n
2
)
,
Var
⁡
(
σ
^
TW
2
)
=
1
4
N
2
{
U
1
+
4
U
2
+
4
f
′
D
~
Σ
D
~
′
f
σ
2
}
hhhhhhhh
=
C
2
n
Var
⁡
(
ε
2
)
+
o
(
1
n
)
,
where C
2
=
(
n
/
4
N
2
)
∑
i
=
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
~
k
+
∑
j
=
0
min
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
~
j
)
2 and d
~
0
=
0. This completes the proof of the theorem.


## 6.3. Proof of Theorem3

By lemma 2, we know(70)
Bias
(
σ
^
1
2
)
=
E
(
σ
^
1
2
)
-
σ
2
=
1
2
N
f
′
D
f
=
O
(
m
2
n
2
)
.
According to (63), for max
⁡
1
≤
i
≤
n
c
i
=
O
(
n
s
), we have
(71)
Cov
(
ε
′
D
ε
,
f
′
D
ε
)
=
O
(
m
4
n
1
-
(
3
/
2
)
s
)
+
o
(
m
3
n
(
3
/
2
)
s
)
.
Note that, under the condition ∑
i
=
1
n
‍
c
i
=
n and max
⁡
1
≤
i
≤
n
c
i
=
O
(
n
s
), it can be shown that
(72)
∑
i
=
1
n
‍
c
i
2
=
O
(
n
1
+
s
)
.
In addition, by Cauchy-Schwarz inequality, we know
(73)
∑
k
=
1
m
‍
d
k
2
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
=
O
(
m
n
1
+
s
)
.
Thus,
(74)
Var
⁡
(
σ
^
1
2
)
=
1
4
N
2
Var
⁡
(
ε
′
D
ε
)
+
1
N
2
f
′
D
Var
⁡
(
ε
)
D
′
f
+
1
N
2
Cov
(
ε
′
D
ε
,
f
′
D
ε
)
=
O
(
1
n
1
-
s
)
+
O
(
1
m
n
1
-
s
)
+
o
(
m
2
n
3
-
s
)
+
o
(
m
n
2
-
(
3
/
2
)
s
)
.
When r and s satisfy r
-
s
>
0 and r
+
(
3
/
2
)
s
≤
1, namely, 0
<
s
<
2
/
5 and 0
<
r
<
1, then
(75)
Var
⁡
(
σ
^
1
2
)
=
O
(
1
n
1
-
s
)
.Next, we consider the order of the bias and variance ofσ
^
TW
2. By (55), we have
(76)
tr
⁡
(
D
~
Σ
)
=
∑
k
=
1
m
‍
d
~
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
.
Note that
(77)
∑
k
=
1
m
‍
d
~
k
=
m
-
5
m
2
16
n
+
o
(
m
2
n
)
,
∑
i
=
1
n
‍
c
i
=
n
.
By (
iv
) in Lemma 6, then
(78)
Bias
(
σ
^
TW
2
)
=
{
(
(
2
n
∑
k
=
1
m
d
~
k
-
∑
k
=
1
m
d
~
k
∑
i
=
1
k
c
i
-
∑
k
=
1
m
d
~
k
∑
i
=
n
-
k
+
1
n
c
i
)
(
2
N
)
-
1
)
-
1
}
σ
2
+
o
(
m
2
n
2
)
=
{
-
5
m
16
n
-
(
∑
k
=
1
m
d
~
k
∑
i
=
1
k
c
i
+
∑
k
=
1
m
d
~
k
∑
i
=
n
-
k
+
1
n
c
i
)
(
2
N
)
-
1
}
σ
2
+
o
(
m
n
)
.
Consequently, it shows that
(79)
|
Bias
(
σ
^
TW
2
)
|
≤
{
5
m
16
n
+
∑
k
=
1
m
‍
k
d
~
k
O
(
n
s
)
N
}
σ
2
+
o
(
m
n
)
.
Thus, we get
(80)
Bias
(
σ
^
TW
2
)
=
O
(
m
n
1
-
s
)
.
By (68) and (
v
) in Lemma 6, for 0
<
s
<
2
/
5 and 0
<
r
<
1, it is similar with (74) to get
(81)
Var
⁡
(
σ
^
TW
2
)
=
1
4
N
2
Var
⁡
(
ε
′
D
~
ε
)
+
1
N
2
f
′
D
~
Var
⁡
(
ε
)
D
~
′
f
+
1
N
2
Cov
(
ε
′
D
~
ε
,
f
′
D
~
ε
)
=
O
(
1
n
1
-
s
)
.This completes the proof of the theorem.


## 6.4. Proof of Theorem4

By Theorem1, we know
(82)
σ
^
1
2
=
1
2
N
y
′
D
y
=
1
2
N
f
′
D
f
+
1
N
f
′
D
ε
+
1
2
N
ε
′
D
ε
.
Note that the first term corresponds to the bias term. By Lemma 6, we know (
1
/
2
N
)
f
′
D
f
=
O
(
m
2
/
n
2
). Thus, for any m
=
n
r with 0
<
r
<
3
/
4,
(83)
1
2
N
f
′
D
f
=
o
(
n
-
1
/
2
)
.
For the second term, by Lemma 6, we have
(84)
E
(
f
′
D
ε
N
)
2
=
f
′
D
Σ
D
′
f
σ
2
N
2
=
o
(
m
2
n
3
-
s
)
=
o
(
m
2
n
2
)
.
Thus, for any m
=
n
r with 0
<
r
<
1
/
2, we have
(85)
f
′
D
ε
N
=
o
p
(
n
-
1
/
2
)
.Now we consider the third term. Letε
~
i
=
ε
i
/
c
i, ε
~
=
(
ε
~
1
,
ε
~
2
,
…
,
ε
~
n
)
T, and C
=
diag
⁡
(
c
1
,
c
2
,
…
,
c
n
), then E
(
ε
~
i
)
=
0
,
Var
⁡
(
ε
~
i
)
=
σ
2, and ε
~
i are i.i.d. random variables, ε
=
C
ε
~. So we have
(86)
ε
T
D
ε
=
ε
~
T
C
T
D
C
ε
~
=
ε
~
T
T
ε
~
=
∑
i
,
j
=
1
n
‍
t
i
j
ε
~
i
ε
~
j
,
where T
=
C
T
D
C
=
(
t
i
j
)
n
×
n is a real symmetric matrix with t
i
j
=
c
i
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
{
i
-
1
,
n
-
i
,
m
}
‍
d
k
) for i
=
j, t
i
j
=
-
c
i
c
i
+
|
i
-
j
|
d
|
i
-
j
| for 0
<
|
i
-
j
|
≤
m, and t
i
j
=
0 otherwise.We know that the Euclidean norm of the matrixT can be denoted as
(87)
∥
T
∥
=
(
∑
i
,
j
=
1
n
‍
t
i
j
2
)
1
/
2
=
{
∑
i
=
1
m
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
i
-
1
‍
d
j
)
2
+
∑
i
=
m
+
1
n
-
m
‍
c
i
2
(
2
∑
k
=
1
m
‍
d
k
)
2
+
∑
i
=
n
-
m
+
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
)
2
+
2
∑
j
=
1
m
‍
(
∑
i
=
1
n
-
j
‍
c
i
c
i
+
j
d
j
2
)
}
1
/
2
,
where d
0
=
0. By the definition of spectral norm, we have
(88)
∥
T
∥
sp
=
max
⁡
∥
x
∥
=
1
∥
T
x
∥
=
(
maximum
eigenvalue
of
T
C
T
)
1
/
2
,
where T
C is the conjugate transpose of the matrix T. Since T is a real symmetric matrix, T
C
=
T. Then, T
C
T
=
T
2.Letλ
1
,
λ
2
,
…
,
λ
n be eigenvalues of matrix T and ordered to be nonincreasing in absolute value. Then, λ
i
2
,
i
=
1
,
…
,
n are eigenvalues of the matrix T
2. Thus, |
λ
1
| is the spectral norm of matrix T. Namely, ∥
T
∥
sp
=
|
λ
1
|. Let
(89)
L
n
2
=
max
⁡
1
≤
i
≤
n
∑
j
=
1
n
‍
t
i
j
2
,
Γ
n
=
max
⁡
1
≤
i
≤
n
∑
j
=
1
n
‍
|
t
i
j
|
.
It is well known that
(90)
L
n
≤
|
λ
1
|
≤
Γ
n
.Letmax
⁡
1
≤
i
≤
n
c
i
=
O
(
n
s
) and ω
=
min
⁡
1
≤
i
≤
m
,
n
-
m
+
1
≤
i
≤
n
{
|
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
i
-
1
‍
d
j
|
,
2
|
∑
k
=
1
m
‍
d
k
|
,
|
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
|
}. Then, combining (87) and (90), we have
(91)
∥
T
∥
sp
∥
T
∥
≤
max
⁡
1
≤
i
≤
n
∑
j
=
1
n
|
t
i
j
|
∥
T
∥
≤
max
⁡
1
≤
i
≤
n
(
2
c
i
|
∑
k
=
1
m
d
k
|
+
2
∑
k
=
1
m
c
i
c
i
+
k
|
d
k
|
)
∥
T
∥
≤
(
2
(
|
∑
k
=
1
m
‍
d
k
|
+
∑
k
=
1
m
‍
|
d
k
|
)
O
(
n
s
)
)
×
(
+
∑
i
=
n
-
m
+
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
)
2
}
1
/
2
{
∑
i
=
1
m
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
i
-
1
‍
d
j
)
2
+
∑
i
=
m
+
1
n
-
m
‍
c
i
2
(
2
∑
k
=
1
m
‍
d
k
)
2
+
∑
i
=
n
-
m
+
1
n
‍
c
i
2
(
∑
k
=
1
m
‍
d
k
+
∑
j
=
0
n
-
i
‍
d
j
)
2
}
1
/
2
)
≤
2
(
|
∑
k
=
1
m
d
k
|
+
∑
k
=
1
m
|
d
k
|
)
ω
O
(
n
s
)
∑
i
=
1
n
c
i
2
.
Note that ∑
i
=
1
n
‍
c
i
=
n. By Cauchy-Schwarz inequality, we have
(92)
∑
i
=
1
n
‍
c
i
2
≥
n
.
Note also that
(93)
2
(
|
∑
k
=
1
m
d
k
|
+
∑
k
=
1
m
|
d
k
|
)
ω
=
O
(
1
)
.
Consequently, for 0
<
s
<
1
/
2, we have
(94)
∥
T
∥
sp
∥
T
∥
≤
2
(
|
∑
k
=
1
m
d
k
|
+
∑
k
=
1
m
|
d
k
|
)
ω
O
(
n
s
)
n
1
/
2
⟶
0
,
as
n
⟶
∞
.Note thatE
ε
~
4
<
∞. Then, by Lemma 7, we obtain
(95)
Var
⁡
(
ε
~
T
T
ε
~
)
-
1
/
2
(
ε
~
T
T
ε
~
-
E
(
ε
~
T
T
ε
~
)
)
⟶
d
N
(
0,1
)
.
That is,
(96)
Var
⁡
(
ε
T
D
ε
)
-
1
/
2
(
ε
T
D
ε
-
E
(
ε
T
D
ε
)
)
⟶
d
N
(
0,1
)
,
where
(97)
E
(
ε
T
D
ε
)
=
tr
⁡
(
D
Σ
)
σ
2
=
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
σ
2
,
Var
⁡
(
ε
T
D
ε
)
=
(
γ
4
-
1
)
σ
4
∑
i
=
1
n
‍
d
i
i
2
c
i
2
+
4
∑
1
≤
i
<
j
≤
n
‍
d
i
j
2
c
i
c
j
σ
4
=
(
γ
4
-
1
)
σ
4
∑
i
=
1
n
‍
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
k
)
2
c
i
2
+
4
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
σ
4
.
Hence, we get
(98)
n
(
(
1
/
2
N
)
ε
T
D
ε
-
μ
0
σ
2
)
σ
0
⟶
d
N
(
0,1
)
,
where μ
0
=
∑
k
=
1
m
‍
d
k
∑
i
=
k
+
1
n
‍
(
c
i
+
c
i
-
k
)
/
(
2
N
), σ
0
2
=
n
(
γ
4
-
1
)
σ
4
/
4
N
2
∑
i
=
1
n
‍
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
k
)
2
c
i
2
+
(
n
σ
4
/
N
2
)
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
). Combining (83), (85), and (98), by Slutsky’s theorem, we have
(99)
n
(
σ
^
1
2
-
μ
0
σ
2
)
σ
0
2
⟶
d
N
(
0,1
)
.Note thatμ
0
=
1
+
O
(
m
/
n
1
-
s
) and (
n
σ
4
/
N
2
)
∑
k
=
1
m
‍
d
k
2
(
∑
i
=
1
n
-
k
‍
c
i
c
i
+
k
)
=
O
(
n
s
/
m
). Then, when (
1
/
2
)
-
r
-
s
>
0 and r
>
s, namely, 0
<
s
<
1
/
4 and 0
<
r
<
1
/
2, we have n
(
μ
0
-
1
)
=
o
(
1
) and
(100)
σ
0
2
=
n
(
γ
4
-
1
)
σ
4
4
N
2
∑
i
=
1
n
‍
(
∑
k
=
1
m
‍
d
k
+
∑
k
=
0
min
⁡
⁡
(
i
-
1
,
n
-
i
,
m
)
‍
d
k
)
2
c
i
2
+
o
(
1
)
=
δ
(
γ
4
-
1
)
σ
4
+
o
(
1
)
,
where δ
=
∑
i
=
1
n
‍
c
i
2
/
n. By Slutsky’s theorem, we obtain
(101)
n
(
σ
^
1
2
-
σ
2
)
δ
(
γ
4
-
1
)
σ
4
=
σ
0
δ
(
γ
4
-
1
)
σ
4
×
{
n
(
σ
^
1
2
-
μ
0
σ
2
)
σ
0
+
n
(
μ
0
-
1
)
σ
2
σ
0
}
⟶
d
N
(
0,1
)
,
as
n
⟶
∞
.This proves the theorem.



---

*Source: 585146-2014-07-03.xml*