# Direct Determination of Smoothing Parameter for Penalized Spline Regression

Authors: Takuma Yoshida
Journal: Journal of Probability and Statistics
 (2014)

---

## Abstract

Penalized spline estimator is one of the useful smoothing methods. To construct the estimator, having goodness of fit and smoothness, the smoothing parameter should be appropriately selected. The purpose of this paper is to select the smoothing parameter using the asymptotic property of the penalized splines. The new smoothing parameter selection method is established in the context of minimization asymptotic form of MISE of the penalized splines. The mathematical and the numerical properties of the proposed method are studied. First we organize the new method in univariate regression model. Next we extend to the additive models. A simulation study to confirm the efficiency of the proposed method is addressed.


---

## Body

## 1. Introduction

Penalized spline methods are a well-known efficient technique for nonparametric smoothing. Penalized splines were suggested by O’Sullivan [1] and Eilers and Marx [2]. In O’Sullivan [1], they used a cubic B-spline function and the penalty was the integrated squared second derivative of the B-spline function. On the other hand, Eilers and Marx [2] use a cubic B-spline function and a difference penalty on the spline coefficients. The Eilers and Marx’s estimator is computationally efficient compared to smoothing splines and O’Sullivan’s estimator since it removes the integration part of the penalty. Hence this paper focuses on the penalized spline estimator provided via Eilers and Marx [2]. The penalized spline method is efficient for both univariate regression and multiple regressions such as the additive model (see Marx and Eilers [3]). General properties usages and a description of the flexibility of penalized splines are described in Ruppert et al. [4].When using penalized splines, the determination of the smoothing parameter is very important since it controls the trade-off between the goodness of fit and the smoothness of the fitted curve. As the classical method for achieving this, the grid search method is often used. The grid search method is selected by minimizing one criterion from candidate points of the smoothing parameter. Criteria for grid searches include cross-validation, generalized cross-validation, Mallow’sC
p, and so forth. Although the grid search selection generally finds one optimal smoothing parameter, it is possible that the worth curve is obtained when not all the candidates are good. This tendency is especially striking in additive models since the number of the smoothing parameter is the same as that of the covariates. Several smoothing parameter selection methods using the grid search criteria have been developed by many authors such as Krivobokova [5], Reiss and Ogden [6], Wood [7], Wood [8], and Wood [9]. On the other hand, the mixed model representation of the spline smoothing has also been studied (see Lin and Zhang [10], Wand [11], and Ruppert et al. [4]). In mixed models, the grid search method is not necessary to obtain the final fit curve. The smoothing parameter in the mixed model can be written as the ratio of the variance of the random coefficient and the error. By estimating these unknown variances using a maximum likelihood method or a restricted maximum likelihood method (REML), the final fitted curves are obtained, yielding the estimated best linear unbiased predictor (EBLUP). Therefore the EBLUP does not require a grid search. However the fitted curve tends to theoretically oversmooth and the numerical stability is not guaranteed if a cubic spline is used (see Section 3). The Bayesian approach to select the smoothing parameter has been studied by Fahrmeir et al. [12], Fahrmeir and Kneib [13], and Heinzl et al. [14]. Kauermann [15] compared some smoothing parameter selection methods.In this paper, we propose a new method to determining the smoothing parameter using the asymptotic properties of the penalized splines. For the remainder of this paper, our new method will be known as the direct method. Before describing the outline of the direct method, we will briefly introduce the asymptotic studies of penalized splines. First, Hall and Opsomer [16] showed the consistency of the penalized spline estimator in white noise representation. Subsequently, Li and Ruppert [17], Claeskens et al. [18], Kauermann et al, [19], and Wang et al. [20] have developed the asymptotics for the penalized spline estimator in univariate regression. Yoshida and Naito [21] and Yoshida and Naito [22] have studied the asymptotics for penalized splines in additive regression models and generalized additive models, respectively. Xiao et al. [23] suggested a new penalized spline estimator, and developed its asymptotic properties in bivariate regression. Thus, the developments of the asymptotic theories of the penalized splines are relatively recent events. In addition, the smoothing parameter selection methods using asymptotic properties have not yet been studied. This motivates us to try to establish such methods.The direct method is conducted by minimizing the mean integrated squared error (MISE) of the penalized spline estimator. In general, the MISE of the nonparametric estimator is divided into the integrated squared bias and the integrated variance of the estimator. Of course the penalized spline estimator is no exception and hence the direct method is stated by utilizing the expression of the asymptotic bias and variance of the penalized spline estimator, which have been derived by Claeskens et al. [18], Kauermann et al. [19], and Yoshida and Naito [22]. From their result, we see that the asymptotic order of the variance of the penalized spline estimator is only dependent on the sample size and the number of knots but not the smoothing parameter. However the second term of the asymptotic variance of the penalized spline estimator contains the smoothing parameter and we can see that the variance becomes small when the smoothing parameter increases. On the other hand, the squared bias of the penalized spline estimator increases if the smoothing parameter is reduced. Therefore the minimizer of the MISE of the penalized spline estimator can be seen as one of the optimal smoothing parameters. Since the MISE is asymptotically convex with respect to the smoothing parameter, the global minimum of the MISE can be found. This detection has been sufficiently developed for bandwidth selection in kernel regression (see Ruppert et al. [24], Wand and Jones [25], etc.). First the present paper focuses on univariate regression, and we next extend the direct method to the additive models. In both models, the mathematical and the numerical properties of the direct method are studied. In additive models, we need to select a smoothing parameter of the same number as the explanatory variable, such that the computational cost of the grid search becomes large. We expect that the computational cost of the direct method is dramatically reduced compared to the grid search.The structure of this paper is as follows. In Section2, we introduce a penalized spline estimator in a univariate regression model. Section 3 provides the direct method and related properties. Section 4 extends the direct method to the additive model. In Section 5, we confirm the performance of the direct method using a numerical study. We provide a discussion on the outlook and further studies in Section 6. The proofs of our theorems are provided in the appendix.


## 2. Penalized Spline Estimator

Consider the regression problem withn observations,
(1)
Y
i
=
f
(
x
i
)
+
ε
i
,
i
=
1
,
…
,
n
,
where Y
i is the response variable, x
i is the explanatory variable, f is the true regression function, and ε
i is the random error which is assumed to be independently distributed with mean 0 and variance σ
2. Throughout the paper we assume the explanatory variable x
i
∈
[
a
,
b
]  (
i
=
1
,
…
,
n
) is not a random variable from which the expectation of Y
i can be expressed as E
[
Y
i
∣
x
i
]
=
f
(
x
i
). The support of the explanatory x
i can be relaxed as the real space ℝ. In order to simplify the way of locating the knots in the following, the support of the explanatory is assumed to be with compact space. We aim to estimate f via a nonparametric penalized spline method. We consider the knots a
=
κ
0
<
κ
1
<
⋯
<
κ
K
<
κ
K
+
1
=
b and, for k
=
-
p
,
…
,
K
-
1, let B
k
[
p
]
(
x
) be the pth degree B-spline basis function defined as
(2)
B
k
[
0
]
(
x
)
=
{
1
,
κ
k
<
x
≤
κ
k
+
1
,
0
,
otherwise
,
B
k
[
p
]
(
x
)
=
x
-
κ
k
κ
k
+
p
-
κ
k
B
k
[
p
-
1
]
(
x
)
+
κ
k
+
p
+
1
-
x
κ
k
+
p
+
1
-
κ
k
+
1
B
k
+
1
[
p
-
1
]
(
x
)
associated with the above knots and the additional knots κ
-
p
=
κ
-
p
+
1
=
⋯
=
κ
0 and κ
K
+
1
=
⋯
=
κ
K
+
p
+
1. The B-spline function B
k
[
p
]
(
x
) is a piecewise pth degree polynomial on an interval [
κ
k
,
κ
k
+
p
+
1
]. The details of the B-spline basis functions are described in de Boor [26]. For simplicity, we write B
k
(
x
)
=
B
k
[
p
]
(
x
) since we do not specify the p in the following sentence.We use the linear combination of{
B
k
(
x
)
∣
k
=
-
p
,
…
,
K
-
1
} and the unknown parameters b
k
(
k
=
-
p
,
…
,
K
-
1
) to approximate the regression function and consider the B-spline regression problem,
(3)
Y
i
=
s
(
x
i
)
+
ε
i
,
i
=
1
,
…
,
n
,
where
(4)
s
(
x
)
=
∑
k
=
-
p
K
-
1
‍
B
k
(
x
)
b
k
.
The purpose is to estimate the parameter b
=
(
b
-
p
⋯
b
K
-
1
)
T included in s
(
x
) instead of f directly. The penalized spline estimator b
^
=
(
b
^
-
p
⋯
b
^
K
-
1
)
T of b is defined as the minimizer of
(5)
∑
i
=
1
n
‍
{
Y
i
-
∑
k
=
-
p
K
-
1
‍
B
k
(
x
)
b
k
}
2
+
λ
∑
k
=
m
-
p
K
-
1
‍
{
Δ
m
(
b
k
)
}
2
,
where λ is the smoothing parameter and Δ is the backward difference operator defined as Δ
b
k
=
b
k
-
b
k
-
1 and
(6)
Δ
m
(
b
k
)
=
Δ
m
-
1
(
Δ
b
k
)
=
∑
j
=
0
m
‍
(
-
1
)
m
-
j
m
C
j
b
k
-
m
+
j
.
Let D
m
=
(
d
i
j
(
m
)
)
i
j be a (
K
+
p
-
m
)
×
(
K
+
p
) matrix, where d
i
j
(
m
)
=
(
-
1
)
|
i
-
j
|
m
C
|
i
-
j
| for i
≤
j
≤
m
+
1 and 0 otherwise. Using the notation Y
=
(
Y
1
⋯
Y
n
)
T and Z
=
(
B
-
p
+
j
-
1
(
x
i
)
)
i
j, (5) can then be expressed as
(7)
(
Y
-
Z
b
)
T
(
Y
-
Z
b
)
+
λ
b
T
D
m
T
D
m
b
.
The minimum of (7) is obtained when
(8)
b
^
=
(
Z
T
Z
+
λ
D
m
T
D
m
)
-
1
Z
T
Y
.
The penalized spline estimator f
^
(
x
) of f
(
x
) for x
∈
[
a
,
b
] is defined as
(9)
f
^
(
x
)
=
∑
k
=
-
p
K
-
1
‍
B
k
(
x
)
b
^
k
=
B
(
x
)
T
b
^
,
where B
(
x
)
=
(
B
-
p
(
x
)
⋯
B
K
-
1
(
x
)
)
T.Ifλ
≡
0, f
^
(
x
) is reduced to the regression spline estimator, which is the spline estimator obtained via the least squares method. The regression spline estimator will lead to an oscillatory fit if the number of knots K is large. However the determination of K and the location of knots are very difficult problems. The advantage of the penalized spline smoothing is that the good smoothing parameter brings the estimator to the curve with the fitness and smoothness simultaneously without choosing the number and location of knots precisely. In the present paper, we use equidistant knots κ
k
=
a
+
k
/
K and focus on the determination of the smoothing parameter. As the location of knots other than above, the quantiles of the data points {
x
1
,
…
,
x
n
} are often used (see Ruppert [27]). However it is known that the penalized spline estimator is hardly affected by the location of knots if K is not too small. Therefore we do not discuss the location of knots. We suggest the direct method for this in the next section.


## 3. Direct Determination of Smoothing Parameter

In this section, we provide the direct method for determining the smoothing parameter without a grid search. This direct method is given theoretical justification by asymptotic theory of the penalized spline estimator. To investigate the asymptotic property of the penalized spline estimator, we assume thatf
∈
C
p
+
1, K
=
o
(
n
1
/
2
), and λ
=
O
(
n
/
K
1
-
m
).For convenience we first give some notation. LetG
n
=
n
-
1
Z
T
Z and Λ
n
=
G
n
+
(
λ
/
n
)
D
m
T
D
m. Let b
* be a best L
∞ approximation to the true function f. This means that b
* satisfies
(10)
sup
⁡
x
∈
(
a
,
b
)
|
f
(
x
)
+
K
-
(
p
+
1
)
b
a
(
x
)
-
B
(
x
)
T
b
*
|
=
o
(
K
-
(
p
+
1
)
)
,
as
K
⟶
∞
,
where
(11)
b
a
(
x
)
=
-
f
(
p
+
1
)
(
x
)
(
p
+
1
)
!
∑
k
=
1
K
‍
I
(
κ
k
-
1
≤
x
<
κ
k
)
B
r
p
+
1
(
x
-
κ
k
-
1
K
-
1
)
,
I
(
a
<
x
<
b
) is the indicator function of an interval (
a
,
b
), and B
r
p
(
x
) is the pth Bernoulli polynomial (see Zhou et al. [28]). It can be easily shown that b
a
(
x
)
=
O
(
1
) as K
→
∞.The penalized spline estimator can be written as(12)
f
^
(
x
)
=
B
(
x
)
T
(
Z
T
Z
)
-
1
Z
T
Y
-
λ
n
B
(
x
)
T
Λ
n
-
1
D
m
T
D
m
G
n
-
1
(
Z
n
)
T
Y
.
The first term of the right hand side of (12) is equal to the regression spline estimator denoted by f
^
rs
(
x
). The asymptotics for the regression spline estimator has been developed by Zhou et al. [28] and can be expressed as
(13)
E
[
f
^
rs
(
x
)
]
=
f
(
x
)
+
K
-
(
p
+
1
)
b
a
(
x
)
+
o
(
K
-
(
p
+
1
)
)
,
V
[
f
^
rs
(
x
)
]
=
σ
2
n
B
(
x
)
T
G
n
-
1
B
(
x
)
{
1
+
o
(
1
)
}
=
O
(
K
n
)
.
From Theorem 2(a) of Claeskens et al. [18], we have
(14)
E
[
f
^
(
x
)
]
-
E
[
f
^
rs
(
x
)
]
=
-
λ
n
B
(
x
)
T
Λ
n
-
1
D
m
T
D
m
b
*
+
o
(
λ
K
1
-
m
n
)
=
O
(
λ
K
1
-
m
n
)
,
V
[
f
^
(
x
)
]
=
V
[
f
^
rs
(
x
)
]
-
λ
K
n
C
(
x
∣
n
,
K
,
λ
)
K
+
o
(
λ
K
2
n
2
)
,
where
(15)
C
(
x
∣
n
,
K
,
λ
)
=
2
σ
2
n
B
(
x
)
T
Λ
n
-
1
D
m
T
D
m
G
n
-
1
B
(
x
)
is the covariance of f
^
rs
(
x
) and the second term of the right hand side of (12). The variance of the second term of (12) can be shown to be negligible (see the appendix). The following theorem leads to λ controlling the trade-off between the squared bias and variance of the penalized spline estimator.Theorem 1.
The covarianceC
(
x
∣
n
,
K
,
λ
)
/
K in (15) is positive. Furthermore, as n
→
∞, C
(
x
∣
n
,
K
,
λ
)
=
C
(
x
∣
n
,
K
,
0
)
(
1
+
o
(
1
)
) and C
(
x
∣
n
,
K
,
0
)
/
K
=
O
(
K
/
n
).From the asymptotic form ofE
[
f
^
(
x
)
] and V
[
f
^
(
z
)
] and Theorem 1, we see that, for small λ, the bias of f
^
(
x
) is small and the variance becomes large. On the other hand, the large λ indicates the bias of f
^
(
x
) increases and the variance decreases. From Theorem 1, the MISE of f
^
(
x
) can be expressed as
(16)
∫
a
b
‍
E
[
{
f
^
(
x
)
-
f
(
x
)
}
2
]
d
x
=
∫
a
b
‍
{
K
-
(
p
+
1
)
b
a
(
x
)
-
λ
n
B
(
x
)
T
Λ
n
-
1
D
m
T
D
m
b
*
}
2
d
x
+
σ
2
n
∫
a
b
‍
B
(
x
)
T
G
n
B
(
x
)
d
x
-
λ
K
n
∫
a
b
‍
C
(
x
∣
n
,
K
,
0
)
d
x
K
+
r
1
n
(
K
)
+
r
2
n
(
K
,
λ
)
,
where r
1
n
(
K
) and r
2
n
(
K
,
λ
) are of negligible order, respectively, compared to the regression spline and penalized spline of the second term of the right hand side of (12). Actually we have r
1
n
(
K
)
=
o
(
K
/
n
) and r
2
n
(
K
,
λ
)
=
o
(
λ
2
K
2
(
1
-
m
)
/
n
2
). The MISE of f
^
(
x
) is asymptotically quadratic and a global minimum exists. Let
(17)
MISE
(
λ
)
=
λ
2
n
2
∫
a
b
‍
{
B
(
x
)
T
Λ
n
-
1
D
m
T
D
m
b
*
}
2
d
x
-
λ
n
∫
a
b
‍
{
2
b
a
(
x
)
B
(
x
)
T
Λ
n
-
1
D
m
T
D
m
b
*
K
p
+
1
-
λ
n
∫
a
b
‍
+
C
(
x
∣
n
,
K
,
0
)
b
a
(
x
)
B
(
x
)
T
Λ
n
-
1
D
m
T
D
m
b
*
K
p
+
1
}
d
x
.
And let λ
opt be the minimizer of MISE(λ). We suggest the use of λ
opt as the optimal smoothing parameter,
(18)
λ
opt
=
n
2
∫
a
b
‍
D
2
n
(
x
∣
f
(
p
+
1
)
,
b
*
)
d
x
∫
a
b
‍
D
1
n
(
x
∣
b
*
)
d
x
,
where
(19)
D
1
n
(
x
∣
b
*
)
=
{
B
(
x
)
T
G
n
-
1
D
m
T
D
m
b
*
}
2
,
D
2
n
(
x
∣
f
(
p
+
1
)
,
b
*
)
=
2
b
a
(
x
)
B
(
x
)
T
G
n
-
1
D
m
T
D
m
b
*
K
p
+
1
+
C
(
x
∣
n
,
K
,
0
)
.
However it is easy to see that MISE(λ) and λ
opt contain an unknown function and parameters, and hence these must be estimated. We construct the estimator of λ
opt by using the consistent estimator of f
(
p
+
1
) and b
*. We can use the penalized spline estimator and its derivative as the pilot estimator of f
(
p
+
1
) and b
*. If we then use another smoothing parameter λ
0, it should be chosen appropriately. Therefore we use the regression spline estimator as the pilot estimator of f
(
p
+
1
)
(
x
) and b
*. First we establish b
~
=
(
Z
T
Z
)
-
1
Z
T
Y. Next we construct the pilot estimator with f
(
p
+
1
)
(
x
) by using the (
p
+
2
)th degree B-spline basis. Let B
[
p
]
(
x
)
=
(
B
-
p
[
p
]
(
x
)
⋯
B
K
-
1
[
p
]
(
x
)
)
T. Using the fundamental property of the B-spline function, f
(
m
)
(
x
) can be written as f
~
(
m
)
(
x
)
=
B
[
p
-
m
]
(
x
)
T
D
m
b
* asymptotically. Hence the regression spline estimator f
~
(
p
+
1
) can be constructed as f
~
(
p
+
1
)
(
x
)
=
B
[
p
+
1
]
(
x
)
T
D
p
+
1
b
(
p
+
2
), where b
(
p
+
2
)
=
(
(
Z
[
p
+
2
]
)
T
Z
[
p
+
2
]
)
-
1
(
Z
[
p
+
2
]
)
T
Y and Z
[
p
+
2
]
=
(
B
-
p
+
j
-
1
[
p
+
2
]
(
x
i
)
)
i
j. Since the regression spline estimator tends to be oscillatory with a higher order pth spline function, the fewer knots are used to construct f
~
(
p
+
1
)
(
x
). The variance σ
2 included in C
(
x
∣
n
,
K
,
0
) is estimated via
(20)
σ
^
2
=
1
n
-
(
K
+
p
)
∑
i
=
1
n
‍
{
Y
i
-
B
(
x
i
)
T
b
~
}
2
.
Using the above pilot estimators,
(21)
λ
^
opt
=
n
2
∑
j
=
1
J
D
2
(
z
j
∣
f
~
(
p
+
1
)
,
b
~
)
∑
j
=
1
J
D
1
(
z
j
∣
b
~
)
,
with some finite grid points {
z
j
}
1
J on [
a
,
b
].Consequently the final penalized spline estimator is defined as(22)
f
^
(
x
)
=
B
(
x
)
T
b
^
opt
,
where
(23)
b
^
opt
=
(
Z
T
Z
+
λ
^
opt
D
m
T
D
m
)
-
1
Z
T
Y
.It is known that the optimal order ofK of the penalized spline estimator is the same as that of the regression spline estimator, K
=
O
(
n
1
/
(
2
p
+
3
)
) (see Kauermann et al. [19] and Zhou et al. [28]). Using this, we show the asymptotic property of λ
opt in following theorem.Theorem 2.
Letf
∈
C
p
+
1. Suppose that K
=
o
(
n
1
/
2
) and λ
=
o
(
n
/
K
1
-
m
). Then λ
o
p
t given in (21) exists, and λ
o
p
t
=
O
(
n
K
m
-
p
-
2
+
K
2
m
) as n
→
∞. Furthermore K
=
O
(
n
1
/
(
2
p
+
3
)
) leads to the optimal order
(24)
λ
o
p
t
=
O
(
n
(
p
+
m
+
1
)
/
(
2
p
+
3
)
)
+
O
(
n
2
m
/
(
2
p
+
3
)
)
and the rate of convergence of MISE of f
^
(
x
) becomes O
(
n
-
2
(
p
+
1
)
/
(
2
p
+
3
)
).The proof of Theorem2 is given in the appendix. At the end of this section, we give a few remarks.Remark 3.
The asymptotic order of the squared bias and variance of the penalized splines areO
(
K
-
2
(
p
+
1
)
)
+
O
(
λ
2
K
2
(
1
-
m
)
/
n
2
) and O
(
K
/
n
), respectively. Therefore under K
=
(
n
1
/
(
2
p
+
3
)
) and λ
=
O
(
n
(
p
+
m
+
1
)
/
(
2
p
+
3
)
), the optimal rate of convergence of MISE of the penalized splines is O
(
n
-
2
(
p
+
1
)
/
(
2
p
+
3
)
). From Theorem 2, we see that the asymptotic order of λ
opt yields the optimal rate of convergence.Remark 4.
O’Sullivan [1] used μ
∫
a
b
‍
{
s
(
m
)
(
x
)
}
2
d
x as the penalty term, where μ is the smoothing parameter. When equidistant knots are used, the penalty ∫
a
b
‍
{
s
(
m
)
(
x
)
}
2
d
x can be expressed as K
2
m
b
T
D
m
T
R
D
m
b, where R
=
(
∫
a
b
‍
B
i
[
p
-
m
]
(
x
)
B
j
[
p
-
m
]
(
x
)
)
i
j. The penalty b
T
D
m
T
D
m
b provided by Eilers and Marx [2] can be seen as the simple version of K
2
m
b
T
D
m
T
R
D
m
b by replacing R with K
-
1
I and λ
=
μ
K
2
m
-
1, where I is the identity matrix. So the order m of the difference matrix D
m controls the smoothness of the pth B-spline function s
(
x
), and hence m should be set such that m
≤
p to give a theoretical justification although the penalized spline estimator can be also calculated for m
>
p. Actually, (
p
,
m
)
=
(
3,2
) is often used by many authors.Remark 5.
The penalized spline regression is often considered as the mixed model representation (see Ruppert et al. [4]). In this frame work, we use the pth truncated spline model
(25)
s
(
x
)
=
β
0
+
β
1
x
+
⋯
+
β
p
x
p
+
∑
j
=
1
K
‍
u
k
(
x
-
κ
k
)
+
p
,
where (
x
-
a
)
+
=
max
⁡
{
x
-
a
,
0
} and the β’s are unknown parameters. Each u
k is independently distributed as u
k
~
N
(
0
,
σ
u
2
), where σ
u
2 is an unknown variance parameter of u
k. The penalized spline fit of f
(
x
) is defined as the estimated BLUP (see Robinson [29]). The smoothing parameter in the ordinal spline regression model corresponds to σ
2
/
σ
u
2 in the spline mixed model. Since the σ
2 and σ
u
2 are estimated using the ML or REML method, we do not need to choose the smoothing parameter via a grid search. It is known that the estimated BLUP fit is linked to the penalized spline estimator (9) with m
=
p
+
1 (see Kauermann et al. [19]). Hence the estimated BLUP tends to have a theoretically underfit (see Remark 4).Remark 6.
From Lyapunov’s theorem, the asymptotic normality of the penalized spline estimatorf
^
(
x
) with λ
^
opt can be derived under the same assumption as Theorem 2 and some additional mild conditions. Although the proof is omitted, it is straightforward since λ
^
opt is the consistent estimator of λ
opt and the asymptotic order of λ
opt satisfies Lyapunov’s condition.


## 4. Extension to Additive Models

We extend the direct method to the regression model with multidimensional explanatory variables. In particular, we consider additive models in this section. For the dataset{
(
Y
i
,
x
i
)
:
i
=
1
,
…
,
n
} with 1-dimensional response Y
i and d-dimensional explanatory x
i
=
(
x
i
1
,
…
,
x
i
d
), the additive models are connected via unknown regression functions f
j
(
j
=
1
,
…
,
d
) and the mean parameter μ such that
(26)
Y
i
=
μ
+
f
1
(
x
i
1
)
+
⋯
+
f
d
(
x
i
d
)
+
ε
i
.
We assume that x
i
j is located on an interval [
a
j
,
b
j
] and f
1
,
…
,
f
d are normalized as
(27)
∫
a
j
b
j
‍
f
j
(
u
)
d
u
=
0
,
j
=
1
,
…
,
d
,
to ensure the identifiability of f
j. Then the intercept μ is typically estimated via
(28)
μ
^
=
1
n
∑
i
=
1
n
‍
Y
i
.
Hence we replace Y
i with Y
i
-
μ
^ in (26) and set μ
=
0, redefining the additive model as
(29)
Y
i
=
f
1
(
x
i
1
)
+
⋯
+
f
d
(
x
i
d
)
+
ε
i
,
i
=
1
,
…
,
n
,
where each Y
i is centered. We aim to estimate f
j via a penalized spline method. Let
(30)
s
j
(
x
)
=
∑
k
=
-
p
K
j
-
1
‍
B
j
,
k
(
x
)
b
j
,
k
,
j
=
1
,
…
,
d
,
be the B-spline model, where B
j
,
k
(
x
) is the pth B-spline basis function with knots {
κ
j
,
k
∣
k
=
-
p
,
…
,
K
j
+
p
+
1
,
κ
j
,
0
=
a
j
,
κ
j
,
K
j
+
1
=
b
j
,
j
=
1
,
…
,
d
} and b
j
,
k’s are unknown parameters. We consider the B-spline additive models
(31)
Y
i
=
s
1
(
x
i
1
)
+
⋯
+
s
d
(
x
i
d
)
+
ε
i
,
i
=
1
,
…
,
n
,
and estimate s
j and b
j
,
k. For j
=
1
,
…
,
d, the penalized spline estimator b
^
j
=
(
b
^
j
,
-
p
⋯
b
^
j
,
K
-
1
)
T of b
j
=
(
b
j
,
-
p
⋯
b
j
,
K
-
1
)
T is defined as
(32)
[
b
^
1
⋮
b
^
d
]
=
argmin
b
1
,
…
,
b
d
∑
i
=
1
n
‍
{
Y
i
-
∑
j
=
1
d
‍
∑
k
=
-
p
K
j
‍
B
j
,
k
(
x
i
)
b
j
,
k
}
2
+
∑
j
=
1
d
‍
λ
j
b
j
T
D
j
m
T
D
j
m
b
j
,
where λ
j are the smoothing parameters and D
j
m is the mth order difference matrix with size (
K
j
+
p
-
m
)
×
(
K
j
+
p
) for j
=
1
,
…
,
d. By using b
^
j, the penalized spline estimator of f
j
(
x
j
) is defined as
(33)
f
^
j
(
x
j
)
=
∑
k
=
-
p
K
j
-
1
‍
B
j
,
k
(
x
j
)
b
^
j
,
k
,
j
=
1
,
…
,
d
.The asymptotics forf
^
j
(
x
j
) have been studied by Yoshida and Naito [22] who derived the asymptotic bias and variance to be
(34)
E
[
f
^
j
(
x
j
)
]
=
f
j
(
x
j
)
+
K
j
-
(
p
+
1
)
b
j
a
(
x
j
)
(
1
+
o
(
1
)
)
+
λ
j
n
B
j
(
x
j
)
T
Λ
j
n
-
1
D
j
m
T
D
j
m
b
j
*
+
o
(
λ
j
K
j
1
-
m
n
)
,
V
[
f
^
j
(
x
j
)
]
=
σ
2
n
B
j
(
x
j
)
T
G
j
n
-
1
B
j
(
x
j
)
-
λ
K
n
σ
2
C
j
(
x
j
∣
n
,
K
j
)
K
+
o
(
λ
j
K
j
2
n
2
)
,
where B
j
(
x
)
=
(
B
-
p
(
x
)
⋯
B
K
j
-
1
(
x
)
)
T, G
j
n
=
n
-
1
Z
j
T
Z
j, Λ
j
n
=
G
j
n
+
(
λ
j
/
n
)
D
m
T
D
m, Z
j
=
(
B
-
p
+
k
-
1
(
x
i
j
)
)
i
k, and b
j
* is the best L
∞ approximation of f
j,
(35)
b
j
a
(
x
)
=
-
f
j
(
p
+
1
)
(
x
)
(
p
+
1
)
!
∑
k
=
0
K
j
-
1
‍
I
(
κ
j
,
k
≤
x
<
κ
j
,
k
+
1
)
×
B
p
+
1
(
x
-
κ
j
,
k
K
j
-
1
)
,
C
j
(
x
j
∣
n
,
K
j
)
=
2
n
B
j
(
x
j
)
T
G
j
n
-
1
D
j
m
T
D
j
m
G
n
-
1
B
j
(
x
j
)
.
The above asymptotic bias and variance of f
^
j
(
x
j
) are similar to that of the penalized spline estimator in univariate regression with {
(
Y
i
,
x
i
j
)
:
i
=
1
,
…
,
n
}. Furthermore the asymptotic normality of [
f
^
1
(
x
1
)
⋯
f
^
d
(
x
d
)
]
T has been shown by Yoshida and Naito [22]. From their paper, we find that f
^
i
(
x
i
) and f
^
j
(
x
j
) are then asymptotically independent for i
≠
j. This indicates some theoretical justification to select λ
j in minimizing the MISE of f
^
j. Similar to the discussion in Section 3, the minimizer λ
j
,
opt of the MISE of f
^
j
(
x
j
) can be obtained for j
=
1
,
…
,
d,
(36)
λ
j
,
opt
=
n
2
∫
a
j
b
j
‍
L
j
n
(
x
∣
f
j
(
p
+
1
)
,
b
j
*
,
σ
2
)
d
x
∫
a
j
b
j
‍
H
j
n
(
x
∣
b
j
*
)
d
x
,
where
(37)
H
j
n
(
x
∣
b
j
*
)
=
{
B
j
(
x
j
)
T
G
j
n
-
1
D
j
m
T
D
j
m
b
j
*
}
2
,
L
j
n
(
x
∣
f
j
(
p
+
1
)
,
b
j
*
,
σ
2
)
=
2
b
a
(
x
)
B
j
(
x
j
)
T
G
j
n
-
1
D
j
m
T
D
j
m
b
j
*
K
j
p
+
1
+
σ
2
C
j
(
x
∣
n
,
K
j
)
.
Since f
j, b
j
*, and σ
2 are unknown, they should be estimated. The pilot estimators of f
j, b
j
*, and σ
2 are constructed based on the regression spline method. By using the pilot estimators f
~
j, b
~
j of f
j, b
j
* and the estimator of σ
2, we construct the estimator of λ
j
,
opt:
(38)
λ
^
j
,
opt
=
n
2
∑
r
=
1
R
L
j
n
(
z
r
∣
f
~
j
(
p
+
1
)
,
b
~
j
,
σ
^
2
)
∑
r
=
1
R
H
j
n
(
z
r
∣
b
~
j
)
,
where {
z
r
}
1
R is some finite grid point sequence on [
a
j
,
b
j
]. We obtain for j
=
1
,
…
,
d, the penalized spline estimator f
^
j
(
x
j
) of f
j
(
x
j
),
(39)
f
^
j
(
x
j
)
=
B
j
(
x
j
)
T
b
^
j
,
opt
,
where b
^
j
,
opt is the penalized spline estimator of b
j using λ
^
j
,
opt. From Theorem 2 and the proof of Theorem 3.4 of Yoshida and Naito [22], the asymptotic normality of [
f
^
1
(
x
1
)
⋯
f
^
d
(
x
d
)
]
T using (
λ
1
,
opt
,
…
,
λ
d
,
opt
) can be shown.Remark 7.
Since the true regression functions are normalized, the estimatorf
^
j should be also centered as
(40)
f
^
j
(
x
j
)
-
1
n
∑
i
=
1
n
‍
f
^
j
(
x
i
j
)
.Remark 8.
The penalized spline estimator ofb
1
,
…
,
b
d can be obtained using a backfitting algorithm (Hastie and Tibshirani [30]). The backfitting algorithm for the penalized splines in additive regression is detailed in Marx and Eilers [3] and Yoshida and Naito [21].Remark 9.
Although we focus on the nonparametric additive regression in this paper, the direct method can be also applied to the generalized additive models. However we omit this discussion because the procedure is similar to the case of the additive models discussed in this section.Remark 10.
The direct method is quite computationally efficient when compared to the grid search method in additive models. In grid searches, we prepare the candidate ofλ
j. Let M
j be the set of all possible candidate grid value of λ
j for j
=
1
,
…
,
d. Then we need to compute the backfitting algorithm {
M
1
×
⋯
×
M
d
} times. On the other hand, it is sufficient to perform the backfitting algorithm for only two steps for the pilot estimator and the final penalized spline estimator. Thus, compared with the conventional grid search method, the direct method can drastically reduce computation time.


## 5. Numerical Study

In this section, we investigate the finite sample performance of the proposed direct method in a Monte Carlo simulation. Let us first consider the univariate regression model (1) for the data {
(
Y
i
,
x
i
)
:
i
=
1
,
…
,
n
}. Then we use the three types of true regression function f
(
x
)
=
cos
⁡
(
π
(
x
-
0.3
)
), f
(
x
)
=
ϕ
(
(
x
-
0.8
)
/
0.05
)
/
0.05
-
ϕ
(
(
x
-
0.2
)
/
0.04
)
/
0.04, and f
(
x
)
=
2
x
3
+
3
sin
(
2
π
(
x
-
0.8
)
3
)
+
3
exp
⁡
[
-
(
x
-
0.5
)
2
/
0.1
], which are labeled by F1, F2, and F3, respectively. Here ϕ
(
x
) is the density function of the normal distribution. The explanatory x
i and error ε
i are independently generated from uniform distribution on [
0,1
] and N
(
0,0
.
5
2
), respectively. We estimate each true regression function via the penalized spline method. We then use the linear and cubic B-spline bases with equidistant knots and the second order difference penalty. In addition we set K
=
5
n
2
/
5 equidistant knots and the smoothing parameter is determined by the direct method. The penalized spline estimator with the linear spline and cubic spline are denoted as L-Direct and C-Direct, respectively. For comparison with L-Direct, the same studies are also implemented for the penalized spline estimator with a linear spline and the smoothing parameter selected via GCV and restricted maximum likelihood method (REML) in mixed model representation, and the local polynomial estimator with normal kernel and Plug-in bandwidth (see Ruppert et al. [24]). In GCV, we set the candidate values of λ as {
i
/
10
:
i
=
0,1
,
…
,
99
}. The above three estimators are denoted by L-GCV, L-REML, and local linear, respectively. Furthermore we compare C-Direct with C-GCV and C-REML, which are the penalized spline estimator with the cubic spline and the smoothing parameter determined by GCV and REML. Let
(41)
sMISE
=
1
J
∑
j
=
1
J
‍
[
1
R
∑
r
=
1
R
‍
{
f
^
r
(
z
j
)
-
f
(
z
j
)
}
2
]
be the sample MISE of any estimator f
^
(
x
) of f
(
x
), where f
^
r
(
z
) is f
^
(
z
) with rth replication and z
j
=
j
/
J. We calculate the sample MISE of the penalized spline estimator with the direct method, GCV, and REML and the local linear estimator. In this simulation, we use J
=
100 and R
=
1000. We have simulated n
=
50 and 200.The sMISE of all estimators for each model andn are given in Table 1. The penalized spline estimator using the direct method shows good performance in each setting. In comparison with other smoothing parameter methods, the direct method is a little better than the GCV as a whole. However for n
=
200, C-GCV is better than C-Direct in F1 and F3 though its difference is very small. We see that the sMISE of C-Direct is smaller than local linear, whereas local linear behaves better than L-Direct in some case. In F2, C-Direct is the smallest of all estimators for n
=
50 and 200. Although the performance totally seems to depend on situation in which data sets are generated, we believe that the proposed method is one of the efficient methods.Table 1
Results of sample MISE forn
=
50 and n
=
200. All entries for MISE are 1
0
2 times their actual values.
True
F1
F2
F3
Sample size
n
=
50
n
=
200
n
=
50
n
=
200
n
=
50
n
=
200
L-Direct
0.526
0.322
3.684
1.070
1.160
0.377
L-GCV
0.726
0.621
5.811
1.082
1.183
0.379
L-REML
2.270
1.544
9.966
3.520
1.283
0.873
Local linear
0.751
0.220
4.274
0.901
1.689
0.503
C-Direct
0.401
0.148
3.027
0.656
1.044
0.326
C-GCV
0.514
0.137
3.526
0.666
1.043
0.290
C-REML
1.326
0.732
8.246
4.213
3.241
0.835Next the difference betweenλ
^
opt and λ
opt is investigated empirically. Let λ
^
opt
,
r be the λ
^
opt with rth replications for r
=
1
,
…
,
1000. Then we calculate the sample MSE of λ
^
opt:
(42)
sMSE
=
1
R
∑
r
=
1
R
‍
{
λ
^
opt
,
r
-
λ
opt
}
2
for F1, F2, and F3 and n
=
50 and 200. To construct λ
opt for F1, F2, and F3, we use true f
(
4
)
(
x
) and σ
2 and an approximate b
*. Here the approximate b
* means the sample average of b
~ with n
=
200 and 1000 replications.In Table2, the sMSE of λ
^
opt for each true function are described. For comparison, the sMSE of the smoothing parameter obtained via GCV and REML are calculated. The sMSE of the L-Direct and the C-Direct is small even in n
=
50 for all true functions. Therefore it seems that the accuracy of λ
^
opt is guaranteed. It indicates that the pilot estimator constructed via least squares method is not bad. The sMSE with the direct method are smaller than that with GCV and REML. This result is not surprising since GCV and REML are not concerned with λ
opt. However together with Table 1, it seems that the sample MSE of the smoothing parameter is reflected in the sample MISE of the estimator.Table 2
Results of sample MSE of smoothing parameter obtained by direct method, GCV, and REML forn
=
50 and n
=
200. All entries for MSE are 1
0
2 times their actual values.
True
F1
F2
F3
Sample size
n
=
50
n
=
200
n
=
50
n
=
200
n
=
50
n
=
200
L-Direct
0.331
0.193
0.113
0.043
0.025
0.037
L-GCV
0.842
0.342
0.445
0.101
0.072
0.043
L-REML
1.070
1.231
0.842
0.437
0.143
0.268
C-Direct
0.262
0.014
0.082
0.045
0.053
0.014
C-GCV
0.452
0.123
0.252
0.092
0.085
0.135
C-REML
0.855
0.224
0.426
0.152
0.093
0.463The proposed direct method was derived based on the MISE. On the other hand the GCV and the REML are obtained in context of minimizing prediction squared error (PSE) and prediction error. Hence we compare the sample PSE of the penalized spline estimator with the direct method, GCV and REML, and the local linear estimator. Since the prediction error is almost the same as MISE (see Section 4 of Ruppert et al. [4]), we omit the comparison of the prediction error. Let
(43)
sPSE
=
1
n
∑
i
=
1
n
‍
[
1
R
∑
r
=
1
R
‍
{
y
i
r
-
f
^
r
(
x
i
)
}
2
]
be the sample PSE for any estimator f
^
(
x
), where y
i
r
(
r
=
1
,
…
,
R
) is independently generated from Y
i
~
N
(
f
(
x
i
)
,
(
0.5
)
2
) for all i.In Table3, the modified sPSE, |
sPSE
-
(
0.5
)
2
|, of all estimators for each model and n are described. In Remark 11, we discuss the modified sPSE. From the result, we can confirm that the direct method has good predictability. GCV can be regarded as the estimator of sPSE. Therefore in some case, sPSE with GCV is smaller than that with the direct method. It seems that the cause is the accuracy of the estimates of variance (see Remark 11). However its difference is very small.Table 3
Results of sample PSE forn
=
50 and n
=
200. All entries for PSE are 1
0
2 times their actual values.
True
F1
F2
F3
Sample size
n
=
50
n
=
200
n
=
50
n
=
200
n
=
50
n
=
200
L-Direct
0.424
0.052
0.341
0.070
0.360
0.117
L-GCV
0.331
0.084
0.381
0.089
0.333
0.139
L-REML
0.642
0.124
0.831
1.002
0.733
0.173
Local linear
0.751
0.220
0.474
0.901
0.489
0.143
C-Direct
0.341
0.48
0.237
0.63
0.424
0.086
C-GCV
0.314
0.43
0.326
0.76
0.533
0.089
C-REML
0.863
1.672
0.456
1.21
1.043
0.125We admit that the computational time (in second) taken to obtainf
^
(
x
) for F1, p
=
3, and n
=
200. The fits with the direct method, GCV and REML took 0.04, 1.22, and 0.34. Although the difference is small, the computational time of the direct method was faster than that of GCV and REML.Next we confirm the behavior of the penalized spline estimator with the direct method in the additive model. For the data{
(
Y
i
,
x
i
1
,
x
i
2
,
x
i
3
)
:
i
=
1
,
…
,
n
}, we assume the additive model with true functions f
1
(
x
1
)
=
sin
(
2
π
x
1
), f
2
(
x
2
)
=
ϕ
(
(
x
2
-
0.5
)
/
0.2
), and f
3
(
x
3
)
=
0.4
ϕ
(
(
x
3
-
0.1
)
/
0.2
)
+
0.6
ϕ
(
(
x
2
-
0.8
)
/
0.2
). The error is similar to the first simulation. The design points (
x
i
1
,
x
i
2
,
x
i
3
) are generated by
(44)
[
x
i
1
x
i
2
x
i
3
]
=
[
(
1
+
ρ
+
ρ
2
)
-
1
0
0
0
(
1
+
2
ρ
)
-
1
0
0
0
(
1
+
ρ
+
ρ
2
)
-
1
]
×
[
1
ρ
ρ
2
ρ
1
ρ
ρ
2
ρ
1
]
[
z
i
1
z
i
2
z
i
3
]
,
where z
i
j
(
i
=
1
,
…
,
n
,
j
=
1,2
,
3
) are generated independently from the uniform distribution on [
0,1
]. In this simulation, we adopt ρ
=
0 and ρ
=
0.5. We then corrected to satisfy
(45)
∫
0
1
‍
f
j
(
z
)
d
z
=
0
,
j
=
1,2
,
3
in each ρ. We construct the penalized spline estimator via the backfitting algorithm with the cubic spline and second order difference penalty. The number of equidistant knots is K
j
=
5
n
2
/
5 and the smoothing parameters are determined using the direct method. The pilot estimator to construct λ
^
j
,
opt is the regression spline estimator with fifth spline and K
j
=
n
2
/
5. We calculate the sample MISE of each f
^
j for n
=
200 and 1000 Monte Carlo iterations. Then we calculate the sample MISE of f
^
1, f
^
2, and f
^
3. In order to compare with the direct method, we also conduct the same simulation with GCV and REML. In GCV, we set the candidate values of λ
j as {
i
/
10
:
i
=
0,1
,
…
,
9
} for j
=
1,2
,
3. Table 4 summarizes the sample MISE of f
^
j
(
j
=
1,2
,
3
) denoted by MISEj for direct method, GCV, and REML. The penalized spline estimator with the direct method performs like that with the GCV in both the uncorrelated and correlated design cases. For ρ
=
0, the behaviors of MISE1, MISE2, and MISE3 with the direct method are similar. On the other hand, for GCV, the MISE1 is slightly larger than MISE2 and MISE3. The direct method leads to an efficient estimate of all covariates. On the whole, the direct method is better than REML. From above, we believe that the direct method is preferable in practice.Table 4
Results of sample MISE of the penalized spline estimator. For each direct method, GCV andρ, MISE1, MISE2, and MISE3 are the sample MISE of f
^
1, f
^
2, and f
^
3, respectively. All entries for MISE are 1
0
2 times their actual values.
Method
ρ
=
0
ρ
=
0.5
MISE1
MISE2
MISE3
MISE1
MISE2
MISE3
Direct
0.281
0.209
0.205
0.916
0.795
0.972
GCV
0.687
0.237
0.390
0.929
0.857
1.112
REML
1.204
0.825
0.621
2.104
1.832
2.263To confirm the consistency of the direct method, the sample MSE ofλ
^
j
,
opt is calculated the same manner as that given in univariate regression case. For comparison, the sample MSE of GCV and REML are obtained. Here the true λ
j
,
opt
(
j
=
1,2
,
3
) is defined in the same way as that for univariate case. In Table 5, the results for each f
1, f
2, and f
3, and ρ
=
0 and 0.5 are shown. We see from the results that the behavior of the direct method is good. The sample MSE of the direct method is smaller than that of GCV and REML for all j. For the random design ρ
=
0.5, such tendency can be also seen.Table 5
Results of sample MSE of the selected smoothing parameter via direct method, GCV, and REML forρ
=
0 and ρ
=
0.5. All entries for MSE are 1
0
2 times their actual values.
ρ
=
0
ρ
=
0.5
True
f
1
f
2
f
3
f
1
f
2
f
3
Direct
0.124
0.042
0.082
0.312
0.105
0.428
GCV
1.315
0.485
0.213
0.547
0.352
0.741
REML
2.531
1.237
2.656
1.043
0.846
1.588Table6 shows the sPSE of f
^
(
x
i
)
=
f
^
1
(
x
i
1
)
+
f
^
2
(
x
i
2
)
+
f
^
3
(
x
i
3
) with each smoothing parameter selection method. We see from result that the efficiency of the direct method is guaranteed in the context of prediction accuracy.Table 6
Results of sample PSE of the penalized spline estimator. All entries for PSE are1
0
2 times their actual values.
n
=
200
ρ
=
0
ρ
=
0.5
Direct
0.281
0.429
GCV
0.387
0.467
REML
1.204
0.925Finally we show the computational time (in second) required to construct the penalized spline estimator with each method. The computational times with the direct method, GCV and REML are 11.87 s, 126.43 s, and 43.5 s, respectively. We see that the direct method is more efficient than other methods in context of the computation (see Remark11). All of computations in the simulation were done using a software R and a computer with 3.40 GHz CPU and 24.0 GB memory. Though this is only few examples, the direct method can be seen as one of the good methods to select the smoothing parameter.Remark 11.
We calculate|
sPSE
-
(
0.5
)
2
| as the criteria to confirm the prediction squared error. The ordinal PSE is defined as
(46)
PSE
=
1
n
∑
i
=
1
n
‍
E
[
{
Y
i
-
f
^
(
x
i
)
}
2
]
=
σ
2
+
1
n
∑
i
=
1
n
‍
E
[
{
f
^
(
x
i
)
-
f
(
x
i
)
}
2
]
,
where Y
i’s are the test data. The second term of the PSE is similar to MISE. So it can be said that the sample PSE evaluates the accuracy of the variance part and MISE part. To see the detail of the difference of the sample PSE between the direct method and other method, we calculated |
sPSE
-
σ
2
| in this section.


## 6. Discussion

In this paper, we proposed a new direct method for determining the smoothing parameter for a penalized spline estimator in a regression problem. The direct method is based on minimizing MISE of the penalized spline estimator. We studied the asymptotic property of the direct method. The asymptotic normality of the penalized spline estimator usingλ
^
opt is theoretically guaranteed when the consistent estimator is used as the pilot estimator to obtain λ
^
opt. In numerical study, for the additive model, the computational cost of this direct method is dramatically reduced when compared to grid search methods such as GCV. Furthermore we find that the performance of the direct method is better than or at least similar to that of other methods.The direct method will be developed for other regression models such as varying-coefficient models, Cox proportional hazards models, single-index models, and others if the asymptotic bias and variance of the penalized spline estimator are derived. It is not limited to the mean regression; it can be applied to quantile regression. Actually, Yoshida [31] has presented the asymptotic bias and variance of the penalized spline estimator in univariate quantile regressions. Furthermore, it is seen that improving the direct method is important for various situations and datasets. In particular, the development of the determination of locally adaptive λ is an interesting avenue of further research.



---

*Source: 203469-2014-04-22.xml*