# A Stochastic Restricted Principal Components Regression Estimator in the Linear Model

Authors: Daojiang He; Yan Wu
Journal: The Scientific World Journal
 (2014)

---

## Abstract

We propose a new estimator to combat the multicollinearity in the linear model when there are stochastic linear restrictions on the regression coefficients. The new estimator is constructed by combining the ordinary mixed estimator (OME) and the principal components regression (PCR) estimator, which is called the stochastic restricted principal components (SRPC) regression estimator. Necessary and sufficient conditions for the superiority of the SRPC estimator over the OME and the PCR estimator are derived in the sense of the mean squared error matrix criterion. Finally, we give a numerical example and a Monte Carlo study to illustrate the performance of the proposed estimator.


---

## Body

## 1. Introduction

In linear regression analysis, the presence of multicollinearity among regressor variables may cause highly unstable least squares estimates of the regression parameters. With multicollinear data, some coefficients may be statistically insignificant and may have the wrong signs. To overcome this problem, different remedial methods have been proposed. One estimation technique designed to combat collinearity is using biased estimators, most notable of which are the Stein estimator by Stein [1], the principal components regression (PCR) estimator by Massy [2], the ordinary ridge regression (ORR) estimator by Hoerl and Kennard [3], and the Liu estimator by Liu [4]. Another method to combat multicollinearity is through the collection and use of additional information, which can be exact or stochastic restrictions [5]. When it comes to stochastic linear restrictions, Durbin [6], Theil and Goldberger [7], and Theil [8] proposed the ordinary mixed estimator (OME) by combining the sample model with stochastic restrictions. Some other important references on this subject are Li and Yang [9, 10], Xu and Yang [11], Yang and Cui [12], Yang and Wu [13], Yang and Xu [14], and so on.In this paper, we will introduce a stochastic restricted principal components (SRPC) regression estimator, which is defined by combining in a special way the ordinary mixed estimator and the principal components regression estimator. We will compare the new estimator with the PCR estimator and the OME, respectively, in the sense of the criterion of the mean squared error matrix (MSEM).The rest of the paper is organized as follows. In Section2, the new estimator is introduced. In Section 3, some properties of the new estimator are discussed. A numerical example and a Monte Carlo simulation study are given in Section 4.


## 2. The New Estimator

Let us consider the general linear model(1)
Y
=
X
β
+
ɛ
,
E
(
ɛ
)
=
0
,
Cov
(
ɛ
)
=
σ
2
I
n
,
where Y is an n
×
1 observable random vector with the expectation E
(
Y
)
=
X
β and the covariance matrix Cov
(
Y
)
=
σ
2
I
n, X is an n
×
p known design matrix of rank p, I
n is the identity matrix of order n, β is a p
×
1 vector of unknown parameters, and ɛ is an n
×
1 vector of random errors. As is well known, the ordinary least squares estimator (OLSE) of β is
(2)
β
^
OLSE
=
S
-
1
X
′
Y
,
where S
=
X
′
X.LetT
=
(
t
1
,
t
2
,
…
,
t
p
) be an orthogonal matrix such that T
′
X
′
X
T
=
Λ, where Λ
=
diag
⁡
(
λ
1
,
λ
2
,
…
,
λ
p
) and λ
1
≥
λ
2
≥
⋯
≥
λ
p
>
0 are the eigenvalues of X
′
X. Further, let T
k
=
(
t
1
,
t
2
,
…
,
t
k
) be the remaining columns of T after having deleted the last p
-
k columns, where 0
≤
k
≤
p. Thus, we have
(3)
T
k
′
X
′
X
T
k
=
Λ
k
=
diag
⁡
(
λ
1
,
λ
2
,
…
,
λ
k
)
,
T
p
-
k
′
X
′
X
T
p
-
k
=
Λ
p
-
k
=
diag
⁡
(
λ
k
+
1
,
λ
k
+
2
,
…
,
λ
p
)
,
where T
p
-
k
=
(
t
k
+
1
,
t
k
+
2
,
…
,
t
p
).In addition to model (1), let us give some prior information about β in the form of a set of j independent stochastic linear restrictions as follows:
(4)
r
=
R
β
+
υ
,
E
(
υ
)
=
0
,
Cov
(
υ
)
=
σ
2
W
,
where r is a j
×
1 vector, R is a j
×
p matrix with rank (
R
)
=
j, υ is a j
×
1 vector of disturbances, and W is assumed to be known and positive definite. Furthermore, it is also assumed that the random vector υ is independent of ɛ.For model (1), Massy [2] introduced the PCR estimator as
(5)
β
^
PCR
=
T
k
(
T
k
′
S
T
k
)
-
1
T
k
′
X
′
Y
.
Xu and Yang [11] showed that the PCR estimator could be rewritten as follows:
(6)
β
^
PCR
=
T
k
T
k
′
β
^
OLSE
.For model (1) with the stochastic restrictions (4), the OME is given by
(7)
β
^
OME
=
(
S
+
R
′
W
-
1
R
)
-
1
(
X
′
Y
+
R
′
W
-
1
r
)
.
Özkale [15] showed that the OME could be rewritten as
(8)
β
^
OME
=
β
^
OLSE
+
S
-
1
R
′
(
W
+
R
S
-
1
R
′
)
-
1
(
r
-
R
β
^
OLSE
)
.
Noting that T
k
′
T
=
(
I
k
⋮
0
)
T
′
T
=
(
I
k
⋮
0
), it can be shown that
(9)
β
^
PCR
=
T
k
(
T
k
′
S
T
k
)
-
1
T
k
′
X
′
Y
=
S
-
1
T
k
T
k
′
X
′
Y
.Now, the stochastic restricted principal components (SRPC) regression estimator can be obtained by combing the OME and PCR estimator. Substituting OLSE with PCR estimator in (8), we can get the new estimator as follows:
(10)
β
^
SRPC
=
β
^
PCR
+
S
-
1
R
′
(
W
+
R
S
-
1
R
′
)
-
1
(
r
-
R
β
^
PCR
)
=
S
-
1
T
k
T
k
′
X
′
Y
+
S
-
1
R
′
(
W
+
R
S
-
1
R
′
)
-
1
×
(
r
-
R
S
-
1
T
k
T
k
′
X
′
Y
)
=
(
S
-
1
-
S
-
1
R
′
(
W
+
R
S
-
1
R
′
)
-
1
R
S
-
1
)
×
(
T
k
T
k
′
X
′
Y
+
R
′
W
-
1
r
)
=
(
S
+
R
′
W
-
1
R
)
-
1
(
T
k
T
k
′
X
′
Y
+
R
′
W
-
1
r
)
.Now, we can see thatβ
^
SRPC is a general estimator which includes the PCR estimator and OME as special cases: if R
=
0, then β
^
SRPC
=
β
^
PCR; if k
=
p, then β
^
SRPC
=
β
^
OME.For the sake of convenience, we list some notations and important lemmas needed in the following discussions. For ann
×
n matrix M, M
≥
0 means that M is symmetric and positive semidefinite and M
>
0 means that M is symmetric and positive definite.Note that for any estimatorβ
^ of β, its MSEM is defined as
(11)
MESM
(
β
^
)
=
E
[
(
β
^
-
β
)
(
β
^
-
β
)
′
]
=
Cov
(
β
^
)
+
Bias
(
β
^
)
Bias
(
β
^
)
′
,
where Bias
(
β
^
)
=
E
(
β
^
)
-
β is the bias of β
^.Lemma 1.
LetM
>
0, and α a vector; then M
-
α
α
′
≥
0 if and only if α
′
M
-
1
α
<
1.Proof.
See Farebrother [16].By Lemma1, the following lemma is straightforward.Lemma 2.
Letβ
^
1
=
A
1
Y, β
^
2
=
A
2
Y be two homogeneous linear estimators of β such that D
:
=
(
A
1
A
1
′
-
A
2
A
2
′
)
>
0. Then
(12)
M
S
E
M
(
β
^
1
)
-
M
S
E
M
(
β
^
2
)
=
σ
2
D
+
b
1
b
1
′
-
b
2
b
2
′
>
0
if and only if
(13)
b
2
T
(
σ
2
D
+
b
1
b
1
′
)
-
1
b
2
<
1
,
where b
i
=
B
i
a
s
(
β
^
i
)
=
(
A
i
X
-
I
)
β, i
=
1,2.Lemma 3.
LetM, N be two n
×
n matrices with M
>
0 and N
≥
0; then M
>
N
⇔
λ
max
⁡
(
N
M
-
1
)
<
1.Proof.
See Rao and Toutenburg [5].


## 3. The Superiority of the New Estimator

The bias vector and the covariance matrix of the SRPC estimator are given by(14)
E
(
β
^
SRPC
)
=
(
S
+
R
′
W
-
1
R
)
-
1
(
T
k
T
k
′
S
+
R
′
W
-
1
R
)
β
=
(
S
+
R
′
W
-
1
R
)
-
1
T
k
T
k
′
S
β
+
(
S
+
R
′
W
-
1
R
)
-
1
×
(
S
+
R
′
W
-
1
R
-
S
)
β
=
(
S
+
R
′
W
-
1
R
)
-
1
T
k
T
k
′
S
β
+
[
I
-
(
S
+
R
′
W
-
1
R
)
-
1
S
]
β
=
β
+
(
S
+
R
′
W
-
1
R
)
-
1
(
T
k
T
k
′
-
I
)
S
β
,
(15)
b
1
∶
=
Bias
(
β
^
SRPC
)
=
E
(
β
^
SRPC
)
-
β
=
(
S
+
R
′
W
-
1
R
)
-
1
(
T
k
T
k
′
-
I
)
S
β
,
Cov
(
β
^
SRPC
)
=
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
(
T
k
T
k
′
S
T
k
T
k
′
+
R
′
W
-
1
R
)
×
(
S
+
R
′
W
-
1
R
)
-
1
.
From (15), we can obtain that
(16)
MSEM
(
β
^
SRPC
)
=
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
(
T
k
T
k
′
S
T
k
T
k
′
+
R
′
W
-
1
R
)
×
(
S
+
R
′
W
-
1
R
)
-
1
+
b
1
b
1
′
.Following the above procedure, we can get(17)
MSEM
(
β
^
PCR
)
=
σ
2
T
k
Λ
k
-
1
T
k
′
+
b
2
b
2
′
,
MSEM
(
β
^
OME
)
=
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
,
where b
2
=
(
T
k
T
k
′
-
I
p
)
β.In order to compareβ
^
SRPC with β
^
PCR and β
^
OME in the MSEM sense, now we investigate the following differences:
(18)
Δ
1
=
MSEM
(
β
^
PCR
)
-
MSEM
(
β
^
SRPC
)
=
σ
2
[
T
k
Λ
k
-
1
T
k
′
-
(
S
+
R
′
W
-
1
R
)
-
1
×
(
T
k
T
k
′
S
T
k
T
k
′
+
R
′
W
-
1
R
)
(
S
+
R
′
W
-
1
R
)
-
1
]
+
b
2
b
2
′
-
b
1
b
1
′
∶
=
σ
2
D
1
+
b
2
b
2
′
-
b
1
b
1
′
,
where
(19)
D
1
=
T
k
Λ
k
-
1
T
k
′
-
(
S
+
R
′
W
-
1
R
)
-
1
×
(
T
k
T
k
′
S
T
k
T
k
′
+
R
′
W
-
1
R
)
(
S
+
R
′
W
-
1
R
)
-
1
,
Δ
2
=
MSEM
(
β
^
OME
)
-
MSEM
(
β
^
SRPC
)
=
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
-
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
×
(
T
k
T
k
′
S
T
k
T
k
′
+
R
′
W
-
1
R
)
·
(
S
+
R
′
W
-
1
R
)
-
1
-
b
1
b
1
′
=
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
(
S
-
T
k
T
k
′
S
T
k
T
k
′
)
×
(
S
+
R
′
W
-
1
R
)
-
1
-
b
1
b
1
′
∶
=
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
D
2
(
S
+
R
′
W
-
1
R
)
-
1
-
b
1
b
1
′
,
where
(20)
D
2
=
S
-
T
k
T
k
′
S
T
k
T
k
′
.In the following theorems, we will give the necessary and sufficient conditions for the new estimator to be superior to the PCR estimator and OME in the MSEM sense.Definition 4.
Suppose thatβ
^
1 and β
^
2 are two estimators of β; then β
^
1 is said to be superior to β
^
2 in the MSEM sense if MSEM
(
β
^
2
)
-
MSEM
(
β
^
1
)
>
0.Theorem 5.
Assume thatλ
max
⁡
(
FT
Λ
T
′
)
<
1; then the SRPC estimator β
^
S
R
P
C is superior to the PCR estimator β
^
P
C
R in the MSEM sense if and only if
(21)
b
1
′
(
σ
2
D
1
+
b
2
b
2
′
)
-
1
b
1
<
1
,
where
(22)
F
=
(
S
+
R
′
W
-
1
R
)
-
1
(
T
k
T
k
′
S
T
k
T
k
′
+
R
′
W
-
1
R
)
×
(
S
+
R
′
W
-
1
R
)
-
1
+
T
p
-
k
Λ
p
-
k
-
1
T
p
-
k
′
.Proof.
Noting thatT
Λ
-
1
T
′
=
T
k
Λ
k
-
1
T
k
′
+
T
p
-
k
Λ
p
-
k
-
1
T
p
-
k
′, it follows that
(23)
D
1
=
T
k
Λ
k
-
1
T
k
′
-
(
S
+
R
′
W
-
1
R
)
-
1
×
(
T
k
T
k
′
S
T
k
T
k
′
+
R
′
W
-
1
R
)
(
S
+
R
′
W
-
1
R
)
-
1
=
T
Λ
-
1
T
′
-
F
.
By Lemma 3 and the assumption that λ
max
⁡
(
F
T
Λ
T
′
)
<
1, we have D
1
>
0. Consequently, by Lemma 2, Δ
1
=
σ
2
D
1
+
b
2
b
2
′
-
b
1
b
1
′
>
0 if and only if
(24)
b
1
′
(
σ
2
D
1
+
b
2
b
2
′
)
-
1
b
1
<
1
.
Thus, the proof of Theorem 5 is completed.Theorem 6.
Assume thatλ
max
⁡
(
T
k
T
k
′
S
T
k
T
k
′
S
-
1
)
<
1; then the SRPC estimator β
^
S
R
P
C is superior to the the mixed estimator β
^
O
M
E in the MSEM sense if and only if
(25)
b
1
′
(
(
S
+
R
′
W
-
1
R
)
-
1
D
2
(
S
+
R
′
W
-
1
R
)
-
1
)
-
1
b
1
<
σ
2
.Proof.
By Lemma3, the assumption that λ
max
⁡
(
T
k
T
k
′
S
T
k
T
k
′
S
-
1
)
<
1 implies that D
2
=
S
-
T
k
T
k
′
S
T
k
T
k
′
>
0. It follows that
(26)
(
S
+
R
′
W
-
1
R
)
-
1
D
2
(
S
+
R
′
W
-
1
R
)
-
1
>
0
.
Therefore, Δ
2
=
σ
2
(
S
+
R
′
W
-
1
R
)
-
1
D
2
(
S
+
R
′
W
-
1
R
)
-
1
-
b
1
b
1
′
>
0 if and only if
(27)
b
1
′
(
(
S
+
R
′
W
-
1
R
)
-
1
D
2
(
S
+
R
′
W
-
1
R
)
-
1
)
-
1
b
1
<
σ
2
.


## 4. Numerical Example and Monte Carlo Simulation

In order to illustrate the performance of the proposed estimator, we first consider the real data example which was discussed in Gruber [17], and the data has also been analyzed by Akdeniz and Erol [18], Li and Yang [9], and Chang and Yang, [19] among others. Now we assemble the data as follows:
(28)
X
=
(
1
1.9
2.2
1.9
3.7
1
1.8
2.2
2.0
3.8
1
1.8
2.4
2.1
3.6
1
1.8
2.4
2.2
3.8
1
2.0
2.5
2.3
3.8
1
2.1
2.6
2.4
3.7
1
2.1
2.6
2.6
3.8
1
2.2
2.6
2.6
4.0
1
2.3
2.8
2.8
3.7
1
2.3
2.7
2.8
3.8
)
,
Y
=
(
2.3
2.2
2.2
2.3
2.4
2.5
2.6
2.6
2.7
2.7
)
.In this experiment, we can note from the theorems that the comparison results depend on the unknown parametersβ and σ
2. Consequently, we cannot exclude that our obtained results in the theorems will be held and the results may be changeable. For this, we replace them by their unbiased estimators, that is, the OLS estimators. The results below are all computed by R2.8.0.From the data, we can obtain the following results:(i)
the eigenvalues ofX
′
X: 312.9320, 0.7536, 0.0453, 0.0372, and 0.0019.
(ii)
the OLS estimator ofβ: (
0.6921,0.6258,0.1154,0.2866
,
and
0.0256
)
′,
(iii)
the OLS estimator ofσ
2: σ
^
2
=
0.0016,
(iv)
the condition number ofX
′
X: κ
=
1.6107
e
+
005.Following Chang and Yang [19], we choose the number of the principal components k
=
3, and consider the following stochastic linear restriction:
(29)
r
=
R
β
+
e
,
R
=
(
1,1
,
2
,
-
2
,
-
2
)
,
e
~
N
(
0
,
σ
^
2
)
.The estimated MSE values of PCR, OME, and SRPC are obtained by replacing all unknown parameters by their OLS estimators, respectively. Table1 gives the results.Table 1
Estimated MSE values of the OME, PCR, and SRPC.
β
0
β
1
β
2
β
3
β
4
MSE
β
^
OME
0.4014
0.5695
-
0.1543
0.3402
0.1252
0.8202
β
^
PCR
-
0.0182
0.5359
-
0.0344
0.2838
0.2104
0.0461
β
^
SRPC
0.0885
0.5566
-
0.0201
0.2641
0.1739
0.0398From Table1, we can observe that the estimated MSE value of the new estimator is smaller than those of PCR and OME, which is in accordance with the theoretical findings in Theorems 5 and 6.To further identify the MSE performance of the new estimator, we are to perform a Monte Carlo simulation study. Specifically, the explanatory variables and the observations are generated by(30)
x
i
j
=
(
1
-
γ
2
)
1
/
2
ω
i
j
+
γ
ω
i
,
p
,
i
=
1,2
,
…
,
n
,
j
=
1,2
,
…
,
p
-
1
,
y
i
=
β
1
x
i
1
+
β
2
x
i
2
+
β
3
x
i
3
+
β
4
x
i
4
+
β
5
x
i
5
+
e
i
,
ɛ
i
~
N
(
0
,
σ
2
)
,
i
=
1,2
,
…
,
n
,
where ω
i
j are independent standard normal pseudorandom numbers, ɛ
i are independent normal pseudorandom numbers with mean zero and variance σ
2, and γ is specified so that the correlation between any two explanatory variables is given by γ
2. In addition, a stochastic linear constraint to the model is considered:
(31)
r
=
R
β
+
e
,
R
=
(
1,3
,
-
2
,
-
2,1
)
,
e
~
N
(
0
,
W
)
.In the simulation, we chooseσ
2
=
1, p
=
6, n
=
50, 100, and W
=
2,   5. Four different sets of correlations, namely, γ
=
0.9, γ
=
0.99, γ
=
0.999, and γ
=
0.9999, are considered to show the weak, strong, and severe collinearity between the explanatory variables following Liu [20]. We choose the normalized eigenvector corresponding to the largest eigenvalue of X
′
X as the true value of β following Chang and Yang [19]. The experiment is replicated 10000 times by generating new error terms. Then, the estimated MSE for an estimator β
~ is calculated as follows:
(32)
MSE
(
β
~
)
=
1
N
∑
m
=
1
N
(
β
~
(
m
)
-
β
)
′
(
β
~
(
m
)
-
β
)
,
where β
~
(
m
) is the estimator of β in the mth replication of the experiment and N
=
10000. The simulation results are summarized in Tables 2 and 3, where the condition number of X
′
X, that is, κ
=
λ
1
/
λ
5, is also given.Table 2
Estimated MSE withn
=
100.
γ
=
0.9
γ
=
0.99
γ
=
0.999
γ
=
0.9999
W
=
2
W
=
5
W
=
2
W
=
5
W
=
2
W
=
5
W
=
2
W
=
5
β
^
OME
0.1419
0.3463
2.6338
5.0907
11.884
12.911
129.02
220.67
β
^
PCR
0.0948
0.0936
0.8798
0.8825
8.7618
8.7304
87.503
85.314
β
^
SRPC
0.0994
0.1190
0.7177
1.1484
5.8144
6.7482
55.871
55.936
κ
21.4
123.8
1230.4
12326.7Table 3
Estimated MSE withn
=
50.
γ
=
0.9
γ
=
0.99
γ
=
0.999
γ
=
0.9999
W
=
2
W
=
5
W
=
2
W
=
5
W
=
2
W
=
5
W
=
2
W
=
5
β
^
OME
0.3155
0.3250
3.3823
4.0796
25.185
46.592
347.39
348.43
β
^
PCR
0.1629
0.1610
1.5253
1.4910
14.818
14.815
146.88
149.92
β
^
SRPC
0.1470
0.2053
0.8802
1.4796
7.0234
8.1218
69.063
69.919
κ
21.6
275.3
1946.0
19348.4From the simulation results shown in Tables2 and 3, we can see that, with the increase of the level of multicollinearity, the estimated MSE values of the three estimators increase in general. However, the proposed estimator SRPC behaves better than the competing estimators in most of the cases. In addition, the more severe the collinearity is, the more pronounced the superiority of SRPC is. Therefore, the proposed estimator is recommended when the explanatory variables are moderately or severely collinear.



---

*Source: 231506-2014-01-23.xml*