# Performance of Some Stochastic Restricted Ridge Estimator in Linear Regression Model

Authors: Jibo Wu; Chaolin Liu
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

This paper considers several estimators for estimating the stochastic restricted ridge regression estimators. A simulation study has been conducted to compare the performance of the estimators. The result from the simulation study shows that stochastic restricted ridge regression estimators outperform mixed estimator. A numerical example has been also given to illustrate the performance of the estimators.


---

## Body

## 1. Introduction

In regression analysis, researchers often encounter the problem of multicollinearity. Multicollinearity leads to high variance and instable parameter estimates when estimating linear regression models using ordinary least squares (OLS). So many researchers propose many ways to overcome this problem.One method to overcome multicollinearity is to consider biased estimator, such as principal component regression estimator [1], ridge estimator [2], and Liu estimator (Liu, 1993). Ridge estimator is used by many researchers. When we use ridge estimator, how to choose the parameter k is very important. A lot of ways of estimating the ridge parameter k have been proposed. To mention a few, Hoerl and Kennard [2], Hoerl and Kennard [3], Hoerl et al. [4], McDonald and Galarneau [5], Kibria [6], Kibria et al. [7], and Najarian et al. [8].Another method to overcome multicollinearity is to consider the restrictions. In the literature, the authors usually discuss two restrictions; one is linear restriction and the other is stochastic linear restriction. For the linear model with linear restriction, Sarkar [9], Kaciranlar et al. [10], and Xu and Yang [11–13] combined the restricted least squares estimator with the ridge estimator, the Liu estimator, almost unbiased ridge and Liu estimator, and r
-
k and r
-
d class estimator and proposed the restricted ridge estimator, restricted Liu estimator, restricted almost unbiased ridge and Liu estimator, and restricted r
-
k and r
-
d class estimator. For the linear model with stochastic linear restriction, Özkale [14] and Yang and Xu [15] proposed the stochastic restricted ridge estimator and the stochastic restricted Liu estimator.The main objective of this paper is to propose some stochastic restricted ridge estimators. These estimators are introduced since they are expected to reduce the mean squared error (MSE) of the mixed method. In order to judge the performance of the estimators, the MSE is calculated. In this paper, we give a Monte simulation study to show the performance of the stochastic restricted ridge estimators. We will also analyze some important properties of the different estimators of the ridge parameterk.The paper is organized as follows. In Section2, we describe the statistical methodology. The simulation results are discussed in Section 3 and a numerical example is given in Section 4. In Section 5 we give a brief summary and conclusions.


## 2. Methodology

In this section we introduce some stochastic restricted ridge estimators for estimating the ridge parameterk based on the work of Hoerl and Kennard [2], Hoerl et al. [4], Schaeffer et al. [16], Kibria [6], and Kibria et al. [7].


### 2.1. Stochastic Restricted Ridge Estimator

Consider the following linear model:(1)
y
=
X
β
+
ɛ
,
where y is an n
×
1 vector of observation, X is an n
×
p design matrix of rank p,  β is a p
×
1 vector denoting unknown coefficients, and ε is an n
×
1 random error vector with E
(
ɛ
)
=
0 and Cov
(
ɛ
)
=
σ
2
I.Suppose thatβ satisfies the following stochastic restriction; that is,
(2)
r
=
R
β
+
e
,
where R is a j
×
p nonzero matrix with rank
⁡
(
R
)
=
j and r is a known vector, and E
(
e
)
=
0 and Cov
(
e
)
=
σ
2
W. In this paper, we assumed that ɛ is independent of e.Using the mixed approach, Durbin [17], Theil and Goldberger [18], and Theil [19] introduced the mixed estimator (ME), which is defined as follows:
(3)
β
^
=
(
X
′
X
+
R
′
W
-
1
R
)
-
1
(
X
′
y
+
R
′
W
-
1
r
)
.
The mixed estimator is an unbiased estimator. However, when multicollinearity exists, the mixed estimator is no longer a good estimator.Özkale [14] proposed the following stochastic restricted ridge estimator (SRRE):
(4)
β
^
(
k
)
=
(
X
′
X
+
R
′
W
-
1
R
+
k
I
)
-
1
(
X
′
y
+
R
′
W
-
1
r
)
,
k
>
0
.For any particular estimatorβ
^ of β, the total MSE is defined as
(5)
MSE
(
β
^
)
=
E
(
β
^
-
β
)
′
(
β
^
-
β
)
.
Thus we have
(6)
MSE
(
β
^
(
k
)
)
=
σ
2
tr
⁡
{
M
k
-
1
M
0
M
k
-
1
}
+
k
2
tr
⁡
{
M
k
-
1
β
β
′
M
k
-
1
}
,
where M
k
=
X
′
X
+
R
′
W
-
1
R
+
k
I.


### 2.2. Estimatingk

The most classical ridge estimator for linear regression is the following:(7)
K
1
=
σ
^
2
α
^
max
⁡
2
,
proposed by Hoerl and Kennard [2, 3], where α
^
max
⁡
2 denote the maximum element of Q
β
^
OLS, Q
′
X
′
X
Q
=
diag
⁡
(
λ
1
,
…
,
λ
p
), β
^
OLS
=
(
X
′
X
)
-
1
X
′
y, and σ
^
2 is the estimator of σ
2. Hoerl et al. [4] introduce an alternative of the estimator of k, which is defined as follows:
(8)
K
2
=
p
σ
^
2
β
^
OLS
′
β
^
OLS
.In Schaeffer et al. [16] a modified version of this estimator is proposed as follows:
(9)
K
3
=
1
α
^
max
⁡
2
.In Kibria et al. [7], a new estimator is proposed as follows:
(10)
K
4
=
median
(
α
^
i
2
σ
^
2
)
.
Finally the following estimators are considered:
(11)
K
5
=
1
p
σ
^
2
(
∏
i
=
1
p
α
^
i
2
)
1
/
p
,
K
6
=
σ
^
2
∑
i
=
1
p
λ
i
α
^
i
2
.


## 2.1. Stochastic Restricted Ridge Estimator

Consider the following linear model:(1)
y
=
X
β
+
ɛ
,
where y is an n
×
1 vector of observation, X is an n
×
p design matrix of rank p,  β is a p
×
1 vector denoting unknown coefficients, and ε is an n
×
1 random error vector with E
(
ɛ
)
=
0 and Cov
(
ɛ
)
=
σ
2
I.Suppose thatβ satisfies the following stochastic restriction; that is,
(2)
r
=
R
β
+
e
,
where R is a j
×
p nonzero matrix with rank
⁡
(
R
)
=
j and r is a known vector, and E
(
e
)
=
0 and Cov
(
e
)
=
σ
2
W. In this paper, we assumed that ɛ is independent of e.Using the mixed approach, Durbin [17], Theil and Goldberger [18], and Theil [19] introduced the mixed estimator (ME), which is defined as follows:
(3)
β
^
=
(
X
′
X
+
R
′
W
-
1
R
)
-
1
(
X
′
y
+
R
′
W
-
1
r
)
.
The mixed estimator is an unbiased estimator. However, when multicollinearity exists, the mixed estimator is no longer a good estimator.Özkale [14] proposed the following stochastic restricted ridge estimator (SRRE):
(4)
β
^
(
k
)
=
(
X
′
X
+
R
′
W
-
1
R
+
k
I
)
-
1
(
X
′
y
+
R
′
W
-
1
r
)
,
k
>
0
.For any particular estimatorβ
^ of β, the total MSE is defined as
(5)
MSE
(
β
^
)
=
E
(
β
^
-
β
)
′
(
β
^
-
β
)
.
Thus we have
(6)
MSE
(
β
^
(
k
)
)
=
σ
2
tr
⁡
{
M
k
-
1
M
0
M
k
-
1
}
+
k
2
tr
⁡
{
M
k
-
1
β
β
′
M
k
-
1
}
,
where M
k
=
X
′
X
+
R
′
W
-
1
R
+
k
I.


## 2.2. Estimatingk

The most classical ridge estimator for linear regression is the following:(7)
K
1
=
σ
^
2
α
^
max
⁡
2
,
proposed by Hoerl and Kennard [2, 3], where α
^
max
⁡
2 denote the maximum element of Q
β
^
OLS, Q
′
X
′
X
Q
=
diag
⁡
(
λ
1
,
…
,
λ
p
), β
^
OLS
=
(
X
′
X
)
-
1
X
′
y, and σ
^
2 is the estimator of σ
2. Hoerl et al. [4] introduce an alternative of the estimator of k, which is defined as follows:
(8)
K
2
=
p
σ
^
2
β
^
OLS
′
β
^
OLS
.In Schaeffer et al. [16] a modified version of this estimator is proposed as follows:
(9)
K
3
=
1
α
^
max
⁡
2
.In Kibria et al. [7], a new estimator is proposed as follows:
(10)
K
4
=
median
(
α
^
i
2
σ
^
2
)
.
Finally the following estimators are considered:
(11)
K
5
=
1
p
σ
^
2
(
∏
i
=
1
p
α
^
i
2
)
1
/
p
,
K
6
=
σ
^
2
∑
i
=
1
p
λ
i
α
^
i
2
.


## 3. The Monte Carlo Simulation

The main purpose of this paper is to see the effect of multicollinearity on mixed estimator and SRRE.


### 3.1. Simulation Technique

To achieve different degrees of collinearity, following McDonald and Galarneau [5] and Liu [20], the explanatory variables were generated using the following method:
(12)
x
i
j
=
(
1
-
γ
2
)
1
/
2
z
i
j
+
γ
z
i
p
,
i
=
1
,
…
,
n
,
j
=
1
,
…
,
p
,
where z
i
j are independent standard normal pseudorandom numbers and γ is specified so that the correlation between any two explanatory variables is given by γ
2. These variables are then standardized so that X
′
X and X
′
y are in correlation forms. Four different sets of correlations corresponding to γ
=
0.75,0.85,0.95,0.99 are discussed. Then the dependent variable are determined by
(13)
y
i
=
β
1
x
i
1
+
⋯
+
β
p
x
i
p
+
ε
i
,
where ε
i are independent normal (
0
,
σ
2
) pseudorandom numbers. In this paper, we consider σ
2
=
0.5,1
,
2,5
,
10, n
=
50,100,150, and p
=
4,6
,
8. Suppose that
(14)
r
=
R
β
+
e
,
e
~
N
(
0
,
σ
2
I
2
)
,
where
(15)
R
=
(
1
1
3
1
0
1
1
1
)
,
R
=
(
1
1
0
3
1
0
0
1
1
1
1
1
)
,
R
=
(
1
1
0
1
3
1
0
0
0
1
1
1
1
1
1
1
)
.


### 3.2. Results and Discussion

Tables1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, and 20 present the results of our Monte Carlo experiment concerning the MSEs of the different proposed stochastic restricted ridge estimators and the mixed estimator.Table 1
Estimated MSE withn
=
50, p
=
4, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0159
0.0157
0.0156
0.0157
0.0157
0.0159
0.0159
σ
2
=
1
0.0637
0.0605
0.0594
0.0608
0.0629
0.0636
0.0637
σ
2
=
2
0.2549
0.2099
0.1989
0.2412
0.2532
0.2543
0.2545
σ
2
=
5
1.5934
0.6732
0.0604
1.5035
1.5891
1.5605
1.5771
σ
2
=
10
6.3734
0.9396
0.8603
6.0119
6.3649
5.8729
6.1203Table 2
Estimated MSE withn
=
50, p
=
4, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0167
0.0165
0.0164
0.0165
0.0164
0.0167
0.0167
σ
2
=
1
0.0669
0.0633
0.0621
0.0637
0.0659
0.0668
0.0668
σ
2
=
2
0.2675
0.2183
0.2066
0.2524
0.2656
0.2666
0.2670
σ
2
=
5
1.6720
0.6863
0.6157
1.5732
1.6673
1.6358
1.6352
σ
2
=
10
6.6879
0.9442
0.8661
6.2905
6.6785
6.1384
6.3959Table 3
Estimated MSE withn
=
50, p
=
4, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0198
0.0195
0.0193
0.0193
0.0194
0.0198
0.0198
σ
2
=
1
0.0791
0.0753
0.0725
0.0757
0.0782
0.0790
0.0790
σ
2
=
2
0.3164
0.2625
0.2349
0.3013
0.3146
0.3155
0.3157
σ
2
=
5
1.9774
0.8369
0.6546
1.8802
1.9730
1.9421
1.9507
σ
2
=
10
7.9098
1.0976
0.8844
7.5192
7.9009
7.3696
7.4955Table 4
Estimated MSE withn
=
50, p
=
4, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0244
0.0240
0.0237
0.0238
0.0239
0.0244
0.0244
σ
2
=
1
0.0977
0.0919
0.0879
0.0926
0.0964
0.0976
0.0976
σ
2
=
2
0.3907
0.3114
0.2743
0.3678
0.3880
0.3893
0.3893
σ
2
=
5
2.4416
0.9010
70.7006
2.2944
2.4349
2.3879
2.3910
σ
2
=
10
9.7665
1.1034
0.9042
9.1752
9.7530
8.9516
8.9960Table 5
Estimated MSE withn
=
100, p
=
4, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0129
0.0128
0.0127
0.0127
0.0128
0.0129
0.0129
σ
2
=
1
0.0516
0.0499
0.0482
0.0508
0.0514
0.0516
0.0516
σ
2
=
2
0.2063
0.1821
0.1619
0.2030
0.2060
0.2061
0.2062
σ
2
=
5
1.2893
0.6857
0.5110
1.2683
1.2886
1.2804
1.2845
σ
2
=
10
5.1572
1.0328
0.7930
5.0731
5.1557
5.0179
5.0806Table 6
Estimated MSE withn
=
100, p
=
4, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0135
0.0134
0.0133
0.0134
0.0135
0.0135
0.0135
σ
2
=
1
0.0541
0.0523
0.0504
0.0533
0.0540
0.0541
0.0541
σ
2
=
2
0.2166
0.1901
0.1683
0.2129
0.2163
0.2163
0.2164
σ
2
=
5
1.3537
0.7025
0.5222
1.3305
1.3529
1.3439
1.3481
σ
2
=
10
5.4147
1.0382
0.8007
5.3220
5.4131
5.2613
5.3261Table 7
Estimated MSE withn
=
100, p
=
4, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0160
0.0159
0.0157
0.0158
0.0159
0.0160
0.0160
σ
2
=
1
0.0642
0.0617
0.0590
0.0629
0.0639
0.0641
0.0641
σ
2
=
2
0.2567
0.2202
0.1919
0.2516
0.2562
0.2563
0.2564
σ
2
=
5
1.6043
0.7599
0.5612
1.5719
1.6032
1.5906
1.5950
σ
2
=
10
6.4172
1.0533
08258
6.2874
6.4150
6.2028
6.2700Table 8
Estimated MSE withn
=
100, p
=
4, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0199
0.0197
0.0193
0.0194
0.0197
0.0199
0.0199
σ
2
=
1
0.0795
0.0749
0.0717
0.0771
0.0791
0.0794
0.0794
σ
2
=
2
0.3179
0.2544
0.2248
0.3081
0.3171
0.3168
0.3175
σ
2
=
5
1.9866
0.7464
0.6094
1.9247
1.9847
1.9470
1.9713
σ
2
=
10
7.9465
0.9623
0.8539
7.6984
7.9427
7.3423
7.7064Table 9
Estimated MSE withn
=
150, p
=
4, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0032
0.0032
0.0032
0.0032
0.0032
0.0032
0.0032
σ
2
=
1
0.0128
0.0127
0.0126
0.0127
0.0128
0.0128
0.0128
σ
2
=
2
0.0512
0.0499
0.0484
0.0508
0.0512
0.0512
0.0512
σ
2
=
5
0.3200
0.2740
0.2355
0.3174
0.3199
0.3192
0.3199
σ
2
=
10
1.2802
0.7474
0.5404
1.2697
1.2800
1.2604
1.2774Table 10
Estimated MSE withn
=
150, p
=
4, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0034
0.0034
0.0033
0.0034
0.0033
0.0034
0.0034
σ
2
=
1
0.0134
0.0133
0.0132
0.0133
0.0134
0.0134
0.0134
σ
2
=
2
0.0538
0.0517
0.0506
0.0530
0.0537
0.0537
0.0538
σ
2
=
5
0.3361
0.2686
0.2442
0.3317
0.3359
0.3340
0.3359
σ
2
=
10
1.3445
0.6541
0.5522
1.3267
1.3440
1.3113
1.3408Table 11
Estimated MSE withn
=
150, p
=
4, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0040
0.0040
0.0040
0.0040
0.0040
0.0040
0.0040
σ
2
=
1
0.0160
0.0158
0.0157
0.0158
0.0159
0.0159
0.0160
σ
2
=
2
0.0638
0.0618
0.0594
0.0632
0.0637
0.0637
0.0637
σ
2
=
5
0.3988
0.3294
0.2761
0.3947
0.3956
0.3974
0.3984
σ
2
=
10
1.5951
0.8382
0.5927
1.5788
1.5947
1.5737
1.5897Table 12
Estimated MSE withn
=
150, p
=
4, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
σ
2
=
1
0.0198
0.0195
0.0193
0.0194
0.0197
0.0198
0.0198
σ
2
=
2
0.0791
0.0744
0.0725
0.0775
0.0789
0.0790
0.0790
σ
2
=
5
0.4946
0.3508
0.3197
0.4837
0.4941
0.4927
0.4938
σ
2
=
10
1.9782
0.7271
0.6421
1.9345
1.9773
1.9486
1.9660Table 13
Estimated MSE withn
=
50, p
=
6, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0181
0.0178
0.0176
0.0175
0.0178
0.0180
0.0180
σ
2
=
1
0.0722
0.0684
0.0651
0.0682
0.0715
0.0721
0.0721
σ
2
=
2
0.2889
0.2358
0.2053
0.2710
0.2875
0.2885
0.2885
σ
2
=
5
1.8058
0.7359
0.5831
1.6902
1.8022
1.7895
1.7886
σ
2
=
10
7.2233
0.9977
0.8417
6.7591
7.2160
6.9678
6.9535Table 14
Estimated MSE withn
=
50, p
=
6, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0189
0.0188
0.0184
0.0184
0.0186
0.0188
0.0188
σ
2
=
1
0.0758
0.0716
0.0680
0.0714
0.0750
0.0757
0.0757
σ
2
=
2
0.3031
0.2452
0.2127
0.2834
0.3015
0.3026
0.3026
σ
2
=
5
1.8944
0.7497
0.5943
1.7676
1.8904
1.8764
1.8764
σ
2
=
10
7.5775
1.0016
0.8479
7.0682
7.5694
7.2969
7.2666Table 15
Estimated MSE withn
=
50, p
=
6, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0224
0.0221
0.0216
0.0216
0.0220
0.0223
0.0223
σ
2
=
1
0.0895
0.0850
0.0790
0.0847
0.0886
0.0894
0.0894
σ
2
=
2
0.3581
0.2944
0.2400
0.3373
0.3562
0.3573
0.3574
σ
2
=
5
2.2380
0.9162
0.6324
2.1053
2.2333
2.2068
2.2102
σ
2
=
10
8.9518
1.1747
0.8681
8.4195
8.9425
8.4697
8.5206Table 16
Estimated MSE withn
=
50, p
=
6, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0276
0.0270
0.0265
0.0264
0.0269
0.0275
0.0275
σ
2
=
1
0.1104
0.1017
0.0950
0.1013
0.1087
0.1103
0.1103
σ
2
=
2
0.4415
0.3288
0.2775
0.4009
0.4381
0.4404
0.4400
σ
2
=
5
2.7591
0.8493
0.6783
2.4979
2.7506
2.7213
2.6973
σ
2
=
10
11.0363
1.0223
0.8903
9.9873
11.0193
10.4518
10.1007Table 17
Estimated MSE withn
=
50, p
=
8, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0694
0.0648
0.0621
0.0631
0.0685
0.0692
0.0692
σ
2
=
1
0.2778
0.2162
0.1940
0.2469
0.2759
0.2776
0.2775
σ
2
=
2
1.1112
0.5257
0.4531
0.9819
1.1074
1.1079
1.1065
σ
2
=
5
6.9449
0.8932
0.8221
6.1268
6.9354
6.8179
6.7664
σ
2
=
10
27.7795
0.9781
0.9476
24.5017
2.77606
25.8426
25.1061Table 18
Estimated MSE withn
=
50, p
=
8, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0729
0.0678
0.0649
0.0660
0.0719
0.0728
0.0728
σ
2
=
1
0.2915
0.2244
0.2010
0.2576
0.2894
0.2912
0.2911
σ
2
=
2
1.1658
0.5382
0.4642
1.0243
1.1617
1.1622
1.1605
σ
2
=
5
7.2865
0.8989
0.8291
6.3908
7.2761
7.1469
7.0805
σ
2
=
10
29.1462
0.9794
0.9500
25.5570
29.1253
27.0208
26.0808Table 19
Estimated MSE withn
=
50, p
=
8, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0861
0.0830
0.0754
0.0815
0.0855
0.0869
0.0860
σ
2
=
1
0.3446
0.2994
0.2267
0.3248
0.3432
0.3445
0.3442
σ
2
=
2
1.3782
0.8508
0.5033
1.2979
1.3756
1.3768
1.3725
σ
2
=
5
8.6138
1.5585
0.8517
8.1096
8.6072
8.5589
8.3965
σ
2
=
10
34.4552
1.3911
0.9575
32.4371
34.4419
33.5908
31.1970Table 20
Estimated MSE withn
=
50, p
=
8, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.1063
0.1009
0.0906
0.0984
0.1048
0.1062
0.1062
σ
2
=
1
0.4252
0.3494
0.2622
0.3908
0.4222
0.4251
0.4243
σ
2
=
2
1.7009
0.8972
0.5531
1.5603
1.6949
1.6988
1.6857
σ
2
=
5
10.6308
1.3896
0.8768
9.7471
10.6157
10.5469
10.0593
σ
2
=
10
42.5233
1.2263
0.9656
38.9855
42.4929
41.2088
34.5263


#### 3.2.1. The Performance as a Function ofn

Increasing the number of observations has a negative effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are decreasing, although SRRE is always better than the ME. However, the performance of the SSRE over the ME is not very obvious. Thus, both measurements of performances show that the advantage of SRRE decreases with the increase of the number of observations.


#### 3.2.2. The Performance as a Function ofp

Increasing the independent variables has a positive effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are increasing. When increasing the independent variables, the SRRE is better than the ME, and the performance of the SRRE over the ME is very obvious. Thus, both measurements of performances show that the advantage of SRRE increases with the increase of the number of observations.


#### 3.2.3. The Performance as a Function ofγ

Increasing the degree of correlation has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase asγ increases. Thus, the gain of applying SRRE increases as the degree of correlation increases.


#### 3.2.4. The Performance as a Function ofσ
2

Increasingσ
2 has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase as σ
2 increases. When σ
2 is big, the performance of the SRRE over the ME is very obvious.


## 3.1. Simulation Technique

To achieve different degrees of collinearity, following McDonald and Galarneau [5] and Liu [20], the explanatory variables were generated using the following method:
(12)
x
i
j
=
(
1
-
γ
2
)
1
/
2
z
i
j
+
γ
z
i
p
,
i
=
1
,
…
,
n
,
j
=
1
,
…
,
p
,
where z
i
j are independent standard normal pseudorandom numbers and γ is specified so that the correlation between any two explanatory variables is given by γ
2. These variables are then standardized so that X
′
X and X
′
y are in correlation forms. Four different sets of correlations corresponding to γ
=
0.75,0.85,0.95,0.99 are discussed. Then the dependent variable are determined by
(13)
y
i
=
β
1
x
i
1
+
⋯
+
β
p
x
i
p
+
ε
i
,
where ε
i are independent normal (
0
,
σ
2
) pseudorandom numbers. In this paper, we consider σ
2
=
0.5,1
,
2,5
,
10, n
=
50,100,150, and p
=
4,6
,
8. Suppose that
(14)
r
=
R
β
+
e
,
e
~
N
(
0
,
σ
2
I
2
)
,
where
(15)
R
=
(
1
1
3
1
0
1
1
1
)
,
R
=
(
1
1
0
3
1
0
0
1
1
1
1
1
)
,
R
=
(
1
1
0
1
3
1
0
0
0
1
1
1
1
1
1
1
)
.


## 3.2. Results and Discussion

Tables1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, and 20 present the results of our Monte Carlo experiment concerning the MSEs of the different proposed stochastic restricted ridge estimators and the mixed estimator.Table 1
Estimated MSE withn
=
50, p
=
4, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0159
0.0157
0.0156
0.0157
0.0157
0.0159
0.0159
σ
2
=
1
0.0637
0.0605
0.0594
0.0608
0.0629
0.0636
0.0637
σ
2
=
2
0.2549
0.2099
0.1989
0.2412
0.2532
0.2543
0.2545
σ
2
=
5
1.5934
0.6732
0.0604
1.5035
1.5891
1.5605
1.5771
σ
2
=
10
6.3734
0.9396
0.8603
6.0119
6.3649
5.8729
6.1203Table 2
Estimated MSE withn
=
50, p
=
4, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0167
0.0165
0.0164
0.0165
0.0164
0.0167
0.0167
σ
2
=
1
0.0669
0.0633
0.0621
0.0637
0.0659
0.0668
0.0668
σ
2
=
2
0.2675
0.2183
0.2066
0.2524
0.2656
0.2666
0.2670
σ
2
=
5
1.6720
0.6863
0.6157
1.5732
1.6673
1.6358
1.6352
σ
2
=
10
6.6879
0.9442
0.8661
6.2905
6.6785
6.1384
6.3959Table 3
Estimated MSE withn
=
50, p
=
4, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0198
0.0195
0.0193
0.0193
0.0194
0.0198
0.0198
σ
2
=
1
0.0791
0.0753
0.0725
0.0757
0.0782
0.0790
0.0790
σ
2
=
2
0.3164
0.2625
0.2349
0.3013
0.3146
0.3155
0.3157
σ
2
=
5
1.9774
0.8369
0.6546
1.8802
1.9730
1.9421
1.9507
σ
2
=
10
7.9098
1.0976
0.8844
7.5192
7.9009
7.3696
7.4955Table 4
Estimated MSE withn
=
50, p
=
4, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0244
0.0240
0.0237
0.0238
0.0239
0.0244
0.0244
σ
2
=
1
0.0977
0.0919
0.0879
0.0926
0.0964
0.0976
0.0976
σ
2
=
2
0.3907
0.3114
0.2743
0.3678
0.3880
0.3893
0.3893
σ
2
=
5
2.4416
0.9010
70.7006
2.2944
2.4349
2.3879
2.3910
σ
2
=
10
9.7665
1.1034
0.9042
9.1752
9.7530
8.9516
8.9960Table 5
Estimated MSE withn
=
100, p
=
4, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0129
0.0128
0.0127
0.0127
0.0128
0.0129
0.0129
σ
2
=
1
0.0516
0.0499
0.0482
0.0508
0.0514
0.0516
0.0516
σ
2
=
2
0.2063
0.1821
0.1619
0.2030
0.2060
0.2061
0.2062
σ
2
=
5
1.2893
0.6857
0.5110
1.2683
1.2886
1.2804
1.2845
σ
2
=
10
5.1572
1.0328
0.7930
5.0731
5.1557
5.0179
5.0806Table 6
Estimated MSE withn
=
100, p
=
4, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0135
0.0134
0.0133
0.0134
0.0135
0.0135
0.0135
σ
2
=
1
0.0541
0.0523
0.0504
0.0533
0.0540
0.0541
0.0541
σ
2
=
2
0.2166
0.1901
0.1683
0.2129
0.2163
0.2163
0.2164
σ
2
=
5
1.3537
0.7025
0.5222
1.3305
1.3529
1.3439
1.3481
σ
2
=
10
5.4147
1.0382
0.8007
5.3220
5.4131
5.2613
5.3261Table 7
Estimated MSE withn
=
100, p
=
4, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0160
0.0159
0.0157
0.0158
0.0159
0.0160
0.0160
σ
2
=
1
0.0642
0.0617
0.0590
0.0629
0.0639
0.0641
0.0641
σ
2
=
2
0.2567
0.2202
0.1919
0.2516
0.2562
0.2563
0.2564
σ
2
=
5
1.6043
0.7599
0.5612
1.5719
1.6032
1.5906
1.5950
σ
2
=
10
6.4172
1.0533
08258
6.2874
6.4150
6.2028
6.2700Table 8
Estimated MSE withn
=
100, p
=
4, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0199
0.0197
0.0193
0.0194
0.0197
0.0199
0.0199
σ
2
=
1
0.0795
0.0749
0.0717
0.0771
0.0791
0.0794
0.0794
σ
2
=
2
0.3179
0.2544
0.2248
0.3081
0.3171
0.3168
0.3175
σ
2
=
5
1.9866
0.7464
0.6094
1.9247
1.9847
1.9470
1.9713
σ
2
=
10
7.9465
0.9623
0.8539
7.6984
7.9427
7.3423
7.7064Table 9
Estimated MSE withn
=
150, p
=
4, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0032
0.0032
0.0032
0.0032
0.0032
0.0032
0.0032
σ
2
=
1
0.0128
0.0127
0.0126
0.0127
0.0128
0.0128
0.0128
σ
2
=
2
0.0512
0.0499
0.0484
0.0508
0.0512
0.0512
0.0512
σ
2
=
5
0.3200
0.2740
0.2355
0.3174
0.3199
0.3192
0.3199
σ
2
=
10
1.2802
0.7474
0.5404
1.2697
1.2800
1.2604
1.2774Table 10
Estimated MSE withn
=
150, p
=
4, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0034
0.0034
0.0033
0.0034
0.0033
0.0034
0.0034
σ
2
=
1
0.0134
0.0133
0.0132
0.0133
0.0134
0.0134
0.0134
σ
2
=
2
0.0538
0.0517
0.0506
0.0530
0.0537
0.0537
0.0538
σ
2
=
5
0.3361
0.2686
0.2442
0.3317
0.3359
0.3340
0.3359
σ
2
=
10
1.3445
0.6541
0.5522
1.3267
1.3440
1.3113
1.3408Table 11
Estimated MSE withn
=
150, p
=
4, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0040
0.0040
0.0040
0.0040
0.0040
0.0040
0.0040
σ
2
=
1
0.0160
0.0158
0.0157
0.0158
0.0159
0.0159
0.0160
σ
2
=
2
0.0638
0.0618
0.0594
0.0632
0.0637
0.0637
0.0637
σ
2
=
5
0.3988
0.3294
0.2761
0.3947
0.3956
0.3974
0.3984
σ
2
=
10
1.5951
0.8382
0.5927
1.5788
1.5947
1.5737
1.5897Table 12
Estimated MSE withn
=
150, p
=
4, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
0.0049
σ
2
=
1
0.0198
0.0195
0.0193
0.0194
0.0197
0.0198
0.0198
σ
2
=
2
0.0791
0.0744
0.0725
0.0775
0.0789
0.0790
0.0790
σ
2
=
5
0.4946
0.3508
0.3197
0.4837
0.4941
0.4927
0.4938
σ
2
=
10
1.9782
0.7271
0.6421
1.9345
1.9773
1.9486
1.9660Table 13
Estimated MSE withn
=
50, p
=
6, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0181
0.0178
0.0176
0.0175
0.0178
0.0180
0.0180
σ
2
=
1
0.0722
0.0684
0.0651
0.0682
0.0715
0.0721
0.0721
σ
2
=
2
0.2889
0.2358
0.2053
0.2710
0.2875
0.2885
0.2885
σ
2
=
5
1.8058
0.7359
0.5831
1.6902
1.8022
1.7895
1.7886
σ
2
=
10
7.2233
0.9977
0.8417
6.7591
7.2160
6.9678
6.9535Table 14
Estimated MSE withn
=
50, p
=
6, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0189
0.0188
0.0184
0.0184
0.0186
0.0188
0.0188
σ
2
=
1
0.0758
0.0716
0.0680
0.0714
0.0750
0.0757
0.0757
σ
2
=
2
0.3031
0.2452
0.2127
0.2834
0.3015
0.3026
0.3026
σ
2
=
5
1.8944
0.7497
0.5943
1.7676
1.8904
1.8764
1.8764
σ
2
=
10
7.5775
1.0016
0.8479
7.0682
7.5694
7.2969
7.2666Table 15
Estimated MSE withn
=
50, p
=
6, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0224
0.0221
0.0216
0.0216
0.0220
0.0223
0.0223
σ
2
=
1
0.0895
0.0850
0.0790
0.0847
0.0886
0.0894
0.0894
σ
2
=
2
0.3581
0.2944
0.2400
0.3373
0.3562
0.3573
0.3574
σ
2
=
5
2.2380
0.9162
0.6324
2.1053
2.2333
2.2068
2.2102
σ
2
=
10
8.9518
1.1747
0.8681
8.4195
8.9425
8.4697
8.5206Table 16
Estimated MSE withn
=
50, p
=
6, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0276
0.0270
0.0265
0.0264
0.0269
0.0275
0.0275
σ
2
=
1
0.1104
0.1017
0.0950
0.1013
0.1087
0.1103
0.1103
σ
2
=
2
0.4415
0.3288
0.2775
0.4009
0.4381
0.4404
0.4400
σ
2
=
5
2.7591
0.8493
0.6783
2.4979
2.7506
2.7213
2.6973
σ
2
=
10
11.0363
1.0223
0.8903
9.9873
11.0193
10.4518
10.1007Table 17
Estimated MSE withn
=
50, p
=
8, and γ
=
0.75.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0694
0.0648
0.0621
0.0631
0.0685
0.0692
0.0692
σ
2
=
1
0.2778
0.2162
0.1940
0.2469
0.2759
0.2776
0.2775
σ
2
=
2
1.1112
0.5257
0.4531
0.9819
1.1074
1.1079
1.1065
σ
2
=
5
6.9449
0.8932
0.8221
6.1268
6.9354
6.8179
6.7664
σ
2
=
10
27.7795
0.9781
0.9476
24.5017
2.77606
25.8426
25.1061Table 18
Estimated MSE withn
=
50, p
=
8, and γ
=
0.85.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0729
0.0678
0.0649
0.0660
0.0719
0.0728
0.0728
σ
2
=
1
0.2915
0.2244
0.2010
0.2576
0.2894
0.2912
0.2911
σ
2
=
2
1.1658
0.5382
0.4642
1.0243
1.1617
1.1622
1.1605
σ
2
=
5
7.2865
0.8989
0.8291
6.3908
7.2761
7.1469
7.0805
σ
2
=
10
29.1462
0.9794
0.9500
25.5570
29.1253
27.0208
26.0808Table 19
Estimated MSE withn
=
50, p
=
8, and γ
=
0.95.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.0861
0.0830
0.0754
0.0815
0.0855
0.0869
0.0860
σ
2
=
1
0.3446
0.2994
0.2267
0.3248
0.3432
0.3445
0.3442
σ
2
=
2
1.3782
0.8508
0.5033
1.2979
1.3756
1.3768
1.3725
σ
2
=
5
8.6138
1.5585
0.8517
8.1096
8.6072
8.5589
8.3965
σ
2
=
10
34.4552
1.3911
0.9575
32.4371
34.4419
33.5908
31.1970Table 20
Estimated MSE withn
=
50, p
=
8, and γ
=
0.99.
ME
K
1
K
2
K
3
K
4
K
5
K
6
σ
2
=
0.5
0.1063
0.1009
0.0906
0.0984
0.1048
0.1062
0.1062
σ
2
=
1
0.4252
0.3494
0.2622
0.3908
0.4222
0.4251
0.4243
σ
2
=
2
1.7009
0.8972
0.5531
1.5603
1.6949
1.6988
1.6857
σ
2
=
5
10.6308
1.3896
0.8768
9.7471
10.6157
10.5469
10.0593
σ
2
=
10
42.5233
1.2263
0.9656
38.9855
42.4929
41.2088
34.5263


### 3.2.1. The Performance as a Function ofn

Increasing the number of observations has a negative effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are decreasing, although SRRE is always better than the ME. However, the performance of the SSRE over the ME is not very obvious. Thus, both measurements of performances show that the advantage of SRRE decreases with the increase of the number of observations.


### 3.2.2. The Performance as a Function ofp

Increasing the independent variables has a positive effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are increasing. When increasing the independent variables, the SRRE is better than the ME, and the performance of the SRRE over the ME is very obvious. Thus, both measurements of performances show that the advantage of SRRE increases with the increase of the number of observations.


### 3.2.3. The Performance as a Function ofγ

Increasing the degree of correlation has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase asγ increases. Thus, the gain of applying SRRE increases as the degree of correlation increases.


### 3.2.4. The Performance as a Function ofσ
2

Increasingσ
2 has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase as σ
2 increases. When σ
2 is big, the performance of the SRRE over the ME is very obvious.


## 3.2.1. The Performance as a Function ofn

Increasing the number of observations has a negative effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are decreasing, although SRRE is always better than the ME. However, the performance of the SSRE over the ME is not very obvious. Thus, both measurements of performances show that the advantage of SRRE decreases with the increase of the number of observations.


## 3.2.2. The Performance as a Function ofp

Increasing the independent variables has a positive effect on all of the estimators. With the increasing of observations, the MSE of SRRE and ME are increasing. When increasing the independent variables, the SRRE is better than the ME, and the performance of the SRRE over the ME is very obvious. Thus, both measurements of performances show that the advantage of SRRE increases with the increase of the number of observations.


## 3.2.3. The Performance as a Function ofγ

Increasing the degree of correlation has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase asγ increases. Thus, the gain of applying SRRE increases as the degree of correlation increases.


## 3.2.4. The Performance as a Function ofσ
2

Increasingσ
2 has a clear negative impact on the ME and SRRE. MSEs of these estimators actually increase as σ
2 increases. When σ
2 is big, the performance of the SRRE over the ME is very obvious.


## 4. Numerical Example

In this example, we discuss the dataset on Portland cement originally due to Woods et al. [21], and the dataset has then been widely analyzed in the literature such as Kaciranlar et al. [10]. Firstly, we assemble the data as follows:
(16)
X
=
(
7
26
6
60
1
29
15
52
11
56
8
20
11
31
8
47
7
52
6
33
11
55
9
22
3
71
17
6
1
31
22
44
2
54
18
22
21
47
4
26
1
40
23
34
11
66
9
12
10
68
8
12
)
r
=
(
78.5
74.3
104.3
87.6
95.9
109.2
102.7
72.5
93.1
115.9
83.8
113.3
109.4
)
.Consider the following matrix restriction:(17)
r
=
R
β
+
e
,
e
~
N
(
0
,
σ
2
I
2
)
,
where
(18)
R
=
(
1
1
3
1
0
1
1
1
)
,
r
=
(
1
1
)
.
The estimated regression coefficients along with the MSE are presented in Table 21.Table 21
Comparative performance of the estimators.
ME
K
1
K
2
K
3
K
4
K
5
K
6
MSE
0.0611
0.0604
0.0609
0.0609
0.0610
0.0608
0.0611
β
1
2.1172
2.1056
2.0762
2.1152
2.1165
2.1130
2.1172
β
2
1.1741
1.1767
1.1834
1.1745
1.1742
1.1751
1.7740
β
3
0.6830
0.6749
0.6543
0.6816
0.6825
0.6800
0.6829
β
4
0.5006
0.5023
0.5067
0.5009
0.5007
0.5012
0.5006From Table21, we can see that the proposed SRRE are performing better than the ME in the sense of smaller MSE. We also noted that the estimator K
1 performed the best followed by K
5.


## 5. Concluding Remarks

In this paper, we discussed the usefulness of the RR to obtain better parameter estimators in linear regressions model with stochastic restrictions when the independent variables are highly correlated. By a Monte Carlo experiment, we evaluate some recently proposed methods of estimating the ridge parameter that we apply in linear regressions model. The results show that the newly proposed estimators outperform the mixed estimator.



---

*Source: 508793-2014-04-30.xml*