# A New Type Iterative Ridge Estimator: Applications and Performance Evaluations

Authors: Aydın Karakoca
Journal: Journal of Mathematics
 (2022)

---

## Abstract

The usage of the ridge estimators is very common in presence of multicollinearity in multiple linear regression models. The ridge estimators are used as an alternative to ordinary least squares in case of multicollinearity as they have lower mean square error. Choosing the optimal value of the biasing parameterk is vital in ridge regression in terms of bias-variance trade off. Since the theoretical comparisons among the ridge estimators are not possible, it is general practice to carry out a Monte Carlo study to compare them. When the Monte Carlo designs on the existing ridge estimators are examined, it is seen that the performances of the ridge estimators are only considered for the same level of relationship between the independent variables. However, it is more likely to encounter different levels of relationships between the independent variables in real data sets. In this study, a new type iterative ridge estimator is proposed based on a modified form of the estimated mean square error function. Furthermore, a novel search algorithm is provided to achieve the estimations. The performance of the proposed estimator is compared with that of the ordinary least squares estimator and existing 18 ridge estimators through an extensive Monte Carlo design. In the design of the Monte Carlo, both data generation techniques were taken into account, based on the constant and varying correlation levels between the independent variables. Two illustrative real data examples are presented. The proposed estimator outperforms the existing estimators in the sense of the mean squared error for both data generating types. Moreover, it is also superior with respect to the k-fold cross-validation method in the real data examples.


---

## Body

## 1. Introduction

Let us consider the general form of the multiple linear regression model:(1)Y=Xβ+ε,where Y is a n×1 vector of response (dependent) variable, β is a p×1 unknown regression coefficient vector, X is a n×p design matrix of rank p, and ε is a n×1 random error vector, which is multivariate normal distributed with zero mean vector and σ2I variance-covariance matrix. The ordinary least square (OLS) estimator of β is given by β^OLS=X′X−1X′Y and the variance-covariance matrix of β^OLS is σ2X′X−1 [1].Recall that the OLS estimator is unbiased and has minimum variance among all unbiased estimators. The OLS estimator maintains its unbiasedness but loses the property of minimum variance in the presence of multicollinearity. Consequently, large standard errors lead to wider confidence intervals and serious errors in the interpretation of model parameters. Several estimators have been proposed to solve the multicollinearity problem [1–10]. These studies have focused on reducing the variance by sacrificing unbiasedness and thus finding estimators that have smaller mean square error (MSE). The ridge regression aims to solve the multicollinearity problem by adding small positive values to the diagonal elements of X′X matrix. The ridge estimator kHK=σ^2/α^max2 where α=α1,α2,…,αp′ is an unknown parameter vector with size p×1 given by equation (2) is proposed by [4]. Most of the ridge estimators were derived from transformations of previously proposed estimators. For example, k^HKB is the harmonic mean of ki=σ2/αi2; furthermore, kAM, kGM, and kMED estimators were also obtained using the arithmetic mean, geometric mean, and the median of ki=σ2/αi2, respectively [1, 5]. kGM and kKSM estimators are suggested by [1, 11], respectively. Six ridge estimators proposed by adding the multiplier include different quantiles of Wi=tmax/α^i (where tmax is maximum eigenvalue of X′X matrix) to kGM and kKSM estimators by [12]. These examples can be extended by studies where new estimators were obtained by applying minimum, maximum, and square root transformations to existing estimators [8, 9].In addition to the estimators in which Ridge and Liu type estimators are combined, the estimators proposed in recent studies for the solution of problems involving multicollinearity and outlier problems have come to the fore [13–16]. There are studies that involve modifying different estimation methods or using ridge estimators in different regression models. For example, the authors in [17] proposed four ridge estimators based on OLS estimates obtained by the deleted-d jackknife method, which are proposed as an alternative to the ridge estimator based on the classical principal components estimator. The authors in [18] also proposed an extention of Kibria Lukman estimator for the gamma regression model in the presence of multicollinearity.In this study, a new estimator is proposed based on a modified form of the estimated MSE function, rather than using any transformations on existing estimators and also a new fast search algorithm is given to obtain the ridge estimates. We examined the available ridge estimators by simulation and found that the estimated MSEs of these estimators are much larger than theoretical MSE. Furthermore, we also observed similar estimated MSEs for very large or very small ridge estimates. To overcome the stated shortcomings of these estimators, we proposed the new estimator whose estimated MSE is close to the theoretical MSE. When the Monte Carlo studies on ridge estimators are examined, it is seen that they are carried out by assuming that the relationships between all independent variables are equal. However, it is more likely to encounter different levels of relationships between variables with real data sets. In this study, the data generation technique can generate data according to the relationships at different levels between the independent variables as like in real data sets used for assessing the performance of estimators in addition to existing artificial data generation technique. The organization of the paper is as follows: the ridge estimators of concern and proposed estimator are given in Section2. In Section 3, the Monte Carlo simulation and findings on the performance evaluations of the estimators are discussed. Two real data examples and results are presented in Section 4. Finally, some concluding remarks are given in Section 5.


## 2. Statistical Methodology

The ridge estimator ofβ is denoted by β^k=X′X+kIp−1X′Y where k is known as “ridge” or “shrinkage” parameter. It is also clear that β^OLS is a special case of β^k for k=0kOLS=0 [4].The canonical form of equation (1) can be expressed using Z=XD and α=Dβ as(2)Y=Zα+ε,where Λ=diagλ1,λ2,…,λp represents the eigenvalues of X′X. D is defined as an orthogonal eigenvector matrix of size p×p such that D′X′XD=Λ. The ridge estimator of α is given as follows:(3)α^k=Z′Z+kI−1Z′Y,where I is the identity matrix of size p×p, k>0, and α^=Λ−1Z′Y is the OLS estimator of α. The MSE of α^k is given in equation (4) and minimized by ki=σ2/αi2 (where αi is the i th element of α) [3, 4].(4)MSEα^k=σ2∑i=1pλiλi+k2+∑i=1pk2αi2λi+k2.In real data examples, the estimated MSE given in equation (5) is used for performance comparisons of ridge estimators since σ and α are unknown [1].(5)MSE^α^k=σ^2∑i=1pλiλi+k2+∑i=1pk2α^i2λi+k2.In Table1, some of the ridge estimators in the literature are given.Table 1
Some of the ridge estimators.
FormulationkOLS=0kHK=σ^2/α^max2kHKB=pσ^2/α^′α^kLW=pσ^2/α^′X′Xα^kHSL=σ^2∑i=1pλiα^i2/∑i=1pλiα^i22kAM=1/p∑i=1pσ^2/α^i2kGM=σ^2/∏i=1pα^i21/pkMED=medianσ^2/α^i2i=1,2,…,pkGMKS=∏i=1pλiσ^2/n−pσ^2+λiα^i21/pkKM2=max1/σ^2/α^i2i=1,2,…,pkKM3=maxσ^2/α^i2i=1,2,…,pkKM5=∏i=1pσ^2/α^i21/pkKM6=median1/σ^2/α^i2i=1,2,…,pkKM11=∏i=1pλmaxσ^2/n−pσ^2+λmaxα^i21/pkHMO=pσ^2/∑i=1pα^i2/1+1+λiα^i2/σ^21/2kY1=1/p∑i=1pσ^2/λiα^i2kY3=medianσ^2/λiα^i2i=1,2,…,pkY4=maxσ^2/λiα^i2i=1,2,…,pα^maxis the maximum element ofα^,σ^2=ε^′ε^′/n−pandλiisi−th eigenvalue ofX′X.


### 2.1. The Proposed Estimator

Generally, in the literature, the performance of ridge estimators on real datasets is compared by the least squares-based estimated MSE [1, 2, 6, 8, 9, 11, 12, 14–18]. From this perspective, we focused on the estimated MSE based on OLS, and considering the simulation results, we found that this approach fails and the estimated MSEs of these estimators are much larger than theoretical MSE. The fact that different ridge estimators give similar estimated MSEs for very large or very small ridge estimates indicates that optimal ridge estimation cannot be obtained considering the convex nature of the MSE function. For the reasons stated, a new ridge estimator named kMEM is proposed, based on the optimization of a modified but still convex function of the estimated MSE function. kMEM given in equation (6) is obtained by modifying MSE^α^k in equation (5). The proposed estimator differs from MSE^α^k in three aspects. First, α^i in equation (6) uses the ridge estimates of canonical parameter for given ridge parameter not OLS estimates. All α^i values in each estimator given in Table 1 are the OLS estimates of α. Second, the n−p multiplier in variance component is added to take into account the sample size effect. Finally, the k2/σ^2 term ensures the small ridge estimates; it is clear that large ridge estimates will increase the MSE.(6)kMEM=argmink>0n−pσ^2∑i=1pλiλi+k2+k2∑i=1pα^i2λi+k2+k2σ^2.Since the solution ofk in (6) cannot be obtained analytically, we proposed a fast search algorithm. The steps of the algorithm are as follows at the t-th iteration.Algorithm 1: The search algorithm.
(1)
The tolerance value set to Tol = 10−6.(2)
Letk1t=10−8 and k2t=2k1t.(3)
Calculate thatMEMkjtα^i are canonical ridge estimates for kjtj=1,2,MEMkjt=n−pσ^2∑i=1pλi/λi+kjt2+kjt2∑i=1pα^i2/λi+kjt2+kjt2/σ^2,j=1,2.(4)
LetD=MEMk1t−MEMk2t.(5)
IfD< Tol, then go to 9; else 6.(6)
IfMEMk1t>MEMk2t, then k1t+1=k2t and k2t+1=2k2t.(7)
IfMEMk1t<MEMk2t, then k1t+1=k1t and k2t+1=k1t+k2t/2.(8)
Go to 3(9)
IfMEMk1t<MEMk2t, then k^MEM=k1t else k^MEM=k2t.


## 2.1. The Proposed Estimator

Generally, in the literature, the performance of ridge estimators on real datasets is compared by the least squares-based estimated MSE [1, 2, 6, 8, 9, 11, 12, 14–18]. From this perspective, we focused on the estimated MSE based on OLS, and considering the simulation results, we found that this approach fails and the estimated MSEs of these estimators are much larger than theoretical MSE. The fact that different ridge estimators give similar estimated MSEs for very large or very small ridge estimates indicates that optimal ridge estimation cannot be obtained considering the convex nature of the MSE function. For the reasons stated, a new ridge estimator named kMEM is proposed, based on the optimization of a modified but still convex function of the estimated MSE function. kMEM given in equation (6) is obtained by modifying MSE^α^k in equation (5). The proposed estimator differs from MSE^α^k in three aspects. First, α^i in equation (6) uses the ridge estimates of canonical parameter for given ridge parameter not OLS estimates. All α^i values in each estimator given in Table 1 are the OLS estimates of α. Second, the n−p multiplier in variance component is added to take into account the sample size effect. Finally, the k2/σ^2 term ensures the small ridge estimates; it is clear that large ridge estimates will increase the MSE.(6)kMEM=argmink>0n−pσ^2∑i=1pλiλi+k2+k2∑i=1pα^i2λi+k2+k2σ^2.Since the solution ofk in (6) cannot be obtained analytically, we proposed a fast search algorithm. The steps of the algorithm are as follows at the t-th iteration.Algorithm 1: The search algorithm.
(1)
The tolerance value set to Tol = 10−6.(2)
Letk1t=10−8 and k2t=2k1t.(3)
Calculate thatMEMkjtα^i are canonical ridge estimates for kjtj=1,2,MEMkjt=n−pσ^2∑i=1pλi/λi+kjt2+kjt2∑i=1pα^i2/λi+kjt2+kjt2/σ^2,j=1,2.(4)
LetD=MEMk1t−MEMk2t.(5)
IfD< Tol, then go to 9; else 6.(6)
IfMEMk1t>MEMk2t, then k1t+1=k2t and k2t+1=2k2t.(7)
IfMEMk1t<MEMk2t, then k1t+1=k1t and k2t+1=k1t+k2t/2.(8)
Go to 3(9)
IfMEMk1t<MEMk2t, then k^MEM=k1t else k^MEM=k2t.


## 3. Simulation Study

This section is about the Monte Carlo simulation including the factors having a significant impact on the estimators and the performance evaluation criteria of the estimators under consideration. We examined the structure of the correlation matrix (type of the data generation), sample size, degree of the multicollinearity, and variances of the error terms as the factors that affect the performance of the estimators.


### 3.1. Simulation Layout

Many authors assumed that the relationship between all the variables is equal for numerical evaluations of ridge estimators [1, 2, 6–9]. These works used the data generation method proposed by [19]. We denoted this type of data generation with DG and applied it as follows. The data matrix X of size nxp whose columns is the explanatory variable which has bivariate correlations with ρ2 generated by(7)xij=1−ρ2zij+ρzip;i=1,2,…,n,j=1,2,…,p,where zij are produced from the standard normal distribution [19].We also examined data generating according to different levels of correlations between independent variables. We denoted this type of data generation with CG and applied as follows. The equationX=ZCR can be written provided that R is the correlation matrix of X and CR is the Cholesky decomposition of R, and Z is the pseudo random matrix with standard normal distribution. The CG type data generation is examined by considering the correlation matrices determined such as R1 and R2. These correlation matrices are generated according to the partial correlations specified by the Vine method [20]. The correlation coefficients given in R1 are spread over to a narrow range, while R2 is in a wider range in absolute value.(8)R1=1.0000−0.94260.9956−0.9932−0.94261.0000−0.94730.95890.9956−0.94731.00000.98450.99560.95890.98451.0000,R2=1.0000−0.9215−0.75420.5679−0.92151.00000.9289−0.2073−0.75420.92891.00000.08380.5679−0.20730.08381.0000.The simulation study is conducted as follows. First, the four-variable (p = 4) design matrix X is generated for each type of data generation. The correlations are considered corresponding to ρ2=0.95,0.99, and 0.999for DG and according to R1 and R2 for CG. Then, for both types of data generations, the following procedure is applied in the same manner. Each column of X is centered and standardized by calculating the z-score and divided by n−1. The dependent variable is also standardized. Thus, X′X and X′y are in a correlation form. β is chosen as the normalized eigenvector corresponding to the largest eigenvalue of the X′X matrix. The values of the other selected factors in simulation study are considered as sample size n = 30, 50, and 100 and standard deviation of error term σ=0.1,0.5,1,5, and10.The observations for the dependent variable are generated by (9) where εi are independent and identically distributed sample from the normal distribution with zero mean and σ2 variance.(9)yi=β0+β1xi1+β2xi2+β3xi3+β4xi4+εi,i=1,2,…,n.For selected data generating type and values ofn and σ, different samples are generated using equation (9) and the simulation is repeated 10000 times. For each replicate, the average MSE (AMSE) of the α^k estimator given in (10) is used for performance evaluation criteria:(10)AMSEα^k=1p×110000×∑t=110000α^k,t−α′α^k,t−α,where α^k,t is estimated parameter vector for ridge estimator at the t-th iteration for the true parameter vector α.


### 3.2. Results and Discussion

The results of the simulation based on DG are given in Tables2–4. We observed from the results of DG type data generating that kMEM has the lowest AMSE value for all given values of ρ2,σ, and n. The OLS estimator is performed the worst in terms of AMSE criteria. We also observed from Tables 2–4 that as the degree of correlation increases, AMSEs of kAM, kY1, kY3, kY4, and kMEM estimators are decreased for all sample size except σ = 0.1 and n = 30.Table 2
AMSEs of the estimators for DG andn = 30.
Estimatorρ2=0.95CN=15.86ρ2=0.99CN=36.34ρ2=0.999CN=115.66σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.020.120.260.911.060.030.080.220.901.060.010.070.210.901.05kKM30.110.210.383.2112.310.060.120.261.695.700.020.090.231.022.15kY30.091.140.392.629.420.080.460.292.177.700.080.120.252.067.27kY10.170.160.384.3616.690.160.130.303.2112.260.160.120.282.9211.13kHSL0.540.311.423.9714.940.740.170.913.6213.770.260.160.383.6113.80kY40.330.220.436.0923.770.320.200.404.1016.050.320.190.393.6014.08kKM20.111.070.298.9932.770.220.940.406.4223.880.390.280.655.0519.44kKM60.340.351.68118.38511.270.480.340.95115.69512.230.670.332.03141.69343.77kAM0.210.236.3798.27379.090.490.377.81272.881052.090.990.6418.381131.354338.15kMED0.282.626.8898.27379.090.798.0820.30272.881052.093.8137.8389.761131.354338.15kGM0.232.626.88458.641910.230.798.0820.301073.604447.764.2737.8389.763689.9413123.13kKM50.234.9318.17439.351755.470.7912.3144.561069.494272.334.2753.44190.064529.8718092.32kKM110.332.227.77188.94755.220.919.3036.89920.863683.314.4591.51366.019150.1236600.45kHMO0.112.519.94247.24988.760.5012.3449.281230.914923.494.94123.23492.8312319.7449278.79kLW0.336.4125.39632.732530.781.3431.67126.473160.3412641.4213.04324.261296.8132418.59129674.18kHKB0.437.5729.81741.542965.711.6437.48149.443732.4014929.1615.14374.681498.2437452.24149808.49kHK0.598.8034.87869.883479.171.9643.94175.684391.2317564.7117.64439.981759.9043996.96175987.68kGMKS0.9019.8877.751925.627700.004.1987.90340.648404.3233602.1135.30675.682579.6963258.87252870.24kOLS1.0826.97107.872696.6810786.725.48136.91547.6313690.7554762.9955.071376.715506.84137670.92550683.67Table 3
AMSEs of the estimators for DG andn = 50.
Estimatorρ2=0.95CN=20.96ρ2=0.99CN=47.45ρ2=0.999CN=150.12σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.100.240.861.010.020.070.210.861.010.010.060.200.861.01kKM30.080.170.312.439.110.050.110.241.404.380.020.080.220.951.73kY30.090.820.332.368.410.080.280.272.087.340.080.090.252.017.09kY10.160.140.333.6814.040.170.120.292.9811.370.160.110.282.8210.75kHSL0.650.281.213.6913.940.640.180.663.4713.230.150.130.323.5113.41kY40.320.210.414.8919.100.320.200.393.7514.690.320.190.393.4913.66kKM20.141.050.317.5027.250.260.690.455.5120.690.410.180.704.8318.64kKM60.380.341.55119.33510.790.520.340.93102.73525.230.720.341.01128.79477.35kAM0.250.266.06128.51490.990.640.438.15350.181330.370.800.7023.481391.355261.81kMED0.393.629.36128.51490.991.0110.9727.11350.181330.376.2449.28115.211391.355261.81kGM0.333.629.36585.182506.281.1310.9727.111310.765854.455.8849.28115.214507.1318831.97kKM50.337.8329.67726.702904.761.1319.2972.271762.267042.965.8881.33301.277315.9529234.78kKM110.493.3212.49309.051235.851.3415.4761.701541.646166.436.23153.75615.0015374.7361498.90kHMO0.174.0616.14402.401609.420.8120.1780.592013.598054.188.04200.81803.1420077.4980309.77kLW0.5010.7442.721066.404265.532.2253.64214.335357.0721428.3321.65539.122156.0253894.11215573.89kHKB0.6312.4649.371230.414921.142.6462.24248.496208.5824833.8524.97620.592481.8862043.20248172.32kHK0.8314.4257.471435.485741.663.0572.54290.137252.5329009.9429.46735.852943.4173584.64294338.39kGMKS1.5936.59144.803604.9014417.587.54168.03661.9416451.9065794.3265.651373.465367.45132995.34531817.28kOLS1.8245.60182.394559.7618239.049.23230.80923.2023080.0192320.0392.292307.179228.70230717.41922869.64Table 4
AMSEs of the estimators for DG andn = 100.
Estimatorρ2=0.95CN=17.25ρ2=0.99CN=39.84ρ2=0.999CN=127.23σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.110.270.861.010.010.080.240.861.000.010.060.220.851.00kKM30.110.220.393.3312.920.060.120.261.695.820.020.080.221.012.18kY30.091.080.392.649.520.080.440.282.167.680.080.110.252.057.23kY10.170.170.384.3816.820.160.130.303.1912.180.160.110.282.8911.03kHSL0.500.371.323.9514.870.690.250.863.5713.570.240.130.373.5413.54kY40.330.220.436.1423.970.320.200.404.0815.980.320.190.393.5613.97kKM20.100.970.288.5430.980.210.880.396.1922.910.380.260.644.9619.06kKM60.330.351.94125.87515.470.470.341.17117.10507.790.660.340.8099.60501.97kAM0.200.215.9885.12329.330.430.367.10236.04906.670.910.6314.65970.843705.63kMED0.232.265.9685.12329.330.687.0217.71236.04906.673.2533.0278.42970.843705.63kGM0.202.265.96416.491768.920.687.0217.71969.554382.033.7133.0278.423058.1215466.54kKM50.207.8530.75763.023051.300.6817.4567.771676.886705.253.7167.90261.986467.8125860.67kKM110.411.896.47156.01623.381.007.8030.87770.473081.694.2577.29309.147728.3430913.32kHMO0.112.439.60238.83955.130.4912.0248.001198.834795.154.83120.51481.9612047.8148191.07kLW0.336.5425.89645.432581.551.3431.58126.043148.3012592.5612.76316.881267.1631674.16126694.95kHKB0.417.2728.65712.572849.801.5936.29144.733614.7014458.3314.71364.261456.6136411.76145646.58kHK0.578.4233.35831.573325.901.9243.22172.804319.2117276.6317.49436.131744.5243612.48174449.75kGMKS1.0024.1696.312404.999619.574.91116.15462.2611535.9046140.7745.031027.454076.76101627.22406468.89kOLS1.1027.44109.762744.1110976.435.63140.72562.8714071.6656286.6356.871421.715686.83142170.76568683.04Generally, the standard deviation of the error terms increases, and the AMSE values also increase. However, the change of AMSE values forkKM6, kHSL, kAM, and kKM2 estimators is irregular. As the standard deviation of the error terms increases, the kMEM estimator is the most homogeneous estimator in terms of AMSE values for ρ2 = 0.95 and ρ2 = 0.99, while it is the second best homogeneous estimator for ρ2 = 0.999.We observed from the CG type data generating results given in Tables5–7 that the kMEM estimator has the lowest AMSE value for all values of ρ2, σ, and n. The OLS estimator is also performed the worst among the estimators in terms of AMSE criteria.Table 5
AMSEs of the estimators for CG andn = 30.
EstimatorCase 1 withR1 (CN = 61.38)Case 2 withR2 (CN = 50.71)σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.050.120.260.901.060.050.230.461.021.13kKM60.360.230.292.9211.180.430.260.499.7938.91kY30.080.160.375.8822.920.140.290.7112.0447.35kY10.160.220.374.2416.270.240.350.648.5333.10kKM30.100.200.383.9314.790.300.400.646.1723.27kY40.320.350.432.589.260.420.470.624.8217.86kAM0.201.061.398.7031.630.300.530.959.5035.95kMED0.282.247.94193.53773.560.100.641.4223.0390.36kGM0.252.867.66110.66429.370.060.782.3443.00169.59kKM50.252.867.66110.66429.370.060.782.3443.00169.59kKM20.110.291.74120.75489.400.080.893.95144.96526.75kHSL0.551.076.41513.642144.040.250.883.90167.07645.75kKM110.355.1719.07461.341843.370.142.168.22201.50805.50kHMO0.174.0416.06400.201600.620.256.0424.08601.042404.04kLW0.4810.5341.921047.324189.660.9522.1088.142201.298804.88kHKB0.5911.7746.641162.684650.290.9622.1588.322205.938823.46kHK0.7613.2752.771317.255268.641.2628.87115.102873.1311491.49kGMKS1.3026.34101.612502.6710005.720.719.6335.85872.553487.31kOLS1.8345.70182.814570.3318281.314.64116.02464.0911602.2846409.11Table 6
AMSEs of the estimators for CG andn = 50.
EstimatorCase 1 withR1 (CN = 61.38)Case 2 withR2 (CN = 50.71)σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.110.260.881.030.040.240.480.991.09kKM60.370.240.302.8210.740.450.260.5211.0544.13kY30.090.160.365.7422.430.150.310.7312.3948.76kY10.170.220.364.1315.740.250.360.668.6333.22kKM30.100.200.373.9514.860.320.420.666.1122.96kY40.330.350.432.518.920.430.480.644.8217.70kAM0.200.961.298.4230.640.330.530.939.0734.07kMED0.312.057.35179.52717.630.100.571.3221.3883.80kGM0.252.757.28105.25403.820.050.672.0438.81152.09kKM50.252.757.28105.25403.820.050.672.0438.81152.09kKM20.120.341.89123.21518.770.060.602.8594.90479.27kHSL0.560.896.07489.112069.940.200.783.30111.20471.55kKM110.416.4724.56601.972406.270.162.519.66238.44953.40kHMO0.174.1616.55412.581650.140.276.6426.45660.382641.38kLW0.5211.1344.201101.754406.171.0825.45101.562537.0410147.98kHKB0.6212.4949.551235.464941.431.0825.48101.682540.0210159.82kHK0.8215.0359.881495.425981.361.4133.12132.143298.8813194.45kGMKS1.6135.63140.283485.7513940.080.9814.9857.351411.725644.07kOLS2.0150.30201.195029.8320119.335.70142.42569.6814242.0456968.15Table 7
AMSEs of the estimators for CG andn = 100.
EstimatorCase 1 withR1 (CN = 61.38)Case 2 withR2 (CN = 50.71)σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.110.270.871.020.040.250.510.981.07kKM60.380.240.302.7010.280.450.270.5210.8943.35kY30.090.150.355.5421.640.160.310.7212.2748.18kY10.160.220.354.0815.650.260.370.668.5633.27kKM30.090.190.363.9414.920.330.430.666.1123.00kY40.320.350.422.549.110.440.490.644.7917.77kAM0.211.001.288.3930.750.330.530.929.0734.18kMED0.292.338.51208.72834.420.110.591.3220.8581.71kGM0.273.048.14121.27470.690.060.682.0238.08150.82kKM50.273.048.14121.27470.690.060.682.0238.08150.82kKM20.120.281.54120.01478.490.070.693.09109.30429.72kHSL0.530.866.12494.512076.730.220.834.14115.16455.29kKM110.519.2636.12895.303580.180.295.0819.89493.661974.17kHMO0.235.5021.90546.492185.810.235.4821.82544.392177.40kLW0.6815.1560.251501.626004.890.8820.4681.582036.998147.50kHKB0.7616.2464.561610.936443.300.8920.5381.862044.418177.36kHK1.0019.9179.401983.337932.991.1426.14104.202600.3610400.37kGMKS2.3053.27211.645278.4921112.291.6432.41127.663174.6712696.42kOLS2.7969.65278.596964.8127859.224.52112.95451.8011295.0645180.25It is also observed that as the number of observations increases, the AMSE values ofkGMKS and kKM11 increase for each correlation structure and σ values. However, as the number of observations increase for σ= 0.1, σ= 5, and σ = 10 values, AMSE of the kMEM estimator decreases for both correlation structures.Generally, we conclude that whenσ increases, AMSE values of all the estimators are increased for all sample size and both correlation structures. It is also observed that AMSE values are higher when the correlation matrix has high variability.


## 3.1. Simulation Layout

Many authors assumed that the relationship between all the variables is equal for numerical evaluations of ridge estimators [1, 2, 6–9]. These works used the data generation method proposed by [19]. We denoted this type of data generation with DG and applied it as follows. The data matrix X of size nxp whose columns is the explanatory variable which has bivariate correlations with ρ2 generated by(7)xij=1−ρ2zij+ρzip;i=1,2,…,n,j=1,2,…,p,where zij are produced from the standard normal distribution [19].We also examined data generating according to different levels of correlations between independent variables. We denoted this type of data generation with CG and applied as follows. The equationX=ZCR can be written provided that R is the correlation matrix of X and CR is the Cholesky decomposition of R, and Z is the pseudo random matrix with standard normal distribution. The CG type data generation is examined by considering the correlation matrices determined such as R1 and R2. These correlation matrices are generated according to the partial correlations specified by the Vine method [20]. The correlation coefficients given in R1 are spread over to a narrow range, while R2 is in a wider range in absolute value.(8)R1=1.0000−0.94260.9956−0.9932−0.94261.0000−0.94730.95890.9956−0.94731.00000.98450.99560.95890.98451.0000,R2=1.0000−0.9215−0.75420.5679−0.92151.00000.9289−0.2073−0.75420.92891.00000.08380.5679−0.20730.08381.0000.The simulation study is conducted as follows. First, the four-variable (p = 4) design matrix X is generated for each type of data generation. The correlations are considered corresponding to ρ2=0.95,0.99, and 0.999for DG and according to R1 and R2 for CG. Then, for both types of data generations, the following procedure is applied in the same manner. Each column of X is centered and standardized by calculating the z-score and divided by n−1. The dependent variable is also standardized. Thus, X′X and X′y are in a correlation form. β is chosen as the normalized eigenvector corresponding to the largest eigenvalue of the X′X matrix. The values of the other selected factors in simulation study are considered as sample size n = 30, 50, and 100 and standard deviation of error term σ=0.1,0.5,1,5, and10.The observations for the dependent variable are generated by (9) where εi are independent and identically distributed sample from the normal distribution with zero mean and σ2 variance.(9)yi=β0+β1xi1+β2xi2+β3xi3+β4xi4+εi,i=1,2,…,n.For selected data generating type and values ofn and σ, different samples are generated using equation (9) and the simulation is repeated 10000 times. For each replicate, the average MSE (AMSE) of the α^k estimator given in (10) is used for performance evaluation criteria:(10)AMSEα^k=1p×110000×∑t=110000α^k,t−α′α^k,t−α,where α^k,t is estimated parameter vector for ridge estimator at the t-th iteration for the true parameter vector α.


## 3.2. Results and Discussion

The results of the simulation based on DG are given in Tables2–4. We observed from the results of DG type data generating that kMEM has the lowest AMSE value for all given values of ρ2,σ, and n. The OLS estimator is performed the worst in terms of AMSE criteria. We also observed from Tables 2–4 that as the degree of correlation increases, AMSEs of kAM, kY1, kY3, kY4, and kMEM estimators are decreased for all sample size except σ = 0.1 and n = 30.Table 2
AMSEs of the estimators for DG andn = 30.
Estimatorρ2=0.95CN=15.86ρ2=0.99CN=36.34ρ2=0.999CN=115.66σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.020.120.260.911.060.030.080.220.901.060.010.070.210.901.05kKM30.110.210.383.2112.310.060.120.261.695.700.020.090.231.022.15kY30.091.140.392.629.420.080.460.292.177.700.080.120.252.067.27kY10.170.160.384.3616.690.160.130.303.2112.260.160.120.282.9211.13kHSL0.540.311.423.9714.940.740.170.913.6213.770.260.160.383.6113.80kY40.330.220.436.0923.770.320.200.404.1016.050.320.190.393.6014.08kKM20.111.070.298.9932.770.220.940.406.4223.880.390.280.655.0519.44kKM60.340.351.68118.38511.270.480.340.95115.69512.230.670.332.03141.69343.77kAM0.210.236.3798.27379.090.490.377.81272.881052.090.990.6418.381131.354338.15kMED0.282.626.8898.27379.090.798.0820.30272.881052.093.8137.8389.761131.354338.15kGM0.232.626.88458.641910.230.798.0820.301073.604447.764.2737.8389.763689.9413123.13kKM50.234.9318.17439.351755.470.7912.3144.561069.494272.334.2753.44190.064529.8718092.32kKM110.332.227.77188.94755.220.919.3036.89920.863683.314.4591.51366.019150.1236600.45kHMO0.112.519.94247.24988.760.5012.3449.281230.914923.494.94123.23492.8312319.7449278.79kLW0.336.4125.39632.732530.781.3431.67126.473160.3412641.4213.04324.261296.8132418.59129674.18kHKB0.437.5729.81741.542965.711.6437.48149.443732.4014929.1615.14374.681498.2437452.24149808.49kHK0.598.8034.87869.883479.171.9643.94175.684391.2317564.7117.64439.981759.9043996.96175987.68kGMKS0.9019.8877.751925.627700.004.1987.90340.648404.3233602.1135.30675.682579.6963258.87252870.24kOLS1.0826.97107.872696.6810786.725.48136.91547.6313690.7554762.9955.071376.715506.84137670.92550683.67Table 3
AMSEs of the estimators for DG andn = 50.
Estimatorρ2=0.95CN=20.96ρ2=0.99CN=47.45ρ2=0.999CN=150.12σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.100.240.861.010.020.070.210.861.010.010.060.200.861.01kKM30.080.170.312.439.110.050.110.241.404.380.020.080.220.951.73kY30.090.820.332.368.410.080.280.272.087.340.080.090.252.017.09kY10.160.140.333.6814.040.170.120.292.9811.370.160.110.282.8210.75kHSL0.650.281.213.6913.940.640.180.663.4713.230.150.130.323.5113.41kY40.320.210.414.8919.100.320.200.393.7514.690.320.190.393.4913.66kKM20.141.050.317.5027.250.260.690.455.5120.690.410.180.704.8318.64kKM60.380.341.55119.33510.790.520.340.93102.73525.230.720.341.01128.79477.35kAM0.250.266.06128.51490.990.640.438.15350.181330.370.800.7023.481391.355261.81kMED0.393.629.36128.51490.991.0110.9727.11350.181330.376.2449.28115.211391.355261.81kGM0.333.629.36585.182506.281.1310.9727.111310.765854.455.8849.28115.214507.1318831.97kKM50.337.8329.67726.702904.761.1319.2972.271762.267042.965.8881.33301.277315.9529234.78kKM110.493.3212.49309.051235.851.3415.4761.701541.646166.436.23153.75615.0015374.7361498.90kHMO0.174.0616.14402.401609.420.8120.1780.592013.598054.188.04200.81803.1420077.4980309.77kLW0.5010.7442.721066.404265.532.2253.64214.335357.0721428.3321.65539.122156.0253894.11215573.89kHKB0.6312.4649.371230.414921.142.6462.24248.496208.5824833.8524.97620.592481.8862043.20248172.32kHK0.8314.4257.471435.485741.663.0572.54290.137252.5329009.9429.46735.852943.4173584.64294338.39kGMKS1.5936.59144.803604.9014417.587.54168.03661.9416451.9065794.3265.651373.465367.45132995.34531817.28kOLS1.8245.60182.394559.7618239.049.23230.80923.2023080.0192320.0392.292307.179228.70230717.41922869.64Table 4
AMSEs of the estimators for DG andn = 100.
Estimatorρ2=0.95CN=17.25ρ2=0.99CN=39.84ρ2=0.999CN=127.23σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.110.270.861.010.010.080.240.861.000.010.060.220.851.00kKM30.110.220.393.3312.920.060.120.261.695.820.020.080.221.012.18kY30.091.080.392.649.520.080.440.282.167.680.080.110.252.057.23kY10.170.170.384.3816.820.160.130.303.1912.180.160.110.282.8911.03kHSL0.500.371.323.9514.870.690.250.863.5713.570.240.130.373.5413.54kY40.330.220.436.1423.970.320.200.404.0815.980.320.190.393.5613.97kKM20.100.970.288.5430.980.210.880.396.1922.910.380.260.644.9619.06kKM60.330.351.94125.87515.470.470.341.17117.10507.790.660.340.8099.60501.97kAM0.200.215.9885.12329.330.430.367.10236.04906.670.910.6314.65970.843705.63kMED0.232.265.9685.12329.330.687.0217.71236.04906.673.2533.0278.42970.843705.63kGM0.202.265.96416.491768.920.687.0217.71969.554382.033.7133.0278.423058.1215466.54kKM50.207.8530.75763.023051.300.6817.4567.771676.886705.253.7167.90261.986467.8125860.67kKM110.411.896.47156.01623.381.007.8030.87770.473081.694.2577.29309.147728.3430913.32kHMO0.112.439.60238.83955.130.4912.0248.001198.834795.154.83120.51481.9612047.8148191.07kLW0.336.5425.89645.432581.551.3431.58126.043148.3012592.5612.76316.881267.1631674.16126694.95kHKB0.417.2728.65712.572849.801.5936.29144.733614.7014458.3314.71364.261456.6136411.76145646.58kHK0.578.4233.35831.573325.901.9243.22172.804319.2117276.6317.49436.131744.5243612.48174449.75kGMKS1.0024.1696.312404.999619.574.91116.15462.2611535.9046140.7745.031027.454076.76101627.22406468.89kOLS1.1027.44109.762744.1110976.435.63140.72562.8714071.6656286.6356.871421.715686.83142170.76568683.04Generally, the standard deviation of the error terms increases, and the AMSE values also increase. However, the change of AMSE values forkKM6, kHSL, kAM, and kKM2 estimators is irregular. As the standard deviation of the error terms increases, the kMEM estimator is the most homogeneous estimator in terms of AMSE values for ρ2 = 0.95 and ρ2 = 0.99, while it is the second best homogeneous estimator for ρ2 = 0.999.We observed from the CG type data generating results given in Tables5–7 that the kMEM estimator has the lowest AMSE value for all values of ρ2, σ, and n. The OLS estimator is also performed the worst among the estimators in terms of AMSE criteria.Table 5
AMSEs of the estimators for CG andn = 30.
EstimatorCase 1 withR1 (CN = 61.38)Case 2 withR2 (CN = 50.71)σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.050.120.260.901.060.050.230.461.021.13kKM60.360.230.292.9211.180.430.260.499.7938.91kY30.080.160.375.8822.920.140.290.7112.0447.35kY10.160.220.374.2416.270.240.350.648.5333.10kKM30.100.200.383.9314.790.300.400.646.1723.27kY40.320.350.432.589.260.420.470.624.8217.86kAM0.201.061.398.7031.630.300.530.959.5035.95kMED0.282.247.94193.53773.560.100.641.4223.0390.36kGM0.252.867.66110.66429.370.060.782.3443.00169.59kKM50.252.867.66110.66429.370.060.782.3443.00169.59kKM20.110.291.74120.75489.400.080.893.95144.96526.75kHSL0.551.076.41513.642144.040.250.883.90167.07645.75kKM110.355.1719.07461.341843.370.142.168.22201.50805.50kHMO0.174.0416.06400.201600.620.256.0424.08601.042404.04kLW0.4810.5341.921047.324189.660.9522.1088.142201.298804.88kHKB0.5911.7746.641162.684650.290.9622.1588.322205.938823.46kHK0.7613.2752.771317.255268.641.2628.87115.102873.1311491.49kGMKS1.3026.34101.612502.6710005.720.719.6335.85872.553487.31kOLS1.8345.70182.814570.3318281.314.64116.02464.0911602.2846409.11Table 6
AMSEs of the estimators for CG andn = 50.
EstimatorCase 1 withR1 (CN = 61.38)Case 2 withR2 (CN = 50.71)σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.110.260.881.030.040.240.480.991.09kKM60.370.240.302.8210.740.450.260.5211.0544.13kY30.090.160.365.7422.430.150.310.7312.3948.76kY10.170.220.364.1315.740.250.360.668.6333.22kKM30.100.200.373.9514.860.320.420.666.1122.96kY40.330.350.432.518.920.430.480.644.8217.70kAM0.200.961.298.4230.640.330.530.939.0734.07kMED0.312.057.35179.52717.630.100.571.3221.3883.80kGM0.252.757.28105.25403.820.050.672.0438.81152.09kKM50.252.757.28105.25403.820.050.672.0438.81152.09kKM20.120.341.89123.21518.770.060.602.8594.90479.27kHSL0.560.896.07489.112069.940.200.783.30111.20471.55kKM110.416.4724.56601.972406.270.162.519.66238.44953.40kHMO0.174.1616.55412.581650.140.276.6426.45660.382641.38kLW0.5211.1344.201101.754406.171.0825.45101.562537.0410147.98kHKB0.6212.4949.551235.464941.431.0825.48101.682540.0210159.82kHK0.8215.0359.881495.425981.361.4133.12132.143298.8813194.45kGMKS1.6135.63140.283485.7513940.080.9814.9857.351411.725644.07kOLS2.0150.30201.195029.8320119.335.70142.42569.6814242.0456968.15Table 7
AMSEs of the estimators for CG andn = 100.
EstimatorCase 1 withR1 (CN = 61.38)Case 2 withR2 (CN = 50.71)σ=0.1σ=0.5σ=1σ=5σ=10σ=0.1σ=0.5σ=1σ=5σ=10kMEM0.030.110.270.871.020.040.250.510.981.07kKM60.380.240.302.7010.280.450.270.5210.8943.35kY30.090.150.355.5421.640.160.310.7212.2748.18kY10.160.220.354.0815.650.260.370.668.5633.27kKM30.090.190.363.9414.920.330.430.666.1123.00kY40.320.350.422.549.110.440.490.644.7917.77kAM0.211.001.288.3930.750.330.530.929.0734.18kMED0.292.338.51208.72834.420.110.591.3220.8581.71kGM0.273.048.14121.27470.690.060.682.0238.08150.82kKM50.273.048.14121.27470.690.060.682.0238.08150.82kKM20.120.281.54120.01478.490.070.693.09109.30429.72kHSL0.530.866.12494.512076.730.220.834.14115.16455.29kKM110.519.2636.12895.303580.180.295.0819.89493.661974.17kHMO0.235.5021.90546.492185.810.235.4821.82544.392177.40kLW0.6815.1560.251501.626004.890.8820.4681.582036.998147.50kHKB0.7616.2464.561610.936443.300.8920.5381.862044.418177.36kHK1.0019.9179.401983.337932.991.1426.14104.202600.3610400.37kGMKS2.3053.27211.645278.4921112.291.6432.41127.663174.6712696.42kOLS2.7969.65278.596964.8127859.224.52112.95451.8011295.0645180.25It is also observed that as the number of observations increases, the AMSE values ofkGMKS and kKM11 increase for each correlation structure and σ values. However, as the number of observations increase for σ= 0.1, σ= 5, and σ = 10 values, AMSE of the kMEM estimator decreases for both correlation structures.Generally, we conclude that whenσ increases, AMSE values of all the estimators are increased for all sample size and both correlation structures. It is also observed that AMSE values are higher when the correlation matrix has high variability.


## 4. Real Data Examples

In the previous section, a detailed simulation study is given to compare the performances of selected estimators. To see the real performances of the estimators, two real data examples are given. The presence of collinearity is determined by the condition number (CN). The square root of the ratio of the largest eigenvalue of the matrixX′X to the smallest eigenvalue is determined as CN. Generally, CN values between 10 and 30 indicate the moderate multicollinearity. The values of the CN greater than 30 were accepted as indicative of strong collinearity [2]. Multicollinearity is high if the CN is between 30 and 100 and severe when its greater than 100 [13].Unlike Monte Carlo studies, the performance of the estimators was evaluated with the k-fold cross validation (CV) method, considering that the true values of the model parameters are not known in real data applications. In the CV method, the dataset is randomly divided intok parts (folds), and each part consists of selected rows of X and Y. For the partitions created, the data set is randomly divided into training and test data k times. The number of observations in training and test data is n−n/k and n/k, respectively. The model estimated from training data is used for estimating the test data [21]. The CV statistic is given in equation (11) where yi,j is the j-th observation in the i-th fold, y^i,j is prediction of j-th observation in the i-th fold using remaining (k−1) fold as training data and is used for evaluating the performance of the each ridge estimator.(11)CV=1k∑i=1k∑j=1n/kyi,j−y^i,j2.


### 4.1. Gasoline Consumption Data

Gasoline consumption and automotive data are used in the first example [22]. The variables considered in the model are miles/gallon Y, displacement (cube inches) X1, torque (feet/pound) X2, carburetor (barrels) X3, number of transmission speed X4, and overall length (inches) X5. Correlations between all the variables are given in R3. The correlations changing moderate to high level among the independent variables are observed in R3. The condition number is 1132, which indicates the serious concern about multicollinearity. The results of the CV for different partitions are also given in Table 8.(12)R3=10.9900.640−0.7720.865−0.8720.99010.653−0.7460.864−0.8490.6400.6531−0.2760.422−0.472−0.772−0.746−0.2761−0.6550.7080.8650.8640.422−0.6551−0.752−0.872−0.849−0.4720.708−0.7521.Table 8
CV results of the estimators for gasoline consumption data.
5 fold6 fold10 foldEstimatorCVEstimatorCVEstimatorCVkMEM96.225kMEM75.111kMEM47.356kHKB96.331kHKB75.207kHKB47.404kKM1196.366kKM1175.240kKM1147.422kKM596.371kKM575.245kKM547.425kHK96.912kHK76.031kHK47.960kGMKS98.197kGMKS77.129kGMKS48.553kY498.602kHMO77.842kHMO49.000kHMO98.866kY478.368kY449.303kY199.607kY179.970kY150.145kY399.808kY380.314kY350.318kKM6100.443kKM380.754kKM650.872kKM2100.548kKM681.450kKM350.901kOLS100.572kKM281.649kKM250.966kLW100.572kLW81.693kOLS50.987kKM3101.540kOLS81.693kLW50.987kGM102.779kGM82.136kGM51.870kMED103.353kAM82.775kMED52.382kAM103.666kMED82.788kAM52.887kHSL106.075kHSL84.046kHSL53.599


### 4.2. Stack Loss Data

We used Brownlee’s stack loss data that contain 21-day data on the oxidation of ammonia to nitric acid in the second example [23]. The response variable and independent variables considered in the model are NH3 stack loss percentage Y, air flow X1, cooling water inlet temperature X2, and acid concentration X3, respectively. Correlations between all the variables are given in R4 where the correlations changing moderate to high level are observed among the independent variables. The condition number is 57.51, which indicates a serious concern about multicollinearity. The results of CV for two partitions are given in Table 9.(13)R4=10.7820.5000.9200.78210.3910.8760.5000.39110.4000.9200.8760.4001.Table 9
CV results of the estimators for stack loss data.
7 fold3 foldEstimatorCVEstimatorCVkMEM67.630kMEM112.097kY467.638kY4112.110kKM1167.642kY1112.121kKM567.643kKM6112.122kY167.645kY3112.126kKM667.646kKM2112.129kY367.648kLW112.131kKM267.650kOLS112.131kLW67.651kKM11112.210kOLS67.651kKM5112.213kHK67.765kHK112.513kMED68.282kMED113.602kHKB68.332kHKB113.702kKM368.489kKM3114.010kGMKS68.628kGMKS114.277kGM70.931kGM118.446kHMO78.785kHMO132.755kAM88.190kAM151.672kHSL110.478kHSL200.644According to the real data application results from Tables8 and 9, the kMEM estimator has the smallest CV value among the others for both gasoline consumption and stack loss data examples. The kHSL estimator has the largest CV for both applications for all partitions and performed the worst among all the estimators.A total of 75 different scenarios were evaluated in the simulations, 45 for DG and 30 for CG. ThekMEM estimator performed the best considering the number of takes places in top three for each scenario in terms of AMSE. When Figure 1 is examined, it is seen that the rankings of the other estimators for both DG and CG are inconsistent. We observe that the second best estimators after kMEM are kKM6 for DG and kY4 for CG.Figure 1
The frequency distribution of the estimators that takes places in top three for DG and CG.


## 4.1. Gasoline Consumption Data

Gasoline consumption and automotive data are used in the first example [22]. The variables considered in the model are miles/gallon Y, displacement (cube inches) X1, torque (feet/pound) X2, carburetor (barrels) X3, number of transmission speed X4, and overall length (inches) X5. Correlations between all the variables are given in R3. The correlations changing moderate to high level among the independent variables are observed in R3. The condition number is 1132, which indicates the serious concern about multicollinearity. The results of the CV for different partitions are also given in Table 8.(12)R3=10.9900.640−0.7720.865−0.8720.99010.653−0.7460.864−0.8490.6400.6531−0.2760.422−0.472−0.772−0.746−0.2761−0.6550.7080.8650.8640.422−0.6551−0.752−0.872−0.849−0.4720.708−0.7521.Table 8
CV results of the estimators for gasoline consumption data.
5 fold6 fold10 foldEstimatorCVEstimatorCVEstimatorCVkMEM96.225kMEM75.111kMEM47.356kHKB96.331kHKB75.207kHKB47.404kKM1196.366kKM1175.240kKM1147.422kKM596.371kKM575.245kKM547.425kHK96.912kHK76.031kHK47.960kGMKS98.197kGMKS77.129kGMKS48.553kY498.602kHMO77.842kHMO49.000kHMO98.866kY478.368kY449.303kY199.607kY179.970kY150.145kY399.808kY380.314kY350.318kKM6100.443kKM380.754kKM650.872kKM2100.548kKM681.450kKM350.901kOLS100.572kKM281.649kKM250.966kLW100.572kLW81.693kOLS50.987kKM3101.540kOLS81.693kLW50.987kGM102.779kGM82.136kGM51.870kMED103.353kAM82.775kMED52.382kAM103.666kMED82.788kAM52.887kHSL106.075kHSL84.046kHSL53.599


## 4.2. Stack Loss Data

We used Brownlee’s stack loss data that contain 21-day data on the oxidation of ammonia to nitric acid in the second example [23]. The response variable and independent variables considered in the model are NH3 stack loss percentage Y, air flow X1, cooling water inlet temperature X2, and acid concentration X3, respectively. Correlations between all the variables are given in R4 where the correlations changing moderate to high level are observed among the independent variables. The condition number is 57.51, which indicates a serious concern about multicollinearity. The results of CV for two partitions are given in Table 9.(13)R4=10.7820.5000.9200.78210.3910.8760.5000.39110.4000.9200.8760.4001.Table 9
CV results of the estimators for stack loss data.
7 fold3 foldEstimatorCVEstimatorCVkMEM67.630kMEM112.097kY467.638kY4112.110kKM1167.642kY1112.121kKM567.643kKM6112.122kY167.645kY3112.126kKM667.646kKM2112.129kY367.648kLW112.131kKM267.650kOLS112.131kLW67.651kKM11112.210kOLS67.651kKM5112.213kHK67.765kHK112.513kMED68.282kMED113.602kHKB68.332kHKB113.702kKM368.489kKM3114.010kGMKS68.628kGMKS114.277kGM70.931kGM118.446kHMO78.785kHMO132.755kAM88.190kAM151.672kHSL110.478kHSL200.644According to the real data application results from Tables8 and 9, the kMEM estimator has the smallest CV value among the others for both gasoline consumption and stack loss data examples. The kHSL estimator has the largest CV for both applications for all partitions and performed the worst among all the estimators.A total of 75 different scenarios were evaluated in the simulations, 45 for DG and 30 for CG. ThekMEM estimator performed the best considering the number of takes places in top three for each scenario in terms of AMSE. When Figure 1 is examined, it is seen that the rankings of the other estimators for both DG and CG are inconsistent. We observe that the second best estimators after kMEM are kKM6 for DG and kY4 for CG.Figure 1
The frequency distribution of the estimators that takes places in top three for DG and CG.


## 5. Conclusion

In this paper, we introduced a new type iterative ridge estimator. In previous studies, the performance of estimators is discussed only for the same relationships between independent variables [1, 2, 6, 8, 9]. In this study, the performances of the estimators were also compared for the relationships between the independent variables at different levels. We evaluated the performance of the estimator via an extensive Monte Carlo simulation study and two real data examples. Results of DG, which is used as a standard data generation technique in similar studies and CG, which provides more similar data to the real data structures are compared. Generally, in the literature, the performance of ridge estimators on real datasets is compared with that of the estimated MSE [1–3, 6–10]. A comparison based on mean square error (MSE) may be appropriate since the actual values of the parameters are known in simulation studies. However, since the actual values of the parameters are unknown in real data sets, performance comparison by estimated MSE based on a single sample may cause erroneous inferences. These situations have question marks in revealing the actual performances of the current estimators. However, since the actual parameter value is not known in practice, this approach will not be as reliable as in the simulation study. For this reason, we compared the performance of the estimators in real data applications with the k-fold cross-validation method. These results show that the proposed estimator outperforms than the others in the sense of AMSE in simulation study and the k-fold cross-validation for the real data examples. Therefore, we suggest researchers who encounter the multicollinearity problem to use kMEM as an alternative ridge estimator to the other estimators examined in this study.In the future, the work may be extended by comparing the proposed estimator with other estimators for multiple linear regressions. In these comparisons, the presence of multicollinearity and outlier can be evaluated simultaneously. The performance of the proposed estimator in different regression models such as Gamma regression can be examined. The proposed estimator can be used to obtain new estimators using the approach including deleted-d Jackknife by [17].



---

*Source: 3781655-2022-05-18.xml*