# Method for Solving LASSO Problem Based on Multidimensional Weight

Authors: Chen ChunRong; Chen ShanXiong; Chen Lin; Zhu YuChen
Journal: Advances in Artificial Intelligence
 (2017)

---

## Abstract

In the data mining, the analysis of high-dimensional data is a critical but thorny research topic. The LASSO (least absolute shrinkage and selection operator) algorithm avoids the limitations, which generally employ stepwise regression with information criteria to choose the optimal model, existing in traditional methods. The improved-LARS (Least Angle Regression) algorithm solves the LASSO effectively. This paper presents an improved-LARS algorithm, which is constructed on the basis of multidimensional weight and intends to solve the problems in LASSO. Specifically, in order to distinguish the impact of each variable in the regression, we have separately introduced part of principal component analysis (Part_PCA), Independent Weight evaluation, and CRITIC, into our proposal. We have explored that these methods supported by our proposal change the regression track by weighted every individual, to optimize the approach direction, as well as the approach variable selection. As a consequence, our proposed algorithm can yield better results in the promise direction. Furthermore, we have illustrated the excellent property of LARS algorithm based on multidimensional weight by the Pima Indians Diabetes. The experiment results show an attractive performance improvement resulting from the proposed method, compared with the improved-LARS, when they are subjected to the same threshold value.


---

## Body

## 1. Introduction

Data mining has shown its charm in the era of big data; it has gained much attention in academia regarding how to mine useful information from mass data by mathematical statistics model [1, 2]. In linear model, model error usually is a result of the lack of key variable. At the beginning of the modeling, generally, the more variables (attribute set) chosen are, the less the model error is. But, in the process of modeling, we need to find the attribute set which has the largest explanatory ability to the response, that is, improving the prediction precision and accuracy of the model through selecting variable [3]. Linear regression analysis is the most widely used of all statistical techniques [4]; the accuracy of that analysis mainly depends on the selection of variables and values of regression coefficients [5]. LASSO is an estimate method which can simplify the index set. In 1996, inspired by the ridge regression (Frank and Friedman, 1993) [6] and Nonnegative Garrote (Breiman, 1995) [7], Tibshirani proposed one new method of variable selection. The idea of this method is minimizing the square of the residuals with the constraint that the sum of the absolute values of regression coefficient β is less than a constant by construct a penalty function to shrinkage coefficient [8]. As a kind of compression estimates, the LASSO method has higher detection accuracy and better parameter convergence consistency. Efron et al. (2002) proposed the LARS algorithm to support the solution of LASSO [9]. And they proposed improved-LARS algorithm (2004) to eliminate the opposite sign of regression coefficient β and solve LASSO better [10]. The improved-LARS algorithm regresses stepwise; each path keeps the correlation between current residual individual and all the variables the same. It also satisfies the solution of LASSO with the same current approach direction and ensures the optimal results and algorithm complexity.Zou (2006) introduced the adaptive-LASSO by using the different tuning parameters for different regression coefficients. He suggests minimizing the following objective function [11]: (1)∑j=1nyj-xjTβ2+n∑i=1pλiβi.Keerthi and Shevade (2007) proposed a fast tracking algorithm for LASSO/LARS [12]; it approximates the logistic regression loss by a piecewise quadratic function.Charbonnier et al. (2010) suggest thatβ owns an internal structure that describes classes of connectivity between the variables [13]. They present the weighted-LASSO method to infer the parameters of a first-order vector autoregressive model that describes time course expression data generated by directed gene-to-gene regulation networks.Since the LASSO method minimizes the sum of squared residual errors, even though the least absolute deviation (LAD) estimator is an alternative to the OLS estimate, Jung (2011) proposed a robust-LASSO-estimator that is not sensitive to outliers, heavy-tailed errors, or leverage points [14].Bergersen et al. (2011) found that a large value ofwj, the regression coefficient for variable j, is subject to a larger penalty and therefore is less likely to be included in the model, and vice versa [15]. They proposed to use weighted-LASSO with integrated relevant external information on the covariates to guide the selection towards more stable results.Arslan (2012) found that, compared with the LAD-LASSO method, the weighted LAD-LASSO (WLAD-LASSO) method will resist the heavy-tailed errors and outliers in explanatory variables [16].LASSO problem is a convex minimization problem; the forward-backward splitting operator method is important to solving it. Salzo and Villa (2012) proposed accelerated version to improve the method’s convergence ability [17].Zhou et al. (2013) proposed an alternative selection procedure based on the kernelized LARS-LASSO method [18]. By formulating the RBF neural network as a linear-in-the-parameters model, they derived a l1-constrained objective function for training the network.Zhao et al. (2015) added two tuning parametersλ and j0 to the wavelet-based weighted-LASSO methods. The tuning parameter λ controls the model sparsity. The choice of j0 controls the optimal level of wavelet decomposition for the functional data. They improved wavelet-based LASSO by adding a prescreening step prior to the model fitting or, alternatively, by using a weighted version of wavelet-based LASSO [19].Salama et al. (2016) proposed a new LASSO algorithm, the minimum variance distortionless response (MVDR) LARS-LASSO [20], which solves the DOA problem in the CS framework.In light of superior performance achieved in [10] for solving LASSO problem, a new idea is extended in this paper into the uses of multidimensional weight LARS. Our main contributions are as follows:(i)
In the solving process of LASSO, each attribute in the evaluation population has different relative importance to the overall evaluation. The relative importance include the following: not all attributes influence the regression results and each individual in the regression model has different weight. When improved-LARS algorithm calculated the equiangular vector, we distinguish the effect resulting from different attribute variable, considering joint correlation between regression variables and surplus variable.(ii)
We discuss the method proposed in this paper by the experimental evidence of the Pima Indians Diabetes Data and two sets of evaluation index.In Section2, we introduce the LASSO problem and improved-LARS algorithm briefly, including theory and definition. In Section 3 we put forward the LARS algorithm based on multidimensional weighting model, which calculates the direction and variables based on the weighting variables and accelerates the approximation process in promising direction. We introduce the data sets and evaluation indicators when we verify algorithm and discuss the experimental results in Section 4. Section 5 is the summary and prospect of this paper.


## 2. LASSO Problem and Improved-LARS Algorithm

### 2.1. The Definition of LASSO

Suppose that there are the multidimensional variablesXj∈Rn,j=1,2,…,m, and response Y∈Rn. Each group of Xj has a corresponding yi. Regression coefficient β is estimated where βl1≤t when the sum of squared residuals is minimal. The LASSO linear regression model is defined by(2)y=Xβ+e.β is j-dimensional column vector, the parameter to be estimated. Error vector e meets E(e)=0 and Vare=σ2. Suppose sparse model Ey∣x=β1x1+β2x2+⋯+βjxj; most of regression coefficients β are 0 in β1,β2,…,βj. Based on obtaining data, variable selection can identify which coefficient is zero and estimate other nonzero parameters; it is looking for parameters to build a sparse model. The problem we need to solve in matrix is defined by(3)α,β=argmin⁡y-Xβ-αl2,βl1≤t,where t is the threshold value of the sum of regression coefficient and l1 and l2 are two types of regularization norms.


### 2.2. The Improved-LARS Algorithm

The improved-LARS algorithm can solve LASSO problem well, which is based on the Forward Selection algorithm and Forward Gradient algorithm. The improved-LRAS has appropriate forward distance, lower complexity, and more relevance of information. Figure1 shows the basic steps of algorithm.(i)
The improved-LARS calculates the correlation betweenXj and Y constantly and finds the individual xK most correlated with the response. It takes the largest step possible in the direction of this individual, using xK to approximate yi.(ii)
Until some other individual, sayxP, has the same correlation with the current residual individual, rxKy=rxPy. Improved-LARS process is in an equiangular direction xU (xU is the direction between the two predictors xP and xK).(iii)
When a third individualxT earns its way into the “most correlated” set, improved-LARS then proceeds equiangularly between xK, xP, and xT, that is, along the “least angle direction,” until a fourth individual enters, and so forth the direction equiangular means the bisector of each vector in high dimension.(iv)
The LARS procedure works until the residual error is less than a threshold or all the variables are involved in the approach, the algorithm stop.Figure 1
Basic steps of the LARS.In Figure1, an example of two-dimensional problem, it starts with both coefficients equal to zero, firstly, finding the individual more correlated with the response y, rx1y>rx2y, approximating y along the x1 direction until residual of β1x1 and y has the same correlation with x1 and x2. Then the approximating direction changes to the equiangular between x1 and x2.


## 2.1. The Definition of LASSO

Suppose that there are the multidimensional variablesXj∈Rn,j=1,2,…,m, and response Y∈Rn. Each group of Xj has a corresponding yi. Regression coefficient β is estimated where βl1≤t when the sum of squared residuals is minimal. The LASSO linear regression model is defined by(2)y=Xβ+e.β is j-dimensional column vector, the parameter to be estimated. Error vector e meets E(e)=0 and Vare=σ2. Suppose sparse model Ey∣x=β1x1+β2x2+⋯+βjxj; most of regression coefficients β are 0 in β1,β2,…,βj. Based on obtaining data, variable selection can identify which coefficient is zero and estimate other nonzero parameters; it is looking for parameters to build a sparse model. The problem we need to solve in matrix is defined by(3)α,β=argmin⁡y-Xβ-αl2,βl1≤t,where t is the threshold value of the sum of regression coefficient and l1 and l2 are two types of regularization norms.


## 2.2. The Improved-LARS Algorithm

The improved-LARS algorithm can solve LASSO problem well, which is based on the Forward Selection algorithm and Forward Gradient algorithm. The improved-LRAS has appropriate forward distance, lower complexity, and more relevance of information. Figure1 shows the basic steps of algorithm.(i)
The improved-LARS calculates the correlation betweenXj and Y constantly and finds the individual xK most correlated with the response. It takes the largest step possible in the direction of this individual, using xK to approximate yi.(ii)
Until some other individual, sayxP, has the same correlation with the current residual individual, rxKy=rxPy. Improved-LARS process is in an equiangular direction xU (xU is the direction between the two predictors xP and xK).(iii)
When a third individualxT earns its way into the “most correlated” set, improved-LARS then proceeds equiangularly between xK, xP, and xT, that is, along the “least angle direction,” until a fourth individual enters, and so forth the direction equiangular means the bisector of each vector in high dimension.(iv)
The LARS procedure works until the residual error is less than a threshold or all the variables are involved in the approach, the algorithm stop.Figure 1
Basic steps of the LARS.In Figure1, an example of two-dimensional problem, it starts with both coefficients equal to zero, firstly, finding the individual more correlated with the response y, rx1y>rx2y, approximating y along the x1 direction until residual of β1x1 and y has the same correlation with x1 and x2. Then the approximating direction changes to the equiangular between x1 and x2.


## 3. LARS Based on Multidimensional Weight

### 3.1. Algorithm Analysis

In the process of improved-LARS stepwise regression, the angle regression takes all selected variables with the same importance. However, each individual ofXj has different weight in the regression model that the indicators in its overall evaluation have different relative importance. We take the correlation between individual Xj and surplus variables into consideration, taking it and the correlation between Xj and yi as the condition to select approximation individual.In Figure3, originally, we choose x1 as the first approximating variable for rx1y>rx2y. We take the predictor Xj’s contribution rate to the whole system as one approximating condition; the new correlation is (4)RXiy=rXiy∗WXjvu,where WXj is the contribution rate; we will describe the calculation method of it in detail later; u are v are constants.Because of the addition condition, it will inevitably increase the range of values about judgment condition. In order to keep the stability of the system, we limit the product in[v,u].After transformation, it may indicate the possibility thatxj could have been chosen to be increasing. For example, Figure 2 shows that the transformed x2 gets θ2 closer to y, reaching x2′, and the transformed x1 gets θ1 closer to y, reaching x1′. It also may indicate the possibility that xj could be selected to be reducing; the transformed x1 gets θ2 far away toy, reaching x1′′.Figure 2
The correlation ofX and Y change when adding the multidimensional weight.Figure 3
The predictor direction when adding the multidimensional weight.On the basis of Figure1, the new correlation will significantly change the approximation process. Figure 3 shows the predictor direction when adding the multidimensional weight: the possibility is selected of two predictors’ change. It starts with x2′; the correlation of x1′ is the same as that of residual e=y-β1′x2′; when moving to β1′x2, the approximation direction changes to the equiangular direction of x1′ and x2′; then it moves forward β2′x1′+x2′; the approximation process is complete. Approximating path changes, and the regression variables β1 and β2 also change; we can get the new β calculated by improved regression method.Applying the aforementioned process to multidimensional high-order system, the collectedm feature indicators and n objects are expressed as(5)XN×m=x11⋯x⋯x1m⋮⋱⋰⋮xi1xijxim⋮⋰⋱⋮xN1⋯xNj⋯xNm,i=1,2,…,N,j=1,2,…,mor(6)X=X1⋯Xj⋯Xm,j=1,2,…,m.The collected result response is(7)Y=y1⋮yi⋮yN,i=1,2,…,N.There are many calculation methods ofWxi in regression process. Without loss of generality, the calculation methods should be unartificial, relative number, quantitative, and independent. We use part_PCA, Independence Weight, and CRITIC to control regression process and verify superiority of the algorithm.


#### 3.1.1. Part of Principle Components Analysis

PCA uses orthogonal transformation for dimension reduction in statistics [21, 22]. It transforms data into a new coordinate system. The biggest variance of the data projection is in the first coordinate, called the 1st principal component. The second of the data projection is in the second coordinate, called the 2nd principal component, and so on. It keeps the low-order principal component of data set, ignores the high-order principal component after transform, identifies the dominating factors, and keeps topm principal component whose overall information utilization rate is higher than 85%. Inspired by the principle of PCA, we preserve the value of all the components simultaneously and identify the variance contribution of each component; the part_PCA algorithm steps are as follows.Sample Standardization
(8)
Z
i
j
=
x
i
j
-
x
-
j
s
j
,
i
=
1,2
,
…
,
N
;
j
=
1,2
,
…
,
m
,where(9)x-j=∑i=1nxijn,sj2=∑i=1nxij-x-j2n-1.The correlation coefficient matrixR is information utilization of each feature(10)R=rijmxp=ZTZn-1,where(11)rij=∑zkj·zkjn-1,i,j=1,2,…,m.


#### 3.1.2. Independence Weight

We sort the multiple correlation coefficient of individual by multiple regression in statistical methods [23, 24]; the greater the multiple correlation coefficient is, the more repeated the information should be, the smaller the information utilization should be, and the smaller the weight should be. Calculation steps are(12)Rj′=∑Xj-X-X⌢-X-∑Xj-X-2∑X⌢-X-2,j=1,2,…,m.X
⌢ is the rest matrix in X except Xj(13)X-=meanX.ForR is negatively related with weight, we take the reciprocal of multiple correlation coefficient as score, getting the weighting coefficient through normalized processing(14)R=1R1′⋯1Rj′⋯1Rm′,j=1,2,…,m.


#### 3.1.3. CRITIC

Based on Independence Weight, CRITIC is a kind of objective weighting method proposed by Diakoulaki et al. [25]. It is based on identifying the objective weight of individual to evaluate the contrast and conflict between indicators [26, 27]. The standard deviation indicates the gap of each scheme in the same index, which can express contrast intensity. The conflict between the indicators is based on the correlation between indicators; relevancy can express conflict.The quantitative conflict between thej indicator and other indicators is (15)∑t=1m1-rtj.r
t
j is the correlation coefficient between Xt and Xj(16)rtj=∑Xt-X-Xj-X-∑Xt-X-2∑Xj-X-2.C
j is the amount of information the j indicator includes (17)Cj=σj∑t=1m1-rtj,j=1,2,…,m.The greaterCj, the greater amount of information and the more relative important of that indicator. The objective weight of j indicator is (18)Rj=Cj∑t=1mCj,j=1,2,…,m.


### 3.2. Algorithm Steps

In order to obtain the numerical solution of stability,X and y in (3) are standardized and preprocessed to omit α, so that ∑jxij/N=0, Xl1=1, Y-=0.ForA=sj1xj1,…,sjlxjl,…,sjkxjk⊆1,2,…,m, define the matrix(19)XA=sj1xj1,…,sjlxjl,…,sjkxjk∈Rn×k.X
A is the column vector selected from X to satisfy A, with the same direction to current Y, where the signs sjl equal to(20)sjl=1,xjl,y>0,0,xjl,y=0,-1,xjl,y<0.The equiangularuA is as the following: (21)GA=XATXA,AA=1ATGA-11A-1/2,wA=AAGA-11A,uA=XAwA.u
A is the unit vector making equal angles, less than 90°, with the columns of XA. 1A is a vector of 1’s of length equaling A, the size of A. wA is the equiangular contribution of each attribute Xi which is selected in XA. wA is processed by weighting to change approach direction and approach variable selection.We now can further describe the improved-LARS based on multidimensional algorithm; we begin atu0=0 and build up newuby steps. Suppose that u⌢A is the current computation and c is the current correlation between predictor and response vector(22)c=XTy-u⌢Aorcj=xj,y-u⌢A.Theactive setA is the set of indices corresponding to covariates with the greatest absolute current correlations, when =maxj⁡cj; uA corresponding to A is the approximate direction. Let (23)sj=signC,j∈A,a=XTuAoraj=xj,uA.The length of approach along theuA direction now is (24)γ=minj∉A+⁡C-cjAA-aj,C+cjAA+aj.“min+” indicates the calculation of the minimum of positive components within each choice of j in this approximating process. Each predictor in A corresponds to increase in γωA; we add weight to control approach direction.Part_PCA, Independent Weight evaluation, and CRITIC are added in the process of improved-LARS algorithm, respectively. Three parallel algorithms are established to calculate the weight of each attribute.(25)Rjpart_PCA=part_PCAX1:roww,:,(26)RjIW=IWX1:roww,:,(27)RjCRITIC=CRITICX1:roww,:.The centralized weightRj indicates the impact of weight on the approach direction after the aforementioned three methods. (28)Rj=centralizationRjpart_PCA/IW/CRITIC,w′=w×Rj⌢,βA=βA+γw′.R
j is the same-dimension weight matrix from Rj∗; the approach direction estimate is as the following:(29)u~A+=u⌢A+γuARj⌢.Then the newactive set is(30)A+=A∪j′.j
′ is the minimizing index in (25) of j. In order to conform to the requirements of the LASSO solution that the track should keep the same direction with current approach direction, the step size of the first opposite sign is (31)γ⌢=minj∉A+⁡-βjwj.Whenγ⌢<γ, there is opposite sign, removing j′ from A. Then the algorithm enters the next approximation process. When using A+ instead of A, repeat the above steps, until the residual error is less than a threshold or all the variables are involved in the approach. The pseudocode is shown in Pseudocode 1.Pseudocode 1
Input. Response y dependent on predictor X, error tolerance ε
Output. LASSO solution β for (3)
(1)begin
(2)data preprocessing: Normalization (X,y)
(3)initialization: u=0, y⌢=y-u
(4) c=XTy⌢
(5) C=maxj{|cj|}
(6) j⌢=arg⁡maxj{|cj|}, A={j⌢}
(7)calculating weight by  Rweight=part_PCA(X)   or  Rweight=IW(X)  or  Rweight=CRITIC(X)
(8) centralizationRweight;
(9)wheny⌢l2<εandA≤mdoβcycle;
(10)in LARS cycle
rate=Rweight1:row(w),:
w′=w∗rate
(11)end cycle;
(12)returnβ.This improved algorithm increases the calculation steps for adding the weighting analysis, so the calculation time increases. But the approach mechanism of each variable in the statistical model stays the same, so the space complexity is consistent with the original algorithm.


## 3.1. Algorithm Analysis

In the process of improved-LARS stepwise regression, the angle regression takes all selected variables with the same importance. However, each individual ofXj has different weight in the regression model that the indicators in its overall evaluation have different relative importance. We take the correlation between individual Xj and surplus variables into consideration, taking it and the correlation between Xj and yi as the condition to select approximation individual.In Figure3, originally, we choose x1 as the first approximating variable for rx1y>rx2y. We take the predictor Xj’s contribution rate to the whole system as one approximating condition; the new correlation is (4)RXiy=rXiy∗WXjvu,where WXj is the contribution rate; we will describe the calculation method of it in detail later; u are v are constants.Because of the addition condition, it will inevitably increase the range of values about judgment condition. In order to keep the stability of the system, we limit the product in[v,u].After transformation, it may indicate the possibility thatxj could have been chosen to be increasing. For example, Figure 2 shows that the transformed x2 gets θ2 closer to y, reaching x2′, and the transformed x1 gets θ1 closer to y, reaching x1′. It also may indicate the possibility that xj could be selected to be reducing; the transformed x1 gets θ2 far away toy, reaching x1′′.Figure 2
The correlation ofX and Y change when adding the multidimensional weight.Figure 3
The predictor direction when adding the multidimensional weight.On the basis of Figure1, the new correlation will significantly change the approximation process. Figure 3 shows the predictor direction when adding the multidimensional weight: the possibility is selected of two predictors’ change. It starts with x2′; the correlation of x1′ is the same as that of residual e=y-β1′x2′; when moving to β1′x2, the approximation direction changes to the equiangular direction of x1′ and x2′; then it moves forward β2′x1′+x2′; the approximation process is complete. Approximating path changes, and the regression variables β1 and β2 also change; we can get the new β calculated by improved regression method.Applying the aforementioned process to multidimensional high-order system, the collectedm feature indicators and n objects are expressed as(5)XN×m=x11⋯x⋯x1m⋮⋱⋰⋮xi1xijxim⋮⋰⋱⋮xN1⋯xNj⋯xNm,i=1,2,…,N,j=1,2,…,mor(6)X=X1⋯Xj⋯Xm,j=1,2,…,m.The collected result response is(7)Y=y1⋮yi⋮yN,i=1,2,…,N.There are many calculation methods ofWxi in regression process. Without loss of generality, the calculation methods should be unartificial, relative number, quantitative, and independent. We use part_PCA, Independence Weight, and CRITIC to control regression process and verify superiority of the algorithm.


### 3.1.1. Part of Principle Components Analysis

PCA uses orthogonal transformation for dimension reduction in statistics [21, 22]. It transforms data into a new coordinate system. The biggest variance of the data projection is in the first coordinate, called the 1st principal component. The second of the data projection is in the second coordinate, called the 2nd principal component, and so on. It keeps the low-order principal component of data set, ignores the high-order principal component after transform, identifies the dominating factors, and keeps topm principal component whose overall information utilization rate is higher than 85%. Inspired by the principle of PCA, we preserve the value of all the components simultaneously and identify the variance contribution of each component; the part_PCA algorithm steps are as follows.Sample Standardization
(8)
Z
i
j
=
x
i
j
-
x
-
j
s
j
,
i
=
1,2
,
…
,
N
;
j
=
1,2
,
…
,
m
,where(9)x-j=∑i=1nxijn,sj2=∑i=1nxij-x-j2n-1.The correlation coefficient matrixR is information utilization of each feature(10)R=rijmxp=ZTZn-1,where(11)rij=∑zkj·zkjn-1,i,j=1,2,…,m.


### 3.1.2. Independence Weight

We sort the multiple correlation coefficient of individual by multiple regression in statistical methods [23, 24]; the greater the multiple correlation coefficient is, the more repeated the information should be, the smaller the information utilization should be, and the smaller the weight should be. Calculation steps are(12)Rj′=∑Xj-X-X⌢-X-∑Xj-X-2∑X⌢-X-2,j=1,2,…,m.X
⌢ is the rest matrix in X except Xj(13)X-=meanX.ForR is negatively related with weight, we take the reciprocal of multiple correlation coefficient as score, getting the weighting coefficient through normalized processing(14)R=1R1′⋯1Rj′⋯1Rm′,j=1,2,…,m.


### 3.1.3. CRITIC

Based on Independence Weight, CRITIC is a kind of objective weighting method proposed by Diakoulaki et al. [25]. It is based on identifying the objective weight of individual to evaluate the contrast and conflict between indicators [26, 27]. The standard deviation indicates the gap of each scheme in the same index, which can express contrast intensity. The conflict between the indicators is based on the correlation between indicators; relevancy can express conflict.The quantitative conflict between thej indicator and other indicators is (15)∑t=1m1-rtj.r
t
j is the correlation coefficient between Xt and Xj(16)rtj=∑Xt-X-Xj-X-∑Xt-X-2∑Xj-X-2.C
j is the amount of information the j indicator includes (17)Cj=σj∑t=1m1-rtj,j=1,2,…,m.The greaterCj, the greater amount of information and the more relative important of that indicator. The objective weight of j indicator is (18)Rj=Cj∑t=1mCj,j=1,2,…,m.


## 3.1.1. Part of Principle Components Analysis

PCA uses orthogonal transformation for dimension reduction in statistics [21, 22]. It transforms data into a new coordinate system. The biggest variance of the data projection is in the first coordinate, called the 1st principal component. The second of the data projection is in the second coordinate, called the 2nd principal component, and so on. It keeps the low-order principal component of data set, ignores the high-order principal component after transform, identifies the dominating factors, and keeps topm principal component whose overall information utilization rate is higher than 85%. Inspired by the principle of PCA, we preserve the value of all the components simultaneously and identify the variance contribution of each component; the part_PCA algorithm steps are as follows.Sample Standardization
(8)
Z
i
j
=
x
i
j
-
x
-
j
s
j
,
i
=
1,2
,
…
,
N
;
j
=
1,2
,
…
,
m
,where(9)x-j=∑i=1nxijn,sj2=∑i=1nxij-x-j2n-1.The correlation coefficient matrixR is information utilization of each feature(10)R=rijmxp=ZTZn-1,where(11)rij=∑zkj·zkjn-1,i,j=1,2,…,m.


## 3.1.2. Independence Weight

We sort the multiple correlation coefficient of individual by multiple regression in statistical methods [23, 24]; the greater the multiple correlation coefficient is, the more repeated the information should be, the smaller the information utilization should be, and the smaller the weight should be. Calculation steps are(12)Rj′=∑Xj-X-X⌢-X-∑Xj-X-2∑X⌢-X-2,j=1,2,…,m.X
⌢ is the rest matrix in X except Xj(13)X-=meanX.ForR is negatively related with weight, we take the reciprocal of multiple correlation coefficient as score, getting the weighting coefficient through normalized processing(14)R=1R1′⋯1Rj′⋯1Rm′,j=1,2,…,m.


## 3.1.3. CRITIC

Based on Independence Weight, CRITIC is a kind of objective weighting method proposed by Diakoulaki et al. [25]. It is based on identifying the objective weight of individual to evaluate the contrast and conflict between indicators [26, 27]. The standard deviation indicates the gap of each scheme in the same index, which can express contrast intensity. The conflict between the indicators is based on the correlation between indicators; relevancy can express conflict.The quantitative conflict between thej indicator and other indicators is (15)∑t=1m1-rtj.r
t
j is the correlation coefficient between Xt and Xj(16)rtj=∑Xt-X-Xj-X-∑Xt-X-2∑Xj-X-2.C
j is the amount of information the j indicator includes (17)Cj=σj∑t=1m1-rtj,j=1,2,…,m.The greaterCj, the greater amount of information and the more relative important of that indicator. The objective weight of j indicator is (18)Rj=Cj∑t=1mCj,j=1,2,…,m.


## 3.2. Algorithm Steps

In order to obtain the numerical solution of stability,X and y in (3) are standardized and preprocessed to omit α, so that ∑jxij/N=0, Xl1=1, Y-=0.ForA=sj1xj1,…,sjlxjl,…,sjkxjk⊆1,2,…,m, define the matrix(19)XA=sj1xj1,…,sjlxjl,…,sjkxjk∈Rn×k.X
A is the column vector selected from X to satisfy A, with the same direction to current Y, where the signs sjl equal to(20)sjl=1,xjl,y>0,0,xjl,y=0,-1,xjl,y<0.The equiangularuA is as the following: (21)GA=XATXA,AA=1ATGA-11A-1/2,wA=AAGA-11A,uA=XAwA.u
A is the unit vector making equal angles, less than 90°, with the columns of XA. 1A is a vector of 1’s of length equaling A, the size of A. wA is the equiangular contribution of each attribute Xi which is selected in XA. wA is processed by weighting to change approach direction and approach variable selection.We now can further describe the improved-LARS based on multidimensional algorithm; we begin atu0=0 and build up newuby steps. Suppose that u⌢A is the current computation and c is the current correlation between predictor and response vector(22)c=XTy-u⌢Aorcj=xj,y-u⌢A.Theactive setA is the set of indices corresponding to covariates with the greatest absolute current correlations, when =maxj⁡cj; uA corresponding to A is the approximate direction. Let (23)sj=signC,j∈A,a=XTuAoraj=xj,uA.The length of approach along theuA direction now is (24)γ=minj∉A+⁡C-cjAA-aj,C+cjAA+aj.“min+” indicates the calculation of the minimum of positive components within each choice of j in this approximating process. Each predictor in A corresponds to increase in γωA; we add weight to control approach direction.Part_PCA, Independent Weight evaluation, and CRITIC are added in the process of improved-LARS algorithm, respectively. Three parallel algorithms are established to calculate the weight of each attribute.(25)Rjpart_PCA=part_PCAX1:roww,:,(26)RjIW=IWX1:roww,:,(27)RjCRITIC=CRITICX1:roww,:.The centralized weightRj indicates the impact of weight on the approach direction after the aforementioned three methods. (28)Rj=centralizationRjpart_PCA/IW/CRITIC,w′=w×Rj⌢,βA=βA+γw′.R
j is the same-dimension weight matrix from Rj∗; the approach direction estimate is as the following:(29)u~A+=u⌢A+γuARj⌢.Then the newactive set is(30)A+=A∪j′.j
′ is the minimizing index in (25) of j. In order to conform to the requirements of the LASSO solution that the track should keep the same direction with current approach direction, the step size of the first opposite sign is (31)γ⌢=minj∉A+⁡-βjwj.Whenγ⌢<γ, there is opposite sign, removing j′ from A. Then the algorithm enters the next approximation process. When using A+ instead of A, repeat the above steps, until the residual error is less than a threshold or all the variables are involved in the approach. The pseudocode is shown in Pseudocode 1.Pseudocode 1
Input. Response y dependent on predictor X, error tolerance ε
Output. LASSO solution β for (3)
(1)begin
(2)data preprocessing: Normalization (X,y)
(3)initialization: u=0, y⌢=y-u
(4) c=XTy⌢
(5) C=maxj{|cj|}
(6) j⌢=arg⁡maxj{|cj|}, A={j⌢}
(7)calculating weight by  Rweight=part_PCA(X)   or  Rweight=IW(X)  or  Rweight=CRITIC(X)
(8) centralizationRweight;
(9)wheny⌢l2<εandA≤mdoβcycle;
(10)in LARS cycle
rate=Rweight1:row(w),:
w′=w∗rate
(11)end cycle;
(12)returnβ.This improved algorithm increases the calculation steps for adding the weighting analysis, so the calculation time increases. But the approach mechanism of each variable in the statistical model stays the same, so the space complexity is consistent with the original algorithm.


## 4. Experiment and Result Analysis

### 4.1. Introducing Data Set

For the characteristics of the compression of the regression coefficient, the experiment set should be sparse, as well as one dependent variable which is easy to distinguish. We take Pima Indians Diabetes Data Set provided by Applied Physics Laboratory of the Johns Hopkins University, for example [28]. The data record 768 performance descriptions, negative and positive diabetes sample, including 8 attributes variables and one classification value. In the classification value, “1” represents that the diabetes test result is positive; “0” represents negative. We verify algorithm by predictor of 8 attributes variables and response of one classification value. The goal of this test is improving on veracity comparing to original LARS algorithm.


### 4.2. Verification Condition

We use ROC curve to show results in order to evaluate more intuitively the performance of the proposed method. That is a binary classification problem whether the participants' diabetes is positive or negative; the testing results have the following four types:TP (true positive), the testing results are positive and are positive actuallyFP (false positive), the testing results are positive but are negative actuallyTN (true negative), the testing results are negative and are negative actuallyFN (false negative), the testing results are negative but are positive actuallyWe take the following three characters as the inspection standards through basic four types of statistics by ROC space:ACC (accuracy),ACC=TP+TN/P+NTPR (true positive rate),TPR=TP/P=TP/TP+FNNPV (negative predictive value),NPV=TN/TN+FNNPV is the proportion of correct detecting about negative; it means that the people who tested negative actually are negative. ACC represents the proportion of correct estimating in the sum of positive and negative. NPV is the proportion of people who tested negative in actually negative population. Compared to NPV, TPR is also called Hit Rate; it is the proportion of correctly detecting the people who actually are positive in the tested positive population. ACC, TPR, and NPV tell us the result is better or worse than LARS.Another character we judge the result with is SSR. The smooth turning point of SSR is corresponding to the optimal regression coefficient of predictor variable.


### 4.3. Experimental Result

Thresholdt starting from 0 increases to 1 for the step length 0.01; we draw the changing curve of ACC, TPR, and NPV, with the negative and positive as dependent variables and the 8 attribute variables as the independent variables.Figure4 shows the accuracy and the comprehensive optimal value of three inspection standards after each cycle of LASSO with Pima Indians Diabetes.Figure 4
Curves about three kinds of inspection standard.Figure4 shows that NPV are all improved when adding the weighting to LASSO’s solution; NPV is improved 5.16% when adding part_PCA, 5.58% when adding Independence Weight, and 5.1% when adding CRITIC. We can find that TPR is improved 13% when adding part_PCA, 14% when adding Independence Weight, and 13% when adding CRITIC for those methods changing the approach direction of algorithm. It is observed that ACC is improved 0.32% when adding part_PCA and Independence Weight. And there is no effect to ACC when adding CRITIC so the ACC keeps the original number.Therefore, the solutions of LASSO when adding the Independence Weight are optimal, followed by the CRITIC and part_PCA. This improved algorithm significantly increases the NPV and TPR ensuring the ACC is not reduced. The veracity of LASSO’s solutions is improved through changing approach direction when adding the weighting.Figure5 shows the SSR (Sum of Squared Residuals) of response and equiangular direction after each cycle of LASSO with Pima Indians Diabetes. It is obvious that the general trend of SSR completely remains consistent and the residual of optimal coefficient almost remains consistent when adding the different weight judgment. It will not change the advantages of LASSO when adding the weighting. If the optimal regression coefficient of predictor increases significantly, the SSR will change too. It increases 0.051 residual of LASSO solution when adding part_PCA, reduces 0.021 residual when adding CRITIC and Independence Weight. The results show that the final regression result is closer to the real response when adding CRITIC and Independence Weight.Figure 5
The curve about the SSR of response and equiangular direction.It can be found when synthesizing these three inspection standards that, adding the approach with multidimensional weight, the threshold value of the optimal solution is mainly reduced (except CRITIC), which means that the sum of absolute values of system’s regression coefficient is less than a smaller threshold; this algorithm meets the requirements in more extreme threshold range.Table1 shows the original regression coefficient β, βpart_PCA which adds part_PCA, βIW which adds Independence Weight, and βCRITIC which adds CRITIC.Table 1
Regression coefficient.
β1
β2
β3
β4
β5
β6
β7
β8
Β
1.7211
3.5409
−0.4862
−0.3665
−0.6504
2.2141
0.7071
−0.1810
Βpart_PCA
1.5300
3.5311
−0.4269
−0.3018
−0.6181
2.0409
0.6727
−0.0567
βIW
1.5050
3.5294
−0.4253
−0.2863
−0.6072
2.0066
0.6651
−0.0356
βCRITIC
1.6449
3.4482
−0.5019
−0.3705
−0.6302
2.1668
0.5471
−0.1597Table2 shows the difference and innovation of the multidimensional weight LARS, improved-LARS associated with part_PCA; Independence Weight and CRITIC act on β, changing the regression track and getting more accurate results.Table 2
Principles ofw in different algorithm.
Comparison ofw
Improved-LARS
w=wA=AAGA-11A
Multidimensional weight LARS
part_PCA
w=w∗centralization(part_PCAX)
Independence Weight
w=w∗centralization(IWX)
CRITIC
w=w∗centralization(CRITICX)


## 4.1. Introducing Data Set

For the characteristics of the compression of the regression coefficient, the experiment set should be sparse, as well as one dependent variable which is easy to distinguish. We take Pima Indians Diabetes Data Set provided by Applied Physics Laboratory of the Johns Hopkins University, for example [28]. The data record 768 performance descriptions, negative and positive diabetes sample, including 8 attributes variables and one classification value. In the classification value, “1” represents that the diabetes test result is positive; “0” represents negative. We verify algorithm by predictor of 8 attributes variables and response of one classification value. The goal of this test is improving on veracity comparing to original LARS algorithm.


## 4.2. Verification Condition

We use ROC curve to show results in order to evaluate more intuitively the performance of the proposed method. That is a binary classification problem whether the participants' diabetes is positive or negative; the testing results have the following four types:TP (true positive), the testing results are positive and are positive actuallyFP (false positive), the testing results are positive but are negative actuallyTN (true negative), the testing results are negative and are negative actuallyFN (false negative), the testing results are negative but are positive actuallyWe take the following three characters as the inspection standards through basic four types of statistics by ROC space:ACC (accuracy),ACC=TP+TN/P+NTPR (true positive rate),TPR=TP/P=TP/TP+FNNPV (negative predictive value),NPV=TN/TN+FNNPV is the proportion of correct detecting about negative; it means that the people who tested negative actually are negative. ACC represents the proportion of correct estimating in the sum of positive and negative. NPV is the proportion of people who tested negative in actually negative population. Compared to NPV, TPR is also called Hit Rate; it is the proportion of correctly detecting the people who actually are positive in the tested positive population. ACC, TPR, and NPV tell us the result is better or worse than LARS.Another character we judge the result with is SSR. The smooth turning point of SSR is corresponding to the optimal regression coefficient of predictor variable.


## 4.3. Experimental Result

Thresholdt starting from 0 increases to 1 for the step length 0.01; we draw the changing curve of ACC, TPR, and NPV, with the negative and positive as dependent variables and the 8 attribute variables as the independent variables.Figure4 shows the accuracy and the comprehensive optimal value of three inspection standards after each cycle of LASSO with Pima Indians Diabetes.Figure 4
Curves about three kinds of inspection standard.Figure4 shows that NPV are all improved when adding the weighting to LASSO’s solution; NPV is improved 5.16% when adding part_PCA, 5.58% when adding Independence Weight, and 5.1% when adding CRITIC. We can find that TPR is improved 13% when adding part_PCA, 14% when adding Independence Weight, and 13% when adding CRITIC for those methods changing the approach direction of algorithm. It is observed that ACC is improved 0.32% when adding part_PCA and Independence Weight. And there is no effect to ACC when adding CRITIC so the ACC keeps the original number.Therefore, the solutions of LASSO when adding the Independence Weight are optimal, followed by the CRITIC and part_PCA. This improved algorithm significantly increases the NPV and TPR ensuring the ACC is not reduced. The veracity of LASSO’s solutions is improved through changing approach direction when adding the weighting.Figure5 shows the SSR (Sum of Squared Residuals) of response and equiangular direction after each cycle of LASSO with Pima Indians Diabetes. It is obvious that the general trend of SSR completely remains consistent and the residual of optimal coefficient almost remains consistent when adding the different weight judgment. It will not change the advantages of LASSO when adding the weighting. If the optimal regression coefficient of predictor increases significantly, the SSR will change too. It increases 0.051 residual of LASSO solution when adding part_PCA, reduces 0.021 residual when adding CRITIC and Independence Weight. The results show that the final regression result is closer to the real response when adding CRITIC and Independence Weight.Figure 5
The curve about the SSR of response and equiangular direction.It can be found when synthesizing these three inspection standards that, adding the approach with multidimensional weight, the threshold value of the optimal solution is mainly reduced (except CRITIC), which means that the sum of absolute values of system’s regression coefficient is less than a smaller threshold; this algorithm meets the requirements in more extreme threshold range.Table1 shows the original regression coefficient β, βpart_PCA which adds part_PCA, βIW which adds Independence Weight, and βCRITIC which adds CRITIC.Table 1
Regression coefficient.
β1
β2
β3
β4
β5
β6
β7
β8
Β
1.7211
3.5409
−0.4862
−0.3665
−0.6504
2.2141
0.7071
−0.1810
Βpart_PCA
1.5300
3.5311
−0.4269
−0.3018
−0.6181
2.0409
0.6727
−0.0567
βIW
1.5050
3.5294
−0.4253
−0.2863
−0.6072
2.0066
0.6651
−0.0356
βCRITIC
1.6449
3.4482
−0.5019
−0.3705
−0.6302
2.1668
0.5471
−0.1597Table2 shows the difference and innovation of the multidimensional weight LARS, improved-LARS associated with part_PCA; Independence Weight and CRITIC act on β, changing the regression track and getting more accurate results.Table 2
Principles ofw in different algorithm.
Comparison ofw
Improved-LARS
w=wA=AAGA-11A
Multidimensional weight LARS
part_PCA
w=w∗centralization(part_PCAX)
Independence Weight
w=w∗centralization(IWX)
CRITIC
w=w∗centralization(CRITICX)


## 5. Conclusions

In this paper, a method considering the variables choosing and the approach direction of LARS algorithm is used to solve LASSO; we propose the LARS algorithm based on multidimensional weight to improve the veracity of LASSO’s solutions and keep the advantage of LASSO’s parameter estimation, which has stable regression coefficient, reduces the number of parameters, and has good consistency of parameter convergence. We verify the efficiency of the algorithm with Pima Indians Diabetes Data Set. The precision of the calculated weight was flawed for the greater dimension of individual, so we need to further optimize the embedding weight algorithm in the later studies, to improve the accuracy and precision of regression algorithm in approach variable and direction choosing which is changed by weighting.



---

*Source: 1736389-2017-05-04.xml*