# Inverse Matrix Problem in Regression for High-Dimensional Data Sets

Authors: Namra Shakeel; Tahir Mehmood
Journal: Mathematical Problems in Engineering
 (2023)

---

## Abstract

Forhigh-dimensional chemometric data, the inverse matrixXtX−1 problem in regression models is a difficulty. Multicollinearity and identification result from the inverse matrix problem. The usage of the least absolute shrinkage and selection operator (LASSO) and partial least squares are two existing ways of dealing with the inverse matrix problem (PLS). For regressing the chemometric data sets, we used extended inverse and beta cube regression. The existing and proposed methods are compared over near-infrared spectra of biscuit dough and Raman spectra analysis of contents of polyunsaturated fatty acids (PUFA). For reliable estimation, Monte Carlo cross-validation has been used. The proposed methods outperform based on the root mean square error, indicating that cube regression and inverse regression are reliable and can be used for diverse high-dimensional data sets.


---

## Body

## 1. Introduction

Predictions have long been a key component of modern data science, whether in statistical analysis or machine learning. Modern technology allows for massive data expansion, yet this data frequently contains meaningless information, making prediction difficult. Researchers are employing novel methodologies and algorithms to extract information and build robust prediction models. Predictor variables that are either directly or indirectly related to other predictor factors are commonly included in such models. Multiple linear regression is widely employed in modern research fields such as chemometrics, econometrics, and bioinformatics [1]. Multiple linear regression models determine the relationship between multiple independent variables X1,X2,X3, … , Xp i.e., Xnxp and dependent variable Y. It can be written as Y=Xβ+μ, where β is the vector of model coefficients estimated by ordinary least square β^=XtX−1XtY that represents the relation between response and predictors while μ is the error term in the model.Certain traits or assumptions are associated with multivariate regression models for prediction. The correlation between predictor factors and the increased number of predictor variables are two of the characteristics. In presence of multicollinearity and a larger number of predictors, the inverse matrix that isXtX−1 does not hold. In chemometrics, most of the data sets faced such issues. For instance, near-infrared (NIR) and Raman spectroscopy calibrations of chemical components in food samples [2], the fatty acid composition and quantities of major constituents in a complex food model system [3] by using Raman spectroscopy. NIR is a vibrational spectroscopy method based on overtones and combinations of basic vibrational modes, whereas Raman is a vibrational spectroscopy technique based on fundamental stretching and deformation modes [4]. In comparison to Raman, the latter technique’s spectral bands are usually wider, giving NIR a poor chemical selectivity. The Raman and near-infrared spectroscopies are both appropriate for food analysis that is quick and useful. Here, measurements are feasible with fiber optics [5]. Both techniques conduct the qualitative, quantitative, and structural information about the samples [6]. For solving the inverse matrix problem penalized regression models are being used. Examples of the penalized regression models include ridged regression [7] and least absolute selection and shrinkage operator (LASSO) [8]. An alternative way is to use partial least squares [9] which utilizes the latent variables. Another dimension is to use the Moore–Penrose generalized inverse [10] for solving the inverse matrix issue. We have introduced the cube penalized regression and have reused the generalized inverse in multiple linear regression. Moore–Penrose generalized inverse is a method for solving singular matrices of high-dimensional regression data. Like other well-known LASSO and ridge regression methods, the concept of beta cube regression is established by considering the cube of the penalty parameter. These proposed methods are compared with reference methods i.e., LASSO and PLS over chemometric high-dimensional data [11–13]. Even though there is a vast literature on the topic of collinearity in linear regression, there is still a need to enhance the outcomes of traditional regression methods by incorporating new regression approaches. Newly discovered methods can save time by efficiently executing programs. Furthermore, these techniques can produce precise and effective outcomes.


## 2. Materials

Two data sets have been chosen for comparison. These data sets are then subdivided into each response, and all regression techniques will be run independently to each response variable with independent variables.


### 2.1. NIR (Near-Infrared) Spectra of Biscuit Dough

The data set contains 700 NIR spectra wavelengths (1100–2498 nm in 2 nm increments) that have been utilized as predictor variables. Fat, sugar, flour, and water yield percentages are arranged into four response variables. As a univariate response, the assessment of each answer is predicted independently [1, 14].Figure1 represents the flow for getting data from NIR spectra [15].Figure 1
This figure represents the NIR spectra of biscuit dough.


### 2.2. Raman Spectra Analysis of Contents of PUFA

The fatty acid composition across the 69 samples in the data set, when using Raman spectroscopy to extract specific chemical information from small components in meals [16]. The fatty acid content in this data set is provided as a percentage of the total sample weight and total fat content. As predictors, the samples produced 1096 wavelength variables. Figure 2 depicts the procedure for obtaining data from Raman spectra.Figure 2
This figure represents the Raman spectroscopy of fatty acids.


## 2.1. NIR (Near-Infrared) Spectra of Biscuit Dough

The data set contains 700 NIR spectra wavelengths (1100–2498 nm in 2 nm increments) that have been utilized as predictor variables. Fat, sugar, flour, and water yield percentages are arranged into four response variables. As a univariate response, the assessment of each answer is predicted independently [1, 14].Figure1 represents the flow for getting data from NIR spectra [15].Figure 1
This figure represents the NIR spectra of biscuit dough.


## 2.2. Raman Spectra Analysis of Contents of PUFA

The fatty acid composition across the 69 samples in the data set, when using Raman spectroscopy to extract specific chemical information from small components in meals [16]. The fatty acid content in this data set is provided as a percentage of the total sample weight and total fat content. As predictors, the samples produced 1096 wavelength variables. Figure 2 depicts the procedure for obtaining data from Raman spectra.Figure 2
This figure represents the Raman spectroscopy of fatty acids.


## 3. Methods

We have considered two reference methods LASSO and PLS while we have proposed two methods beta cube regression and generalized inverse regression.


### 3.1. Least Absolute Shrinkage and Selection Operator

The LASSO operator stands for least absolute shrinkage and selection operator. It has been introduced by Santosa and Symes in 1986 [17]. It came into prominence in 2006 by Tibshirani [18]. This is the type of regression model that executes variable selection as well as regularization to increase the estimation of accuracy. LASSO shrink β coefficients exactly equal to zero. LASSO is also called the L1 norm [19]. The LASSO estimator of β^ can be written as(1)β^LASSO=argminβ1n∑i=1nY−Xβ2subjectto∑j=1pβ1≤t,where t is the penalty on L1 norm. Equation 1 can also be written in the following form:(2)β^LASSO=argminβ1nY−Xβ22+λ∑j=1pβ1.Here,L1 norm and L2 norm are defined by β1=∑i=1pβi and β22=∑i=1pβi2. There is one-to-one correspondence in t and λ. This relation is because of duality. Thus, for every t≥ 0, there exists λ≥ 0 such that both problems play the same role [20]. The selection of λ can be done by cross validation. If λ = 0, LASSO estimator behaves similarly to the ordinary least square. If λ value rises, a number of nonzero β^ coefficients decreases and if λ value approaches to ∞ , then β^ becomes zero and LASSO provides null model [21]. Due to the nondifferentiable objective function, the LASSO does not provide a closed-form solution to the problem. Still, there is a possibility of obtaining closed-form by adding a soft threshold operator. That is how the soft-thresholding operator is defined for LASSO regression.(3)signλx=x+λ,ifx<−λ,0,ifx≤λ,x−λ,ifx>λ.Theβ^ can be written as(4)β^=1nXtY−λ2signβ,β^LASSO=1nXtYj+λ2,if1nXtYj<−λ2,0,if1nXtYj≤λ2,1nXtYj−λ2,if1nXtYj>λ2.


### 3.2. Partial Least Square (PLS) Regression

PLS is a substitute of a multiple linear regression model [22]. It is an iterative procedure. In PLS, the objective is to optimize the covariance between X and Y. The PLS regression can be written as β^PLS=WPtW−1Q. Here, P is the X-loading, Q is the Y-loading, and W is the loading weights. PLS even performs when data is noisy and missing.


### 3.3. Generalized Inverse Regression

ForA is a n×p matrix of rank ≤p, then A− of dimension p×n is called generalized inverse matrix AA−A=A.The generalized inverse is not unique in general, but it always exists. As the inverse of a matrix does not exist if its determinant becomes zero. The solution to finding the inverse of such a matrix is presented by an American mathematician, Moore in 1935, and later in 1955, a scientist named Penrose developed a Moore inverse in a different method [10]. Hence, is called Moore–Penrose Inverse. Although Moore–Penrose generalized inverse is not unique in the case of the nonsquare and singular matrix, it provides accurate results in minimum time for regression data sets. In the meanwhile, an author named Rao contributed a computational method of a singular matrix called pseudo inverse and used it to solve the least square theory to know estimators of linear equations [23]. Finding the system of linear equations is one of the most common uses of the generalized inverse. Let Y=Xβ then β=X−1Y, it is only true if X is invertible matrix. If X is nonsquare, singular matrix, then its Moore–Penrose generalized inverse can be represented as(5)β^ginv=ginvXtXXtY,where “ginv” presents the generalized inverse.


### 3.4. Beta Cube Regression

The concept of beta cube regression is basically taken from [24] by considering the L3 penalty in ordinary least squares. By considering some conditions, this regression method can be applied to real-life data sets. The regression coefficients are estimated using this approach by solving the following constraint, expressed as(6)β^Cube=argminβ∑i=1nY−Xβ2subjectto∑j=1pβ3≤t.Equation 6 can also be described as(7)β^Cube=argminβ∑i=1nY−Xβ2+λ∑j=1pβ3.Equating the derivative of equation 7, we will obtain the following equation:(8)β^Cube=2XtX+3λβ−12XtY.β vector comes within the β^ after the derivation of constraint. β vector in the previous equations can first be generated randomly by normal random distribution and against each value of λ; β^ is computed. Then, from all computed β^, one optimal β^ is selected. This optimal β^ is generated again and again by applying a loop in it to get the best and minimum β^.


## 3.1. Least Absolute Shrinkage and Selection Operator

The LASSO operator stands for least absolute shrinkage and selection operator. It has been introduced by Santosa and Symes in 1986 [17]. It came into prominence in 2006 by Tibshirani [18]. This is the type of regression model that executes variable selection as well as regularization to increase the estimation of accuracy. LASSO shrink β coefficients exactly equal to zero. LASSO is also called the L1 norm [19]. The LASSO estimator of β^ can be written as(1)β^LASSO=argminβ1n∑i=1nY−Xβ2subjectto∑j=1pβ1≤t,where t is the penalty on L1 norm. Equation 1 can also be written in the following form:(2)β^LASSO=argminβ1nY−Xβ22+λ∑j=1pβ1.Here,L1 norm and L2 norm are defined by β1=∑i=1pβi and β22=∑i=1pβi2. There is one-to-one correspondence in t and λ. This relation is because of duality. Thus, for every t≥ 0, there exists λ≥ 0 such that both problems play the same role [20]. The selection of λ can be done by cross validation. If λ = 0, LASSO estimator behaves similarly to the ordinary least square. If λ value rises, a number of nonzero β^ coefficients decreases and if λ value approaches to ∞ , then β^ becomes zero and LASSO provides null model [21]. Due to the nondifferentiable objective function, the LASSO does not provide a closed-form solution to the problem. Still, there is a possibility of obtaining closed-form by adding a soft threshold operator. That is how the soft-thresholding operator is defined for LASSO regression.(3)signλx=x+λ,ifx<−λ,0,ifx≤λ,x−λ,ifx>λ.Theβ^ can be written as(4)β^=1nXtY−λ2signβ,β^LASSO=1nXtYj+λ2,if1nXtYj<−λ2,0,if1nXtYj≤λ2,1nXtYj−λ2,if1nXtYj>λ2.


## 3.2. Partial Least Square (PLS) Regression

PLS is a substitute of a multiple linear regression model [22]. It is an iterative procedure. In PLS, the objective is to optimize the covariance between X and Y. The PLS regression can be written as β^PLS=WPtW−1Q. Here, P is the X-loading, Q is the Y-loading, and W is the loading weights. PLS even performs when data is noisy and missing.


## 3.3. Generalized Inverse Regression

ForA is a n×p matrix of rank ≤p, then A− of dimension p×n is called generalized inverse matrix AA−A=A.The generalized inverse is not unique in general, but it always exists. As the inverse of a matrix does not exist if its determinant becomes zero. The solution to finding the inverse of such a matrix is presented by an American mathematician, Moore in 1935, and later in 1955, a scientist named Penrose developed a Moore inverse in a different method [10]. Hence, is called Moore–Penrose Inverse. Although Moore–Penrose generalized inverse is not unique in the case of the nonsquare and singular matrix, it provides accurate results in minimum time for regression data sets. In the meanwhile, an author named Rao contributed a computational method of a singular matrix called pseudo inverse and used it to solve the least square theory to know estimators of linear equations [23]. Finding the system of linear equations is one of the most common uses of the generalized inverse. Let Y=Xβ then β=X−1Y, it is only true if X is invertible matrix. If X is nonsquare, singular matrix, then its Moore–Penrose generalized inverse can be represented as(5)β^ginv=ginvXtXXtY,where “ginv” presents the generalized inverse.


## 3.4. Beta Cube Regression

The concept of beta cube regression is basically taken from [24] by considering the L3 penalty in ordinary least squares. By considering some conditions, this regression method can be applied to real-life data sets. The regression coefficients are estimated using this approach by solving the following constraint, expressed as(6)β^Cube=argminβ∑i=1nY−Xβ2subjectto∑j=1pβ3≤t.Equation 6 can also be described as(7)β^Cube=argminβ∑i=1nY−Xβ2+λ∑j=1pβ3.Equating the derivative of equation 7, we will obtain the following equation:(8)β^Cube=2XtX+3λβ−12XtY.β vector comes within the β^ after the derivation of constraint. β vector in the previous equations can first be generated randomly by normal random distribution and against each value of λ; β^ is computed. Then, from all computed β^, one optimal β^ is selected. This optimal β^ is generated again and again by applying a loop in it to get the best and minimum β^.


## 4. Model Building and Comparison

For model building, the model parameters are required to tune. For this, we have used cross validation [25, 26]. Moreover, cross validation is used for the reliable comparison of reference regression models and proposed regression models.


### 4.1. Cross Validation

Cross validation (CV) is a fundamental approach for verifying the dependability of a regression model. The fundamental idea underlying CV is to assess a model’s prediction performance using data that was not used to develop the model. Typically, technique performance is evaluated using new data. To eliminate this reliance, the data set is divided into training and test sets numerous times. The model parameters for allλ possibilities using the training data are derived for each split, and the estimated parameters are evaluated on the corresponding test set. The penalty parameter that performs best (in specific respects) across all train sets is then selected [28].


### 4.2. Monte Carlo Cross Validation

Data splitting can be carried out by Monte Carlo cross validation. In Monte Carlo cross validation, each data point is tested arbitrary times and partitions can be possible many times, and this method's result is high bias but low in variance. It splits the training data at random (maybe 70–30 percent, or 60–40 percent). Data are split randomly to avoid overestimating or underestimating the results [29].


### 4.3. Selection of Penalty Parameter

The penalized term in regression is determined by a tuning parameterλ, also known as a penalty parameter. When data values are shrunk towards a central point, such as the mean, it refers to the degree of shrinking that happens.


### 4.4. Performance Estimation

We employed root mean square error (RMSE), a method for measuring prediction quality, to estimate the performance of regression models. Using Euclidean distance, it predicts how far estimations differ from actual values.(9)RMSE=∑i=1nYi−Y^i2n,where n is sample size, Yi is the ith actual response and Y^i is the ith predicted response [30].


## 4.1. Cross Validation

Cross validation (CV) is a fundamental approach for verifying the dependability of a regression model. The fundamental idea underlying CV is to assess a model’s prediction performance using data that was not used to develop the model. Typically, technique performance is evaluated using new data. To eliminate this reliance, the data set is divided into training and test sets numerous times. The model parameters for allλ possibilities using the training data are derived for each split, and the estimated parameters are evaluated on the corresponding test set. The penalty parameter that performs best (in specific respects) across all train sets is then selected [28].


## 4.2. Monte Carlo Cross Validation

Data splitting can be carried out by Monte Carlo cross validation. In Monte Carlo cross validation, each data point is tested arbitrary times and partitions can be possible many times, and this method's result is high bias but low in variance. It splits the training data at random (maybe 70–30 percent, or 60–40 percent). Data are split randomly to avoid overestimating or underestimating the results [29].


## 4.3. Selection of Penalty Parameter

The penalized term in regression is determined by a tuning parameterλ, also known as a penalty parameter. When data values are shrunk towards a central point, such as the mean, it refers to the degree of shrinking that happens.


## 4.4. Performance Estimation

We employed root mean square error (RMSE), a method for measuring prediction quality, to estimate the performance of regression models. Using Euclidean distance, it predicts how far estimations differ from actual values.(9)RMSE=∑i=1nYi−Y^i2n,where n is sample size, Yi is the ith actual response and Y^i is the ith predicted response [30].


## 5. Results and Discussion

We have considered 2 data sets NIR spectra of biscuit dough and Raman spectra analysis of contents of PUFFA for modeling and comparison purposes. The descriptive summary of these data sets is presented in Tables1 and 2. Notably, the NIR data set has 4 response variables fat, flour, sucrose, and water while the Raman data set has 2 response variables weight and fat.Table 1
A descriptive description of the NIR biscuit dough data is provided.
ResponsesFatFlourSucroseWaterMean18.3116.5948.9814.19Variance3.8715.247.462.20Training data50505050Testing data22222222Table 2
The content of polyunsaturated fatty acids is described in detail.
ResponsesSample weightFat contentMean4.3933.64Variance7.89251.85Training data4848Testing data2121In Figure3, the suggested technique beta cube and generalized inverse approaches are compared to reference methods PLS and LASSO for prediction capabilities. The fact that the RMSE of the generalized inverse is almost equal to zero for cookie dough components demonstrates that it works well for each data set. In comparison to PLS, the beta cube performs best, with an RMSE value between 0 and 0.25. LASSO has the lowest RMSE among all approaches for fat and flour data sets, but it does not provide adequate results for sugar and water data sets. For all data sets of biscuit dough, in Figure 4 the generalized inverse is performing well for threshold = 0. This means for threshold = 0, the models are including all the β^ coefficients. Beta Cube is picking different penalty parameters between 0.1 and 1 and thus getting the mean value of λ around 0.5. This means the model is picking approximately half of the relevant β^ coefficients. The PLS model is considering optimal components in between the range 0.1 to 0.8 and providing a mean value of thresholds around 0.45. LASSO is predicting suitable results for threshold values equal to 0.1 and 0.2 for fat and flour data sets. For sucrose and water data sets, LASSO is not performing well even at a threshold value of 1.Figure 3
The RMSE of components of biscuit dough for each prediction method is presented.Figure 4
The distribution of threshold and components of biscuit dough is presented for each prediction method.Other methodologies’ prediction accuracy is determined using PLS and LASSO algorithms. The RMSE of the generalized inverse for fatty acid is almost equal to zero, suggesting that it performs best for each data set, as shown in 5. In comparison to PLS, the beta cube performs best, with an RMSE value between 0 and 0.15. When compared to all other techniques, LASSO delivers the lowest RMSE for these data sets.For all data sets of fatty acid, in Figures5, the generalized inverse is performing well for threshold = 0. This means for threshold = 0, the models are including all the β^ coefficients. Beta Cube is picking penalty parameters 0.1 and 0.2. In PLS, the models are considering optimal components in between the range of 0.1 to 0.75. For these data sets, LASSO is not performing well even at the threshold value of 1 (Figure 6).Figure 5
For each prediction approach, the distribution of fatty acid thresholds as a percentage of total sample weight and total fat content is shown.Figure 6
The distribution of RMSE of fatty acids as a percentage of total sample weight and total fat content is presented for each prediction method.


## 6. Limitations

Only the used data set makes the suggested results valid. Furthermore, the suggested approach is not robust and cannot be guaranteed to perform well in the presence of outliers.


## 7. Conclusions

In terms of high-dimensional chemometric data set prediction capabilities, the presented approaches outperform. Among the data sets modelled are NIR of biscuit dough and Raman of polyunsaturated fatty acids. In addition, the suggested extended inverse and Beta Cube regression techniques yield more consistent results. These regression methods are designed for data sets with a large number of independent variables relative to the sample size. Furthermore, they are effective for multicollinear data sets and identification difficulties. Future studies should delve into the extra features and many uses of the approaches presented.



---

*Source: 2308541-2023-02-23.xml*