# Error-Function-Based Penalized Quantile Regression in the Linear Mixed Model
###### Abstract  
We study a novel Doubly penalized ERror Function regularized Quantile Regression (DERF-QR) in this paper. This is a method of variable selection by ERror Function (ERF) regularization in the linear effects model. We introduce a two-stage iterative algorithm combining the iterative reweighted \(L_{1}\) approach and the alternating direction method of multipliers (ADMM) to estimate the model parameters. Numerical simulations show that by using this method we remove redundant variables effectively and obtain accurate coefficient estimations. Our method outperforms two existing penalized quantile regression methods in various error conditions by comparison. Finally, we apply the methodology in a financial dataset and showcase its practicality. alternating direction method of multipliers; error function; quantile regression; linear mixed-effects models 2020 IEEEexample: accepted: 28 2020 IEEEexample: accepted  
effectively in high-dimensional settings. Several regularization techniques have been developed to overcome this difficulty in existing work. [PERSON] introduced the Lasso method in [10] and simultaneously obtained the coefficient estimation and variable selection by \(L_{1}\) regularization. [PERSON] et al. proposed the non-convex SCAD penalty with Oracle properties in [11]. Based on these foundations, many regularization methods and extensions have been developed to handle more complex models. In quantile regression for the linear mixed-effects model, neglecting random effects can lead to biased estimates by failing to capture individual heterogeneity. Including too many random-effect covariates may cause overfitting, resulting in unstable estimates and reduced interpretability and generalizability. Thus, effective variable selection is crucial for simplifying models, eliminating redundancy, and improving computational efficiency. However, it is more challenging to estimate the coefficients and select the variables due to the interaction between fixed and random effects in the linear mixed-effects model. [PERSON] proposed a quantile regression approach in random-effects models and applied the Lasso penalty in random effects to eliminate redundant effects, but this method does not allow for selection among fixed-effect variables [12]. [PERSON] et al. proposed using \(L_{1}\) penalties to simultaneously select fixed and random effects in quantile regression for linear mixed-effects models, but the estimates obtained with the Lasso penalty are biased [13]. [PERSON] et al. used improved Cholesky decomposition for fixed- and random-effects estimation and variable selection in linear mixed-effects models [14]. [PERSON] et al. proposed a double-penalized likelihood method for finite mixture regression models, imposing penalties on both the mixing proportions and regression coefficients. This approach enables simultaneous parameter estimation and variable selection [15]. Inspired by the above proposition, we propose a Doubly penalized ERror Function regularized Quantile Regression (DERF-QR) method in a linear mixed-effects model in this paper. The innovations of this paper are as follows. First, we improve the existing doubly penalized quantile regression method by incorporating a novel regularization term based on the error function in a linear mixed-effects model [16]. This penalty function approximates the \(L_{0}\) penalty under certain conditions and substantially enhances the performance of coefficient estimation and variable selection. Second, we propose an efficient algorithm for this method by combining a two-step iterative method with the iterative reweighted \(L_{1}\) proximal to the alternating direction method of multipliers algorithm (IRW-pADMM) [17]. The structure of this paper is as follows. In Section 2, we introduce the properties of penalty functions and present an estimation method for the doubly penalized method. We give the Monte Carlo simulations to validate the performance of this method in coefficient estimation and variable selection in Section 3.