# Implementation Aspects in Regularized Structural Equation Models
## 2 Estimation of Regularized Structural Equation Models
To circumvent this issue, the smoothly clipped absolute deviation (SCAD [24]) penalty has been proposed. \[\mathcal{P}_{\text{SCAD}}(x,\lambda)=\left\{ \begin{array}{ll}\lambda|x|& \text{if }|x|<\lambda\\ -(x^{2}-2a\lambda|x|^{2}+\lambda^{2})(2(a-1))^{-1}&\text{if }\lambda\leq|x|\leq a \lambda\\ (a+1)\lambda^{2}&\text{if }|x|>a\lambda\end{array} \right. \tag{10}\]  
In many studies, the recommended value of \(a=3.7\) (see [24]) has been adopted (e.g., [25; 26]). The SCAD penalty retains the penalization rate and the induced bias of the lasso for model parameters close to zero, but continuously relaxes the rate of penalization as the absolute value of the model parameters increases. Note that \(\mathcal{P}_{\text{SCAD}}\) has the property of the lasso penalty around zero, but has zero derivatives for \(x\) values strongly differing from zero. Note that the minimizer of \(l_{\rm pen}\) is a function of the fixed regularization parameter \(\lambda\); that is,  
\[\widetilde{\theta}(\lambda)=\operatorname*{arg\,min}_{\Theta}\,l_{\rm pen}( \theta,\lambda;\xi). \tag{11}\]  
Hence, the parameter estimate \(\widetilde{\theta}(\lambda)\) of \(\theta\) depends on a parameter that must be known. To circumvent this issue, the regularized SEM can be repeatedly estimated on a finite grid of regularization parameters \(\lambda\) (e.g., on an equidistant grid between 0.01 and 1.00 with increments of 0.01). The Bayesian information criterion (BIC), defined by \(\text{BIC}=2l(\theta,\xi)+\log(N)H\), where \(H\) denotes the number of parameters, can be used to select an optimal regularization parameter. Because the minimization of BIC is equivalent to the minimization of BIC/2, the final parameter estimate \(\widehat{\theta}\) is obtained as  
\[\widehat{\theta}=\widetilde{\theta}(\widehat{\lambda})\text{ with }\widehat{ \lambda}=\operatorname*{arg\,min}_{\lambda}\left\{l(\widetilde{\theta}( \lambda),\xi)+\frac{\log(N)}{2}\left(\sum_{k=1}^{K}\delta_{k}\chi_{0}( \widetilde{\theta}_{k}(\lambda))\right)\right\}\,, \tag{12}\]  
where the function \(\chi_{z}\) as an indicator whether \(|x|\) is larger than \(z\) for any \(z\geq 0\):  
\[\chi_{z}(x)=\left\{ \begin{array}{ll}1&\text{if }|x|>z\\ 0&\text{if }|x|\leq z\end{array} \right.\,. \tag{13}\]  
In particular, the quantity \(\sum_{k=1}^{K}\delta_{k}\chi_{0}(\widetilde{\theta}_{k}(\lambda))\) in (12) counts the number of parameter estimates \(\widetilde{\delta}_{k}(\lambda)\) for \(k=1,\ldots,K\) for which the penalty function is applied (i.e., \(\delta_{k}=1\)) and which differ from 0. Note that the minimization of the BIC depends on two components. First, the model fit can be improved by minimizing the negative log-likelihood function while freely estimating more parameters. Second, sparse models are preferred in BIC minimization because the second term in (12) minimizes the number of estimated model parameters that are different from zero. Hence, there is always a trade-off between model fit improvement and parsimonious model estimation. It should be emphasized that BIC is frequently preferred over the Akaike information criterion (AIC) in regularized estimation [11; 27]. In typical sample sizes, BIC imposes stronger penalization of the number of estimated parameters than AIC. In fact, alternative information criteria with even stronger penalization are discussed in regularization [25; 28; 29]. Regularized estimation of single-group and multiple-group SEMs are widespread in the methodological literature [11; 21; 30; 31; 32; 33; 34]. In these applications, cross-loadings, entries in the covariance matrix of residuals, or the vector of item intercepts are regularized. Applying regularized estimation in SEMs allows for flexible yet parsimonious model specifications.