# Implementation Aspects in Regularized Structural Equation Models
## 2 Estimation of Regularized Structural Equation Models  
We now describe regularized maximum likelihood (ML) estimation approach for multiple-group SEMs. Note that some identification constraints must be imposed to estimate the covariance structure model (5) (see [2]). For modeling multivariate normally distributed data without missing values, the empirical mean vector \(\overline{x}\) and the empirical covariance matrix \(\mathbf{\varsigma}\) are sufficient statistics for estimating \(\mu\) and \(\mathbf{\Sigma}\). Hence, they are also sufficient statistics for the parameter vector \(\theta=(\theta_{1},\dots,\theta_{K})\). Now, assume that there exist \(G\) groups with sample sizes \(N_{g}\) and empirical means \(\overline{\pi}_{g}\) and covariance matrices \(S_{g}\) (\(g=1,\ldots,G\)). The population mean vectors and covariance matrices are denoted by \(\mu_{g}\) and \(\Sigma_{g}\), respectively. The model-implied mean vectors and covariance matrices are denoted by \(\mu_{g}(\theta)\) and \(\Sigma_{g}(\theta)\), respectively. Note that the parameter vector \(\theta\) does not have an index \(g\) to indicate that there can be common and unique parameters across groups. In a multiple-group CFA, equal factor loadings and item intercepts across groups are frequently imposed (i.e., measurement invariance holds). Let \(\xi_{g}=(\overline{x}_{g},S_{g})\) be the sufficient statistics of group \(g\). The combined vector containing all sufficient statistics for the multiple-group SEM is denoted by \(\xi=(\xi_{1},\ldots,\xi_{G})\). The negative log-likelihood function \(l\) for the multiple-group SEM (see [2; 4]) is given by  
\[l(\theta;\xi)=\sum_{g=1}^{G}\frac{N_{g}}{2}\Big{(}-I\log(2\pi)+\log|\Sigma_{g} (\theta)|+\text{tr}(S_{g}\Sigma_{g}(\theta)^{-1})+(\overline{x}_{g}-\mu_{g}( \theta))^{\top}\Sigma_{g}(\theta)^{-1}(\overline{x}_{g}-\mu_{g}(\theta))\Big{)}. \tag{6}\]  
In empirical applications, the model-implied mean vectors covariance matrices will frequently be misspecified [18; 19; 20], and \(\theta\) can be interpreted as a pseudo-true parameter defined as the maximizer of the fitting function \(l\) in (6). In regularized SEM estimation, a penalty function is added to the log-likelihood function that imposes some sparsity assumption on a subset of model parameters [11; 12; 21]. Frequently, the penalty function \(\mathcal{P}\) is non-differentiable in order to impose sparsity. We define a known parameter \(\delta_{k}\in\{0,1\}\) for all parameters \(\theta_{k}\), where \(\delta_{k}=1\) indicates that for the \(k\)th entry \(\theta_{k}\) in \(\theta\), a penalty function is applied. The penalized log-likelihood function is given by  
\[l_{\text{pen}}(\theta,\lambda;\xi)=l(\theta;\xi)+N^{*}\sum_{k=1}^{K}\delta_{k }\mathcal{P}(|\theta_{k}|^{p},\lambda)\, \tag{7}\]  
where \(\lambda\) is a nonnegative regularization parameter, and \(N^{*}\) a scaling factor that frequently equals the total sample size \(N=\sum_{g=1}^{G}N_{g}\). The power \(p\) in the penalty function usually takes values in \([0,2]\). Most of the literature on regularized SEMs employs the power \(p=1\), but \(p=0.5\) has been recently suggested [16] (but see also [22]). The minimizer of \(l(\theta)\) is denoted as the regularized (or penalized) ML estimate. We now discuss typical choices of the penalty function \(\mathcal{P}\). For a scalar parameter \(x\), the least absolute shrinkage and selection operator (LASSO) penalty is a popular penalty function used in regularization [23], and it is defined as  
\[\mathcal{P}_{\text{LASSO}}(x,\lambda)=\lambda|x|\, \tag{8}\]  
where \(\lambda\) is a nonnegative regularization parameter that controls the extent of sparsity in the obtained parameter estimate. Note that the LASSO penalty function combined with \(p=0.5\) is equivalent to the alignment loss function (ALF [16]):  
\[\mathcal{P}_{\text{ALF}}(x,\lambda)=\lambda\sqrt{|x|}. \tag{9}\]  
It is known that the LASSO penalty introduces bias in estimated parameters.