# Implementation Aspects in Regularized Structural Equation Models
## 2 Estimation of Regularized Structural Equation Models
### Regularized SEM Estimation Approaches  
Regularized estimation of (11) typically involves a non-differentiable optimization function because the penalty function is non-differentiable. In [14], exact and approximate solutions are distinguished for minimizing the penalized log-likelihood function \(l_{\rm pen}\) in (11).  
Exact estimation operates on the non-differentiable penalized log-likelihood function. In coordinate descent (CD), the penalized log-likelihood function is cyclically minimized across all entries of the parameter vector \(\theta\) (see [23]). If the function \(l_{\rm pen}\) is minimized in the \(k\)th coordinate \(\theta_{k}\) of \(\Theta\), the remaining entries in \(\theta\) are fixed to the estimate from the previous iteration. This coordinate-wise estimation can be repeated for all parameters and iterated until convergence is reached. The advantage of CD when using the LASSO or the SCAD penalty is that regularized parameters are exactly zero, while nonregularized parameters differ from zero. Hence, a sparse estimate \(\theta\) is obtained. However, CD can be computationally demanding [14]. In addition, it can also not be generally ensured that a global minimum (instead of a local minimum) is found with CD estimation.  
Alternatively, the non-differentiable optimization function can be replaced by a differentiable one [12; 35; 36; 37; 38]. The penalty function involves the non-differentiable absolute value function that can be replaced by  
\[|x|\approx(x^{2}+\varepsilon)^{1/2}\text{ or more generally }|x|^{p}\approx(x^{2}+ \varepsilon)^{p/2} \tag{14}\]  
for a sufficiently small \(\varepsilon>0\), such as \(\varepsilon=10^{-3}\) or \(\varepsilon=10^{-4}\). Fortunately, general-purpose optimizers that rely on derivatives can be relied on when using differentiable approximations based on the penalized log-likelihood function. These optimizers are widely available in software and are reliable if good starting values are available. The disadvantage of the differentiable approximation (DA) approach is that there are no estimated parameters that are exactly zero. To determine a parameter estimate \(\widehat{\theta}\), a threshold \(\tau\)[14] must be specified that defines which small parameter entries should be set to zero. Hence, the final parameter estimate in DA is given by  
\[\widehat{\theta}=\widehat{\theta}(\widehat{\lambda})\text{ with }\widehat{ \lambda}=\operatorname*{arg\,min}_{\lambda}\left\{l(\widehat{\theta}(\lambda ),\xi)+\frac{\log(N)}{2}\left(\sum_{k=1}^{K}\delta_{k}\chi_{\tau}(\widehat{ \theta}_{k}(\lambda))\right)\right\}\,. \tag{15}\]  
Note that the threshold \(\tau\) is typically a function of \(\varepsilon\)[14], and \(\tau\) should be (much) larger than \(\varepsilon\). In general, the penalized ML estimate based on DA defined in (15) relies on two tuning parameters, \(\varepsilon\) and \(\tau\), that must be properly chosen. [PERSON] et al. [14] argue that there is typically not enough knowledge on how to choose these tuning parameters in practical applications. Therefore, they generally prefer CD over DA.