# Coordinate Descent for Variance-Component Models
## 5 Conclusions  
The MLE solution for variance-component models requires the optimization of a non-convex log-likelihood function. In this paper, we showed that a basic implementation of the cyclic CD algorithm is computationally expensive to run and is not amenable to traditional convergence analysis. To remedy this, we proposed a novel parameter-expanded CD (PX-CD) algorithm, which is both computationally faster and also subject to theoretical guarantees. PX-CD optimizes a higher-dimensional surrogate function that attains a coordinate-wise minimum with respect to each of the variance component parameters. The extra speed is derived from the fact that required quantities (such as first and second-order derivatives) are evaluated via the conjugate-gradient algorithm. Additionally, we propose an alternative updating regime called PXI-CD, where the expanded block parameters are updated immediately after each coordinate update. This new updating regime requires more computation for each iteration as compared to PX-CD. However, numerically, we observed that, for large-scale models, where the number of variance components \(m+1\) is large and \(\mathbf{V}_{i}\) are full-rank, the number of iterations needed to converge greatly offsets the additional computational cost per coordinate update cycle. Our numerical experiments suggest that PX-CD and PXI-CD outperform the best current alternative--the MM algorithm. When the number of variance components \(m\) is large, we observed that PXI-CD was significantly faster than the MM algorithm and tended to converge to more optimal stationary points. A potential extension of this work is to apply parameter-expanded CD algorithms to the multivariate-response variance-component model. Instead of the univariate response, one considers the multivariate response model with a \(n\times d\) response matrix \(\mathbf{Y}\). In this setup, \(\mathbb{E}[\mathbf{Y}]=\mathbf{X}\mathbf{B}\) where \(\mathbf{B}\) is a \(p\times d\) matrix. The \(nd\times nd\) covariance matrix is of the form  
\[\mathbf{\Omega}=\mathrm{cov}(\mathrm{vec}(\mathbf{Y}))=\sum_{i=0}^{m} \mathbf{\Gamma}_{i}\otimes\mathbf{V}_{i},\]  
where \(\mathbf{\Gamma}_{i}\) are unknown \(d\times d\) variance components and \(\mathbf{V}_{i}\) are the known \(n\times n\) covariance matrices. The challenging aspect of this problem is that each optimization with respect to a parameter \(\mathbf{\Gamma}_{i}\) is not univariate but is rather a search over positive semi-definite matrices--itself, a difficult optimization problem. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & \(m\) & \(r\) & **Iterations** & **Time (s)** & **Objective** \\ \hline PXI-CD & 100 & 102 & 95.1 (9.9) & 23.1 (2.66) & \(-43.6\) \\ MM & & & 1580.2 (194.76) & 108.7 (15.37) & \(-43.6\) \\ PXI-CD & 200 & 51 & 180.0 (13.41) & 58.8 (4.4) & \(-82.5\) \\ MM & & & 2797.8 (401.1) & 466.0 (88.95) & \(-82.6\) \\ PXI-CD & 500 & 20 & 397.8 (53.73) & 258.1 (34.56) & \(-100.3\) \\ MM & & & 2700.8 (366.57) & 1055.4 (179.97) & \(-106.6\) \\ PXI-CD & 1000 & 10 & 434.7 (35.7) & 507.5 (39.91) & \(-82.1\) \\ MM & & & 3004.8 (343.61) & 2329.4 (296.37) & \(-91.2\) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The running times (s)—mouse data. **Author Contributions:** Conceptualization, A.M., S.M. and Z.B. ; methodology, A.M., S.M. and Z.B. ; software, A.M.; formal analysis, A.M., S.M. and Z.B. data curation, A.M.; writing--original draft preparation, A.M.; writing--review and editing, A.M., S.M. and Z.B. All authors have read and agreed to the published version of the manuscript. **Funding:** This research received no external funding. **Data Availability Statement:** Publicly available datasets were analyzed in this study. This data can be found here: [[https://openmended.github.io/SnpArrays.jl/latest](https://openmended.github.io/SnpArrays.jl/latest)]([https://openmended.github.io/SnpArrays.jl/latest](https://openmended.github.io/SnpArrays.jl/latest)) (accessed on 1 July 2022) mouse single nucleotide polymorphism (SNP) array data set.