# Coordinate Descent for Variance-Component Models
## 3 Parameter-Expanded CD
### Updating Regime
We can rewrite the optimization problem (3) in the penalized form:\[f(\mathbf{x}):=f_{0}(\mathbf{x})+\sum_{k=1}^{m+1}f_{k}(x_{k})+f_{m+2}(\mathbf{x}_{m+2}),\]  
where \(f_{0}(\mathbf{x}):=H(\mathbf{\Omega},\mathbf{C})\) and  
\[f_{k}(x):= \begin{cases}0,&x\geq 0\\ \infty,&x<0\end{cases} ,\quad k=1,\ldots,m+1,\quad f_{m+2}(\mathbf{x}):= \begin{cases} 0,&\mathbf{x}\in[\mathbf{0},\mathbf{\infty}]\\ \infty,&\mathbf{x}\
ot\in[\mathbf{0},\mathbf{\infty})\end{cases} .\]  
We can then apply the results in [18], which state that every limit point of \(\{\mathbf{\gamma}^{(t)}\}_{t\geq 0}\) is a coordinate-wise minimum provided that:  
1. Each function \(x_{k}\mapsto f(\mathbf{x})\), \(k=1,\ldots,m+1\) and \(\mathbf{x}_{m+2}\mapsto f(\mathbf{x})\) has a unique minimum. For \(k=1,\ldots,m+1\) this has already been verified in Lemma 1. For \(\mathbf{x}_{m+2}\mapsto f(\mathbf{x})\), we simply recall that \(H(\mathbf{\Omega},\mathbf{C})\geq G(\mathbf{\Omega})\) with equality achieved if and only if \(\mathbf{C}=\mathbf{\Omega}\), or equivalently \(\mathbf{x}_{m+2}=(x_{1},\ldots,x_{m+1})\). 2. Each \(f_{k},k=1,\ldots,m+2\) is lower semi-continuous. This is clearly true for \(k=1,\ldots,m+1\) because, at the point of discontinuity, we have \(\liminf_{x\to 0}f_{k}(x)\geq f_{k}(0)=0\). For \(f_{m+2}\), we simply check that \(\{\mathbf{x}:f_{m+2}(\mathbf{x})\leq c\}\) for a given \(c\in\mathbb{R}\) is a closed set. 3. The domain of \(f_{0}\) is a Cartesian product and \(f_{0}\) is continuous on its domain. Clearly, the domain is the \(2(m+1)\) Cartesian product \(\mathcal{D}:=[0,\infty)\times\cdots\times[0,\infty)\) and \(f_{0}\) is continuous on its effective domain \(\{\mathbf{x}\in\mathcal{D}:f(\mathbf{x})<\infty\}\). 4. The updating rule is _essentially cyclic_--that is, there exists a constant \(T\geq m+2\) such that every block in \((x_{1},\ldots,x_{m+1},\mathbf{x}_{m+2})\) is updated at least once between the \(r\)-th iteration and the \((r+T-1)\)-th iteration for all \(r\). In our case, each block is updated at least once in one iteration of Algorithm 2 so that we satisfy the essentially cyclic condition. In the PXI-CD, we actually update \((m+1)\) times the block \(\mathbf{x}_{m+2}\). Thus, we can conclude by Proposition 5.1 in [18] that either \(f_{0}(\mathbf{\gamma}^{(t)})\to-\infty\) or the limit points of \(\{\mathbf{\gamma}^{(t)}\}_{t\geq 0}\) are coordinate-wise minima of \(H(\mathbf{\Omega},\mathbf{C})\). If we further assume that \(\mathbf{y}\
ot\in\operatorname{span}\{\mathbf{\mathrm{V}}_{1},\ldots,\mathbf{\mathrm{V}}_ {m}\}<n\), then we can show that set \(\{\mathbf{\gamma}\geq 0:f_{0}(\mathbf{\gamma})\leq f_{0}(\mathbf{\gamma}^{(0)})\}\) is compact, which ensures that the sequence \(\{\mathbf{\gamma}^{(t)}\}_{t\geq 0}\) is bounded and rules out the possibility that \(\lim_{t}f_{0}(\mathbf{\gamma}^{(t)})=-\infty\). To see that this is the case, note that \(G(\mathbf{\Omega})\) provides a lower bound to \(H(\mathbf{\Omega},\mathbf{C})\), and this is sufficient to show that \(\{\mathbf{\gamma}\geq 0:G(\sum_{k}\gamma_{k}\mathbf{\mathrm{V}}_{k})\leq c\}\) is compact for any \(c\in\mathbb{R}\) under the assumption that \(\mathbf{\mathrm{V}}_{0}:=\mathbf{\mathrm{I}}\) and \(\mathbf{y}\
ot\in\operatorname{span}\{\mathbf{\mathrm{V}}_{1},\ldots,\mathbf{\mathrm{V}}_ {m}\}<n\). However, these are precisely the conditions of Lemma 3 in [9], which ensure that \(\{\mathbf{\gamma}\geq 0:L(\mathbf{\gamma})\geq c\}\) is compact for a likelihood \(L(\mathbf{\gamma}):=-G(\sum_{k}\gamma_{k}\mathbf{\mathrm{V}}_{k})\). Finally, note that since we update the entire block \(\tilde{\mathbf{\gamma}}\) simultaneously (in both PXI-CD and PXI-CD), this means that a coordinate-wise minimum of \(H(\mathbf{\Omega},\mathbf{C})\) is also a coordinate-wise minimum for \(G(\mathbf{\Omega})\). We again emphasize that, as with all the competitor methods, the theorem does not guarantee convergence of the coordinate-descent sequence \(\{\mathbf{\gamma}^{(t)}\}_{t\geq 0}\) or that the convergence will be to a local minimum.