SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jian-Feng Cai $^{1,2}$ , José Vinícius de M. Cardoso $^{1}$ , Daniel P. Palomar $^{1}$ , Jiaxi Ying $^{1,2*}$
Hong Kong University of Science and Technology<sup>1</sup>
HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute
We study the problem of estimating precision matrices in Gaussian distributions that are multivariate totally positive of order two  $(\mathrm{MTP}_2)$ . The precision matrix in such a distribution is an  $M$ -matrix. This problem can be formulated as a sign-constrained log-determinant program. Current algorithms are designed using the block coordinate descent method or the proximal point algorithm, which becomes computationally challenging in high-dimensional cases due to the requirement to solve numerous nonnegative quadratic programs or large-scale linear systems. To address this issue, we propose a novel algorithm based on the two-metric projection method, incorporating a carefully designed search direction and variable partitioning scheme. Our algorithm substantially reduces computational complexity, and its theoretical convergence is established. Experimental results on synthetic and real-world datasets demonstrate that our proposed algorithm provides a significant improvement in computational efficiency compared to the state-of-the-art methods.
We consider the problem of estimating the precision matrix (i.e., inverse covariance matrix) in a multivariate Gaussian distribution, where all the off-diagonal elements of the precision matrix are nonpositive. The resulting precision matrix is a symmetric  $M$ -matrix. Such property is also known as multivariate totally positive of order two  $(\mathrm{MTP}_2)$  [1]. For ease of presentation, we call the nonpositivity constraints on the off-diagonal elements of the precision matrix as  $\mathrm{MTP}_2$  constraints. This model arises in a variety of applications such as taxonomic reasoning [2], graph signal processing [3], factor analysis in psychometrics [4], and financial markets [5].
Estimating precision matrices under  $\mathrm{MTP}_2$  constraints is an active research topic. Recent results in [2, 4, 6] show that  $\mathrm{MTP}_2$  constraints lead to a drastic reduction on the number of observations required for the maximum likelihood estimator (MLE) to exist in Gaussian distributions and Ising models. This advantage is crucial in high-dimensional regimes with limited observations. Growing interest in estimating precision matrices under  $\mathrm{MTP}_2$  constraints is seen in graph signal processing [7-9]. A precision matrix fulfilling  $\mathrm{MTP}_2$  constraints can be regarded as a generalized graph Laplacian, where eigenvalues and eigenvectors are interpreted as spectral frequencies and Fourier bases, facilitating the computation of graph Fourier transform [10]. The  $\mathrm{MTP}_2$  property has also been studied in portfolio allocation [5] and structure recovery [11, 12].
Estimating precision matrices under  $\mathrm{MTP}_2$  constraints can be formulated as a sign-constrained log-determinant program. Existing algorithms for solving this problem, such as the block coordinate descent (BCD) [2, 3, 8] and proximal point algorithm (PPA) [9], are efficient for low-dimensional
problems. However, they become time-consuming in high-dimensional scenarios due to the necessity of solving numerous nonnegative quadratic programs or large-scale linear systems. An alternative is the gradient projection method [13], which offers computational efficiency per iteration. Nevertheless, it often grapples with slow convergence rates in high-dimensional cases. Thus, there is a demand for efficient and scalable algorithms for precision matrix estimation under  $\mathrm{MTP}_2$  constraints.
In this paper, we propose a fast projected Newton-like algorithm for estimating precision matrices under  $\mathrm{MTP}_2$  constraints. While second-order algorithms generally require fewer iterations than first-order methods, they often encounter computational challenges due to the necessity of computing a large (approximate) Hessian matrix inverse or equivalently solving the linear system. Our main contributions in this paper are threefold:
1. Our proposed algorithm, rooted in the two-metric projection method, stands apart from current BCD and PPA approaches [2, 3, 8, 9] for solving the target problem. Utilizing a well-designed search direction and variable partitioning scheme, our algorithm avoids the need to solve nonnegative quadratic programs or linear systems, yielding a significant computational reduction compared to BCD and PPA algorithms. As a second-order method, our algorithm maintains the same per-iteration computational complexity as the gradient projection method.
2. We establish that our algorithm converges to the minimizer of the target problem. Furthermore, under a mild assumption, we prove the convergence of the set of free variables to the support of the minimizer within finite iterations and provide the convergence rate of our algorithm.
3. Numerical experiments on both synthetic and real-world datasets provide compelling evidence that our algorithm converges to the minimizer considerably faster than the compared methods. We apply the proposed method to financial time-series data and observe significant performance in terms of modularity value on the learned financial networks.
Notation: Lower case bold letters denote vectors and upper case bold letters denote matrices. Both  $X_{ij}$  and  $[X]_{ij}$  denote the  $(i,j)$ -th entry of  $X$ .  $[p]$  denotes the set  $\{1,\dots,p\}$ , and  $[p]^2$  denotes the set  $\{1,\dots,p\} \times \{1,\dots,p\}$ . Let  $\otimes$  be the Kronecker product and  $\odot$  be the entry-wise product.  $\operatorname{supp}(X) = \{(i,j) \in [p]^2 | X_{ij} \neq 0\}$ .  $\| X \|_{\max} = \max_{i,j} |X_{ij}|$  and  $\| X \|_{\min} = \min_{i,j} |X_{ij}|$ .  $\mathbb{S}_+^p$  and  $\mathbb{S}_{++}^p$  denote the sets of symmetric positive semi-definite and positive definite matrices with dimension  $p \times p$ .  $\operatorname{vec}(X)$  and  $X^\top$  denote the vectorized version and transpose of  $X$ .
2 Problem Formulation and Related Work
In this section, we first introduce the problem formulation, then present related works.
Let  $\pmb{y} = (y_{1},\dots ,y_{p})$  be a  $p$ -dimensional random vector following  $\pmb{y}\sim \mathcal{N}(\pmb {0},\pmb {\Sigma})$ , where  $\pmb{\Sigma}$  is the covariance matrix. We focus on the problem of estimating the precision matrix  $\Theta \coloneqq \pmb{\Sigma}^{-1}$  given  $n$  i.i.d. observations  $\pmb{y}^{(1)},\ldots ,\pmb{y}^{(n)}$ . Let  $\pmb{S} = \frac{1}{n}\sum_{i = 1}^{n}\pmb{y}^{(i)}\bigl (\pmb{y}^{(i)}\bigr)^{\top}$  be the sample covariance matrix. Throughout the paper, the sample covariance matrix is assumed to have strictly positive diagonal elements, which holds with probability one. This is because some diagonal element  $S_{jj}$  is zero if and only if the  $j$ -th element of  $\pmb{y}^{(i)}$  must be zero for every  $i\in [p]$ , which holds with probability zero.
We consider solving the following sign-constrained log-determinant program:
$$
\boldsymbol {X} ^ {\star} := \underset {\boldsymbol {X} \in \mathcal {M} ^ {p}} {\arg \min } - \log \det  (\boldsymbol {X}) + \operatorname {t r} (\boldsymbol {X} \boldsymbol {S}) + \sum_ {i \neq j} \lambda_ {i j} | X _ {i j} |, \tag {1}
$$
$$
\text {s u b j e c t} X _ {i j} = 0, \forall (i, j) \in \mathcal {E},
$$
where  $\lambda_{ij}$  is the regularization parameter,  $\mathcal{E}$  is the disconnectivity set with each node pair forced to disconnect, and  $\mathcal{M}^p$  is the set of all  $p$ -dimensional, symmetric, non-singular  $M$ -matrices defined by
$$
\mathcal {M} ^ {p} := \left\{\boldsymbol {X} \in \mathbb {S} _ {+ +} ^ {p} \mid X _ {i j} \leq 0, \forall i \neq j \right\}. \tag {2}
$$
The disconnectivity set in (1) can be obtained in several ways: (i) it is often the case that some edges between nodes must not exist due to prior knowledge; (ii) it can be estimated from initial estimators; (iii) it can be obtained in some tasks of learning structured graphs such as bipartite graph [14-16].
2.2 Related work
Estimating precision matrices under Gaussian graphical models has been extensively studied in the literature. One well-known method is graphical lasso [17-20], which minimizes the  $\ell_1$ -regularized Gaussian negative log-likelihood. Various algorithms were proposed to solve this problem including first-order methods [17, 18, 21-29] and second-order methods [30-33]. The graphical lasso optimization problem is unconstrained and nonsmooth, while Problem (1) is smooth and constrained. The difficulties in solving the two problems are inherently different, and the algorithms mentioned above cannot be directly extended to solve Problem (1).
Recent studies [2, 3, 8, 9] employed BCD and PPA-type algorithms to estimate precision matrices under  $\mathrm{MTP}_2$  constraints. In [2], the primal variable is updated one column/row at a time by solving a nonnegative quadratic program, cycling until convergence. The work [8] follows a similar approach but addresses the dual problem, improving memory efficiency. Both works target Problem (1) without the disconnectivity constraint. A BCD-type algorithm was proposed in [3] to accommodate disconnectivity constraints. However, the computational complexity of these algorithms, at  $O(p^4)$  operations per cycle, becomes prohibitive for high-dimensional problems. Alternatively, recent work [9] introduced an inexact PPA algorithm to solve a transformed problem, derived using the soft-thresholding technique. However, this algorithm demands the computation of an inexact Newton direction from a  $p^2 \times p^2$  linear system at every iteration within the inner loop, presenting computational difficulties in high-dimensional scenarios.
The proposed algorithm adopts the two-metric projection framework [34], incorporating distinct metrics for search direction and projection. A representative method in this framework is the projected Newton algorithm [35], originally designed for nonnegativity constrained problems. However, it is unsuitable for Problem (1) due to its  $O(p^6)$  operations needed to compute the inverse of the Hessian at each iteration. To mitigate computation and memory costs, the projected quasi-Newton algorithm with limited-memory Boyden-Fletcher-Goldfarb-Shanno (L-BFGS) was introduced in [36], requiring  $O((m + p)p^2)$  operations per iteration, with  $m$  the number of iterations stored for Hessian approximation. By leveraging the structure of Problem (1), this paper carefully designs the search direction and variable partitioning scheme to substantially reduce computation and memory costs, achieving the same orders per iteration as the gradient projection method [13].
3 Proposed Algorithm
In this section, we propose a fast projected Newton-like algorithm to solve Problem (1). The constraints in Problem (1) can be rewritten as  $\mathbf{X} \in \Omega \cap \mathbb{S}_{++}^{p}$ , where  $\Omega$  is defined as
$$
\Omega := \left\{\boldsymbol {X} \in \mathbb {R} ^ {p \times p} \mid X _ {i j} = 0, \forall (i, j) \in \mathcal {E}; X _ {i j} \leq 0, \forall i \neq j \right\}.
$$
The set  $\Omega$  is convex and closed, thus this constraint can be handled by a projection  $\mathcal{P}_{\Omega}$  defined by
$$
\left[ \mathcal {P} _ {\Omega} (\boldsymbol {A}) \right] _ {i j} = \left\{ \begin{array}{l l} 0 & \text {i f} (i, j) \in \mathcal {E}, \\ A _ {i j} & \text {i f} i = j, \\ \min  \left(A _ {i j}, 0\right) & \text {i f} (i, j) \notin \mathcal {E} \text {a n d} i \neq j. \end{array} \right. \tag {3}
$$
The positive definite set  $\mathbb{S}_{++}^p$  is not closed and cannot be managed by a projection, which will be handled using the line search method in Section 3.3. Let  $f$  denote the objective function of Problem (1). To address Problem (1), we start with the gradient projection method, expressed as:
$$
\boldsymbol {X} _ {k + 1} = \mathcal {P} _ {\Omega} \left(\boldsymbol {X} _ {k} - \gamma_ {k} \nabla f \left(\boldsymbol {X} _ {k}\right)\right), \tag {4}
$$
where  $\gamma_{k}$  is the step size. To accelerate convergence, one may consider
$$
\boldsymbol {X} _ {k + 1} = \mathcal {P} _ {\Omega} \left(\boldsymbol {X} _ {k} - \gamma_ {k} \boldsymbol {P} _ {k}\right), \tag {5}
$$
in which  $P_{k}\in \mathbb{R}^{p\times p}$  is a search direction defined by
$$
\operatorname {v e c} \left(\boldsymbol {P} _ {k}\right) = \boldsymbol {M} _ {k} ^ {- 1} \operatorname {v e c} \left(\nabla f \left(\boldsymbol {X} _ {k}\right)\right), \tag {6}
$$
where  $M_{k}\in \mathbb{R}^{p^{2}\times p^{2}}$  is a positive definite symmetric matrix, incorporating second-order derivative information. If we adopt  $M_{k}$  as the Hessian matrix, then  $P_{k}$  becomes the Newton direction as shown in Proposition 3.1, with the proof provided in Appendix B.1, and iterate (5) can be viewed as a natural adaptation of the unconstrained Newton's method. Regrettably, the convergence of such an iterate to the minimizer cannot be guaranteed, as  $P_{k}$  may not be a descent direction here, which is supported by numerical results in Section 5.1.1. Similar observations have also been reported in [35, 37].
Proposition 3.1. If  $M_{k}$  is constructed as the Hessian matrix  $H_{k}$  of Problem (1), then the search direction  $P_{k}$  defined in (6) can be written as  $P_{k} = X_{k}\nabla f(X_{k})X_{k}$ .
Remark 3.2. We refer to iterate (5) as the two-metric projection method [34], as it adopts two distinct metrics: the search direction  $P_{k}$  induced by the quadratic norm  $\| \cdot \|_{M_k}$  defined by  $\| A\|_{M_k} = \langle \mathrm{vec}(A), M_k\mathrm{vec}(A)\rangle^{\frac{1}{2}}$ , and the projection  $\mathcal{P}_{\Omega}$  with respect to the Frobenius norm  $\| \cdot \|_{\mathrm{F}}$ .
3.1 Identifying the sets of restricted and free variables
In order to guarantee iterate (5) to converge to the minimizer, we partition the variables into two groups, i.e., restricted and free variables, and update the two groups separately. We first define a set  $\mathcal{T}(\boldsymbol{X},\epsilon)$  with respect to  $\boldsymbol{X}\in \mathbb{S}_{++}^{p}$  and  $\epsilon \in \mathbb{R}_+$ ,
$$
\mathcal {T} (\boldsymbol {X}, \epsilon) := \left\{(i, j) \in [ p ] ^ {2} \mid - \epsilon \leq X _ {i j} \leq 0, [ \nabla f (\boldsymbol {X}) ] _ {i j} <   0 \right\}. \tag {7}
$$
Then at the  $k$ -th iteration, we identify the set of restricted variables based on  $\mathbf{X}_k$  as follows,
$$
\mathcal {I} _ {k} := \mathcal {T} \left(\boldsymbol {X} _ {k}, \epsilon_ {k}\right) \cup \mathcal {E}, \tag {8}
$$
where  $\mathcal{E}$  is the disconnectivity set from Problem (1), and  $\epsilon_{k}$  is a small positive scalar. For any  $(i,j)\in \mathcal{T}(X_k,\epsilon_k)$ ,  $X_{ij}$  in the next iterate is likely to be outside the feasible set (i.e.,  $X_{ij} > 0$ ) if we remove the projection  $\mathcal{P}_{\Omega}$ , as it is near zero and moves towards the positive direction if using the negative of the gradient as the search direction. Therefore, we set all restricted variables to zero.
To establish the theoretical convergence of the algorithm, the positive scalar  $\epsilon_{k}$  in (8) is specified as
$$
\epsilon_ {k} := \min  \left(2 (1 - \alpha) m ^ {2} \| [ \nabla f (\boldsymbol {X} _ {k}) ] _ {\mathcal {T} _ {\delta} \backslash \mathcal {E}} \| _ {\min }, \delta\right), \tag {9}
$$
where  $m$  is a positive scalar (See Lemma B.1 in Appendix),  $\alpha \in (0,1)$  is a parameter in the line search condition, and  $\mathcal{T}_{\delta}$  represents the set  $\mathcal{T}(\mathbf{X}_k,\delta)$ . In the rare event that  $\mathcal{T}_{\delta}$  is empty, particularly in sparse settings, we define  $\epsilon_{k} = \delta$ , implying an empty  $\mathcal{T}(\mathbf{X}_k,\epsilon_k)$  in (8). The parameter  $\delta$  satisfies  $0 < \delta < \min_{(i,j)\in \mathrm{supp}(\mathbf{X}^{\star})}|[[\mathbf{X}^{\star}]_{ij}]|$ , where  $\mathbf{X}^{\star}$  is the minimizer of Problem (1). Such a condition can be ensured by setting a sufficiently small positive  $\delta$ . Then  $\epsilon_{k}$  in (9) is nearly equal to  $\delta$ . From an implementation view, we can directly set a small positive  $\epsilon_{k}$ , resulting in the algorithm performing well in practice. The set of free variables, denoted by  $\mathcal{T}_{k}^{c}$ , is the complement of  $\mathcal{T}_{k}$ .
3.2 Computing approximate Newton direction
While the (approximate) Newton direction usually demands a considerably higher computational cost than the gradient, our designed direction maintains the same computational order as the gradient.
At the  $k$ -th iteration, we first partition  $\mathbf{X}_k$  into two groups,  $[\mathbf{X}_k]_{\mathcal{I}_k}$  and  $[\mathbf{X}_k]_{\mathcal{I}_k^c}$ , where  $[\mathbf{X}_k]_{\mathcal{I}_k} \in \mathbb{R}^{|\mathcal{I}_k|}$  and  $[\mathbf{X}_k]_{\mathcal{I}_k^c} \in \mathbb{R}^{|\mathcal{I}_k^c|}$  denote two vectors containing all elements of  $\mathbf{X}_k$  in the sets  $\mathcal{I}_k$  and  $\mathcal{I}_k^c$ , respectively. Then we can rewrite the search direction  $\mathbf{P}_k$  in (6) as follows,
$$
\operatorname {p v e c} _ {k} \left(\boldsymbol {P} _ {k}\right) = \boldsymbol {Q} _ {k} \operatorname {p v e c} _ {k} (\nabla f \left(\boldsymbol {X} _ {k}\right)), \tag {10}
$$
where  $\mathrm{pvec}_k(P_k)$  stacks  $P_{k}$  into a vector, similar to  $\operatorname {vec}(P_k)$ , but places elements from  $\mathcal{I}_k^c$  first, followed by those from  $\mathcal{I}_k$ .  $Q_{k}$  is obtained by permuting the rows and columns of  $M_{k}^{-1}$  in (6). To enhance computational efficiency, we propose constructing  $Q_{k}$  and  $M_{k}^{-1}$  as follows:
$$
\boldsymbol {Q} _ {k} = \left[ \begin{array}{c c} {\left[ M _ {k} ^ {- 1} \right] _ {\mathcal {I} _ {k} ^ {c} \mathcal {I} _ {k} ^ {c}}} & {\left[ M _ {k} ^ {- 1} \right] _ {\mathcal {I} _ {k} ^ {c} \mathcal {I} _ {k}}} \\ {\left[ M _ {k} ^ {- 1} \right] _ {\mathcal {I} _ {k} \mathcal {I} _ {k} ^ {c}}} & {\left[ M _ {k} ^ {- 1} \right] _ {\mathcal {I} _ {k} \mathcal {I} _ {k}}} \end{array} \right] = \left[ \begin{array}{c c} {\left[ H _ {k} ^ {- 1} \right] _ {\mathcal {I} _ {k} ^ {c} \mathcal {I} _ {k} ^ {c}}} & \boldsymbol {0} \\ \boldsymbol {0} & D _ {k} \end{array} \right], \tag {11}
$$
where  $D_{k}\in \mathbb{R}^{|\mathcal{I}_{k}|\times |\mathcal{I}_{k}|}$  is a positive definite diagonal matrix, and  $\left[H_k^{-1}\right]_{\mathcal{I}_k^c\mathcal{I}_k^c}\in \mathbb{R}^{|\mathcal{I}_k^c | \times |\mathcal{I}_k^c |}$  is a principal submatrix of  $H_{k}^{-1}$ , preserving rows and columns indexed by  $\mathcal{I}_k^c$ . Here,  $H_{k}$  is the Hessian matrix at  $X_{k}$ . The construction of  $M_{k}^{-1}$  in (11) is crucial for defining the search direction, enabling computation and memory costs comparable to the gradient projection method while effectively incorporating second-order derivative information.
Next, we compute the approximate Newton direction  $P_{k}$  over the set  $\mathcal{I}_k^c$  and present the iterate  $[X_{k + 1}]_{\mathcal{I}_k^c}$ . We define a projection  $\mathcal{P}_{\mathcal{I}_k^c}(A)$  as follows,
$$
\left[ \mathcal {P} _ {\mathcal {I} _ {k} ^ {c}} (\boldsymbol {A}) \right] _ {i j} = \left\{ \begin{array}{l l} A _ {i j} & \text {i f} (i, j) \in \mathcal {I} _ {k} ^ {c}, \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {12}
$$
Leveraging the well-designed gradient scaling matrix  $Q_{k}$  in (11), we can efficiently compute the approximate Newton direction, as demonstrated in Proposition 3.3, with proof in Appendix B.2.
Proposition 3.3. If  $M_{k}$  is constructed by (11), then the search direction  $\pmb{P}_{k}$  defined in (6) over  $\mathcal{I}_k^c$  can be written as  $[\pmb{P}_k]_{\mathcal{I}_k^c} = [\pmb{X}_k\mathcal{P}_{\mathcal{I}_k^c}(\nabla f(\pmb{X}_k))\pmb{X}_k]_{\mathcal{I}_k^c}$ .
Using the search direction from Proposition 3.3, we update  $X_{k + 1}$  over  $\mathcal{I}_k^c$  as follows,
$$
\left[ \boldsymbol {X} _ {k + 1} \right] _ {\mathcal {I} _ {k} ^ {c}} = \mathcal {P} _ {\Omega} \left(\left[ \boldsymbol {X} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c}} - \gamma_ {k} \left[ \boldsymbol {X} _ {k} \mathcal {P} _ {\mathcal {I} _ {k} ^ {c}} (\nabla f (\boldsymbol {X} _ {k})) \boldsymbol {X} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c}}\right). \tag {13}
$$
For the restricted variables in the set  $\mathcal{I}_k$ , we directly set them to zero, i.e.,  $[\pmb{X}_{k + 1}]_{\mathcal{I}_k} = \mathbf{0}$ .
3.3 Computing step size
We adopt an Armijo-like rule for step size selection, ensuring the global convergence of our algorithm. Based on the iterate proposed in Section 3.2, we define  $\mathbf{X}_k(\gamma_k)$  with  $[\mathbf{X}_k(\gamma_k)]_{\mathcal{I}_k} = \mathbf{0}$  and
$$
\left[ \boldsymbol {X} _ {k} \left(\gamma_ {k}\right) \right] _ {\mathcal {I} _ {k} ^ {c}} = \mathcal {P} _ {\Omega} \left(\left[ \boldsymbol {X} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c}} - \gamma_ {k} \left[ \boldsymbol {X} _ {k} \mathcal {P} _ {\mathcal {I} _ {k} ^ {c}} (\nabla f (\boldsymbol {X} _ {k})) \boldsymbol {X} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c}}\right). \tag {14}
$$
We test step sizes  $\gamma_{k} \in \{\beta^{0}, \beta^{1}, \beta^{2}, \ldots\}$  with  $\beta \in (0, 1)$ , until we find the smallest  $t \in \mathbb{N}$  such that  $X_{k}(\gamma_{k})$ , with  $\gamma_{k} = \beta^{t}$ , satisfies  $X_{k}(\gamma_{k}) \in \mathbb{S}_{++}^{p}$  and the line search condition:
$$
f \left(\boldsymbol {X} _ {k} \left(\gamma_ {k}\right)\right) \leq f \left(\boldsymbol {X} _ {k}\right) - \alpha \gamma_ {k} \left\langle \left[ \nabla f \left(\boldsymbol {X} _ {k}\right) \right] _ {\mathcal {I} _ {k} ^ {c}}, \left[ \boldsymbol {P} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c}} \right\rangle - \alpha \left\langle \left[ \nabla f \left(\boldsymbol {X} _ {k}\right) \right] _ {\mathcal {I} _ {k}}, \left[ \boldsymbol {X} _ {k} \right] _ {\mathcal {I} _ {k}} \right\rangle , \tag {15}
$$
where  $\alpha \in (0,1)$  is a scalar. We then set  $X_{k + 1} = X_k(\gamma_k)$ . Positive definiteness of  $X_{k + 1}$  can be verified during Cholesky factorization for objective function evaluation. It is worth mentioning that working with the positive semi-definiteness constraint on  $X$  instead of positive definiteness would not change anything in the algorithm if we keep the line search, as the positive definiteness is automatically enforced due to the form of the objective function.
The line search condition (15) is a variant of the Armijo rule. Condition (15) can be always satisfied for a small enough step size as shown in Proposition 3.4. Define the feasible set of Problem (1) as
$$
\mathcal {U} ^ {p} := \left\{\boldsymbol {X} \in \mathbb {R} ^ {p \times p} \mid \boldsymbol {X} \in \mathcal {M} ^ {p}, X _ {i j} = 0, \forall (i, j) \in \mathcal {E} \right\}. \tag {16}
$$
For any given  $X^o \in \mathcal{U}^p$ , define the lower level set of the objective function  $f$  for Problem (1) as:
$$
L _ {f} := \left\{\boldsymbol {X} \in \mathcal {U} ^ {p} \mid f (\boldsymbol {X}) \leq f \left(\boldsymbol {X} ^ {o}\right) \right\}. \tag {17}
$$
Proposition 3.4. For any  $X_{k}\in L_{f}$ , there exists a  $\bar{\gamma}_k > 0$  such that  $X_{k}(\gamma_{k})\in \mathbb{S}_{++}^{p}$  and the line search condition (15) holds for any  $\gamma_k\in (0,\bar{\gamma}_k)$ .
The proof of Proposition 3.4 is available in Appendix B.3. We demonstrate that  $X_{k}(\gamma_{k})$  ensures a decrease of the objective function value in Proposition 3.5, proved in Appendix B.4.
Proposition 3.5. For any  $X_{k}\in L_{f}$  , if  $X_{k}(\gamma_{k})$  satisfies the line search condition (15), then we have
$$
f \left(\boldsymbol {X} _ {k} (\gamma_ {k})\right) \leq f \left(\boldsymbol {X} _ {k}\right) - \alpha \gamma_ {k} m ^ {2} \big \| \left[ \nabla f (\boldsymbol {X} _ {k}) \right] _ {\mathcal {I} _ {k} ^ {c}} \big \| ^ {2},
$$
where  $m$  is a positive scalar (See Lemma B.1 in Appendix).
3.4 Computation and memory costs