SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In each iteration, our algorithm calculates the gradient, performs two matrix multiplications, and conducts two projections, with respective computational costs of  $O(p^3)$ ,  $O(p^3)$ , and  $O(p^2)$ . In our current implementation of the line search method, we first conduct the Cholesky factorization  $\mathbf{X} = \mathbf{L}\mathbf{L}^\top$  using MATLAB's "chol" function. This function can simultaneously verify the positive definiteness of  $\mathbf{X}$ . Next, we calculate the log-determinant function as  $\log \det(\mathbf{X}) = 2\sum_{i}\log (L_{ii})$ . The Cholesky factorization is the most computationally demanding step, generally requiring  $O(p^3)$  costs for a  $p\times p$  matrix.
Algorithm 1 Fast Projected Newton-like (FPN) algorithm
1: Input: Regularization parameter  $\lambda_{ij}$ ,  $\alpha$ ,  $\beta$ ,  $S$ , and  $X_0$ ;  
2: for  $k = 0,1,2,\ldots$  do  
3: Identify the restricted set  $\mathcal{I}_k$  and free set  $\mathcal{I}_k^c$  according to (8);  
4: Compute the search direction over  $\mathcal{I}_k^c$ :  $[P_k]_{\mathcal{I}_k^c} = [X_k\mathcal{P}_{\mathcal{I}_k^c}(\nabla f(X_k))X_k]_{\mathcal{I}_k^c}$ ;  
5:  $t \gets 0$ ;  
6: repeat  
7: Update  $X_{k+1}$ :  $[X_{k+1}]_{\mathcal{I}_k} = 0$ , and  $[X_{k+1}]_{\mathcal{I}_k^c} = \mathcal{P}_{\Omega}([X_k]_{\mathcal{I}_k^c} - \beta^t[P_k]_{\mathcal{I}_k^c})$ ;  
8:  $t \gets t + 1$ ;  
9: until  $X_{k+1} \succ 0$ , and  
[ f(X_{k+1}) \leq f(X_k) - \alpha \beta^t \langle [\nabla f(X_k)]_{\mathcal{I}_k^c}, [P_k]_{\mathcal{I}_k^c} \rangle - \alpha \langle [\nabla f(X_k)]_{\mathcal{I}_k}, [X_k]_{\mathcal{I}_k} \rangle. ]
To mitigate the computational burden associated with Cholesky factorization, we suggest a more efficient method for evaluating the log-determinant function and verifying positive definiteness, as presented in [38]. This method, which leverages Schur complements and sparse linear system solving, can tackle problems of up to  $10^{6}$  dimensions. Furthermore, it is worthwhile to investigate more efficient strategies for computing an approximate log-determinant function. In this context, the approach proposed in [39] offers a nearly linear scaling of execution time with the number of non-zero entries, while maintaining a high level of accuracy.
In summary, our algorithm has an overall complexity of  $O(p^3)$  per iteration. BCD-type algorithms [2, 3, 8] need  $O(p^4)$  operations per cycle, while projected quasi-Newton with L-BFGS [36] requires  $O((m + p)p^2)$  operations per iteration, with  $m$  as the stored iteration count for Hessian approximation. The PPA algorithm [9] requires computing an inexact Newton direction from a  $p^2 \times p^2$  linear system during each inner loop iteration, with the exact complexity not established. In addition, our algorithm, gradient projection and BCD-type methods [2, 3, 8] need  $O(p^2)$  memory costs, while projected quasi-Newton with L-BFGS [36] requires  $O(mp^2)$  and PPA [9] demands  $O(p^4)$ .
Prior to delving into the convergence analysis, we first establish the uniqueness of the minimizer for Problem (1) and determine the sufficient and necessary conditions for a point to be the minimizer.
Theorem 4.1. The minimizer of Problem (1) is unique, and a point  $X^{\star} \in \mathcal{M}^{p}$  is the minimizer if and only if it satisfies
$$
[ \boldsymbol {X} ^ {\star} ] _ {i j} = 0 \forall (i, j) \in \mathcal {E}, [ \nabla f (\boldsymbol {X} ^ {\star}) ] _ {\mathcal {V} \backslash \mathcal {E}} \leq \mathbf {0}, a n d [ \nabla f (\boldsymbol {X} ^ {\star}) ] _ {\mathcal {V} ^ {c}} = \mathbf {0}, \tag {18}
$$
where  $\mathcal{V} = \{(i,j)\in [p]^2\mid [X^{\star}]_{ij} = 0\}$
The proof of Theorem 4.1 is available in Appendix B.5. The following theorem shows that our algorithm converges to the minimizer of Problem (1).
Theorem 4.2. The sequence  $\{\pmb{X}_k\}$  generated by Algorithm 1 with  $\pmb{X}_0 \in L_f$  converges to the minimizer  $\pmb{X}^{\star}$  of Problem (1), with  $\{f(\pmb{X}_k)\}$  monotonically decreasing.
The proof of Theorem 4.2 is available in Appendix B.6. It is worth noting that constructing an initial point  $X_0 \in L_f$ , as defined in (17), is straightforward. Please refer to the proof of Theorem 4.2 for more details on this. The theoretical analysis on support set convergence and sequence convergence rate relies on the following assumption.
Assumption 4.3. For the minimizer  $X^{\star}$  of Problem (1), we assume that the gradient of the objective function  $f$  at  $X^{\star}$  satisfies
$$
\left[ \nabla f (\boldsymbol {X} ^ {\star}) \right] _ {i j} <   0, \quad \forall (i, j) \in \mathcal {V} \setminus \mathcal {E},
$$
where  $\mathcal{V} = \left\{(i,j)\in [p]^2\big|\left[X^\star \right]_{ij} = 0\right\}$ , and  $\mathcal{E}$  is the disconnectivity set.
Theorem 4.4. Under Assumption 4.3, the set of free variables  $\mathcal{I}_k^c$  generated by Algorithm 1 converges to the support of the minimizer  $X^{\star}$  of Problem (1) in finite iterations. In other words, there exists some  $k_{o} \in \mathbb{N}_{+}$  such that  $\mathcal{I}_k^c = \mathrm{supp}(X^{\star})$  for any  $k \geq k_{o}$ .
The proof of Theorem 4.4 is provided in Appendix B.7. Theorem 4.4 demonstrates that the set of free variables constructed by our variable partitioning scheme can exactly identify the support of  $X^{\star}$  in finite iterations. Now we establish the convergence rate of our algorithm. Define
$$
\boldsymbol {R} _ {k} = \left[ \boldsymbol {H} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c} \mathcal {I} _ {k} ^ {c}} - \left[ \boldsymbol {H} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c} \mathcal {I} _ {k}} \left[ \boldsymbol {H} _ {k} \right] _ {\mathcal {I} _ {k} \mathcal {I} _ {k}} ^ {- 1} \left[ \boldsymbol {H} _ {k} \right] _ {\mathcal {I} _ {k} ^ {c} \mathcal {I} _ {k}} ^ {\top}. \tag {19}
$$
Theorem 4.5. Under Assumption 4.3, the sequence  $\{X_k\}$  generated by Algorithm 1 satisfies
$$
\limsup_{k\to \infty}\frac{\left\|X_{k + 1} - X^{\star}\right\|_{M_{k}}^{2}}{\left\|X_{k} - X^{\star}\right\|_{M_{k}}^{2}}\leq \limsup_{k\to \infty}\left(1 - \min \left(m_{k},\frac{2(1 - \alpha)\beta m_{k}}{M_{k}}\right)\right)^{2},
$$
where  $m_{k}$  and  $M_{k}$  as the smallest and largest eigenvalues of  $\pmb{R}_{k}^{-\frac{1}{2}}\left[\pmb{H}_{k}\right]_{\mathcal{T}_{k}^{c}\mathcal{T}_{k}^{c}}\pmb{R}_{k}^{-\frac{1}{2}}$ , respectively.
The proof of Theorem 4.5 is provided in Appendix B.8. Theorem 4.5 reveals that the convergence rate of our algorithm depends on the condition number  $m_{k} / M_{k}$  of  $\pmb{R}_{k}^{-\frac{1}{2}}[\pmb{H}_{k}]_{\mathcal{I}_{k}^{c}\mathcal{I}_{k}^{c}}\pmb{R}_{k}^{-\frac{1}{2}}$ . Replacing  $\pmb{R}_{k}$  with an identify matrix, (i.e., using the projected gradient method) results in a rate dependent on the condition number of  $[\pmb{H}_{k}]_{\mathcal{I}_{k}^{c}\mathcal{I}_{k}^{c}}$ . The condition number of  $\pmb{R}_{k}^{-\frac{1}{2}}[\pmb{H}_{k}]_{\mathcal{I}_{k}^{c}\mathcal{I}_{k}^{c}}\pmb{R}_{k}^{-\frac{1}{2}}$  could be larger, as  $\pmb{R}_{k}$  could approximate  $[\pmb{H}_{k}]_{\mathcal{I}_{k}^{c}\mathcal{I}_{k}^{c}}$  well. Thus, the gradient scaling matrix  $\pmb{R}_{k}^{-1}$ , i.e.,  $[\pmb{H}_{k}^{-1}]_{\mathcal{I}_{k}^{c}\mathcal{I}_{k}^{c}}$  in (11), leads our algorithm to converge faster than the projected gradient method.
It is important to note that our algorithm generally does not achieve superlinear convergence, despite the incorporation of second-order information. Superlinear convergence necessitates that the inverse gradient scaling matrix progressively approximate the Hessian at the minimizer [40]. However, this is a condition that our constructed scaling matrix does not meet. Despite this, we should note that constructing a search direction to achieve superlinear convergence proves significantly more computationally demanding than our approach, as it cannot leverage the special structure of the Hessian to decrease the computational load.
We conduct experiments on synthetic and real-world data to verify the performance of our algorithm. All experiments were conducted on 2.10GHZ Xeon Gold 6152 machines and Linux OS, and all methods were implemented in MATLAB. State-of-the-art methods for comparisons include:
- BCD [2]: Updates each column/row of primal variable using a nonnegative quadratic program.
- optGL [8]: Similar to BCD but solves nonnegative quadratic programs on the dual variable.
- GGL [3]: Similar to BCD but handles disconnectivity constraints, while BCD and optGL cannot.
- PGD [13]: Projected gradient descent method with backtracking line search.
- APGD [41]: Accelerated projected gradient algorithm with extrapolation step.
- PPA [9]: Inexact proximal point algorithm with Newton-CG method.
- PQN-LBFGS[36]: Projected quasi-Newton method using limited-memory BFGS.
Note that all state-of-the-art methods listed above can converge to the minimizer of Problem (1), and we focus on the comparison of computational time for those methods. To that end, we report the relative error of the objective function value as a function of the run time, which is calculated by
$$
| f (\boldsymbol {X} _ {k}) - f (\boldsymbol {X} ^ {\star}) | / | f (\boldsymbol {X} ^ {\star}) |, \tag {20}
$$
where  $f$  is the objective function of Problem (1), and  $X^{\star}$  is its minimizer. The  $X^{\star}$  is computed by running the state-of-the-art method GGL [3] until it converges to a point  $X_{k}\in \mathcal{M}^{p}$  satisfying
$$
\left[ \boldsymbol {X} _ {k} \right] _ {i j} = 0 \quad \forall (i, j) \in \mathcal {E}, \quad \left[ \nabla f (\boldsymbol {X} _ {k}) \right] _ {\mathcal {A} \backslash \mathcal {E}} \leq \mathbf {0}, \quad \| [ \nabla f (\boldsymbol {X} ^ {\star}) ] _ {\mathcal {A} ^ {c}} \| _ {\max } \leq 1 0 ^ {- 8}, \tag {21}
$$
where  $\mathcal{A} := \{(i,j) \in [p]^2 \mid |[X_k]_{ij}| \leq 10^{-8}\}$ . Through the comparison with the sufficient and necessary conditions of the unique minimizer of Problem (1) presented in Theorem 4.1, we can see that any point  $X_k$  satisfying the conditions in (21) is sufficiently close to the minimizer.
(a) BA graph of degree one
(b) BA graph of degree two
Figure 1: Illustration of BA graphs of degree one (a) and degree two (b) with 500 nodes. (c) Convergence comparison between our algorithm and the Newton direction from Proposition 3.1.
(c) Convergence comparison
We generate independent samples  $\pmb{y}^{(1)},\dots,\pmb{y}^{(n)}\in \mathbb{R}^p$  from a multivariate Gaussian distribution with zero mean and precision matrix  $\Theta$ , where  $\Theta \in \mathcal{M}^p$  is the underlying precision matrix associated with a graph consisting of  $p$  nodes. We use the Barabási-Albert (BA) model [42] to generate the support of the underlying precision matrix. To help readers to know well the BA graphs, we present two examples in Figure 1. More details about experimental setting are provided in Appendix A.
5.1.1 Comparisons of search directions
We evaluate the convergence of algorithms with different search directions. Figure 1 (c) demonstrates that our algorithm, using the direction from Proposition 3.3, converges to the minimizer, aligning with our theoretical convergence results in Theorem 4.2. In contrast, the algorithm using iterate (5) and the Newton direction from Proposition 3.1 stops decreasing the objective function value after a few iterations, indicating that this direction cannot be consistently regarded as a descent direction. The algorithm selects the step size through the Armijo rule, i.e., it is continually reduced until a decrease in the objective function is achieved.
5.1.2 Comparisons of computational time
We evaluate the computational time of our algorithm and state-of-the-art methods on synthetic datasets, averaging results over 10 realizations. We plot markers every 10 iterations for PGD, APGD, PQN-LBFGS, and FPN, while marking each cycle of updating all columns/rows for BCD-type algorithms (BCD, GGL, and optGL) and each outer iteration for PPA.
Figure 2 compares the computational time of various methods for solving Problem (1) on BA graphs of degree one. Our proposed FPN significantly outperforms all state-of-the-art methods in convergence time for node counts ranging from 1000 to 5000. BCD and GGL are efficient at 1000 nodes, being faster than PGD and APGD, and competitive with PQN-LBFGS and PPA. However, at 5000 nodes, BCD and GGL become slower due to the  $O(p^4)$  operations per cycle required to solve  $p$  nonnegative quadratic programs, leading to rapidly increasing computational costs.
(a)  $p = 1000$
Figure 2: Relative errors of the objective function values versus time on BA graphs of degree one.
(b)  $p = 3000$
(c)  $p = 5000$
(a)  $p = 1000$
Figure 3: Relative errors of the objective function value versus time on BA graphs of degree two.
(a)
(c)
Figure 4: Relative errors of objective function values versus time on data sets: (a) BA graph of degree one and (b) BA graph of degree two, imposing disconnectivity constraints. (c) Run time versus numbers of nonzero elements in precision matrices across varying regularization parameter values, with the underlying precision matrix having 2998 nonzero elements (indicated by vertical red line).
Figure 3 presents the computational time of different methods on BA graphs of degree two. As with degree one graphs, FPN outperforms state-of-the-art methods in computational time for varying node counts. PQN-LBFGS and FPN require fewer iterations to converge to the minimizer than PGD and APGD, particularly in high dimensions (e.g.,  $p = 5000$ ), indicating faster convergence. This is because both PQN-LBFGS and FPN utilize second-order information and approximate Newton direction, overcoming low convergence rates of first-order methods in high-dimensional cases.
Figure 4 (a) and (b) compare the computational time of various algorithms solving Problem (1) with disconnectivity constraints, where FPN consistently converges fastest. (c) evaluates the impact of the estimated precision matrix's sparsity level on run time. BCD, GGL, PPA, and FPN exhibit stable run time across varying sparsity levels, highlighting their robustness regarding regularization parameter settings, while other methods display increased run time as sparsity decreases.
We perform experiments on two real-world datasets: the concepts dataset and a financial time-series dataset. For the concepts dataset, we compare the computational time of different algorithms solving Problem (1). The experimental results on the financial time-series dataset are provided in Appendix A, where we examine the performance of our method in graph edge recovery.
The concepts dataset [43], from Intel Labs, comprises 1000 nodes and 218 semantic features, with  $p = 1000$  and  $n = 218$ . Nodes represent concepts like "house," "coat," and "whale," while semantic features are questions like "Can it fly?", "Is it alive?", and "Can you use it?". Responses, collected via Amazon Mechanical Turk, range from "definitely no" to "definitely yes" on a five-point scale.
Figure 5 (a) compares the run time of various algorithms solving Problem (1) on the concepts dataset. Our proposed algorithm converges to the minimizer considerably faster than state-of-the-art algorithms, which is consistent with the observations in synthetic experiments. Note that all compared algorithms can reach the minimizer of Problem (1), and thus learn the same graph.
(b)
Figure 5: (a) Relative errors of the objective function values versus time for different algorithms in solving Problem (1) on the concepts data set, consisting of 1000 nodes. (b) The connected subgraph illustrated by the minimizer on the concepts data set, which consists of 132 nodes.
Figure 5 (b) displays a connected subgraph illustrated by the minimizer of Problem (1). Interestingly, it is observed that the learned graph forms a semantic network, where related concepts are closely connected. For instance, insect concepts such as "bee", "butterfly", "flea", "mosquito", and "spider" are grouped together, while human-related concepts like "baby", "husband", "child", "girls", and "man" form another group. Moreover, the network connects "penguin" closely to birds like "owl" and "crow" and sea animals like "goldfish" and "seal", highlighting its aquatic bird nature. Overall, the learned network effectively captures concept relationships.
6 Conclusions and Discussions
In this paper, we have introduced a fast projected Newton-like method for estimating precision matrices under  $\mathrm{MTP}_2$  constraints. Our algorithm, leveraging the two-metric projection method, stands out from existing BCD and PPA-type approaches for addressing the target problem. The proposed algorithm is not only straightforward to implement but also efficient in terms of computation and memory usage. We have provided theoretical convergence analysis and conducted extensive experiments, which clearly demonstrate the superior efficiency of our algorithm in computational time, outperforming state-of-the-art methods. Moreover, we have observed significant performance of our method in terms of modularity value on the learned financial time-series graphs.
Finally, we discuss the limitations of our paper. Our algorithm is proven to converge to the minimizer without any assumptions; however, we require Assumption 4.3 to establish support set convergence in finite iterations and to determine the convergence rate. This assumption is relatively mild, as Theorem 4.1 shows that the minimizer  $X^{\star}$  must satisfy  $[\nabla f(X^{\star})]_{ij} \leq 0$  for each  $(i,j) \in \mathcal{V} \backslash \mathcal{E}$ . The only additional requirement in Assumption 4.3 is the strictness of this inequality. However, the conditions for ensuring this strictness remain unclear. As this assumption is equivalent to the strict complementary slackness condition in optimization theory, exploring verifiable conditions to guarantee Assumption 4.3 could enrich our algorithm's insights.
This work was supported by the Hong Kong Research Grants Council GRF 16207820, 16310620, and 16306821, the Hong Kong Innovation and Technology Fund (ITF) MHP/009/20, and the Project of Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone under Grant HZQB-KCZYB-2020083. We would also like to thank the anonymous reviewers for their valuable feedback on the manuscript.
[39] I. Han, D. Malioutov, and J. Shin, "Large-scale log-determinant computation through stochastic Chebyshev expansions," in International Conference on Machine Learning. PMLR, 2015, pp. 908-917.
[40] J. Nocedal and S. Wright, Numerical optimization. Springer Science & Business Media, 2006.
[41] Y. Nesterov, Introductory lectures on convex optimization: A basic course. Springer Science & Business Media, 2003, vol. 87.
[42] A.-L. Barabási and R. Albert, “Emergence of scaling in random networks,” Science, vol. 286, no. 5439, pp. 509–512, 1999.
[43] B. Lake and J. Tenenbaum, “Discovering structure by learning sparse graphs,” in Proceedings of the 33rd Annual Cognitive Science Conference, 2010, pp. 778—783.
[44] J. Ying, J. V. d. M. Cardoso, and D. P. Palomar, “Nonconvex sparse graph learning under Laplacian constrained graphical model,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 7101–7113.
[45] E. J. Candes, M. B. Wakin, and S. P. Boyd, “Enhancing sparsity by reweighted  $\ell_1$  minimization,” Journal of Fourier analysis and applications, vol. 14, no. 5, pp. 877–905, 2008.
[46] J. Ying, J. V. d. M. Cardoso, and D. P. Palomar, “Minimax estimation of Laplacian constrained precision matrices,” in International Conference on Artificial Intelligence and Statistics, vol. 130, 2021, pp. 3736–3744.
[47] ——, “A fast algorithm for graph learning under attractive Gaussian Markov random fields,” in 2021 55th Asilomar Conference on Signals, Systems, and Computers, 2021, pp. 1520–1524.
[48] M. E. Newman, "Modularity and community structure in networks," Proceedings of the National Academy of Sciences, vol. 103, no. 23, pp. 8577-8582, 2006.
[49] R. J. Plemmons, “M-matrix characterizations. I-nonsingular M-matrices,” Linear Algebra and its applications, vol. 18, no. 2, pp. 175–188, 1977.
[50] P. Drábek and J. Milota, Methods of nonlinear analysis: applications to differential equations. Springer Science & Business Media, 2007.
[51] D. P. Bertsekas, Nonlinear programming, 3rd ed. Athena Scientific, Belmont, MA, 2016.
[52] ——, “On the Goldstein-Levitin-Polyak gradient projection method,” IEEE Transactions on Automatic Control, vol. 21, no. 2, pp. 174–184, 1976.
In the following sections, we provide more details on experimental settings and additional experimental results for financial time-series data in Appendix A and the proofs for Propositions 3.1, 3.3, 3.4, and 3.5, as well as Theorems 4.1, 4.2, 4.4, and 4.5 in Appendix B.
We first provide an in-depth description of the experimental settings, then present numerical experiments carried out on the financial time-series data.
A.1 Experimental settings
We examine Barabasi-Albert (BA) graphs [42] as a model for the support structure of the underlying precision matrix. BA models hold a significant position in network science due to their ability to generate random scale-free networks via a preferential attachment mechanism. This mechanism ensures that newly introduced nodes are more likely to connect with nodes that possess a higher degree during the network's evolution. Scale-free networks serve as suitable models for various systems, including the Internet, protein interaction networks, citation networks, as well as the majority of social and online networks [42].
In a BA graph with degree  $r$ , each new node connects to  $r$  pre-existing nodes, with the probability of connection being proportional to the number of edges the existing nodes currently have. In this paper, we consider  $r$  values of 1 and 2.
Following the procedures detailed in [44], we assign a positive weight to each edge of a graph and set a zero weight for disconnected nodes. Positive weights are uniformly sampled from  $U(2,5)$ . This process results in a weighted adjacency matrix  $A$  containing all the graph weights. Then we adopt the procedures outlined in [2] for generating the underlying precision matrix  $\Theta$ . We first set
$$
\tilde {\boldsymbol {\Theta}} = \delta \boldsymbol {I} - \boldsymbol {A}, \quad \text {w i t h} \quad \delta = 1. 0 5 \lambda_ {\max } (\boldsymbol {A}), \tag {22}
$$
where  $\lambda_{\max}(A)$  represents the largest eigenvalue of  $A$ . In this context, the weight is zero if two nodes are disconnected, and follows a uniform distribution  $U(2,5)$  when nodes are connected. Lastly, we define  $\Theta = D\tilde{\Theta}D$ , where  $D$  is a diagonal matrix selected such that the covariance matrix  $\Theta^{-1}$  has unit diagonal elements.
We set the regularization parameter  $\lambda_{ij}$  in Problem (1) as follows
$$
\lambda_ {i j} = \frac {\sigma}{\left| \left[ \hat {\boldsymbol {X}} \right] _ {i j} \right| + \epsilon}, \quad \forall i \neq j, \tag {23}
$$
where  $\hat{\pmb{X}}$  represents an estimator,  $\epsilon$  is set to  $10^{-3}$ , and  $\sigma > 0$  is a parameter that adjusts the sparsity. The function in (23) is closely related to the re-weighted  $\ell_1$ -norm regularization [14, 45, 46], which effectively enhances the sparsity of the solution and reduces the estimation bias resulting from the  $\ell_1$ -norm [47]. We employ the maximum likelihood estimator as  $\hat{\pmb{X}}$ , i.e., the minimizer of Problem (1) without the sparsity regularization. Note that one can solve the maximum likelihood estimator with a relatively large tolerance to obtain a coarse estimator, and alternative monotonically decreasing functions may be explored in (23). For PQN-LBFGS, we utilize the previous 50 updates to compute the search direction. For FPN, we set  $\epsilon_k = 10^{-15}$  in (8) for identifying the set of restricted variables.