Title: Fast Projected Newton-like Method for Precision Matrix Estimation under Total Positivity
Venue: NeurIPS 2023
Accepted: yes
Keywords: MTP2

Total Positivity

Generalized graph Laplacian

Precision matrix estimation

Nonnegative partial correlations
Abstract: We study the problem of estimating precision matrices in Gaussian distributions that are multivariate totally positive of order two ($\mathrm{MTP}_2$). The precision matrix in such a distribution is an M-matrix. This problem can be formulated as a sign-constrained log-determinant program. Current algorithms are designed using the block coordinate descent method or the proximal point algorithm, which becomes computationally challenging in high-dimensional cases due to the requirement to solve numerous nonnegative quadratic programs or large-scale linear systems. To address this issue, we propose a novel algorithm based on the two-metric projection method, incorporating a carefully designed search direction and variable partitioning scheme. Our algorithm substantially reduces computational complexity, and its theoretical convergence is established. Experimental results on synthetic and real-world datasets demonstrate that our proposed algorithm provides a significant improvement in computational efficiency compared to the state-of-the-art methods.

Fast Projected Newton-like Method for Precision Matrix Estimation under Total Positivity

Abstract

We study the problem of estimating precision matrices in Gaussian distributions that are multivariate totally positive of order two ( MTP 2 ). The precision matrix in such a distribution is an M -matrix. This problem can be formulated as a signconstrained log-determinant program. Current algorithms are designed using the block coordinate descent method or the proximal point algorithm, which becomes computationally challenging in high-dimensional cases due to the requirement to solve numerous nonnegative quadratic programs or large-scale linear systems. To address this issue, we propose a novel algorithm based on the two-metric projection method, incorporating a carefully designed search direction and variable partitioning scheme. Our algorithm substantially reduces computational complexity, and its theoretical convergence is established. Experimental results on synthetic and real-world datasets demonstrate that our proposed algorithm provides a significant improvement in computational efficiency compared to the state-of-the-art methods.

1 Introduction

We consider the problem of estimating the precision matrix ( i.e., inverse covariance matrix) in a multivariate Gaussian distribution, where all the off-diagonal elements of the precision matrix are nonpositive. The resulting precision matrix is a symmetric M-matrix . Such property is also known as multivariate totally positive of order two ( MTP 2 ) [1]. For ease of presentation, we call the nonpositivity constraints on the off-diagonal elements of the precision matrix as MTP 2 constraints. This model arises in a variety of applications such as taxonomic reasoning [2], graph signal processing [3], factor analysis in psychometrics [4], and financial markets [5].

Estimating precision matrices under MTP 2 constraints is an active research topic. Recent results in [2, 4, 6] show that MTP 2 constraints lead to a drastic reduction on the number of observations required for the maximum likelihood estimator (MLE) to exist in Gaussian distributions and Ising models. This advantage is crucial in high-dimensional regimes with limited observations. Growing interest in estimating precision matrices under MTP 2 constraints is seen in graph signal processing [7-9]. A precision matrix fulfilling MTP 2 constraints can be regarded as a generalized graph Laplacian, where eigenvalues and eigenvectors are interpreted as spectral frequencies and Fourier bases, facilitating the computation of graph Fourier transform [10]. The MTP 2 property has also been studied in portfolio allocation [5] and structure recovery [11, 12].

Estimating precision matrices under MTP 2 constraints can be formulated as a sign-constrained log-determinant program. Existing algorithms for solving this problem, such as the block coordinate descent (BCD) [2, 3, 8] and proximal point algorithm (PPA) [9], are efficient for low-dimensional

problems. However, they become time-consuming in high-dimensional scenarios due to the necessity of solving numerous nonnegative quadratic programs or large-scale linear systems. An alternative is the gradient projection method [13], which offers computational efficiency per iteration. Nevertheless, it often grapples with slow convergence rates in high-dimensional cases. Thus, there is a demand for efficient and scalable algorithms for precision matrix estimation under MTP 2 constraints.

1.1 Contributions

In this paper, we propose a fast projected Newton-like algorithm for estimating precision matrices under MTP 2 constraints. While second-order algorithms generally require fewer iterations than first-order methods, they often encounter computational challenges due to the necessity of computing a large (approximate) Hessian matrix inverse or equivalently solving the linear system. Our main contributions in this paper are threefold:

Our proposed algorithm, rooted in the two-metric projection method, stands apart from current BCD and PPA approaches [2, 3, 8, 9] for solving the target problem. Utilizing a well-designed search direction and variable partitioning scheme, our algorithm avoids the need to solve nonnegative quadratic programs or linear systems, yielding a significant computational reduction compared to BCD and PPA algorithms. As a second-order method, our algorithm maintains the same per-iteration computational complexity as the gradient projection method.

We establish that our algorithm converges to the minimizer of the target problem. Furthermore, under a mild assumption, we prove the convergence of the set of free variables to the support of the minimizer within finite iterations and provide the convergence rate of our algorithm.

Numerical experiments on both synthetic and real-world datasets provide compelling evidence that our algorithm converges to the minimizer considerably faster than the compared methods. We apply the proposed method to financial time-series data and observe significant performance in terms of modularity value on the learned financial networks.

̸

Notation: Lower case bold letters denote vectors and upper case bold letters denote matrices. Both X ij and [ X ] ij denote the ( i, j ) -th entry of X . [ p ] denotes the set { 1 , . . . , p } , and [ p ] 2 denotes the set { 1 , . . . , p } × { 1 , . . . , p } . Let ⊗ be the Kronecker product and ⊙ be the entry-wise product. supp( X ) = { ( i, j ) ∈ [ p ] 2 | X ij = 0 } . ∥ X ∥ max = max i,j | X ij | and ∥ X ∥ min = min i,j | X ij | . S p + and S p ++ denote the sets of symmetric positive semi-definite and positive definite matrices with dimension p × p . vec( X ) and X ⊤ denote the vectorized version and transpose of X .

2 Problem Formulation and Related Work

In this section, we first introduce the problem formulation, then present related works.

2.1 Problem formulation

Let y = ( y 1 , . . . , y p ) be a p -dimensional random vector following y ∼ N ( 0 , Σ ) , where Σ is the covariance matrix. We focus on the problem of estimating the precision matrix Θ := Σ -1 given n i.i.d. observations y (1) , . . . , y ( n ) . Let S = 1 n ∑ n i =1 y ( i ) ( y ( i ) ) ⊤ be the sample covariance matrix. Throughout the paper, the sample covariance matrix is assumed to have strictly positive diagonal elements, which holds with probability one. This is because some diagonal element S jj is zero if and only if the j -th element of y ( i ) must be zero for every i ∈ [ p ] , which holds with probability zero.

We consider solving the following sign-constrained log-determinant program:

̸

where λ ij is the regularization parameter, E is the disconnectivity set with each node pair forced to disconnect, and M p is the set of all p -dimensional, symmetric, non-singular M-matrices defined by

̸

The disconnectivity set in (1) can be obtained in several ways: (i) it is often the case that some edges between nodes must not exist due to prior knowledge; (ii) it can be estimated from initial estimators; (iii) it can be obtained in some tasks of learning structured graphs such as bipartite graph [14-16].

2.2 Related work

Estimating precision matrices under Gaussian graphical models has been extensively studied in the literature. One well-known method is graphical lasso [17-20], which minimizes the ℓ 1 -regularized Gaussian negative log-likelihood. Various algorithms were proposed to solve this problem including first-order methods [17, 18, 21-29] and second-order methods [30-33]. The graphical lasso optimization problem is unconstrained and nonsmooth, while Problem (1) is smooth and constrained. The difficulties in solving the two problems are inherently different, and the algorithms mentioned above cannot be directly extended to solve Problem (1).

Recent studies [2, 3, 8, 9] employed BCD and PPA-type algorithms to estimate precision matrices under MTP 2 constraints. In [2], the primal variable is updated one column/row at a time by solving a nonnegative quadratic program, cycling until convergence. The work [8] follows a similar approach but addresses the dual problem, improving memory efficiency. Both works target Problem (1) without the disconnectivity constraint. A BCD-type algorithm was proposed in [3] to accommodate disconnectivity constraints. However, the computational complexity of these algorithms, at O ( p 4 ) operations per cycle, becomes prohibitive for high-dimensional problems. Alternatively, recent work [9] introduced an inexact PPA algorithm to solve a transformed problem, derived using the soft-thresholding technique. However, this algorithm demands the computation of an inexact Newton direction from a p 2 × p 2 linear system at every iteration within the inner loop, presenting computational difficulties in high-dimensional scenarios.

The proposed algorithm adopts the two-metric projection framework [34], incorporating distinct metrics for search direction and projection. A representative method in this framework is the projected Newton algorithm [35], originally designed for nonnegativity constrained problems. However, it is unsuitable for Problem (1) due to its O ( p 6 ) operations needed to compute the inverse of the Hessian at each iteration. To mitigate computation and memory costs, the projected quasi-Newton algorithm with limited-memory Boyden-Fletcher-Goldfarb-Shanno (L-BFGS) was introduced in [36], requiring O (( m + p ) p 2 ) operations per iteration, with m the number of iterations stored for Hessian approximation. By leveraging the structure of Problem (1), this paper carefully designs the search direction and variable partitioning scheme to substantially reduce computation and memory costs, achieving the same orders per iteration as the gradient projection method [13].

3 Proposed Algorithm

In this section, we propose a fast projected Newton-like algorithm to solve Problem (1). The constraints in Problem (1) can be rewritten as X ∈ Ω ∩ S p ++ , where Ω is defined as

̸

The set Ω is convex and closed, thus this constraint can be handled by a projection P Ω defined by

̸

The positive definite set S p ++ is not closed and cannot be managed by a projection, which will be handled using the line search method in Section 3.3. Let f denote the objective function of Problem (1). To address Problem (1), we start with the gradient projection method, expressed as:

where γ k is the step size. To accelerate convergence, one may consider

in which P k ∈ R p × p is a search direction defined by

where M k ∈ R p 2 × p 2 is a positive definite symmetric matrix, incorporating second-order derivative information. If we adopt M k as the Hessian matrix, then P k becomes the Newton direction as shown in Proposition 3.1, with the proof provided in Appendix B.1, and iterate (5) can be viewed as a natural adaptation of the unconstrained Newton's method. Regrettably, the convergence of such an iterate to the minimizer cannot be guaranteed, as P k may not be a descent direction here, which is supported by numerical results in Section 5.1.1. Similar observations have also been reported in [35, 37].

Proposition 3.1. If M k is constructed as the Hessian matrix H k of Problem (1) , then the search direction P k defined in (6) can be written as P k = X k ∇ f ( X k ) X k .

Remark 3.2 . We refer to iterate (5) as the two-metric projection method [34], as it adopts two distinct metrics: the search direction P k induced by the quadratic norm ∥·∥ M k defined by ∥ A ∥ M k = ⟨ vec ( A ) , M k vec ( A ) ⟩ 1 2 , and the projection P Ω with respect to the Frobenius norm ∥·∥ F .

3.1 Identifying the sets of restricted and free variables

In order to guarantee iterate (5) to converge to the minimizer, we partition the variables into two groups, i.e. , restricted and free variables, and update the two groups separately. We first define a set T ( X , ϵ ) with respect to X ∈ S p ++ and ϵ ∈ R + ,

Then at the k -th iteration, we identify the set of restricted variables based on X k as follows,

where E is the disconnectivity set from Problem (1), and ϵ k is a small positive scalar. For any ( i, j ) ∈ T ( X k , ϵ k ) , X ij in the next iterate is likely to be outside the feasible set ( i.e. , X ij > 0 ) if we remove the projection P Ω , as it is near zero and moves towards the positive direction if using the negative of the gradient as the search direction. Therefore, we set all restricted variables to zero.

To establish the theoretical convergence of the algorithm, the positive scalar ϵ k in (8) is specified as

where m is a positive scalar (See Lemma B.1 in Appendix), α ∈ (0 , 1) is a parameter in the line search condition, and T δ represents the set T ( X k , δ ) . In the rare event that T δ is empty, particularly in sparse settings, we define ϵ k = δ , implying an empty T ( X k , ϵ k ) in (8). The parameter δ satisfies 0 < δ < min ( i,j ) ∈ supp( X ⋆ ) | [ X ⋆ ] ij | , where X ⋆ is the minimizer of Problem (1). Such a condition can be ensured by setting a sufficiently small positive δ . Then ϵ k in (9) is nearly equal to δ . From an implementation view, we can directly set a small positive ϵ k , resulting in the algorithm performing well in practice. The set of free variables, denoted by I c k , is the complement of I k .

3.2 Computing approximate Newton direction

While the (approximate) Newton direction usually demands a considerably higher computational cost than the gradient, our designed direction maintains the same computational order as the gradient.

At the k -th iteration, we first partition X k into two groups, [ X k ] I k and [ X k ] I c k , where [ X k ] I k ∈ R |I k | and [ X k ] I c k ∈ R |I c k | denote two vectors containing all elements of X k in the sets I k and I c k , respectively. Then we can rewrite the search direction P k in (6) as follows,

where pvec k ( P k ) stacks P k into a vector, similar to vec( P k ) , but places elements from I c k first, followed by those from I k . Q k is obtained by permuting the rows and columns of M -1 k in (6). To enhance computational efficiency, we propose constructing Q k and M -1 k as follows:

where D k ∈ R |I k |×|I k | is a positive definite diagonal matrix, and [ H -1 k ] I c k I c k ∈ R |I c k |×|I c k | is a principal submatrix of H -1 k , preserving rows and columns indexed by I c k . Here, H k is the Hessian matrix at X k . The construction of M -1 k in (11) is crucial for defining the search direction, enabling computation and memory costs comparable to the gradient projection method while effectively incorporating second-order derivative information.

Next, we compute the approximate Newton direction P k over the set I c k and present the iterate [ X k +1 ] I c k . We define a projection P I c k ( A ) as follows,

Leveraging the well-designed gradient scaling matrix Q k in (11), we can efficiently compute the approximate Newton direction, as demonstrated in Proposition 3.3, with proof in Appendix B.2.

Proposition 3.3. If M k is constructed by (11) , then the search direction P k defined in (6) over I c k can be written as [ P k ] I c k = [ X k P I c k ( ∇ f ( X k )) X k ] I c k .

Using the search direction from Proposition 3.3, we update X k +1 over I c k as follows,

For the restricted variables in the set I k , we directly set them to zero, i.e. , [ X k +1 ] I k = 0 .

3.3 Computing step size

We adopt an Armijo-like rule for step size selection, ensuring the global convergence of our algorithm. Based on the iterate proposed in Section 3.2, we define X k ( γ k ) with [ X k ( γ k )] I k = 0 and

We test step sizes γ k ∈ { β 0 , β 1 , β 2 , . . . } with β ∈ (0 , 1) , until we find the smallest t ∈ N such that X k ( γ k ) , with γ k = β t , satisfies X k ( γ k ) ∈ S p ++ and the line search condition:

where α ∈ (0 , 1) is a scalar. We then set X k +1 = X k ( γ k ) . Positive definiteness of X k +1 can be verified during Cholesky factorization for objective function evaluation. It is worth mentioning that working with the positive semi-definiteness constraint on X instead of positive definiteness would not change anything in the algorithm if we keep the line search, as the positive definiteness is automatically enforced due to the form of the objective function.

The line search condition (15) is a variant of the Armijo rule. Condition (15) can be always satisfied for a small enough step size as shown in Proposition 3.4. Define the feasible set of Problem (1) as

For any given X o ∈ U p , define the lower level set of the objective function f for Problem (1) as:

Proposition 3.4. For any X k ∈ L f , there exists a ¯ γ k > 0 such that X k ( γ k ) ∈ S p ++ and the line search condition (15) holds for any γ k ∈ (0 , ¯ γ k ) .

The proof of Proposition 3.4 is available in Appendix B.3. We demonstrate that X k ( γ k ) ensures a decrease of the objective function value in Proposition 3.5, proved in Appendix B.4.

Proposition 3.5. For any X k ∈ L f , if X k ( γ k ) satisfies the line search condition (15) , then we have

where m is a positive scalar (See Lemma B.1 in Appendix).

3.4 Computation and memory costs

In each iteration, our algorithm calculates the gradient, performs two matrix multiplications, and conducts two projections, with respective computational costs of O ( p 3 ) , O ( p 3 ) , and O ( p 2 ) . In our current implementation of the line search method, we first conduct the Cholesky factorization X = LL ⊤ using MATLAB's 'chol' function. This function can simultaneously verify the positive definiteness of X . Next, we calculate the log-determinant function as log det( X ) = 2 ∑ i log( L ii ) . The Cholesky factorization is the most computationally demanding step, generally requiring O ( p 3 ) costs for a p × p matrix.

Algorithm 1 Fast Projected Newton-like (FPN) algorithm

To mitigate the computational burden associated with Cholesky factorization, we suggest a more efficient method for evaluating the log-determinant function and verifying positive definiteness, as presented in [38]. This method, which leverages Schur complements and sparse linear system solving, can tackle problems of up to 10 6 dimensions. Furthermore, it is worthwhile to investigate more efficient strategies for computing an approximate log-determinant function. In this context, the approach proposed in [39] offers a nearly linear scaling of execution time with the number of non-zero entries, while maintaining a high level of accuracy.

In summary, our algorithm has an overall complexity of O ( p 3 ) per iteration. BCD-type algorithms [2, 3, 8] need O ( p 4 ) operations per cycle, while projected quasi-Newton with L-BFGS [36] requires O (( m + p ) p 2 ) operations per iteration, with m as the stored iteration count for Hessian approximation. The PPA algorithm [9] requires computing an inexact Newton direction from a p 2 × p 2 linear system during each inner loop iteration, with the exact complexity not established. In addition, our algorithm, gradient projection and BCD-type methods [2, 3, 8] need O ( p 2 ) memory costs, while projected quasi-Newton with L-BFGS [36] requires O ( mp 2 ) and PPA [9] demands O ( p 4 ) .

4 Convergence Analysis

Prior to delving into the convergence analysis, we first establish the uniqueness of the minimizer for Problem (1) and determine the sufficient and necessary conditions for a point to be the minimizer.

Theorem 4.1. The minimizer of Problem (1) is unique, and a point X ⋆ ∈ M p is the minimizer if and only if it satisfies

The proof of Theorem 4.1 is available in Appendix B.5. The following theorem shows that our algorithm converges to the minimizer of Problem (1).

Theorem 4.2. The sequence { X k } generated by Algorithm 1 with X 0 ∈ L f converges to the minimizer X ⋆ of Problem (1) , with { f ( X k ) } monotonically decreasing.

The proof of Theorem 4.2 is available in Appendix B.6. It is worth noting that constructing an initial point X 0 ∈ L f , as defined in (17), is straightforward. Please refer to the proof of Theorem 4.2 for more details on this. The theoretical analysis on support set convergence and sequence convergence rate relies on the following assumption.

Assumption 4.3. For the minimizer X ⋆ of Problem (1), we assume that the gradient of the objective function f at X ⋆ satisfies

where V = { ( i, j ) ∈ [ p ] 2 ∣ ∣ [ X ⋆ ] ij = 0 } , and E is the disconnectivity set.

Theorem 4.4. Under Assumption 4.3, the set of free variables I c k generated by Algorithm 1 converges to the support of the minimizer X ⋆ of Problem (1) in finite iterations. In other words, there exists some k o ∈ N + such that I c k = supp( X ⋆ ) for any k ≥ k o .

The proof of Theorem 4.4 is provided in Appendix B.7. Theorem 4.4 demonstrates that the set of free variables constructed by our variable partitioning scheme can exactly identify the support of X ⋆ in finite iterations. Now we establish the convergence rate of our algorithm. Define

Theorem 4.5. Under Assumption 4.3, the sequence { X k } generated by Algorithm 1 satisfies

where m k and M k as the smallest and largest eigenvalues of R -1 2 k [ H k ] I c k I c k R -1 2 k , respectively.

The proof of Theorem 4.5 is provided in Appendix B.8. Theorem 4.5 reveals that the convergence rate of our algorithm depends on the condition number m k /M k of R -1 2 k [ H k ] I c k I c k R -1 2 k . Replacing R k with an identify matrix, ( i.e. , using the projected gradient method) results in a rate dependent on the condition number of [ H k ] I c k I c k . The condition number of R -1 2 k [ H k ] I c k I c k R -1 2 k could be larger, as R k could approximate [ H k ] I c k I c k well. Thus, the gradient scaling matrix R -1 k , i.e. , [ H -1 k ] I c k I c k in (11), leads our algorithm to converge faster than the projected gradient method.

It is important to note that our algorithm generally does not achieve superlinear convergence, despite the incorporation of second-order information. Superlinear convergence necessitates that the inverse gradient scaling matrix progressively approximate the Hessian at the minimizer [40]. However, this is a condition that our constructed scaling matrix does not meet. Despite this, we should note that constructing a search direction to achieve superlinear convergence proves significantly more computationally demanding than our approach, as it cannot leverage the special structure of the Hessian to decrease the computational load.

5 Experimental Results

We conduct experiments on synthetic and real-world data to verify the performance of our algorithm. All experiments were conducted on 2.10GHZ Xeon Gold 6152 machines and Linux OS, and all methods were implemented in MATLAB. State-of-the-art methods for comparisons include:

BCD [2]: Updates each column/row of primal variable using a nonnegative quadratic program.

optGL [8]: Similar to BCD but solves nonnegative quadratic programs on the dual variable.

GGL [3]: Similar to BCD but handles disconnectivity constraints, while BCD and optGL cannot.

PGD [13]: Projected gradient descent method with backtracking line search.

APGD [41]: Accelerated projected gradient algorithm with extrapolation step.

PPA [9]: Inexact proximal point algorithm with Newton-CG method.

PQN-LBFGS[36]: Projected quasi-Newton method using limited-memory BFGS.

Note that all state-of-the-art methods listed above can converge to the minimizer of Problem (1), and we focus on the comparison of computational time for those methods. To that end, we report the relative error of the objective function value as a function of the run time, which is calculated by

where f is the objective function of Problem (1), and X ⋆ is its minimizer. The X ⋆ is computed by running the state-of-the-art method GGL [3] until it converges to a point X k ∈ M p satisfying

where A := { ( i, j ) ∈ [ p ] 2 ∣ ∣ | [ X k ] ij | ≤ 10 -8 } . Through the comparison with the sufficient and necessary conditions of the unique minimizer of Problem (1) presented in Theorem 4.1, we can see that any point X k satisfying the conditions in (21) is sufficiently close to the minimizer.

(a) BA graph of degree one

(b) BA graph of degree two

It erations

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

10 0

10 1

10 2

0

10

20

30

40

50

Newton direction

Proposed

(c) Convergence comparison

5.1 Synthetic data

We generate independent samples y (1) , . . . , y ( n ) ∈ R p from a multivariate Gaussian distribution with zero mean and precision matrix Θ , where Θ ∈ M p is the underlying precision matrix associated with a graph consisting of p nodes. We use the Barabási-Albert (BA) model [42] to generate the support of the underlying precision matrix. To help readers to know well the BA graphs, we present two examples in Figure 1. More details about experimental setting are provided in Appendix A.

5.1.1 Comparisons of search directions

We evaluate the convergence of algorithms with different search directions. Figure 1 (c) demonstrates that our algorithm, using the direction from Proposition 3.3, converges to the minimizer, aligning with our theoretical convergence results in Theorem 4.2. In contrast, the algorithm using iterate (5) and the Newton direction from Proposition 3.1 stops decreasing the objective function value after a few iterations, indicating that this direction cannot be consistently regarded as a descent direction. The algorithm selects the step size through the Armijo rule, i.e. , it is continually reduced until a decrease in the objective function is achieved.

5.1.2 Comparisons of computational time

We evaluate the computational time of our algorithm and state-of-the-art methods on synthetic datasets, averaging results over 10 realizations. We plot markers every 10 iterations for PGD, APGD, PQN-LBFGS, and FPN, while marking each cycle of updating all columns/rows for BCD-type algorithms (BCD, GGL, and optGL) and each outer iteration for PPA.

Figure 2 compares the computational time of various methods for solving Problem (1) on BA graphs of degree one. Our proposed FPN significantly outperforms all state-of-the-art methods in convergence time for node counts ranging from 1000 to 5000 . BCD and GGL are efficient at 1000 nodes, being faster than PGD and APGD, and competitive with PQN-LBFGS and PPA. However, at 5000 nodes, BCD and GGL become slower due to the O ( p 4 ) operations per cycle required to solve p nonnegative quadratic programs, leading to rapidly increasing computational costs.

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

10

20

30

40

50

60

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(a)

p

= 1000

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

200

400

600

800

1000

1200

1400

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(b)

p

= 3000

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

600

1200

1800

2400

3000

3600

4200

4800

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(c)

p

= 5000

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

10

20

30

40

50

60

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(a)

p

= 1000

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

150

300

450

600

750

900

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(b)

p

= 3000

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

700

1400

2100

2800

3500

4200

4900

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(c)

p

= 5000

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

10

20

30

40

50

60

GGL

PPA

PGD

PQN-LBFGS

Proposed

(a)

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

10

20

30

40

50

GGL

PPA

PGD

PQN-LBFGS

Proposed

(b)

Number of nonzero elements

Run time (seconds)

10 3

2

·

10 3

3

·

10 3

5

·

10 3

10 4

2

·

10 4

0

50

100

150

200

250

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(c)

Figure 3 presents the computational time of different methods on BA graphs of degree two. As with degree one graphs, FPN outperforms state-of-the-art methods in computational time for varying node counts. PQN-LBFGS and FPN require fewer iterations to converge to the minimizer than PGD and APGD, particularly in high dimensions ( e.g. , p = 5000 ), indicating faster convergence. This is because both PQN-LBFGS and FPN utilize second-order information and approximate Newton direction, overcoming low convergence rates of first-order methods in high-dimensional cases.

Figure 4 (a) and (b) compare the computational time of various algorithms solving Problem (1) with disconnectivity constraints, where FPN consistently converges fastest. (c) evaluates the impact of the estimated precision matrix's sparsity level on run time. BCD, GGL, PPA, and FPN exhibit stable run time across varying sparsity levels, highlighting their robustness regarding regularization parameter settings, while other methods display increased run time as sparsity decreases.

5.2 Real-world data

We perform experiments on two real-world datasets: the concepts dataset and a financial time-series dataset. For the concepts dataset, we compare the computational time of different algorithms solving Problem (1). The experimental results on the financial time-series dataset are provided in Appendix A, where we examine the performance of our method in graph edge recovery.

The concepts dataset [43], from Intel Labs, comprises 1000 nodes and 218 semantic features, with p = 1000 and n = 218 . Nodes represent concepts like "house," "coat," and "whale," while semantic features are questions like "Can it fly?", "Is it alive?", and "Can you use it?". Responses, collected via Amazon Mechanical Turk, range from "definitely no" to "definitely yes" on a five-point scale.

Figure 5 (a) compares the run time of various algorithms solving Problem (1) on the concepts dataset. Our proposed algorithm converges to the minimizer considerably faster than state-of-the-art algorithms, which is consistent with the observations in synthetic experiments. Note that all compared algorithms can reach the minimizer of Problem (1), and thus learn the same graph.

CPU time (seconds)

|

f

(

X

k

) -

f

(

X

*

) | /  |

f

(

X

*

) |

10 - 10

10 - 9

10 - 8

10 - 7

10 - 6

10 - 5

10 - 4

10 - 3

10 - 2

10 - 1

0

10

20

30

40

50

60

GGL

BCD

optGL

PPA

PGD

PQN-LBFGS

Proposed

(a)

baby

girl

human

man

person

wife

body

woman

circus

band

kid

boy

parade

child

husband

crowd

tribe

male

gang

pony

cat

dog

kitten

puppy

pet

rat

squirrel

cow

horse

hog

donkey

pig

lamb

goat

rabbit

buffalo ox

gazelle

moose

calf

deer

elephant

antelope

giraffe

unicorn

kangaroo

sheep

zebra

cattle

bear

ape

lion

monkey

skunk

otter

raccoon

fox

leopard

gorilla

tiger

rhinoceros

hippo

walrus

whale

alligator

seal

dinosaur

dolphin

shark

stingray

octopus

crocodile

goldfish

fish

snake

haddock

trout

salmon

rattlesnake

clam shrimp

lobster

reptile

turtle

chicken

parrot

lizard

salamander

frog

toad

penguin goose

hen

rooster

sparrow ostrich

peacock

hawk

bird robin

duck

bluebird

seagull

eagle

crow

pigeon

vulture

owl

grasshopper

ladybug

moth

beetle

fly

insect

cricket

flea

earthworms

bee

worm

cockroach

butterfly

ant

caterpillar

spider

snail

(b)

Figure 5 (b) displays a connected subgraph illustrated by the minimizer of Problem (1). Interestingly, it is observed that the learned graph forms a semantic network, where related concepts are closely connected. For instance, insect concepts such as 'bee', 'butterfly', 'flea', 'mosquito', and 'spider' are grouped together, while human-related concepts like 'baby', 'husband', 'child', 'girls', and 'man' form another group. Moreover, the network connects 'penguin' closely to birds like 'owl' and 'crow' and sea animals like 'goldfish' and 'seal', highlighting its aquatic bird nature. Overall, the learned network effectively captures concept relationships.

6 Conclusions and Discussions

In this paper, we have introduced a fast projected Newton-like method for estimating precision matrices under MTP 2 constraints. Our algorithm, leveraging the two-metric projection method, stands out from existing BCD and PPA-type approaches for addressing the target problem. The proposed algorithm is not only straightforward to implement but also efficient in terms of computation and memory usage. We have provided theoretical convergence analysis and conducted extensive experiments, which clearly demonstrate the superior efficiency of our algorithm in computational time, outperforming state-of-the-art methods. Moreover, we have observed significant performance of our method in terms of modularity value on the learned financial time-series graphs.

Finally, we discuss the limitations of our paper. Our algorithm is proven to converge to the minimizer without any assumptions; however, we require Assumption 4.3 to establish support set convergence in finite iterations and to determine the convergence rate. This assumption is relatively mild, as Theorem 4.1 shows that the minimizer X ⋆ must satisfy [ ∇ f ( X ⋆ )] ij ≤ 0 for each ( i, j ) ∈ V\E . The only additional requirement in Assumption 4.3 is the strictness of this inequality. However, the conditions for ensuring this strictness remain unclear. As this assumption is equivalent to the strict complementary slackness condition in optimization theory, exploring verifiable conditions to guarantee Assumption 4.3 could enrich our algorithm's insights.

References

S. Fallat, S. Lauritzen, K. Sadeghi, C. Uhler, N. Wermuth, and P. Zwiernik, 'Total positivity in Markov structures,' The Annals of Statistics , vol. 45, no. 3, pp. 1152-1184, 2017.

M. Slawski and M. Hein, 'Estimation of positive definite M-matrices and structure learning for attractive Gaussian Markov random fields,' Linear Algebra and its Applications , vol. 473, pp. 145-179, 2015.

H. E. Egilmez, E. Pavez, and A. Ortega, 'Graph learning from data under Laplacian and structural constrints,' IEEE Journal of Selected Topics in Signal Processing , vol. 11, no. 6, pp. 825-841, 2017.

S. Lauritzen, C. Uhler, and P. Zwiernik, 'Maximum likelihood estimation in Gaussian models under total positivity,' The Annals of Statistics , vol. 47, no. 4, pp. 1835-1863, 2019.

R. Agrawal, U. Roy, and C. Uhler, 'Covariance Matrix Estimation under Total Positivity for Portfolio Selection,' Journal of Financial Econometrics , vol. 20, no. 2, pp. 367-389, 09 2020.

S. Lauritzen, C. Uhler, and P. Zwiernik, 'Total positivity in exponential families with application to binary variables,' The Annals of Statistics , vol. 49, no. 3, pp. 1436-1459, 2021.

E. Pavez, H. E. Egilmez, and A. Ortega, 'Learning graphs with monotone topology properties and multiple connected components,' IEEE Transactions on Signal Processing , vol. 66, no. 9, pp. 2399-2413, 2018.

E. Pavez and A. Ortega, 'Generalized Laplacian precision matrix estimation for graph signal processing,' in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing , 2016, pp. 6350-6354.

Z. Deng and A. M.-C. So, 'A fast proximal point algorithm for generalized graph laplacian learning,' in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 5425-5429.

D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, 'The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains,' IEEE Signal Processing Magazine , vol. 30, no. 3, pp. 83-98, 2013.

J. Kelner, F. Koehler, R. Meka, and A. Moitra, 'Learning some popular Gaussian graphical models without condition number bounds,' in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 10 986-10 998.

Y. Wang, U. Roy, and C. Uhler, 'Learning high-dimensional Gaussian graphical models under total positivity without adjustment of tuning parameters,' in International Conference on Artificial Intelligence and Statistics , vol. 108, 2020, pp. 2698-2708.

J. Ying, J. V . d. M. Cardoso, and D. P. Palomar, 'Adaptive estimation of graphical models under total positivity,' in International Conference on Machine Learning , 2023, pp. 40 054-40 074.

S. Kumar, J. Ying, J. V. d. M. Cardoso, and D. P. Palomar, 'A unified framework for structured graph learning via spectral constraints,' Journal of Machine Learning Research , vol. 21, no. 22, pp. 1-60, 2020.

J. V. d. M. Cardoso, J. Ying, and D. P. Palomar, 'Learning bipartite graphs: Heavy tails and multiple components,' in Advances in Neural Information Processing Systems , vol. 35, 2022, pp. 14 044-14 057.

S. Kumar, J. Ying, J. V. d. M. Cardoso, and D. P. Palomar, 'Structured graph learning via Laplacian spectral constraints,' in Advances in Neural Information Processing Systems , 2019, pp. 11 647-11 658.

O. Banerjee, L. E. Ghaoui, and A. d'Aspremont, 'Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data,' Journal of Machine Learning Research , vol. 9, no. Mar, pp. 485-516, 2008.

A. d'Aspremont, O. Banerjee, and L. El Ghaoui, 'First-order methods for sparse covariance selection,' SIAM Journal on Matrix Analysis and Applications , vol. 30, no. 1, pp. 56-66, 2008.

P. Ravikumar, M. J. Wainwright, G. Raskutti, and B. Yu, 'High-dimensional covariance estimation by minimizing ℓ 1 -penalized log-determinant divergence,' Electronic Journal of Statistics , vol. 5, pp. 935-980, 2011.

M. Yuan and Y. Lin, 'Model selection and estimation in the Gaussian graphical model,' Biometrika , vol. 94, no. 1, pp. 19-35, 2007.

J. Duchi, S. Gould, and D. Koller, 'Projected subgradient methods for learning sparse Gaussians,' in Conference on Uncertainty in Artificial Intelligence , 2008, p. 153-160.

J. Friedman, T. Hastie, and R. Tibshirani, 'Sparse inverse covariance estimation with the graphical lasso,' Biostatistics , vol. 9, no. 3, pp. 432-441, 2008.

Z. Lu, 'Smooth optimization approach for sparse covariance selection,' SIAM Journal on Optimization , vol. 19, no. 4, pp. 1807-1827, 2009.

Q. Sun, K. M. Tan, H. Liu, and T. Zhang, 'Graphical nonconvex optimization via an adaptive convex relaxation,' in Proceedings of the 35th International Conference on Machine Learning , vol. 80, 2018, pp. 4810-4817.

K. Scheinberg, S. Ma, and D. Goldfarb, 'Sparse inverse covariance selection via alternating linearization methods,' in Advances in Neural Information Processing Systems , vol. 23, 2010, pp. 2101-2109.

A. Eftekhari, D. Pasadakis, M. Bollhöfer, S. Scheidegger, and O. Schenk, 'Block-enhanced precision matrix estimation for large-scale datasets,' Journal of computational science , vol. 53, p. 101389, 2021.

J. Chen, P. Xu, L. Wang, J. Ma, and Q. Gu, 'Covariate adjusted precision matrix estimation via nonconvex optimization,' in Proceedings of the 35th International Conference on Machine Learning , vol. 80, 2018, pp. 921-930.

P. Xu, J. Ma, and Q. Gu, 'Speeding up latent variable Gaussian graphical model estimation via nonconvex optimization,' in Advances in Neural Information Processing Systems , vol. 30, 2017, pp. 1933-1944.

E. Treister and J. S. Turek, 'A block-coordinate descent approach for large-scale sparse inverse covariance estimation,' Advances in neural information processing systems , vol. 27, 2014.

Q. T. Dinh, A. Kyrillidis, and V. Cevher, 'A proximal Newton framework for composite minimization: Graph learning without Cholesky decompositions and matrix inversions,' in International Conference on Machine Learning , vol. 28, 2013, pp. 271-279.

C.-J. Hsieh, M. A. Sustik, I. S. Dhillon, and P. Ravikumar, 'QUIC: quadratic approximation for sparse inverse covariance estimation,' The Journal of Machine Learning Research , vol. 15, no. 1, pp. 2911-2947, 2014.

L. Li and K.-C. Toh, 'An inexact interior point method for L 1 -regularized sparse covariance selection,' Mathematical Programming Computation , vol. 2, no. 3, pp. 291-315, 2010.

C. Wang, D. Sun, and K.-C. Toh, 'Solving log-determinant optimization problems by a NewtonCG primal proximal point algorithm,' SIAM Journal on Optimization , vol. 20, no. 6, pp. 2994-3013, 2010.

E. M. Gafni and D. P. Bertsekas, 'Two-metric projection methods for constrained optimization,' SIAM Journal on Control and Optimization , vol. 22, no. 6, pp. 936-964, 1984.

D. P. Bertsekas, 'Projected Newton methods for optimization problems with simple constraints,' SIAM Journal on Control and Optimization , vol. 20, no. 2, pp. 221-246, 1982.

D. Kim, S. Sra, and I. S. Dhillon, 'Tackling box-constrained optimization via a new projected Quasi-Newton approach,' SIAM Journal on Scientific Computing , vol. 32, no. 6, pp. 3548-3563, 2010.

--, 'Fast Newton-type methods for the least squares nonnegative matrix approximation problem,' in SIAM International Conference on Data Mining , 2007, pp. 343-354.

C.-J. Hsieh, M. A. Sustik, I. S. Dhillon, P. K. Ravikumar, and R. Poldrack, 'BIG & QUIC: Sparse inverse covariance estimation for a million variables,' in Advances in Neural Information Processing Systems , vol. 26, 2013, pp. 3165-3173.

I. Han, D. Malioutov, and J. Shin, 'Large-scale log-determinant computation through stochastic Chebyshev expansions,' in International Conference on Machine Learning . PMLR, 2015, pp. 908-917.

J. Nocedal and S. Wright, Numerical optimization . Springer Science & Business Media, 2006.

Y. Nesterov, Introductory lectures on convex optimization: A basic course . Springer Science &Business Media, 2003, vol. 87.

A.-L. Barabási and R. Albert, 'Emergence of scaling in random networks,' Science , vol. 286, no. 5439, pp. 509-512, 1999.

B. Lake and J. Tenenbaum, 'Discovering structure by learning sparse graphs,' in Proceedings of the 33rd Annual Cognitive Science Conference , 2010, pp. 778--783.

J. Ying, J. V. d. M. Cardoso, and D. P. Palomar, 'Nonconvex sparse graph learning under Laplacian constrained graphical model,' in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 7101-7113.

E. J. Candes, M. B. Wakin, and S. P. Boyd, 'Enhancing sparsity by reweighted ℓ 1 minimization,' Journal of Fourier analysis and applications , vol. 14, no. 5, pp. 877-905, 2008.

J. Ying, J. V . d. M. Cardoso, and D. P. Palomar, 'Minimax estimation of Laplacian constrained precision matrices,' in International Conference on Artificial Intelligence and Statistics , vol. 130, 2021, pp. 3736-3744.

--, 'A fast algorithm for graph learning under attractive Gaussian Markov random fields,' in 2021 55th Asilomar Conference on Signals, Systems, and Computers , 2021, pp. 1520-1524.

M. E. Newman, 'Modularity and community structure in networks,' Proceedings of the National Academy of Sciences , vol. 103, no. 23, pp. 8577-8582, 2006.

R. J. Plemmons, 'M-matrix characterizations. I-nonsingular M-matrices,' Linear Algebra and its applications , vol. 18, no. 2, pp. 175-188, 1977.

P. Drábek and J. Milota, Methods of nonlinear analysis: applications to differential equations . Springer Science & Business Media, 2007.

D. P. Bertsekas, Nonlinear programming , 3rd ed. Athena Scientific, Belmont, MA, 2016.

--, 'On the Goldstein-Levitin-Polyak gradient projection method,' IEEE Transactions on Automatic Control , vol. 21, no. 2, pp. 174-184, 1976.

In the following sections, we provide more details on experimental settings and additional experimental results for financial time-series data in Appendix A and the proofs for Propositions 3.1, 3.3, 3.4, and 3.5, as well as Theorems 4.1, 4.2, 4.4, and 4.5 in Appendix B.

A Additional Experiments

We first provide an in-depth description of the experimental settings, then present numerical experiments carried out on the financial time-series data.

A.1 Experimental settings

We examine Barabasi-Albert (BA) graphs [42] as a model for the support structure of the underlying precision matrix. BA models hold a significant position in network science due to their ability to generate random scale-free networks via a preferential attachment mechanism. This mechanism ensures that newly introduced nodes are more likely to connect with nodes that possess a higher degree during the network's evolution. Scale-free networks serve as suitable models for various systems, including the Internet, protein interaction networks, citation networks, as well as the majority of social and online networks [42].

In a BA graph with degree r , each new node connects to r pre-existing nodes, with the probability of connection being proportional to the number of edges the existing nodes currently have. In this paper, we consider r values of 1 and 2.

Following the procedures detailed in [44], we assign a positive weight to each edge of a graph and set a zero weight for disconnected nodes. Positive weights are uniformly sampled from U (2 , 5) . This process results in a weighted adjacency matrix A containing all the graph weights. Then we adopt the procedures outlined in [2] for generating the underlying precision matrix Θ . We first set

where λ max ( A ) represents the largest eigenvalue of A . In this context, the weight is zero if two nodes are disconnected, and follows a uniform distribution U (2 , 5) when nodes are connected. Lastly, we define Θ = D ˜ Θ D , where D is a diagonal matrix selected such that the covariance matrix Θ -1 has unit diagonal elements.

We set the regularization parameter λ ij in Problem (1) as follows

̸

where ˆ X represents an estimator, ϵ is set to 10 -3 , and σ > 0 is a parameter that adjusts the sparsity. The function in (23) is closely related to the re-weighted ℓ 1 -norm regularization [14, 45, 46], which effectively enhances the sparsity of the solution and reduces the estimation bias resulting from the ℓ 1 -norm [47]. We employ the maximum likelihood estimator as ˆ X , i.e. , the minimizer of Problem (1) without the sparsity regularization. Note that one can solve the maximum likelihood estimator with a relatively large tolerance to obtain a coarse estimator, and alternative monotonically decreasing functions may be explored in (23). For PQN-LBFGS, we utilize the previous 50 updates to compute the search direction. For FPN, we set ϵ k = 10 -15 in (8) for identifying the set of restricted variables.

For calculating the relative error as defined in (20), there is no additional computational cost for PGD, APGD, PQN-LBFGS and FPN, since the objective function value is already evaluated within the backtracking line search during each iteration. In contrast, BCD, optGL, and GGL do not require evaluating the objective function, leading to an extra computational cost when computing the relative error in (20) for comparison purposes. However, this additional cost can be considered negligible, as these methods only compute the objective function value after completing a cycle. Moreover, the number of cycles needed for BCD, optGL, and GGL is substantially fewer than the number of iterations required by other methods.

A.2 Financial time-series data

We carry out numerical experiments on a financial time-series dataset to evaluate the performance of our method in recovering graph edges. The applicability of MTP2 models on financial time-series data is well-established, as market factors lead to positive dependencies among stocks [5].

The dataset consists of 201 stocks composing the S&P 500 index, spanning the period from January 1, 2017, to January 1, 2020, yielding 753 observations per stock, i.e. , p = 201 and n = 753 . We construct the log-returns data matrix as Xi,j = log P i,j -log P i -1 ,j , where P i,j represents the closing price of the j -th stock on the i -th day. The stocks are categorized into five sectors based on the Global Industry Classification Standard (GICS) system: Consumer Staples, Utilities, Industrials, Information Technology, and Energy.

Directly assessing the correctness of the learned graph edges is not feasible in financial time-series data, as the underlying graph structure remains unknown. However, we expect stocks from the same sector to have interconnected edges. To measure the performance of edge recovery, we employ the modularity metric [48]. Given a graph G = ( V, E ) , where V represents the vertex set and E denotes the edge set, the modularity is defined as:

where A ij = 1 if ( i, j ) ∈ E , and 0 otherwise. d i represents the number of edges connected to node i , c i indicates the type of node i , and δ ( · , · ) refers to the Kronecker delta function, with δ ( a, b ) = 1 if a = b and 0 otherwise. A stock graph with high modularity exhibits dense connections among stocks within the same sector and sparse connections between stocks in distinct sectors. A higher modularity value implies a more faithful representation of the underlying stock network.

(a)

Glasso

,

Q

= 0

.

47

(b)

FPN

,

Q

= 0

.

65

(c)

FPN

(

E

)

,

Q

= 0

.

67

Figure 6 demonstrates that the performances of our proposed FPN and FPN ( E ) are better than that of Glasso [22], since the majority of connections in graphs learned through FPN and FPN ( E ) occur between nodes within the same sector. In contrast, only a few connections (depicted as gray-colored edges) exist between nodes from different sectors. Both FPN and FPN ( E ) achieve higher modularity values compared to Glasso, indicating that the former have a higher degree of interpretability than the latter. Furthermore, we observe that FPN ( E ) moderately enhances the performance of FPN.

We fine-tune the sparsity regularization parameter for each method based on the modularity value, allowing only a limited number of isolated nodes. Note that increasing the regularization parameter for Glasso would result in numerous isolated nodes that cannot be grouped. FPN ( E ) refers to the application of FPN for solving Problem (1) with a disconnectivity set E . This set is obtained through hard thresholding on the MLE, which is also used in computing regularization weights in (23).

B Proofs

In this section, we provide the proofs for Propositions 3.1, 3.3, 3.4, and 3.5, as well as Theorems 4.1, 4.2, 4.4, and 4.5.

B.1 Proof of Proposition 3.1

Proof. The Hessian matrix at X k has the form:

Following from the property of the Kronecker product that vec ( ABC ) = ( C ⊤ ⊗ A ) vec( B ) , we can obtain

As a result, we have

completing the proof.

B.2 Proof of Proposition 3.3

Proof. Following from (10) and (11), we obtain

where projection P I c k is defined in (12). For ease of presentation, by a slight abuse of notation, both [ A ] I c k ∈ R |I c k | and [vec ( A )] I c k ∈ R |I c k | represent a vector containing all elements of A in the set I c k . Following from the fact that vec ( ABC ) = ( C ⊤ ⊗ A ) vec( B ) and H -1 k = X k ⊗ X k , we have

By collecting the elements in the set I c k , we obtain

completing the proof.

B.3 Proof of Propositio 3.4

To prove Propositio 3.4, we first establish Lemma B.1 below to show that the lower level set of the objective function is compact. We note that Lemma B.1 depends on the condition that the sample covariance matrix has strictly positive diagonal elements, which is assumed throughout the paper and holds with probability one.

Lemma B.1. The lower level set L f defined in (17) is nonempty and compact, and for any X ∈ L f , we have

where m and M are two positive scalars.

Proof. We first show that the largest eigenvalue of any X ∈ L f can be upper bounded by M . The objective function of (1) can be written as

where G = S -λ with λ defined by

Then we obtain

̸

Define two constants

̸

̸

Note that µ > 0 because S ii > 0 for each i ∈ [ p ] . Then one has

̸

̸

̸

where the inequality follows from the fact that X ij ≤ 0 for i = j , and X ij = 0 for ( i, j ) ∈ E . We denote the largest eigenvalue of X by λ max ( X ) . Then we have

̸

where the second inequality follows from (26). In what follows, we bound the terms tr ( XG ) and ∑ i = j X ij . For any X ∈ L f , one has f ( X o ) ≥ -log det( X ) + tr ( XG ) . Therefore, the term tr ( XG ) can be bounded by

̸

Let λ ′ := min i = j, ( i,j ) / ∈E λ ij . Then one has

̸

̸

̸

where the last inequality follows from ∑ i = j λ ij | X ij | ≤ tr ( XG ) , because tr ( XS ) ≥ 0 since X ∈ S p ++ and S ∈ S p + . Together with (27), (28) and (29), we obtain

Since log ( λ max ( X )) grows much slower than λ max ( X ) , λ max ( X ) can be upper bounded by a constant M , which depends on f ( X o ) , S , and λ .

We denote the smallest eigenvalue of X by λ min ( X ) . For any X ∈ L f , one has

As a result, we have λ min ( X ) ≥ e -f ( X o ) M -( p -1) , which shows that λ min ( X ) can be lower bounded by a positive constant m = e -f ( X o ) M -( p -1) . Finally, we show that the lower level set L f is compact. First, L f is closed because f is a continuous function. Second, L f is also bounded because it is a subset of { X ∈ R p × p | m I ⪯ X ⪯ M I } .

Proof of Propositio 3.4. We first show that X k ( γ k ) ∈ S ++ holds for a small enough step size. The X k ( γ k ) can be equivalently written in the form X k ( γ k ) = Z k -γ k G k , where Z k ∈ R p × p defined by [ Z k ] ij = [ X k ] ij if ( i, j ) ∈ I c k , and 0 otherwise. G k is some search direction only over I c k , which is a symmetric matrix. It is known that a matrix A is a nonsingular M-matrix if and only if A is a Z-matrix and there exists a vector x > 0 with Ax > 0 [49]. Following from the fact that X k is a nonsingular M-matrix , there exists x > 0 such that

Therefore, Z k is also a nonsingular M-matrix , implying that Z k ∈ S ++ . As a result, if γ k < λ min ( Z k ) /ρ ( G k ) , where ρ ( G k ) is the spectral radius of G k , then X k ( γ k ) ∈ S ++ . The definition of X k ( γ k ) further guarantees that X k ( γ k ) ∈ U p . We can verify that f ( X ) = + ∞ for any X ∈ cl( U p ) \ U p , where X is positive semidefinite and singular. Thus, we consider an X o ∈ U p with f ( X o ) is sufficiently large such that f ( X k ( γ k )) ≤ f ( X o ) . Therefore, X k ( γ k ) ∈ L f .

Next we prove that the line search condition (15) holds for a small enough step size. Recall that I c k is the complement of the set I k defined in (8). For any X k ∈ L f , the set I c k can be represented as

where B ( l ) , l = 1 , . . . , 5 , are defined as

Each subset B ( l ) k is disjoint with each other. B (5) k only contains the indexes of the diagonal elements of X k , whiles the other subsets include the indexes of the off-diagonal elements. Then one has, for any ( i, j ) ∈ B (1) k and γ k > 0 ,

For any ( i, j ) ∈ B (2) k and γ k > 0 , one has [ X k ( γ k ) ] ij -[ X k ] ij = -γ k [ P k ] ij . For any ( i, j ) ∈ B (3) k , if the step size satisfies

then one has [ X k ( γ k ) ] ij -[ X k ] ij = -γ k [ P k ] ij . Similarly, for any ( i, j ) ∈ B (4) k and γ k > 0 , [ X k ( γ k ) ] ij -[ X k ] ij = -γ k [ P k ] ij . Finally, for any ( i, j ) ∈ B (5) k , [ X k ] ij must be on the diagonal of X k . Therefore, we can directly remove the projection P Ω , and obtain [ X k ( γ k ) ] ij -[ X k ] ij = -γ k [ P k ] ij . Therefore, if (31) holds, one has

where the inequality follows from (30). Recall that [ X k ( γ k ) ] I k = 0 , which can be equivalently expressed as the form of projected gradient descent:

where we write P Ω ( [ A ] I k ) for [ P Ω ( A ) ] I k with a slight abuse of notation, and ˜ D k is defined as

Then one has

where a k = ϵ k ∥ ∥ [ ∇ f ( X k ) ] T k \E ∥ ∥ min , and T k represents T ( X k , ϵ k ) . Note that I k = T k ∪ E , and [ X k ] ij = [ X k ( γ k )] ij = 0 for any ( i, j ) ∈ E .

It is worth mentioning that we primarily focus on cases with non-empty T k . In the rare event that T k is empty, such as when T δ is empty (in this situation we define ϵ k = δ , leading to an empty T k ), we can simply ignore the terms related to I k in (34) and the following proof.

Furthermore, one has

The largest eigenvalue of [ M -1 k ] I c k I c k can be bounded by

where the first inequality follows from the Eigenvalue Interlacing Theorem, and the second inequality follows from Lemma B.1. Combining (34), (35) and (36) yields

Since X k ( γ k ) ∈ L f , we obtain

where the inequality follows from

where the inequality follows from Lemma B.1.

Let ¯ γ k = min ( 2(1 -α ) m 2 M 2 , min ( i,j ) ∈B (3) k ϵ k ∣ ∣ [ P k ] ij ∣ ∣ , λ min ( Z k ) ρ ( G k ) ) . Note that ¯ γ k is bounded away from zero, because each [ P k ] ij is bounded following from Lemma B.1. To this end, for any γ k ∈ (0 , ¯ γ k )

,

where the inequality follows from (32), the definition of ϵ k in (9), the inequality ∥ ∥ [ ∇ f ( X k ) ] T δ \E ∥ ∥ min ≤ ∥ ∥ [ ∇ f ( X k ) ] T k \E ∥ ∥ min , because T k ⊆ T δ , and the inequality 〈[ ∇ f ( X k ) ] I c k , [ P k ] I c k 〉 ≥ 0 , because [ M -1 k ] I c k I c k is positive definite and

completing the proof.

B.4 Proof of Proposition 3.5

Proof. The line search condition (15) leads to

where the inequality follows from

We further have

where the first and second inequalities follow from Eigenvalue Interlacing Theorem and Lemma B.1, respectively. We complete the proof by combining (40) and (41).

B.5 Proof of Theorem 4.1

Proof. We first prove that Problem (1) has at least one minimizer. It is known by the Weierstrass' extreme value theorem [50] that the set of minima is nonempty for any lower semicontinuous function with a nonempty compact lower level set. Therefore, the existence of the minimizers for Problem (1) can be guaranteed by Lemma B.1.

On the other hand, we show that Problem (1) has at most one minimizer by its strict convexity. We have ∇ 2 f ( X ) = X -1 ⊗ X -1 . Thus ∇ 2 f ( X ) ≻ 0 for any X in the feasible region of Problem

(1) defined in (16). Therefore, Problem (1) is strictly convex, and thus has at most one minimizer. Together with the existence of the minimizers, we conclude that Problem (1) has an unique minimizer.

The Lagrangian of Problem (1) is

where Y is a KKT multiplier with Y ii = 0 for i ∈ [ p ] , and λ is defined in (25).

For a convex optimization problem with Slater's condition holding, a pair is primal and dual optimal if and only if the KKT conditions hold. Thus, ( X ⋆ , Y ⋆ ) is primal and dual optimal if and only if it satisfies the KKT conditions of (1) as below,

̸

̸

̸

We first prove that the minimizer X ⋆ must satisfy all conditions in (18). Note that the KKT conditions (43)-(46) must hold for the minimizer X ⋆ . Let V = { ( i, j ) ∈ [ p ] 2 | X ⋆ ij = 0 } . First, X ⋆ ij = 0 for any ( i, j ) ∈ E following from (44). Second, for any ( i, j ) with i = j and ( i, j ) ∈ V c , we have X ⋆ ij = 0 and ( i, j ) / ∈ E . Following from (45), we further obtain Y ⋆ ij = 0 . Together with (46), we conclude that Y ⋆ ij = 0 for any ( i, j ) ∈ V c . Following from (43), [ ∇ f ( X ⋆ ) ] V c = 0 . Since X ⋆ is positive definite, ( i, i ) ∈ V c for any i ∈ [ p ] . Then, for any ( i, j ) ∈ V\E , we have i = j , thus obtain Y ⋆ ij ≥ 0 according to (45). Following from (43), we get [ ∇ f ( X ⋆ ) ] V\E ≤ 0 .

̸

Now we prove that any point X ⋆ ∈ M p satisfying the conditions in (18) must be the minimizer, i.e. , the KKT conditions (43)-(46) hold for X ⋆ . We construct Y ⋆ by Y ⋆ = -∇ f ( X ⋆ ) . First, it is straightforward to check that the conditions (43) and (44) hold. Second, we have

We know that ( i, i ) ∈ V c for any i ∈ [ p ] , since X ⋆ ∈ M p . Together with (47), we obtain that the condition (46) holds.

Finally, following from the fact that ( i, i ) ∈ V c for any i ∈ [ p ] and E ⊆ V , we obtain

̸

̸

where I 1 = { ( i, j ) ∈ [ p ] 2 | ( i, j ) ∈ V c , i = j } , and I 2 = { ( i, j ) ∈ [ p ] 2 | ( i, j ) ∈ V , ( i, j ) / ∈ E } . For any ( i, j ) ∈ I 1 , we have Y ⋆ ij = 0 according to (47), and X ⋆ ij < 0 since X ⋆ ∈ M p . Thus the condition (45) holds for any ( i, j ) ∈ I 1 . For any ( i, j ) ∈ I 2 , we have X ⋆ ij = 0 , and Y ⋆ ij = -[ ∇ f ( X ⋆ )] ij ≥ 0 according to the second condition in (18). Thus the condition (45) also holds for any ( i, j ) ∈ I 2 . Totally, the condition (45) holds. To sum up, all KKT conditions (43)-(46) hold for any X ⋆ ∈ M p satisfying the conditions in (18), and thus we conclude that X ⋆ is the minimizer of Problem (1).

B.6 Proof of Theorem 4.2

Proof. Following from Proposition 3.4, for any iterate X k ∈ L f , a small enough step size ensures that the line search condition (15) holds, which leads to a sufficient decrease of the objective function as shown in Theorem 3.5, and the next iterate X k +1 ∈ L f . When the sequence starts with X 0 ∈ L f , each point of the sequence { X k } k ≥ 0 admits X k ∈ L f . Note that it is easy to construct an initial point X 0 ∈ L f , because we could consider an X o ∈ U p in (17), which is close to cl( U p ) \ U p such that f ( X o ) sufficiently large, following from the fact that f ( X ) = + ∞ for any X ∈ cl( U p ) \ U p . LemmaB.1 shows that the lower level set L f is compact, thus the sequence { X k } has at least one limit point. For every limit X ⋆ of the sequence { X k } , we have X ⋆ ∈ M p . Define I ⋆ = T ( X ⋆ , ϵ ⋆ ) ∪ E , where T ( X ⋆ , ϵ ⋆ ) is equal to

and ϵ ⋆ is defined as ϵ ⋆ := min ( 2(1 -α ) m 2 ∥ ∥ [ ∇ f ( X ⋆ ) ] T ⋆ δ \E ∥ ∥ min , δ ) , where T ⋆ δ denotes T ( X ⋆ , δ ) . Since f is continuous and { f ( X k ) } keeps decreasing and is bounded, it follows that { f ( X k ) } converges and

Following from Proposition 3.4, the line search condition (15) ensures that each X k ∈ S p ++ and, for a strictly positive γ k , it holds

Recall that I k = T k ∪E . Wenote that the following facts hold: α ∈ (0 , 1) is a constant, [ M -1 k ] I c k I c k is positive definite, [ X k ] ij = 0 for any ( i, j ) ∈ E , [ X k ] ij ≤ 0 for any ( i, j ) ∈ I k \E , and [ ∇ f ( X k )] ij < 0 for any ( i, j ) ∈ I k \E due to the definition of T ( X , ϵ ) . Thus, the two terms on the right-hand side of (48) are both nonnegative. Moreover, the right-hand side of (48) approaches zero if and only if both [ ∇ f ( X k ) ] I c k and [ X k ] I k simultaneously go to zero. Therefore, we deduce that every limit point X ⋆ must satisfy:

We show that every limit point X ⋆ is the minimizer of Problem (1) according to Theorem 4.1. Let V = { ( i, j ) ∈ [ p ] 2 | [ X ⋆ ] ij = 0 } . First, for any ( i, j ) ∈ E , we have [ X ⋆ ] ij = 0 , because of the projection P Ω in each iteration. Note that [ X ⋆ ] ij = 0 for any ( i, j ) ∈ I ⋆ according to (49). For any ( i, j ) ∈ V c , i.e. , [ X ⋆ ] ij = 0 , we must have ( i, j ) ∈ {I ⋆ } c . Together with (49), [ ∇ f ( X ⋆ ) ] V c = 0 holds. For any ( i, j ) ∈ V\E , we must have

Recall that [ ∇ f ( X ⋆ ) ] ij < 0 for any ( i, j ) ∈ T ( X ⋆ , ϵ ⋆ ) , and [ ∇ f ( X ⋆ ) ] ij = 0 for any ( i, j ) {I ⋆ } c . Overall, we obtain

To sum up, all the conditions in Theorem 4.1 hold for every limit point X ⋆ , and thus every limit point is the minimizer of Problem (1).

Since the minimizer of Problem (1) is unique, we obtain that the limit point of the sequence { X k } is also unique, and thus { X k } is convergent. Therefore, we conclude that the sequence { X k } converges to the unique minimizer of Problem (1). The monotone decreasing of the sequence { f ( X k ) } can be established by Proposition 3.5.

B.7 Proof of Theorem 4.4

Proof. We prove that the support of X ⋆ is consistent with the set I c k for a sufficiently large k . Without loss of generality, we specify the constant δ in (9) as

where ω ∈ (0 , 1) is a constant. Note that the sequence { X k } converges to X ⋆ . Under Theorem 4.3 that [ ∇ f ( X ⋆ ) ] ij is strictly negative for any ( i, j ) ∈ supp c ( X ⋆ ) \ E , there must exist some a > 0 and K 1 ∈ N + such that

holds for any k ≥ K 1 , where α ∈ (0 , 1) is a constant, and m is a positive constant defined in Lemma B.1. We consider a neighbourhood of X ⋆ defined by

̸

∈

where r is a positive constant defined as

where c < 1 -ω is a positive constant. There must exist K 2 ∈ N + such that X k ∈ N ( X ⋆ ; r ) holds for any k ≥ K 2 . Take K o = max( K 1 , K 2 ) . For any k ≥ K o and ( i, j ) ∈ supp( X ⋆ ) , one has

Thus one can obtain, for any k ≥ K o ,

where the last inequality follows from (50) and (53). Recall that for any ( i, j ) ∈ T ( X k , δ ) , ∣ ∣ [ X k ] ij ∣ ∣ ≤ δ. Then one has

Then the ϵ k in (9) can be bounded by

where the first and second inequalities follow from (55) and (51), respectively. For any k ≥ K o and ( i, j ) ∈ T ( X k , ϵ k ) ∪ E , one has

where the second inequality follows from the definition of ϵ k . Thus we obtain

On the other hand, for any k ≥ K o and ( i, j ) ∈ supp c ( X ⋆ ) , one has

where the last inequality follows from (53) and (56). Therefore, one has

Note that B (5) k ∩ supp c ( X ⋆ ) = ∅ , because any ( i, j ) ∈ B (5) k corresponds to the element on the diagonal which must be nonzero. Moreover, following from (51), one has

Therefore, we can obtain supp c ( X ⋆ ) ⊆ T ( X k , ϵ k ) ∪ E . Together with (57), we obtain

We can see that the sets I k and I c k are fixed for any k ≥ K o . Therefore, following from the iterate of X k +1 ,

Moreover, together with (54) and (60), we obtain that for any k ≥ K o , [ X k +1 ] I c k +1 = 0 . Take k o = K o +1 . We obtain

Equivalently, we have completing the proof.

̸

B.8 Proof of Theorem 4.5

Proof. Theorem 4.5 is a direct extension of the result in [51]. Let g be twice continuously differentiable and consider the following iterate

where D k is positive definite and symmetric. Then the sequence { x k } admits the following convergence result [51],

where x ⋆ is the limit of the sequence { x k } , which satisfies that ∇ g ( x ⋆ ) = 0 and ∇ 2 g ( x ⋆ ) is positive definite, and m ′ k and M ′ k are the smallest and largest eigenvalues of ( D k ) 1 / 2 ∇ 2 g ( x k )( D k ) 1 / 2 , respectively. This conclusion is extended from the convergence result for the quadratic objective function. Conceptually, this makes sense because a twice continuously differentiable objective function is very close to a positive definite quadratic function in the neighborhood of a non-singular local minimum.

Following from Theorem 4.4, for any k ≥ k o , iterate (13) can be written as

which reduces to an iterate of an unconstrained optimization algorithm on some subspace. Following from the results in (61), we obtain

where m k and M k are the smallest and largest eigenvalues of R -1 2 k [ H k ] I c k I c k R -1 2 k , respectively. Following from (11), Theorem 4.2, and R -1 k = [ H -1 k ] I c k I c k , we obtain

When k ≥ k o , the line search condition reduces to

Similar to the unconstrained case in [52], the step size must satisfy the line search condition if γ k ≥ min ( 1 , 2(1 -α ) β/M k ) as k → ∞ . Then together with (63) and (64), we complete the proof.

Peer reviews:
comments: SUMMARY:
This paper proposed a novel algorithm for solving the problem of estimating precision matrices in Gaussian distributions with MTP2 constraint. The problem formulation the paper considered is a sign-constrained log-determinant program with disconnectivity constraints. The algorithm is based on a carefully designed search direction and variable partitioning scheme. The author provide the choices of approximate Newton direction and the computing step size with the theoretical convergence analysis, where the author finally shows the convergence in finite iterations. The experimental results indicates better computational efficiency of the proposed algorithm compared to the state-of-the-art methods.

STRENGTHS:
The paper is well written and clear, the approach is well supported by the theoretical analysis. The experiment and application seem strong and nice. The authors also made comparisons to other state-of-the-art methods that makes the paper more complete.

WEAKNESSES:
- Some quantities could be elaborated in the main paper. e.g., $D_k$ in line 153. I am not sure if the form of the diagonal matrix is important or a minor issue that will only be useful for theoretical analysis.

- It might also be good to mention the special case of the algorithm without the disconnectivity constraint and how it compares with other existing methods.

- It seems to me that the choice of regularization parameter $\lambda_{ij}$ doesn't affect the results in Theorem 4.4 & 4.5. Could you elaborate more on this?

QUESTIONS:
See weakness.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-20

comments: SUMMARY:
This paper presents a novel projected Newton-like method for computing the inverse covariance matrix of a multivariate Gaussian distribution, for which the off-diagonal elements are non-positive. The method is shown to be an order of magnitude faster than existing state-of-the-art methods for this task, while the authors also prove convergence to the global optimum.

STRENGTHS:
Well-written.
Fast algorithm. 
Complete theory. 
Convincing experiments.

WEAKNESSES:
I would like the authors to spend a little more time motivating and introducing the problem.
In particular, discussing the following questions on the level of introduction, might make the paper more appealing.
1. What features of the data lead to such an inverse covariance matrix structure? 
2. What is the exact "reduction on the number of observations" for a model with the MTP_2 structure? (line 23). 
3. What is the role, in terms of data / applications, of the disconnectivity set Epsilon? 
4. Why is problem (1) preferred to other alternatives?

QUESTIONS:
Please see above.

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

comments: SUMMARY:
(1) This paper investigates the problem of estimating precision matrices in Gaussian distributions  that are multivariate totally positive of order two (MTP2), and propose an algorithm based on the two-metric projection method.
(2) Empirical tests are conducted.

STRENGTHS:
- **Originality:** The techniques used in this paper reduces computational complexity and its theoretical convergence is established.
 
-  **Quality:** The notations, problem statements, and mathematical proofs appear rigorous, although I have not scrutinized them word by word.
 
- **Clarity:** The readability of this article is good.
 
- **Significance:** The techniques (and tricks) may inspire and guide future researchers.

WEAKNESSES:
1. The main issue with this paper is that it does not cite the highly relevant paper [A]. The bound provided by theoretical analysis of this paper is a supremum of a limit point, which generally can not guarantee the convergence, while the theoretical result of [A] is much more attractive. 

2. Also, in the empirical study, the algorithm presented in the paper has not been compared to [A].  

[A] Ying, Jiaxi, José Vinícius De Miranda Cardoso, and Daniel P. Palomar. "Adaptive Estimation of Graphical Models under Total Positivity." (2023).

QUESTIONS:
Please address these questions I posed to in the previous message.

LIMITATIONS:
Yes, they have.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose and analyze a variable metric projected gradient method for a variant of the graphical Lasso problem where the off-diagonal elements are additionally constraint to be non-positive, and in which specific entries of the precision matrix are known to be zero.\nThe proposed metric which is used to precondition the gradients comes from the curvature of the function $X \\mapsto -\\operatorname{log det}(X)$ , which is part of the objective, and thus incorporates second-order information of the objective function, unlike more standard approaches for the problem.\n\nThe authors put their result in the context of the recent literature, and show in numerical experiments on covariances created from multivariate Gaussian whose precision matrix sparsity comes from a Barabási-Albert random graph that the proposed method outperforms other methods used to optimize the same objective function, such as block coordinate descent or the proximal point algorithm in terms of clock-time convergence speed.\n\nFrom a theoretical point of view, the authors show that the proposed algorithm indeed converges to the unique minimizer of the (convex) objective, that the proposed stepsize rule indeed works, and that the algorithm identifies the true support in a finite number of iterations under an additional reasonable assumption on the minimizer.\n\nSTRENGTHS:\nI consider the proposed efficient estimator for the sparse precision matrix with off-diagonal non-negativity constraint of interest for the machine learning community due to wide potential applicability of the setting to engineering problems. \nThe specific gradient preconditioning through the weighted metric induced by the second derivatives of the $\\ell_2$-error is original for the problem, and the computational advantages compared to state-of-the-art methods are well explained ($O(p^3)$ time complexity) and showcased in sufficiently timing experiments. The choice of the comparison methods is adequate and the experimental setups are informative.\n\nThe paper is clearly and well written. A strength of the paper is the suitable co-design that heavily takes into account the problem structure, which makes the Hessian approximation used in the variable metric preconditioning simple enough to compute in a reasonable amount of time, while at the same time delivering a benefit in per-iteration improvement.\n\nWEAKNESSES:\nWhile the problem as formulated in (1) is convex as stated, the underlying estimator pipeline is somewhat inherently non-convex due to the fact that the $\\lambda_{ij}$ correspond to weights of a reweighted $\\ell_1$-term whose weights are chosen based on the MLE. This detail could be mentioned more prominently in the paper. \n\nFrom a computational point of view, the line search method deployed has the computational disadvantage that a lot of function evaluations, which might be expensive due to their need for spectral decompositions, might be needed to make the method work.\n\nIt remains unclear whether the convergence rate result Theorem 4.5 is optimal. Due to the incorporation of second-order information, a discussion of why superlinear convergence cannot be shown or observed would be further informative.\n\nQUESTIONS:\n- When discussing the positive definiteness constraint in line 112, you mention that it is not closed and furthermore, cannot be managed by a projection. However, by looking at (1), it seems that due to the objective, it is clear that the minimizer will not be singular due to the existence of the term $- \\operatorname{log det}(X)$  in the minimization: If at least one of the eigenvalues of X is 0, $\\operatorname{log det}(X)$ is finite. Lemma B.1 touches upon this, but couldn't the positive definiteness constraint be relaxed to a psd constraint?\n- In the considered model (1), the role of the disconnectivity constraint remains slightly unclear. Appendix A.1. and also the discussion of the financial time series example leaves the impression that the disconnectivity set is found by a similar method as the weights of the $\\ell_1$-term $\\lambda_{ij}$. Can you make the distinction clearer?\n - In Theorem 4.2, it will be convenient if one the authors can refer definition equation for $L_f$.\n\nLIMITATIONS:\nThe this discussion of limitations is adequate in general. However, it would have been beneficial if the \"hidden - $\\lambda$-dependence\" on the sparsity of the estimated covariance matrix had been explained better in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}