Title: Learning Large-Scale MTP$_2$ Gaussian Graphical Models via Bridge-Block Decomposition
Venue: NeurIPS 2023
Accepted: yes
Keywords: MTP2 Gaussian Graphical Model

High-dimensional precision matrix estimation

Bridge-block decomposition.
Abstract: This paper studies the problem of learning the large-scale Gaussian graphical models that are multivariate totally positive of order two ($\text{MTP}_2$). By introducing the concept of bridge, which commonly exists in large-scale sparse graphs, we show that the entire problem can be equivalently optimized through (1) several smaller-scaled sub-problems induced by a \emph{bridge-block decomposition} on the thresholded sample covariance graph and (2) a set of explicit solutions on entries corresponding to  \emph{bridges}. From practical aspect, this simple and provable discipline can be applied to break down a large problem into small tractable ones, leading to enormous reduction on the computational complexity and substantial improvements for all existing algorithms.  The synthetic and real-world experiments demonstrate that our proposed method presents a significant speed-up compared to the state-of-the-art benchmarks.

Learning Large-Scale MTP 2 Gaussian Graphical Models via Bridge-Block Decomposition

Abstract

This paper studies the problem of learning the large-scale Gaussian graphical models that are multivariate totally positive of order two (MTP 2 ). By introducing the concept of bridge, which commonly exists in large-scale sparse graphs, we show that the entire problem can be equivalently optimized through (1) several smallerscaled sub-problems induced by a bridge-block decomposition on the thresholded sample covariance graph and (2) a set of explicit solutions on entries corresponding to bridges . From practical aspect, this simple and provable discipline can be applied to break down a large problem into small tractable ones, leading to enormous reduction on the computational complexity and substantial improvements for all existing algorithms. The synthetic and real-world experiments demonstrate that our proposed method presents a significant speed-up compared to the state-of-the-art benchmarks.

1 Introduction

In recent decades, the surge in data availability has posed challenges for analyzing and understanding large-scale datasets. A key challenge is examining pairwise relationships among numerous variables, which can be represented using Gaussian graphical models (GGMs) [1] that depict variable connections as graphs. The precision matrix, which is the inverse of the covariance matrix, helps determine the non-zero patterns of GGMs [2, 3]. A traditional approach to estimate the precision matrix is the graphical lasso [4, 5], which is formulated as a regularized log-determinant program. The solutions of graphical lasso possess an appealing property known as sparsity, which severs as a common assumption particularly in large graphical models and has been shown to offer numerous benefits by a significant amount of research. For example, the sparsity can reduce the model size and improve the interpretability of the model by highlighting the most important variables and their relationships, allowing better understanding the underlying causal structure [6-8].

In this paper, we solve the large-scale sparse precision matrix estimation problem for GGMs that follow a multivariate totally positive of order two (MTP 2 ) Gaussian distribution [9, 10], or equivalently, possess a precision matrix that is a symmetric M -matrix [11], where all off-diagonal elements are non-positive. The so-called MTP 2 property is a special form of positive dependence and plays an essential role in applications where all interaction potentials are considered non-negative or the focus is on emphasizing positive associations between variables [12-15]. The MTP 2 property has been applied in various fields. Here are some examples. In finance, MTP 2 structures are often exploited as the asset returns are often considered positively correlated [16-18]; In machine learning, MTP 2 GGMs are recognized as attractive Markov random field and have been used in applications such as taxonomic reasoning [19] and psychometrics [13]. The MTP 2 precision matrix, also referred to

as the generalized graph Laplacian [20-22], along with the combinatorial graph Laplacian [23-27], have found broad applications across a variety of fields due to their unique characteristics. These applications include, but are not limited to, graph Fourier transform [28], electrical circuits analysis [29], image and video coding [30], financial time-series analysis [31, 32], and structured graph learning [33, 34].

The objective of this paper is to build a theoretical foundation and devise effective approaches for learning large-scale, sparse MTP 2 GGMs. The contributions of this paper are threefold.

This is the first work in the literature that introduces the notion of bridges in the context of learning MTP 2 GGMs to the best of our knowledge. Building upon this notion, we prove that the presence of explicit solutions for some entries depends on whether an edge functions as a bridge. Meanwhile, we demonstrate that the optimal solution exhibits a decomposed structure through a vertex-partition known as bridge-block decomposition.

Based on theoretical findings, we propose an efficient framework that decomposes a large problem into several small tractable ones, each of which can be solved using any existing algorithm. With some negligible extra cost for bridge-block decomposition, the proposed method results in a dimension reduction that significantly cuts down the computational complexity.

The synthetic and real-world experiments demonstrate that our proposed methods prompt a considerable speed-up for all existing methods and enable the solving of large-scale MTP 2 GGMs that were previously considered infeasible.

1.1 Notation and Organization

Vectors and matrices are written as lower and upper case bold letters, respectively. An undirected graph is denoted as G = ( V , E ) , where V is the set of nodes with size |V| = p and E ⊂ V × V is the set of edges. Note that we shall not distinguish between ( i, j ) ∈ G and ( i, j ) ∈ E . Some graph terminology frequently used throughout the paper are introduced as follows.

̸

Support graph Given a symmetric matrix A ∈ S p , the support graph of A , denoted as supp ( A ) , is defined as an undirected graph with the vertex set V = { 1 , . . . , p } and the edge set E ∈ V × V such that ( i, j ) ∈ E if and only if A ij = 0 for every two different vertices i, j ∈ V .

Neighbors The neighbors of a node i refer to the subset of vertices that are connected to this node, i.e., N ( i ) = { j | ( i, j ) ∈ E} .

Path and Cycle A path from node i 1 ∈ V to node i T ∈ V is a sequence (i.e., an ordered set) of edges denoted as d i 1 ,i T , each one incident to the next, i.e., { ( i t , i t +1 ) } T -1 t =1 ⊆ E . A cycle is a path from a node to itself when it does not duplicately include the same edge.

Partition A partition P = {V 1 , . . . , V K } is a collection of non-empty, disjoint vertex sets V 1 , . . . , V K called cluster such that ⋃ K i =1 V k = V .

Component A component of G refers to a subset of vertices that are connected to each other by edges, but are not connected to any other vertices in the graph.

The paper is organized as follows. In Section 2, we introduce the problem formulation and related works. In Section 3, we present our main result along with its potential impact. Section 4 presents the numerical experiments to show that the proposed methods can facilitate the performance of existing algorithms and solve large-scale MTP 2 GGMs problems 2 . In Section 5 we draw the conclusions.

2 Problem Formulation and Related Works

2.1 Problem Formulation

Let y be a random vector that follows a Gaussian distribution with zero mean and a precision matrix Θ , a.k.a. the inverse of covariance matrix Σ . A Gaussian graphical model represents the conditional dependency between random variables via a graph G , in which nodes correspond to y and edges between these variables represent conditional dependencies, which can be equivalently characterized by the non-zero patterns of the precision matrix Θ [2].

̸

This paper considers estimating the precision matrix Θ given n independent and identically distributed observations { y 1 , . . . , y n } that follow an MTP 2 Gaussian distribution. Equivalently, Θ is assumed to be a symmetric M -matrix, i.e., Θ ij ≤ 0 for any i = j . The problem is formulated as

̸

̸

where S is the sample covariance matrix, Λ ij ≥ 0 are the regularization coefficients, the objective is to minimize the negative log-likelihood of the data subject to a weighted ℓ 1 -norm penalty on the precision matrix, and M p refers to a set of M -matrices with dimension p , i.e.,

̸

We refer to Θ ⋆ as the optimal solution of (1) and to the support graph of Θ ⋆ , i.e., supp ( Θ ⋆ ) , as the optimal graph . Particularly, we define the thresholded sample covariance matrix T as

̸

and the support graph of T as the thresholded sample covariance graph (short for thresholded graph ). Clearly, entries corresponding to S ij ≤ λ ij will be thresholded to zero. Since the thresholded graph holds immense importance throughout the paper, unless explicitly mentioned otherwise, we denote G = supp ( T ) .

̸

The involvement of MTP 2 constraints, which require that Θ ij ≤ 0 for all i = j , confers several advantages [11]. For example, all M -matrices are inverse-positive, i.e., Θ -1 ≥ 0 , and the nonsmoothness from ℓ 1 norms can be eliminated via ∑ i = j Λ ij | Θ ij | = -⟨ Λ , Θ ⟩ , where Λ = (Λ ij ) and diag ( Λ ) = 0 . Meanwhile, the MTP 2 constraints maintain the sparsity in the estimated precision matrix as an implicit regularizer [19]. Throughout the paper, we require the following assumption.

̸

Assumption 2.1 holds with probability 1 if the number of observations n ≥ 2 [13, 19]. Under that assumption, the minimizer of Problem (1) exists and is unique according to [19, Theorem 1].

2.2 Related Works

In literature, devising algorithms for learning MTP 2 GGMs has garnered considerable attentions. For instance, block coordinate descent (BCD) methods [19, 26, 35] update a single row/column of the precision matrix in a cyclic manner by solving non-negative least squares problems. Projectionbased methods, such as projected gradient descent [36] and projected quasi-Newton algorithms [37], iteratively take steps along the descent direction and then project the solutions back to the feasible region. Despite these efforts, existing research has not fully explored the potential of exploiting MTP 2 properties to reduce computational complexity. With complexities of O ( p 4 ) for BCD methods and O ( p 3 ) for projection-based techniques, addressing large-scale problems continues to be challenging, particularly when manipulating full-dimensional matrices without problem reduction.

Instead of devising efficient algorithms, recently, leveraging the properties of the thresholded sample covariance graph has emerged as a popular approach for learning GGMs. Specifically, the existence of closed-form solution has been established for graphical lasso when the thresholded sample covariance graph is acyclic (i.e., contains no cycles) [38, 39]. However, the applicability of their main results is limited by certain conditions that are difficult to verify. Interestingly, those conditions are unnecessary for the existence of closed-from solutions in MTP 2 GGMs [20]. Despite the theoretical establishment on closed-form solutions, an exact acyclic structure is rarely observed in practice. Therefore, more research dives into the relationship between supp ( T ) and supp ( Θ ⋆ ) . Research found that supp ( Θ ⋆ ) is a subset of supp ( T ) [13], the number of connected components in supp ( T ) is identical to that in supp ( Θ ⋆ ) [20], and there exist necessary conditions for the presence of edges in supp ( Θ ⋆ ) via analyzing the path products on thresholded matrix [13].

This paper advances prior research in two ways. Firstly, unlike previous studies that only provided an explicit solution for Θ ij in the case of acyclic thresholded graphs, we reveal that this explicit solution for Θ ij consistently applies to all ( i, j ) pairs acting as bridges, regardless of whether the thresholded graph is acyclic or non-acyclic. Secondly, we highlight that the optimal graph can be represented in an equivalent decomposed form through a vertex partition, termed as the bridge-block decomposition of the thresholded graph by leveraging MTP 2 properties.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

3 Proposed Methods

The goal of this paper is not about developing an algorithm but to shed light on the remarkable properties concealed in the thresholded graph. This section is organized as follows. In Section 3.1, we introduce bridge and bridge-block decomposition. Section 3.2 presents our main result. Then in Section 3.3, we elaborate on the contributions and connections to existing research.

3.1 Bridge and Bridge-Block Decomposition

Bridge is one of the important concepts in graph theory. Technically, an edge is called a bridge if and only if its deletion increases the number of graph components. Therefore, an edge is a bridge only when it is not contained in any cycles [40]. Notably, when a graph consists solely of bridges, it is referred to as acyclic . In this paper, we denote B as the set of all bridges, i.e,

Remark 3.1 . Bridges are frequently observed, particularly in large-scale sparse graphs [41-43]. This is attributed to the fact that in sparse graphs, only the most significant relationships between variables are retained, and the removal of edges can create additional connected components, giving rise to the presence of bridges.

Remark 3.2 . In practice, the set B can be efficiently obtained via various bridge-finding algorithms [40, 44]. These algorithms employ a depth-first search approach, resulting in a computational complexity of O ( |V| + |E| ) . In the sparse graphs that are of interest to us, the number of edges |E| typically scales similarly to the number of nodes |V| . As a result, the computational cost of identifying bridges in large-scale sparse graphs remains low.

Using Figure 1 as an illustration, we perform the bridge-block decomposition [45] as follows. By removing all the bridges, we compute the clusters V k corresponding to the components of G = ( V , E \ B ) . This process results in a vertex-partition, known as the bridge-block decomposition:

where K refers to the number of clusters, also the number of components in the graph ( V , E \ B ) .

Without loss of generality, we define the operator ψ as ψ ( i ) = k if node i belongs to the k -th cluster. In the literature, the bridge-block decomposition is also known as the 2-edge-connected decomposition [46, 47], a fundamental concept in graph theory with numerous applications such as community search [48], social network mining [49], and transmission networks [50].

In practice, the cost of obtaining the bridge-block decomposition P bbd , which involves calculating the thresholded graph, bridges, and clusters, is negligible. With the aforementioned preliminary knowledge, we formally introduce our main result as follows.

3.2 Main Results

Considering a square matrix A ∈ S p , we define A V k ∈ S p k as the principal sub-matrix of A keeping the rows and columns indexed by V k , in which p k = |V k | is the number of nodes in k -th cluster and

̸

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

we have p = ∑ k p k . For each i ∈ V k , we mark π ( i ) as its corresponding index in V k . Hence, we define ̂ Θ k as the optimal solution of k -th sub-problem, i.e.,

The main result is then given as follows.

Theorem 3.3. Under Assumption 2.1, given the bridge-block decomposition of the thresholded graph supp ( T ) as P bbd , and the optimal solution of each sub-problem (46) as ̂ Θ k , the optimal solution of Problem (1) , i.e., Θ ⋆ , can be obtained as

̸

in which ζ i = 1 S ii ∑ ( i,m ) ∈B T 2 im S ii S mm -T 2 im and ζ i = 0 if ∀ m : ( i, m ) / ∈ B .

Figure 2 depicts an example of how to apply Theorem 3.3 to obtain the optimal solution more efficiently. Theoretically, the key to show the optimality of (7) is via an explicit expression of the inverse of Θ , i.e., R = Θ -1 using following theorem.

Theorem 3.4. Given S , T ∈ S p , the bridge-block decomposition P bbd = {V 1 , . . . , V K } of supp ( T ) , and a set of matrices { ̂ Θ k ∈ S |V k | ++ } K k =1 , the inverse of Θ in the form of (7) is derived as

where ̂ R k = [ ̂ Θ k ] -1 and for each i , j in different clusters, g ij is given as

in which T -1 is the number of bridges in d ij , g ij = 0 if d ij = ∅ , and u denotes a sequence of 'incident' bridges, i.e., d ij ∩ B = { ( u 1 , u 2 ) , . . . , ( u 2 T -1 , u 2 T ) } following the orders of d ij while preserving the elements that are bridges.

We note that all terms in (9) can be computed via (8) and Theorem 3.4 itself generally holds for arbitrary S , T , and { ̂ Θ k } K k =1 . Detailed proofs and discussions are deferred to Appendix A.

Theorem 3.4 is theoretically critical as it provides an explicit form of the inverse of Θ , which was previously difficult to compute. It is vital for demonstrating the optimality of Θ by connecting the KKT system of the original large-scale problem to the KKT systems of smaller sub-problems through the bridge-block decomposition. Consequently, together with MTP 2 properties, we show that Θ in the form of (7) satisfies the KKT condition of Problem (1) in Appendix B.

The role of MTP 2 constraints: Although Theorem 3.4 can be applied without requiring MTP 2 properties, it is important to highlight that these properties serve as sufficient conditions to demonstrate

the optimality of Θ = R -1 . By incorporating the MTP 2 constraints, the non-smoothness in graphical lasso is eliminated, resulting in simplified optimality conditions as shown below:

̸

Simultaneously, R becomes non-negative ( ≥ 0 ). As a result, The most challenging part of the KKT conditions -R ij + S ij -λ ij ≤ 0 holds under MTP 2 properties. In Appendix C, we will elaborate the details and explore additional conditions under which we can extend the applicability of Theorem 3.3 to the traditional graphical lasso.

Proposed solving framework: In practical applications, it is often more efficient to employ the bridge-block decomposition technique instead of directly manipulating full-sized matrices. This approach involves breaking down the problem into smaller isolated sub-problems. By addressing these sub-problems individually, we can compute the optimal solution by utilizing the solutions obtained from each sub-problem, as outlined in Theorem 3.3. This approach offers several advantages:

Significant reduction in computational cost . Foremost, the cost of bridge-block decomposition is cheap. Suppose that we use a BCD solver of complexity O ( p 4 ) , then the total cost is reduced to ∑ K k =1 O ( |V k | 4 ) ≪O ( p 4 ) , where ∑ k |V k | = p . This can prompt an enormous difference.

Considerable reduction in memory cost . The memory cost is typically troublesome for largescale problems as each full-dimensional matrix contains p 2 elements. Theorem 3.3 can avoid generating a number of full-dimensional intermediate variables during computation.

Potential speed-up via parallel computing . The sub-problems can be optimized independently, which allows parallel computing for significant speed-up.

3.3 Connection to Existing Research

From Theorem 3.3, we obtain a very interesting result that the ( i, j ) -th entries of Θ admits an explicit solution, i.e., Θ ij = -T ij /( S ii S jj -T 2 ij ) if edge ( i, j ) is a bridge in the thresholded graph supp ( T ) . To the best of our knowledge, this result, shown in Corollary 3.5, is the first sufficient condition for an edge belonging to optimal graph supp ( Θ ⋆ ) .

Corollary 3.5. If edge ( i, j ) is a bridge in the thresholded graph supp ( T ) , then ( i, j ) is also a bridge in the optimal graph supp ( Θ ⋆ ) .

̸

Corollary 3.5 can be obtained as follows. By the definition of a bridge, from nodes i to j the path d ij = { ( i, j ) } is the only path in the thresholded graph, and removing it would result in an increase in the number of components. Given that supp ( Θ ⋆ ) ⊆ supp ( T ) and Θ ij = 0 , it follows that d ij = { ( i, j ) } remains the unique path from nodes i to j in the optimal graph. This indicates that the edge ( i, j ) continues to function as a bridge.

In the literature, only some necessary conditions for edges in the thresholded graph supp ( T ) to be retained in the optimal graph supp ( Θ ⋆ ) have been established. For instance, [13] and [20] indicate that supp ( Θ ⋆ ) ⊆ supp ( T ) , which implies that ( i, j ) / ∈ supp ( Θ ⋆ ) if ( i, j ) / ∈ supp ( T ) .

Another explicit solution is also applicable when k -th cluster only contains one node, i.e., V k = { i } , then ̂ Θ k = 1 /S ii . As a consequence, Theorem 3.3 generalizes to the following corollary for acyclic graph which admits a bridge-block decomposition into p clusters.

Corollary 3.6. Suppose that the thresholded graph supp ( T ) is acyclic, then the optimal graph supp ( Θ ⋆ ) is also acyclic and Θ ⋆ admits a closed-form solution as

The proof directly follows Theorem 3.3. Corollary 3.6 coincides with [20, Theorem 3] and [38]. Compared to our results in Theorem 3.3, explicit solutions in literature heavily depend on the graph

structure. Hence, our theory appears as the first to reveal that, fundamentally, it is the edge property, i.e., whether the edge is a bridge, that decides the existence of explicit solutions.

In conclusion, our proposed methods generalize the existing results with more profound understandings, offering significant potential for solving large-scale precision matrix estimation problems.

4 Numerical Simulations

We conduct experiments on synthetic and real-world data to evaluate how the proposed method accelerates the convergence of existing algorithms compared to algorithms that do not exploit it. All experiments were conducted on 2.60GHZ Xeon Gold 6348 machines and Linux OS. All methods are implemented in MATLAB and the state-of-the-art methods we consider includes

BCD : Block Coordinate Descent [19] of complexity O ( p 4 ) .

PGD : Projected Gradient Descent [36] of complexity O ( p 3 ) .

PQN-LBFGS : A projected quasi-Newton method with limited memory BFGS [51]. The complexity is O (( m + p ) p 2 ) , where m is the number of iterations stored for approximating the Hessian.

FPN : Fast projected Newton-like method [37] of complexity O ( p 3 ) .

Note that all methods converge to the optimal solution. The target here is not to compare these methods but to evaluate how our proposed framework promotes the efficiency of these methods.

4.1 Synthetic Data Experiments

We use the processes described in [19] to generate the data. We begin with an underlying graph that has an adjacency matrix A ∈ S p , and define Θ = δ I -A , where δ = 1 . 05 · λ max ( A ) and λ max ( A ) represents the largest eigenvalue of A . his ensures that Θ is a positive definite matrix with off-diagonal elements being negative, making Θ a randomly generated M -matrix. Next, we normalize Θ by substituting it with D Θ D , where D is a suitably chosen diagonal matrix, ensuring that diag ( Θ -1 ) = 1 . We then sample n = 10 p data points from a Gaussian distribution N ( 0 , Θ -1 ) and calculate the sample covariance matrix as S .

̸

Following [25], we construct the regularization matrix Λ . Briefly, given an initial estimate Θ (0) , we set Λ ij = χ / (Θ (0) ij + ϵ ) when i = j and Λ ij = 0 when i = j . Here, χ > 0 determines the sparsity level and ε is a small positive constant, such as 10 -3 . Generally, a large penalty is expected on Θ ij when Θ (0) ij is small. We efficiently compute Θ (0) using the following equation:

̸

and then appropriately adjust the value of χ so that supp ( T ) has only one connected component, while supp ( Θ ⋆ ) can roughly recover the underlying structure.

We consider two scale-free random graphs as underlying with their typical structures in Figure 4.

Barabasi-Albert (BA) graph [52, 53] of order one. BA models generate random scale-free networks via preferential attachment, suitable for modeling various networks like the Internet, world wide web, protein interactions, citations, and social/online networks.

Stochastic Block Model (SBM) of networks [54], a.k.a. the community graph [55]. Stochastic block models serve as fundamental tools in network science, creating random networks based on community structures, where nodes within the same group are more likely to form connections.

In the experiments, we consider p = 5000 and p = 20000 and calculate the relative errors as follows:

where we compare the relative errors at each iteration against the computational time. Here, Θ ⋆ represents the optimal solution, and f signifies the objective function of (1). For methods employing bridge-block decomposition (bbd), we first calculate ̂ Θ ( s ) k for each cluster at the s -th iteration, followed by calculating the intermediate solution Θ ( s ) using (7). The computational time at the s -th iteration is then the sum of the costs to obtain P bbd , calculate all ̂ Θ ( s ) k , and compute Θ ( s ) via (7).

10

-6

10

-4

10

-2

10

0

10

0

10

1

10

2

10

3

CPU time (seconds)

Relative Errors

BA graph, p = 5000

10

-6

10

-4

10

-2

10

0

10

2

10

1

10

2

10

3

10

4

CPU time (seconds)

Relative Errors

BA graph, p = 20000

10

-6

10

-4

10

-2

10

0

10

2

10

0

10

1

10

2

10

3

10

4

CPU time (seconds)

Relative Errors

SBM graph, p = 5000

10

-6

10

-4

10

-2

10

0

10

2

10

1

10

2

10

3

10

4

CPU time (seconds)

Relative Errors

SBM graph, p = 20000

BCD

BCD (+ bbd)

PGD

PGD (+ bbd)

FPN

FPN (+ bbd)

PQN-LBFGS

PQN-LBFGS (+ bbd)

The results for p = 5000 and p = 20000 are displayed in Figure 3, with all results averaged over five realizations. We highlight that the extra computational cost compared to methods without acceleration is negligible as shown in Table 1. From the figures, we observe that:

Significant Speed-up: Our proposed framework substantially accelerates state-of-the-art methods by one to four orders of magnitude.

Solving Otherwise Infeasible Problems: When p = 20000 , existing methods cannot optimize the problem within 10 4 seconds. In contrast, our framework greatly speeds up convergence, making it possible to solve large-scale problems.

Higher Speed-up for Sparser Graph: Proposed method achieves greater acceleration on the BA graph than the SBM graph, as the former exhibits a stronger sparsity pattern.

Higher Speed-up for Methods of Higher-complexity: Figure 3 demonstrates that our method has a more significant impact on improving the efficiency of the BCD method, which has higher computational complexity and thus benefits more from dimension reduction.

To further shed light on the factors that determine the magnitudes of improvement, in the next sub-section, we conduct additional experiments that dive into how bridge-block decomposition boosts the performance of existing methods subject to different structures of the thresholded graph.

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

l

(1) BA graph

(2) SBM graph

10

30

100

300

1000

2000

3000

Ratio of Improvement

K

5

10

50

4.2 Further Experiments on Ratios of Improvement

We investigate a scenario where the underlying graph is a community graph with a block-tridiagonal adjacency matrix, and each cluster contains an equal number of nodes ( p 1 = |V k | , ∀ k ). Without loss of generality, we assume that the internal edges within clusters form a cycle. To generate S and Λ , we follow the procedures in Section 4.1. We then modify Λ to α ( 11 T -A ) +(1 -α ) Λ , ensuring that supp ( T ) = supp ( A ) , where α ∈ [0 , 1] and A is the adjacency matrix of the considered graph.

We experiment with various values of K (the number of clusters) and p 1 (the number of nodes in each cluster). For each configuration, we calculate the ratio of improvement, defined as the quotient of the time required to achieve RE ( Θ ) < 10 -6 without using bridge-block decomposition to the time when applying the decomposition. We conduct multiple trials of the BCD method and display the results in Figure 5. The results in Figure 5 suggest that the number of sub-graphs K is a primary factor influencing the ratio values. The improvement is deemed tremendous, especially for large-scale settings. The involvement of bridge-block decomposition enables solving an MTP 2 GGMs with hundreds of millions of variables as long as we can conduce bridge-block decomposition. Consequently, many previously unattainable real-life applications can now be optimized.

4.3 Real-word Data Experiment

We consider learning the MTP 2 GGM for the Crop Image dataset available from the UCR Time Series Archive [56]. This dataset comprises p = 24000 pixels, where each pixel corresponds to a time series record capturing spectral information in n = 46 instances. These instances represent geometrically and radiometrically corrected satellite images. By examining this dataset, we can observe the temporal changes in the observed areas through the recorded measurements. To facilitate our analysis, the data has been pre-processed by [57] using an indexing technique. This preprocessing step enables us to work with compressed informative indexes instead of handling large image files.

Our goal is to perform graph-based clustering to the time series data that contains 46 observations, denoted as { y 1 , . . . , y 46 } using MTP 2 GGMs, where y i ∈ R 24000 . The data includes 24 classes corresponding to different indexed land covers, such as corn, barley, or lake. It is important to note that these labels are not involved in the clustering process but help evaluate the quality of the final graph-based clusters by computing the graph modularity [58]. Via analyzing the data, in Appendix D, we discuss that why learning MTP 2 GGMs is statistically meaningful in the context of clustering and show that the MTP 2 assumptions approximately hold for this dataset.

The initial estimate Θ (0) is computed using the method proposed in [59]. The regularization matrix Λ is determined using the approach in Section 4.1 with ϵ = 0 . 01 and χ = 0 . 2 . We then compute the sample correlation matrix as S ∈ S 24000 . This results in a precision matrix estimation problem involving 5 . 76 × 10 8 variables to optimize, which is quite challenging for state-of-the-art methods without bridge-block decomposition.

Figure 6b shows the results of empirical convergence, and Figure 6a visualizes a local structure of supp ( Θ ) ⋆ (modularity = 0 . 6849 ). In line with previous findings, our suggested approach greatly speeds up the convergence of all current algorithms by a minimum of three orders of magnitude. This demonstrates immense potential for managing large-scale sparse graphical models.

Corn

Temporal

Meadow

Hardwood

Pasture

Repeseed

Barley

(a) Visualizations.

10

-8

10

-6

10

-4

10

-2

10

0

10

2

10

4

10

1

10

2

10

3

10

4

CPU time (seconds)

Relative Errors

BCD

BCD (+ bbd)

PGD

PGD (+ bbd)

FPN

FPN (+ bbd)

PQN-LBFGS

PQN-LBFGS (+ bbd)

(b) Experimental results on Crop data

(

p

= 24000)

.

While our paper's primary focus is not to uncover insights into the estimated MTP 2 GGMs for a deeper understanding of the underlying data, interesting phenomena can still be observed in the structures presented in Figure 6a. Notably, we find that the majority of edges exist within the same crop type, while connections between nodes associated with different crops are relatively sparse. This observation is valuable for clustering processes and aligns with our expectations, as stronger positive dependencies are often observed within the same class, while dependencies between different classes tend to be weaker.

Moreover, our graphical representation reveals more nuanced patterns. For instance, in Figure 6a of our manuscript, we observe a dense network of edges between two crop types, 'temporary meadow' and 'pasture', indicating a significantly stronger conditional dependency compared to other categories. This observation further supports our understanding. Hence, the inferred conditional dependency structure encapsulates the inherent interrelationships among different crops. As a result, our proposed framework based on bridge-block decomposition effectively facilitates learning in large-scale MTP 2 graphical models.

5 Conclusions and Discussions

Real-world sparse Gaussian graphical models often comprise subsets of variables that are densely connected to one another, while variables in different clusters maintain weak connections. However, standard estimation algorithms do not account for this property. In this paper, by introducing the concept of bridge, we leverage these characteristics to reveal an interesting finding: the optimal solution for Problem (1) equivalently admits a decomposed form via the bridge-block decomposition of the thresholded graph. We provide theoretical insights into the separability of optimal solutions and the existence of explicit expressions based on edge properties, specifically whether an edge is a bridge. This method surpasses conventional approaches that depend on unique graph structures, such as determining if a graph is acyclic. From the practical aspect, our proposed method offers a handy architect for accelerating any existing algorithms by handling a large-scale learning problem via a number of small and tractable sub-problems. Although our method is mainly developed for sparse large-scale graphs and therefore seems inapplicable for dense graph, as elaborated in the appendix C, it can also provide a quick way to obtain a solution, which can serve as either a starting point (warm-start) for numerical algorithms or an alternative for acquiring approximate solutions. Overall, our simple and provable approach demonstrates exceptional performance and paves a novel way for more extensive designs of large-scale MTP 2 Gaussian graphical models.

References

H. Rue and L. Held, 'Gaussian Markov random fields: theory and applications,' Monographs on Statistics and Applied Probability , vol. 104, 2005.

S. L. Lauritzen, Graphical models . Clarendon Press, 1996, vol. 17.

J. Whittaker, Graphical models in applied multivariate statistics . Wiley Publishing, 2009.

O. Banerjee, L. El Ghaoui, and A. d'Aspremont, 'Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data,' The Journal of Machine Learning Research , vol. 9, pp. 485-516, 2008.

J. Friedman, T. Hastie, and R. Tibshirani, 'Sparse inverse covariance estimation with the graphical lasso,' Biostatistics , vol. 9, no. 3, pp. 432-441, 2008.

M. Yuan and Y. Lin, 'Model selection and estimation in the Gaussian graphical model,' Biometrika , vol. 94, no. 1, pp. 19-35, 2007.

H. Liu, K. Roeder, and L. Wasserman, 'Stability approach to regularization selection for high dimensional graphical models,' Advances in Neural Information Processing Systems , vol. 23, 2010.

P. Danaher, P. Wang, and D. M. Witten, 'The joint graphical lasso for inverse covariance estimation across multiple classes,' Journal of the Royal Statistical Society: Series B (Statistical Methodology) , vol. 76, no. 2, pp. 373-397, 2014.

E. Bølviken, 'Probability inequalities for the multivariate normal with non-negative partial correlations,' Scandinavian Journal of Statistics , pp. 49-58, 1982.

D. M. Malioutov, J. K. Johnson, and A. S. Willsky, 'Walk-sums and belief propagation in Gaussian graphical models,' The Journal of Machine Learning Research , vol. 7, pp. 2031-2064, 2006.

R. J. Plemmons, 'M-matrix characterizations. I-nonsingular M-matrices,' Linear Algebra and its Applications , vol. 18, no. 2, pp. 175-188, 1977.

S. Fallat, S. Lauritzen, K. Sadeghi, C. Uhler, N. Wermuth, and P. Zwiernik, 'Total positivity in Markov structures,' The Annals of Statistics , pp. 1152-1184, 2017.

S. Lauritzen, C. Uhler, and P. Zwiernik, 'Maximum likelihood estimation in Gaussian models under total positivity,' The Annals of Statistics , vol. 47, no. 4, pp. 1835-1863, 2019.

F. Röttger, S. Engelke, and P. Zwiernik, 'Total positivity in multivariate extremes,' The Annals of Statistics , vol. 51, no. 3, pp. 962-1004, 2023.

D. Rossell and P. Zwiernik, 'Dependence in elliptical partial correlation graphs,' Electronic Journal of Statistics , vol. 15, no. 2, pp. 4236-4263, 2021.

A. Müller and M. Scarsini, 'Archimedean copulae and positive dependence,' Journal of Multivariate Analysis , vol. 93, no. 2, pp. 434-445, 2005.

R. Agrawal, U. Roy, and C. Uhler, 'Covariance matrix estimation under total positivity for portfolio selection,' Journal of Financial Econometrics , vol. 20, no. 2, pp. 367-389, 2022.

R. Zhou, J. Ying, and D. P. Palomar, 'Covariance matrix estimation under low-rank factor model with nonnegative correlations,' IEEE Transactions on Signal Processing , vol. 70, pp. 4020-4030, 2022.

M. Slawski and M. Hein, 'Estimation of positive definite M-matrices and structure learning for attractive Gaussian markov random fields,' Linear Algebra and its Applications , vol. 473, pp. 145-179, 2015.

E. Pavez, H. E. Egilmez, and A. Ortega, 'Learning graphs with monotone topology properties and multiple connected components,' IEEE Transactions on Signal Processing , vol. 66, no. 9, pp. 2399-2413, 2018.

J. Ying, J. V. d. M. Cardoso, and D. P. Palomar, 'A fast algorithm for graph learning under attractive Gaussian Markov random fields,' in 2021 55th Asilomar Conference on Signals, Systems, and Computers , 2021, pp. 1520-1524.

Z. Deng and A. M.-C. So, 'A fast proximal point algorithm for generalized graph Laplacian learning,' in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 5425-5429.

J. Ying, J. V . d. M. Cardoso, and D. P. Palomar, 'Does the ℓ 1 -norm learn a sparse graph under Laplacian constrained graphical models?' arXiv preprint arXiv:2006.14925 , 2020.

--, 'Nonconvex sparse graph learning under Laplacian constrained graphical model,' in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 7101-7113.

--, 'Minimax estimation of Laplacian constrained precision matrices,' in Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , vol. 130, 2021, pp. 3736-3744.

H. E. Egilmez, E. Pavez, and A. Ortega, 'Graph learning from data under Laplacian and structural constraints,' IEEE Journal of Selected Topics in Signal Processing , vol. 11, no. 6, pp. 825-841, 2017.

J. Ying, X. Han, R. Zhou, X. Wang, and H. C. So, 'Network topology inference with sparsity and Laplacian constraints,' arXiv preprint arXiv:2309.00960 , 2023.

D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, 'The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains,' IEEE signal processing magazine , vol. 30, no. 3, pp. 83-98, 2013.

F. Dorfler and F. Bullo, 'Kron reduction of graphs with applications to electrical networks,' IEEE Transactions on Circuits and Systems I: Regular Papers , vol. 60, no. 1, pp. 150-163, 2012.

H. E. Egilmez, Y.-H. Chao, A. Ortega, B. Lee, and S. Yea, 'GBST: Separable transforms based on line graphs for predictive video coding,' in 2016 IEEE International Conference on Image Processing (ICIP) . IEEE, 2016, pp. 2375-2379.

J. V. D. M. Cardoso, J. Ying, and D. P. Palomar, 'Graphical models in heavy-tailed markets,' in Advances in Neural Information Processing Systems , vol. 34, 2021, pp. 19 989-20 001.

--, 'Learning bipartite graphs: Heavy tails and multiple components,' in Advances in Neural Information Processing Systems , vol. 35, 2022, pp. 14 044-14 057.

S. Kumar, J. Ying, J. V. d. M. Cardoso, and D. P. Palomar, 'A unified framework for structured graph learning via spectral constraints,' Journal of Machine Learning Research , vol. 21, no. 22, pp. 1-60, 2020.

--, 'Structured graph learning via Laplacian spectral constraints,' in Advances in Neural Information Processing Systems , vol. 32, 2019, pp. 11 647-11 658.

E. Pavez and A. Ortega, 'Generalized Laplacian precision matrix estimation for graph signal processing,' in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2016, pp. 6350-6354.

J. Ying, J. V . D. M. Cardoso, and D. P. Palomar, 'Adaptive estimation of graphical models under total positivity,' in International Conference on Machine Learning , 2023, pp. 40 054-40 074.

J.-F. Cai, J. V . d. M. Cardoso, D. P. Palomar, and J. Ying, 'Fast projected Newton-like method for precision matrix estimation under total positivity,' arXiv preprint arXiv:2112.01939 , 2021.

S. Sojoudi, 'Equivalence of graphical lasso and thresholding for sparse graphs,' The Journal of Machine Learning Research , vol. 17, no. 1, pp. 3943-3963, 2016.

S. Fattahi and S. Sojoudi, 'Graphical lasso and thresholding: Equivalence and closed-form solutions,' Journal of Machine Learning Research , 2019.

J. M. Schmidt, 'A simple test on 2-vertex-and 2-edge-connectivity,' Information Processing Letters , vol. 113, no. 7, pp. 241-244, 2013.

S. Højsgaard and S. L. Lauritzen, 'Graphical Gaussian models with edge and vertex symmetries,' Journal of the Royal Statistical Society: Series B (Statistical Methodology) , vol. 70, no. 5, pp. 1005-1027, 2008.

M. L. Stein, 'Space-time covariance functions,' Journal of the American Statistical Association , vol. 100, no. 469, pp. 310-321, 2005.

C. Li and H. Li, 'Network-constrained regularization and variable selection for analysis of genomic data,' Bioinformatics , vol. 24, no. 9, pp. 1175-1182, 2008.

R. E. Tarjan, 'A note on finding the bridges of a graph,' Information Processing Letters , vol. 2, no. 6, pp. 160-161, 1974.

J. Westbrook and R. E. Tarjan, 'Maintaining bridge-connected and biconnected components on-line,' Algorithmica , vol. 7, no. 1, pp. 433-464, 1992.

L. Chang, J. X. Yu, L. Qin, X. Lin, C. Liu, and W. Liang, 'Efficiently computing k-edge connected components via graph decomposition,' in Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data , 2013, pp. 205-216.

R. Zhou, C. Liu, J. X. Yu, W. Liang, B. Chen, and J. Li, 'Finding maximal k-edge-connected subgraphs from a large graph,' in Proceedings of the 15th International Conference on Extending Database Technology , 2012, pp. 480-491.

Y. Fang, X. Huang, L. Qin, Y. Zhang, W. Zhang, R. Cheng, and X. Lin, 'A survey of community search over big graphs,' The VLDB Journal , vol. 29, no. 1, pp. 353-392, 2020.

R. Agrawal, S. Rajagopalan, R. Srikant, and Y. Xu, 'Mining newsgroups using networks arising from social behavior,' in Proceedings of the 12th International Conference on World Wide Web , 2003, pp. 529-535.

L. Lan and A. Zocca, 'Refining bridge-block decompositions through two-stage and recursive tree partitioning,' arXiv preprint arXiv:2110.06998 , 2021.

D. Kim, S. Sra, and I. S. Dhillon, 'Tackling box-constrained optimization via a new projected quasi-Newton approach,' SIAM Journal on Scientific Computing , vol. 32, no. 6, pp. 3548-3563, 2010.

A.-L. Barabási and R. Albert, 'Emergence of scaling in random networks,' Science , vol. 286, no. 5439, pp. 509-512, 1999.

R. Albert and A.-L. Barabási, 'Statistical mechanics of complex networks,' Reviews of Modern Physics , vol. 74, no. 1, p. 47, 2002.

P. W. Holland, K. B. Laskey, and S. Leinhardt, 'Stochastic blockmodels: First steps,' Social Networks , vol. 5, no. 2, pp. 109-137, 1983.

S. Fortunato, 'Community detection in graphs,' Physics Reports , vol. 486, no. 3-5, pp. 75-174, 2010.

H. A. Dau, A. Bagnall, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, and E. Keogh, 'The UCR time series archive,' IEEE/CAA Journal of Automatica Sinica , vol. 6, no. 6, pp. 1293-1305, 2019.

C. W. Tan, G. I. Webb, and F. Petitjean, 'Indexing and classifying gigabytes of time series under time warping,' in SIAM International Conference on Data Mining , 2017, pp. 1-10.

M. E. Newman, 'Modularity and community structure in networks,' Proceedings of the National Academy of Sciences , vol. 103, no. 23, pp. 8577-8582, 2006.

F. Nie, X. Wang, M. Jordan, and H. Huang, 'The constrained Laplacian rank algorithm for graph-based clustering,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 30, no. 1, 2016.

Appendix

In what follows, we present technical proofs omitted in the main content. In Section A, we introduce important properties of matrix R and prove Theorem 3.4. In Section B, we prove the main result, i.e., Theorem 3.3. In Section C, we introduce some extension of the proposed framework to broaden its applicability. In Section D, we will comment on real-world experiments and explain why MTP 2 assumption approximately holds for Crop Image dataset.

A Proof of Theorem 3.4

This section contains a detailed description of how we construct the matrix R and prove Theorem 3.4 based on the properties of R . The proof relies on the concept of a 'generalized path product', which involves a set of edges labeled as a 'bridge path'. The matrix R has several notable properties that are essential in demonstrating Theorem 3.3.

A.1 Definitions and Important Properties

̸

To begin, let us define the 'bridge path' B ij for nodes i and j that belong to different clusters, i.e., ψ ( i ) = ψ ( j ) based on the bridge set B in the thresholded graph G . The bridge path B ij is defined as the set of bridges in any path d ij from node i to node j , where each element in B ij is a bridge in B and the elements in B ij follow the order of d ij while preserving the elements in B . Symbolically, we can express this as:

where T denotes the number of clusters in any path from nodes i to j . It is important to note the following:

Each element in B ij is a bridge in the thresholded graph.

The elements in B ij follow the orders of d ij . This means that for any path from nodes i to j , the path will visit ( u 2 i -1 , u 2 i ) , then visit ( u 2 j -1 , u 2 j ) whenever we have i < j . consequently, we have

For any path d ij , the bridge path B ij = d ij ∩ B is unique.

B ij is defined as an empty set if there is no path from nodes i to j .

It is important to mention that if the thresholded graph is acyclic, the bridge path can be reduced to a traditional path. In this scenario, all edges in the path d ij become bridges, and the bridge path B ij is equivalent to the path d ij , which can be expressed as B ij = d ij .

We define the matrix R ∈ S p based on the definition of bridge-path, as shown below:

where ̂ R k = [ ̂ Θ k ] -1 and for each i , j in different clusters, g ij is given as

where u denotes the bridge path B ij and g ij ( R ) = 0 if B ij = ∅ .

Remark A.1 . Clearly, we have R ij = R ji for any i and j .

Example : For instance, using Figure 7 as an example, we consider a path from node 1 to 10 as

the bridge path is then given as

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

and R 1 , 10 is computed as

Particularly, we note that

The bridge path B 1 , 10 can not be written as { (9 , 10) , (5 , 6) } as d 1 , 10 would always visit the bridge (5 , 6) before visiting (9 , 10) .

The bridge path B 1 , 10 can not be written as { (6 , 5) , (9 , 10) } as d 1 , 10 leave first cluster at node 5 , then enter second cluster at node 6 . Therefore, we have ψ (1) = ψ (5) .

It is easy to observe that B 10 , 1 = { (10 , 9) , (6 , 5) } . Based on definitions, we can obtain R 10 , 1 = R 1 , 10 .

Specially, if the matrix S is given as a correlation matrix, i.e., diag ( S ) = 1 , then R 1 , 10 = R 1 , 5 · R 5 , 6 · R 6 , 9 · R 9 , 10 · R 10 , 10 , which can be seen as a continued produce on a path, known as path product.

The matrix R shares some important properties introduced as follows.

Lemma A.2. For any i, j ∈ V k , ̂ R k together with ̂ Θ k would satisfy the following equations

̸

̸

̸

Proof. Let's denote Γ π ( i ) ,π ( j ) as the dual variables associated with the constraints [ ̂ Θ k ] π ( i ) ,π ( j ) ≤ 0 . With ̂ R k = ̂ Θ -1 k , the KKT conditions include

̸

̸

̸

We can eliminate the dual variables as follows:

For [ ̂ Θ k ] π ( i ) ,π ( j ) < 0 , the complementary slackness leads to Γ π ( i ) ,π ( j ) = 0 , which implies -[ ̂ R k ] π ( i ) ,π ( j ) + S i,j -Λ i,j = 0 .

For [ ̂ Θ k ] π ( i ) ,π ( j ) = 0 , we have Γ π ( i ) ,π ( j ) ≥ 0 , which implies -[ ̂ R k ] π ( i ) ,π ( j ) + S i,j -Λ i,j ≤ 0 . Hence, Lemma A.2 is obtained.

In particular, we note that [ ̂ R k ] π ( i ) ,π ( i ) = S i,i holds for any i ∈ V k according to (21a). As a result, the following property can be easily verified via the definitions.

̸

Corollary A.3. Suppose that two nodes i and j are in different clusters, i.e., ψ ( i ) = ψ ( j ) , and the bridge path is not empty and denoted by B ij with form of (14), then we have

Proof. Corollary A.3 can be easily verified via definitions. If t = 0 or t = 2 T + 1 , Corollary A.3 holds as R i,i = S i,i and R j,j = S j,j . Let s be a number that satisfies 0 ≤ s ≤ 2 T . For any ( u s , u s +1 ) ∈ B ij , we have

Clearly, Corollary A.3 is obtained.

The aforementioned properties of matrix R are important for establishing the proof of Theorem 3.4 and Theorem 3.3.

A.2 Proof of Theorem 3.4

Proof. Let F = Θ R , where Θ and R have the forms of (7) and (8), respectively. The target is to show F ii = 1 , ∀ i and F ij = 0 , ∀ i = j using the following results we have derived.

(a) For any i, j ∈ V k , we have

and R ij = [ ̂ R k ] π ( i ) ,π ( j ) according to the definitions.

(b) For any nodes i and j in different clusters, suppose that B ij is not empty and edge ( u, v ) is a bridge in B ij , then R ij S uu = R iu R uj and R ij S vv = R iv R vj hold according to Corollary A.3.

(c) The matrix ̂ F k , which is defined as ̂ F k = ̂ R k ̂ Θ k is equal to the identity matrix, i.e, ̂ F k = I . In other words, we have

̸

Based on (a), (b), and (c), we first show that F ii = 1 holds for any i ∈ { 1 , . . . , p } . We suppose that node i is in k -the cluster, i.e., k = ψ ( i ) . Then, by using ∀ m / ∈ ( V k ∪ N ( i )) : Θ im = 0 from definitions in (7), we have

̸

̸

Therefore, T ii = 1 holds for any i ∈ { 1 , . . . , p } .

Let us proceed with the proof that F ij = ∑ p m =1 R im Θ mj = 0 for any i = j . Without loss of generality, we suppose that node j is in the k -th cluster, i.e., k = ψ ( j ) . According to the definitions, we have

̸

Here, there would be three cases to analyze.

The first case happens where i ∈ V k , which means that nodes i and j are in the same cluster. In this case, we have

The second case happens where i / ∈ V k , but there is no path between nodes i and j . In such a case, for any node m in V k ∪ N ( j ) , there is also no path from nodes i to m . Consequently, based on the definition of R , we have

The third case happens where i / ∈ V k and there exists a path from nodes i to j . Then we have ψ ( i ) = ψ ( j ) and the bridge path B ij is denoted as

where ψ ( u 1 ) = ψ ( i ) and ψ ( u 2 T ) = ψ ( j ) = k . In particular, we emphasize that u 2 T ∈ V k as ( u 2 T -1 , u 2 T ) is a bridge to enter the cluster that contains j . Therefore, we have

where the last equality is from (29) by using u 2 T ∈ V k . In conclusion, via enumerating all the possible cases, we have F = I , which means that R Θ = I .

Remark A.4 . Theorem 3.4 generally holds for any T , S and { ̂ Θ k ∈ S |V k | ++ } K k =1 . It does not utilize MTP 2 properties from Θ . Hence, Theorem 3.4 itself may be extended to graphical models beyond MTP 2 assumption.

As a result, we obtain

̸

B Proof of Main Results

To prove Theorem 3.3, we require some preliminaries on MTP 2 properties.

Lemma B.1. [11, Theorem 1] A real symmetric matrix X ∈ S p , which admits the form:

̸

is a non-singular M -matrix if and only if one of the following statements holds: (1) Every eigenvalue of X is positive; (2) X is inverse-positive. That is, X -1 exists and X -1 ≥ 0 .

A direct outcome of Lemma B.1 is that the matrix R is a non-negative matrix.

Corollary B.2. The matrix R given by (8) satisfies R ≥ 0 .

Corollary B.2 holds according to the fact that

Now, we begin proving Theorem 3.3.

Proof of Theorem 3.3: First of all, we show that the matrix Θ , given as

̸

̸

is primal feasible and positive definite. Based on Assumption 2.1, which states that S ij < √ S ii S jj , we know that T 2 ij ≤ ( S ij -Λ ij ) 2 ≤ S 2 ij < S ii S jj . Hence, for any i = j , we have Θ ij ≤ 0 due to -T ij / ( S ii S jj -T 2 ij ) ≤ 0 and the primal feasibility of each ̂ Θ k . Additionally, since we have R ≥ 0 according to Corollary B.2, we can conclude that Θ is inverse-positive according to Theorem 3.4, which implies that it is a positive-definite matrix. In other words, Θ is both primal feasible and inverse-positive, and we can express this as Θ ≻ 0 based on Lemma B.1.

In second step, we show that Θ = R -1 satisfies the optimality condition listed as follows, i.e.,

̸

̸

For (37a), we let k = ψ ( i ) . Then, according to (21a) in Lemma A.2, we have

For (37b), as Θ ij = 0 , the pair ( i, j ) either satisfies ( i, j ) ∈ B or ψ ( i ) = ψ ( j ) ∆ = k . If ( i, j ) ∈ B , then by using R ij = T ij , we have

If nodes i and j are in the same cluster, then according to (21b) in Lemma A.2, we have Θ ij = [ ̂ Θ k ] π ( i ) ,π ( j ) = 0 such that

̸

̸

For (37c), the pair ( i, j ) either satisfies ψ ( i ) = ψ ( j ) or ψ ( i ) = ψ ( j ) . For first case, we have S ij -Λ ij ≤ 0 as ( i, j ) / ∈ supp ( T ) . Consequently, we obtain

as R ≥ 0 . For second case where ψ ( i ) = ψ ( j ) , we have Θ ij = [ ̂ Θ k ] π ( i ) ,π ( j ) = 0 . Then, applying (21c) in Lemma A.2, we have

Hence, together with the fact that Θ is primal feasible and positive definite, we conclude that Θ satisfies the KKT conditions of convex Problem (1). Therefore, Θ is the optimal solution of Problem (1) and Theorem 3.3 is obtained.

̸

C Extensions and Discussions

C.1 Extensions to traditional graphical lasso

The traditional graphical lasso is formulated as

The KKT conditions of graphical lasso include:

̸

̸

In the realm of graphical lasso, the thresholded sample covariance matrix is usually defined as

Our main results Theorem 3.3 can be generalized to graphical lasso with some modifications.

Theorem C.1. Suppose that P is the bridge-block decomposition of the thresholded graph, ̂ Θ k is the optimal solution of k -th sub-problem, i.e.,

̸

The decomposed form, which is given as

̸

in which ζ i = 1 S ii ∑ ( i,m ) ∈B T 2 im S ii S mm -T 2 im and ζ i = 0 if ∀ m : ( i, m ) / ∈ B , achieves optimality if Θ ≻ 0 and the following inequality is satisfied:

̸

It is important to note that the decomposed form (47) aligns with the structure of (7), with the key distinction lying in the computation of the thresholded matrix T . The proofs of Theorem C.1 can be conducted similar to Section B.

̸

The condition (48) can be guaranteed under some circumstances. For example, we can set large values for Λ ij whenever i = j and i , j are in different clusters. In this case, the thresholded graph is sparse and the condition (48) is satisfied. Nevertheless, in the majority of practical situations, verifying this condition poses a substantial challenge. Conversely, these conditions are inherently satisfied when MTP 2 constraints are applied.

C.2 Extensions to dense graphs

The proposed method operates under an implicit assumption of sparsity, which is a common characteristic of high-dimensional applications where the focus is on identifying only the most significant relationships among variables. However, it is possible to encounter scenarios where the thresholded graph is dense, making it difficult to identify any bridges for conducting bridge-block decomposition. In such cases, the proposed framework can still be useful, and this section will primarily focus on exploring its potential applications under these circumstances. We will discuss various strategies and techniques for adapting the proposed method to deal with dense graphs, enabling a broader applicability of the proposed bridge-block decomposition approach.

̸

One of typical approaches to make the proposed method applicable is to tune the values of regularization coefficients. As the thresholded matrix is computed as

we can always increase the value of Λ ij to make the thresholded graph supp ( T ) sufficiently sparse. When tuning the hyper-parameters is not recommended, a more general strategy is to use the solution of proposed framework as an initial point of numerical algorithms.

Warm-start algorithm : One of the main possible strengths of the proposed decomposed-form solution is that it can be treated as an initial point (warm-start) for the numerical algorithms specialized for solving Problem 1. To apply the decomposition approach, we first need to derive a suitable vertex partition of the graph based on the given data matrix S and regularization matrix Λ . There are several possible vertex partitioning techniques that can be used, including:

k -edge-connected decomposition: This method divides a connected graph into its k -edge-connected sub-graphs, which can be useful for identifying bridges or bottlenecks in the graph structure.

k -vertex-connected decomposition: This method decomposes a graph into its k -vertex-connected sub-graphs, which can be useful for identifying clusters or communities of vertices that are tightly connected to each other.

k -core decomposition: This method identifies the maximal subgraphs of the graph in which each vertex has a degree of at least k , which can be useful for identifying the most densely connected regions of the graph.

Community detection: This method identifies groups of vertices that are densely connected within the group and sparsely connected to vertices outside the group, which can be useful for identifying clusters or communities of vertices with similar properties or behavior.

Once we obtain a vertex partition P = {V 1 , . . . , V K } , where K refers to the number of clusters, we can compute the initial point using the following form:

̸

in which E e refers to a set of external edges among clusters, i.e.,

̸

and ̂ Θ k is computed as the optimal solution of k -th sub-problem:

Remark C.2 . The initial point is the optimal solution if the vertex-partition P is a bridge-block decomposition of the optimal graph.

Remark C.3 . When the vertex partition P is given as P = {{ 1 } , { 2 } , . . . , { p }} , i.e., V k = { k } for k = 1 , . . . , p , the initial point (50) reduces to a closed-form expression:

In practice, utilizing the warm-start strategy can also enhance the performance of certain numerical algorithms.

ζ i is computed as

-150

-125

-100

-75

-50

0.25

0.50

0.75

1.00

l

Average Test Log-likelihood

MTP2

GL

D Additional Comments on Real-World Experiments

To show that MTP 2 assumption approximately holds for Crop dataset, we selected 20 random subsets from the Crop dataset. For each subset, we computed the Graphical Lasso and the MTP 2 graphical model for different values of using the first 10 observations. The remaining 36 observations were used to calculate the out-of-sample log-likelihood, which was then averaged across all datasets. This process allows us to evaluate how well these models generalize to unseen data.

As depicted in Figure 8 of the attached PDF, the MTP 2 graphical model outperforms the Graphical Lasso, providing a higher test log-likelihood. We present one instance of the estimated graphical lasso model in Figure 9. It reveals that most conditional correlations are positive (red edges, 90%), with a few being negative (blue edges, 10%). This pattern implies strong positive dependence in the Crop data, aligning with the characteristics of MTP 2 .

These results are not unexpected, given that the Crop dataset comprises multiple clusters. Within the same cluster, we expect data points to exhibit greater similarity compared to those in different clusters. This situation signifies a form of positive dependence, thereby justifying the plausibility of the MTP 2 assumption in this problem.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that, in Gaussian MTP2 distributions, bridges in the graph structure have a closed form solution.  They use this observation to suggest practical solutions that can be applied whenever such models are being fit.\n\nSTRENGTHS:\nA nice, more or less self-contained theoretical work that unifies and extends some existing lines of research into MTP2 Gaussian distributions.\n\nWEAKNESSES:\nSome room for improvement in terms of presentation and some minor typos.  It's harder for me to judge practical utility.  While it is true that MTP2 distributions have some practical utility (I've used them myself), it isn't clear that many real datasets meet that requirement (even approximately).\n\nQUESTIONS:\n- \"calculating the thresholded graph, bridges, and clusters, is negligible\" Sure, but I feel that a precise statement and citation is needed here.  In particular, you still have to compute the sample covariance matrix for this.\n\n- I actually had to read it a couple times to understand the exact procedure that was being proposed.  I suggest adding some clarifying details to help the reader.\n\n-  What is the primary limitation that makes this result only apply to MTP2 distributions?  Is it possible that something similar could be true more generally?  I didn't really get a sense of why MTP2 was required in the main text.\n\n-  Do you need Gaussian distributions here?  Can something be done for more general MTP2 distributions, e.g., those that can still be represented as pairwise graphical models?\n\nMisc. typos (only a few listed):\n- \"which severs as a common assumption\"\n- \"make possibilities of solving...\" (revise)\n- \"As thresholded graph plays\"\n- \"garnered considerable attentions\"\n- \"Bridge is one of the important concepts\"\n- \"closed-form solutions in literature heavily\"\n- \"seems inapplicable for dense graph\"\n\nLIMITATIONS:\nSome discussion of limitations, though it would have been nice to see a more clearly identified set of problems yet to be solved.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a graphical lasso problem where the precision matrix is restricted to be symmetric M-matrix and the associated GMRF graph has a special structure. Specifically, the authors consider the situation when the graph allows bridge-block decomposition so that vertices can be partitioned into k parts by cutting k-1 \"bridges\".  Under this situation, the authors show that the original problem can be decomposed into k subproblems, illustrated in theorems in section 3.2. Since the existing algorithm's complexity quickly increases as the dimensionality increases, decomposition into subproblems offers large computational benefits. Authors compare the performance of four different algorithms, using synthetic and real data, with and without leveraging the bridge-block decomposition and show that bridge-block decomposition provides huge computational benefits if the given graph has many edges that are “bridges”.\n\nSTRENGTHS:\nAuthors provide a closed form of the elements of precision matrix corresponding to bridges, and make a connection with existing literature when underlying GMRF graph is an acyclic graph. This method is useful when one wants to solve MTP2 constrained graphical lasso problem with a penalty parameter that allows block-bridge decomposition. This work is resembles the existing literature on graphical lasso (without MTP2 constraint), such as Witten et al (2011); Mazumder and Hastie (2012), where graphical lasso problem can be decomposed to the smaller subproblems when graph has many connected components, and adds a contribution in the context of MTP2 constrained graphical lasso when graph has many bridges.\n\n\nWitten, D. M., Friedman, J. H., & Simon, N. (2011). New insights and faster computations for the graphical lasso. Journal of Computational and Graphical Statistics, 20(4), 892-900.\n\nMazumder, R., & Hastie, T. (2012). Exact covariance thresholding into connected components for large-scale graphical lasso. The Journal of Machine Learning Research, 13(1), 781-794.\n\nWEAKNESSES:\nSimulation settings and real data examples have some discrepancy from usual (graphical) lasso settings and its motivation. The main goal of graphical lasso is to discover the underlying conditional independency structure from the multivariate Gaussian data with various choice of penalty parameter (aka lasso path diagram). However authors first fix the graph in their settings such as preferential attachment graph and stochastic block model and choose penalty parameter according to the graph. It is understandable in the simulation study setting to show the computational benefits with block-bridge decomposition, but fixing the graph in the real data is not convincing, and also it is not clearly stated why MTP2 graphical lasso is appropriate to this crop image data problem (is it reasonable to assume that crop image data comes from high-dimensional multivariate Gaussian distribution? what is the interpretation of the resulting graph? why MTP2 constraint is necessary / plausible in this problem?)\n\nIn practice, graphical lasso is ran under various settings of penalty parameters and chosen appropriately such as cross validation. The existing graphical lasso decomposition (Witten et al (2011); Mazumder and Hastie (2012)) are useful since graphical lasso with high penalty parameter often leads to a graph with many connected components. In similar spirit, I suggest authors to illustrate how often the graph that allows block bridge decomposition appears by the choice of penalty parameter.\n\nQUESTIONS:\n- Does the main result, decomposition into smaller graphical lasso subproblems under block-bridge decomposition, holds without MTP2 constraint? \n- If not, please clarify the the role of the MTP2 constraint in the main results; does MTP2 constraint is necessary for the assumption 2.1. to be hold?\n- What happens if graph has many connected components (without bridges) in MTP2 constrained graphical lasso problem? Does similar decomposition holds?\n- Section 4.1: Sample size $n$ is 10 times the dimension $p$? This setting completely violates the main motivation of graphical lasso where $\\ell_1$ penalty has been proposed since MLE does not exist when $n<p$. \n- Section 4.3: Shouldn't be data $y_i$ is 24000-dimensional with $i=1,\\dots,46$ observations?\n\nLIMITATIONS:\nThe main results are meaningful but simulation results and real data settings are less convincing. No potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of estimating the precision matrix, which is the inverse of the correlation matrix, of a given Gaussian random vector $y$. The precision matrix $\\Theta$ is assumed to satisfy a technical condition called MTP2 which states that $\\Theta$ is symmetric and $\\Theta_{i,j} \\le 0$. This seems to be a well motivated assumption from various applications. The contribution of this paper is a technique for estimating $\\Theta$ as follows: given a predicted sparsity pattern on $\\Theta$ in the form of a graph $G$ , the natural optimization problem for estimating $\\Theta$ can be solved by first solving the optimization problem on smaller 'blocks' and combining them across 'bridges'. They are defined as follows. Bridges are single edge cuts in  $G$ and the resulting connected vertices are called blocks.\n\nSTRENGTHS:\nThe paper shows that given a block bridge decomposition, the optimization problem of estimating $\\Theta$ can be efficiently solved by first solving the problem on the individual blocks and then combining the solution across bridges. The paper gives an explicit formula for doing so. In the case where $G$ is sparse, this can represent significant computational savings over estimating the entire precision matrix at once. Furthermore, since the work provides a structural theorem, any optimization algorithm can be used in conjunction with their observation.\n\nWEAKNESSES:\nI am not familiar with the literature but it seems like a big assumption to know the threshold graph explicitly. What happens if this graph is unknown? It seems to be more natural that the graph is unknown and one must estimate it.\n\nAn intermediate setting which also seems interesting is in the case where we know a noisy approximation to the block bridge structure. How do the proposed methods perform under such noisy information? Are the derived formulas robust?\n\nWhat is the motivation for even assuming the block bridge structure? I can see social networks being one motivation but it would be more convincing if there were experiments on (real) social networks.\n\nQUESTIONS:\nWhat happens to the SBM experiments if the edge probabilities between the different blocks are increased? The quality of the method should deteriorate as the probabilities increase, since the block structure increasingly deteriorates. It would be interesting to see how much the proposed method can tolerate.\n\nLIMITATIONS:\nNo ethical concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning Gaussian Graphical Models (GGMs) satisfying a certain positive associativity condition among the variables, namely that the precision matrix has nonnegative off-diagonal elements. This condition is known as being \"multivariate totally positive of order two\", or MTP$_2$, and has applications in ML (where it corresponds to attractive Markov random fields), finance, and more.\n\nMTP$_2$ GGMs come with the benefit that the traditional optimization procedure used to estimate the precision matrix of a GGM, namely the graphical Lasso, takes on a particularly simple and smooth form. Prior work had shown various polynomial-time convergence guarantees for the graphical Lasso under the MTP$_2$ assumption, but these are still not very suitable for practical applications (scaling with the dimension $p$ as $O(p^3)$ or $O(p^4)$). Other prior work had shown a closed-form solution for the graphical Lasso under the assumption that the \"thresholded sample covariance graph\" (an object defined in terms of the sample covariance and the regularization parameters used in the graphical Lasso), or thresholded graph for short, is acyclic.\n\nThe main contribution of this paper is essentially to generalize the latter closed-form result in terms of the \"bridge-block decomposition\" of the thresholded graph. The bridge-block decomposition of a graph is essentially a partition of the graph into components connected only by \"bridge\" edges. Formally, a bridge edge is an edge such that deleting it increases the number of connected components in a graph; it is effectively \"the only edge\" bridging two different components (see Fig 1). The paper's main theorem (Thm 3.3) essentially says the following: to solve the graphical Lasso for an MTP$_2$ GGM, compute the bridge-block decomposition of the thresholded graph, run the graphical Lasso for each component separately, and stitch them together using a simple closed-form formula. Moreover, this theorem also readily recovers as a special case the prior closed-form result for acyclic thresholded graphs. This is because in an acyclic graph, every edge is a bridge, and the bridge-block decomposition is particularly simple.\n\nThus the main result amounts to a divide-and-conquer recipe for learning MTP$_2$ GGMs, and the authors show various numerical experiments suggesting the practical superiority of this method over all prior methods (which operate on the entire graph). A key benefit is that the subproblems may be solved using any graphical Lasso implementation whatsoever, and potentially in parallel.\n\nSTRENGTHS:\nDisclaimer: I am not very familiar with the literature in this area, and my review should be taken as that of a relative outsider.\n\nThe paper's main result is both an interesting structural result about MTP$_2$ GGMs as well as a genuinely practical algorithmic advance in learning such models. From a conceptual point of view, the idea of leveraging the bridge-block decomposition seems novel. The overall result seems like a useful and nice contribution to the literature on this problem, and to the extent that one considers MTP$_2$ GGMs significant, one should consider this result significant as well.\n\nThe paper is largely clear and easy to follow (modulo some occasionally confusing bits; see the Questions section). It does a good job of setting up the main problem as well as the necessary context. I did not manage to verify the proofs in detail, but they seemed fairly clean, relying on an analysis of the KKT conditions of the graphical Lasso as well as some clever algebraic manipulation.\n\nWEAKNESSES:\nI think the main things to really evaluate about this paper are its novelty and significance. As an outsider to this area, I find this hard to accurately gauge, but I think the paper scores well on these fronts.\n\nI do think the paper could benefit from a better conceptual overview of the main proof and the role of the bridge-block decomposition. The context and benefits are discussed adequately, but the key ideas in the proof do not come through very well, and the main proof seemed slightly magical to me. Why would one have expected the bridge-block decomposition to help? Was its role surprising?\n\nQUESTIONS:\nI think the biggest question I have is about the conceptual role of the bridge-block decomposition, as explained above. A few other questions:\n- One of the key parts of the eventual proof of Thm 3.3 is Lemma A.2, which is simply described as following from the KKT conditions. Presumably the authors mean the KKT conditions associated with Problem (5)? Since it plays a relatively important conceptual role, I feel this could definitely use more elaboration.\n- What exactly is the time required to compute the bridge-block decomposition? This is described as negligible in Sec 3.1, but is it e.g. $O(p^2)$? Similarly, what about the quantities used in the closed-form formula? Currently the only discussion of the overall asymptotic running time appears in the first bullet of the \"Proposed Framework\" list on page 5.\n- In the experimental section, it would be helpful to say more about it is natural to synthesize/define $\\Theta$ and $\\Lambda$ in the specific way described in the first two paragraphs of Section 4.1, especially for readers who are unfamiliar with the prior work mentioned in those paragraphs.\n\nA couple other nits regarding the presentation:\n- It is not immediately obvious how Corollary 3.5 follows from Thm 3.3, and this could use a line of explanation.\n- The notation $\\mathbb{S}^p$ is not formally defined in the paper, and MTP$_2$'s expansion is only mentioned in the abstract.\n\nLIMITATIONS:\nThe paper could use a couple additional lines in the final section about the technical limitations of this work and what the major next steps could be. I am not aware of any significant potential negative societal impact of this theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

comments: SUMMARY:
The paper focuses on the problem of learning large-scale Gaussian graphical models (GGMs) that are multivariate totally positive of order two (MTP2). The high-dimensional, sparse MTP2 GGMs are not easily manageable due to their size and complexity. The authors propose a novel approach, introducing the concept of a "bridge", to optimize the entire problem into several smaller, more manageable sub-problems and a set of closed-form solutions. The approach is based on the bridge-block decomposition of the thresholded sample covariance graph, which leads to reductions in computational complexity and improvements in existing algorithms.

STRENGTHS:
The proposed bridge-block decomposition framework on Gaussian graphical models seems novel. The problem is motivated nicely, and according to the authors, the framework could significantly reduce computational and memory cost. 

The proposed method seems to subsume various network structures, including the BA graph and the SBM, which are common models used in network analysis.

Experimental results are provided, and the computational results look promising.

WEAKNESSES:
As the authors mentioned in the paper, the proposed method might not generalize to dense cases. Still I feel in many settings like BA and SBM, sparsity is a reasonable assumption.

QUESTIONS:
Is there any way to characterize the effect of the proposed approach in terms of some global graphical properties, for example, the edge expansion? Asking this because bridge is related.

LIMITATIONS:
See Above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27