Title: LogSpecT: Feasible Graph Learning Model from Stationary Signals with Recovery Guarantees
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Signal Processing

Spectral Template

Network Inference

Optimization

Linearized ADMM
Abstract: Graph learning from signals is a core task in graph signal processing (GSP). A significant subclass of graph signals called the stationary graph signals that broadens the concept of stationarity of data defined on regular domains to signals on graphs is gaining increasing popularity in the GSP community. The most commonly used model to learn graphs from these stationary signals is SpecT, which forms the foundation for nearly all the subsequent, more advanced models. Despite its strengths, the practical formulation of the model, known as rSpecT, has been identified to be susceptible to the choice of hyperparameters. More critically, it may suffer from infeasibility as an optimization problem. In this paper, we introduce the first condition that ensures the infeasibility of rSpecT and design a novel model called LogSpecT, along with its practical formulation rLogSpecT to overcome this issue. Contrary to rSpecT, our novel practical model rLogSpecT is always feasible. Furthermore, we provide recovery guarantees of rLogSpecT from modern optimization tools related to epi-convergence, which could be of independent interest and significant for various learning problems. To demonstrate the practical advantages of rLogSpecT, a highly efficient algorithm based on the linearized alternating direction method of multipliers (L-ADMM) that allows closed-form solutions for each subproblem is proposed with convergence guarantees. Extensive numerical results on both synthetic and real networks not only corroborate the stability of our proposed methods, but also highlight their comparable and even superior performance than existing models.

LogSpecT: Feasible Graph Learning Model from Stationary Signals with Recovery Guarantees

Abstract

Graph learning from signals is a core task in graph signal processing (GSP). A significant subclass of graph signals called the stationary graph signals that broadens the concept of stationarity of data defined on regular domains to signals on graphs is gaining increasing popularity in the GSP community [10, 20, 25]. The most commonly used model to learn graphs from these stationary signals is SpecT [31], which forms the foundation for nearly all the subsequent, more advanced models. Despite its strengths, the practical formulation of the model, known as rSpecT , has been identified to be susceptible to the choice of hyperparameters. More critically, it may suffer from infeasibility as an optimization problem. In this paper, we introduce the first condition that ensures the infeasibility of rSpecT and design a novel model called LogSpecT , along with its practical formulation rLogSpecT to overcome this issue. Contrary to rSpecT, our novel practical model rLogSpecT is always feasible. Furthermore, we provide recovery guarantees of rLogSpecT from modern optimization tools related to epi-convergence, which could be of independent interest and significant for various learning problems. To demonstrate the practical advantages of rLogSpecT, a highly efficient algorithm based on the linearized alternating direction method of multipliers (L-ADMM) that allows closed-form solutions for each subproblem is proposed with convergence guarantees. Extensive numerical results on both synthetic and real networks not only corroborate the stability of our proposed methods, but also highlight their comparable and even superior performance than existing models.

1 Introduction

Learning with graphs has proved its relevance in many practical areas, such as life science [36, 37], signal processing [13, 14, 38, 39], and financial engineering [1, 21], to just name a few. However, there are many cases that the graphs are not readily prepared and only the data closely related to the graphs can be observed. Hence, a core task in Graph Signal Processing (GSP) is to learn the underlying graph topology based on the interplay between data and graphs [22].

Among these commonly used data properties, an assumption that is referred to as the graph signal stationarity [10] is gaining attention [30, 20, 25]. Graph signal stationarity is the extension of the notion of stationarity from the regular space/time domain to the irregular domain (i.e. graphs). The latter is a traditional hypothesis in signal processing to capture a special type of statistical relationship between samples of a temporal signal. It is a cornerstone of many signal processing methods. Although the graph signal stationarity property is proposed more from a theoretic end, the

techniques based on this idealistic assumption pave the way for the development of the following more advanced and applicable models [35, 34]. Furthermore, several works have shown that some real datasets can be approximately viewed as stationary or partially explained by the stationarity assumption. For instance, [25] revealed that the well-known USPS dataset and the CMUPIE set of cropped faces exhibit near stationarity. [10] found that certain weather data could be explained by stationary graph signals. Additionally, [31] highlighted the relevance of the stationarity assumption in learning protein structure. [43] applied the stationary assumption to US Senate Roll Calls and achieved good performance.

The predominant methods to process stationary graph signals and learn topology under the stationarity assumption are the spectral-template-based models. The start of this line of works is [31], which proposed a vanilla model called SpecT [31] to learn graphs from stationary graph signals. Many extensions of this fundamental model have been made since then to accommodate the more realistic and complicated settings, e.g. joint inference of multiple graphs [32, 27], learning with streaming graph signals [33], learning with hidden nodes [4], learning from both smooth and stationary graph signals [5]. In practice, SpecT requires the unknown data covariance matrix. Hence, a robust formulation called rSpecT is proposed [31], which replaces the unknown covariance matrix with its estimate and introduces a hyperparameter to reflect the estimation inaccuracy. Unfortunately, the model is sensitive to this hyperparameter and improper tuning of it may jeopardize the model performance or lead to model infeasibility.

The current approach to selecting an appropriate value is heuristic. It finds the smallest one that allows for a feasible solution. This method has two shortcomings. On one hand, It is computationally costly. The smallest value is the exact solution to a second-order conic programming, which is solved approximately with computational methods. Any approximate solution that is smaller than the exact one will make rSpecT infeasible. On the other hand, such an approach lacks theoretical guarantees as how the smallest value behaves as the number of observations increases is unknown. This issue makes the existing recovery guarantees invalid. To the best of our knowledge, only a few works have focused on how to understand the performance of the robust formulation rSpecT. [24] empirically showed that the feasible set of the robust formulation approximates the original one as more samples are collected. [23] provided the first recovery guarantee for rSpecT. Their work exploits the fact that the objective is the ℓ 1 -norm and the constraints are linear [44]. However, the conditions needed in their work are not only restrictive but also hard to check, as they require the full rankness of a large-scale matrix related to random samples. Another model [34] tries to circumvent the model infeasibility issue by turning a constraint into a penalty. However, this approach introduces another hyperparameter that is neither easy to tune nor amenable to theoretical analysis. Also, this hyperparameter should go to infinity heuristically when more and more samples are collected, which may exacerbate model instability.

In this paper, we propose an alternative formulation of SpecT that can bypass the aforementioned issues. We first provide a condition that ensures the infeasibility of the fundamental model rSpecT. Then, we propose a novel formulation called LogSpecT and its practical formulation rLogSpecT based on the spectral template and incorporating a log barrier. The inclusion of a log barrier as a regularizer on the degrees is inspired by its extensive application in graph learning from smooth graph signals [15, 16, 19, 47] and the discussion in [7] emphasizing that restricting the degrees to preclude isolated nodes is crucial to learning graphs with special structures. It is worth noting that the approach in [7], which directly imposes constraints, may exacerbate the infeasibility issue in our problem. Although rLogSpecT still requires a parameter to reflect the estimation error of the unknown covariance matrix, it no longer suffers from the infeasibility issue associated with the improper choice of the parameter. Furthermore, we provide the decaying rate of the parameter that can guarantee the recovery of rLogSpecT. This not only aids in the parameter tuning, but also establishes a connection between rLogSpecT and LogSpecT through non-asymptotic recovery analysis. These theoretical results are derived using contemporary optimization tools related to epi-convergence [29]. Unlike the current guarantees for rSpecT that are built on an ℓ 1 analysis model [44], our approach based on epi-convergence is not confined to the specific types of optimization problems (e.g. the combination of the ℓ 1 -norm and log barrier in the objective function) and consequently admits broader applications that can be of independent interest.

In the algorithmic aspect, we design a linearized alternating direction method of multipliers (LADMM) to solve rLogSpecT. The subproblems of L-ADMM admit closed-form solutions and can be implemented efficiently due to the linearization technique. Also, we provide the convergence result of the proposed method. Finally, we conduct extensive experiments on both synthetic data and

real networks. Notably, the infeasibility issue of rSpecT frequently occurs in experiments on real networks, thereby emphasizing its practical significance beyond theoretical concerns. Furthermore, we observe that even when the classical models are set to be feasible, our novel models (LogSpecT and rLogSpecT) exhibit improved accuracy and stability, particularly in the case of synthetic BA graphs. This empirically illustrates that although our models are initially proposed to circumvent the infeasibility issue, they are superior alternatives to classical methods in the general cases.

1.1 Notation

The notation we use in this paper is standard. We use [ m ] to denote the set { 1 , 2 , . . . , m } for any positive integer m . Let the Euclidean space of all real matrices be equipped with the inner product ⟨ X , Y ⟩ := tr( X ⊤ Y ) for any matrices X , Y and denote the induced Frobenius norm by ∥ · ∥ F (or ∥ · ∥ 2 when the argument is a vector). For any X ∈ R m × n , we use B ( X , ρ ) to denote the closed Euclidean ball centering at X with radius ρ . Let ∥ · ∥ be the operator norm, ∥ X ∥ 1 , 1 := ∑ i,j | X ij | , ∥ X ∥ ∞ , ∞ := max i,j | X ij | , and let diag( X ) be vector formed by the diagonal entries of X . For a column vector x , let Diag( x ) be the diagonal matrix whose diagonal elements are given by x . Given a closed and convex set C , we use Π C ( w ) := argmin v ∈C ∥ v -w ∥ 2 to denote the projection of the point w onto the set C . We use 1 (resp. 0 ) to denote an all-one vector (resp. all-zero vector) whose dimension will be clear from the context. For a set S and any real number α , let α S := { αx | x ∈ S} , S 1 := { x 1 | x ∈ S} , and ι S ( · ) be the indicator function of S . For two non-empty and compact sets X and Y , the distance between them is defined as dist ( X , Y ) := sup X ∈X inf Y ∈Y ∥ X -Y ∥ F 1 .

2 Preliminaries

Let G = ( V , E ) be a graph, where V = [ m ] is the set of nodes and E ⊂ V×V is the set of edges. Let S be the weight matrix associated with the graph G , where S ij represents the weight of the edge between nodes i and j . In this paper, we consider undirected graphs without self-loops. The set of adjacency matrices for such graphs is S := { S ∈ R m × m | S = S ⊤ , diag( S ) = 0 , S ≥ 0 } . Suppose that the adjacency matrix S admits the eigen-decomposition S = U Λ U ⊤ , where Λ is a diagonal matrix and U is an orthogonal matrix. A graph filter is a linear operator h : R m × m → R m × m defined as h ( S ) = ∑ p i =0 h i S i = U (∑ p i =0 h i Λ i ) U ⊤ , where p > 0 is the order of the graph filter and { h i } p i =0 are the filter coefficients. According to the convention, we have S 0 = I .

A graph signal can be represented by a vector x ∈ R m , where the i -th element x i is the signal value associated with node i . A signal x is said to be stationary if it is generated from

where w satisfies E [ w ] = 0 and E [ w ⊤ w ] = I . Simple calculations give that the covariance matrix of x , which is denoted by C ∞ , shares the same eigenvectors with S . Hence, we have the constraint

Based on this, the following fundamental model SpecT is proposed to learn graphs from stationary signals without the knowledge of graph filters [31, 34]:

where the constraint ∑ m j =1 S 1 j = 1 is used to preclude the trivial optimal solution S ∗ = 0 . When C ∞ is unknown and only n i.i.d samples of { x i } from (1) are available, the robust formulation, which is based on the estimate C n of C ∞ and called rSpecT , is used:

For this robust formulation, the recovery guarantee is studied empirically in [24] and theoretically in [23] under some conditions that are restrictive and hard to check.

3 Infeasibility of rSpecT and Novel Models

Even though rSpecT has gained much popularity, there are few works that discuss the choice of the hyperparameter δ and how it affects the model feasibility. In this section, we present a condition under which rSpecT is always infeasible and then propose an alternative formulation. To motivate our results, let us consider the following 2-node example.

̸

Example 3.1. Consider a simple graph containing 2 nodes. Then, the set given by the second constraint of rSpecT is S ∩ { S ∈ R m × m : ∑ m j =1 S 1 j = 1 } = [0 , 1; 1 , 0] , which is a singleton. Suppose that the sample covariance matrix is C n = [ h 11 , h 12 ; h 12 , h 22 ] . Then, the constraint ∥ C n S -SC n ∥ F ≤ δ is reduced to 2( h 11 -h 22 ) 2 ≤ δ 2 . Hence, when h 11 = h 22 and δ < √ 2 | h 11 -h 22 | , rSpecT has no feasible solution.

Before delving into the general case, we introduce a linear operator B ∈ R m 2 × m ( m -1) / 2 that maps the vector x ∈ R ( m -1) m/ 2 + to the vectorization of an adjacency matrix S ∈ S of a simple, undirected graph:

where i, j ∈ [ m ] . We also define A n := I ⊗ C n -C n ⊗ I ∈ R m 2 × m 2 , so that the first constraint of rSpecT can be rewritten as ∥ C n S -SC n ∥ F ≤ δ ⇐⇒ ∥ A n vec ( S ) ∥ 2 ≤ δ , where vec ( · ) is the vectorization operator. We now give a condition that guarantees the infeasibility of rSpecT.

Theorem 3.2. Consider the linear system

̸

where y ∈ R m ( m -1) 2 . If (4) has no feasible solution, then there exists a ¯ δ n > 0 such that rSpecT is infeasible for all δ n ∈ [0 , ¯ δ n ) .

Remark 3.3. From Theorem 3.2 we can infer that for any fixed n , the linear system (4) has no solution when A n B has full column rank (e.g., Example 3.1). This leads to the infeasibility of rSpecT with δ n ∈ [0 , ¯ δ n ) . As remarked in [31], one should tackle the feasibility issue of rSpecT with caution.

The failure of rSpecT (SpecT) lies in the existence of the constraint ( S 1 ) 1 = 1 , which is used to preclude the trivial solution S ∗ = 0 . For this reason, we resort to an alternative approach to bypassing the trivial solution. When the graphs are assumed to have no isolated node, the log barrier is commonly applied [15]. In these cases, the zero solution is naturally precluded. This observation inspires us to propose the following novel formulation, which combines the log barrier with the spectral template (2) to learn graphs without isolated nodes from stationary signals:

where ∥ · ∥ 1 , 1 is a convex relaxation of ∥ · ∥ 0 promoting graph sparsity, 1 ⊤ log( S 1 ) is the penalty to guarantee the nonexistence of isolated nodes, and α is the tuning parameter. As can be seen from the following proposition, the hyperparameter α in LogSpecT only affects the scale of edge weights instead of the graph connectivity structure.

Proposition 3.4. Let Opt( C , α ) be the optimal solution set of LogSpecT with input covariance matrix C and parameter α > 0 . Then, for any γ > 0 , it follows that

Remark 3.5. The result of Proposition 3.4 spares us from tuning the hyperparameter α when we are coping with binary graphs. In fact, certain normalization will eliminate the impact of different values of α and preserve the connectivity information. Hence, we may simply set α = 1 in implementation.

Note that the true covariance matrix C ∞ in LogSpecT is usually unknown and an estimate C n from n i.i.d samples is available. To tackle the estimation inaccuracy, we introduce the following robust

formulation:

This formulation substitutes C ∞ with C n and relaxes the equality constraint to an inequality constraint with a tolerance threshold δ n . Contrary to rSpecT, we prove that rLogSpecT is always feasible.

Proposition 3.6. For δ n > 0 and C n with any fixed n , rLogSpecT always has a nontrivial feasible solution.

4 Recovery Guarantee of rLogSpecT

In this section, we investigate the non-asymptotic behavior of rLogSpecT when more and more i.i.d samples are collected. For the sake of brevity, we denote f := ∥·∥ 1 , 1 -α 1 ⊤ log ( · 1 ) . The theoretical recovery guarantee is as follows:

Theorem 4.1. If δ n ≥ 2 ρ ∥ C n -C ∞ ∥ with given ρ := max { f ∗ , αm, αm (log α -1) } , then there exist constants c 1 , ˜ c 1 , ˜ c 2 > 0 such that

where ε n := c 1 ( δ n +2 ρ ∥ C n -C ∞ ∥ ) , S n, ∗ is the optimal solution set of rLogSpecT, f ∗ (resp. f ∗ n ) denotes the optimal value of LogSpecT (resp. rLogSpecT), and

is the ε -suboptimal solution set of LogSpecT.

Remark 4.2. (i) Compared with the conclusion (ii) in the Theorem 4.1, the conclusion (iii) links the optimal solution sets of rLogSpecT and LogSpecT instead of the sub-optimal solutions.

(ii) A byproduct of the proof (see Appendix E.2 for details) shows that the optimal node degree vector set ( S ∗ 0 1 ) is a singleton. However, there is no guarantee of the uniqueness of S ∗ itself. We will discuss the impact of such non-uniqueness on model performance in Section 6.4.

Remark 4.3. The proof of Theorem 4.1 relies on two important optimization concepts: Epiconvergence and truncated Hausdorff distance. Epi-convergence is closely related to the asymptotic solution behavior of approximate minimization problems, and the truncated Hausdorff distance is used to characterize the epi-convergence non-asymptotically. With the help of the Kenmochi condition, which allows us to explicitly calculate the truncated Hausdorff distance, we are able to study the non-asymptotic behavior of the optimal value and optimal solutions of the models. We refer the reader to Appendix E and also [28, Chapter 7], [29, Chapter 6.J] for more details.

Corollary 4.4. Under the assumptions in Theorem 4.1, it follows that if δ n → 0 , then

Remark 4.5. From the strong law of large numbers, if we choose δ n = O ( ∥ C n -C ∞ ∥ ) , then almost surely one has lim n →∞ ∥ C n -C ∞ ∥ = 0 and consequently δ n → 0 . Hence, Corollary 4.4 shows that (5) holds almost surely.

For the remaining part of this section, we study the choice of δ n under certain statistical assumptions. A large number of distributions (e.g., Gaussian distributions, exponential distributions and any bounded distributions) can be covered by the sub-Gaussian distribution (cf. [41, Section 2.5]), whose formal definition is as follows.

Definition 4.6 (Sub-Gaussian Distributions) . The probability distribution of a random vector w is called sub-Gaussian if there are C, v > 0 such that for every t > 0 , P ( ∥ w ∥ 2 > t ) ≤ Ce -vt 2 .

Consider the case that w in the generative model (1) follows a sub-Gaussian distribution. The following result is adapted from [40, Proposition 2.1].

Lemma 4.7. Suppose that w in the generative model (1) follows a sub-Gaussian distribution. Then, x follows a sub-Gaussian distribution, and with probability larger than 1 -O ( 1 n ) , one has ∥ C n -C ∞ ∥ ≤ O ( √ log n/n ) .

Equipped with non-asymptotic results in Theorem 4.1, we choose δ n with a specific decaying rate.

Corollary 4.8. If the input w in (1) follows a sub-Gaussian distribution and δ n = O ( √ log n/n ) , then the assumptions in Theorem 4.1 hold with probability larger than 1 -O ( 1 n ) .

Corollary 4.8 together with Theorem 4.1 illustrates the non-asymptotic convergence of optimal function value/suboptimal solution set/optimal node degree vector of rLogSpect from n i.i.d samples to the ideal model LogSpect with the convergence rate O ( ε n ) / O ( ε n ) / O ( √ ε n ) , where ε n = O ( | C n -C ∞ | ) ≤ O ( √ log n/n ) for sub-Gaussian ω . This matches the convergence rate O ( √ log n/n ) of classic spectral template models, e.g., Proposition 2 in [31] and Theorem 2 in [23] for the SpecT model, which shows that LogSpecT is also competitive on recovery rate.

5 Linearized ADMM for Solving rLogSpecT

In this section, we design a linearized ADMM algorithm to solve rLogSpecT that admits closed-form solutions for the subproblems. The ADMM-type algorithms have been successfully applied to tackle various graph learning tasks [46, 42].

Firstly, we reformulate rLogSpecT such that it fits the ADMM scheme:

The augmented Lagrangian function of problem (6) is

Update of primal variables Z and q Since Z and q are separable given S , we update the primal variables in two blocks: S and ( Z , q ) . More specifically, the update of ( Z , q ) in the k -th iteration is

which admits the following closed-form solution:

Proposition 5.1. The update (7) can be explicitly rewritten as

Update of primal variables S The ADMM update of S does not admit a closed-form solution. Hence, we solve the linearized version ˆ L k of the augmented Lagrangian function L when all the variables except S are fixed. It is given by

where C ( k ) := 11 ⊤ + C n Λ ( k ) -Λ ( k ) C n -λ ( k ) 2 1 ⊤ + ρ ( S ( k ) C 2 n + C 2 n S ( k ) -2 C n S ( k ) C n + Z ( k +1) C n -C n Z ( k +1) -q ( k +1) 1 ⊤ + S ( k ) 11 ⊤ ) and τ is the parameter for linearization. Therefore, the update of S in the k -th iteration is

where Π S ( X ) is the projection of X onto the closed and convex set S and can be computed by

̸

Update of dual variables and parameters Finally, we update the dual variables Λ and λ 2 via gradient ascent steps of the augmented Lagrangian function:

Together with the updating rule for the augmented parameter ρ , we complete the whole algorithm. The details are presented in Appendix F.

Complexity of each iteration To illustrate the efficiency of L-ADMM, we calculate the complexity of each subproblem. In the update of Z and q , the most time consuming part is to calculate C n S ( k ) . When the number of nodes m is larger than the sample size n , we rewrite C n S ( k ) as (1 /n X )( X ⊤ S ( k ) ) , where X ∈ R m × n is the data matrix. In this case, the complexity is in the order of O ( nm 2 ) . When m is smaller than n , we first calculate C n and then multiply it with S ( k ) . This gives the complexity O ( nm 2 ) . Similarly, the complexity needed to update S is O ( nm 2 + mn 2 ) . The update of the dual variables can be decomposed as the summation of components calculated when updating the primal variables. In summary, the total complexity of each iteration is O ( nm 2 + mn 2 ) .

Convergence of L-ADMM For the convergence analysis of L-ADMM, we treat ( Z , q ) as one variable, and the two constraints C n S -SC n = Z and q = S 1 can be written into a single one:

Then we can apply a two-block proximal ADMM (i.e., L-ADMM) to the problem by alternatingly updating S (see details in Appendix F.4 for this part) and ( Z , q ). Consequently, the convergence result in [45, Theorem 4.2] can be invoked directly to derive the following theorem.

6 Experiments

In this section, we evaluate our proposed algorithm and models via numerical experiments. The experiments are conducted on both synthetic and real networks. We apply the standard metrics [18]: F-measure, Precision, and Recall to assess the quality of learned graphs. The source code is available at: https://github.com/StevenSYL/NeurIPS2023-LogSpecT .

6.1 Data Generation

Random Graphs . In the experiments, we consider two types of synthetic graphs, namely, the Erd˝ os-Rényi (ER) graph [9] and the Barabasi-Albert model graph (BA) [2]. The ER graphs are generated by placing an edge between each pair of nodes independently with probability p = 0 . 2 and the weight on each edge is set to 1. The BA graphs are generated by having two connected nodes initially and then adding new nodes one at a time, where each new node is connected to exactly one previous node that is randomly chosen with a probability proportional to its degree at the time.

Graph Filters and Signals . Three graph filters are used in the experiments. The first one is the low-pass graph filter (lowpass-EXP) h ( S ) = exp( 1 2 S ) . The second one is the high-pass graph filter (highpass-EXP) h ( S ) = exp( -S ) . The last one is a quadratic graph filter (QUA) h ( S ) = S 2 + S + I . Note that the quadratic graph filter is neither low-pass nor high-pass. We will also use random graph filters developed from the above ones for experiments on real networks. They will be introduced when applied. The stationary graph signals are generated from the generative model (1). The input graph signal w ∼ N ( 0 , I m ) is a random vector following the normal distribution.

6.2 How to Infer Binary Graphs

Many practical tasks require binary graphs instead of weighted graphs. However, the results from LogSpect and rLogSpecT are generally weighted. In this section, we tackle the issue of how to convert a weighted graph to a binary one. Firstly, we normalize each edge in the graph W by dividing it over the maximal weight, which yields a collection of values ranging from 0 to 1 . Our task is then to select a threshold ε to round these values to 0 or 1 and construct a binary graph W ∗ accordingly. Mathematically, this procedure is captured by the formula:

To choose the threshold ε , we either use a training-based strategy or a searching-based strategy. The training-based one searches the best ε on k graphs and applies the value to the newly learned graph. The searching-based one simply searches for ε that yields the best performance for a graph.

6.3 Convergence of L-ADMM

We present the convergence performance of L-ADMM in this section. Since there does not literally exist customized algorithm for rLogSpecT, we compare our L-ADMM with CVXPY [8]. The solver used in CVXPY is MOSEK. We conduct the experiments on 100-node BA graphs with 10000 stationary graph signals generated from low-pass EXP graph filter. The parameter δ is set as 10 √ log n/n . For L-ADMM algorithm, the target accuracy is set as 10 -6 and the initialization is set as zero. The running time for L-ADMM takes around 20 seconds while the solver takes over 110 seconds. The primal residual and dual residual of L-ADMM in each iteration are plot in Figure 1. The results corroborate the efficiency of our proposed algorithm. Furthermore, the locally linear convergence can be observed from the experiment. This implies that L-ADMM converges with fast speed to high accuracy. It remains an open question to prove the property theoretically.

0

500

1000

1500

2000

2500

3000

iteration

10 -5

10 -3

10 -1

10 1

10 3

Residuals

primal residual

dual residual

10 1

10 2

10 3

10 4

10 5

10 6

Sample Size

0.0

0.2

0.4

0.6

0.8

1.0

F-measure

rLogSpecT Performance

lowpass-EXP

highpass-EXP

QUA

20

30

40

50

60

0.0

0.2

0.4

0.6

0.8

1.0

Average F-measure

LogSpecT performance

20

30

40

50

60

0.0

0.2

0.4

0.6

0.8

1.0

Average F-measure

SpecT performance

ER-lowpass-EXP

ER-highpass-EXP

ER-QUA

BA-lowpass-EXP

BA-highpass-EXP

BA-QUA

Number of Nodes

Number of Nodes

6.4 Experiments on Synthetic Networks

To evaluate the efficacy of LogSpecT and rLogSpecT, we compare them with SpecT and rSpecT on synthetic data. Notice that the prior of no isolated nodes has been incorporated in practice by adding restrictions on the smallest degree. This approach introduces an additional hyperparameter and may exacerbate the infeasibility issue in robust formulation. Also, no recovery guarantees are developed for this trick. For these reasons, we still compare LogSpect & rLogSpecT with SpecT & rSpecT.

Performance of LogSpecT . We first compare the performance of the ideal models: LogSpecT and SpecT on the two types of random graphs with three graph filters. We conduct experiments on graphs with nodes ranging from 20 to 60 and use the searching-based strategy to set threshold ε ∗ . After 10 repetitions, we report the average results in Figure 3. The left column presents the performance of LogSpecT and the right column presents that of SpecT. Firstly, we observe that on ER graphs, both models achieve good performance. They nearly recover the graphs perfectly. However, the performance on BA graphs differs. In this case, LogSpecT can work efficiently. The learned graphs from LogSpecT enjoy much higher F-measure values than those from SpecT. Secondly, the comparison between different graph filters shows that different graph filters have few impacts on recovery performance for both LogSpecT and SpecT. Finally, we observe that the outputs of LogSpecT on BA graphs tend to possess higher F-measure when the number of nodes increases. This suggests that LogSpecT may behave better on larger networks. However, such phenomena cannot be observed from SpecT results.

Performance of rLogSpecT . Since LogSpecT persuasively outperforms SpecT, we only present the performance of rLogSpecT when more samples are collected. As we have mentioned before, the optimal solution to LogSpecT is not necessarily unique. Thus, we do not expect to be able to show how rLogSpecT's optimal solution converges to LogSpecT's optimal solution. Moreover, the non-uniqueness may jeopardize the performance of rLogSpecT intuitively. However, the following experiment on BA graphs shows that the learned graphs from rLogSpecT with the binary-graphtransforming strategy tend to approach the ground truth when enough samples are collected.

In this experiment, the sample size n is chosen from 10 to 10 6 and δ n is set as 0 . 2 √ log n/n . We rely on the training-based strategy to obtain the best threshold ε ∗ from 10 randomly chosen training graphs. We then calculate the F-measure of the learned binary graph in the testing set. The result is reported in Figure 2. It shows that rLogSpecT works for all three graph filters on BA graphs and tends to possess higher F-measure values when more and more signals are collected. This indicates that similar to LogSpecT, rLogSpecT is efficient for different types of graph filters on BA graphs, including the high-pass ones. For more experiments and discussions, we refer readers to Appendix G.

6.5 Experiments on Real Networks

In this set of experiments, we compare the performance of LogSpecT (resp. rLogSpecT) with SpecT (resp. rSpecT) and other methods from the statistics and GSP communities on Protein database and Reddit database from [6]. The Protein database is a Bioinformatics dataset, where nodes are secondary structure elements (SSEs) and there is an edge between two nodes if they are neighbors in the amino-acid sequence or in 3D space. The Reddit database is a social network dataset. In this dataset, each graph corresponds to an online discussion thread, where nodes correspond to users, and there is an edge between two nodes if at least one of them responds to another's comments. We choose graphs in databases whose numbers of nodes are smaller than 50 and generate stationary graph signals from the generative model (1), with the graph filtering chosen as H ( S ) = exp 1 2 S . Our experiments include in total 871 testing graphs in the Protein database and 309 in Reddit database.

Infeasibility of rSpecT . For these real networks, we first check whether the infeasibility issue encountered by rSpecT is significant. To this end, we adopt the random graph filters t 1 S 2 + t 2 S + t 3 I , where t i , i = 1 , 2 , 3 are random variables following a Gaussian distribution with µ = 0 and σ = 2 . Then, we calculate the smallest δ n such that rSpecT is feasible. If the smallest δ n > 0 , rSpecT is likely to be encountered with the infeasibility issue. The results with different numbers of graph signals observed are shown in Table 1.

The experiment results present a decrease in the mean value of δ min when more samples are used. However, a high frequency of the infeasibility issue occurring in these real datasets cannot be mitigated by the increasing sample size. These observations highlight the necessity of the careful treatment for choosing δ in rSpecT again.

Performance of different graph learning models . In this section, the stationary graph signals are generated by the low-pass-EXP filter. We choose the famous statistical method called (thresholded) correlation [17] and the first GSP method that applies the log barrier to graph inference [15] as the baselines. The optimal threshold for the correlation method is selected from 0 . 1 to 0 . 6 and we search in { 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000 } to obtain the best hyperparameter in Kalofolias' model. The parameter δ n in rLogSpecT is set as 10 √ log n/n and in rSpecT it is set as the smallest value that allows a feasible solution [31]. We also rely on the searching-based strategy to convert the learned weighted graphs from (r)SpecT & (r)LogSpecT.

The results of the ideal models with the true covariance matrix C ∞ applied are collected in Figure 4. We observe that on the real graphs, LogSpecT achieves the best performance on average (median represented by the red line). Also, compared with SpecT, LogSpecT performs more stably. We remark that since the graph signals are not necessarily smooth, Kalofolias' model cannot provide guaranteed performance, especially on the Reddit networks.

LogSpecT

SpecT

Correlation Kalofolias

0.0

0.2

0.4

0.6

0.8

1.0

F-measure

Protein Networks

LogSpecT

SpecT

Correlation Kalofolias

0.0

0.2

0.4

0.6

0.8

1.0

F-measure

Reddit Networks

10 1

10 2

10 3

10 4

10 5

10 6

Sample Size

0.65

0.70

0.75

0.80

0.85

0.90

0.95

Average F-measure

Protein Networks

rLogSpecT

rSpecT

Correlation

Kalofolias

10 1

10 2

10 3

10 4

10 5

10 6

Sample Size

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Average F-measure

Reddit Network

Figure 5 compares the performance of four different methods when different numbers of signals are observed. 2 When the sample size increases, the models focusing on stationarity property can recover the graphs more accurately while correlation method and Kalofolias' method fail to exploit the signal information. This can also be inferred from the experiment results in Figure 4 since the models fail to achieve good performance from full information, let alone from partial information. The experiment also shows that when a fixed number of samples are observed, the learned graphs from rLogSpecT approximate the ground truth better than rSpecT. This further corroborates the superiority of rLogSpecT on graph learning from stationary signals.

7 Conclusion

In this paper, we directly tackle the infeasibility issue encountered in graph learning from stationary graph signals [31] and propose the first condition that guarantees the model infeasibility. To overcome this, we propose an efficient alternative. The recovery guarantees of its robust formulation are analyzed with advanced optimization tools, which may find broader applications in learning tasks. Compared with current literature [23], these theoretical results require less stringent conditions. We also design an L-ADMM algorithm that allows for efficient implementation and theoretical convergence. Extensive experiments on both synthetic and real data are conducted in this paper. The results show that our proposed model can significantly outperform the existing ones.

We believe this work represents an important step beyond the fundamental model SpecT. Its general formulation allows for the transfer of the current extensions made on SpecT. Testing our proposed models with these extensions is one future direction. Also, we notice that although the recovery guarantees for robust formulations are clear, the estimation performance analysis for the ideal-case models (i.e. SpecT and LogSpecT) is still incomplete. Investigating the exact recovery conditions is another future direction.

Acknowledgments and Disclosure of Funding

This work was supported in part by the Hong Kong Research Grants Council (RGC) General Research Fund (GRF) Project CUHK 14203920.

References

Daron Acemoglu, Asuman Ozdaglar, and Alireza Tahbaz-Salehi. Systemic risk and stability in financial networks. American Economic Review , 105(2):564-608, 2015.

Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science , 286(5439):509-512, 1999.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Andrei Buciulea, Samuel Rey, Cristobal Cabrera, and Antonio G Marques. Network reconstruction from graph-stationary signals with hidden variables. In Proceedings of the 53rd Asilomar Conference on Signals, Systems, and Computers , pages 56-60. IEEE, 2019.

Andrei Buciulea, Samuel Rey, and Antonio G Marques. Learning graphs from smooth and graph-stationary signals with hidden variables. IEEE Transactions on Signal and Information Processing over Networks , 8:273-287, 2022.

Samir Chowdhury and Tom Needham. Generalized spectral clustering via gromov-wasserstein learning. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS 2021) , pages 712-720. PMLR, 2021.

Jose Vinicius de Miranda Cardoso, Jiaxi Ying, and Daniel Palomar. Graphical models in heavy-tailed markets. Advances in Neural Information Processing Systems , 34:19989-20001, 2021.

Steven Diamond and Stephen Boyd. Cvxpy: A python-embedded modeling language for convex optimization. The Journal of Machine Learning Research , 17(1):2909-2913, 2016.

Paul Erd˝ os, Alfréd Rényi, et al. On the evolution of random graphs. Publication of the Mathematical Institute of the Hungarian Academy of Sciences , 5(1):17-60, 1960.

Benjamin Girault. Stationary graph signals using an isometric graph translation. In Proceedings of the 23rd European Signal Processing Conference (EUSIPCO 2015) , pages 1516-1520. IEEE, 2015.

Yiran He and Hoi-To Wai. Detecting central nodes from low-rank excited graph signals via structured factor analysis. IEEE Transactions on Signal Processing , 70:2416-2430, 2022.

Alan J Hoffman. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards , 49(4):263-265, 1952.

Alexander Jung, Alfred O Hero III, Alexandru Cristian Mara, Saeed Jahromi, Ayelet Heimowitz, and Yonina C Eldar. Semi-supervised learning in network-structured data via total variation minimization. IEEE Transactions on Signal Processing , 67(24):6256-6269, 2019.

Alexander Jung and Nguyen Tran. Localized linear regression in networked data. IEEE Signal Processing Letters , 26(7):1090-1094, 2019.

Vassilis Kalofolias. How to learn a graph from smooth signals. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) , pages 920929. PMLR, 2016.

Vassilis Kalofolias, Andreas Loukas, Dorina Thanou, and Pascal Frossard. Learning time varying graphs. In Proceedings of 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017) , pages 2826-2830. IEEE, 2017.

Eric D. Kolaczyk. Statistical Analysis of Network Data: Methods and Models . Springer-Verlag, New York, NY, USA, 2009.

D. Christopher Manning, Hinrich Schütze, and Prabhakar Raghavan. Introduction to Information Retrieval . Cambridge University Press, 2008.

Hermina Petric Maretic and Pascal Frossard. Graph laplacian mixture model. IEEE Transactions on Signal and Information Processing over Networks , 6:261-270, 2020.

Antonio G Marques, Santiago Segarra, Geert Leus, and Alejandro Ribeiro. Stationary graph processes and spectral estimation. IEEE Transactions on Signal Processing , 65(22):5911-5926, 2017.

Gautier Marti, Frank Nielsen, Mikołaj Bi´ nkowski, and Philippe Donnat. A review of two decades of correlations, hierarchies, networks and clustering in financial markets. Progress in Information Geometry , pages 245-274, 2021.

Gonzalo Mateos, Santiago Segarra, Antonio G Marques, and Alejandro Ribeiro. Connecting the dots: Identifying network structure via graph signal processing. IEEE Signal Processing Magazine , 36(3):16-43, 2019.

Madeline Navarro, Yuhao Wang, Antonio G Marques, Caroline Uhler, and Santiago Segarra. Joint inference of multiple graphs from matrix polynomials. Journal of Machine Learning Research , 23(76):1-35, 2022.

Bastien Pasdeloup, Vincent Gripon, Grégoire Mercier, Dominique Pastor, and Michael G Rabbat. Characterization and inference of graph diffusion processes from observations of stationary signals. IEEE transactions on Signal and Information Processing over Networks , 4(3):481-496, 2017.

Nathanaël Perraudin and Pierre Vandergheynst. Stationary signal processing on graphs. IEEE Transactions on Signal Processing , 65(13):3462-3477, 2017.

Raksha Ramakrishna, Hoi-To Wai, and Anna Scaglione. A user guide to low-pass graph signal processing and its applications: Tools and applications. IEEE Signal Processing Magazine , 37(6):74-85, 2020.

Samuel Rey, Andrei Buciulea, Madeline Navarro, Santiago Segarra, and Antonio G Marques. Joint inference of multiple graphs with hidden variables from stationary graph signals. In Proceedings of 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022) , pages 5817-5821. IEEE, 2022.

R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis , volume 317. Springer Science &Business Media, 2009.

Johannes O Royset and Roger JB Wets. An Optimization Primer . Springer, 2021.

Santiago Segarra, Sundeep Prabhakar Chepuri, Antonio G Marques, and Geert Leus. Statistical graph signal processing: Stationarity and spectral estimation. Cooperative and Graph Signal Processing , pages 325-347, 2018.

Santiago Segarra, Antonio G Marques, Gonzalo Mateos, and Alejandro Ribeiro. Network topology inference from spectral templates. IEEE Transactions on Signal and Information Processing over Networks , 3(3):467-483, 2017.

Santiago Segarra, Yuhao Wang, Caroline Uhler, and Antonio G Marques. Joint inference of networks from stationary graph signals. In Proceedings of the 51st Asilomar Conference on Signals, Systems, and Computers , pages 975-979. IEEE, 2017.

Rasoul Shafipour, Abolfazl Hashemi, Gonzalo Mateos, and Haris Vikalo. Online topology inference from streaming stationary graph signals. In 2019 IEEE Data Science Workshop (DSW) , pages 140-144. IEEE, 2019.

Rasoul Shafipour and Gonzalo Mateos. Online topology inference from streaming stationary graph signals with partial connectivity information. Algorithms , 13(9):228, 2020.

Rasoul Shafipour, Santiago Segarra, Antonio G Marques, and Gonzalo Mateos. Network topology inference from non-stationary graph signals. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5870-5874. IEEE, 2017.

Stephen M Smith, Karla L Miller, Gholamreza Salimi-Khorshidi, Matthew Webster, Christian F Beckmann, Thomas E Nichols, Joseph D Ramsey, and Mark W Woolrich. Network modelling methods for fmri. Neuroimage , 54(2):875-891, 2011.

Oliver Stegle, Sarah A Teichmann, and John C Marioni. Computational and analytical challenges in single-cell transcriptomics. Nature Reviews Genetics , 16(3):133-145, 2015.

Yuichi Tanaka and Yonina C Eldar. Generalized sampling on graphs with subspace and smoothness priors. IEEE Transactions on Signal Processing , 68:2272-2286, 2020.

Yuichi Tanaka, Yonina C Eldar, Antonio Ortega, and Gene Cheung. Sampling signals on graphs: From theory to applications. IEEE Signal Processing Magazine , 37(6):14-30, 2020.

Roman Vershynin. How close is the sample covariance matrix to the actual covariance matrix? Journal of Theoretical Probability , 25(3):655-686, 2012.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge University Press, 2018.

Xiaolu Wang, Chaorui Yao, Haoyu Lei, and Anthony Man-Cho So. An efficient alternating direction method for graph learning from smooth signals. In Proceedings of 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2021) , pages 5380-5384. IEEE, 2021.

Chenyue Zhang, Yiran He, and Hoi-To Wai. Product graph learning from multi-attribute graph signals with inter-layer coupling. In Proceedings of 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023) , pages 1-5. IEEE, 2023.

Hui Zhang, Ming Yan, and Wotao Yin. One condition for solution uniqueness and robustness of both l1-synthesis and l1-analysis minimizations. Advances in Computational Mathematics , 42(6):1381-1399, 2016.

Xiaoqun Zhang, Martin Burger, and Stanley Osher. A unified primal-dual algorithm framework based on bregman iteration. Journal of Scientific Computing , 46(1):20-46, 2011.

Licheng Zhao, Yiwei Wang, Sandeep Kumar, and Daniel P Palomar. Optimization algorithms for graph laplacian estimation via admm and mm. IEEE Transactions on Signal Processing , 67(16):4231-4244, 2019.

Shuai Zheng, Zhenfeng Zhu, Zhizhe Liu, Zhenyu Guo, Yang Liu, Yuchen Yang, and Yao Zhao. Multi-modal graph learning for disease prediction. IEEE Transactions on Medical Imaging , 41(9):2207-2216, 2022.

A Organization of the Appendix

The appendix includes the missing proofs, detailed discussions of some argument in the main body and more numerical experiments. We organize the appendix as follows:

The proof of infeasibility condition (Theorem 3.2) is provided in Section B.

Explanations on conditions derived in Theorem 3.2 are included in Section C.

The proof of properties of the proposed model (r)LogSpecT (Proposition 3.4 & 3.6) is given in Section D and some additional properties are discussed.

The truncated Hausdorff distance based proof details of Theorem 4.1 and Corollary 4.4 are given in Section E.

Details of L-ADMM and its convergence analysis are in Section F.

Additional experiments and discussions on synthetic and real data are included in Section G.

B Proof of Theorem 3.2

Since the linear system (4) has no solution, we know from Farkas' lemma that the following system has solutions:

Let x ∗ ∈ R m 2 be a solution to (11). Denote x + := max { x ∗ , 0 } , x -:= max {-x ∗ , 0 } . Then, there exists c ∈ (0 , 1] such that

Define y := -1 ⊤ m 2 ( x + + x -) , z := c 1 ⊤ m 2 ( x + + x -) and set ¯ δ = c . For all δ ∈ [0 , ¯ δ ) , ( x + , x -, y, z ) is a solution to the following linear system:

Again, from Farkas' lemma, this implies that the following linear system does not have a solution:

where s ∈ R m ( m -1) / 2 and t ∈ R . Since (12) is equivalent to:

the above argument indicates that (13) does not have a solution. Suppose rSpecT has a feasible solution S ′ , then

Hence, S ′ is also a solution to (13). However, (13) does not have a solution. We can conclude that rSpecT is infeasible in this case.

C Explanations on Sufficient Conditions in Theorem 3.2

We elaborate more on the infeasibility condition that A n B has full column rank. An application of the condition is Example 3.1. Specifically, we know that in this case,

This implies that

̸

Hence, when h 11 = h 22 , A n B has full column rank. This means that when δ is small enough (from Example 3.1 we know ˜ δ = √ 2 | h 11 -h 22 | ), the model rSpecT is infeasible.

D Proofs of Properties of (r)LogSpecT

D.1 Proof of Proposition 3.4

Since the constraint set S is a cone, it follows that for all γ > 0 , γ S = S . Then, we know that

where the third equality is from the basic calculus rule of the logarithm function. Set γ = α and then Opt( C , α ) = α Opt( C , 1) , which completes the proof.

D.2 Proof of Proposition 3.6

The proof will be conducted by constructing a feasible solution for rLogSpecT. Recall that A n = I ⊗ C n -C n ⊗ I and the matrix B ∈ R m 2 × m ( m -1) / 2 that maps a non-negative vector to the vectorization of a valid adjacency matrix. Let S = min { δ ∥ A n Bs ∥ 2 , 1 }· mat ( Bs ) with s ∈ R ( m -1) m/ 2 being a non-negative vector, where mat ( · ) is the matricization operator. Note that

Then, we know that

Thus, the given S is a feasible solution for rLogSpecT and it completes the proof.

D.3 Properties of optimal solutions and values of (r)LogSpecT

In this section, we further discuss some properties of the optimal solutions/value of the proposed models, which are useful for deriving the recovery guarantee. More specifically, we obtain an upper bound on the optimal solutions (which may not be unique) independent of the sample size n and the inaccuracy parameter δ n . Also, a lower bound of optimal values follows.

Proposition D.1. The following statements hold:

For an optimal solution S ∗ (resp. S ∗ n ) to LogSpecT (resp. rLogSpecT with any given sample size n ), it follows that

If δ n ≥ 2 αm ∥ C n -C ∞ ∥ , then

where f ∗ (resp. f ∗ n ) denotes the optimal value of LogSpecT (resp. rLogSpecT).

For the first statement, let us consider the Karush-Kuhn-Tucker (KKT) conditions of LogSpecT and rLogSpecT. Since the LogSpecT is a convex problem and Slater's condition holds, the KKT conditions are necessary and sufficient for the optimality, i.e., there exists ( Λ 1 , Λ 2 ) ∈ R m × m ×N S ( S ∗ ) such that

where N S ( S ∗ ) := { N ∈ R m × m : sup X ∈S ⟨ X -S ∗ , N ⟩ ≤ 0 } is the normal cone of S at S ∗ , and ∇∥ S ∗ ∥ 1 , 1 is well-defined since ∥ · ∥ 1 , 1 = ⟨· , 11 ⊤ ⟩ at S ∗ ≥ 0 , which is differentiable. Taking further calculation gives that

Combining this with (14) by taking inner product of both sides with S ∗ , we obtain that

From the structure of S and the fact that Λ 2 ∈ N S ( S ∗ ) , one has that ⟨ Λ 2 , S ∗ ⟩ = 0 . Also, note that C ∞ S ∗ = S ∗ C ∞ . Hence, the equation (15) can be simplified as the desired result:

The KKT conditions of rLogSpecT indicate that there exist λ 1 ≥ 0 , Λ 2 ∈ N S ( S ∗ n ) and Q ∈ ∂ ∥ C n S ∗ n -S ∗ n C n ∥ F (i.e., the subgradient of the function S ↦→∥ C n S -SC n ∥ F at S ∗ n ) such that

Moreover, from the definition of the convex subdifferential we know that 0 ≥ ∥ C n S ∗ n -S ∗ n C n ∥ F -⟨ Q , S ∗ n ⟩ . Thus, after taking inner product of both sides of the equation (16) with S ∗ n , it follows that:

which implies that ∑ i,j ( S ∗ n ) ij ≤ αm -λ 1 δ n ≤ αm . This completes the proof of the first statement.

For the second statement, we first prove that v ∗ n and v ∗ are larger than αm (1 -log α ) . Define the auxiliary function g : R → R such that g ( x ) := x -α log x for any x ∈ R + , whose minimum is attained at α . Since for any S ∈ S ,

where f is the objective in LogSpecT, it follows that

This implies that v ∗ n and v ∗ are larger than αm (1 -log α ) . Next, we will show v ∗ n ≤ v ∗ . Consider any optimal solution S ∗ to LogSpecT. We show that it is feasible for rLogSpecT.

where the equality comes from C ∞ S ∗ = S ∗ C ∞ , the first inequality comes from the fact that ∥ XY ∥ ≤ ∥ X ∥ F ∥ Y ∥ ≤ ∥ X ∥ 1 , 1 ∥ Y ∥ , the second one comes from the first statement and the last one is due to δ n ≥ 2 αm ∥ C n -C ∞ ∥ . Hence, S ∗ is feasible for rLogSpecT, which indicates that v ∗ n ≤ v ∗ . The proof is completed.

E Proof of Theorem 4.1 & Corollary 4.4

E.1 Truncated Hausdorff distance

In this section, we introduce an advanced technique in optimization that is efficient in analyzing the recovery guarantee of robust formulations. Before that, we introduce the concept of truncated Hausdorff distance between two sets.

Definition E.1 (Truncated Hausdorff Distance [29, 6.J]) . For any ρ ≥ 0 , the truncated Hausdorff distance between two sets C and D is defined as:

It turns out that the distance between the optimum of two minimization problems can be bounded with the truncated Hausdorff distance of the epigraphs under some conditions. The result is captured in the following lemma.

Lemma E.2 ([29, Theorem 6.56]) . Let ρ ∈ [0 , ∞ ) . Suppose that the extended-real-valued lower semicontinuous functions f, g : R n → R satisfy

inf f, inf g ∈ [ -ρ, ρ ] ,

argmin f, argmin g ⊆ B ( 0 , ρ ) .

Then, it follows that

Suppose further that ε ≥ 2 ˆ d ρ (epi f, epi g ) , then one has

where ε -argmin f is the ε -suboptimal solution set of f that is defined as ε -argmin f := { x ∈ dom f : f ( x ) ≤ inf f + ε } , and x ∗ g is a minimizer of g .

From the above lemma, we know that if two optimization problems are close enough (in the sense of truncated Hausdorff distance), then the optimum of them should be close to each other. Hence, in order to apply this result, we need to bound the truncated Hausdorff distance in an explicit way, which is solved by the following Kenmochi condition.

Lemma E.3 (Kenmochi Condition [29, Proposition 6.58]) . Let ρ ∈ [0 , ∞ ) . Then, for f, g : R n → R with nonempty epigraphs, one has that

where [ f ≤ ρ ] := { x ∈ R n : f ( x ) ≤ ρ } .

E.2 Proof of Theorem 4.1

Before presenting the proof, we first introduce the following lemma.

Lemma E.4 (Hoffman's Error Bound [12]) . Consider the set S := { x ∈ R n : Ax ≤ b } . There exists C > 0 such that for any x ∈ R n , one has

For the sake of brevity, we denote

Hence, the optimization problem LogSpecT (resp. rLogSpecT) is equivalent to inf ¯ f (resp. inf ¯ f n ).

Now, we aim to use Lemma E.3 to bound ˆ d ρ (epi ¯ f, epi ¯ f n ) . Let S ∈ S ∩ B ( 0 , ρ ) satisfy

Then, we know that

and consequently S is in the domain of ¯ f n . Then, it follows that for any η > 0 , we have

Before verifying the reverse side of the Kenmochi condition, we first consider the non-emptiness of [ ¯ f n ≤ ρ ] ∩ B ( 0 , ρ ) . Since

it follows from Proposition D.1 that ∥ S ∗ n ∥ 1 , 1 ≤ αm ≤ ρ and f ∗ n ≤ f ∗ ≤ ρ , which implies that [ ¯ f n ≤ ρ ] ∩ B ( 0 , ρ ) is nonempty. Let S n ∈ [ ¯ f n ≤ ρ ] ∩ B ( 0 , ρ ) . Then, one has that

Hence, it follows that

Also, note that there exists β > 0 such that ( S n 1 ) i ≥ β for all i ∈ [ m ] as ¯ f n ≤ ρ and ∥ S n ∥ 1 , 1 -α 1 ⊤ log( S n 1 ) →∞ when S n → 0 . Thus, applying Lemma E.4 to the linear system

yields that there exists ˜ c > 0 such that

Hence, there exists ˜ S in the domain of ¯ f such that

Since the function S ↦→∥ S ∥ 1 , 1 -α 1 ⊤ log( S 1 ) is locally Lipschitz continuous when ( S 1 ) i ≥ β , there exists L > 0 such that

Setting c 1 ≥ max { 1 , L } · ˜ c , one can obtain that for any S n ∈ [ ¯ f n ≤ ρ ] ∩ B ( 0 , ρ )

where η := c 1 · (2 ρ ∥ C ∞ -C n ∥ + δ n ) . Combining inequality (19) and (20), we can conclude that

In order to derive the conclusion (i) and (ii), it remains to check the requirements in Lemma E.2. Since ρ ≥ αm , the first statement of Proposition D.1 shows that the optimal solutions to inf ¯ f and inf ¯ f n lie in B ( 0 , ρ ) . Since ρ ≥ f ∗ and -ρ ≤ αm (1 -log α ) , the second statement of the proposition shows that inf ¯ f, inf ¯ f n ∈ [ -ρ, ρ ] . Hence, applying Lemma E.2 completes the proof of the first two statements.

To prove conclusion (iii), we first make the following two claims:

(a) S ∗ 0 1 is a singleton, whose element is denoted by S ∗ 1 ,

(b) For any ¯ ε ∈ [0 , ∞ ) , there exists a δ (¯ ε ) > 0 such that for all 0 ≤ ε ≤ ¯ ε and S ε ∈ S ∗ ε , one has that

Granting these and with the help of Theorem 4.1 (ii), we can derive that for all S ∗ n ∈ S n, ∗

where ˜ c 1 , ˜ c 2 are positive constants, and S 2 ε n ∈ S ∗ 2 ε n satisfies ∥ S ∗ n -S 2 ε n ∥ F = dist( S ∗ n , S ∗ 2 ε n ) (whose existence is guaranteed since S ∗ ε is convex and compact). Hence,

To proceed, it remains to prove the claims. Define an auxiliary function h : R m → R as h ( x ) = ∑ m i =1 x i -α ∑ m i =1 log x i for each x ∈ R m + . Consider the following optimization problem:

For the sake of brevity, denote the ε -suboptimal solution set of (23) as H ∗ ε . In the remaining part, we will first show that S ∗ ε 1 = H ∗ ε and then, by the strict convexity of h , the desired two claims hold.

The first step is to show that the optimal function value of the problem (23) satisfies h ∗ = f ∗ . Since it is obvious that ˜ x = S ∗ 1 is feasible for (23), h ∗ ≤ h (˜ x ) = f ( S ∗ ) = f ∗ . Suppose to the contrary that h ∗ < f ∗ , from the fact that the objective function is coercive and continuous and the feasible set is closed, there exists ˜ S such that it is feasible for LogSpecT and x ∗ = ˜ S 1 , where x ∗ is an optimal solution to (23). Since h ∗ = h ( x ∗ ) = h ( ˜ S 1 ) = f ( ˜ S ) , this contradicts the fact that f ( ˜ S ) ≥ f ∗ . Hence, h ∗ = f ∗ . Next, we will show that S ∗ ε 1 = H ∗ ε . Consider any ε -suboptimal solution S ∈ S ∗ ε , i.e.,

Hence, S 1 ∈ H ∗ ε and it implies that S ∗ ε 1 ⊆ H ∗ ε . On the other hand, for any ε -suboptimal solution x ∈ H ∗ ε , there exists S that is feasible for LogSpecT such that x = S 1 . Thus,

This implies that S ∈ S ∗ ε and consequently H ∗ ε ⊆ S ∗ ε 1 . Hence, H ∗ ε = S ∗ ε 1 .

Since h is strictly convex, its optimal solution set H ∗ 0 is a singleton. Then, S ∗ 0 1 = H ∗ 0 is a singleton, which proves the first claim. For the second claim, we know that for any S ε ∈ S ∗ ε there exists x ε ∈ H ∗ ε such that

where x ∗ ∈ H ∗ 0 . The coerciveness of h asserts that x ε and x ∗ are bounded. This together with the fact that h is strongly convex on any bounded set, illustrates that there exists µ > 0 such that

where the second inequality comes from the global optimality of x ∗ . Combining (24) and (25) gives that

This completes the proof of the claims.

E.3 Proof of Corollary 4.4

Suppose to the contrary that there exists a sequence { S ∗ n } n , where the n th element is an optimal solution to rLogSpecT with sample size n , such that

From Proposition D.1, we know that { S ∗ n } n is bounded, and consequently, has a convergent subsequence. Without loss of generality, we may assume that the sequence itself is convergent and the limiting point is S ∗ . Note that

Hence, C ∞ S ∗ = S ∗ C ∞ . This indicates that S ∗ is feasible for LogSpecT. Then, from Theorem 4.1, we know that f ( S ∗ n ) = f ∗ n → f ∗ , which leads to f ( S ∗ ) = f ∗ since f ( · ) = ∥ · ∥ 1 , 1 -α 1 ⊤ log( · 1 ) is continuous. Together with the fact that S ∗ is feasible, we conclude that S ∗ is an optimal solution to LogSpecT. This further implies that dist( S ∗ n , S ∗ 0 ) → 0 , which is a contradiction.

E.4 Proof of Lemma 4.7

Recall the generative model (1). Since w follows a sub-Gaussian distribution, it can be shown that for every t > 0 ,

for some positive constant v ′ , which means that x also follows a sub-Gaussian distribution. Thus, due to the sub-Gaussian property, ∥ C n -C ∞ ∥ can be explictly bounded by the following lemma.

Lemma E.5 ([40, Proposition 2.1]) . Consider sub-Gaussian, identical, independent random vectors x 1 , x 2 , . . . , x n ∈ R m with n > m . Then for all ε > 0 , it follows that

for some constant l > 0 .

Setting ε 2 = (4 /l ) log(2 n ) m/n , Lemma E.5 indicates that with high probability (lower bounded by 1 -n -1 ),

F Derivations of L-ADMM and Convergence Analysis

This section includes the details of L-ADMM for rLogSpecT.

F.1 Proof of Proposition 5.1

Note that the minimization problem (7) is separable for Z and q , and can be split into two subproblems:

For problem (26), the optimal solution is the projection of C n S ( k ) -S ( k ) C n + Λ ( k ) /ρ onto B ( 0 , δ n ) , which is given by

For problem (27), the first-order optimality condition gives

This together with the fact that the objective function is convex implies that

F.2 Calculation of Π S ( · )

The projection of X to S can be calculated via an optimization problem:

̸

which is equivalent to

Hence

F.3 Stopping criterion and updating rule of ρ

We follow the procedures in [3] to update ρ in each iteration. Similarly, we define the primal residual and dual residual as follows:

The aim of updating ρ is to control the decaying speed of p res and d res such that their difference is not too large. To this end, we update ρ adaptively following the scheme:

When p res and d res are both smaller than the threshold ε = 10 -5 , we stop the algorithm.

F.4 Convergence analysis

Define D := Diag( 1 ⊤ m , . . . , 1 ⊤ m ) ∈ R m × m 2 . Then, D satisfies D vec ( S ) = S 1 and ∥ D ⊤ D ∥ = m . Denote

Then the linearized ADMM update (8) of S can be written as:

where ∥ x ∥ Q := x ⊤ Qx . Since τ > m + ∥ A n ∥ 2 , we know that Q is positively definite. Consequently, by treating ( Z , q ) as one variable, we can apply Theorem 4.2 in [45] and directly obtain the result.

G More Experiments and Discussions on Synthetic and Real Data

G.1 Influence of graph filters

To make a fair comparison between rSpecT and rLogSpecT, we test rSpecT on BA graphs with the same graph filters and the results are reported in Figure 6. It is obvious that rSpecT fails in these cases and cannot benefit from the increase in sample size. This is reasonable since SpecT fails on BA graphs as indicated in Figure 3, let alone the approximation formulation rSpecT.

rLogSpecT Performance

10 1

10 2

10 3

10 4

10 5

10 6

Sample Size

0.1

0.2

0.3

0.4

0.5

0.6

0.7

F-measure

rSpecT Performance

lowpass-EXP

highpass-EXP

QUA

10 1

10 2

10 3

10 4

10 5

10 6

Sample Size

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

F-measure

lowpass-EXP

highpass-EXP

QUA

We further test rLogSpecT on ER graphs with different numbers of signals observed. The parameter δ n is set as 20 √ log n/n and the results are reported in Figure 7. The figure shows that for graph filters that are not high-pass, rLogSpecT can achieve nearly perfect recovery when the sample size is large enough. Also, compared with the performance on BA graphs, rLogSpecT works better on ER graphs. This observation is in accordance with the conclusion from Figure 3 that LogSpecT performs better on ER graphs than BA ones. We further notice that the difference between the low-pass graph filter and the high-pass one is huge. To check the conjecture that rLogSpecT generally performs better on low-pass graph filters, we choose different graph filters exp( t S ) with t ranging from -2 to 2 and conduct the experiments on ER graphs. When the graph shifting operator is the adjacency matrix, the positive low-pass parameter t corresponds to low-pass graph filters and the negative t corresponds to the high-pass ones [26, 11]. We omit the case when t = 0 since this filter does not contain any graph information (note that exp(0 S ) = I ).

Wethen repeat the experiments for 50 times and report the average results in Figure 8. The comparison between the performance of low-pass graph filters and high-pass graph filters indicates that the lowpass graph filters generally outperforms the high-pass ones. A closer look at the results shows that the performance grows faster when the absolute value of t is smaller. And eventually, the graph filter with smaller absolute value of t prevails. This observation is interesting since Figure 3 indicates that the choice of graph filters has few impacts on the model performance. One explanation is that both low-pass graph filters and high-pass graph filters attenuate some frequencies of the graph and the larger absolute value of t leads to the more loss of information carried by finite signals.

10 1

10 2

10 3

10 4

10 5

10 6

Sample Size

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Average F-measure

low-pass parameter = -2

low-pass parameter = -1.5

low-pass parameter = -1

low-pass parameter = -0.5

low-pass parameter = 0.5

low-pass parameter = 1

low-pass parameter = 1.5

low-pass parameter = 2

G.2 Experiments on USPS dataset

USPS dataset is a handwritten digit dataset. As shown in [25], it is nearly stationary with respect to the nearest neighbour graph. This dataset collects images of handwritten digits. In the experiment, each pixel is viewed as a node and the value on the pixel is view as the graph signal. We follow the approach in [25] to construct the 20 nearest neighbour graph, on which the data is verified to be nearly stationary. More specifically, we pick the 1296 digit 1 images. The weights between two nodes are decided by the Gaussian radial basis function. It turns out that the stationarity measure s := ∥ diag( U ⊤ C n U ) ∥ 2 / ∥ U ⊤ C n U ∥ F equals 0.78 in this dataset. Here U is the eigen-matrix of the constructed graph and C n is the covariance matrix of the observed graph signals. We view the 20 nearest neighbour graph as the groundtruth and compare the learned graph from rLogSpecT with it. We present the subgraphs consisting of the top 10 nodes with the largest degrees in the two graphs respectively in Figure 9. The F-measure of these two subgraphs is 0.96. This result corroborates the efficiency of our proposed rLogSpecT model.

-1.00

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

-1.00

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

GroundTruth graph,10 nodes,42 edges

-1.00

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

-1.00

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

LearnedGraph,10 nodes, 45 edges

Peer reviews:
comments: SUMMARY:
This paper studies the problem of graph learning from stationary signals. The authors propose an algorithmic framework, LogSpecT, which addresses the shortcomings of an existing body of work that relies on graph learning using spectral templates.

STRENGTHS:
1. The problem setting is graph learning for stationary graphs using spectral templates. This is a problem of interest in the graph signal processing domain and is motivated by practical applications. The major technical contribution of the paper is to solve the problem by introducing log barrier in the objective function. Introduction of log barrier facilitates tackling the graph learning scenarios that were not feasible by existing algorithms in this domain.
2. The paper is well written and the technical content is easy to follow.
3. The experiments demonstrate the LogSpecT ourtperforms the previous versions of spectral templates and two other baselines from the literature in the task of graph learning. Moreover, it is demonstrated that previous versions of graph learning algorithms may encounter infeasibility scenarios on real datasets with high likelihood.

WEAKNESSES:
There may be a minor concern that the technical contributions of the paper are narrowly focused on overcoming the shortcomings of graph learning with spectral templates.

QUESTIONS:
Could you comment on the computational complexity of LogSpecT? In particular, I am curious whether the operations involving $C_n$ in Section 5 could be computationally prohibitive for high dimensional datasets.

LIMITATIONS:
Limitations have been adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
In this paper, the authors considered the problem of learning graphs from stationary signals. In order to overcome the infeasibility issue of an existing method called rSpecT, the authors proposed a novel formulation by introducing the log barrier term to learn graphs without isolated nodes. The feasibility can be guaranteed by the new formulation. Furthermore, the recovery guarantees of the new formulation was established, and an efficient algorithm based on linearized ADMM was proposed to solve the formulation.

STRENGTHS:
This paper is very interesting, with novel ideas to solve an issue of a well-known method in graph signal processing. The authors proposed a novel formulation to overcome the infeasibility issue of rSpecT and established the theoretical recovery guarantees. Furthermore, an efficient algorithm based on linearized ADMM was designed to solve the formulation. Numerical results demonstrated the stability and superiority of the new method.

WEAKNESSES:
1. The descriptions of the experiments conducted on real networks require further improvement. Specifically, please provide more details regarding the number of observations, the nature of these observations, and what the edges represent within the networks.

2. As observed in Figures 2 and 4, the F-measure does not increase to 1, even when given a sufficiently large number of observations. The authors should provide commentary on this phenomenon to address any potential concerns.

QUESTIONS:
Please answer questions from Weakness.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of learning a graph from stationary graph signals. To this end, the authors introduce two new optimisation problems called LogSpecT and rLogSpecT.  They prove some recovery guarantees on the optimal value of (r)LogSpecT and give a convergence rate when the signals are sub-Gaussian. They also provide an efficient solver based on L-ADMM that admits closed solutions at each substep.  Finally, some empirical results highlight the performance of their methods.\n\nSTRENGTHS:\n1\\ The paper is very well written and easy to follow\n\n2\\ The problem is well-motivated and the solution seems theoretically sound\n\n3\\ The paper is solid and almost all aspects encountered in the graph learning problem are covered (optimization, recovery guarantees, convergence,...)\n\nWEAKNESSES:\n1\\ There are no error bars. They should be added to really show the results of each method and to be able to compare them.\n\n2\\ A table with all the measurements reported could be a good thing. For example, in all the experiments, only the F measure is given, but precision and recall are also important for assessing the quality of the learned graphs.\n\n3\\ An illustration comparing the real graph with the learned graph (for example by plotting the adjacency matrix) could be interesting.\n\n4\\ The complexity and the time until convergence are not given for the methods. An additional experiment on this aspect should be added.\n\n=======\nRemark\n\nthe step size tau is never defined\n\nQUESTIONS:\n1\\ What is the intuition behind the need for a delta greater than a certain value in Theorem 4.1?\n\n2\\ In the GSP literature, there are many more recent methods than the one of Kalofolias for learning the graph. Why choose to compare your methods only to the one of Kalofolias?\n\n3\\ Furthermore, why exclude this method from the comparison in the results of the section 6.3?\n\n4\\ Experiments are made with a maximum of 60 nodes. What is the maximum number of nodes that can be learned in a reasonable time? if possible, can we have some examples?\n\n=====\n\nMore open questions:\n\n5\\ Is it possible to link the optimization problem to a maximum likelihood estimate?\n\n6\\ It seems that the filter h only has an impact on the covariance estimate. Is it true? If the answer is \"no\", would it be a good idea (if possible) to include information about the h filter in the optimization problem?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors improve upon an existing method, rSpecT, to impute graphs from stationary signals. The latter frames the recovery of the (weighted) adjacency matrix as a convex optimization problem with constraints —- namely the commutativity of the covariance matrix and the adjacency matrix, as well as the first row sum to be equal to 1, in order to avoid the trivial 0 solution. Here, the authors show that this yields infeasible solutions in a number of cases. Consequently, the authors propose to replace the row sum constraint by a log barrier. They show that the solution is always feasible (however, it ceases to be unique).  They then show through a set of experiments that their solution outperforms the state of the art approaches in recovering the underlying graph.\n\nSTRENGTHS:\nFirst, a disclaimer: although familiar with GSP, I am not an expert in the graph recovery literature. My comments must thus be taken sparingly.\n\nOverall, the paper is clearly written and motivated.\nThe discussion on the infeasibility of the current state of the art method, rSpecT, was interesting. The solution that the authors bring is therefore well motivated.\n\nWEAKNESSES:\nAs highlighted before, I am not an expert in the field and I am not able to put this paper in context of the general literature on the topic.\n\nA few questions however:\n- the authors claim that their method is more computationally efficient with the use of ADMM. Do they have plots showing the running time? ADMM, although fast at each iteration, can be slow to converge.\n- the experiments could have been expanded: in particular, the authors only present ER and BA graphs, but they could have varied the parameters to try and assess whether their method is robust across all sorts of topologies.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe method could have been more extensively evaluated.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
This is an emergency review, regrettably, the paper is outside of my expertise.

Review:
This is a theoretical work concerned with graph learning from signals. The work identifies an issue with the feasibility of a previous well known model in the field, SpecT, then introduces a novel model, LogSpecT, for which this issue does not occur.

While I regrettably could not comprehend the main content, the topic, general presentation, and references make me believe that this paper is topically not suited for neurIPS. I concede that the CfP of neurIPS is somewhat vague and broad, but based on the common topics in the field, I believe for the conference this is very much a niche topic to which the vast majority of attendees would not be able to connect.

Superficial observations
 - There isn't any mention of the word "neural"
 - From 45 references, the vast majority is in signal processing. A single one is at neurIPS
 - The intro is exceedingly dry to read, with no motivating example, nor any illustration. Especially, what are examples of signals considered for graph learning? What would the stationarity assumption mean in these examples?

I understand that reading a superficial review like this may be frustrating to the authors, but can do no better. I would much suggest the authors consider a specialist forum, where this type of content might be better understood and appreciated.

STRENGTHS:
see above

WEAKNESSES:
see above

QUESTIONS:
see above

LIMITATIONS:
see above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-12