SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 4.8 together with Theorem 4.1 illustrates the non-asymptotic convergence of optimal function value/suboptimal solution set/optimal node degree vector of rLogSpect from  $n$  i.i.d samples to the ideal model LogSpect with the convergence rate  $\mathcal{O}(\varepsilon_n) / \mathcal{O}(\varepsilon_n) / \mathcal{O}(\sqrt{\varepsilon_n})$ , where  $\varepsilon_{n} = \mathcal{O}(|C_{n} - C_{\infty}|)\leq \mathcal{O}(\sqrt{\log n / n})$  for sub-Gaussian  $\omega$ . This matches the convergence rate  $\mathcal{O}(\sqrt{\log n / n})$  of classic spectral template models, e.g., Proposition 2 in [31] and Theorem 2 in [23] for the SpecT model, which shows that LogSpecT is also competitive on recovery rate.
5 Linearized ADMM for Solving rLogSpecT
In this section, we design a linearized ADMM algorithm to solve rLogSpecT that admits closed-form solutions for the subproblems. The ADMM-type algorithms have been successfully applied to tackle various graph learning tasks [46, 42].
Firstly, we reformulate rLogSpecT such that it fits the ADMM scheme:
$$
\min  _ {\boldsymbol {S}, \boldsymbol {Z}, \boldsymbol {q}} \left\langle \boldsymbol {S}, \mathbf {1} \boldsymbol {1} ^ {\top} \right\rangle - \alpha \mathbf {1} ^ {\top} \log \boldsymbol {q} + \iota_ {\mathcal {S}} (\boldsymbol {S}) + \iota_ {\mathbb {B} (\boldsymbol {0}, \delta_ {n})} (\boldsymbol {Z}) \tag {6}
$$
$$
\begin{array}{l} \text {s . t .} \quad C _ {n} S - S C _ {n} = Z, q = S 1. \end{array}
$$
The augmented Lagrangian function of problem (6) is
$$
\begin{array}{l} L (\boldsymbol {S}, \boldsymbol {Z}, \boldsymbol {q}, \boldsymbol {\Lambda}, \boldsymbol {\lambda} _ {2}) = \left\langle \mathbf {1 1} ^ {\top}, \boldsymbol {S} \right\rangle - \alpha \mathbf {1} ^ {\top} \log \boldsymbol {q} + \iota_ {\mathcal {S}} (\boldsymbol {S}) + \iota_ {\mathbb {B} (\boldsymbol {0}, \delta_ {n})} (\boldsymbol {Z}) + \left\langle \boldsymbol {\Lambda}, \boldsymbol {C} _ {n} \boldsymbol {S} - \boldsymbol {S C} _ {n} - \boldsymbol {Z} \right\rangle \\ + \boldsymbol {\lambda} _ {2} ^ {\top} (\boldsymbol {q} - \boldsymbol {S 1}) + \frac {\rho}{2} \| \boldsymbol {C} _ {n} \boldsymbol {S} - \boldsymbol {S C} _ {n} - \boldsymbol {Z} \| _ {F} ^ {2} + \frac {\rho}{2} \| \boldsymbol {q} - \boldsymbol {S 1} \| _ {2} ^ {2}. \\ \end{array}
$$
Update of primal variables  $Z$  and  $q$  Since  $Z$  and  $q$  are separable given  $S$ , we update the primal variables in two blocks:  $S$  and  $(Z, q)$ . More specifically, the update of  $(Z, q)$  in the  $k$ -th iteration is
$$
\left(\boldsymbol {Z} ^ {(k + 1)}, \boldsymbol {q} ^ {(k + 1)}\right) := \operatorname {a r g m i n} _ {\left(\boldsymbol {Z}, \boldsymbol {q}\right)} L \left(\boldsymbol {S} ^ {(k)}, \boldsymbol {Z}, \boldsymbol {q}, \boldsymbol {\Lambda} ^ {(k)}, \boldsymbol {\lambda} _ {2} ^ {(k)}\right), \tag {7}
$$
which admits the following closed-form solution:
Proposition 5.1. The update (7) can be explicitly rewritten as
$$
\boldsymbol {Z} ^ {(k + 1)} = \min  \left\{1, \frac {\delta_ {n}}{\| \tilde {\boldsymbol {Z}} \| _ {F}} \right\} \tilde {\boldsymbol {Z}}, \quad \boldsymbol {q} ^ {(k + 1)} = \frac {\tilde {\boldsymbol {q}} + \sqrt {\tilde {\boldsymbol {q}} ^ {2} + 4 \alpha / \rho \mathbf {1}}}{2},
$$
where  $\tilde{\pmb{Z}} = \pmb{C}_n\pmb{S}^{(k)} - \pmb{S}^{(k)}\pmb{C}_n + \frac{\pmb{\Lambda}^{(k)}}{\rho}$ ,  $\tilde{\pmb{q}} = \pmb{S}^{(k)}\pmb{1} - \frac{1}{\rho}\pmb{\lambda}_2^{(k)}$ .
Update of primal variables  $S$  The ADMM update of  $S$  does not admit a closed-form solution. Hence, we solve the linearized version  $\hat{L}_k$  of the augmented Lagrangian function  $L$  when all the variables except  $S$  are fixed. It is given by
$$
\hat {L} _ {k} (\boldsymbol {S}) = \left\langle \boldsymbol {C} ^ {(k)}, \boldsymbol {S} \right\rangle + \frac {\rho \tau}{2} \| \boldsymbol {S} - \boldsymbol {S} ^ {(k)} \| _ {F} ^ {2} + \iota_ {\mathcal {S}} (\boldsymbol {S}),
$$
where  $C^{(k)} \coloneqq 11^{\top} + C_{n}\Lambda^{(k)} - \Lambda^{(k)}C_{n} - \lambda_{2}^{(k)}\mathbf{1}^{\top} + \rho (S^{(k)}C_{n}^{2} + C_{n}^{2}S^{(k)} - 2C_{n}S^{(k)}C_{n} + Z^{(k+1)}C_{n} - C_{n}Z^{(k+1)} - q^{(k+1)}\mathbf{1}^{\top} + S^{(k)}\mathbf{1}\mathbf{1}^{\top})$  and  $\tau$  is the parameter for linearization. Therefore, the update of  $S$  in the  $k$ -th iteration is
$$
\boldsymbol {S} ^ {(k + 1)} := \operatorname {a r g m i n} _ {\boldsymbol {S} \in \mathcal {S}} \hat {L} _ {k} (\boldsymbol {S}) = \Pi_ {\mathcal {S}} \left(\boldsymbol {S} ^ {(k)} - \boldsymbol {C} ^ {(k)} / \rho \tau\right), \tag {8}
$$
where  $\Pi_S(X)$  is the projection of  $X$  onto the closed and convex set  $S$  and can be computed by
$$
(\Pi_ {\mathcal {S}} (\boldsymbol {X})) _ {i j} = \left\{ \begin{array}{l l} \frac {1}{2} \max  \{0, X _ {i j} + X _ {j i} \}, & \text {i f} i \neq j, \\ 0, & \text {i f} i = j. \end{array} \right.
$$
Update of dual variables and parameters Finally, we update the dual variables  $\Lambda$  and  $\lambda_{2}$  via gradient ascent steps of the augmented Lagrangian function:
$$
\boldsymbol {\Lambda} ^ {(k + 1)} := \boldsymbol {\Lambda} ^ {(k)} + \rho \left(\boldsymbol {C} _ {n} \boldsymbol {S} ^ {(k + 1)} - \boldsymbol {S} ^ {(k + 1)} \boldsymbol {C} _ {n} - \boldsymbol {Z} ^ {(k + 1)}\right), \tag {9}
$$
$$
\boldsymbol {\lambda} _ {2} ^ {(k + 1)} := \boldsymbol {\lambda} _ {2} ^ {(k)} + \rho (\boldsymbol {q} ^ {(k + 1)} - \boldsymbol {S} ^ {(k + 1)} \mathbf {1}). \tag {10}
$$
Together with the updating rule for the augmented parameter  $\rho$ , we complete the whole algorithm. The details are presented in Appendix F.
Complexity of each iteration To illustrate the efficiency of L-ADMM, we calculate the complexity of each subproblem. In the update of  $\mathbf{Z}$  and  $\mathbf{q}$ , the most time consuming part is to calculate  $C_n S^{(k)}$ . When the number of nodes  $m$  is larger than the sample size  $n$ , we rewrite  $C_n S^{(k)}$  as  $(1 / nX)(X^\top S^{(k)})$ , where  $X \in \mathbb{R}^{m \times n}$  is the data matrix. In this case, the complexity is in the order of  $\mathcal{O}(nm^2)$ . When  $m$  is smaller than  $n$ , we first calculate  $C_n$  and then multiply it with  $S^{(k)}$ . This gives the complexity  $\mathcal{O}(nm^2)$ . Similarly, the complexity needed to update  $S$  is  $\mathcal{O}(nm^2 + mn^2)$ . The update of the dual variables can be decomposed as the summation of components calculated when updating the primal variables. In summary, the total complexity of each iteration is  $\mathcal{O}(nm^2 + mn^2)$ .
Convergence of L-ADMM For the convergence analysis of L-ADMM, we treat  $(Z,q)$  as one variable, and the two constraints  $C_n S - SC_n = Z$  and  $q = S1$  can be written into a single one:
$$
[ \boldsymbol {I} \otimes \boldsymbol {C} _ {n} - \boldsymbol {C} _ {n} \otimes \boldsymbol {I}; \boldsymbol {I} \otimes \mathbf {1} ^ {\top} ] \operatorname {v e c} (\boldsymbol {S}) = [ \operatorname {v e c} (\boldsymbol {Z}); \boldsymbol {q} ].
$$
Then we can apply a two-block proximal ADMM (i.e., L-ADMM) to the problem by alternatingly updating  $S$  (see details in Appendix F.4 for this part) and  $(Z, q)$ . Consequently, the convergence result in [45, Theorem 4.2] can be invoked directly to derive the following theorem.
Theorem 5.2. If  $\tau > m + \|C_n \otimes I_m - I_m \otimes C_n\|^2$ , then  $\lim_{k \to \infty} f(S^{(k)}) = f_n^*$ .
In this section, we evaluate our proposed algorithm and models via numerical experiments. The experiments are conducted on both synthetic and real networks. We apply the standard metrics [18]: F-measure, Precision, and Recall to assess the quality of learned graphs. The source code is available at: https://github.com/StevenSYL/NeurIPS2023-LogSpecT.
6.1 Data Generation
Random Graphs. In the experiments, we consider two types of synthetic graphs, namely, the Erdős-Rényi (ER) graph [9] and the Barabasi-Albert model graph (BA) [2]. The ER graphs are generated by placing an edge between each pair of nodes independently with probability  $p = 0.2$  and the weight on each edge is set to 1. The BA graphs are generated by having two connected nodes initially and then adding new nodes one at a time, where each new node is connected to exactly one previous node that is randomly chosen with a probability proportional to its degree at the time.
Graph Filters and Signals. Three graph filters are used in the experiments. The first one is the low-pass graph filter (lowpass-EXP)  $h(\mathbf{S}) = \exp \left(\frac{1}{2}\mathbf{S}\right)$ . The second one is the high-pass graph filter (highpass-EXP)  $h(\mathbf{S}) = \exp (-\mathbf{S})$ . The last one is a quadratic graph filter (QUA)  $h(\mathbf{S}) = \mathbf{S}^2 +\mathbf{S} + \mathbf{I}$ . Note that the quadratic graph filter is neither low-pass nor high-pass. We will also use random graph filters developed from the above ones for experiments on real networks. They will be introduced when applied. The stationary graph signals are generated from the generative model (1). The input graph signal  $\boldsymbol {w}\sim \mathcal{N}(\mathbf{0},\boldsymbol{I}_m)$  is a random vector following the normal distribution.
6.2 How to Infer Binary Graphs
Many practical tasks require binary graphs instead of weighted graphs. However, the results from LogSpect and rLogSpecT are generally weighted. In this section, we tackle the issue of how to convert a weighted graph to a binary one. Firstly, we normalize each edge in the graph  $W$  by dividing it over the maximal weight, which yields a collection of values ranging from 0 to 1. Our task is then to select a threshold  $\varepsilon$  to round these values to 0 or 1 and construct a binary graph  $W^{*}$  accordingly. Mathematically, this procedure is captured by the formula:
$$
(W ^ {*}) _ {i j} = \left\{ \begin{array}{l l} 1, & \text {i f} W _ {i j} / \max  \{W _ {i ^ {\prime} j ^ {\prime}} \} \geq \varepsilon , \\ 0, & \text {i f} W _ {i j} / \max  \{W _ {i ^ {\prime} j ^ {\prime}} \} <   \varepsilon . \end{array} \right.
$$
To choose the threshold  $\varepsilon$ , we either use a training-based strategy or a searching-based strategy. The training-based one searches the best  $\varepsilon$  on  $k$  graphs and applies the value to the newly learned graph. The searching-based one simply searches for  $\varepsilon$  that yields the best performance for a graph.
6.3 Convergence of L-ADMM
We present the convergence performance of L-ADMM in this section. Since there does not literally exist customized algorithm for rLogSpecT, we compare our L-ADMM with CVXPY [8]. The solver used in CVXPY is MOSEK. We conduct the experiments on 100-node BA graphs with 10000 stationary graph signals generated from low-pass EXP graph filter. The parameter  $\delta$  is set as  $10\sqrt{\log n / n}$ . For L-ADMM algorithm, the target accuracy is set as  $10^{-6}$  and the initialization is set as zero. The running time for L-ADMM takes around 20 seconds while the solver takes over 110 seconds. The primal residual and dual residual of L-ADMM in each iteration are plot in Figure 1. The results corroborate the efficiency of our proposed algorithm. Furthermore, the locally linear convergence can be observed from the experiment. This implies that L-ADMM converges with fast speed to high accuracy. It remains an open question to prove the property theoretically.
6.4 Experiments on Synthetic Networks
To evaluate the efficacy of LogSpecT and rLogSpecT, we compare them with SpecT and rSpecT on synthetic data. Notice that the prior of no isolated nodes has been incorporated in practice by adding restrictions on the smallest degree. This approach introduces an additional hyperparameter and may exacerbate the infeasibility issue in robust formulation. Also, no recovery guarantees are developed for this trick. For these reasons, we still compare LogSpecT & rLogSpecT with SpecT & rSpecT.
Performance of LogSpecT. We first compare the performance of the ideal models: LogSpecT and SpecT on the two types of random graphs with three graph filters. We conduct experiments on graphs with nodes ranging from 20 to 60 and use the searching-based strategy to set threshold  $\varepsilon^{*}$ . After 10 repetitions, we report the average results in Figure 3. The left column presents the performance of LogSpecT and the right column presents that of SpecT. Firstly, we observe that on ER graphs, both models achieve good performance. They nearly recover the graphs perfectly. However, the performance on BA graphs differs. In this case, LogSpecT can work efficiently. The learned graphs from LogSpecT enjoy much higher F-measure values than those from SpecT. Secondly, the comparison between different graph filters shows that different graph filters have few impacts on recovery performance for both LogSpecT and SpecT. Finally, we observe that the outputs of LogSpecT on BA graphs tend to possess higher F-measure when the number of nodes increases. This suggests that LogSpecT may behave better on larger networks. However, such phenomena cannot be observed from SpecT results.
Performance of rLogSpecT. Since LogSpecT persuasively outperforms SpecT, we only present the performance of rLogSpecT when more samples are collected. As we have mentioned before, the optimal solution to LogSpecT is not necessarily unique. Thus, we do not expect to be able to show how rLogSpecT's optimal solution converges to LogSpecT's optimal solution. Moreover, the non-uniqueness may jeopardize the performance of rLogSpecT intuitively. However, the following experiment on BA graphs shows that the learned graphs from rLogSpecT with the binary-graph-transforming strategy tend to approach the ground truth when enough samples are collected.
In this experiment, the sample size  $n$  is chosen from 10 to  $10^{6}$  and  $\delta_{n}$  is set as  $0.2\sqrt{\log n / n}$ . We rely on the training-based strategy to obtain the best threshold  $\varepsilon^{*}$  from 10 randomly chosen training graphs. We then calculate the F-measure of the learned binary graph in the testing set. The result is reported in Figure 2. It shows that rLogSpecT works for all three graph filters on BA graphs and tends to possess higher F-measure values when more and more signals are collected. This indicates that similar to LogSpecT, rLogSpecT is efficient for different types of graph filters on BA graphs, including the high-pass ones. For more experiments and discussions, we refer readers to Appendix G.
6.5 Experiments on Real Networks
In this set of experiments, we compare the performance of LogSpecT (resp. rLogSpecT) with SpecT (resp. rSpecT) and other methods from the statistics and GSP communities on Protein database and Reddit database from [6]. The Protein database is a Bioinformatics dataset, where nodes are secondary structure elements (SSEs) and there is an edge between two nodes if they are neighbors in the amino-acid sequence or in 3D space. The Reddit database is a social network dataset. In this dataset, each graph corresponds to an online discussion thread, where nodes correspond to users, and there is an edge between two nodes if at least one of them responds to another's comments. We choose graphs in databases whose numbers of nodes are smaller than 50 and generate stationary graph signals from the generative model (1), with the graph filtering chosen as  $H(S) = \exp \frac{1}{2} S$ . Our experiments include in total 871 testing graphs in the Protein database and 309 in Reddit database.
Infeasibility of rSpecT. For these real networks, we first check whether the infeasibility issue encountered by rSpecT is significant. To this end, we adopt the random graph filters  $t_1S^2 + t_2S + t_3I$ , where  $t_i, i = 1, 2, 3$  are random variables following a Gaussian distribution with  $\mu = 0$  and  $\sigma = 2$ . Then, we calculate the smallest  $\delta_n$  such that rSpecT is feasible. If the smallest  $\delta_n > 0$ , rSpecT is likely to be encountered with the infeasibility issue. The results with different numbers of graph signals observed are shown in Table 1.
The experiment results present a decrease in the mean value of  $\delta_{\mathrm{min}}$  when more samples are used. However, a high frequency of the infeasibility issue occurring in these real datasets cannot be mitigated by the increasing sample size. These observations highlight the necessity of the careful treatment for choosing  $\delta$  in rSpecT again.
Performance of different graph learning models. In this section, the stationary graph signals are generated by the low-pass-EXP filter. We choose the famous statistical method called (thresholded) correlation [17] and the first GSP method that applies the log barrier to graph inference [15] as the baselines. The optimal threshold for the correlation method is selected from 0.1 to 0.6 and we search in  $\{0.01, 0.1, 1, 10, 100, 1000\}$  to obtain the best hyperparameter in Kalofolias' model. The parameter  $\delta_{n}$  in rLogSpecT is set as  $10\sqrt{\log n / n}$  and in rSpecT it is set as the smallest value that allows a feasible solution [31]. We also rely on the searching-based strategy to convert the learned weighted graphs from (r)SpecT & (r)LogSpecT.
The results of the ideal models with the true covariance matrix  $C_{\infty}$  applied are collected in Figure 4. We observe that on the real graphs, LogSpecT achieves the best performance on average (median represented by the red line). Also, compared with SpecT, LogSpecT performs more stably. We remark that since the graph signals are not necessarily smooth, Kalofolias' model cannot provide guaranteed performance, especially on the Reddit networks.
Figure 5 compares the performance of four different methods when different numbers of signals are observed. When the sample size increases, the models focusing on stationarity property can recover the graphs more accurately while correlation method and Kalofolias' method fail to exploit the signal information. This can also be inferred from the experiment results in Figure 4 since the models fail to achieve good performance from full information, let alone from partial information. The experiment also shows that when a fixed number of samples are observed, the learned graphs from rLogSpecT approximate the ground truth better than rSpecT. This further corroborates the superiority of rLogSpecT on graph learning from stationary signals.
In this paper, we directly tackle the infeasibility issue encountered in graph learning from stationary graph signals [31] and propose the first condition that guarantees the model infeasibility. To overcome this, we propose an efficient alternative. The recovery guarantees of its robust formulation are analyzed with advanced optimization tools, which may find broader applications in learning tasks. Compared with current literature [23], these theoretical results require less stringent conditions. We also design an L-ADMM algorithm that allows for efficient implementation and theoretical convergence. Extensive experiments on both synthetic and real data are conducted in this paper. The results show that our proposed model can significantly outperform the existing ones.
We believe this work represents an important step beyond the fundamental model SpecT. Its general formulation allows for the transfer of the current extensions made on SpecT. Testing our proposed models with these extensions is one future direction. Also, we notice that although the recovery guarantees for robust formulations are clear, the estimation performance analysis for the ideal-case models (i.e. SpecT and LogSpecT) is still incomplete. Investigating the exact recovery conditions is another future direction.
2The model in [34] is a substitute of rSpecT to approximate SpecT and its hyperparameter is hard to determine. Hence, we omit the performance of that model.
This work was supported in part by the Hong Kong Research Grants Council (RGC) General Research Fund (GRF) Project CUHK 14203920.
[1] Daron Acemoglu, Asuman Ozdaglar, and Alireza Tahbaz-Salehi. Systemic risk and stability in financial networks. American Economic Review, 105(2):564-608, 2015.
[2] Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science, 286(5439):509-512, 1999.
[3] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning, 3(1):1-122, 2011.
[4] Andrei Buciulea, Samuel Rey, Cristobal Cabrera, and Antonio G Marques. Network reconstruction from graph-stationary signals with hidden variables. In Proceedings of the 53rd Asilomar Conference on Signals, Systems, and Computers, pages 56-60. IEEE, 2019.
[5] Andrei Bucielea, Samuel Rey, and Antonio G Marques. Learning graphs from smooth and graph-stationary signals with hidden variables. IEEE Transactions on Signal and Information Processing over Networks, 8:273-287, 2022.
[6] Samir Chowdhury and Tom Needham. Generalized spectral clustering via gromov-wasserstein learning. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS 2021), pages 712-720. PMLR, 2021.
[7] Jose Vinicius de Miranda Cardoso, Jiaxi Ying, and Daniel Palomar. Graphical models in heavy-tailed markets. Advances in Neural Information Processing Systems, 34:19989-20001, 2021.
[8] Steven Diamond and Stephen Boyd. Cvxpy: A python-embedded modeling language for convex optimization. The Journal of Machine Learning Research, 17(1):2909–2913, 2016.
[9] Paul Erdős, Alfred Rényi, et al. On the evolution of random graphs. Publication of the Mathematical Institute of the Hungarian Academy of Sciences, 5(1):17-60, 1960.
[10] Benjamin Girault. Stationary graph signals using an isometric graph translation. In Proceedings of the 23rd European Signal Processing Conference (EUSIPCO 2015), pages 1516-1520. IEEE, 2015.
[11] Yiran He and Hoi-To Wai. Detecting central nodes from low-rank excited graph signals via structured factor analysis. IEEE Transactions on Signal Processing, 70:2416-2430, 2022.
[12] Alan J Hoffman. On approximate solutions of systems of linear inequalities. Journal of Research of the National Bureau of Standards, 49(4):263-265, 1952.
[13] Alexander Jung, Alfred O Hero III, Alexandru Cristian Mara, Saeed Jahromi, Ayelet Heimowitz, and Yonina C Eldar. Semi-supervised learning in network-structured data via total variation minimization. IEEE Transactions on Signal Processing, 67(24):6256-6269, 2019.
[14] Alexander Jung and Nguyen Tran. Localized linear regression in networked data. IEEE Signal Processing Letters, 26(7):1090-1094, 2019.
[15] Vassilis Kalofolias. How to learn a graph from smooth signals. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016), pages 920-929. PMLR, 2016.
[16] Vassilis Kalofolias, Andreas Loukas, Dorina Thanou, and Pascal Frossard. Learning time varying graphs. In Proceedings of 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), pages 2826-2830. IEEE, 2017.