SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shangyuan Liu
shangyuanliu@link.cuhk.edu.hk
Linglingzhi Zhu
llzzhu@se.cuhk.edu.hk
Anthony Man-Cho So
manchoso@se.cuhk.edu.hk
Graph learning from signals is a core task in graph signal processing (GSP). A significant subclass of graph signals called the stationary graph signals that broadens the concept of stationarity of data defined on regular domains to signals on graphs is gaining increasing popularity in the GSP community [10, 20, 25]. The most commonly used model to learn graphs from these stationary signals is  $SpecT$  [31], which forms the foundation for nearly all the subsequent, more advanced models. Despite its strengths, the practical formulation of the model, known as  $rSpecT$ , has been identified to be susceptible to the choice of hyperparameters. More critically, it may suffer from infeasibility as an optimization problem. In this paper, we introduce the first condition that ensures the infeasibility of  $rSpecT$  and design a novel model called  $LogSpecT$ , along with its practical formulation  $rLogSpecT$  to overcome this issue. Contrary to  $rSpecT$ , our novel practical model  $rLogSpecT$  is always feasible. Furthermore, we provide recovery guarantees of  $rLogSpecT$  from modern optimization tools related to epi-convergence, which could be of independent interest and significant for various learning problems. To demonstrate the practical advantages of  $rLogSpecT$ , a highly efficient algorithm based on the linearized alternating direction method of multipliers (L-ADMM) that allows closed-form solutions for each subproblem is proposed with convergence guarantees. Extensive numerical results on both synthetic and real networks not only corroborate the stability of our proposed methods, but also highlight their comparable and even superior performance than existing models.
Learning with graphs has proved its relevance in many practical areas, such as life science [36, 37], signal processing [13, 14, 38, 39], and financial engineering [1, 21], to just name a few. However, there are many cases that the graphs are not readily prepared and only the data closely related to the graphs can be observed. Hence, a core task in Graph Signal Processing (GSP) is to learn the underlying graph topology based on the interplay between data and graphs [22].
Among these commonly used data properties, an assumption that is referred to as the graph signal stationarity [10] is gaining attention [30, 20, 25]. Graph signal stationarity is the extension of the notion of stationarity from the regular space/time domain to the irregular domain (i.e. graphs). The latter is a traditional hypothesis in signal processing to capture a special type of statistical relationship between samples of a temporal signal. It is a cornerstone of many signal processing methods. Although the graph signal stationarity property is proposed more from a theoretic end, the
techniques based on this idealistic assumption pave the way for the development of the following more advanced and applicable models [35, 34]. Furthermore, several works have shown that some real datasets can be approximately viewed as stationary or partially explained by the stationarity assumption. For instance, [25] revealed that the well-known USPS dataset and the CMUPIE set of cropped faces exhibit near stationarity. [10] found that certain weather data could be explained by stationary graph signals. Additionally, [31] highlighted the relevance of the stationarity assumption in learning protein structure. [43] applied the stationary assumption to US Senate Roll Calls and achieved good performance.
The predominant methods to process stationary graph signals and learn topology under the stationarity assumption are the spectral-template-based models. The start of this line of works is [31], which proposed a vanilla model called  $SpecT$  [31] to learn graphs from stationary graph signals. Many extensions of this fundamental model have been made since then to accommodate the more realistic and complicated settings, e.g. joint inference of multiple graphs [32, 27], learning with streaming graph signals [33], learning with hidden nodes [4], learning from both smooth and stationary graph signals [5]. In practice, SpecT requires the unknown data covariance matrix. Hence, a robust formulation called  $rSpecT$  is proposed [31], which replaces the unknown covariance matrix with its estimate and introduces a hyperparameter to reflect the estimation inaccuracy. Unfortunately, the model is sensitive to this hyperparameter and improper tuning of it may jeopardize the model performance or lead to model infeasibility.
The current approach to selecting an appropriate value is heuristic. It finds the smallest one that allows for a feasible solution. This method has two shortcomings. On one hand, it is computationally costly. The smallest value is the exact solution to a second-order conic programming, which is solved approximately with computational methods. Any approximate solution that is smaller than the exact one will make rSpecT infeasible. On the other hand, such an approach lacks theoretical guarantees as how the smallest value behaves as the number of observations increases is unknown. This issue makes the existing recovery guarantees invalid. To the best of our knowledge, only a few works have focused on how to understand the performance of the robust formulation rSpecT. [24] empirically showed that the feasible set of the robust formulation approximates the original one as more samples are collected. [23] provided the first recovery guarantee for rSpecT. Their work exploits the fact that the objective is the  $\ell_1$ -norm and the constraints are linear [44]. However, the conditions needed in their work are not only restrictive but also hard to check, as they require the full rankness of a large-scale matrix related to random samples. Another model [34] tries to circumvent the model infeasibility issue by turning a constraint into a penalty. However, this approach introduces another hyperparameter that is neither easy to tune nor amenable to theoretical analysis. Also, this hyperparameter should go to infinity heuristically when more and more samples are collected, which may exacerbate model instability.
In this paper, we propose an alternative formulation of SpecT that can bypass the aforementioned issues. We first provide a condition that ensures the infeasibility of the fundamental model rSpecT. Then, we propose a novel formulation called LogSpecT and its practical formulation  $r\text{LogSpecT}$  based on the spectral template and incorporating a log barrier. The inclusion of a log barrier as a regularizer on the degrees is inspired by its extensive application in graph learning from smooth graph signals [15, 16, 19, 47] and the discussion in [7] emphasizing that restricting the degrees to preclude isolated nodes is crucial to learning graphs with special structures. It is worth noting that the approach in [7], which directly imposes constraints, may exacerbate the infeasibility issue in our problem. Although rLogSpecT still requires a parameter to reflect the estimation error of the unknown covariance matrix, it no longer suffers from the infeasibility issue associated with the improper choice of the parameter. Furthermore, we provide the decaying rate of the parameter that can guarantee the recovery of rLogSpecT. This not only aids in the parameter tuning, but also establishes a connection between rLogSpecT and LogSpecT through non-asymptotic recovery analysis. These theoretical results are derived using contemporary optimization tools related to epi-convergence [29]. Unlike the current guarantees for rSpecT that are built on an  $\ell_1$  analysis model [44], our approach based on epi-convergence is not confined to the specific types of optimization problems (e.g. the combination of the  $\ell_1$ -norm and log barrier in the objective function) and consequently admits broader applications that can be of independent interest.
In the algorithmic aspect, we design a linearized alternating direction method of multipliers (L-ADMM) to solve rLogSpecT. The subproblems of L-ADMM admit closed-form solutions and can be implemented efficiently due to the linearization technique. Also, we provide the convergence result of the proposed method. Finally, we conduct extensive experiments on both synthetic data and
real networks. Notably, the infeasibility issue of rSpecT frequently occurs in experiments on real networks, thereby emphasizing its practical significance beyond theoretical concerns. Furthermore, we observe that even when the classical models are set to be feasible, our novel models (LogSpecT and rLogSpecT) exhibit improved accuracy and stability, particularly in the case of synthetic BA graphs. This empirically illustrates that although our models are initially proposed to circumvent the infeasibility issue, they are superior alternatives to classical methods in the general cases.
1.1 Notation
The notation we use in this paper is standard. We use  $[m]$  to denote the set  $\{1,2,\ldots,m\}$  for any positive integer  $m$ . Let the Euclidean space of all real matrices be equipped with the inner product  $\langle \mathbf{X},\mathbf{Y}\rangle \coloneqq \mathrm{tr}(\mathbf{X}^{\top}\mathbf{Y})$  for any matrices  $\mathbf{X},\mathbf{Y}$  and denote the induced Frobenius norm by  $\| \cdot \| _F$  (or  $\| \cdot \| _2$  when the argument is a vector). For any  $\mathbf{X}\in \mathbb{R}^{m\times n}$ , we use  $\mathbb{B}(\mathbf{X},\rho)$  to denote the closed Euclidean ball centering at  $\mathbf{X}$  with radius  $\rho$ . Let  $\| \cdot \|$  be the operator norm,  $\| \mathbf{X}\|_{1,1}\coloneqq \sum_{i,j}|X_{ij}|$ ,  $\| \mathbf{X}\|_{\infty ,\infty}\coloneqq \max_{i,j}|\mathbf{X}_{ij}|$ , and let  $\mathrm{diag}(\pmb {X})$  be the vector formed by the diagonal entries of  $\pmb{X}$ . For a column vector  $\pmb{x}$ , let  $\mathrm{Diag}(\pmb {x})$  be the diagonal matrix whose diagonal elements are given by  $\pmb{x}$ . Given a closed and convex set  $\mathcal{C}$ , we use  $\Pi_{\mathcal{C}}(\pmb {w})\coloneqq \operatorname *{argmin}_{\pmb {v}\in \mathcal{C}}\| \pmb {v} - \pmb {w}\| _2$  to denote the projection of the point  $\pmb{w}$  onto the set  $\mathcal{C}$ . We use 1 (resp. 0) to denote an all-one vector (resp. all-zero vector) whose dimension will be clear from the context. For a set  $\mathcal{S}$  and any real number  $\alpha$ , let  $\alpha S\coloneqq \{\alpha x\mid x\in S\}$ ,  $S1\coloneqq \{\pmb {x1}\mid \pmb {x}\in S\}$ , and  $\iota_S(\cdot)$  be the indicator function of  $S$ . For two non-empty and compact sets  $\mathcal{X}$  and  $\mathcal{Y}$ , the distance between them is defined as  $\mathrm{dist}(\mathcal{X},\mathcal{Y})\coloneqq \sup_{\pmb {X}\in \mathcal{X}}\inf_{\pmb {Y}\in \mathcal{Y}}\| \pmb {X} - \pmb {Y}\| _F^1$ .
Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  be a graph, where  $\mathcal{V} = [m]$  is the set of nodes and  $\mathcal{E}\subset \mathcal{V}\times \mathcal{V}$  is the set of edges. Let  $S$  be the weight matrix associated with the graph  $\mathcal{G}$ , where  $S_{ij}$  represents the weight of the edge between nodes  $i$  and  $j$ . In this paper, we consider undirected graphs without self-loops. The set of adjacency matrices for such graphs is  $S\coloneqq \{S\in \mathbb{R}^{m\times m}\mid S = S^{\top}$ ,  $\mathrm{diag}(S) = 0$ ,  $S\geq 0\}$ . Suppose that the adjacency matrix  $S$  admits the eigen-decomposition  $S = U\Lambda U^{\top}$ , where  $\Lambda$  is a diagonal matrix and  $U$  is an orthogonal matrix. A graph filter is a linear operator  $h:\mathbb{R}^{m\times m}\to \mathbb{R}^{m\times m}$  defined as  $h(S) = \sum_{i = 0}^{p}h_{i}S^{i} = U\left(\sum_{i = 0}^{p}h_{i}\Lambda^{i}\right)U^{\top}$ , where  $p > 0$  is the order of the graph filter and  $\{h_i\}_{i = 0}^p$  are the filter coefficients. According to the convention, we have  $S^0 = I$ .
A graph signal can be represented by a vector  $\pmb{x} \in \mathbb{R}^m$ , where the  $i$ -th element  $x_{i}$  is the signal value associated with node  $i$ . A signal  $\pmb{x}$  is said to be stationary if it is generated from
$$
\boldsymbol {x} = h (\boldsymbol {S}) \boldsymbol {w}, \tag {1}
$$
where  $\pmb{w}$  satisfies  $\mathbb{E}[\pmb{w}] = \mathbf{0}$  and  $\mathbb{E}[\pmb{w}^\top \pmb{w}] = \pmb{I}$ . Simple calculations give that the covariance matrix of  $\pmb{x}$ , which is denoted by  $C_{\infty}$ , shares the same eigenvectors with  $\pmb{S}$ . Hence, we have the constraint
$$
\boldsymbol {C} _ {\infty} \boldsymbol {S} = \boldsymbol {S} \boldsymbol {C} _ {\infty}. \tag {2}
$$
Based on this, the following fundamental model SpecT is proposed to learn graphs from stationary signals without the knowledge of graph filters [31, 34]:
$$
\min  _ {\boldsymbol {S}} \| \boldsymbol {S} \| _ {1, 1}, \text {s . t .} \quad C _ {\infty} \boldsymbol {S} = \boldsymbol {S} C _ {\infty}, \boldsymbol {S} \in \mathcal {S} \cap \left\{\boldsymbol {S} \in \mathbb {R} ^ {m \times m}: \sum_ {j = 1} ^ {m} \boldsymbol {S} _ {1 j} = 1 \right\}, \quad (\text {S p e c T})
$$
where the constraint  $\sum_{j=1}^{m} S_{1j} = 1$  is used to preclude the trivial optimal solution  $S^{*} = 0$ . When  $C_{\infty}$  is unknown and only  $n$  i.i.d samples of  $\{\pmb{x}_i\}$  from (1) are available, the robust formulation, which is based on the estimate  $C_n$  of  $C_{\infty}$  and called  $rSpecT$ , is used:
$$
\min  _ {\boldsymbol {S}} \| \boldsymbol {S} \| _ {1, 1}, \text {s . t .} \| \boldsymbol {C} _ {n} \boldsymbol {S} - \boldsymbol {S C} _ {n} \| _ {F} \leq \delta , \boldsymbol {S} \in \mathcal {S} \cap \left\{\boldsymbol {S} \in \mathbb {R} ^ {m \times m}: \sum_ {j = 1} ^ {m} \boldsymbol {S} _ {1 j} = 1 \right\}. (\text {r S p e c T})
$$
For this robust formulation, the recovery guarantee is studied empirically in [24] and theoretically in [23] under some conditions that are restrictive and hard to check.
It reduces to the classic point-to-set (resp. point-to-point) distance when  $\mathcal{V}$  (resp.  $\mathcal{X}$  and  $\mathcal{Y}$ ) is a singleton.
3 Infeasibility of rSpecT and Novel Models
Even though rSpecT has gained much popularity, there are few works that discuss the choice of the hyperparameter  $\delta$  and how it affects the model feasibility. In this section, we present a condition under which rSpecT is always infeasible and then propose an alternative formulation. To motivate our results, let us consider the following 2-node example.
Example 3.1. Consider a simple graph containing 2 nodes. Then, the set given by the second constraint of  $rSpecT$  is  $S \cap \{S \in \mathbb{R}^{m \times m} : \sum_{j=1}^{m} S_{1j} = 1\} = [0,1;1,0]$ , which is a singleton. Suppose that the sample covariance matrix is  $C_n = [h_{11}, h_{12}; h_{12}, h_{22}]$ . Then, the constraint  $\| C_n S - SC_n \|_F \leq \delta$  is reduced to  $2(h_{11} - h_{22})^2 \leq \delta^2$ . Hence, when  $h_{11} \neq h_{22}$  and  $\delta < \sqrt{2} |h_{11} - h_{22}|$ ,  $rSpecT$  has no feasible solution.
Before delving into the general case, we introduce a linear operator  $\pmb{B} \in \mathbb{R}^{m^2 \times m(m-1)/2}$  that maps the vector  $\pmb{x} \in \mathbb{R}_+^{(m-1)m/2}$  to the vectorization of an adjacency matrix  $S \in S$  of a simple, undirected graph:
$$
\left(\boldsymbol {B} \boldsymbol {x}\right) _ {m (i - 1) + j} = \left\{ \begin{array}{l l} x _ {i - j + \frac {j - 1}{2} (2 m - j)}, & \text {i f} i > j, \\ 0, & \text {i f} i = j, \\ x _ {j - i + \frac {i - 1}{2} (2 m - i)}, & \text {i f} i <   j, \end{array} \right. \tag {3}
$$
where  $i,j\in [m]$ . We also define  $\pmb{A}_{n}\coloneqq \pmb{I}\otimes \pmb{C}_{n} - \pmb{C}_{n}\otimes \pmb{I}\in \mathbb{R}^{m^{2}\times m^{2}}$ , so that the first constraint of rSpecT can be rewritten as  $\| C_nS - SC_n\| _F\leq \delta \iff \| A_n\mathrm{vec}(S)\| _2\leq \delta$ , where  $\mathrm{vec}(\cdot)$  is the vectorization operator. We now give a condition that guarantees the infeasibility of rSpecT.
Theorem 3.2. Consider the linear system
$$
\boldsymbol {A} _ {n} \boldsymbol {B} \boldsymbol {y} = \mathbf {0}, \quad \boldsymbol {y} \leq \mathbf {0}, \quad \sum_ {i = 1} ^ {m - 1} y _ {i} \neq \mathbf {0}, \tag {4}
$$
where  $\pmb{y} \in \mathbb{R}^{\frac{m(m - 1)}{2}}$ . If (4) has no feasible solution, then there exists a  $\bar{\delta}_n > 0$  such that  $rSpecT$  is infeasible for all  $\delta_n \in [0, \bar{\delta}_n)$ .
Remark 3.3. From Theorem 3.2 we can infer that for any fixed  $n$ , the linear system (4) has no solution when  $A_{n}B$  has full column rank (e.g., Example 3.1). This leads to the infeasibility of rSpecT with  $\delta_n \in [0, \bar{\delta}_n)$ . As remarked in [31], one should tackle the feasibility issue of rSpecT with caution.
The failure of rSpecT (SpecT) lies in the existence of the constraint  $(S1)_1 = 1$ , which is used to preclude the trivial solution  $S^{*} = 0$ . For this reason, we resort to an alternative approach to bypassing the trivial solution. When the graphs are assumed to have no isolated node, the log barrier is commonly applied [15]. In these cases, the zero solution is naturally precluded. This observation inspires us to propose the following novel formulation, which combines the log barrier with the spectral template (2) to learn graphs without isolated nodes from stationary signals:
$$
\min  _ {\boldsymbol {S}} \| \boldsymbol {S} \| _ {1, 1} - \alpha \mathbf {1} ^ {\top} \log (\boldsymbol {S} \mathbf {1})
$$
(LogSpecT)
$$
\begin{array}{l} \text {s . t .} \quad \boldsymbol {S} \in \mathcal {S}, \quad \boldsymbol {S C} _ {\infty} = \boldsymbol {C} _ {\infty} \boldsymbol {S}, \end{array}
$$
where  $\| \cdot \|_{1,1}$  is a convex relaxation of  $\| \cdot \|_0$  promoting graph sparsity,  $\mathbf{1}^\top \log(S\mathbf{1})$  is the penalty to guarantee the nonexistence of isolated nodes, and  $\alpha$  is the tuning parameter. As can be seen from the following proposition, the hyperparameter  $\alpha$  in LogSpecT only affects the scale of edge weights instead of the graph connectivity structure.
Proposition 3.4. Let  $\mathrm{Opt}(C, \alpha)$  be the optimal solution set of LogSpecT with input covariance matrix  $C$  and parameter  $\alpha > 0$ . Then, for any  $\gamma > 0$ , it follows that
$$
\operatorname {O p t} (\boldsymbol {C}, \alpha) = \gamma \operatorname {O p t} (\boldsymbol {C}, \alpha / \gamma) = \alpha \operatorname {O p t} (\boldsymbol {C}, 1).
$$
Remark 3.5. The result of Proposition 3.4 spares us from tuning the hyperparameter  $\alpha$  when we are coping with binary graphs. In fact, certain normalization will eliminate the impact of different values of  $\alpha$  and preserve the connectivity information. Hence, we may simply set  $\alpha = 1$  in implementation.
Note that the true covariance matrix  $C_{\infty}$  in LogSpecT is usually unknown and an estimate  $C_n$  from  $n$  i.i.d samples is available. To tackle the estimation inaccuracy, we introduce the following robust
formulation:
(rLogSpecT)
$$
\begin{array}{l} \text {s . t .} \quad \boldsymbol {S} \in \mathcal {S}, \quad \| \boldsymbol {S C} _ {n} - \boldsymbol {C} _ {n} \boldsymbol {S} \| _ {F} ^ {2} \leq \delta_ {n} ^ {2}. \end{array}
$$
This formulation substitutes  $C_{\infty}$  with  $C_n$  and relaxes the equality constraint to an inequality constraint with a tolerance threshold  $\delta_n$ . Contrary to rSpecT, we prove that rLogSpecT is always feasible.
Proposition 3.6. For  $\delta_{n} > 0$  and  $C_n$  with any fixed  $n$ , rLogSpecT always has a nontrivial feasible solution.
4 Recovery Guarantee of rLogSpecT
In this section, we investigate the non-asymptotic behavior of rLogSpecT when more and more i.i.d samples are collected. For the sake of brevity, we denote  $f \coloneqq \| \cdot \|_{1,1} - \alpha \mathbf{1}^\top \log (\cdot \mathbf{1})$ . The theoretical recovery guarantee is as follows:
Theorem 4.1. If  $\delta_n \geq 2\rho\|C_n - C_\infty\|$  with given  $\rho \coloneqq \max\{f^*, \alpha m, \alpha m(\log \alpha - 1)\}$ , then there exist constants  $c_1, \tilde{c}_1, \tilde{c}_2 > 0$  such that
$$
\left. \left| f _ {n} ^ {*} - f ^ {*} \right| \leq \varepsilon_ {n}; \quad \text {(i i)} \operatorname {d i s t} \left(\mathcal {S} ^ {n, *}, \mathcal {S} _ {2 \varepsilon_ {n}} ^ {*}\right) \leq \varepsilon_ {n}; \quad \text {(i i i)} \operatorname {d i s t} \left(\mathcal {S} ^ {n, *} \mathbf {1}, \mathcal {S} _ {0} ^ {*} \mathbf {1}\right) \leq \tilde {c} _ {1} \varepsilon_ {n} + \tilde {c} _ {2} \sqrt {\varepsilon_ {n}}, \right.
$$
where  $\varepsilon_{n} := c_{1}(\delta_{n} + 2\rho \| C_{n} - C_{\infty}\|)$ ,  $S^{n,*}$  is the optimal solution set of rLogSpecT,  $f^{*}$  (resp.  $f_{n}^{*}$ ) denotes the optimal value of LogSpecT (resp. rLogSpecT), and
$$
\mathcal {S} _ {\varepsilon} ^ {*} := \left\{\boldsymbol {S} \in \mathcal {S} \mid \boldsymbol {S C} _ {\infty} = \boldsymbol {C} _ {\infty} \boldsymbol {S}, f (\boldsymbol {S}) \leq f ^ {*} + \varepsilon \right\}
$$
is the  $\varepsilon$ -suboptimal solution set of LogSpecT.
Remark 4.2. (i) Compared with the conclusion (ii) in the Theorem 4.1, the conclusion (iii) links the optimal solution sets of  $r\mathrm{LogSpec}T$  and  $\mathrm{LogSpec}T$  instead of the sub-optimal solutions.
(ii) A byproduct of the proof (see Appendix E.2 for details) shows that the optimal node degree vector set  $(S_0^* \mathbf{1})$  is a singleton. However, there is no guarantee of the uniqueness of  $S^*$  itself. We will discuss the impact of such non-uniqueness on model performance in Section 6.4.
Remark 4.3. The proof of Theorem 4.1 relies on two important optimization concepts: Epi-convergence and truncated Hausdorff distance. Epi-convergence is closely related to the asymptotic solution behavior of approximate minimization problems, and the truncated Hausdorff distance is used to characterize the epi-convergence non-asymptotically. With the help of the Kenmochi condition, which allows us to explicitly calculate the truncated Hausdorff distance, we are able to study the non-asymptotic behavior of the optimal value and optimal solutions of the models. We refer the reader to Appendix E and also [28, Chapter 7], [29, Chapter 6.J] for more details.
Corollary 4.4. Under the assumptions in Theorem 4.1, it follows that if  $\delta_n \to 0$ , then
$$
\lim  _ {n \rightarrow \infty} \operatorname {d i s t} \left(\mathcal {S} ^ {n, *}, \mathcal {S} _ {0} ^ {*}\right) = 0. \tag {5}
$$
Remark 4.5. From the strong law of large numbers, if we choose  $\delta_{n} = \mathcal{O}(\| C_{n} - C_{\infty}\|)$ , then almost surely one has  $\lim_{n\to \infty}\| C_n - C_\infty \| = 0$  and consequently  $\delta_{n}\rightarrow 0$ . Hence, Corollary 4.4 shows that (5) holds almost surely.
For the remaining part of this section, we study the choice of  $\delta_{n}$  under certain statistical assumptions. A large number of distributions (e.g., Gaussian distributions, exponential distributions and any bounded distributions) can be covered by the sub-Gaussian distribution (cf. [41, Section 2.5]), whose formal definition is as follows.
Definition 4.6 (Sub-Gaussian Distributions). The probability distribution of a random vector  $\mathbf{w}$  is called sub-Gaussian if there are  $C, v > 0$  such that for every  $t > 0$ ,  $\mathbb{P}(\| \mathbf{w}\|_2 > t) \leq Ce^{-vt^2}$ .
Consider the case that  $w$  in the generative model (1) follows a sub-Gaussian distribution. The following result is adapted from [40, Proposition 2.1].
Lemma 4.7. Suppose that  $\mathbf{w}$  in the generative model (1) follows a sub-Gaussian distribution. Then,  $\mathbf{x}$  follows a sub-Gaussian distribution, and with probability larger than  $1 - \mathcal{O}(\frac{1}{n})$ , one has  $\| C_n - C_\infty \| \leq \mathcal{O}(\sqrt{\log n / n})$ .
Equipped with non-asymptotic results in Theorem 4.1, we choose  $\delta_{n}$  with a specific decaying rate.
Corollary 4.8. If the input  $\mathbf{w}$  in (1) follows a sub-Gaussian distribution and  $\delta_n = \mathcal{O}(\sqrt{\log n / n})$ , then the assumptions in Theorem 4.1 hold with probability larger than  $1 - \mathcal{O}\left(\frac{1}{n}\right)$ .