SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seiyun Shin<sup>1</sup>, Ilan Shomorony<sup>1</sup>, and Han Zhao<sup>2</sup>
$^{1}$ Department of ECE and  $^{2}$ Department of CS
University of Illinois at Urbana-Champaign, IL
{seiyuns2, ilans, hanzhao}@illinois.edu
Graph Neural Networks (GNNs) are a powerful class of machine learning models with applications in recommender systems, drug discovery, social network analysis, and computer vision. One challenge with their implementation is that GNNs often take large-scale graphs as inputs, which imposes significant computational/storage costs in the training and testing phases. In particular, the message passing operations of a GNN require multiplication of the graph adjacency matrix  $A \in \mathbb{R}^{n \times n}$  and the data matrix  $X \in \mathbb{R}^{n \times d}$ , and the  $O(n^2 d)$  time complexity can be prohibitive for large  $n$ . Thus, a natural question is whether it is possible to perform the GNN operations in (quasi-)linear time by avoiding the full computation of  $AX$ . To study this question, we consider the setting of a regression task on a two-layer Linear Graph Convolutional Network (GCN). We develop an efficient training algorithm based on (1) performing node subsampling, (2) estimating the leverage scores of  $AX$  based on the subsampled graph, and (3) performing leverage score sampling on  $AX$ . We show that our proposed scheme learns the regression model observing only  $O(nd\varepsilon^{-2}\log n)$  entries of  $A$  in time  $O(nd^2\varepsilon^{-2}\log n)$ , with the guarantee that the learned weights deviate by at most  $\varepsilon$  under the  $\ell_2$  norm from the model learned using the entire adjacency matrix  $A$ . We present empirical results for regression problems on real-world graphs and show that our algorithm significantly outperforms other baseline sampling strategies that exploit the same number of observations.
Graph Neural Networks (GNNs) have gained popularity as a powerful machine learning method for graph-structured data. By learning rich representations of graph data, GNNs can solve a variety of prediction tasks on graphs [1-7]. GNNs have delivered impressive results across many different areas, including social network analysis [8], bioinformatics [9-11], and recommendation systems [12].
Given their remarkable performance, being able to train GNNs efficiently is an important task. However, training GNNs can be quite challenging in the context of large-scale graphs, which impose significant computational costs. In particular, the message passing scheme in GNNs requires, for each node, summing up the feature vectors of all neighboring nodes into one feature vector. For a multi-layer Graph Convolutional Network (GCN) [3], the layer-wise propagation rule is
$$
H ^ {(\ell + 1)} = \sigma \left(A H ^ {(\ell)} W ^ {(\ell)}\right), \tag {1}
$$
where  $H^{(\ell)}$  denotes the feature representation at the  $\ell$ th layer, with  $H^{(0)} = X$  (i.e., the data matrix consisting of nodes' features of an input graph  $G$ );  $W^{(\ell)}$  denotes the weight matrix;  $\sigma(\cdot)$  is a non-linear activation function like the ReLU; and  $A$  denotes the adjacency matrix of  $G$ .
If the graph has  $n$  nodes and the feature dimension is  $d$ , computing the matrix multiplication  $AX$  requires  $O(n^{2}d)$  time and can be prohibitive in big data settings. A natural question is whether it is possible to train a GCN to avoid the quadratic complexity scaling with  $n$ .
Another motivation for avoiding the full computation of  $AX$  is applications where the adjacency matrix  $A$  is not fully known a priori, and it must be learned via node/edge queries. For example, in large social networks (the current number of social media users is over 4.89 billion [13]), one may need to access the adjacency matrix  $A$  by querying the adjacency list of specific nodes. As another example, certain biological networks must be learned through the physical probing of pairwise interactions, which may make obtaining the entire adjacency matrix  $A$  prohibitively expensive. For example, the mapping of the neural network of living organisms (such as the connectome of C. elegans [14]) requires physical probing of the connectivity between neurons.
A natural approach to avoid the quadratic complexity of computing  $AX$  is via the subsampling of the graph  $G$  [15, 6, 16, 17]. If the resulting subgraph is sparse enough,  $AX$  can be computed very efficiently. But how many entries of  $A$  need to be observed in order to guarantee that the message passing step  $AX$  can be computed accurately enough in the context of GNNs? What kinds of graph subsampling strategies are amenable to theoretical performance guarantees on the training of GNNs?
We consider the setting of a regression task to be learned via a GCN. Let  $A \in \mathbb{R}^{n \times n}$  be the weighted adjacency matrix of a graph  $G = (V, A)$  with  $|V| = n$ . Each node  $v_{i} \in V$  has an associated feature vector of dimension  $d (< n)$  and a label, denoted by  $(\mathbf{x}_i, y_i) \in \mathbb{R}^d \times \mathbb{R}$ , which can also be represented as a  $n \times d$  data matrix  $X$  and label vector  $\mathbf{y}$ . Training the GCN corresponds to minimizing the loss
$$
\mathcal {L} (W, A, X, \mathbf {y}) := \frac {1}{2 n} \| \mathbf {y} - f _ {\mathrm {G C N}} (W, A, X) \| _ {2} ^ {2}, \tag {2}
$$
on the training data  $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$ , where  $W$  denotes the GCN network weights, and  $f_{\mathrm{GCN}}(W, A, X)$  denotes a feed-forward computation of a GCN's output. As a first step to studying whether (2) can be solved accurately on a sparse subgraph of  $G$ , we focus on a simple linear GCN, where there is no non-linearity  $\sigma(\cdot)$ . Specifically, the feed-forward output is given by  $f_{\mathrm{GCN}}^{\mathrm{linear}}(\mathbf{w}, A, X) \coloneqq AX\mathbf{w}$ , where we use  $\mathbf{w} \in \mathbb{R}^d$  (instead of  $W \in \mathbb{R}^{1 \times d}$ ) to indicate that the learnable parameters are in the form of a vector. Hence, our goal is to solve
$$
\min  _ {\mathbf {w}} \quad \frac {1}{2 n} \| \mathbf {y} - A X \mathbf {w} \| _ {2} ^ {2}. \tag {3}
$$
Note that one can view this optimization problem as a graph-weighted linear regression problem.
The setting of linear regression provides natural suggestions for a subsampling strategy. It is known that leverage score sampling allows one to solve the linear regression problem with  $(1 + \varepsilon)$  accuracy using only  $O(d\varepsilon^{-2}\log n)$  subsampled rows of the data matrix  $X$  and in time  $O(nd^2 +d^3)^1$  [18-22]. Nevertheless, to apply this strategy to (3), one would need to first compute  $AX$ , requiring  $O(n^{2}d)$  time, so that the row leverage scores of  $AX$  can then be computed. This motivates us to propose a two-step approach that (1) performs node subsampling to obtain an approximate observation of  $AX$  and estimate the leverage scores of the rows of the augmented matrix  $[AX| - \mathbf{y}] \in \mathbb{R}^{n\times (d + 1)}$  and (2) performs leverage score sampling on the rows of  $[AX| - \mathbf{y}]$  using the estimated scores.
This two-step approach is illustrated in Figure 1. In order to obtain an estimator of  $AX$ , our sampling scheme in the first step builds  $O(\log n)$  rank-1 matrices by sampling the same amount of columns of  $A$  and the corresponding rows of  $X$ . The key idea behind this approach is taking a random subset of nodes and propagating their feature vectors to their neighbors. We provide a spectral approximation guarantee for the resulting estimate of  $AX$  and an approximation guarantee for the leverage scores of the augmented matrix  $[\widehat{AX} - \mathbf{y}]$  computed on this estimate. In the second step, we adopt the standard leverage score sampling for sampling rows of  $A$ , using the leverage score estimates of  $[AX] - \mathbf{y}]$  obtained from the first step. With  $O(d\varepsilon^{-2}\log n)$  sampled rows of  $\tilde{A}$  and  $\tilde{y}$ , the training algorithm then computes  $\tilde{A} X$  and uses it to solve the regression problem. We show that our proposed scheme learns the regression model with learned weights deviated by at most  $\varepsilon$  from those with full information, by observing only  $O(nd\varepsilon^{-2}\log n)$  entries of  $A$ , and in time  $O(nd^2\varepsilon^{-2}\log n)$ .
On real-world benchmark datasets, we demonstrate the performance improvements of the two proposed schemes over other baseline sampling schemes via numerical simulations.
1More precisely, the time complexity is  $\tilde{O} ((\mathrm{nnz}(AX) + d^3)\log (\varepsilon^{-1}))$  , where  $\tilde{O} (\cdot)$  hides factors polylogarithmic in  $(n,d)$
Figure 1: Two-step Algorithm: (1) perform node subsampling to obtain an estimate  $\widehat{AX}$ , from which we can compute  $\ell_i([\widehat{AX}|-y]$  for  $i = 1, \ldots, n$ ; and (2) use the leverage score estimates to perform leverage score sampling on  $A$  and  $y$  and perform regression using  $\tilde{A}X$  and  $\tilde{y}$  instead of  $AX$  and  $y$ .
Notation: Throughout the paper, we use  $A = (a_{ij}) \in \mathbb{R}_{\geq 0}^{n \times n}$  to denote the graph adjacency matrix, where each entry value is nonnegative and bounded (i.e.,  $\bar{0} \leq a_{ij} \leq M$  for some constant  $M > 0$ ). We write  $A_{i}$  and  $A_{j}$  to indicate the  $i$ th row and  $j$ th column of the matrix  $A$  respectively. In addition,  $X \in \mathbb{R}^{n \times d}$  denotes the data (design) matrix, where the  $i$ th row in  $X$ ,  $\mathbf{x}_i$ , corresponds to the node vector at the  $i$ th node  $v_i$  in the graph  $G$ . We also assume that the absolute value of each entry value of  $X$  is bounded by  $M$  (i.e.,  $|X_{ij}| \leq M$ ). We denote vectors by a lowercase bold letter (e.g.,  $\mathbf{x} \in \mathbb{R}^n$ ), and by default, all the vectors will be in column format, hence  $X = (\mathbf{x}_1^\top; \ldots; \mathbf{x}_n^\top)^\top$ , where  $(\cdot)^\top$  denotes the transpose. We indicate the  $i$ th entry of  $\mathbf{x}$  with either  $[\mathbf{x}]_i$  or  $x_i$ , and we use  $[\mathbf{x}]_S$  to denote the concatenation of  $x_i$ s for all  $i \in S$ . We also let  $[n] := \{1, 2, \ldots, n\}$ . We denote by  $\langle \mathbf{x}, \mathbf{y} \rangle := \sum_{i} x_i y_i \in \mathbb{R}$  the inner product between  $\mathbf{x}$  and  $\mathbf{y}$ . Unless otherwise mentioned, we use  $\|\mathbf{v}\|$  for the  $\ell_2$ -norm for vector  $\mathbf{v}$ . We let  $\|A\| := \max_{\|\mathbf{v}\|=1} \|A\mathbf{v}\|$  denote the operator norm of a matrix, and  $\|A\|_F := \sqrt{\sum_{i,j} a_{ij}^2}$  denote the Frobenius norm. We denote the standard basis vector by  $\mathbf{e}_i$ . We write  $\mathrm{Bern}(p)$  for a Bernoulli distribution with parameter  $p$ . Lastly, we denote by  $\mathrm{nz}(A)$  the number of non-zero entries in matrix  $A$ .
Related Work: We provide a detailed discussion of related works in Appendix A.
2 Motivation: Leverage Score Sampling for Linear Regression
First, we briefly describe key results on leverage score sampling that will be relevant to our discussion. We refer to [23] for a detailed discussion on the subject, and we provide key proofs in Appendix D. Definition 1. The leverage score of the  $i$ th row of  $X$  is  $\ell_i(X) \coloneqq \mathbf{x}_i^\top (X^\top X)^\dagger \mathbf{x}_i$ , where  $(\cdot)^\dagger$  denotes the Moore-Penrose pseudoinverse.
Intuitively, a row's leverage score measures how important it is in composing the row space of  $X$ . If a row has a component orthogonal to all the other rows, its leverage score is 1. Removing it would decrease the rank of  $X$ , completely changing its row space. The leverage scores are the diagonal entries of the projection matrix  $X(X^{\top}X)^{-1}X^{\top}$ , which can be used to show that  $\sum_{i=1}^{n} \ell_i(X) = \operatorname{rank}(X)$ . The following alternative characterizations of  $\ell_i(X)$  will be useful.
Proposition 1. Let  $X = U\Sigma V^{\top}$  be the singular value decomposition (SVD) of  $X$ , where  $U \in \mathbb{R}^{n\times d}$ ,  $\Sigma \in \mathbb{R}^{d\times d}$  and  $V \in \mathbb{R}^{d\times d}$ . Then  $\ell_i(X) = \| U_{i:}\| _2^2$ , where  $U_{i}$  is the  $i$ th row of  $U$ .
Proposition 2. The leverage score can be alternatively computed as  $\ell_i(X) = \max_{\mathbf{v}} \frac{(\mathbf{x}_i^\top \mathbf{v})^2}{\|X \mathbf{v}\|_2^2}$ .
Through the use of leverage scores, it is possible to approximately solve a linear regression task using a subset of the data points [18-22]. In particular, leverage score sampling allows the spectral approximation of a matrix.
Definition 2. A matrix  $\widehat{X} \in \mathbb{R}^{n \times d}$  is an  $\varepsilon$ -spectral approximation of  $X \in \mathbb{R}^{n \times d}$  if, for all  $\mathbf{v} \in \mathbb{R}^d$ ,
$$
(1 - \varepsilon) \cdot \| X \mathbf {v} \| _ {2} \leq \| \widehat {X} \mathbf {v} \| _ {2} \leq (1 + \varepsilon) \cdot \| X \mathbf {v} \| _ {2}. \tag {4}
$$
Suppose we are given a data matrix  $X = (\mathbf{x}_1^\top; \ldots; \mathbf{x}_n^\top)^\top$  and a vector  $\mathbf{u} = (u_1, \ldots, u_n)$  of leverage score overestimates of  $X$  (i.e.,  $\ell_i(X) \leq u_i$ ,  $\forall i \in [n]$ ). We create a matrix  $\tilde{X}$  by including the  $i$ th row
of  $X$  in  $\tilde{X}$  with probability  $p_i = \min[1, cu_i \varepsilon^{-2} \log n]$  for some constant  $c$ , scaled by  $1 / \sqrt{p_i}$ . The following is a slight modification of Lemma 4 in [21] (which we prove in Appendix D):
Lemma 1. With probability at least  $1 - n^{-\Omega (1)}$ , for any  $\mathbf{v}$ , we have
$$
(1 - \varepsilon) \cdot \| X \mathbf {v} \| \leq \| \tilde {X} \mathbf {v} \| \leq (1 + \varepsilon) \cdot \| X \mathbf {v} \|.
$$
Notice that the expected number of rows in  $\tilde{X}$  is  $c\| \mathbf{u}\| _1\varepsilon^{-2}\log n$ . Hence, as long as  $\| \mathbf{u}\| _1 = O(d)$ , we only need to sample  $O(d\varepsilon^{-2}\log n)$  rows of  $X$  to obtain a good spectral approximation.
Now suppose we want to solve the OLS problem  $\min_{\mathbf{w}} \frac{1}{2} \| \mathbf{y} - X \mathbf{w} \|_2^2$  where  $\mathbf{y} \in \mathbb{R}^n$  is the response vector corresponding to  $X$ . The optimal solution is given by  $\mathbf{w}^* = (X^\top X)^{-1} X^\top \mathbf{y}$ . Exactly computing  $\mathbf{w}^*$  requires  $O(nd^2 + d^3)$  time.
An alternative approach using leverage score sampling would be as follows. First, we note that the OLS objective can be rewritten using  $\| \mathbf{y} - X\mathbf{w}\| = \| [X| - \mathbf{y}]\cdot \left[ \begin{array}{l}\mathbf{w}\\ 1 \end{array} \right]\|$ . If we now have leverage score overestimates  $u_{1},\ldots ,u_{n}$  for the augmented matrix  $[X| - \mathbf{y}]\in \mathbb{R}^{n\times (d + 1)}$ , we can construct the subsampled matrix  $[X_S| - \mathbf{y}_S]\in \mathbb{R}^{|S|\times (d + 1)}$  (with rows rescaled by  $1 / \sqrt{p_i}$ ). Then, by Lemma 1, a solution to
$$
\tilde {\mathbf {w}} = \arg \min  _ {\mathbf {w}} \left. \left\| [ X _ {S} | - \mathbf {y} _ {S} ] \cdot \left[ \begin{array}{l} \mathbf {w} \\ 1 \end{array} \right] \right\| _ {2}, \right. \tag {5}
$$
would satisfy  $\| \mathbf{y}_S - X_S\tilde{\mathbf{w}}\| _2\leq \| \mathbf{y}_S - X_S\mathbf{w}^*\| _2\leq (1 + \varepsilon)\| \mathbf{y} - X\mathbf{w}^*\| _2$
The exact leverage scores of  $X$  can be computed in time  $O(nd^2)$  (see Algorithm 3 in Appendix B) and, since  $|S| = O(d\varepsilon^{-2}\log n)$ , the reduced linear system (5) can be solved in time  $O(d^{3}\varepsilon^{-2}\log n)$ . The overall complexity for solving the OLS problem in this way is given by  $O(nd^2 + d^3\varepsilon^{-2}\log n)$ . Notice that there is no real acceleration obtained by this procedure. But also notice that here we are using the exact leverage scores for sampling while, from Lemma 1, we know that this approach works as long as we have good leverage score overestimates.
3 Efficient Training of a Linear GCN via Leverage Scores
In this section, we introduce an efficient training algorithm that approximately solves the graph-weighted regression task (3) by observing only  $O(nd\varepsilon^{-2}\log n)$  entries of  $A$ , and in time  $O(nd^2\varepsilon^{-2}\log n)$ . The necessity for introducing the efficient training algorithm comes from the  $O(n^{2}d)$  time complexity of computing  $AX$  via matrix multiplication.
Motivated by the discussion in Section 2 for standard linear regression, a natural approach is to attempt to compute leverage score overestimates for  $AX$  efficiently without the full computation of  $AX$ . Our proposed algorithm uses the following three steps to approximately solve the graph-weighted linear regression problem.
1. ESTIMATELEVERAGESCORES(A,X) (See Algorithm 1 and Algorithm 2)
2. LEVERAGESCORESAMPLING(A,  $\{\widehat{\ell_i} ([AX| - \mathbf{y}]),\forall i\in [n]\}$ ) (See Algorithm 4 in Appendix B)
3. REGRESSIONSOLVER(  $\tilde{A} X,\tilde{\mathbf{y}}$  ) (See Algorithm 5 in Appendix B)
In ESTIMATELEVERAGESCORES  $(A,X)$ , a node subsampling technique is used to build an estimate  $\widehat{AX}$  of  $AX$  with an approximation guarantee. The estimate  $\widehat{AX}$  can then be used to produce provably good overestimates of the leverage scores of the augmented matrix  $[AX| - y]$ . Next, LEVERAGESCORESAMPLING  $(A,\{\widehat{\ell_i} ([AX| - y]),\forall i\in [n]\})$  uses these overestimates to produce a matrix  $\tilde{A}$  and scaled labels  $\tilde{\mathbf{y}}$  consisting of a reduced number of rows of  $A$  and  $\mathbf{y}$  respectively. Using Lemma 1, we then show that  $\tilde{A} X$  provides a spectral approximation for  $AX$ , which allows approximately solving the regression problem (3) using REGRESSIONSOLVER  $(\tilde{A} X,\tilde{\mathbf{y}})$ . We note that none of the algorithmic procedures requires fully computing  $AX$  and the end-to-end algorithm has an  $O(n\log n)$  time dependence on  $n$ .
Algorithm 1 ESTIMATELEVERAGESCORES(A, X) via Uniform Sampling
Input: Adjacency matrix  $A \in \mathbb{R}^{n \times n}$ , data matrix  $X \in \mathbb{R}^{n \times d}$ , budget  $B$ , threshold  $\varepsilon > 0$
Output: Leverage score overestimates for  $AX$
1: Draw  $I_{j} \sim \mathrm{Bern}(p)$ ,  $\forall j \in [n]$  independently, where  $p = \min \left[\frac{B}{n}, 1\right]$
2:  $\widehat{A_{:j}X_{j}}\gets \frac{I_j}{p} A_{:j}X_{j};\forall j\in [n]$
3:  $\widehat{AX} \gets \sum_{j=1}^{n} \widehat{A_{:j}X_{j}}$ .  $\triangleright O(Bd)$
4:  $S\gets$  LEVERAGESCORE([AX- y]) See Alg. 3 in App. B;  $\tilde{O} (\mathrm{nnz}([\widehat{AX} ] - \mathbf{y}] + d^3)$
5:  $\{\widehat{\ell_i} ([AX| - \mathbf{y}]),\forall i\in [n]\} \gets \frac{(1 + \varepsilon)^2}{(1 - \varepsilon)^2}\cdot S$
6: return  $\widehat{\ell}_i([AX| - \mathbf{y}])$ ,  $\forall i \in [n]$
3.1 First Stage: Uniform Sampling of  $A_{:j}X_{j}$
In order to efficiently estimate the leverage scores of  $AX$ , a key observation we make is that  $AX$  can be decomposed as a sum of  $n$  rank-1 matrices as  $AX = \sum_{j=1}^{n} A_{:j} X_{j}$ , where  $A_{:j}$  and  $X_{j}$  denote  $j$ th column of the adjacency matrix  $A$  and  $j$ th row of the data matrix  $X$ . Notice that one can view  $A_{:j} X_{j}$  as the effect of node  $j$ 's feature vector propagated to other nodes after the one-step message passing operation. Hence, a natural approximation strategy is to take a subset of these rank-1 matrices. First, we consider a uniform sampling strategy that samples node indices  $j$  uniformly at random and computes  $A_{:j} X_{j}$ . See Algorithm 1 for the detailed procedure.
Given a budget  $B$  (the total number of nodes that should be observed), the algorithm draws sampling indicator variables  $I_{j} \sim \mathrm{Bern}(p)$  independently with  $p = \min \left[\frac{B}{n}, 1\right]$ . For  $j = 1, \dots, n$ , we build
$$
\widehat {A _ {: j} X _ {j :}} = \frac {I _ {j}}{p} A _ {: j} X _ {j:},
$$
and then set  $\widehat{AX} := \sum_{j} \widehat{A_{:j}X_{j}}$ . Notice that  $\widehat{AX}$  is an unbiased estimate of  $AX$ , because for all  $i, j$ ,
$$
\mathbb {E} \left[ \widehat {[ A X ] _ {i j}} \right] = \sum_ {k = 1} ^ {n} p \cdot \frac {1}{p} a _ {i k} x _ {k j} = \sum_ {k = 1} ^ {n} a _ {i k} x _ {k j} = [ A X ] _ {i j}. \tag {6}
$$
In addition, one can readily see that  $B$  rank-1 matrices are sampled in expectation, and, with high probability, the actual number of sampled rank-1 matrices is less than  $(1 + \alpha)B$  for  $\alpha > 0$ .
Next, we present a spectral approximation guarantee for  $\widehat{AX}$ . In order to establish that result, we make the following mild assumption on the density of the input graph.
Assumption 1. We assume that the input graph  $G$  satisfies
$$
\left\| A X \right\| \geq n ^ {3 / 2} d ^ {1 / 2}. \tag {7}
$$
Strictly speaking, (7) is not just an assumption about the density of  $G$ , as it takes into consideration the interaction between  $A$  and  $X$ . Nevertheless, this assumption would be expected to hold in cases where the graph  $G$  is dense. In particular, since  $\| AX\| \geq d^{-1 / 2}\| AX\| _F$ , Assumption 1 holds if  $\| AX\| _F\geq n^{3 / 2}d$ . Assuming that  $A$  has  $\delta n^2$  non-zero entries and that entries of  $X$  are drawn i.i.d. from some distribution with positive second moment  $\beta^2$ , independently of  $A$ , we have  $\| AX\| _F\approx \sqrt{\delta n^2d\beta^2} = \beta \delta^{1 / 2}n^{3 / 2}d^{1 / 2}$ . Treating  $d$  as a constant and rescaling  $X$  by a constant, we have that the assumption holds. More generally, if  $d$  is a constant and
$$
\left| \left\{(i, j) \in [ n ] \times [ d ]: \mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j} \geq c _ {1} n \right\} \right| \geq c _ {2} n
$$
for constants  $c_{1}$  and  $c_{2}$ , then the assumption holds (after a constant rescaling of the data matrix). The following stronger assumption will be needed to establish guarantees for the leverage score estimates:
Assumption 2. For all  $(i,j)\in [n]\times [d],\mathbf{e}_i^\top AX\mathbf{e}_j\geq c_1n$  for some constant  $c_{1}$
Notice that if the graph  $G$  is instead sparse, the problem we focus on due to the computation of  $AX$  is less severe since one can efficiently compute  $AX$  in time that depends on  $\mathrm{nnz}(A)$ .
The proof of the following theorem is in Appendix E.
Theorem 1. Under Assumption 1, if we run Algorithm 1 with budget  $B = C \varepsilon^{-2} \log n$  for large enough  $C > 0$ , the estimate  $\widehat{AX}$  satisfies
$$
(1 - \varepsilon) \cdot \| A X \mathbf {v} \| \leq \| \widehat {A X} \mathbf {v} \| \leq (1 + \varepsilon) \cdot \| A X \mathbf {v} \|, \quad \forall \mathbf {v} \in \mathbb {R} ^ {d}, \tag {8}
$$
with probability at least  $1 - n^{-\Omega (1)}$
Given the spectral approximation guarantee in Theorem 1, we expect the leverage scores computed on the augmented matrix  $\widehat{[AX| - y]}$  to be close to the leverage scores of  $[AX| - y]$ . Since we want an overestimate of the leverage scores, we use the multiplicative constant  $\left(\frac{1 + \varepsilon}{1 - \varepsilon}\right)^2$  to obtain
$$
\widehat {\ell} _ {i} ([ A X | - \mathbf {y} ]) := \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {2} \cdot \ell_ {i} ([ \widehat {A X} | - \mathbf {y} ]), \quad \forall i \in [ n ]. \tag {9}
$$
The following lemma, whose proof is in Appendix F, asserts that  $\widetilde{\ell}_i([AX| - \mathbf{y}])$  is indeed an overestimate of  $\ell_i([AX| - \mathbf{y}])$ .
Lemma 2. Under Assumption 2, Algorithm 1 with budget  $B = C\varepsilon^{-2}\log n$  for large enough  $C > 0$  outputs leverage score overestimates satisfying
$$
\ell_ {i} ([ A X | - \mathbf {y} ]) \leq \widehat {\ell_ {i}} ([ A X | - \mathbf {y} ]) \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {4} \ell_ {i} ([ A X | - \mathbf {y} ]), \quad \forall i \in [ n ], \tag {10}
$$
3.2 Second Stage: Leverage Score Sampling and Approximate Regression
The second step of the proposed algorithm exploits the leverage score estimates  $\widehat{\ell}_i([AX| - \mathbf{y}])$ ,  $\forall i \in [n]$  to perform the standard leverage score sampling. With Lemma 1, one can readily obtain a  $(1 + \varepsilon)$ -approximation guarantee for the graph-weighted linear regression task.