SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To mitigate the "neighbor explosion" problem, most of the related works [6, 16, 15, 43] take layerwise sampling approaches to sample neighbors of the nodes in the previous layer in a top-down manner. Specifically, layer-wise samplers [16, 15, 43] adopt importance sampling to sample neighbors in the lower layer given each node in the upper layer, assuming that the sample size for each layer is independent of each other.
Second, node-wise samplers [6] randomly sample neighbors in the lower layer given each node in the upper layer. In addition, [17] proposed a bandit-based approach to perform node-wise sampling in order to optimize the variance in the vein of layer sampling approaches. See also [44], which proposed a variance-reduced estimator to reduce the receptive field of a single node.
Third, there have been several works [45, 32] that adopt graph-wise sampling approaches. In particular, the proposed schemes first partition the node set of a graph [45] or sample subgraphs [32], and then train models on those partitions or subgraphs in a batch mode [3]. They show that the training time of each epoch may be much faster compared with layer-wise sampling approaches.
Graph-weighted linear regression: We note that there have been a few works that address linear regression on network-linked data [46-48]. Yet neither sample complexity analysis nor time complexity analysis is provided.
B Basic Algorithms
Algorithm 3 LEVERAGESCORE(X)
Input: Data matrix  $X\in \mathbb{R}^{n\times d}$
Output: Leverage score  $\ell_i(X)^4$ ,  $\forall i \in [n]$
1: Compute  $K\coloneqq X^{\top}X$ $\triangleright O(nd^{2})$
2:  $(V,\Sigma ,V^{\top})\gets \mathrm{SVD}(K)$
3:  $U\gets X(V\Sigma^{-1})$ $\triangleright O(nd^{2})$
4: return  $\ell_i(X) := \| U_i\| _2^2, \forall i \in [n]$
4The time complexity is  $O(nd^2 +d^3)$  in total, and more precisely, it is  $\tilde{O} ((\mathrm{nnz}(AX) + d^3)\log (\varepsilon^{-1}))$  see [18-22] for details.
Algorithm 4 LEVERAGESCORESAMPLING  $(A, \{\widehat{\ell}_i([AX| - y]), \forall i \in [n]\})$
Input: Adjacency matrix  $A\in \mathbb{R}^{n\times n}$ , leverage score estimate of  $AX\left\{\widehat{\ell}_i([AX| - \mathbf{y}])\right.$ $\forall i\in [n]\}$  Output:  $\tilde{A}\in \mathbb{R}^{O(d\varepsilon^{-2}\log n)\times n}$ , scaled labels  $\tilde{\mathbf{y}}\in \mathbb{R}^{O(d\varepsilon^{-2}\log n)}$
1: Draw  $I_{j} \sim \mathrm{Bern}(p_{j})$ ,  $\forall j \in [n]$  independently, where  $p_{j} = c\widehat{\ell_{j}}([AX| - y])\varepsilon^{-2}\log n$
2:  $\tilde{A}\gets \sum_{j = 1}^{n}\frac{I_j}{\sqrt{p_j}}\mathbf{e}_jA_j$
3:  $\tilde{\mathbf{y}}\gets \sum_{j = 1}^{n}\frac{I_j}{\sqrt{p_j}} y_j\mathbf{e}_j$
4: return  $(\hat{A}, \hat{\mathbf{y}})$
Algorithm 5 REGRESSIONSOLVER  $(\tilde{A} X,\tilde{\mathbf{y}})$
Input: Matrix  $\tilde{A} X\in \mathbb{R}^{O(d\varepsilon^{-2}\log n)\times d}$ , scaled labels  $\tilde{\mathbf{y}}\in \mathbb{R}^{O(d\varepsilon^{-2}\log n)}$
Output: Weight vector  $\tilde{\mathbf{w}}\in \mathbb{R}^d$
1:  $\tilde{\mathbf{w}}\gets \left((\tilde{A} X)^{\top}(\tilde{A} X)\right)^{\dagger}(\tilde{A} X)^{\top}\tilde{\mathbf{y}}$
2: return  $\tilde{\mathbf{w}}$
C Proof of Proposition 2
Define  $t\coloneqq \max_{\mathbf{v}}\frac{(\mathbf{x}_i^\top\mathbf{v})^2}{\|X\mathbf{v}\|_2^2}$  . Hence, it follows that
$$
\forall \mathbf {v}, \mathbf {v} ^ {\top} \left(\mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top}\right) \mathbf {v} \leq t \cdot \mathbf {v} ^ {\top} \left(X ^ {\top} X\right) \mathbf {v}.
$$
Equivalently, this means that  $t$  is the smallest value such that
$$
\mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} \preceq t \cdot X ^ {\top} X.
$$
Let  $\mathbf{w} = X\mathbf{v}$ . Since  $X$  has full column rank, we know that  $\mathbf{v} = X^{\dagger}\mathbf{w}$ . Without loss of generality, assume  $\| \mathbf{w}\| _2 = 1$ , then we can equivalently transform the above constraint as
$$
\forall \mathbf {w}, \| \mathbf {w} \| _ {2} = 1, \quad \mathbf {w} ^ {\top} (X ^ {\dagger}) ^ {\top} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} X ^ {\dagger} \mathbf {w} \leq t.
$$
However, note that  $(X^{\dagger})^{\top}\mathbf{x}_i\mathbf{x}_i^{\top}X^{\dagger}$  is a rank-one matrix, hence it suffices to choose  $t$  such that
$$
\begin{array}{l} t \geq \operatorname {t r} \left(\left(X ^ {\dagger}\right) ^ {\top} \mathbf {x} _ {i} \mathbf {x} _ {i} ^ {\top} X ^ {\dagger}\right) = \operatorname {t r} \left(\mathbf {x} _ {i} ^ {\top} X ^ {\dagger} \left(X ^ {\dagger}\right) ^ {\top} \mathbf {x} _ {i}\right) = \operatorname {t r} \left(\mathbf {x} _ {i} ^ {\top} \left(X ^ {\top} X\right) ^ {\dagger} \mathbf {x} _ {i}\right) \\ = \operatorname {t r} \left(\mathbf {x} _ {i} ^ {\top} (X ^ {\top} X) ^ {\dagger} \mathbf {x} _ {i}\right) = \mathbf {x} _ {i} ^ {\top} (X ^ {\top} X) ^ {\dagger} \mathbf {x} _ {i} = \ell_ {i} (X). \\ \end{array}
$$
In this section, we prove Lemma 1, which yields a good approximation by sampling with probability proportional to the leverage score overestimates:
Lemma 3. Given a data matrix  $X = (\mathbf{x}_1^\top; \ldots; \mathbf{x}_n^\top)^\top \in \mathbb{R}^{n \times d}$  and a vector  $\mathbf{u} = (u_1, \ldots, u_n)$  of leverage score overestimates of  $X$  (i.e.,  $\ell_i(X) \leq u_i$ ,  $\forall i \in [n]$ ), suppose we create a matrix  $\tilde{X}$  by including the  $i$ th row of  $X$  in  $\tilde{X}$  with probability  $p_i = \min[1, cu_i\varepsilon^{-2}\log n]$  for some constant  $c$ , scaled by  $1 / \sqrt{p_i}$ . With probability at least  $1 - n^{-\Omega(1)}$ , for any  $\mathbf{v}$ , we have
To prove the lemma, we rely on the following matrix Chernoff bound, which is a variant of the matrix Chernoff bound in [21, Lemma 11] (also, see [49]).
Suppose  $W_{1}, \ldots, W_{n}$  are independent, random positive semidefinite matrices with size  $d \times d$ . Define  $Z := \sum_{j=1}^{n} W_{j}$ . If  $W_{j} \preceq R \cdot \mathbb{E}[Z]$ , then
$$
\Pr (Z \succeq (1 + \varepsilon) \mathbb {E} [ Z ]) \leq d \cdot \exp \left(- \frac {\varepsilon^ {2}}{3 R}\right); \tag {12}
$$
$$
\Pr \left(Z \preceq (1 - \varepsilon) \mathbb {E} [ Z ]\right) \leq d \cdot \exp \left(- \frac {\varepsilon^ {2}}{2 R}\right), \tag {13}
$$
where  $A \preceq B$  is equivalent to saying  $\mathbf{v}^\top (A - B)\mathbf{v} \geq 0$  for matrices  $A$  and  $B$  and for all  $\mathbf{v}$ .
To apply this inequality, define  $W_{j} \coloneqq \frac{1}{p_{j}}\mathbf{x}_{j}\mathbf{x}_{j}^{\top}$ , which is sampled with probability  $p_j$ . We first know that  $Z$  is an unbiased estimate of  $X^{\top}X$  since
$$
\mathbb {E} [ Z ] = \sum_ {j = 1} ^ {n} p _ {j} \cdot \frac {1}{p _ {j}} \mathbf {x} _ {j} \mathbf {x} _ {j} ^ {\top} = \sum_ {j = 1} ^ {n} \mathbf {x} _ {j} \mathbf {x} _ {j} ^ {\top} = X ^ {\top} X.
$$
First, consider the case where  $p_j < 1$ . This implies that
$$
\ell_ {j} (X) \leq u _ {j} \leq \frac {1}{\varepsilon^ {- 2} \log n}.
$$
In order to invoke the matrix Chernoff bound, we only need to verify that  $W_{j}$  ( $\succeq 0$ ) is spectrally upper bounded:
$$
W _ {j} = \frac {1}{c u _ {j} \varepsilon^ {- 2} \log n} \mathbf {x} _ {j} \mathbf {x} _ {j} ^ {\top} \preceq \frac {1}{c \ell_ {j} (X) \varepsilon^ {- 2} \log n} \mathbf {x} _ {j} \mathbf {x} _ {j} ^ {\top} \preceq \frac {\ell_ {j} (X) \cdot X ^ {\top} X}{c \ell_ {j} (X) \varepsilon^ {- 2} \log n} = \frac {X ^ {\top} X}{c \varepsilon^ {- 2} \log n}, \tag {14}
$$
where the last inequality holds due to the proof of Proposition 2. Now, the statement follows from the matrix Chernoff bound with  $W_{j}^{\prime} = (X^{\top}X)^{-1 / 2}W_{j}(X^{\top}X)^{-1 / 2}$  and  $R = \frac{1}{c\varepsilon^{-2}\log n}$ , to conclude that
$$
(1 - \varepsilon) \cdot X ^ {\top} X \preceq \sum_ {j = 1} ^ {n} W _ {j} \preceq (1 + \varepsilon) \cdot X ^ {\top} X,
$$
with probability at least  $1 - d(n^{-c/3} + n^{-c/2}) = 1 - n^{-\Omega(1)}$ , for some  $c > 0$  and the regime where  $d < n$ . Note that we use a union-bound technique to derive the desired high probability guarantee.
Now consider the case where  $p_j = 1$ , where  $W_j \coloneqq \mathbf{x}_j\mathbf{x}_j^\top$  is sampled with probability 1. In this case, (14) generally does not hold. However, since sampling  $W_j$  with probability 1 is equivalent to selecting  $W_j^{(1)}, \ldots, W_j^{(c\varepsilon^{-2}\log n)}$ , each equal to  $\frac{1}{c\varepsilon^{-2}\log n}\mathbf{x}_i\mathbf{x}^\top$  with probability 1, it follows that
$$
Y _ {j} ^ {(i)} \preceq \frac {X ^ {\top} X}{c \varepsilon^ {- 2} \log n}, \quad i = 1, \ldots , c \varepsilon^ {- 2} \log n.
$$
Here we assume that  $c\varepsilon^{-2}\log n$  is an integer for simplicity. Replacing  $Y_{j}$  with  $\sum_{i = 1}^{c\varepsilon^{-2}}\log nY_{j}^{(i)}$  yields the desired result since it does not change  $\mathbb{E}[Z]$ . This completes the proof.
We will use the following statement of Matrix Bernstein's inequality [49, Theorem 6.1.1]: Suppose  $W_{1},\ldots ,W_{n}$  are independent, random matrices with common dimension  $d_{1}\times d_{2}$  satisfying
$$
\mathbb {E} \left[ W _ {j} \right] = \mathbf {0}, \quad \text {a n d} \quad \| W _ {j} \| \leq L, \forall j \in [ n ].
$$
Define  $Z \coloneqq \sum_{j=1}^{n} W_j$ . Then for all  $t > 0$ ,
$$
\mathbb {E} \left[ \| Z \| \right] \leq \sqrt {2 v (Z) \log \left(d _ {1} + d _ {2}\right) + \frac {1}{3} \log \left(d _ {1} + d _ {2}\right)}; \tag {15}
$$
$$
\Pr \left(\left\| \sum_ {j = 1} ^ {n} W _ {j} \right\| \geq t\right) \leq \left(d _ {1} + d _ {2}\right) \cdot \exp \left(- \frac {\frac {1}{2} t ^ {2}}{v (Z) + \frac {1}{3} L t}\right), \tag {16}
$$
$$
v (Z) = \max  \left\{\left\| \mathbb {E} \left[ Z Z ^ {\top} \right] \right\|, \left\| \mathbb {E} \left[ Z ^ {\top} Z \right] \right\| \right\} = \max  \left\{\left\| \sum_ {j = 1} ^ {n} \mathbb {E} \left[ W _ {j} W _ {j} ^ {\top} \right] \right\|, \left\| \sum_ {j = 1} ^ {n} \mathbb {E} \left[ W _ {j} ^ {\top} W _ {j} \right] \right\| \right\}.
$$
To apply this inequality, first recall that Algorithm 1 samples  $A_{:j}X_{j}$ : to get a rank-1 matrix  $\widehat{A_{:j}X_{j}} := \frac{I_j}{p_j} A_{:j}X_j$ : with probability
$$
p _ {j} = \min \left[ \frac {B}{n}, 1 \right].
$$
Defining  $W_{j} \coloneqq \widehat{A_{:j}X_{j:}} - A_{:j}X_{j:}$ , we note that  $\mathbb{E}[W_j] = 0$  as shown in (6). In addition, assuming that entries of  $A \in \mathbb{R}^{n \times n}$  and  $X \in \mathbb{R}^{n \times d}$  satisfy  $|A_{ij}| \leq M$  and  $|X_{ij}| \leq M$ , we then have
$$
\begin{array}{l} \| W _ {j} \| = \left\| \frac {I _ {j}}{p _ {j}} A _ {: j} X _ {j:} - A _ {: j} X _ {j:} \right\| = \left\| \left(\frac {I _ {j}}{p _ {j}} - 1\right) A _ {: j} X _ {j:} \right\| \\ \leq \frac {1}{p _ {j}} \| A _ {: j} X _ {j:} \| \\ = \max  \left[ \frac {n}{B}, 1 \right] \cdot \| A _ {: j} \| \cdot \| X _ {j:} \| \\ \stackrel {(i)} {=} \frac {n}{B} \| A _ {: j} \| \| X _ {j:} \| \\ \stackrel {(i i)} {\leq} \frac {n ^ {3 / 2} d ^ {1 / 2} M ^ {2}}{B}, \\ \end{array}
$$
where  $(i)$  follows from the fact that we are interested in the regime where  $\frac{B}{n} = o(1)$ ; and  $(ii)$  follows from the fact that  $\| A_{:j}\| \leq \sqrt{n} M$  and  $\| X_{j:}\| \leq \sqrt{d} M$ . We can also bound  $\| \mathbb{E}[Z^{\top}Z]\|$  as
$$
\begin{array}{l} \left\| \mathbb {E} \left[ Z ^ {\top} Z \right] \right\| = \left\| \sum_ {j = 1} ^ {n} \mathbb {E} \left[ W _ {j} ^ {\top} W _ {j} \right] \right\| \leq \sum_ {j = 1} ^ {n} \left\| \mathbb {E} \left[ W _ {j} ^ {\top} W _ {j} \right] \right\| \\ = \sum_ {j = 1} ^ {n} \left\| \mathbb {E} \left[ \left(\frac {I _ {j}}{p _ {j}} - 1\right) ^ {2} X _ {j:} ^ {\top} A _ {: j} ^ {\top} A _ {: j} X _ {j:} \right] \right\| \\ \leq \sum_ {j = 1} ^ {n} \frac {1}{p _ {j}} \| X _ {j:} \| ^ {2} \| A _ {: j} \| ^ {2} \\ \leq \frac {n ^ {3} d M ^ {4}}{B}, \\ \end{array}
$$
and  $\mathbb{E}[ZZ^{\top}]$  can be similarly bounded, implying that  $v(Z)\leq \frac{n^{3}dM^{4}}{B}$ . Setting  $t = \varepsilon \| AX\|$  yields
$$
\begin{array}{l} \Pr \left(\left\| \sum_ {j = 1} ^ {n} W _ {j} \right\| \geq \varepsilon \cdot \| A X \|\right) \leq 2 (n + d) \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} \| A X \| ^ {2}}{\frac {n ^ {3} d M ^ {4}}{B} + \frac {1}{3} \frac {n ^ {3 / 2} d ^ {1 / 2} M ^ {2}}{B} \varepsilon \| A X \|}\right) \\ = 2 (n + d) \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} B}{\frac {n ^ {3} d M ^ {4}}{\| A X \| ^ {2}} + \frac {1}{3} \frac {n ^ {3 / 2} d ^ {1 / 2} M ^ {2}}{\| A X \|} \varepsilon}\right) \\ \stackrel {(i i i)} {\leq} 2 (n + d) \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} B}{M ^ {4} + \frac {1}{3} M ^ {2} \varepsilon}\right) \\ \stackrel {(i v)} {\leq} 4 n \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} B}{M ^ {4} + \frac {1}{3} M ^ {2} \varepsilon}\right), \\ \end{array}
$$
where  $(iii)$  follows from Assumption 1 that  $\|AX\| \geq n^{3/2}d^{1/2}$ ; and  $(iv)$  follows from the setting that we focus on  $d < n$ . Hence, as long as we set  $B = C\varepsilon^{-2}\log n$  for some constant  $C (>3M^4 > 0)$ , the exponent of the Matrix Bernstein inequality can be bounded as  $-C'\log n$  for some  $C' > 0$ , and thus can bound the deviation probability to be  $n^{-\Omega(1)}$ .
Now using the reverse triangle inequality,
$$
\left| \left\| \widehat {A X} \right\| - \left\| A X \right\| \right| \leq \left\| \widehat {A X} - A X \right\| \leq \varepsilon \| A X \|,
$$
we can conclude that the following spectral approximation holds
$$
(1 - \varepsilon) \| A X \| \leq \| \widehat {A X} \| \leq (1 + \varepsilon) \| A X \|,
$$
with probability at least  $1 - n^{-\Omega(1)}$ . This completes the proof.
F Proof of Lemma 2
We will use the following statement of Bernstein's inequality to get the desired result: If  $W_{1}, \ldots, W_{n}$  are independent zero-mean random variables such that  $|W_{i}| \leq M$  with probability 1. Then for all  $t > 0$ ,
$$
\Pr \left(\left| \sum_ {k = 1} ^ {n} W _ {k} \right| \geq t\right) \leq 2 \exp \left(- \frac {\frac {1}{2} t ^ {2}}{\sum_ {i = 1} ^ {n} \mathbb {E} \left[ W _ {i} ^ {2} \right] + \frac {1}{3} M t}\right). \tag {17}
$$
To apply this inequality, we first provide the following proposition which will serve as key ingredients for proving the lemma.
Proposition 3. Fix  $(i,j) \in [n] \times [d]$ . Let  $W_k := \frac{I_k}{p_k} \cdot [A_{:k}X_{k:}]_{ij} - [A_{:k}X_{k:}]_{ij}$ . Then for all  $k \in [n]$ , the following two hold:
1.  $|W_k| \leq \frac{nM^2}{B}$
2.  $\mathbb{E}\left[W_k^2\right] \leq \frac{nM^4}{B}$
Note that these two hold for all  $(i,j)\in [n]\times [d]$
Proof. We first bound  $|W_k|$  as:
$$
\begin{array}{l} \left| W _ {k} \right| = \left| \frac {I _ {k}}{p _ {k}} \left[ A _ {: k} X _ {k:} \right] _ {i j} - \left[ A _ {: k} X _ {k:} \right] _ {i j} \right| = \left| \left(\frac {I _ {k}}{p _ {k}} - 1\right) \left[ A _ {: k} X _ {k:} \right] _ {i j} \right| \\ \leq \frac {1}{p _ {k}} | [ A _ {: k} X _ {k:} ] _ {i j} | \\ = \max  \left[ \frac {n}{B}, 1 \right] \cdot M ^ {2} \\ \stackrel {(i)} {=} \frac {n M ^ {2}}{B}, \\ \end{array}
$$
where  $(i)$  follows from the fact that we are interested in the regime where  $\frac{B}{n} = o(1)$ . We also bound the second moment of  $W_{k}$  as:
$$
\begin{array}{l} \mathbb {E} \left[ W _ {k} ^ {2} \right] = \mathbb {E} \left[ \left(\frac {I _ {k}}{p _ {k}} - 1\right) ^ {2} [ A _ {: k} X _ {k:} ] _ {i j} ^ {2} \right] \\ \leq \frac {1}{p _ {k}} [ A _ {: k} X _ {k:} ] _ {i j} ^ {2} \\ \leq \frac {n M ^ {4}}{B}, \\ \end{array}
$$
where we used the facts that  $\mathbb{E}[I_{ij}^{(\ell)}] = p_j$ . This completes the proof.
Now setting  $t = \varepsilon \cdot \mathbf{e}_i^\top AX\mathbf{e}_j$  and applying the Bernstein's inequality now yields:
$$
\begin{array}{l} \Pr \left((i, j) \in [ n ] \times [ d ]: \left| \sum_ {k = 1} ^ {n} W _ {k} \right| \geq t\right) \\ = \Pr \left((i, j) \in [ n ] \times [ d ]: \left| \mathbf {e} _ {i} ^ {\top} \widehat {A X} \mathbf {e} _ {j} - \mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j} \right| \geq \varepsilon \cdot \mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j}\right) \\ \stackrel {(i)} {\leq} n ^ {2} \cdot \operatorname * {P r} \left(\left| \mathbf {e} _ {i} ^ {\top} \widehat {A X} \mathbf {e} _ {j} - \mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j} \right| \geq \varepsilon \cdot \mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j}\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq 2 n ^ {2} \cdot \exp \left(- \frac {\frac {\varepsilon^ {2}}{2} \left(\mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j}\right) ^ {2}}{n \cdot \frac {n M ^ {4}}{B} + \frac {1}{3} \cdot \frac {n M ^ {2}}{B} \cdot \left(\mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j}\right)}\right) \\ \stackrel {(i i)} {\leq} 2 n ^ {2} \cdot \exp \left(- \frac {\frac {B \varepsilon^ {2}}{2}}{M ^ {4} + \frac {1}{3} M ^ {2}}\right), \\ \end{array}
$$
where  $(i)$  follows from a union-bound technique; and  $(ii)$  follows from Assumption 2 that  $\mathbf{e}_i^\top AX\mathbf{e}_j = \Omega (n),\forall (i,j)\in [n]\times [d]$ . Setting  $B = \frac{C}{\varepsilon^2}\log n$  for sufficiently large  $C > \frac{16}{3} M^4 >0$ , one can observe that the last bound is upper bounded by  $n^{-\Omega (1)}$ . Since the other direction holds similarly, we conclude that with probability  $1 - n^{-\Omega (1)}$ , with budget of  $B = \frac{C}{\varepsilon^2}\log n$ ,
$$
(1 - \varepsilon) \cdot \mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j} \leq \mathbf {e} _ {i} ^ {\top} \widehat {A X} \mathbf {e} _ {j} \leq (1 + \varepsilon) \cdot \mathbf {e} _ {i} ^ {\top} A X \mathbf {e} _ {j}, \quad \forall (i, j) \in [ n ] \times [ d ].
$$
For the augmented matrix  $[AX| - \mathbf{y}]$ , the above then yields
$$
(1 - \varepsilon) \cdot {\bf e} _ {i} ^ {\top} [ A X | - {\bf y} ] {\bf e} _ {j} \leq {\bf e} _ {i} ^ {\top} [ \widehat {A X} | - {\bf y} ] {\bf e} _ {j} \leq (1 + \varepsilon) \cdot {\bf e} _ {i} ^ {\top} [ A X | - {\bf y} ] {\bf e} _ {j}, \quad \forall (i, j) \in [ n ] \times [ d ].
$$
Pick  $\mathbf{v} \in \mathbb{R}^{d + 1}$ , where it can be represented as
$$
\mathbf {v} = \sum_ {k = 1} ^ {d + 1} \alpha_ {k} \mathbf {e} _ {k},
$$
for some  $\alpha_{1},\ldots ,\alpha_{d + 1}$  . From the above, one can readily see that with probability at least  $1 - n^{-\Omega (1)}$
$$
\frac {(\mathbf {e} _ {i} ^ {\top} [ A X | - \mathbf {y} ] \mathbf {v}) ^ {2}}{\| [ A X | - \mathbf {y} ] \mathbf {v} \| _ {2} ^ {2}} \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {2} \cdot \frac {(\mathbf {e} _ {i} ^ {\top} [ \widehat {A X} | - \mathbf {y} ] \mathbf {v}) ^ {2}}{\| [ \widehat {A X} | - \mathbf {y} ] \mathbf {v} \| _ {2} ^ {2}} \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {4} \cdot \frac {(\mathbf {e} _ {i} ^ {\top} [ A X | - \mathbf {y} ] \mathbf {v}) ^ {2}}{\| [ A X | - \mathbf {y} ] \mathbf {v} \| _ {2} ^ {2}}, \quad \forall \mathbf {v}.
$$
Choosing  $\mathbf{v}^* := \arg \max_{\mathbf{v}} \frac{(\mathbf{e}_i^\top[\widehat{AX}|-y]\mathbf{v})^2}{\|\widehat{AX}|-y]\mathbf{v}\|_2^2}$  and  $\mathbf{w}^* := \arg \max_{\mathbf{w}} \frac{(\mathbf{e}_i^\top[AX|-y]\mathbf{w})^2}{\|\widehat{AX}|-y]\mathbf{w}\|_2^2}$ , from the above, we have
$$
\left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {2} \cdot \frac {\left(\mathbf {e} _ {i} ^ {\top} [ \widehat {A X} | - \mathbf {y} ] \mathbf {v} ^ {*}\right) ^ {2}}{\| [ \widehat {A X} | - \mathbf {y} ] \mathbf {v} ^ {*} \| _ {2} ^ {2}} \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {4} \cdot \frac {\left(\mathbf {e} _ {i} ^ {\top} [ A X | - \mathbf {y} ] \mathbf {v} ^ {*} \right. ^ {2}) ^ {2}}{\| [ A X | - \mathbf {y} ] \mathbf {v} ^ {*} \| _ {2} ^ {2}} \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {4} \cdot \frac {\left(\mathbf {e} _ {i} ^ {\top} [ A X | - \mathbf {y} ] \mathbf {w} ^ {*} \right. ^ {2})}{\| [ A X | - \mathbf {y} ] \mathbf {w} ^ {*} \| _ {2} ^ {2}},
$$
where the last formula is  $\left(\frac{1 + \varepsilon}{1 - \varepsilon}\right)^4\ell_i(AX)$  by the choice of  $\mathbf{w}^*$ . In a similar manner, we obtain
$$
\frac {(\mathbf {e} _ {i} ^ {\top} [ A X | - \mathbf {y} ] \mathbf {w} ^ {*}) ^ {2}}{\| [ A X | - \mathbf {y} ] \mathbf {w} ^ {*} \| _ {2} ^ {2}} \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {2} \cdot \frac {(\mathbf {e} _ {i} ^ {\top} [ \widehat {A X} | - \mathbf {y} ] \mathbf {w} ^ {*} ) ^ {2}}{\| [ A X | - \mathbf {y} ] \mathbf {w} ^ {*} \| _ {2} ^ {2}} \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {2} \cdot \frac {(\mathbf {e} _ {i} ^ {\top} [ \widehat {A X} | - \mathbf {y} ]   \mathbf {v} ^ {*} ) ^ {2}}{\| [ A X | - \mathbf {y} ] \mathbf {v} ^ {*} \| _ {2} ^ {2}}.
$$
Referring to Proposition 2 and the leverage score estimate  $\widehat{\ell_i} (AX)$  satisfying
$$
\widehat {\ell} _ {i} ([ A X | - \mathbf {y} ]) = \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {2} \ell_ {i} ([ \widehat {A X} | - \mathbf {y} ]) = \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {2} \cdot \max _ {\mathbf {v}} \frac {(\mathbf {e} _ {i} ^ {\top} [ \widehat {A X} | - \mathbf {y} ] \mathbf {v}) ^ {2}}{\| [ A X | - \mathbf {y} ] \mathbf {v} \| _ {2} ^ {2}},
$$
we get the desired result of
$$
\ell_ {i} ([ A X | - \mathbf {y} ]) \leq \widehat {\ell_ {i}} ([ A X | - \mathbf {y} ]) \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {4} \ell_ {i} ([ A X | - \mathbf {y} ]), \quad \forall i \in [ n ],
$$
G Theoretical Guarantees for Algorithm 2
In this section, we provide theoretical guarantees for Algorithm 2. As mentioned in Section 4, we obtain the same series of theoretical guarantees as those for Algorithm 1. For completeness, we leave the detailed statements below.
Theorem 3. Under Assumption 1, if we run Algorithm 2 with the budget  $B = C \varepsilon^{-2} \log n$  for large enough  $C > 0$ , the estimate  $\widetilde{A X}$  satisfies
$$
(1 - \varepsilon) \cdot \| A X \mathbf {v} \| \leq \| \widehat {A X} \mathbf {v} \| \leq (1 + \varepsilon) \cdot \| A X \mathbf {v} \|, \quad \forall \mathbf {v} \in \mathbb {R} ^ {d} \tag {18}
$$
Lemma 4. Under Assumption 1, Algorithm 2 with the budget  $B = C\varepsilon^{-2}\log n$  for large enough  $C > 0$  outputs the leverage score overestimates with a  $\left(\frac{1 + \varepsilon}{1 - \varepsilon}\right)^4$  multiplicative ratio; i.e.,
$$
\ell_ {i} ([ A X | - \mathbf {y} ]) \leq \widehat {\ell} _ {i} ([ A X | - \mathbf {y} ]) \leq \left(\frac {1 + \varepsilon}{1 - \varepsilon}\right) ^ {4} \ell_ {i} ([ A X | - \mathbf {y} ]), \quad \forall i \in [ n ], \tag {19}
$$
Theorem 4. Under Assumption 1, suppose we run Algorithm 2 with the budget  $B = C\varepsilon^{-2}\log n$  for large enough  $C > 0$ , to obtain  $\{\widehat{\ell}_i([AX| - \mathbf{y}]),\forall i\in [n]\}$ . Subsequently, if we run LEVERAGESCORESAMPLING(  $A,\{\widehat{\ell}_i([AX| - \mathbf{y})],\forall i\in [n]\}$ ), we obtain  $\tilde{A}\in \mathbb{R}^{O(d\varepsilon^{-2}\log n)\times n}$  and  $\tilde{\mathbf{y}}\in \mathbb{R}^{O(d\varepsilon^{-2}\log n)}$  such that REGRESSIONSOLVER  $(\tilde{A} X,\tilde{\mathbf{y}})$  provides
$$
\min  _ {\tilde {\mathbf {w}}} \| \mathbf {y} - \tilde {A} X \tilde {\mathbf {w}} \| _ {2} ^ {2} \leq (1 + \varepsilon) \cdot \min  _ {\mathbf {w}} \| \mathbf {y} - A X \mathbf {w} \| _ {2} ^ {2}.
$$
We note that it suffices to provide the proof of Theorem 3, as the rest follows in a similar manner to those for Algorithm 1.
Proof of Theorem 3: As in Appendix E, we will use the Matrix Bernstein's inequality described in (16). Recall that Algorithm 2 samples  $A_{:j}X_{j}$ : to get a rank-1 matrix  $\widehat{A_{:j}X_{j}} := \frac{I_j}{p_j} A_{:j}X_{j}$ ; with probability as in (11)
$$
p _ {j} = \min  \left[ B \cdot \frac {\| A _ {: j} \| \| X _ {j :} \|}{\sum_ {j} \| A _ {: j} \| \| X _ {j :} \|}, 1 \right],
$$
such that  $\sum_{j}p_{j}\leq B$
Defining  $W_{j} \coloneqq \widehat{A_{:j}X_{j}} - A_{:j}X_{j}$ : and we note that  $\mathbb{E}[W_j] = 0$  as shown in (6). In addition, assuming that entries of  $A \in \mathbb{R}^{n \times n}$  and  $X \in \mathbb{R}^{n \times d}$  satisfy  $|A_{ij}| \leq M$  and  $|X_{ij}| \leq M$ , we then have