SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| W _ {j} \right\| = \left\| \frac {I _ {j}}{p _ {j}} A _ {: j} X _ {j:} - A _ {: j} X _ {j:} \right\| = \left\| \left(\frac {I _ {j}}{p _ {j}} - 1\right) A _ {: j} X _ {j:} \right\| \\ \leq \frac {1}{p _ {j}} \| A _ {: j} X _ {j:} \| \\ = \max  \left[ \frac {1}{B} \cdot \frac {\sum_ {j} \| A _ {: j} \| \| X _ {j :} \|}{\| A _ {: j} \| \| X _ {j :} \|}, 1 \right] \cdot \| A _ {: j} \| \cdot \| X _ {j:} \| \\ = \max  \left[ \frac {1}{B} \cdot \sum_ {j = 1} ^ {n} \| A _ {: j} \| \| X _ {j:} \|, \| A _ {: j} \| \| X _ {j:} \| \right\| \\ \stackrel {(i)} {\leq} \max \left[ \frac {n ^ {3 / 2} d ^ {1 / 2} M ^ {2}}{B}, n ^ {1 / 2} d ^ {1 / 2} M \right] \\ \stackrel {(i i)} {\leq} \frac {n ^ {3 / 2} d ^ {1 / 2} M ^ {2}}{B}, \\ \end{array}
$$
where  $(i)$  follows from the fact that  $\| A_{:j}\| \leq \sqrt{n} M$  and  $\| X_{j:}\| \leq \sqrt{d} M$ ; and  $(ii)$  follows from the fact that we are interested in the regime where  $B = o(n)$ . We can also bound  $\| \mathbb{E}[Z^{\top}Z]\|$  as
$$
\begin{array}{l} \left\| \mathbb {E} \left[ Z ^ {\top} Z \right] \right\| = \left\| \mathbb {E} \left[ \sum_ {j = 1} ^ {n} \left(\frac {I _ {j}}{p _ {j}} A _ {: j} X _ {j:} - A _ {: j} X _ {j:}\right) ^ {\top} \left(\frac {I _ {j}}{p _ {j}} A _ {: j} X _ {j:} - A _ {: j} X _ {j:}\right) \right] \right\| \\ = \left\| \sum_ {j = 1} ^ {n} \left[ \frac {1}{p _ {j} ^ {2}} \left(A _ {: j} X _ {j:}\right) ^ {\top} \left(A _ {: j} X _ {j:}\right) \cdot p _ {j} - 2 \left(A _ {: j} X _ {j:}\right) ^ {\top} \left(A _ {: j} X _ {j:}\right) + \left(A _ {: j} X _ {j:}\right) ^ {\top} \left(A _ {: j} X _ {j:}\right) \right] \right\| \\ \end{array}
$$
$$
\begin{array}{l} = \left\| \sum_ {j = 1} ^ {n} \left(\frac {1}{p _ {j}} - 1\right) \cdot \left(\left\| A _ {: j} \right\| ^ {2} \cdot X _ {j:} ^ {\top} X _ {j:}\right) \right\| \\ \leq \left\| \sum_ {j = 1} ^ {n} \left[ \frac {1}{B} \cdot \left(\frac {\sum_ {j = 1} ^ {n} \| A _ {: j} \| \| X _ {j :} \|}{\| A _ {: j} \| \| X _ {j :} \|}\right) \cdot \left(\| A _ {: j} \| ^ {2} \cdot X _ {j:} ^ {\top} X _ {j:}\right) \right] \right\| \\ \leq \frac {1}{B} \cdot \left(\sum_ {j = 1} ^ {n} \| A _ {: j} \| \| X _ {j:} \|\right) \cdot \left\| \sum_ {j = 1} ^ {n} \frac {\| A _ {: j} \|}{\| X _ {j :} \|} X _ {j:} ^ {\top} X _ {j:} \right\| \\ = \frac {1}{B} \cdot \left(\sum_ {j = 1} ^ {n} \| A _ {: j} \| \| X _ {j:} \|\right) ^ {2} \\ \leq \frac {n ^ {3} d M ^ {4}}{B}. \\ \end{array}
$$
Also, one can readily see that  $\mathbb{E}[ZZ^{\top}]$  can be similarly bounded, implying that  $v(Z)\leq \frac{n^{3}dM^{4}}{B}$ . Setting  $t = \varepsilon \| AX\|$  yields:
$$
\begin{array}{l} \Pr \left(\left\| \sum_ {j = 1} ^ {n} W _ {j} \right\| \geq \varepsilon \cdot \| A X \|\right) \leq 2 (n + d) \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} \| A X \| ^ {2}}{\frac {n ^ {3} d M ^ {4}}{B} + \frac {1}{3} \frac {n ^ {3 / 2} d ^ {1 / 2} M ^ {2}}{B} \varepsilon \| A X \|}\right) \\ \leq 2 (n + d) \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} B}{\frac {n ^ {3} d M ^ {4}}{\| A X \| ^ {2}} + \frac {1}{3} \frac {n ^ {3 / 2} d ^ {1 / 2} M ^ {2}}{\| A X \|} \varepsilon}\right) \\ \stackrel {(i i i)} {\leq} 2 (n + d) \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} B}{M ^ {4} + \frac {1}{3} M ^ {2} \varepsilon}\right) \\ \stackrel {(i v)} {\leq} 4 n \cdot \exp \left(- \frac {\frac {1}{2} \varepsilon^ {2} B}{M ^ {4} + \frac {1}{3} M ^ {2} \varepsilon}\right), \\ \end{array}
$$
H Further Implementation Details
In this section, we provide additional details on the explanations of the datasets, wall-clock time with the adoption of sparse matrix multiplication, and peak memory usage comparisons.
Datasets: We consider benchmark datasets from the Open Graph Benchmark (OGB) [29], Stanford Network Analysis Project (SNAP) [30], and House dataset [31]. For the MSE comparisons, we consider (1) ogbl-ddi dataset from OGB, (2) ego-Facebook dataset from SNAP, and (3) House dataset. We note that while the third dataset has its data matrix (i.e., a concatenation of feature vectors) and labels, the first two datasets do not have feature matrices and target variables. Accordingly, to conduct controlled experiments, we synthetically generate data matrix  $X$ , weight vector  $\mathbf{w}$ , and noisy labels
Table 2: Wall-clock time comparison on end-to-end processes with sparse matrix multiplication
Dataset	# Nodes	# Edges	# Features	Wall-clock time (sec)
use of full AX	use of Algorithm 1 and leverage score sampling
ogbl-ssi [29]	4.3K	1.3M	100	1.02	0.84
ogbn-arxiv [29]	169.3K	1.2M	128	6.69	0.86
Synthetic data (Gaussian)	50.0K	625.0M	500	2452.14	58.30
Table 3: Peak memory usage comparison on sampling and regression steps
Dataset	# Nodes	# Edges	# Features	Peak memory (MiB)
use of full AX	use of Algorithm 1 and leverage score sampling
ogbl-ddi [29]	4.3K	1.3M	100	1.45	0.17
ogbn-arxiv [29]	169.3K	1.2M	128	330.75	160.45
Synthetic data (Gaussian)	50.0K	625.0M	500	240.43	0.17
Synthetic data (Gaussian)	100.0K	2.5B	500	431.82	0.52
Synthetic data (Gaussian)	150.0K	5.6B	500	865.25	19.39
$\mathbf{y}$  by setting the linear relationship between labels and features:  $\mathbf{y} := X\mathbf{w} + \mathbf{n}$ . Here  $\mathbf{n}$  denotes an additive Gaussian noise with parameters  $(\mu, \sigma) = (1, 10)$ . Based upon the size of the two datasets described below, we consider the data matrices  $X \in \mathbb{R}^{n \times d}$  (with  $n \in \{4267, 4039\}$  and  $d = 100$ ), where each row (i.e., each node's feature vector) is drawn according to the Cauchy distribution with parameters  $(x_0, \gamma) = (10, 100)$ , in an i.i.d. manner. Here  $x_0$  denotes the location parameter, indicating the location of the peak of the distribution, and  $\gamma$  denotes the scale parameter, specifying the half-width at half-maximum (HWHM). Also, we consider  $\mathbf{w} \in \mathbb{R}^d$  generated by the Cauchy distribution with  $(x_0, \gamma) = (10, 100)$ .
ogbl-ddi dataset [29]: This dataset represents a undirected and unweighted graph which models the drug-drug interactions. Each node corresponds to an FDA-approved or experimental drug. An edge indicates interactions between drugs that can be interpreted as the difference between the joint effect of taking the two drugs together and the expected effect when drugs act independently of each other. As there are neither feature vectors nor labels that are available, we synthetically generate those as mentioned above.
ego-Facebook dataset [30]: This dataset models a social network that represents friendship of "circles" between users from Facebook. Examples of such relationships between users include students of common universities, sports teams, relatives, etc. Notice that the graph adjacency matrix is more sparse compared to ogbl-ddi dataset. As above, since there exists no feature vectors and labels that are available, we synthetically generate those to perform regression task.
House dataset [31]: This dataset represents a graph in which nodes correspond to the properties; the existence of edges means that two properties are located close to each other; and the target label is the property's price. In addition, there exist node features representing MedInc, HouseAge, AveRooms, AveBedrms, Population, and AveOccup. In particular, we use the refined dataset from [33].
Furthermore, for the wall-clock time and peak memory usage comparisons, we additionally use ogblarxiv [29] dataset, representing the citation network between all Computer Science (CS) arXiv papers (although the dataset is tailored for classification of 40 subject areas) and synthetic dataset where all  $(A,X,\mathbf{y})$  are generated according to Gaussian distribution with size specified in Table 1, Table 2, and Table 3.
Additional results and analysis:
Wall-clock time with the adoption of sparse matrix multiplication: Notice that as compared to results in Table 1, Table 2 demonstrates that the computation time of both our approach and using exact  $AX$  become faster due to the adoption of sparse matrix multiplication and the reduction time was larger for using exact  $AX$  (especially for the ogbn-arxiv dataset which is sparse). However, we observe that our proposed scheme is faster than the regression by using the exact multiplication of  $AX$ . Furthermore, for the synthetic dataset where  $A$  and  $X$  are generated according to a Gaussian
distribution and hence  $A$  is dense, we observe that the sparse matrix multiplication does not help speed up the computation time and the matrix multiplication with scipysparse package took even longer than the matrix multiplication with numpy package.
Peak memory usage: In addition to run-time, we expect our proposed approach to offer significant improvements in terms of memory usage. As a simple experiment in terms of memory usage, we provide the peak memory usage comparison for the same datasets during the sampling and regression steps of the algorithm. As shown in Table 3, in the best case, the proposed algorithm requires  $1414 \times$  less memory than the algorithm using the exact computation of  $AX$ .
Computing architecture for wall-clock time and peak-memory usage comparisons: For a fair comparison on Table 1, Table 2, and Table 3, we use the same regression solver and use the same specification of 48 cores of an x86 64 processor with 503.74GB memory.