SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the first step is by choosing  $\mathbf{x} = \mathbf{x}^{\star}$ , the second step is by the guarantee of  $\ell_{\infty}$  subspace embedding, and the final step is by definition of  $\sigma_{1}(\mathbf{a}_{i^{\star}})$ . Combining Inequality (3.2) and Inequality (3.3) gives the claimed lower bound on  $\| \sigma_1^{\mathbf{S}_1\mathbf{A}}(\mathbf{S}_\infty \mathbf{A})\|_\infty$ . The runtime follows from the cost of computing  $\mathbf{S}_{\infty}\mathbf{A}$  from [75] and that of computing  $\widetilde{O}(d)$  of  $\ell_{1}$  sensitivities with respect to  $\mathbf{S}_1\mathbf{A}$ .
Figure 1: Average absolute log ratios for all  $\ell_p$  sensitivity approximations for wine, with the theoretical additive bound (dashed).
We demonstrate our fast sensitivity approximations on multiple real-world datasets in the UCI Machine Learning Dataset Repository [46], such as the wine and fires datasets, for different  $p$  and varying approximation parameters  $\alpha$ . We focus on the wine dataset here, with full details in Appendix E.
Experimental Setup. For each dataset and each  $p$ , we first apply the Lewis weight subspace embedding to compute a smaller matrix SA. Then, we run our sensitivity computation Algorithm 1 and measure the average and maximum absolute log ratio  $|\log (\sigma_{\mathrm{approx}} / \sigma_{\mathrm{true}})|$  to capture the relative error of the sensitivity approximation. Note that this is stricter than our guarantees, which give only an  $O(\alpha)$  additive error; therefore we have no real upper bound on the maximum absolute log ratio due to small sensitivities. We plot an upper bound of  $\log (\alpha^p)$ , which is the worst case additive error approximation although it does provide relative error guarantees with respect to the average sensitivity. We compare our fast algorithm for computing the total sensitivity with the naive brute-force method by approximating all sensitivities and then averaging.
Analysis. In practice, we found most real-world datasets have easy-to-approximate sensitivities with total  $\ell_p$  sensitivity about 2-5 times lower than the theoretical upper bound. Figure 1 shows that the average absolute log ratios for approximating the  $\ell_p$  sensitivities are much smaller than those suggested by the theoretical upper bound, even for large  $\alpha$ . Specifically, when  $\alpha = 40$ , we incur only a  $16\mathrm{x}$  accuracy deterioration in exchange for a  $40\mathrm{x}$  faster algorithm. This is significantly better than the worst-case accuracy guarantee which for  $p = 3$  would be  $\alpha^p = 40^3$ .
More importantly, we find that empirical total sensitivities are much lower than the theoretical upper bounds suggest, often by a factor of at least 5, especially for  $p > 2$ . This implies that real-world structure can be utilized for improved data compression and justifies the importance of sensitivity estimation for real-world datasets. Furthermore, our novel algorithm approximates the total sensitivity up to an overestimate within a factor of 1.3, in less than a quarter of the time of the brute-force algorithm. Our observation generalizes to other real-world datasets (see Appendix E).
p	Total Sensitivity Upper Bound	Brute-Force/True	Approximation	Brute-Force Runtime (s)	Approximate Runtime (s)
1	14	5.2	6.4	667	105
1.5	14	11.6	14.2	673	131
2.5	27.1	13.8	14.9	693	209
3	52.4	7.2	8.9	686	192
Table 1: Runtime comparison for computing total sensitivities for the wine dataset, which has matrix shape (177, 14).
We are very grateful to: Sagi Perel, Arun Jambulapati, and Kevin Tian for helpful discussions about related work; Taisuke Yasuda for his generous help discussing results in  $\ell_p$  sensitivity sampling; and our NeurIPS reviewers for their time, effort, and many constructive suggestions. Swati gratefully acknowledges funding for her student researcher position from Google Research and research
assistant
assistant
assistant
assistant
assistant
assistant
assistant
[75] David P Woodruff and Taisuke Yasuda. High-dimensional geometric streaming in polynomial space. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 732-743, 2022. 5, 9, 28
[76] David P Woodruff Cameron Musco, Christopher Musco, David P Woodruff, and Taisuke Yasuda. Active sampling for linear regression beyond the 12 norm. CoRR, 2021. 5
[77] Maryam Fazel, Yin Tat Lee, Swati Padmanabhan, and Aaron Sidford. Computing lewis weights to high precision. In Proceedings of the 2022 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), 2022. 5, 6, 7, 8
[78] David P Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1-2), 2014. 5
[79] Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. Journal of the ACM (JACM), 2007. 6
[80] Christian Sohler and David P Woodruff. Subspace embeddings for the  $\ell_1$ -norm with applications. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pages 755-764, 2011. 6
[81] David Woodruff and Qin Zhang. Subspace embeddings and  $\ell_p$ -regression using exponential random variables. In Conference on Learning Theory, pages 546-567. PMLR, 2013. 6
[82] Cameron Musco, Christopher Musco, David P Woodruff, and Taisuke Yasuda. Active linear regression for  $\ell_p$  norms and beyond. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), 2022. 7
[83] Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM), 68(1), 2021. 17
[84] Roman Vershynin. High-dimensional probability: An introduction with applications in data science. Cambridge university press, 2020. 17
A Omitted proofs: general technical results
Fact A.1 (Crude sensitivity approximation via leverage scores). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , let  $\sigma_{1}(\mathbf{a}_{i})$  denote the  $i^{\text{th}} \ell_{1}$  sensitivity of  $\mathbf{A}$  with respect to  $\mathbf{A}$ , and let  $\tau_{i}(\mathbf{A})$  denote its  $i^{\text{th}}$  leverage score with respect to  $\mathbf{A}$ . Then we have  $\sqrt{\frac{\tau_{i}(\mathbf{A})}{n}} \leq \sigma_{1}(\mathbf{a}_{i}) \leq \sqrt{\tau_{i}(\mathbf{A})}$ .
Proof. The proof of this claim follows from a simple application of standard norm inequalities, and we present one here for completeness. For any  $\mathbf{u} \in \mathbb{R}^n$ , we have  $\| \mathbf{u} \|_2 \leq \| \mathbf{u} \|_1 \leq \sqrt{n} \| \mathbf{u} \|_2$ . Therefore, for any  $\mathbf{x} \in \mathbb{R}^d$ , we have
$$
\frac {\left| \mathbf {x} ^ {\top} \mathbf {a} _ {i} \right|}{\sqrt {n} \| \mathbf {A x} \| _ {2}} \leq \frac {\left| \mathbf {x} ^ {\top} \mathbf {a} _ {i} \right|}{\| \mathbf {A x} \| _ {1}} \leq \frac {\left| \mathbf {x} ^ {\top} \mathbf {a} _ {i} \right|}{\| \mathbf {A x} \| _ {2}}. \tag {A.1}
$$
Let  $\mathbf{x}_{\tau}$  be the vector that realizes the  $i^{\mathrm{th}}$  leverage score, and  $\mathbf{x}_{\sigma}$  be the vector that realizes the  $i^{\mathrm{th}}\ell_{1}$  sensitivity (i.e. they are the maximizers of sensitivity). Then, we may conclude from Inequality (A.1) that
$$
\sqrt {\frac {| \mathbf {x} _ {\tau} ^ {\top} \mathbf {a} _ {i} | ^ {2}}{n \| \mathbf {A} \mathbf {x} _ {\tau} \| _ {2} ^ {2}}} \leq \frac {| \mathbf {x} _ {\tau} ^ {\top} \mathbf {a} _ {i} |}{\| \mathbf {A} \mathbf {x} _ {\tau} \| _ {1}} \leq \frac {| \mathbf {x} _ {\sigma} ^ {\top} \mathbf {a} _ {i} |}{\| \mathbf {A} \mathbf {x} _ {\sigma} \| _ {1}} \leq \sqrt {\frac {| \mathbf {x} _ {\sigma} ^ {\top} \mathbf {a} _ {i} | ^ {2}}{\| \mathbf {A} \mathbf {x} _ {\sigma} \| _ {2} ^ {2}}} \leq \sqrt {\frac {| \mathbf {x} _ {\tau} ^ {\top} \mathbf {a} _ {i} | ^ {2}}{\| \mathbf {A} \mathbf {x} _ {\tau} \| _ {2} ^ {2}}},
$$
which gives the claimed result.
Fact A.2. Given an  $n \times d$  matrix, the cost of computing  $k$  of its  $\ell_1$  sensitivities is  $\widetilde{O}(\mathbf{nnz}(\mathbf{A}) + k \cdot d^{\omega})$ .
Proof. Given an  $n \times d$  matrix  $\mathbf{A}$ , we first compute  $\mathbf{SA}$ , its  $\ell_1$  subspace embedding of size  $O(d) \times d$ , and compute the  $i^{\mathrm{th}}$  sensitivity of  $\mathbf{A}$  as follows.
$$
\boldsymbol {\sigma} _ {1} ^ {\mathbf {A}} (\mathbf {a} _ {i}) = \max _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} |}{\| \mathbf {A x} \| _ {1}} = \max _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} |}{\Theta (\| \mathbf {S A x} \| _ {1})} \approx_ {O (1)} \boldsymbol {\sigma} _ {1} ^ {\mathbf {S A}} (\mathbf {a} _ {i}),
$$
where the second step is by the definition of  $\ell_1$  subspace embedding (cf. Definition 2.3). To compute  $\sigma_{1}^{\mathbf{SA}}(\mathbf{a}_{i})$ , we need to compute  $\min_{\mathbf{a}_i^\top \mathbf{x} = 1} \| \mathbf{SAx}\|_1$ , which, by introducing a new variable for each of the rows of  $\mathbf{SAx}$ , can be transformed into a linear program with  $O(d)$  variables and  $d$  constraints. To see the claim on runtime, the cost of computing the subspace embedding is  $\mathbf{nnz}(\mathbf{A})$ . Having computed this once, we can then solve  $k$  linear programs, each at cost  $d^{\omega}$  (which is the current fastest LP solver [83]). Putting together these two costs gives the claimed runtime.
Lemma A.3. Given positive numbers  $a_1, a_2, \ldots, a_m$ , let  $r := \frac{\max_{i \in [m]} a_i}{\min_{i \in [m]} a_i}$  and  $A^{(true)} := \sum_{i \in [m]} a_i$ . Suppose we sample, uniformly at random with replacement, a set  $S \in [m]$  of these numbers, and let  $A^{(est)} := \frac{m}{|S|} \cdot \sum_{i: a_i \in S} a_i$ . Then, if  $|S| \geq 10\left(\frac{r(1 + \gamma)}{\gamma^2} \log \left(\frac{1}{\delta}\right)\right)$  for a large enough absolute constant  $C$ , we can ensure with at least a probability of  $1 - \delta$ , that  $|A^{(est)} - A^{(true)}| \leq \gamma A^{(true)}$ .
Proof. By construction, the expected value of a sample  $a_{i}$  in  $S$  is  $\frac{1}{m} A^{(\mathrm{true})}$ . Denoting by  $\mathcal{P}$  the uniform distribution over  $a_1, a_2, \ldots, a_m$ , and let  $\sigma(\mathcal{P})$  be the standard deviation of this distribution  $\mathcal{P}$ . Then, we can apply Bernstein's inequality [84, Theorem 2.8.1] to see that the absolute error  $|A^{(\mathrm{est})} - A^{(\mathrm{true})}|$  satisfies the following guarantee for all  $t > 0$ :
$$
\begin{array}{l} \Pr \left\{\left| A ^ {(\text {est})} - A ^ {(\text {true})} \right| \geq t \right\} = \Pr \left\{\left| \sum_ {i: a _ {i} \in S} \left(a _ {i} - \frac {1}{m} A ^ {(\text {true})}\right) \right| \geq t \cdot \frac {| S |}{m} \right\} \\ \leq e ^ {- \left\{\frac {\frac {1}{2} t ^ {2} \cdot \frac {| S | ^ {2}}{m ^ {2}}}{| S | \cdot \sigma (\mathcal {P}) ^ {2} + \frac {1}{3} t \cdot \frac {| S |}{m} \cdot \max  _ {i \in [ m ]} a _ {i}} \right\}} \\ = e ^ {- \left\{\frac {\frac {1}{2} t ^ {2} \cdot | S |}{m ^ {2} \cdot \sigma (\mathcal {P}) ^ {2} + \frac {1}{3} \cdot m \cdot t \cdot \max  _ {i \in [ m ]} a _ {i}} \right\}}. \tag {A.2} \\ \end{array}
$$
Since each of the  $a_{i}$  is positive, we note that
$$
\sigma (\mathcal {P}) ^ {2} = \frac {1}{m} \sum_ {i \in [ m ]} \left(a _ {i} - \frac {A ^ {\text {(t r u e)}}}{m}\right) ^ {2} \leq \frac {1}{m} \sum_ {i \in [ m ]} a _ {i} ^ {2} \leq \frac {1}{m} \cdot \max  _ {i \in [ m ]} a _ {i} \cdot \sum_ {i \in [ m ]} a _ {i} = \frac {1}{m} \cdot \max  _ {i \in [ m ]} a _ {i} \cdot A ^ {\text {(t r u e)}}. \tag {A.3}
$$
Combining Inequality (A.2) and Inequality (A.3) for  $t = \gamma A^{(\mathrm{true})}$  implies that
$$
\Pr \left\{\left| A ^ {(\text {e s t})} - A ^ {(\text {t r u e})} \right| \geq \gamma A ^ {(\text {t r u e})} \right\} \leq e ^ {- \left\{\frac {\frac {1}{2} \gamma^ {2} A ^ {(\text {t r u e})} | S |}{m (1 + \frac {1}{3} \cdot \gamma) \cdot \max  _ {i \in [ m ]} a _ {i}} \right\}} \leq \delta \tag {A.4}
$$
holds for the choice of  $|S| \geq 10\left(\frac{m}{\gamma^2}(1 + \gamma)\frac{\max_{i \in [m]} a_i}{A^{(\mathrm{true})}}\log\left(\frac{1}{\delta}\right)\right)$  yields the claimed error guarantee.
B Omitted proofs:  $\ell_1$  sensitivities
B.1 Estimating all  $\ell_1$  sensitivities
$$
\boldsymbol {\sigma} _ {1} (\mathbf {a} _ {i}) \leq \widetilde {\boldsymbol {\sigma}} _ {i} \leq O \left(\boldsymbol {\sigma} _ {1} (\mathbf {a} _ {i}) + \frac {\alpha}{n} \mathfrak {S} _ {1} (\mathbf {A})\right). \tag {3.1}
$$
Proof. Note that in Line 2 of Algorithm 1, the rows of  $\mathbf{A}$  are partitioned into randomly created  $n / \alpha$  blocks. Suppose  $\mathbf{a}_i$ , the  $i^{\mathrm{th}}$  row of  $\mathbf{A}$ , falls into the block  $\mathbf{B}_{\ell}$ . Recall, that in Line 5, the rows from  $\mathbf{B}_{\ell}$  are mapped to those in  $\mathbf{P}$  with row indices in the set  $J$ . Then, we compute the  $i^{\mathrm{th}}\ell_{1}$  sensitivity estimate as  $\widetilde{\pmb{\sigma}}_i = \max_{j\in J}\pmb{\sigma}_1^{\mathrm{SA}}(\mathbf{p}_j)$ . We observe that for all  $\mathbf{x} \in \mathbb{R}^d$ ,
$$
\left\| \mathbf {S} \mathbf {A} \mathbf {x} \right\| _ {1} = \Theta \left(\left\| \mathbf {A} \mathbf {x} \right\| _ {1}\right). \tag {B.1}
$$
We use Equation (B.1) below to establish the claimed bounds. Let  $\mathbf{x}^{\star} = \arg \max_{\mathbf{x}\in \mathbb{R}^{d}}\frac{|\mathbf{a}_{i}^{\top}\mathbf{x}|}{\|\mathbf{A}\mathbf{x}\|_{1}}$  be the vector that realizes the  $i^{\mathrm{th}}\ell_1$  sensitivity of  $\mathbf{A}$ . Further, let the  $j^{\mathrm{th}}$  row of  $\mathbf{P}$  be  $\mathbf{r}_k^{(\ell)}\mathbf{B}_\ell$ . Then, with a probability of at least  $1 / 2$ , we have
$$
\boldsymbol {\sigma} _ {1} ^ {\mathbf {S A}} \left(\mathbf {p} _ {j}\right) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} \right|}{\left\| \mathbf {S A x} \right\| _ {1}} \geq \frac {\left| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} ^ {\star} \right|}{\left\| \mathbf {S A x} ^ {\star} \right\| _ {1}} = \Theta (1) \frac {\left| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} ^ {\star} \right|}{\left\| \mathbf {A x} ^ {\star} \right\| _ {1}} \geq \Theta (1) \frac {\left| \mathbf {a} _ {i} ^ {\top} \mathbf {x} ^ {\star} \right|}{\left\| \mathbf {A x} ^ {\star} \right\| _ {1}} = \Theta (1) \boldsymbol {\sigma} _ {1} (\mathbf {a} _ {i}), \tag {B.2}
$$
where the first step is by definition of  $\sigma_{1}^{\mathbf{SA}}(\mathbf{p}_{j})$ , the second is by evaluating the function being maximized at a specific choice of  $\mathbf{x}$ , the third step is by applying  $\| \mathbf{SAx}\| _1 = \Theta (\| \mathbf{Ax}\| _1)$  from Equation (B.1), and the final step is by definition of  $\mathbf{x}^{\star}$  and  $\sigma_{1}(\mathbf{a}_{i})$ . For the fourth step, we use the fact that  $|\mathbf{r}_k^\ell \mathbf{B}_\ell \mathbf{x}^\star | = |\mathbf{r}_{k,i}^\ell \mathbf{a}_i^\top \mathbf{x}^\star +\sum_{j\neq i}\mathbf{r}_{k,j}^\ell (\mathbf{B}_\ell \mathbf{x}^\star)_j|\geq |\mathbf{a}_i^\top \mathbf{x}^\star |$  with a probability of at least  $1 / 2$  since the vector  $\mathbf{r}_k^\ell$  has coordinates that are  $+1$  or  $-1$  with equal probability. By a union bound over the  $|J|$  independent rows that block  $\mathbf{B}_{\ell}$  is mapped to, we establish the claimed lower bound in Inequality (B.2) with a probability of at least 0.9. To show an upper bound on  $\sigma_{1}^{\mathrm{SA}}(\mathbf{p}_{j})$ , we observe that
$$
\boldsymbol {\sigma} _ {1} ^ {\mathbf {S A}} \left(\mathbf {p} _ {j}\right) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} \right|}{\| \mathbf {S A x} \| _ {1}} \leq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\| \mathbf {B} _ {\ell} \mathbf {x} \| _ {1}}{\| \mathbf {S A x} \| _ {1}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\| \mathbf {B} _ {\ell} \mathbf {x} \| _ {1}}{\| \mathbf {A x} \| _ {1}} \leq \Theta (1) \sum_ {j: \mathbf {a} _ {j} \in \mathbf {B} _ {\ell}} \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left| \mathbf {a} _ {j} ^ {\top} \mathbf {x} \right|}{\| \mathbf {A x} \| _ {1}}, \tag {B.3}
$$
where the second step is by Hölder inequality and the fact that the entries of  $\mathbf{r}_k^{(\ell)}$  are all bounded between  $-1$  and  $+1$ , and the third step is by Equation (B.1). The final term of the above chain of inequalities may be rewritten as  $\Theta(1)\sum_{j:\mathbf{a}_j \in \mathbf{B}_{\ell}}s_j(\mathbf{A})$ , by the definition of  $\mathbf{B}_{\ell}$ . Because  $\mathbf{B}_{\ell}$  is a group of  $\alpha$  rows selected uniformly at random out of  $n$  rows and contains the row  $\mathbf{a}_i$ , we have:
$$
\mathbb{E}\left\{\sum_{\substack{j:\mathbf{a}_{j}\in \mathbf{B}_{\ell},\\ j\neq i}}\boldsymbol{\sigma}_{1}(\mathbf{a}_{j})\right\} = \frac{\alpha - 1}{n - 1}\sum_{j\neq i}\boldsymbol{\sigma}_{1}(\mathbf{a}_{j}).
$$
Therefore, Markov inequality gives us that with a probability of at least 0.9, we have
$$
\sigma_ {1} ^ {\mathbf {S A}} \left(\mathbf {p} _ {j}\right) \leq \sigma_ {1} \left(\mathbf {a} _ {j}\right) + \mathfrak {S} _ {1} (\mathbf {A}) O \left(\frac {\alpha}{n}\right). \tag {B.4}
$$
The median trick then establishes the bounds in Inequality (B.2) and Inequality (B.4) with high probability. The claimed runtime is obtained by the cost of constructing SA plus  $O(n / \alpha)$  computations of  $\ell_1$  sensitivities with respect to SA, for which we may use Fact 2.4.
B.2 Estimating the sum of  $\ell_1$  sensitivities
In this section, we present a randomized algorithm that approximates the total sensitivity  $\mathfrak{S}_1(\mathbf{A})$ , up to a  $\gamma$ -multiplicative approximation for some  $\gamma \in (0,1)$  that is not too small. This algorithm is significantly less general than Algorithm 2 (which holds for all  $p \geq 1$ ) but it uses extremely simple facts about leverage scores and what we think is a new recursion technique in this context, which we believe could be of potential independent interest.
Theorem B.1. Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and an approximation factor  $\gamma \in (0, 1)$  such that  $\gamma \geq \Omega\left(\frac{1}{n^3}\right)$ , there exists an algorithm, which in time  $\widetilde{O}\left(\mathbf{nnz}(\mathbf{A}) + \frac{d^\omega}{\gamma^2} \cdot \max\left(\sqrt{d}, \frac{1}{\gamma^2}\right)\right)$ , returns a positive scalar  $\widehat{s}$  such that, with a probability of 0.99, we have
$$
\mathfrak {S} _ {1} (\mathbf {A}) \leq \widehat {s} \leq (1 + O (\gamma)) \mathfrak {S} _ {1} (\mathbf {A}).
$$
Algorithm 4 Approximating the Sum of  $\ell_1$ -Sensitivities
Input: Matrix  $\mathbf{A}\in \mathbb{R}^{n\times d}$  and approximation factor  $\gamma \in (0,1)$
Output: A positive scalar that satisfies, with probability 0.99, that
$$
\mathfrak {S} _ {1} (\mathbf {A}) \leq \widehat {s} \leq (1 + O (\gamma)) \mathfrak {S} _ {1} (\mathbf {A})
$$
1: Compute  $\mathbf{SA}\in \mathbb{R}^{O(d)\times d}$ , a  $1 / 2$ -approximate  $\ell_1$  subspace embedding of  $\mathbf{A}$  (cf. Definition 2.3)
2: Set  $\rho = O(\gamma / \log (\log (n + d)))$ . Compute  $\mathbf{S}'\mathbf{A}$ , a  $\rho$ -approximate  $\ell_1$  subspace embedding of  $\mathbf{A}$
3: Construct a matrix  $\widehat{\mathbf{A}}$  comprising only those rows of  $\mathbf{A}$  with leverage scores at least  $1/n^{10}$
4: Let  $s$  be the output of Algorithm 5 with inputs  $\widehat{\mathbf{A}}$ ,  $\mathbf{SA}$ ,  $\mathbf{S}'\mathbf{A}$ ,  $n$ , and  $\rho$
5: Return  $\widehat{s} := (1 + \gamma)\left(s + \frac{1}{n^5}(|\mathbf{A}| - |\widehat{\mathbf{A}}|)\right)$
B.2.1 Proof sketch.
We achieve our guarantee via Algorithm 4 and Algorithm 5, which are based on the following key principles. The first is Fact 2.2: the  $i^{\mathrm{th}}$  sensitivity and  $i^{\mathrm{th}}$  leverage score of  $\mathbf{A} \in \mathbb{R}^{n \times d}$  satisfy  $\sqrt{\frac{\tau_i(\mathbf{A})}{n}} \leq \sigma_1(\mathbf{a}_i) \leq \sqrt{\tau_i(\mathbf{A})}$ . The second idea, derived from Bernstein's inequality (and formalized in Lemma A.3), is: the sum of a set of positive numbers with values bounded between  $a$  and  $b$  can be approximated to a  $\gamma$ -multiplicative factor using  $\widetilde{O}((b / a)\gamma^{-2})$  uniformly sampled (and appropriately scaled) samples. The third principle is that  $\sigma_1(\mathbf{a}_i) \approx \sigma_1^{\mathbf{SA}}(\mathbf{a}_i)$  when  $\mathbf{SA}$  is a constant-approximation  $\ell_1$  subspace embedding of  $\mathbf{A}$ .
Based on these ideas, we first divide the rows of  $\mathbf{A}$  into  $\mathcal{B} = \Theta (\log n)$  submatrices based on which of the  $\mathcal{B}$  buckets [1/2, 1], [1/4, 1/2], [1/8, 1/4], etc. their leverage scores (computed with respect to  $\mathbf{A}$ ) fall into. Since each of these  $\mathcal{B}$  submatrices comprises rows with leverage scores in the range  $[a, 2a]$  for some  $a$ , per Fact 2.2, we have  $\sqrt{\frac{a}{n}} \leq \sigma_1(\mathbf{a}_i) \leq \sqrt{2a}$ . Therefore, by Lemma A.3, the total sensitivity of this submatrix may be approximated by the appropriately scaled total sensitivity of  $O(\sqrt{n})$  of its uniformly sampled rows. Our design choice to split rows based on leverage scores is based on the fact that leverage scores (unlike sensitivities) are efficiently computable (using [60]). Computing the total sensitivity of an  $\widetilde{O}(\sqrt{n})$ -sized matrix thus costs  $\widetilde{O}(\mathbf{nnz}(\mathbf{A}) + \sqrt{n} \cdot d^{\omega})$  under this scheme.
Applying the idea sketched above recursively further reduces the cost. Suppose, after dividing the input matrix  $\mathbf{A}$  into  $\mathcal{B}$  submatrices  $\mathbf{M}_i$  and subsampling  $\widetilde{\mathbf{M}}_i$  from these matrices, we try to recurse on each of these  $\mathcal{B}$  submatrices  $\widetilde{\mathbf{M}}_i$ . A key requirement for this idea to work is that the sum of the  $\ell_1$  sensitivities of the rows of  $\widetilde{\mathbf{M}}_i$  in isolation be close enough to the true sum of  $\ell_1$  sensitivities of