SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Computing Approximate  $\ell_p$  Sensitivities
Swati Padmanabhan
pswt@mit.edu
David P. Woodruff
dwoodruf@cs.cmu.edu
Qiuyi (Richard) Zhang
qiuyiz@google.com
Recent works in dimensionality reduction for regression tasks have introduced the notion of sensitivity, an estimate of the importance of a specific datapoint in a dataset, offering provable guarantees on the quality of the approximation after removing low-sensitivity datapoints via subsampling. However, fast algorithms for approximating sensitivities, which we show is equivalent to approximate regression, are known for only the  $\ell_2$  setting, in which they are popularly termed leverage scores. In this work, we provide the first efficient algorithms for approximating  $\ell_p$  sensitivities and related summary statistics of a given matrix. In particular, for a given  $n\times d$  matrix, we compute  $\alpha$ -approximation to its  $\ell_1$  sensitivities at the cost of  $O(n / \alpha)$  sensitivity computations. For estimating the total  $\ell_p$  sensitivity (i.e. the sum of  $\ell_p$  sensitivities), we provide an algorithm based on importance sampling of  $\ell_p$  Lewis weights, which computes a constant factor approximation to the total sensitivity at the cost of roughly  $O(\sqrt{d})$  sensitivity computations. Furthermore, we estimate the maximum  $\ell_1$  sensitivity, up to a  $\sqrt{d}$  factor, using  $O(d)$  sensitivity computations. We generalize all these results to  $\ell_p$  norms for  $p > 1$ . Lastly, we experimentally show that for a wide class of matrices in real-world datasets, the total sensitivity can be quickly approximated and is significantly smaller than the theoretical prediction, demonstrating that real-world datasets have low intrinsic effective dimensionality.
Many modern large-scale machine learning datasets comprise tall matrices  $\mathbf{A} \in \mathbb{R}^{n \times d}$  with  $d$  features and  $n \gg d$  training examples, often making it computationally expensive to run them through even basic algorithmic primitives. Therefore, many applications spanning dimension reduction, privacy, and fairness aim to estimate the importance of a given datapoint as a first step. Such importance estimates afford us a principled approach to focus our attention on a subset of only the most important examples.
In view of this benefit, several sampling approaches have been extensively developed in the machine learning literature. One of the simplest such methods prevalent in practice is uniform sampling. Prominent examples of this technique include variants of stochastic gradient descent methods [1-4] such as [5-14] developed for the "finite-sum minimization" problem
$$
\operatorname {m i n i m i z e} _ {\boldsymbol {\theta} \in \mathcal {X}} \quad \sum_ {i = 1} ^ {n} f _ {i} (\boldsymbol {\theta}). \tag {1.1}
$$
where, in the context of empirical risk minimization, each  $f_{i}:\mathcal{X}\mapsto \mathbb{R}_{\geq 0}$  in (1.1) measures the loss incurred by the  $i$ -th data point from the training set, and in generalized linear models, each  $f_{i}$  represents a link function applied to a linear predictor evaluated at the  $i$ -th data point. The aforementioned algorithms randomly sample a function  $f_{i}$  and make progress via gradient estimation techniques.
However, uniform sampling can lead to significant information loss when the number of important training examples is relatively small. Hence, importance sampling methods that assign higher sampling probabilities to more important examples have emerged both in theory [15, 16] and practice [17-21]. One such technique is based on sensitivity scores, defined [15] for each coordinate  $i \in [n]$  as:
$$
\boldsymbol {\sigma} _ {i} \stackrel {\text {d e f}} {=} \max  _ {\mathbf {x} \in \mathcal {X}} \frac {f _ {i} (\mathbf {x})}{\sum_ {j = 1} ^ {n} f _ {j} (\mathbf {x})}. \tag {1.2}
$$
Sampling each  $f_{i}$  independently with probability  $p_i \propto \sigma_i$  and scaling the sampled row with  $1 / p_i$  preserves the objective in (1.1) in expectation for every  $\mathbf{x} \in \mathcal{X}$ . Further, one gets a  $(1 \pm \varepsilon)$ -approximation to this objective by sampling  $\widetilde{O}(\mathfrak{S}d\varepsilon^{-2})$  functions [22, Theorem 1.1], where  $\mathfrak{S} = \sum_{i=1}^{n} \sigma_i$  is called the total sensitivity, and  $d$  is an associated VC dimension.
Numerous advantages of sensitivity sampling over  $\ell_p$  Lewis weight sampling [23] were highlighted in the recent work [24], prominent among which is that a sampling matrix built using the  $\ell_p$  sensitivities of  $\mathbf{A}$  is significantly smaller than one built using its  $\ell_p$  Lewis weights in many important cases, e.g., when the total sensitivity  $\mathfrak{S}$  is small for  $p > 2$  (which is seen in many structured matrices, such as combinations of Vandermonde, sparse, and low-rank matrices as studied in [25] as well as those in many naturally occurring datasets (cf. Section 4)). Additionally, sensitivity sampling has found use in constructing  $\epsilon$ -approximators for shape-fitting functions, also called strong coresets [15, 16, 26-28], clustering [16, 29, 30], logistic regression [31, 32], and least squares regression [33, 34].
Despite their benefits, the computational burden of estimating these sensitivities has been known to be significant, requiring solving an expensive maximization problem for each datapoint, which is a limiting factor in their use [35]. A key exception is for  $f_{i}(\mathbf{x}) = (\mathbf{a}_{i}^{\top}\mathbf{x})^{2}$ , when the sensitivity score of  $f_{i}$  is exactly the leverage score of  $\mathbf{a}_{i}$ , which can be computed quickly in  $O(\log (n))$  sensitivity calculations, as opposed to the naive  $O(n)$  sensitivity calculations (cf. Section 2).
In this work, we initiate a systematic study of algorithms for approximately computing the  $\ell_p$  sensitivities of a matrix, generally up to a constant factor. We first define  $\ell_p$  sensitivities:
Definition 1.1. [15] Given a full-rank matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  with  $n \geq d$  and a scalar  $p \in (0, \infty)$ , let  $\mathbf{a}_i$  denote the  $i$ 'th row of matrix  $\mathbf{A}$ . Then the vector of  $\ell_p$  sensitivities of  $\mathbf{A}$  is defined as<sup>1</sup> (cf. Section 2 for the notation)
$$
\boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i}) \stackrel {\text {d e f}} {=} \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {A x} \neq \mathbf {0}} \frac {\left| \mathbf {a} _ {i} ^ {\top} \mathbf {x} \right| ^ {p}}{\left\| \mathbf {A x} \right\| _ {p} ^ {p}}, \text {f o r a l l} i \in [ n ], \tag {1.3}
$$
and the total  $\ell_p$  sensitivity is defined as  $\mathfrak{S}_p(\mathbf{A})\stackrel {\mathrm{def}}{=}\sum_{i\in [n]}\pmb{\sigma}_p(\mathbf{a}_i)$ .
These  $\ell_p$  sensitivities yield sampling probabilities for  $\ell_p$  regression, a canonical optimization problem that captures least squares regression, maximum flow, and linear programming, in addition to appearing in applications like low rank matrix approximation [36], sparse recovery [37], graph-based semi-supervised learning [38-40], and data clustering [41]. However, algorithms for approximating  $\ell_p$  sensitivities were known for only  $p = 2$  (when they are called "leverage scores"). As is typical in large-scale machine learning, we assume  $n \gg d$ . We now state our contributions, which are three-fold.
(1) Approximation algorithms. We design algorithms for three prototypical computational tasks associated with  $\ell_p$  sensitivities of a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ . Specifically, for an approximation parameter  $\alpha > 1$ , we can simultaneously estimate all  $\ell_p$  sensitivities up to an additive error of  $O\left(\frac{\alpha^p}{n} \mathfrak{S}_p(\mathbf{A})\right)$  via  $O(n / \alpha)$  individual sensitivity calculations, reducing our runtime by sacrificing some accuracy. We limit most of our results in the main text to  $\ell_1$  sensitivities, but extend these to all  $p \geq 1$  in Appendix C. To state our results for the  $\ell_p$  setting, we use  $\mathbf{nnz}(\mathbf{A})$  to denote the number of non-zero entries of the matrix  $\mathbf{A}$ ,  $\omega$  for the matrix multiplication constant, and we introduce the following piece of notation.
Definition 1.2 (Notation for  $\ell_p$  Results). We introduce the notation  $\mathrm{LP}(m,d,p)$  to denote the cost of approximating one  $\ell_p$  sensitivity of an  $m\times d$  matrix up to an accuracy of a given constant factor.
<sup>1</sup>From hereon, for notational conciseness, we omit stating  $\mathbf{A}\mathbf{x} \neq \mathbf{0}$  in the problem constraint.
Theorem 1.3 (Estimating all sensitivities; informal version of Theorem 3.1 and Theorem C.2). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and  $1 \leq \alpha \ll n$ , there exists an algorithm (Algorithm 1) which returns a vector  $\widetilde{\sigma} \in \mathbb{R}_{\geq 0}^n$  such that, with high probability, for each  $i \in [n]$ , we have
$$
\boldsymbol {\sigma} _ {1} (\mathbf {a} _ {i}) \leq \widetilde {\boldsymbol {\sigma}} _ {i} \leq O (\boldsymbol {\sigma} _ {1} (\mathbf {a} _ {i}) + \frac {\alpha}{n} \mathfrak {S} _ {1} (\mathbf {A})).
$$
Our algorithm's runtime is  $\widetilde{O}\left(\mathbf{nnz}(\mathbf{A}) + \frac{n}{\alpha} \cdot d^{\omega}\right)$ . More generally, for  $p \geq 1$ , we can compute (via Algorithm 6) an estimate  $\widetilde{\sigma} \in \mathbb{R}_{\geq 0}^{n}$  satisfying  $\sigma_{p}(\mathbf{a}_{i}) \leq \widetilde{\sigma}_{i} \leq O(\alpha^{p - 1}\sigma_{p}(\mathbf{a}_{i}) + \frac{\alpha^{p}}{n}\mathfrak{S}_{p}(\mathbf{A}))$  with high probability for each  $i \in [n]$ , at a cost of  $O\left(\mathbf{nnz}(\mathbf{A}) + \frac{n}{\alpha} \cdot \mathsf{LP}(O(d^{\max(1,p/2)}), d, p)\right)$ .
Two closely related properties arising widely in algorithms research are the total sensitivity  $\mathfrak{S}_1(\mathbf{A})$  and the maximum sensitivity  $\| \pmb{\sigma}_{1}(\mathbf{A})\|_{\infty}$ . As stated earlier, one important feature of the total sensitivity is that it determines the total sample size needed for function approximation, thus making its efficient estimation crucial for fast  $\ell_1$  regression. Additionally, it was shown in [24] that the sample complexity of sensitivity sampling is much lower than that of Lewis weight sampling in many practical applications such as when the total sensitivity is small; therefore, an approximation of the total sensitivity can be used as a fast test for whether or not to proceed with sensitivity sampling (involving the costly task of calculating all sensitivities).
While the total sensitivity helps us characterize the whole dataset (as explained above), the maximum sensitivity captures the importance of the most important datapoint and is used in, e.g., experiment design and in reweighting matrices for low coherence [42]. Additionally, it captures the maximum extent to which a datapoint can influence the objective function and is therefore used in differential privacy [43]. While one could naively estimate the total and maximum sensitivities by computing all  $n$  sensitivities using Theorem 1.3, we give faster algorithms that, strikingly, have no polynomial dependence on  $n$ . In the assumed regime of  $n \gg d$ , this is a significant runtime improvement.
Theorem 1.4 (Estimating the total sensitivity; informal version of Theorem 3.5). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , a scalar  $p \geq 1$ , and a small constant  $\gamma < 1$ , there exists an algorithm (Algorithm 2), which returns a positive scalar  $\widehat{s}$  such that, with a probability of at least 0.99, we have
$$
\mathfrak {S} _ {p} (\mathbf {A}) \leq \widehat {s} \leq (1 + O (\gamma)) \mathfrak {S} _ {p} (\mathbf {A}).
$$
Our algorithm's runtime is  $\widetilde{O}\left(\mathbf{nnz}(\mathbf{A}) + \frac{1}{\gamma^2} \cdot d^{|p/2-1|} \mathsf{LP}(O(d^{\max(1,p/2)}, d, p)\right)$ .
In Algorithm 4 (presented in Appendix B.2), we give an alternate method to estimate  $\mathfrak{S}_1(\mathbf{A})$  based on recursive leverage score sampling, without  $\ell_p$  Lewis weight calculations, which we believe to be of independent interest. All of our algorithms utilize approximation properties of  $\ell_p$  Lewis weights and subspace embeddings and recent developments in fast computation of these quantities.
Theorem 1.5 (Estimating the maximum sensitivity; informal version of Theorem 3.7 and Theorem C.3). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , there exists an algorithm (Algorithm 3) which returns a scalar  $\widetilde{\sigma} > 0$  such that
$$
\left\| \boldsymbol {\sigma} _ {1} (\mathbf {A}) \right\| _ {\infty} \leq \tilde {\boldsymbol {\sigma}} \leq O (\sqrt {d} \| \boldsymbol {\sigma} _ {1} (\mathbf {A}) \| _ {\infty}).
$$
Our algorithm's runtime is  $\widetilde{O}(\mathbf{nnz}(\mathbf{A}) + d^{\omega + 1})$ . Furthermore, via Algorithm 7, this result holds for all  $p \in [1,2]$  and for  $p > 2$ , this guarantee costs  $\widetilde{O}\left(\mathbf{nnz}(\mathbf{A}) + d^{p/2} \cdot \mathsf{LP}(O(d^{p/2}), d, p)\right)$ .
(2) Hardness results. In the other direction, while it is known that  $\ell_p$  sensitivity calculation reduces to  $\ell_p$  regression, we show the converse. Specifically, we establish hardness of computing  $\ell_p$  sensitivities in terms of the cost of solving multiple corresponding  $\ell_p$  regression problems, by showing that a  $(1 + \varepsilon)$ -approximation to  $\ell_p$ -sensitivities solves  $\ell_p$  multi-regression up to an accuracy factor of  $1 + \varepsilon$ .
Theorem 1.6 (Leave-One-Out  $\ell_p$  Multiregression Reduces to  $\ell_p$  Sensitivities; informal²). Suppose that we are given an sensitivity approximation algorithm  $\mathcal{A}$ , which for any matrix  $\mathbf{A}' \in \mathbb{R}^{n' \times d'}$  and accuracy parameter  $\varepsilon' \in (0,1)$ , computes  $(1 \pm \varepsilon')\sigma_p(\mathbf{A}')$  in time  $\mathcal{T}(n',d',\mathbf{nnz}(\mathbf{A}',\varepsilon'))$ . Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  with  $n \geq d$ , let  $OPT_i := \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}_{:-i}\mathbf{y} + \mathbf{A}_{:i}\|_p^p$  and  $\mathbf{y}_i^\star := \arg \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}_{:-i}\mathbf{y} + \mathbf{A}_{:i}\|_p$  for all the  $i \in [d]$ . Then, there exists an algorithm that takes  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and computes  $(1 \pm \varepsilon)OPT_i$  for all  $i$  in time  $\mathcal{T}(n+d,d,\mathbf{nnz}(\mathbf{A}),\varepsilon)$ .
2Please see Lemma D.2 for the formal version with a small additive error.
For  $p \neq 2$ , approximating all  $\ell_p$  sensitivities to a high accuracy would therefore require improving  $\ell_p$  multi-regression algorithms with  $\Omega(d)$  instances, for which the current best high-accuracy algorithms take poly(d) nnnz(A) time [44, 45]. More concretely, our hardness results imply that in general one cannot compute all sensitivities as quickly as leverage scores unless there is a major breakthrough in multi-response regression. In order to show this, we introduce a reduction algorithm that efficiently regresses columns of A against linear combinations of other columns of A, by simply adding a row to A capturing the desired linear combination and then computing sensitivities, which will reveal the cost of the corresponding regression problem. We can solve  $\Theta(d)$  of these problems by augmenting the matrix to increase the rows and columns by at most a constant factor. We defer these details to Appendix D.
(3) Numerical experiments. We consolidate our theoretical contributions with a demonstration of the empirical advantage of our approximation algorithms over the naive approach of estimating these sensitivities using the UCI Machine Learning Dataset Repository [46] in Section 4. We found that many datasets have extremely small total sensitivities; therefore, fast sensitivity approximation algorithms utilize this small intrinsic dimensionality of real-world data far better than Lewis weights sampling, with sampling bounds often a factor of  $2 - 5x$  better. We also found these sensitivities to be easy to approximate, with the accuracy-routine tradeoff much better than our theory suggests, with up to  $40x$  faster in runtime. Lastly, we show that our algorithm for estimating the total sensitivity produces accurate estimates, up to small constant factors, with a runtime speedup of at least  $4x$ .
Introduced in the pioneering work of [15], sensitivity sampling falls under the broad framework of "importance sampling". It has found use in constructing  $\epsilon$ -approximators of numerical integrands of several function classes in numerical analysis, statistics (e.g., in the context of VC dimension), and computer science (e.g., in coresets for clustering). The results from [15] were refined and extended by [16], with more advances in the context of shape-fitting problems in [26-28], clustering [16, 29, 30], logistic regression [31, 32], least squares regression [33, 34, 47], principal component analysis [48], reinforcement learning [49, 50], and pruning of deep neural networks [51-53]. Sampling algorithms for  $\ell_p$  subspace embeddings have also been extensively studied in functional analysis [54-59] as well as in theoretical computer science [23, 60, 44, 45].
Another notable line of work in importance sampling uses Lewis weights to determine sampling probabilities. As we explain in Section 2, Lewis weights are closely related to sensitivities. Many modern applications of Lewis weights in theoretical computer science we introduced in [23], which gave input sparsity time algorithms for approximating Lewis weights and used them to obtain fast algorithms for solving  $\ell_p$  regression. They have subsequently been used in row sampling algorithms for data pre-processing [33, 61, 62, 60, 23], computing dimension-free strong coresets for  $k$ -median and subspace approximation [63], and fast tensor factorization in the streaming model [64]. Lewis weights are also used for  $\ell_1$  regression in: [65] for stochastic gradient descent pre-conditioning, [66] for quantile regression, and [67] to provide algorithms for linear algebraic problems in the sliding window model. Lewis weights have also become widely used in convex geometry [68], randomized numerical linear algebra [23, 42, 69, 64, 70], and machine learning [71-73].
We use boldface uppercase and lowercase letters for matrices and vectors respectively. We denote the  $i^{\mathrm{th}}$  row vector of a matrix  $\mathbf{A}$  by  $\mathbf{a}_i$ . Given a matrix  $\mathbf{M}$ , we use  $\operatorname{Tr}(\mathbf{M})$  for its trace,  $\operatorname{rank}(\mathbf{M})$  for its rank,  $\mathbf{M}^\dagger$  for its Moore-Penrose pseudoinverse, and  $|\mathbf{M}|$  for the number of its rows. When  $x$  is an integer, we use  $[x]$  for the set of integers  $1, 2, \ldots, x$ . For two positive scalars  $x$  and  $y$  and a scalar  $\alpha \in (0,1)$ , we use  $x \approx_{\alpha} y$  to denote  $(1 - \alpha)y \leq x \leq (1 + \alpha)y$ ; we sometimes use  $x \approx y$  to mean  $(1 - c)y \leq x \leq (1 + c)y$  for some appropriate universal constant  $c$ . We use  $\widetilde{O}$  to hide dependence on  $n^{o(1)}$ . We acknowledge that there is some notation overloading between  $p$  (when used to denote the scalar in, for example,  $\ell_p$  norms) and  $p_i$  (when used to denote some probabilities) but the distinction is clear from context. We defer the proofs of facts stated in this section to Appendix A.
Notation for sensitivities. We start with a slightly general definition of sensitivities:  $\pmb{\sigma}_p^{\mathbf{B}}(\mathbf{a}_i)$  to denote the  $\ell_p$  sensitivity of  $\mathbf{a}_i$  with respect to  $\mathbf{B}$ ; when computing the sensitivity with respect to the
same matrix that the row is drawn from, we omit the matrix superscript:
$$
\boldsymbol {\sigma} _ {p} ^ {\mathbf {B}} (\mathbf {a} _ {i}) := \max  _ {\mathbf {x}} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} | ^ {p}}{\| \mathbf {B x} \| _ {p} ^ {p}} \text {a n d} \boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i}) := \max  _ {\mathbf {x}} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} | ^ {p}}{\| \mathbf {A x} \| _ {p} ^ {p}}. \tag {2.1}
$$
When referring to the vector of  $\ell_p$  sensitivities of all the rows of a matrix, we omit the subscript in the argument: so,  $\sigma_p^{\mathbf{B}}(\mathbf{A})$  is the vector of  $\ell_p$  sensitivities of rows of the matrix  $\mathbf{A}$ , each computed with respect to the matrix  $\mathbf{B}$  (as in Equation (2.1)), and analogously,  $\sigma_p(\mathbf{A})$  is the vector of  $\ell_p$  sensitivities of the rows of matrix  $\mathbf{A}$ , each computed with respect to itself. Similarly, the total sensitivity is the sum of  $\ell_p$  sensitivities of the rows of  $\mathbf{A}$  with respect to matrix  $\mathbf{B}$ , for which we use the following notation:
$$
\mathfrak {S} _ {p} ^ {\mathbf {B}} (\mathbf {A}) := \sum_ {i \in [ | \mathbf {A} | ]} \sigma_ {p} ^ {\mathbf {B}} \left(\mathbf {a} _ {i}\right) \text {a n d} \mathfrak {S} _ {p} (\mathbf {A}) := \sum_ {i \in [ | \mathbf {A} | ]} \sigma_ {p} \left(\mathbf {a} _ {i}\right), \tag {2.2}
$$
where, analogous to the rest of the notation regarding sensitivities, we omit the superscript when the sensitivities are being computed with respect to the input matrix itself.
While  $\sigma_{p}^{\mathbf{B}}(\mathbf{a}_{i})$  could be infinite, by appending  $\mathbf{a}_{i}$  to  $\mathbf{B}$ , the generalized sensitivity becomes once again contained in [0, 1]. Specifically, a simple rearrangement of the definition gives us the identity  $\sigma_{p}^{\mathbf{B} \cup \mathbf{a}_{i}}(\mathbf{a}_{i}) = 1 / (1 + 1 / \sigma_{p}^{\mathbf{B}}(\mathbf{a}_{i}))$ . We now define leverage scores, which are a special type of sensitivities and also amenable to efficient computation [60], which makes them an ideal "building block" in algorithms for approximating sensitivities.
Leverage scores are  $\ell_2$  sensitivities. The leverage score of the  $i^{\mathrm{th}}$  row  $\mathbf{a}_i$  of  $\mathbf{A}$  is  $\tau_i(\mathbf{A})\stackrel {\mathrm{def}}{=}\mathbf{a}_i^\top (\mathbf{A}^\top \mathbf{A})^\dagger \mathbf{a}_i$ . The  $i^{\mathrm{th}}$  leverage score is also the  $i^{\mathrm{th}}$  diagonal entry of the orthogonal projection matrix  $\mathbf{A}(\mathbf{A}^\top \mathbf{A})^\dagger \mathbf{A}^\top$ . Since the eigenvalues of an orthogonal projection matrix are zero or one, it implies  $\tau_{i}(\mathbf{A}) = \mathbf{1}_{i}^{\top}\mathbf{A}(\mathbf{A}^{\top}\mathbf{A})^{\dagger}\mathbf{A}^{\top}\mathbf{1}_{i}\leq 1$  for all  $i\in [n]$  and  $\sum_{i = 1}^{n}\tau_{i}(\mathbf{A})\leq d$  (see [74]).
It turns out that  $\tau_{i}(\mathbf{A}) = \sigma_{2}(\mathbf{a}_{i})$  (see [75]). This may be verified by applying a change of basis to the variable  $\mathbf{y} = \mathbf{A}\mathbf{x}$  in the definition of sensitivity and working out that the maximizer is the vector parallel to  $\mathbf{y} = \mathbf{A}^{\dagger}\mathbf{a}_{i}$ . This also gives an explicit formula for the total  $\ell_2$  sensitivity:  $\mathfrak{S}_2(\mathbf{A}) = \mathrm{rank}(\mathbf{A})$ .
The fastest algorithm for constant factor approximation to leverage scores is by [60], as stated next.
Fact 2.1 ([60, Lemma 7]). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , we can compute constant-factor approximations to all its leverage scores in time  $O(\mathbf{nnz}(\mathbf{A})\log (n) + d^{\omega}\log (d))$ .
In this paper, the above is the result we use to compute leverage scores (to a constant accuracy). Indeed, since the error accumulates multiplicatively across different steps of our algorithms, a high-accuracy algorithm for leverage score computation does not serve any benefit over this constant-accuracy one. Leverage scores approximate  $\ell_1$  sensitivities up to a distortion factor of  $\sqrt{n}$ , as seen from the below known fact we prove, for completeness, in Appendix A.