SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fact 2.2 (Crude sensitivity approximation via leverage scores). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , let  $\sigma_{1}(\mathbf{a}_{i})$  denote the  $i^{\text{th}} \ell_{1}$  sensitivity of  $\mathbf{A}$  with respect to  $\mathbf{A}$ , and let  $\tau_{i}(\mathbf{A})$  denote its  $i^{\text{th}}$  leverage score with respect to  $\mathbf{A}$ . Then we have  $\sqrt{\frac{\tau_{i}(\mathbf{A})}{n}} \leq \sigma_{1}(\mathbf{a}_{i}) \leq \sqrt{\tau_{i}(\mathbf{A})}$ .
In general,  $\ell_p$  sensitivities of rows of a matrix suffer a distortion factor of at most  $n^{|1/2-1/p|}$  from corresponding leverage scores, which follows from Hölder's inequality. The  $\ell_p$  generalizations of leverages scores are called  $\ell_p$  Lewis weights [54, 23], which satisfy the recurrence  $\mathbf{w}_i = \tau_i(\mathbf{W}^{1/2-1/p}\mathbf{A})$ . However, unlike when  $p = 2$ ,  $\ell_p$  Lewis weights are not equal to sensitivities, and from an approximation perspective, they provide only a one-sided bound on the sensitivities  $\sigma_p^{\mathbf{A}}(\mathbf{a}_i) \leq d^{\max(0,p/2-1)}\mathbf{w}_p^{\mathbf{A}}(\mathbf{a}_i)$  [75, 76]. While we have algorithms for efficiently computing leverage scores [60] and Lewis weights [23, 77], extensions for utilizing these ideas for fast and accurate sensitivity approximation algorithms for all  $p$  have been limited, which is the gap this paper aims to fill. A key regime of interest for many downstream algorithms is a small constant factor approximation to true sensitivities, such as for subsampling [78].
Subspace embeddings. We use sampling-based constant-approximation  $\ell_p$  subspace embeddings that we denote by  $\mathbf{S}_p$ . When the type of embedding is clear from context, we omit the subscript.
Definition 2.3 ( $\ell_p$  Subspace Embedding). Given  $\mathbf{A} \in \mathbb{R}^{n \times d}$  (typically  $n \gg d$ ), we call  $\mathbf{S} \in \mathbb{R}^{r \times n}$  (typically  $d \leq r \ll n$ ) an  $\epsilon$ -approximate  $\ell_p$  subspace embedding of  $\mathbf{A}$  if  $\| \mathbf{S} \mathbf{A} \mathbf{x} \|_p \approx_{\varepsilon} \| \mathbf{A} \mathbf{x} \|_p \forall \mathbf{x} \in \mathbb{R}^d$ .
When the diagonal matrix  $\mathbf{S}$  is defined as  $\mathbf{S}_{ii} = 1 / \sqrt{p_i}$  with probability  $p_i = \Omega (\tau_i(\mathbf{A}))$  (and 0 otherwise), then with high probability  $\mathbf{SA}$  is an  $\ell_2$  subspace embedding for  $\mathbf{A}$  [33, 79]; further,  $\mathbf{S}$  has at most  $\widetilde{O}(d\varepsilon^{-2})$  non-zero entries. For general  $p$ , we use Lewis weights to construct  $\ell_p$  subspace embeddings fast [23, Theorem 7.1]. This fast construction has been made possible by the reduction of Lewis weight computation to a few (polynomial in  $p$ ) leverage score computations for  $p < 4$  [23] as well as  $p \geq 4$  [77]. This efficient construction is the reason for our choice of this subspace embedding. We note that there has been an extensive amount of literature on designing  $\ell_1$  subspace embeddings for example using Cauchy [80] and exponential [81] random variables.
For  $p \leq 2$ , it is known that the expected sample complexity of rows is  $\sum_{i=1}^{n} p_i = O(d)$  [34], and for  $p > 2$ , this is known to be at most  $O(d^{p/2})$  [35]. While these bounds are optimal in the worst case, one important application of faster sensitivity calculations is to compute the minimal subspace embedding dimension for average case applications, particularly those seen in practice. This gap is captured by the total sensitivity  $\mathfrak{S}_p(\mathbf{A})$ , which has been used to perform more sample efficient subspace embeddings [67, 27, 28], as sampling  $\widetilde{O}(\mathfrak{S}_p(\mathbf{A})d/\epsilon^2)$  rows suffice to provide an  $\ell_p$  subspace embedding.
Accurate sensitivity calculations. While leverage scores give a crude approximation, constant-factor approximations of  $\ell_p$  sensitivities seem daunting since we must compute worst-case ratios over the input space. However, we can compute the cost of one sensitivity specific row by reducing it to  $\ell_p$  regression since by scale invariance, we have  $\frac{1}{\sigma_p^{\mathbf{B}}(\mathbf{a}_i)} = \min_{\mathbf{a}_i^\top \mathbf{x} = 1} \| \mathbf{B}\mathbf{x} \|_p^p$ .
This reduction allows us to use recent developments in fast approximate algorithms for  $\ell_p$  regression [44, 25, 45], which utilize subspace embeddings to first reduce the number of rows of the matrix to  $O(\mathrm{poly}(d))$ . For the specific case of  $p = 1$ , which is the focus of the main body of our paper, we may reduce to approximate  $\ell_1$  regression on a  $O(d) \times d$  matrix, which is a form of linear programming.
Fact 2.4. Given an  $n \times d$  matrix, the cost of computing  $k$  of its  $\ell_1$  sensitivities is  $\widetilde{O}(\mathbf{nnz}(\mathbf{A}) + k \cdot d^{\omega})$ .
3 Approximating functions of sensitivities
We first present a constant-probability algorithm approximating the  $\ell_1$  sensitivities in Algorithm 1. Our algorithm is a natural one: Since computing the  $\ell_1$  sensitivities of all the rows simultaneously is computationally expensive, we instead hash  $\alpha$ -sized blocks of random rows into smaller (constant-sized) buckets and compute the sensitivities of the smaller,  $O(n / \alpha)$ -sized matrix  $\mathbf{P}$  so generated, computing the sensitivities with respect to  $\mathbf{SA}$ , the  $\ell_1$  subspace embedding of  $\mathbf{A}$ . Running this algorithm multiple times gives our high-probability guarantee via the standard median trick.
Theorem 3.1. Given a full-rank matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and an approximation factor  $1 < \alpha \ll n$ , let  $\sigma_{1}(\mathbf{a}_{i})$  be the  $\ell_{1}$  sensitivity of the  $i^{\text{th}}$  row of  $\mathbf{A}$ . Then there exists an algorithm that, in time  $\widetilde{O}\left(\mathbf{nnz}(\mathbf{A}) + \frac{n}{\alpha} \cdot d^{\omega}\right)$ , returns  $\widetilde{\sigma} \in \mathbb{R}_{\geq 0}^{n}$  such that with high probability, for each  $i \in [n]$ ,
$$
\boldsymbol {\sigma} _ {1} (\mathbf {a} _ {i}) \leq \widetilde {\boldsymbol {\sigma}} _ {i} \leq O \left(\boldsymbol {\sigma} _ {1} (\mathbf {a} _ {i}) + \frac {\alpha}{n} \mathfrak {G} _ {1} (\mathbf {A})\right). \tag {3.1}
$$
Proof Sketch of Theorem 3.1; full proof in Appendix B.1. We achieve our guarantee via Algorithm 1. To see this, first note that  $\| \mathbf{SAx}\| _1\approx \Theta (\| \mathbf{Ax}\| _1)$  (since as per Line 1, SA is an  $\ell_1$  subspace embedding of A). Combining this with the fact that every row in A is mapped, via the designed randomness, to some row in the matrix P helps establish the desired approximation guarantee. The runtime follows from using Fact 2.4 to compute  $\ell_{1}$  sensitivities with respect to SA of  $|\mathbf{P}| = O(n / \alpha)$  rows.
As seen in Appendix C.1, our techniques described above also generalize to the  $p \geq 1$  case. Specifically, in Theorem C.2, we show that reducing  $\ell_p$  sensitivity calculations by an  $\alpha$  factor gives an approximation guarantee of the form  $O(\alpha^{p-1} \sigma_p(\mathbf{a}_i)) + \frac{\alpha^p}{n} \mathfrak{S}_p(\mathbf{A}))$ .
Remark 3.2. The sensitivities returned by our algorithm are approximate, with relative and additive error, but are useful in certain settings as they preserve  $\ell_p$  regression approximation guarantees while increasing total sample complexity by only a small poly(d) factor compared to true sensitivities while still keeping it much smaller than that obtained via Lewis weights. To see this with a simple toy
Algorithm 1 Approximating  $\ell_1$ -Sensitivities: Row-wise Approximation
Input: Matrix  $\mathbf{A}\in \mathbb{R}^{n\times d}$  , approximation factor  $\alpha \in (1,n)$    
Output: Vector  $\widetilde{\sigma}\in \mathbb{R}_{>0}^{n}$  that satisfies, for each  $i\in [n]$  , with probability 0.9, that  $\sigma_{1}(\mathbf{a}_{i})\leq \widetilde{\sigma}_{i}\leq O(\sigma_{1}(\mathbf{a}_{i}) + \mathfrak{S}_{1}(\mathbf{A})\frac{\alpha}{n})$    
1: Compute for A an  $\ell_1$  subspace embedding SA  $\in O(d)\times d$  (cf. Definition 2.3)   
2: Partition A into  $\frac{n}{\alpha}$  blocks  $\mathbf{B}_1,\dots ,\mathbf{B}_{n / \alpha}$  each comprising  $\alpha$  randomly selected rows   
3: for the  $\ell^{\mathrm{th}}$  block  $\mathbf{B}_{\ell}$  , with  $\ell \in [\frac{n}{\alpha} ]$  do   
4: Sample  $\alpha$  -dimensional independent Rademacher vectors  $\mathbf{r}_1^{(\ell)},\ldots ,\mathbf{r}_{100}^{(\ell)}$    
5: For each  $j\in [100]$  , compute the row vectors  $\mathbf{r}_j^{(\ell)}\mathbf{B}_\ell \in \mathbb{R}^d$    
6: end for   
7: Let  $\mathbf{P}\in \mathbb{R}^{100\frac{n}{\alpha}\times d}$  be the matrix of all vectors from Line 5. Compute  $\sigma_{1}^{\mathrm{SA}}(\mathbf{P})$  using Fact 2.4   
8: for each  $i\in [n]$  do   
9: Denote by  $J$  the set of row indices in P that a is mapped to in Line 5   
10: Set  $\widetilde{\sigma}_i = \max_{j\in J}(\sigma_1^{\mathrm{SA}}(\mathbf{p}_j))$    
11: end for   
12: Return  $\widetilde{\sigma}$
example, consider the case  $p > 2$ . Note that by Theorem 1.5 of [24], the sample complexity using approximate sensitivities is  $\widetilde{O}(\alpha^{2p - 2}\mathfrak{S}_p^{2 - 2 / p}(\mathbf{A}))$ , using the  $\alpha^p$  factor blowup via Theorem C.2, that using true sensitivities is  $\widetilde{O}(\mathfrak{S}_p^{2 - 2 / p}(\mathbf{A}))$ , and that using  $\ell_p$  Lewis weights is  $O(d^{p / 2})$ . Suppose also that the total sensitivity  $\mathfrak{S}_p(\mathbf{A}) = d$ . Further assume that  $n = d^{10}$  and  $\alpha = n^{\frac{1}{10p}} = d^{1 / p}$ ; then the dimension-dependence of sample complexities given by our approximate sensitivities, true sensitivities, and Lewis weights are, respectively,  $\widetilde{O}(d^4)$ ,  $\widetilde{O}(d^2)$ , and  $\widetilde{O}(d^{p / 2})$  (ignoring small factors in the exponent that do not affect the asymptotic analysis). Thus, our approximate sensitivities provide a tradeoff in computational cost and sample complexity between using true sensitivities and Lewis weights.
3.1 Estimating the sum of  $\ell_p$  sensitivities
In this section, we present a one-shot algorithm that provides a constant-approximation to the total  $\ell_p$  sensitivity for all  $p \geq 1$ . This algorithm is based on importance sampling using  $\ell_p$  Lewis weights. For our desired guarantees, we crucially need the approximation bounds of  $\ell_p$  Lewis weights, as provided by Fact 3.3, efficient computation of  $\ell_p$  Lewis weights as provided by [23] for  $p < 4$  and by [77] for  $p \geq 4$ , and the fact that a random  $\ell_p$  sampling matrix  $\mathbf{S}$  (cf. Fact 3.4) yields  $\mathbf{SA}$ , a constant-approximation subspace embedding to  $\mathbf{A}$  [24, Theorem 1.8].
Algorithm 2 Approximating the Sum of  $\ell_p$ -Sensitivities for  $p \geq 1$
Input: Matrix  $\mathbf{A}\in \mathbb{R}^{n\times d}$  , approximation factor  $\gamma \in (0.01,1)$  , and a scalar  $p\geq 1$    
Output: A positive scalar that satisfies, with probability 0.99, that   
 $\mathfrak{S}_p(\mathbf{A})\leq \widehat{s}\leq (1 + O(\gamma))\mathfrak{S}_p(\mathbf{A})$    
1: Compute all  $\ell_p$  Lewis weights of  $\mathbf{A}$  ,denoting the  $i^{\mathrm{th}}$  weight by  $\mathbf{w}_p(\mathbf{a}_i)$    
2: Define the sampling vector  $v\in \mathbb{R}_{\geq 0}^{n}$  by  $v_{i} = \frac{\mathbf{w}_{p}(\mathbf{a}_{i})}{d}$  for all  $i\in [n]$    
3: Sample  $m = O(d^{|1 - p / 2|})$  rows with replacement, picking the  $i^{\mathrm{th}}$  row with a probability of  $v_{i}$    
4: Construct a random  $\ell_p$  sampling matrix  $\mathbf{S}_p\mathbf{A}$  with probabilities  $\{v_i\}_{i = 1}^n$  (cf. Fact 3.4)   
5: For each sampled row  $i_j$ $j\in [m]$  , compute  $r_j = \frac{\sigma_p^{\mathbf{S}_p\mathbf{A}}(\mathbf{a}_{ij})}{v_{ij}}$    
6: Return  $\frac{1}{m}\sum_{j = 1}^{m}r_{j}$
Fact 3.3 (Lewis Weights Bounds Sensitivities, [42, 82]). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and  $p \geq 1$ , the  $\ell_p$  sensitivity  $\sigma_p^{\mathbf{A}}(\mathbf{a}_i)$  and  $\ell_p$  Lewis weight  $\mathbf{w}_p(\mathbf{a}_i)$  of the row  $\mathbf{a}_i$  satisfy, for all  $i \in [n]$ ,
$$
\boldsymbol {\sigma} _ {p} ^ {\mathbf {A}} (\mathbf {a} _ {i}) \leq d ^ {\max  (0, p / 2 - 1)} \mathbf {w} _ {p} (\mathbf {a} _ {i}).
$$
Fact 3.4 (Sampling via Lewis Weights [23, 35]). Given  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and  $p > 0$ , Consider a random diagonal  $\ell_p$  sampling matrix  $\mathbf{S} \in \mathbb{R}^{n \times n}$  with sampling probabilities  $\{p_i\}$  proportional to the  $\ell_p$  Lewis weight of  $\mathbf{A}$ , i.e., for each  $i \in [n]$ , the  $i^{\text{th}}$  diagonal entry is independently set to be
$$
\mathbf {S} _ {i, i} = \left\{ \begin{array}{l l} 1 / p _ {i} ^ {1 / p} & \text {w i t h p r o b a b i l i t y p _ {i}} \\ 0 & \text {o t h e r w i s e} \end{array} \right..
$$
Then, with high probability,  $\mathbf{S}$  with  $O(\varepsilon^{-2}d^{\max(1,p/2)}(\log d)^2\log(d / \varepsilon))$  rows is an  $\ell_p$  subspace embedding for  $\mathbf{A}$  (cf. Definition 2.3).
Theorem 3.5. Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and an approximation factor  $\gamma \in (0,1)$ , there exists an algorithm, which returns a positive scalar  $\widehat{s}$  such that, with a probability 0.99, we have
Proof. Without loss of generality, we may assume  $\mathbf{A}$  to be full rank. Then, its Lewis weights satisfy  $\sum_{i=1}^{n} \mathbf{w}_p(\mathbf{a}_i) = d$ . Per Line 2 of Algorithm 2, our sampling distribution is chosen to be  $v_i = \frac{\mathbf{w}_p(\mathbf{a}_i)}{d}$  for all  $i \in [n]$ . We sample from  $\mathbf{A}$  rows with replacement, with row  $i$  picked with a probability of  $v_i$ . From the definition of  $v_i$  in Line 2 and  $r_j$  in Line 5 and the fact that  $\mathbf{S}_p\mathbf{A}$  is a constant factor  $\ell_p$  subspace embedding of  $\mathbf{A}$ , our algorithm's output satisfies the following unbiasedness condition:
$$
\mathbb {E} \left(\frac {1}{m} \sum_ {j \in [ m ]} r _ {j}\right) = \mathbb {E} \left(\frac {1}{m} \sum_ {j = 1} ^ {m} \sum_ {i _ {j} = 1} ^ {n} \frac {\boldsymbol {\sigma} _ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {a} _ {i _ {j}})}{v _ {i _ {j}}} \cdot v _ {i _ {j}}\right) = \mathfrak {G} _ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {A}) \leq 2 \mathfrak {G} _ {p} (\mathbf {A}).
$$
By independence, we also have the following bound on the variance of  $\frac{1}{m}\sum_{j\in [m]}r_j$ :
$$
\operatorname {V a r} \left(\frac {1}{m} \sum_ {j \in [ m ]} r _ {j}\right) \leq \frac {1}{m} \sum_ {i = 1} ^ {n} \frac {\boldsymbol {\sigma} _ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {a} _ {i}) ^ {2}}{v _ {i}} = \frac {d}{m} \cdot \sum_ {i = 1} ^ {n} \frac {\boldsymbol {\sigma} _ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {a} _ {i}) ^ {2}}{\mathbf {w} _ {p} (\mathbf {a} _ {i})},
$$
with the final step holding by the choice of  $v_{i}$  in Line 2. In the case that  $p \geq 2$ , we have
$$
\operatorname {V a r} \left(\frac {1}{m} \sum_ {j \in [ m ]} r _ {j}\right) \leq \frac {d}{m} \cdot \sum_ {i = 1} ^ {n} \frac {\sigma_ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {a} _ {i}) ^ {2}}{\mathbf {w} _ {p} (\mathbf {a} _ {i})} \leq \frac {d}{m} \cdot \sum_ {i \in [ n ]} \sigma_ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {a} _ {i}) \cdot d ^ {p / 2 - 1} \leq \frac {2 d ^ {p / 2}}{m} \mathfrak {G} _ {p} (\mathbf {A}),
$$
where the first inequality uses Fact 3.3. Therefore, applying Chebyshev's inequality on  $\frac{1}{m}\sum_{j\in [m]}r_j$  (as defined in Line 5) with  $m = O\left(\frac{d^{p / 2}}{\mathfrak{S}_p(\mathbf{A})\gamma^2}\right)$  gives us a  $\gamma$ -multiplicative accuracy in approximating  $\mathfrak{S}_p(\mathbf{A})$  (with the desired constant probability). For  $p\geq 2$ , we additionally have [24, Theorem 1.7] the lower bound  $\mathfrak{S}_p(\mathbf{A})\geq d$ , which when plugged into the value of  $m$  gives the claimed sample complexity. A similar argument may be applied for the case  $p < 2$ ; specifically, we have that
$$
\operatorname {V a r} \left(\frac {1}{m} \sum_ {j \in [ m ]} r _ {j}\right) \leq \frac {d}{m} \cdot \sum_ {i = 1} ^ {n} \frac {\sigma_ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {a} _ {i}) ^ {2}}{\mathbf {w} _ {p} (\mathbf {a} _ {i})} \leq \frac {d}{m} \cdot \sum_ {i \in [ n ]} \sigma_ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {a} _ {i}) \leq \frac {2 d}{m} \mathfrak {S} _ {p} (\mathbf {A}),
$$
where the second step is by  $\mathbf{w}_p(\mathbf{a}_i) \geq \sigma_p(\mathbf{a}_i)$  from Fact 3.3. For  $p < 2$ , we also have  $\mathfrak{S}_p(\mathbf{A}) \geq d^{p/2}$  from [24, Theorem 1.7]. Applying Chebyshev's inequality with this fact gives a sample complexity of  $m = O\left(d^{1 - p/2} / \gamma^2\right)$ . This completes the correctness guarantee.
Runtime. We first compute all  $\ell_p$  Lewis weights of  $\mathbf{A} \in \mathbb{R}^{n \times d}$  up to a constant multiplicative accuracy, the cost of which is  $O\left(\frac{1}{1 - |1 - p/2|} \log (\log (n))\right)$  leverage score computations for  $p < 4$  [23] and  $O(p^3 \log (np))$  leverage score computations for  $p \geq 4$  [77]. Next, we compute  $m = O(d^{|1 - p/2|})$  sensitivities with respect to  $\mathbf{S}_p\mathbf{A}$ . From [23, 35],  $\mathbf{S}_p$  has, with high probability,  $O(d^{p/2} \log d)$  rows when  $p > 2$  and  $O(d \log d)$  rows when  $p \leq 2$ . Summing the cost of computing these  $m$  sensitivities and that of computing the Lewis weights gives the claimed runtime.
Remark 3.6. We present in Appendix B.2 an alternate algorithm, Algorithm 4, for estimating the total  $\ell_p$  sensitivity for  $p = 1$ . Algorithm 4 uses recursive computations of leverage scores, in contrast to Algorithm 2 which uses Lewis weights in a one-shot manner, and may be of independent interest.
3.2 Approximating the maximum  $\ell_{1}$  sensitivity
In this section, we present an algorithm that approximates  $\| \pmb{\sigma}_1(\mathbf{A})\|_{\infty} = \max_{i\in [|\mathbf{A}|]}\pmb{\sigma}_1(\mathbf{a}_i)$ , the maximum of the  $\ell_1$  sensitivities of the rows of a matrix  $\mathbf{A}$ . As alluded to in Section 1, a first approach to this problem would be to simply estimate all  $n$  sensitivities and compute their maximum. To do better than this, one idea, inspired by the random hashing approach of Algorithm 1, is as follows.
If the matrix has a large number of high-sensitivity rows, then, intuitively, the appropriately scaled maximum sensitivity of a uniformly sampled subset of these rows should approximate  $\| \pmb{\sigma}_1(\mathbf{A})\|_{\infty}$ . Specifically, assume that the matrix has at least  $\alpha$  rows of sensitivity at least  $\| \pmb{\sigma}_1(\mathbf{A})\|_{\infty} / \sqrt{\alpha}$ ; uniformly sample  $n / \alpha$  rows and return  $\widetilde{\pmb{\sigma}}_a$ , the (appropriately scaled) maximum of  $\pmb{\sigma}_1^{\mathbf{S}_1\mathbf{A}}(\mathbf{a}_i)$ , for the sampled rows  $i$ . Then,  $\| \pmb{\sigma}_1(\mathbf{A})\|_{\infty} \leq \widetilde{\pmb{\sigma}}_a \leq O(\sqrt{\alpha}\| \pmb{\sigma}_1(\mathbf{A})\|_{\infty})$  with a constant probability. Here the upper bound guarantee is without any condition on the number of rows with large  $\ell_1$  sensitivities.
In the other case, if the matrix does not have too many high-sensitivity rows, we could estimate the maximum sensitivity by hashing rows into small buckets via Rademacher combinations of uniformly sampled blocks of  $\alpha$  rows each (just like in Algorithm 1). Then,  $\widetilde{\sigma}_b$ , the scaled maximum of the  $\ell_1$  sensitivities of these rows satisfies  $\| \pmb{\sigma}_1(\mathbf{A}) \|_{\infty} \leq \widetilde{\sigma}_b \leq O(\sqrt{\alpha} \| \pmb{\sigma}_1(\mathbf{A}) \|_{\infty})$ . Here, it is the lower bound that comes for free (i.e., without any condition on the number of rows with large  $\ell_1$  sensitivities).
Despite the above strategies working for each case, there is no way to combine these approaches without knowledge of whether the input matrix has a enough "high-sensitivity" rows or not. We therefore avoid this approach and instead present Algorithm 3, where we make use of  $\mathbf{S}_{\infty}\mathbf{A}$ , an  $\ell_{\infty}$  subspace embedding of  $\mathbf{A}$ . Our algorithm hinges on the recent development [75, Theorem 1.3] on efficient construction of such an embedding with simply a subset of  $\widetilde{O}(d)$  rows of  $\mathbf{A}$ .
Algorithm 3 Approximating the Maximum of  $\ell_1$ -Sensitivities
Input: Matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$
Output: Scalar  $\widehat{s} \in \mathbb{R}_{\geq 0}$  that satisfies  $\|\pmb{\sigma}_1(\mathbf{A})\|_{\infty} \leq \widehat{s} \leq C \cdot \sqrt{d} \cdot \|\pmb{\sigma}_1(\mathbf{A})\|_{\infty}$ .
1: Compute, for  $\mathbf{A}$ , an  $\ell_{\infty}$  subspace embedding  $\mathbf{S}_{\infty} \mathbf{A} \in \mathbb{R}^{O(d \log^2(d)) \times d}$  such that  $\mathbf{S}_{\infty} \mathbf{A}$  is a subset of the rows of  $\mathbf{A}$  [75, Theorem 1.3]
2: Compute, for  $\mathbf{A}$ , an  $\ell_1$  subspace embedding  $\mathbf{S}_1\mathbf{A} \in \mathbb{R}^{O(d) \times d}$
3: Return  $\sqrt{d} \| \sigma_1^{\mathbf{S}_1\mathbf{A}}(\mathbf{S}_{\infty}\mathbf{A})\|_{\infty}$
Theorem 3.7 (Approximating the Maximum of  $\ell_1$  Sensitivities). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , there exists an algorithm, which in time  $\widetilde{O}(\mathbf{nnz}(\mathbf{A}) + d^{\omega + 1})$ , outputs a positive scalar  $\widehat{s}$  that satisfies
$$
\Omega (\| \sigma_ {1} (\mathbf {A}) \| _ {\infty}) \leq \widehat {s} \leq O (\sqrt {d} \| \sigma_ {1} (\mathbf {A}) \| _ {\infty}).
$$
Proof Sketch of Theorem 3.7; full proof in Appendix B.3. We achieve our guarantee via Algorithm 3. Define  $\mathbf{x}^{\star}$  and  $\mathbf{a}_{i^{\star}}$  as:  $\mathbf{x}^{\star}, i^{\star} = \arg \max_{\mathbf{x} \in \mathbb{R}^{d}, i \in [|\mathbf{A}|]} \frac{|\mathbf{a}_{i}^{\top} \mathbf{x}|}{\|\mathbf{A} \mathbf{x}\|_{1}}$ . Since  $\mathbf{S}_{1} \mathbf{A}$  is an  $\ell_{1}$  subspace embedding of  $\mathbf{A}$ , we have, for any  $\mathbf{x} \in \mathbb{R}^{d}$ , that  $\|\mathbf{S}_{1} \mathbf{A} \mathbf{x}\|_{1} = \Theta(\|\mathbf{A} \mathbf{x}\|_{1})$ . If  $\mathbf{S}_{\infty} \mathbf{A}$  contains the row  $\mathbf{a}_{i^{\star}}$ , then  $\|\mathbf{S}_{1} \mathbf{A} \mathbf{x}\|_{1} = \Theta(\|\mathbf{A} \mathbf{x}\|_{1})$  implies  $\|\sigma_{1}^{\mathbf{S}_{1} \mathbf{A}}(\mathbf{S}_{\infty} \mathbf{A})\|_{\infty} = \Theta(1)\|\sigma_{1}(\mathbf{A})\|_{\infty}$ . In the other case, suppose  $\mathbf{a}_{i^{\star}}$  is not included in  $\mathbf{S}_{\infty} \mathbf{A}$ . Then we observe that
$$
\left\| \boldsymbol {\sigma} _ {1} ^ {\mathbf {S} _ {1} \mathbf {A}} \left(\mathbf {S} _ {\infty} \mathbf {A}\right) \right\| _ {\infty} = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {c} _ {j} \in \mathbf {S} _ {\infty} \mathbf {A}} \frac {\mathbf {c} _ {j} ^ {\top} \mathbf {x}}{\left\| \mathbf {S} _ {1} \mathbf {A x} \right\| _ {1}} \geq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty}}{\left\| \mathbf {S} _ {1} \mathbf {A x} \right\| _ {1}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty}}{\left\| \mathbf {A x} \right\| _ {1}}, \tag {3.2}
$$
where the second step is by choosing a specific vector in the numerator and the third step uses  $\| \mathbf{S}_1\mathbf{A}\mathbf{x}\| _1 = \Theta (\| \mathbf{A}\mathbf{x}\| _1)$ . We further have,
$$
\max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty}}{\left\| \mathbf {A x} \right\| _ {1}} \geq \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} ^ {\star} \right\| _ {\infty}}{\left\| \mathbf {A x} ^ {\star} \right\| _ {1}} \geq \frac {\left\| \mathbf {A x} ^ {\star} \right\| _ {\infty}}{\sqrt {d} \| \mathbf {A x} ^ {\star} \| _ {1}} \geq \frac {\left| \mathbf {a} _ {i ^ {*}} ^ {\top} \mathbf {x} ^ {\star} \right|}{\sqrt {d} \| \mathbf {A x} ^ {\star} \| _ {1}} = \frac {1}{\sqrt {d}} \| \boldsymbol {\sigma} _ {1} (\mathbf {A}) \| _ {\infty},, \tag {3.3}
$$