SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since Inequality (B.19) shows upper and lower bounds on each sensitivity  $\sigma_{1}^{\mathbf{S}^{\prime}\mathbf{A}}(\mathbf{M}_{i})$  of the matrix  $\mathbf{M}_i$ , we may use Lemma A.3 to approximate the sum of these sensitivities. In particular, by Lemma A.3, we may construct a matrix  $\widetilde{\mathbf{M}}_i$  composed of  $\frac{20\sqrt{|C|}(1 + 2\rho)\log(\delta^{-1})}{\rho^2}$  rows of  $\mathbf{M}_i$  sampled uniformly at random with replacement, and we are guaranteed
$$
(1 + \rho) \frac {\left| \mathbf {M} _ {i} \right|}{\left| \widetilde {\mathbf {M}} _ {i} \right|} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\widetilde {\mathbf {M}} _ {i}\right) \approx_ {\rho} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\mathbf {M} _ {i}\right) \text {w i t h a p r o b a b i l i t y a t l e a s t} 1 - \delta . \tag {B.20}
$$
Note that the output  $\widehat{s}$  of Algorithm 5 when applied to node  $\mathcal{N}$  is defined as
$$
\widehat {s} := (1 + \rho) \sum_ {i \in [ \mathcal {B} ]} \frac {\left| \mathbf {M} _ {i} \right|}{\left| \widetilde {\mathbf {M}} _ {i} \right|} \widehat {s} _ {i}. \tag {B.21}
$$
Therefore, by union bound over the failure probability (cf. Fact B.3), we can combine Equation (B.14) and sum Equation (B.20) over all  $i \in \mathcal{B}$  child nodes to conclude that  $\widehat{s}$  from Equation (B.21) satisfies a  $(\widehat{\rho} + \rho, (\widehat{\delta} + \delta)\mathcal{B})$ -approximation guarantee of Definition B.4.
B.2.3 Proof of Theorem B.1
Proof of Theorem B.1. We first sketch our strategy to prove Theorem B.1's correctness guarantee. Essentially, our goal is to establish Definition B.4 for the root node  $\mathcal{T}_{(1,1)}$ . That this goal suffices to prove the theorem is justified in Lemma B.7. We now show that the node  $\mathcal{T}_{(1,1)}$  satisfies a  $(\widehat{\rho},\widehat{\delta})$ -approximation guarantee for some  $\widehat{\rho}$  and  $\widehat{\delta}$ . We use the method of induction.
Correctness guarantee of Theorem B.1. We know that the leaf nodes all satisfy the  $(\rho, 0)$ -approximation guarantee in Definition B.4 since at this level, the base case is triggered, where we compute a  $\rho$ -approximation estimate of the total  $\ell_1$  sensitivity with respect to the subspace embedding  $\mathbf{S}'\mathbf{A}$ . Having established this approximation guarantee at the leaf nodes, we can use Lemma B.8 to inductively propagate the property of  $(\rho, \delta)$ -approximation guarantee up the recursion tree  $\mathcal{T}$ . From Lemma B.6, we have that the recursion tree  $\mathcal{T}$  of Algorithm 5 is of depth at most  $D = O(\log \log (n + d))$ , and the number of child nodes at any given node is at most  $\mathcal{B} = \Theta (\log (n))$ ; we factor these into the approximation guarantee and error probability.
Base case. At the leaf nodes of the recursion tree, we directly compute the total sensitivities with respect to  $\mathbf{S}'\mathbf{A}$ . Recall that we denote the depth of the tree by  $D$ . Then, because  $\mathbf{S}'\mathbf{A}$  is, by definition, a  $\rho$ -approximate subspace embedding for  $\mathbf{A}$ , we have, for each  $i$  that indexes a leaf node,
$$
\widehat {s} _ {(D, i)} \approx_ {\rho} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\mathbf {M} ^ {(D, i)}\right) \text {w i t h a p r o b a b i l i t y o f 1 .} \tag {B.22}
$$
Induction proof. Consider the subtree  $\mathcal{T}_j$  truncated at (but including) the nodes at level  $j$  in the recursion tree  $\mathcal{T}$  corresponding to Algorithm 5. Assume the induction hypothesis that all the leaf nodes in the subtree  $\mathcal{T}_j$  satisfy Definition B.4 with parameters  $\left(\rho \cdot (D + 1 - j),\delta \cdot \left(\frac{\mathcal{B}^{D - j + 1} - \mathcal{B}}{\mathcal{B} - 1}\right)\right)$ . That is, at the matrix of each such node  $i$ , we have an estimate  $\widehat{s}_{(j,i)}$  of its total sensitivity  $\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M}^{(j,i)})$  such that
$$
\widehat {s} _ {(j, i)} \approx_ {\rho \cdot (D + 1 - j)} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\mathbf {M} ^ {(j, i)}\right) \text {w i t h a p r o b a b i l i t y a t l e a s t} 1 - \delta \cdot \left(\frac {\mathcal {B} ^ {D - j + 1} - \mathcal {B}}{\mathcal {B} - 1}\right). \tag {B.23}
$$
Then, each of the leaf nodes of the subtree  $\mathcal{T}_{j-1}$  (i.e., the subtree that stops one level above  $\mathcal{T}_j$ ) satisfies the assumption in Lemma B.8 since its child nodes are either leaf nodes of the entire recursion tree  $\mathcal{T}$  (for which this statement has been shown in Inequality (B.22)) or, if they are not
leaf nodes of the recursion tree, they satisfy Definition B.4 with parameters  $\widehat{\rho} = \rho \cdot (D + 1 - j)$  and  $\widehat{\delta} = \delta \cdot \left(\frac{\mathcal{B}^{D - j + 1} - \mathcal{B}}{\mathcal{B} - 1}\right)$ . Therefore, Lemma B.8 implies that each of the leaf nodes of  $\mathcal{T}_{j - 1}$  satisfies Definition B.4 with approximation parameter  $\widehat{\rho}$  and error probability  $\widehat{\delta}$  defined as follows:
$$
\widehat {\rho} = \rho + \rho \cdot (D + 1 - j) = \rho \cdot (D + 2 - j) \text {a n d} \widehat {\delta} = \mathcal {B} \cdot \left(\delta + \delta \cdot \left(\frac {\mathcal {B} ^ {D - j + 1} - \mathcal {B}}{\mathcal {B} - 1}\right)\right) = \delta \cdot \left(\frac {\mathcal {B} ^ {D - j + 2} - \mathcal {B}}{\mathcal {B} - 1}\right).
$$
In other words, the outputs  $\widehat{s}_{(j - 1,k)}$  of Algorithm 5 on the matrices  $\mathbf{M}^{(j - 1,k)}$  in the leaf nodes of  $\mathcal{T}_{j - 1}$  each satisfy:
$$
\widehat {s} _ {(j - 1, k)} \approx_ {O (\rho \cdot (D + 2 - j))} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\mathbf {M} ^ {(j - 1, k)}\right) \text {w i t h a p r o b a b i l i t y a t l e a s t} 1 - \delta \cdot \left(\frac {\mathcal {B} ^ {D - j + 2} - \mathcal {B}}{\mathcal {B} - 1}\right). \tag {B.24}
$$
The base case in Inequality (B.22) and the satisfaction of the induction hypothesis in Inequality (B.24) together finish the proof of the induction hypothesis.
Finishing the proof of correctness. In light of the above conclusion, for the root node  $\mathcal{T}_{(1,1)}$ , which is at the level  $j = 1$ , we may plug in  $j = 1$  in Inequality (B.23) and obtain that the output  $\widehat{s}_{(1,1)}$  of Algorithm 5 at the root node satisfies
$$
\widehat {s} _ {(1, 1)} \approx_ {O (D \rho)} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\mathbf {M} ^ {(1, 1)}\right) \text {w i t h a p r o b a b i l i t y o f a t l e a s t} 1 - \delta \left(\frac {\mathcal {B} ^ {D} - \mathcal {B}}{\mathcal {B} - 1}\right). \tag {B.25}
$$
In Algorithm 5, we choose the parameters  $\delta = \frac{0.01}{B^D}$  and  $\rho = O\left(\frac{\gamma}{D}\right)$ . Plugging these parameters into Inequality (B.25), we conclude that the root node satisfies Definition B.4 with a  $(\gamma, 0.01)$ -approximation guarantee. This finishes the proof of correctness of the claim.
Proof of runtime. Let  $b$  denote the maximum number of rows of a matrix at a leaf node of  $\mathcal{T}$ . Further, recall from Fact 2.4 that the cost of computing one  $\ell_1$  sensitivity of an  $n \times d$  matrix is,
$$
L = \widetilde {O} (\mathbf {n n z} (\mathbf {A}) + d ^ {\omega}). \tag {B.26}
$$
Then as stated in Line 1, the algorithm computes the  $\rho$ -approximate total sensitivity at the leaf nodes; Fact 2.4, this incurs a cost of
$$
\mathcal {C} (b) = \widetilde {O} (\mathbf {n n z} (\mathbf {A}) + b \cdot d ^ {\omega}). \tag {B.27}
$$
At any other node, the computational cost is the work done at that level plus the total cost of the recursive calls at that level. Let  $\mathcal{C}(r)$  denote the runtime of Algorithm 5 when the input is an  $r\times d$  matrix. Then,
$$
\mathcal {C} (r) = \left\{ \begin{array}{l l} \mathcal {B} \cdot \mathcal {C} \left(\frac {\sqrt {r} (1 + \rho)}{\rho^ {2}} \log (\delta^ {- 1})\right) + L & r > b \\ \mathcal {C} (b) & r \leq b \end{array} \right.,
$$
where  $b$  is the number of rows in the base case, and  $L$  is the cost of computing leverage scores and equals  $L = \widetilde{O}(\mathbf{nnz}(\mathbf{A}) + d^{\omega})$  from Fact 2.1. For the depth of the recursion  $D$  (where the top level is  $D = 1$ ), plugging in our choices of  $\delta = \frac{0.01}{B^{D}}$  and  $\rho = O\left(\frac{\gamma}{D}\right)$ , this may be expressed as via expanding the recursion as:
$$
\mathcal {C} (n) = \mathcal {B} ^ {D} \cdot \mathcal {C} (b) + L \cdot \left(\frac {\mathcal {B} ^ {D} - 1}{\mathcal {B} - 1}\right) \tag {B.28}
$$
for  $n \geq b$ . We plug into this expression the values of  $b$  and  $D$  from Lemma B.6,  $L$  from Equation (B.26),  $\mathcal{C}(b)$  from Equation (B.27), and  $\mathcal{B} = \Theta(\log(n))$  to get the claimed runtime.
B.3 Estimating the maximum  $\ell_1$  sensitivity
$$
\Omega (\| \sigma_ {1} (\mathbf {A}) \| _ {\infty}) \leq \widehat {s} \leq O (\sqrt {\bar {d}} \| \sigma_ {1} (\mathbf {A}) \| _ {\infty}).
$$
$$
\left\| \mathbf {S} _ {1} \mathbf {A} \mathbf {x} \right\| _ {1} = \Theta (\left\| \mathbf {A} \mathbf {x} \right\| _ {1}). \tag {B.29}
$$
We use Equation (B.29) to establish the claimed bounds below. First we set some notation. Define  $\mathbf{x}^{\star}$  and  $\mathbf{a}_{i^{\star}}$  as follows:
$$
\mathbf {x} ^ {\star}, i ^ {\star} = \arg \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, i \in [ | \mathbf {A} | ]} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} |}{\| \mathbf {A x} \| _ {1}} \tag {B.30}
$$
Thus,  $\mathbf{x}^{\star}$  is the vector that realizes the maximum sensitivity of the matrix  $\mathbf{A}$ , and the row  $\mathbf{a}_{i^{\star}}$  is the row of  $\mathbf{A}$  with maximum sensitivity with respect to  $\mathbf{A}$ . Suppose the matrix  $\mathbf{S}_{\infty}\mathbf{A}$  contains the row  $\mathbf{a}_{i^{\star}}$ . Then we have
$$
\max  _ {i: \mathbf {c} _ {i} \in \mathbf {S} _ {\infty} \mathbf {A}} \boldsymbol {\sigma} _ {1} (\mathbf {c} _ {i}) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {c} _ {k} \in \mathbf {S} _ {\infty} \mathbf {A}} \frac {| \mathbf {c} _ {k} ^ {\top} \mathbf {x} |}{\| \mathbf {S} _ {1} \mathbf {A x} \| _ {1}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {c} _ {k} \in \mathbf {S} _ {\infty} \mathbf {A}} \frac {| \mathbf {c} _ {k} ^ {\top} \mathbf {x} |}{\| \mathbf {A x} \| _ {1}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {a} _ {i ^ {*}} ^ {\top} \mathbf {x} |}{\| \mathbf {A x} \| _ {1}}, \tag {B.31}
$$
where the first step is by definition of  $\ell_1$  sensitivity of  $\mathbf{S}_{\infty}\mathbf{A}$  with respect to  $\mathbf{S}_1\mathbf{A}$ , the second step is by Equation (B.29), and the third step by noting that the matrix  $\mathbf{S}_{\infty}\mathbf{A}$  is a subset of the rows of  $\mathbf{A}$  (which includes  $\mathbf{a}_{i^{\star}}$ ). By definition of  $\sigma_{1}(\mathbf{a}_{i^{\star}})$  in Equation (B.30), we have
$$
\max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left| \mathbf {a} _ {i ^ {*}} ^ {\top} \mathbf {x} \right|}{\left\| \mathbf {A x} \right\| _ {1}} = \left\| \boldsymbol {\sigma} _ {1} (\mathbf {A}) \right\| _ {\infty},. \tag {B.32}
$$
Then, combining Equation (B.31) and Inequality (B.32) gives the guarantee in this case. In the other case, suppose  $\mathbf{a}_{i^{\star}}$  is not included in  $\mathbf{S}_{\infty}\mathbf{A}$ . Then we observe that the upper bound from the preceding inequalities still holds. For the lower bound, we observe that
$$
\left\| \boldsymbol {\sigma} _ {1} ^ {\mathbf {S} _ {1} \mathbf {A}} \left(\mathbf {S} _ {\infty} \mathbf {A}\right) \right\| _ {\infty} = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {c} _ {j} \in \mathbf {S} _ {\infty} \mathbf {A}} \frac {\mathbf {c} _ {j} ^ {\top} \mathbf {x}}{\left\| \mathbf {S} _ {1} \mathbf {A x} \right\| _ {1}} \geq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty}}{\left\| \mathbf {S} _ {1} \mathbf {A x} \right\| _ {1}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty}}{\left\| \mathbf {A x} \right\| _ {1}}, \tag {B.33}
$$
$$
\max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty}}{\left\| \mathbf {A x} \right\| _ {1}} \geq \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} ^ {\star} \right\| _ {\infty}}{\left\| \mathbf {A x} ^ {\star} \right\| _ {1}} \geq \frac {\left\| \mathbf {A x} ^ {\star} \right\| _ {\infty}}{\sqrt {d} \| \mathbf {A x} ^ {\star} \| _ {1}} \geq \frac {\left| \mathbf {a} _ {i ^ {\star}} ^ {\top} \mathbf {x} ^ {\star} \right|}{\sqrt {d} \| \mathbf {A x} ^ {\star} \| _ {1}} = \frac {1}{\sqrt {d}} \| \boldsymbol {\sigma} _ {1} (\mathbf {A}) \| _ {\infty}, \quad (B. 3 4)
$$
where the first step is by choosing  $\mathbf{x} = \mathbf{x}^{\star}$ , the second step is by the distortion guarantee of  $\ell_{\infty}$  subspace embedding [35], and the final step is by definition of  $\sigma_{1}(\mathbf{a}_{i^{*}})$ . Combining Inequality (C.10) and Inequality (C.11) gives the claimed lower bound on  $\| \sigma_1^{\mathbf{S}_1\mathbf{A}}(\mathbf{S}_\infty \mathbf{A})\|_\infty$ . The runtime follows from the computational cost and row dimension of  $\mathbf{S}_{\infty}\mathbf{A}$  from [35] and the cost of computing  $\ell_{1}$  sensitivities with respect to  $\mathbf{S}_1\mathbf{A}$  as per Fact 2.4.
C Omitted proofs:  $\ell_p$  sensitivities
We recall the following notation that we use for stating our results in this setting.
Definition C.1 (Notation for  $\ell_p$  Results). We introduce the notation  $\mathsf{LP}(m,d,p)$  to denote the cost of approximating one  $\ell_p$  sensitivity of an  $m\times d$  matrix up to an accuracy of a given constant factor.
C.1 Estimating all  $\ell_p$  sensitivities
We generalize Algorithm 1 to the case  $p \geq 1$  below.
Theorem C.2 (Approximating all  $\ell_p$  sensitivities). Given a full-rank matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , an approximation factor  $1 < \alpha \ll n$ , and a scalar  $p \geq 1$ , let  $\sigma_p(\mathbf{a}_i)$  be the  $i^{\text{th}} \ell_p$  sensitivity. Then there exists an algorithm that returns a vector  $\widetilde{\sigma} \in \mathbb{R}_{\geq 0}^n$  such that with high probability, for each  $i \in [n]$ , we have
$$
\Omega \left(\boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i})\right) \leq \widetilde {\boldsymbol {\sigma}} _ {i} \leq O \left(\alpha^ {p - 1} \boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i})\right) + \frac {\alpha^ {p}}{n} \widetilde {\mathbf {S}} _ {p} (\mathbf {A}). \tag {C.1}
$$
Our algorithm runs in time  $\widetilde{O}\left(\mathbf{nnz}(\mathbf{A}) + \frac{n}{\alpha} \cdot \mathsf{LP}(d^{\max(1,p/2)}, d, p)\right)$  (cf. Definition 1.2).
Algorithm 6 Approximating  $\ell_p$ -Sensitivities: Row-wise Approximation
Inputs: Matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , approximation factor  $\alpha \in (1, n]$ , scalar  $p \geq 1$
Output: Vector  $\widetilde{\sigma} \in \mathbb{R}_{>0}^n$  that satisfies, for each  $i \in [n]$ , with probability 0.9, that
$$
\boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i}) \leq \widetilde {\boldsymbol {\sigma}} _ {i} \leq O \left(\alpha^ {p - 1} \boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i}) + \frac {\alpha^ {p}}{n} \mathfrak {S} _ {p} (\mathbf {A})\right)
$$
1: Compute, for  $\mathbf{A}$ , an  $\ell_p$  subspace embedding  $\mathbf{S}_p\mathbf{A} \in \mathbb{R}^{\widetilde{O}(d^{\max(1,p/2)}) \times d}$
2: Partition  $\mathbf{A}$  into  $\frac{n}{\alpha}$  blocks  $\mathbf{B}_1, \ldots, \mathbf{B}_{n / \alpha}$  each comprising  $\alpha$  randomly selected rows.
3: for the  $\ell^{\mathrm{th}}$  block  $\mathbf{B}_{\ell}$ , with  $\ell \in [\frac{n}{\alpha} ]\mathbf{d}\mathbf{o}$
4: Sample  $\alpha$ -dimensional independent Rademacher vectors  $\mathbf{r}_1^{(\ell)}, \ldots, \mathbf{r}_{100}^{(\ell)}$
5: For each  $j\in [100]$  , compute the row vectors  $\mathbf{r}_j^{(\ell)}\mathbf{B}_\ell \in \mathbb{R}^d$
6: end for
7: Let  $\mathbf{P} \in \mathbb{R}^{100\frac{n}{\alpha} \times d}$  be the matrix of all vectors computed in Line 5. Compute  $\pmb{\sigma}_p^{\mathbf{S}_p\mathbf{A}}(\mathbf{P})$  using Definition 1.2
8: for each  $i \in [n]$  do
9: Denote by  $J$  the set of row indices in  $\mathbf{P}$  that  $\mathbf{a}_i$  is mapped to in Line 5
10: Set  $\widetilde{\pmb{\sigma}}_i = \max_{j\in J}(\pmb{\sigma}_p^{\mathbf{S}_p\mathbf{A}}(\mathbf{p}_j))$
11: end for
12: Return  $\widetilde{\sigma}$
Proof. Suppose the  $i^{\mathrm{th}}$  row of  $\mathbf{A}$  falls into the bucket  $\mathbf{B}_{\ell}$ . Suppose, further, that the rows from  $\mathbf{B}_{\ell}$  are mapped to those in  $\mathbf{P}$  with row indices in the set  $J$ . Then, Algorithm 6 returns a vector of sensitivity estimates  $\widetilde{\sigma} \in \mathbb{R}^{n}$  with the  $i^{\mathrm{th}}$  coordinate defined as  $\widetilde{\sigma}_i = \max_{j \in J} \sigma_p(\mathbf{c}_j)$ . For any  $\mathbf{x} \in \mathbb{R}^d$ , we have
$$
\| \mathbf {S} _ {p} \mathbf {A x} \| _ {p} ^ {p} \approx \| \mathbf {A x} \| _ {p} ^ {p}. \tag {C.2}
$$
We use Equation (C.2) to establish the claimed bounds below. Let  $\mathbf{x}^{\star} = \arg \max_{\mathbf{x}\in \mathbb{R}^{d}}\frac{|\mathbf{a}_{i}^{\top}\mathbf{x}|^{p}}{||\mathbf{A}\mathbf{x}||_{p}^{p}}$  be the vector that realizes the  $i^{\mathrm{th}}\ell_p$  sensitivity of A. Further, let the  $j^{\mathrm{th}}$  row of  $\mathbf{P}$  be  $\mathbf{r}_k^{(\ell)}\mathbf{B}_\ell$ . Then, with probability of at least  $1 / 2$ , we have
$$
\boldsymbol {\sigma} _ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {p} _ {j}) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} | ^ {p}}{\| \mathbf {S} _ {p} \mathbf {A x} \| _ {p} ^ {p}} \geq \frac {| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} ^ {\star} | ^ {p}}{\| \mathbf {S} _ {p} \mathbf {A x} ^ {\star} \| _ {p} ^ {p}} = \Theta (1) \frac {| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} ^ {\star} | ^ {p}}{\| \mathbf {A x} ^ {\star} \| _ {p} ^ {p}} \geq \Theta (1) \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} ^ {\star} | ^ {p}}{\| \mathbf {A x} ^ {\star} \| _ {p} ^ {p}} = \Theta (1) \boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i}), \tag {C.3}
$$
where the first step is by definition of  $\sigma_{p}(\mathbf{p}_{j})$ , the second is by evaluating the function being maximized at a specific choice of  $\mathbf{x}$ , the third step is by Equation (C.2), and the final step is by definition of  $\mathbf{x}^{\star}$  and  $\sigma_{p}(\mathbf{a}_{i})$ . For the fourth step, we use the fact that  $|\mathbf{r}_{k}^{(\ell)}\mathbf{B}_{\ell}\mathbf{x}^{\star}|^{p} = |\mathbf{r}_{k,i}^{(\ell)}\mathbf{a}_{i}^{\top}\mathbf{x}^{\star} + \sum_{j\neq i}\mathbf{r}_{k,j}^{(\ell)}(\mathbf{B}\mathbf{x}^{\star})_{j}|^{p}\geq |\mathbf{a}_{i}^{\top}\mathbf{x}^{\star}|^{p}$  with a probability of at least  $1 / 2$  since the vector  $\mathbf{r}_k^{(\ell)}$  has coordinates that are  $-1$  or  $+1$  with equal probability. By a union bound over  $j\in J$  independent rows that block  $\mathbf{B}_{\ell}$  is mapped to, we establish the claimed lower bound in Inequality (C.3) with probability at least 0.9. To show an upper bound on  $\sigma_{p}^{\mathbf{S}_{p}\mathbf{A}}(\mathbf{p}_{j})$ , we observe that
$$
\max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} | ^ {p}}{\| \mathbf {S} _ {p} \mathbf {A x} \| _ {p} ^ {p}} \leq \alpha^ {p - 1} \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\| \mathbf {B} _ {\ell} \mathbf {x} \| _ {p} ^ {p}}{\| \mathbf {S} _ {p} \mathbf {A x} \| _ {p} ^ {p}} \leq \Theta (\alpha^ {p - 1}) \sum_ {j: \mathbf {a} _ {j} \in \mathbf {B} _ {\ell}} \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {a} _ {j} ^ {\top} \mathbf {x} | ^ {p}}{\| \mathbf {A x} \| _ {p} ^ {p}} = \Theta (\alpha^ {p - 1}) \sum_ {j: \mathbf {a} _ {j} \in \mathbf {B} _ {\ell}} \sigma_ {p} (\mathbf {a} _ {j}),
$$
where the second step is by Equation (C.2) and opening up  $\| \mathbf{B}_{\ell}\mathbf{x}\| _p^p$  in terms of the rows in  $\mathbf{B}_{\ell}$ , and the final step is by definition of  $\mathbf{B}_{\ell}$ . To see the first step, we observe that
$$
\left| \mathbf {r} _ {k} ^ {(\ell)} \mathbf {B} _ {\ell} \mathbf {x} \right| ^ {p} \leq \left\| \mathbf {B} _ {\ell} \mathbf {x} \right\| _ {p} ^ {p} \cdot \left\| \mathbf {r} _ {k} ^ {(\ell)} \right\| _ {q} ^ {p} \leq \left\| \mathbf {B} _ {\ell} \mathbf {x} \right\| _ {p} ^ {p} \cdot \alpha^ {p / q} = \left\| \mathbf {B} _ {\ell} \mathbf {x} \right\| _ {p} ^ {p} \cdot \alpha^ {p - 1},
$$
where  $\frac{1}{p} +\frac{1}{q} = 1$ , the first step is by Hölder's inequality, the second is because each entry of  $\mathbf{r}_k^{(\ell)}$  is either  $+1$  or  $-1$  and  $\mathbf{r}_k^{(\ell)}\in \mathbb{R}^\alpha$ , and the third is because  $\frac{p}{q} = p - 1$ . Because  $\mathbf{B}_{\ell}$  is a group of  $\alpha$  rows
selected uniformly at random out of  $n$  rows and contains the row  $\mathbf{a}_i$ , we have:
$$
\mathbb {E} \left\{\sum_ {j: \mathbf {a} _ {j} \in \mathbf {B} _ {\ell}, j \neq i} \pmb {\sigma} _ {p} (\mathbf {a} _ {j}) \right\} = \frac {\alpha - 1}{n - 1} \sum_ {j \neq i} \pmb {\sigma} _ {p} (\mathbf {a} _ {i}).
$$
$$
\boldsymbol {\sigma} _ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {p} _ {j}) \leq O \left(\alpha^ {p - 1} \boldsymbol {\sigma} _ {p} (\mathbf {a} _ {i}) + \frac {\alpha^ {p}}{n} \mathfrak {S} _ {p} (\mathbf {A})\right). \tag {C.5}
$$
The median trick then establishes Inequality (C.4) and Inequality (C.5) with high probability. To establish the runtime, note that we first construct the subspace embedding  $\mathbf{S}_p\mathbf{A}$  and then compute the  $\ell_p$  sensitivities of  $\mathbf{P}$  (with  $O(n / \alpha)$  rows) with respect to  $\mathbf{S}_p\mathbf{A}$ . The size of the subspace embedding  $\mathbf{S}_p\mathbf{A}$  is  $O(d^{1\vee p / 2}\times d)$  (see [75, Table 1]). This completes the runtime claim.
C.2 Estimating the maximum of  $\ell_p$  sensitivities
In Section 3.2 and Appendix B.3, we showed our result for estimating the maximum  $\ell_1$  sensitivity. In this section, we show how to extend this result to all  $p \geq 1$ . Algorithm 7 generalizes Algorithm 3.
Algorithm 7 Approximating the Maximum of  $\ell_p$ -Sensitivities
Input: Matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and  $p \geq 1$  (with  $p \neq 2$ )
Output: Scalar  $\widehat{s} \in \mathbb{R}_{\geq 0}$  that satisfies that
$$
\| \pmb {\sigma} _ {p} (\mathbf {A}) \| _ {\infty} \leq \widehat {s} \leq C \cdot d ^ {p / 2} \cdot \| \pmb {\sigma} _ {p} (\mathbf {A}) \| _ {\infty}
$$
1: Compute, for  $\mathbf{A}$ , an  $\ell_{\infty}$  subspace embedding  $\mathbf{S}_{\infty} \mathbf{A} \in \mathbb{R}^{O(d \log^2(d)) \times d}$  such that  $\mathbf{S}_{\infty} \mathbf{A}$  is a subset of the rows of  $\mathbf{A}$  [35]
2: Compute, for  $\mathbf{A}$ , an  $\ell_p$  subspace embedding  $\mathbf{S}_p\mathbf{A} \in \left\{ \begin{array}{ll} \mathbb{R}^{O(d) \times d} & p \in [1, 2) \\ \mathbb{R}^{O(d^{p/2}) \times d} & p > 2 \end{array} \right.$
3: Return  $d^{p/2} \|\sigma_p^{\mathbf{S}_p\mathbf{A}}(\mathbf{S}_\infty \mathbf{A})\|_\infty$
Theorem C.3 (Approximating the maximum of  $\ell_p$  sensitivities). Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and  $p \geq 1$  (with  $p \neq 2$ ), there exists an algorithm, which outputs a positive scalar  $\widehat{s}$  that satisfies
$$
\Omega (\| \pmb {\sigma} _ {p} (\mathbf {A}) \| _ {\infty}) \leq \widehat {s} \leq O (d ^ {p / 2} \| \pmb {\sigma} _ {p} (\mathbf {A}) \| _ {\infty}).
$$
The runtime of the algorithm is  $\widetilde{O}(\mathbf{nnz}(\mathbf{A}) + d^{\omega + 1})$  for  $p \in [1,2)$  and  $\widetilde{O}(\mathbf{nnz}(\mathbf{A}) + d^{p/2} \cdot \mathsf{LP}(O(d^{p/2}), d, p))$  for  $p > 2$ .
$$
\left\| \mathbf {S} _ {p} \mathbf {A x} \right\| _ {p} ^ {p} = \Theta \left(\left\| \mathbf {A x} \right\| _ {p} ^ {p}\right). \tag {C.6}
$$
We use Equation (C.6) to establish the claimed bounds below. First we set some notation. Define  $\mathbf{x}^{\star}$  and  $\mathbf{a}_{i^{\star}}$  as follows: