SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 5 Subroutine for Approximating the Sum of  $\ell_1$ -Sensitivities
Input: Matrices  $\mathbf{M} \in \mathbb{R}^{r \times d}$ ,  $\mathbf{S}\mathbf{A} \in \mathbb{R}^{O(d) \times d}$ ,  $\mathbf{S}'\mathbf{A} \in \mathbb{R}^{O(d) \times d}$ , scalars  $n$  and  $\rho \in (0,1)$   
Output: A positive scalar  $\widehat{s}$  that satisfies, with probability 0.99, that  
\[
\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M}) \leq \widehat{s} \leq (1 + O(\rho))\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M}).
\]  
1: Set  $\mathcal{B} = \Theta(\log n)$ ,  $D = O(\log (\log (n + d))$ ,  $\delta = \frac{0.01}{B^D}$  and  $b = \widetilde{O}\left(\frac{D^4}{\gamma^2}\max\left(\frac{D^4}{\gamma^2},\sqrt{d}\right)\right)$   
2: if the number of rows of  $\mathbf{M}$  is at most  $b$  then  
3: Let  $\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M})$  be the sum of  $\ell_1$  sensitivities of the rows of  $\mathbf{M}$  with respect to  $\mathbf{S}'\mathbf{A}$   
4: Using Fact 2.4 on  $\mathbf{M}$ , compute  $\widehat{s}$  such that  $\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M}) \leq \widehat{s} \leq (1 + O(\rho))\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M})$   
5: Return  $\widehat{s}$   
6: else  
7: Construct the matrix  $\mathbf{C} := \begin{bmatrix} \mathbf{M} \\ \mathbf{SA} \end{bmatrix}$ . Set  $r = O(\sqrt{|\mathbf{C}|}(1 + \rho)\rho^{-2}\log (\delta^{-1}))$ .  
8: Compute  $\tau(\mathbf{C})$ , the vector of leverage scores of  $\mathbf{C}$   
9: Divide  $[\frac{1}{n^{20}},1]$  into  $\mathcal{B}$  geometrically decreasing sub-intervals  $[1/2,1]$ ,  $[1/4,1/2],\ldots,$  etc.  
10: for each bucket  $i \in [\mathcal{B}]$  created in the previous step do  
11: Construct matrix  $\mathbf{M}_i$  with those rows of  $\mathbf{M}$  whose leverage scores fall in the  $i^{\text{th}}$  bucket  
12: Construct matrix  $\widetilde{\mathbf{M}}_i$  composed of  $r$  uniformly sampled (with replacement) rows of  $\mathbf{M}_i$   
13: Compute  $\widetilde{s}_i$ , the output of Algorithm 5 with inputs  $\widetilde{\mathbf{M}}_i$ , SA, S'A,  $n$ , and  $\rho$ .  
14: end for  
15: Return  $\widehat{s} := (1 + \rho)\sum_{i\in [\mathcal{B}]}|\frac{\mathbf{M}_i|}{|\widetilde{\mathbf{M}}_i|}\widetilde{s}_i$   
16: end if
those rows with respect to the original matrix  $\mathbf{A}$ . In general, this is not true. However, we can meet our requirement using the  $\ell_1$  subspace embedding  $\mathbf{S}\mathbf{A}$  of the (original) matrix  $\mathbf{A}$ . Specifically, if we vertically concatenate a matrix  $\widetilde{\mathbf{M}}_i$  with  $\mathbf{S}\mathbf{A}$  and call this matrix  $\mathbf{C}$ , then for each row  $\widetilde{\mathbf{M}}_i[j] \in \widetilde{\mathbf{M}}_i$ , we have  $\sigma_1^{\mathbf{C}}(\widetilde{\mathbf{M}}_i[j]) \approx \sigma_1^{\mathbf{A}}(\widetilde{\mathbf{M}}_i[j])$ . Further,  $|\mathbf{C}| \leq O(\sqrt{n})$ , and so in the step subsampling the rows of  $\widetilde{\mathbf{M}}_i$ , we choose  $O(n^{1/4})$  samples. Recursing this exponentially decreases the row dimension of the input matrix at each level, with  $(\log n)^k$  matrices at the  $k^{\text{th}}$  level of recursion.
B.2.2 Helper lemmas for proving Theorem B.1
For our formal proof, we need Definition B.2, Fact B.3, and Definition B.4.
Definition B.2 (Notation for recursion tree of Algorithm 5). Recall that  $\mathcal{B} = \Theta (\log (n))$  as in Line 1 of Algorithm 5. Then we have the following notation corresponding to Algorithm 5.
We define a rooted  $\mathcal{B}$ -ary tree  $\mathcal{T}$  corresponding to the execution of Algorithm 5.
- We use  $\mathcal{T}_j$  to denote the subtree starting from the root node with all the nodes up to and including those at the  $j^{\text{th}}$  level. We denote the  $i^{\text{th}}$  node at the  $j^{\text{th}}$  level by  $\mathcal{T}_{(j,i)}$ . Thus, every node is uniquely specified by its level in the tree and its index within that level.
The root node  $\mathcal{T}_{(1,1)}$  at level 1 corresponds to the first call to Algorithm 5 from Algorithm 4.
For the node  $\mathcal{T}_{(j,i)}$ , we denote by  $\mathbf{M}^{(j,i)}$  the input matrix to the corresponding recursive call; note that all the other inputs are global and remain the same throughout the recursive call.
$\triangleright$  Analogously, we use  $\widehat{s}_{(j,i)}$  to denote our estimate of  $\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M}^{(j,i)})$ , the sum of  $\ell_1$  sensitivities of rows of the matrix  $\mathbf{M}^{(j,i)}$  in the node  $\mathcal{T}_{(j,i)}$ , defined with respect to the sketched matrix  $\mathbf{S}'\mathbf{A}$ .
Fact B.3. If event  $A$  happens with probability  $1 - \tau_{1}$  and event  $B$  happens with probability  $1 - \tau_{2}$  conditioned on event  $A$ , then the probability of both events  $A$  and  $B$  happening is at least  $1 - \tau_{1} - \tau_{2}$ .
Definition B.4. We say that a node  $\mathcal{N}$  with matrix  $\mathbf{M}$  satisfies the  $(\widehat{\rho},\widehat{\delta})$ -approximation guarantee if  $\widehat{s}$ , the output of Algorithm 5 on  $\mathbf{M}$ , satisfies the following guarantee for the true sum  $\mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M})$  of  $\ell_1$  sensitivities of the rows in  $\mathbf{M}$ :
$\widehat{s} \approx_{\widehat{\rho}} \mathfrak{S}_1^{\mathbf{S}'\mathbf{A}}(\mathbf{M})$  with a probability at least  $1 - \widehat{\delta}$
for some approximation parameter  $\widehat{\rho} \in (0,1)$  and an error probability parameter  $\widehat{\delta} \in (0,1)$ .
Lemma B.5 (Partitioning of  $\mathbf{M}$ ). In Algorithm 5, even though we ignore the range  $\left[0, \frac{1}{n^{20}}\right]$  when creating the submatrices  $\mathbf{M}_1, \mathbf{M}_2, \ldots, \mathbf{M}_{\mathcal{B}}$  from  $\mathbf{M}$ , every row in  $\mathbf{M}$  is part of exactly one of the submatrices  $\mathbf{M}_1, \mathbf{M}_2, \ldots, \mathbf{M}_{\mathcal{B}}$ . In other words, the matrix  $\mathbf{M}$  is partitioned into the matrices  $\mathbf{M}_1, \mathbf{M}_2, \ldots, \mathbf{M}_{\mathcal{B}}$ , with no row missed out.
Proof. Our proof strategy is to show that for all row indices  $k \in [|\mathbf{M}|]$ , each of the leverage scores  $\tau_{k}(\mathbf{C})$  satisfy  $\tau_{k}(\mathbf{C}) \geq \frac{1}{n^{20}}$ . Therefore, creating buckets of rows with the range of leverage score starting at  $\frac{1}{n^{20}}$  (instead of 0) does not miss out any rows.
To this end, we note that for every row  $\mathbf{a}_k$  in the input  $\mathbf{M}$  to Algorithm 5, we have:
$$
\sqrt {\frac {1}{n ^ {1 1}}} \leq \sqrt {\frac {\tau_ {k} (\mathbf {A})}{| \mathbf {A} |}} \leq \sigma_ {1} ^ {\mathbf {A}} (\mathbf {a} _ {k}) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {x} ^ {\top} \mathbf {a} _ {k} |}{\| \mathbf {A x} \| _ {1}} \leq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} 3 \frac {| \mathbf {x} ^ {\top} \mathbf {a} _ {k} |}{\| \mathbf {C x} \| _ {1}} = 3 \sigma_ {1} ^ {\mathbf {C}} (\mathbf {a} _ {k}) \leq 3 \sqrt {\tau_ {k} (\mathbf {C})}, \tag {B.5}
$$
where the first step is because in Line 3 of Algorithm 4, we discard those rows  $\mathbf{a}_k$  of  $\mathbf{A}$  with leverage scores smaller than  $1 / n^{10}$  and there are  $n$  rows in  $\mathbf{A}$ , so every row  $\mathbf{a}_k$  in the input matrix  $\mathbf{M}$  to Algorithm 5 satisfies  $\tau_k(\mathbf{A}) \geq \frac{1}{n^{10}}$ ; the second step is by Fact 2.2 applied to  $\mathbf{a}_k$ ; the third step is by the definition of the  $\ell_1$  sensitivity of  $\mathbf{a}_k$  with respect to  $\mathbf{A}$ ; the fourth step is by observing that  $\| \mathbf{C} \mathbf{x} \|_1 = \| \mathbf{S} \mathbf{A} \mathbf{x} \|_1 + \| \mathbf{M} \mathbf{x} \|_1 \leq 2 \| \mathbf{A} \mathbf{x} \|_1 + \| \mathbf{A} \mathbf{x} \|_1 = 3 \| \mathbf{A} \mathbf{x} \|_1$  since  $\mathbf{SA}$  is a constant-factor (with the constant assumed  $1/2$ ).  $\ell_1$  subspace embedding of  $\mathbf{A}$  and because  $\mathbf{M}$  is a submatrix of  $\mathbf{A}$ ; the fifth step is by the definition of  $\ell_1$  sensitivity of  $\mathbf{a}_k$  with respect to  $\mathbf{C}$ ; the final step is by Fact 2.2 applied to  $\mathbf{a}_k$  within  $\mathbf{C}$ . Observing the first and last terms of this inequality chain, we have  $\tau_k(\mathbf{C}) \geq \frac{1}{3n^{11}}$  for all  $k \in [|\mathbf{C}|]$ . This finishes the proof of the claim.
Lemma B.6. For the recursion tree  $\mathcal{T}$  corresponding to Algorithm 5, let the size  $b$  of the input matrices at the leaf nodes be  $b = \frac{50D^3\log(100\mathcal{B})}{\gamma^2} \cdot \left(\frac{2D^3\log(100\mathcal{B})}{\gamma^2} + \sqrt{d\log(d)}\right)$ . Then, the depth of the recursion tree is less than  $D := 1 + \log (\log (2n + 2d\log (d)))$ .
Proof. For now, let  $\beta$  be arbitrary but less than  $\beta^2 \leq n + d'$ , where  $d'$  is the row dimension of the subspace embedding SA. Then, the size of the input matrix to the nodes of the recursion tree as we increase in depth evolves as
$$
\beta \sqrt {d ^ {\prime} + n}, \beta \sqrt {d ^ {\prime} + \beta \sqrt {d ^ {\prime} + n}}, \beta \sqrt {d ^ {\prime} + \beta \sqrt {d ^ {\prime} + \beta \sqrt {d ^ {\prime} + n}}}. \dots
$$
Thus, this sequence evolves as
$$
x _ {t + 1} = \beta \sqrt {d ^ {\prime} + x _ {t}}, \text {w i t h} x _ {0} = n + d ^ {\prime}.
$$
To see the limit of this sequence, we set  $x = \beta \sqrt{d' + x}$ , and solve for this quadratic equation as
$$
x = \frac {\beta^ {2} + \sqrt {\beta^ {4} + 4 \beta^ {2} d ^ {\prime}}}{2} \leq \beta (\beta + \sqrt {d ^ {\prime}}). \tag {B.6}
$$
We set the base case to be  $b^{\star} \coloneqq 7\beta (\beta + \sqrt{d'})$ , a constant times larger than this asymptotic limit. We now compute the depth  $t^{\star}$  at which the recursion attains the value  $b^{\star}$  is attained. To do so, we consider a sequence  $\{y_t\}$  which overestimates the sequence  $\{x_t\}$  of matrix sizes for a big range of  $t$ . We define  $\{y_t\}$  as
$$
y _ {0} = n + d ^ {\prime}, \text {a n d} y _ {t + 1} = \beta \sqrt {2 y _ {t}},
$$
and it can be checked that for  $t$  such that  $y_{t} \geq d'$ , we have  $y_{t+1} \geq x_{t+1}$ . Moreover, for all  $t \geq 1$ ,  $y_{t} = \beta^{1 + 1/2 + \dots + 1/2^{t-1}} (2y_{0})^{1/2^{t}} = \beta^{2 - 2^{1-t}} (2y_{0})^{1/2^{t}}$ . The limit of this sequence can easily be seen to  $\beta^{2}$ . We now split our analysis in two cases based on the limits of these two sequences.
Case 1:  $\beta^2 \geq d'$ . Since the limit of the sequence  $\{y_t\}$  is  $\beta^2$ , it is easy to see that  $y_t$  is always greater  $d'$  (since  $y_t$  is monotonically decreasing as  $y_0 \geq \beta^2$ ). Thus,  $x_t$  will always be less than  $y_t$  in this case. We will now show that  $y_t$  (and thus  $x_t$ ) will be less than  $b^\star$  in  $t_0 = 2\log \log(2y_0)$  steps. Solving the inequality for  $t$  as follows:  $y_t \leq \beta^2 (2y_0)^{1/2^t} \leq 4\beta^2 \leq b^\star$ , it suffices to ensure that  $(2y_0)^{1/2^t} \leq 4$ , which happens if  $t \geq t_0$ . Thus,  $t^\star \leq 2\log \log 2y_0$ .
Case 2:  $\beta^2 < d'$ . In this case, the sequence  $y_{t}$  will eventually go below  $d'$  and thus the inequality  $y_{t} \geq x_{t}$  might eventually break down. Let  $t = t'$  denote the first time step at which  $y_{t} \leq 5d'$  (which will be finite in this case). Then, by definition,  $y_{t} \geq x_{t}$  for all  $t < t'$ . We will now upper bound the value of  $t'$  as follows:  $y_{t} \leq \beta^{2}(2y_{0})^{1/2^{t}} \leq d'(2y_{0})^{1/2^{t}} \leq 4d'$ ; Thus, it suffices to ensure that  $(2y_{0})^{1/2^{t}} \leq 4$ , which happens if  $t \geq t_{0} := 2\log \log(2y_{0})$ . Hence,  $t' \leq t_{0}$ . Moreover, the value of  $y_{t'} \leq 5d'$ , implies that  $x_{t'-1} \leq y_{t'-1} \leq \frac{(5d')^{2}}{2\beta^{2}}$ . We will now show that  $x_{t'+1} \leq b^{\star}$  using the evolution of  $x_{t+1} = \beta \sqrt{d' + x_{t}}$  as follows: First,  $x_{t'} \leq \beta \sqrt{d'} + \beta \sqrt{x_{t'-1}} \leq \beta \sqrt{d'} + 5d' \leq 6d'$ ; Then,  $x_{t'+1} \leq \beta \sqrt{d'} + \beta \sqrt{6d'} \leq 7\beta \sqrt{d'} \leq b^{\star}$ . Thus,  $t^{\star} \leq 1 + 2\log \log 2y_{0}$ .
Therefore, in both of these cases, we have that after at most  $D = 1 + \log \log (2n + 2d')$  depth, the base case of  $b^{\star} = 7\beta (\beta + \sqrt{d'})$  is achieved at the leaf node. Recall that the algorithm sets  $D = 2\log \log (n + d') + 1$  and  $\rho = \gamma / D$  for  $d' = d\log(d)$  which implies that  $\beta := C\frac{(1+\rho)}{\rho^2}\log(\delta^{-1}) = CD\frac{(D+\gamma)}{\gamma^2}\log(100\mathcal{B}^D) \leq \frac{2D^3\log(100\mathcal{B})}{\gamma^2}$ . Since Algorithm 5 sets  $b = \frac{50D^3\log(100\mathcal{B})}{\gamma^2} \cdot \left(\frac{2D^3\log(100\mathcal{B})}{\gamma^2} + \sqrt{d\log(d)}\right)$ , which is larger than  $b^{\star}$ , we may conclude that the choice of  $b$  implies that the recursion tree  $\mathcal{T}$  of Algorithm 5 has a depth less than  $D = 1 + \log(\log(2n + 2d\log(d)))$ .
Lemma B.7. Suppose the root node  $\mathcal{T}_{(1,1)}$  of Algorithm 5 satisfies a  $(\widehat{\rho},\widehat{\delta})$ -approximation guarantee as defined in Definition B.4, and denote by  $\widehat{s}$  the output of Algorithm 4. Then, with a probability of at least  $1 - \widehat{\delta}$ , we have
$$
\frac {1}{1 + \gamma} \widehat {s} \approx_ {\widehat {\rho} + \rho} \mathfrak {S} _ {1} ^ {\mathbf {A}} (\mathbf {A}).
$$
As a result of this lemma, in the final proof of correctness of Algorithm 4 (appearing later as proof of Theorem B.1), it suffices to show that  $\widehat{s}$  satisfies a  $(\widehat{\rho}, \widehat{\delta})$ -approximation guarantee for some appropriate choices of these parameters.
Proof of Lemma B.7. Suppose the root node satisfies Definition B.4 with some  $(\widehat{\rho}, \widehat{\delta})$ -approximation guarantee. This means that there exists an  $\widetilde{s}$  such that the output of Algorithm 5 on the input matrix  $\widehat{\mathbf{A}}$  satisfies
$$
\widetilde {s} \approx_ {\widehat {\rho}} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} (\widehat {\mathbf {A}}) \text {w i t h a p r o b a b i l i t y a t l e a s t} 1 - \widehat {\delta}. \tag {B.7}
$$
Since  $\mathbf{S}'\mathbf{A}$  is, by design, a  $\rho$ -approximation  $\ell_1$  subspace embedding for  $\mathbf{A}$ , it means for all  $\mathbf{x} \in \mathbb{R}^d$  we have
$$
(1 - \rho) \| \mathbf {A} \mathbf {x} \| _ {1} \leq \| \mathbf {S} ^ {\prime} \mathbf {A} \mathbf {x} \| _ {1} \leq (1 + \rho) \| \mathbf {A} \mathbf {x} \| _ {1}. \tag {B.8}
$$
Therefore, for every row  $\mathbf{a}_i\in \hat{\mathbf{A}}$  we have by the definition of  $\ell_1$  sensitivity and Inequality (B.8) that the  $\ell_1$  sensitivity of  $\mathbf{a}_i$  computed with respect to  $\mathbf{S}'\mathbf{A}$  is a  $\rho$ -approximation of the  $\ell_1$  sensitivity of  $\mathbf{a}_i$  with respect to  $\mathbf{A}$ :
$$
\boldsymbol {\sigma} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} (\mathbf {a} _ {i}) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} |}{\| \mathbf {S} ^ {\prime} \mathbf {A x} \| _ {1}} \approx_ {\rho} \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} |}{\| \mathbf {A x} \| _ {1}} = \boldsymbol {\sigma} _ {1} ^ {\mathbf {A}} (\mathbf {a} _ {i}).
$$
Therefore, by linearity, this  $\rho$ -approximation factor transfers over in connecting the total sensitivity of  $\hat{\mathbf{A}}$  with respect to  $\mathbf{S}'\mathbf{A}$  to the total sensitivity of  $\hat{\mathbf{A}}$  with respect to  $\mathbf{A}$ :
$$
\mathfrak {G} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} (\widehat {\mathbf {A}}) = \sum_ {i \in [ | \widehat {\mathbf {A}} | ]} \sigma_ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} (\mathbf {a} _ {i}) \approx_ {\rho} \sum_ {i \in [ | \widehat {\mathbf {A}} | ]} \sigma_ {1} ^ {\mathbf {A}} (\mathbf {a} _ {i}) = \mathfrak {G} _ {1} ^ {\mathbf {A}} (\widehat {\mathbf {A}}). \tag {B.9}
$$
Therefore, chaining Equation (B.7) and Equation (B.9) yields the following approximation guarantee:
$$
\widetilde {s} \approx_ {\widehat {\rho} + \rho} \mathfrak {S} _ {1} ^ {\mathbf {A}} (\widehat {\mathbf {A}}) \text {w i t h a p r o b a b i l i t y a t l e a s t} 1 - \widehat {\delta}. \tag {B.10}
$$
There is no change in the error probability above from Equation (B.7) because Equation (B.9) holds deterministically. Next, since in Line 3 of Algorithm 4, we dropped rows of  $\mathbf{A}$  that have leverage scores less than  $\frac{1}{n^{10}}$ , for each of the dropped rows  $\mathbf{a}_i$ , we have
$$
\boldsymbol {\sigma} _ {1} ^ {\mathbf {A}} \left(\mathbf {a} _ {i}\right) \leq \sqrt {\tau_ {i} (\mathbf {A})} \leq \frac {1}{n ^ {5}}. \tag {B.11}
$$
Therefore, the quantity we return,  $\widehat{s}$ , satisfies the following approximation guarantee:
$$
\frac {1}{1 + \gamma} \widehat {s} := \widetilde {s} + \frac {1}{n ^ {5}} (| \mathbf {A} | - | \widehat {\mathbf {A}} |) \geq (1 - (\widehat {\rho} + \rho)) \mathfrak {S} _ {1} ^ {\mathbf {A}} (\widehat {\mathbf {A}}) + \sum_ {i: \mathbf {a} _ {i} \notin \widehat {\mathbf {A}}} \boldsymbol {\sigma} _ {1} ^ {\mathbf {A}} (\mathbf {a} _ {i}) \geq (1 - (\widehat {\rho} + \rho)) \mathfrak {S} _ {1} ^ {\mathbf {A}} (\mathbf {A}), \tag {B.12}
$$
where the first step is by definition of  $\widehat{s}$  in Algorithm 4; the second step is by Equation (B.10); the third step is by applying Inequality (B.11) to each of the rows not present in  $\widehat{\mathbf{A}}$ . In the other direction,
$$
\frac {1}{1 + \gamma} \widehat {s} = \widetilde {s} + \frac {1}{n ^ {5}} (| \mathbf {A} | - | \widehat {\mathbf {A}} |) \leq (1 + (\widehat {\rho} + \rho)) \mathfrak {S} _ {1} ^ {\mathbf {A}} (\widehat {\mathbf {A}}) + \frac {1}{n ^ {4}} \leq (1 + (\widehat {\rho} + \rho)) \mathfrak {S} _ {1} ^ {\mathbf {A}} (\mathbf {A}), \tag {B.13}
$$
where the first step is by definition of  $\widehat{s}$  in Algorithm 4; the second step is by the fact that  $|\mathbf{A}| \leq n$ ; the final step is by combining the assumption  $\rho \geq \frac{1}{n^4}$  and the facts that all sensitivities are non-negative and that the total  $\ell_1$  sensitivity is at least one. Combining Inequality (B.12) and Inequality (B.13) finishes the proof.
Lemma B.8 (Single-Level Computation in Recursion Tree). Consider a node  $\mathcal{N}$  in the recursion tree corresponding to Algorithm 5, and let  $\mathbf{M}$  be the input matrix for  $\mathcal{N}$ . Further assume that  $\mathcal{N}$  is not a leaf node. Denote by  $\mathcal{N}_1, \mathcal{N}_2, \ldots, \mathcal{N}_B$  (where  $\mathcal{B} = \Theta(\log(n))$ ) is the number of recursive calls at each node in Algorithm 5) the children of node  $\mathcal{N}$ , with each node  $\mathcal{N}_i$  with the input matrix  $\widetilde{\mathbf{M}}_i$  (as constructed in Algorithm 5). Suppose each of these child nodes  $\mathcal{N}_i$  satisfies a  $(\widehat{\rho}, \widehat{\delta})$ -approximation guarantee (cf. Definition B.4). Then  $\mathcal{N}$  satisfies a  $(\widehat{\rho} + \rho, (\widehat{\delta} + \delta) \cdot \mathcal{B})$ -approximation guarantee.
Proof. We first reiterate the steps of Algorithm 5 that are crucial to this proof. As proved in Lemma B.5, Algorithm 5 partitions the rows of the matrix  $\mathbf{M}$  in node  $\mathcal{N}$  into  $\mathcal{B}$  submatrices  $\mathbf{M}_1, \mathbf{M}_2, \ldots, \mathbf{M}_{\mathcal{B}}$  based on their leverage scores. Next, for each matrix  $\mathbf{M}_i$ , we uniformly sample with replacement  $O(\sqrt{|\mathbf{C}|}(1 + \rho)\rho^{-2}\log (\delta^{-1}))$  rows and term this set of rows as matrix  $\widetilde{\mathbf{M}}_i$ , which we then pass to Algorithm 5 recursively in node  $\mathcal{N}_i$  (recall that we assume that the node  $\mathcal{N}$  is not a leaf node). In other words, each child node  $\mathcal{N}_i$  has as its input matrix  $\widetilde{\mathbf{M}}_i$ .
Since we assume that all the child nodes  $\mathcal{N}_1, \mathcal{N}_2, \ldots, \mathcal{N}_B$  satisfy a  $(\widehat{\rho}, \widehat{\delta})$ -approximation guarantee as in Definition B.4, it means that, for all  $i \in [\mathcal{B}]$ , the output  $\widehat{s}_i$  of Algorithm 5 on node  $\mathcal{N}_i$  satisfies the following guarantee:
$$
\widehat {s} _ {i} \approx_ {\widehat {\rho}} \mathfrak {S} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\widetilde {\mathbf {M}} _ {i}\right) \text {w i t h a p r o b a b i l i t y a t l e a s t} 1 - \widehat {\delta}. \tag {B.14}
$$
Next, we recall that each  $\mathbf{M}_i$  is composed of only those rows of  $\mathbf{M}$  whose leverage scores (computed with respect to  $\mathbf{C}$ ) lie in the  $i^{\text{th}}$  bucket  $[2^{-i}, 2^{-i+1}]$ . Combining this with Fact 2.2, this implies that the  $\ell_1$  sensitivity of the  $j^{\text{th}}$  row  $\mathbf{M}_i[j]$  of the matrix  $\mathbf{M}_i$ , computed with respect to  $\mathbf{C}$ , satisfies
$$
\sqrt {\frac {2 ^ {- i}}{| \mathbf {C} |}} \leq \sqrt {\frac {\tau_ {j} ^ {\mathbf {C}} (\mathbf {M} _ {i})}{| \mathbf {C} |}} \leq \boldsymbol {\sigma} _ {1} ^ {\mathbf {C}} \left(\mathbf {M} _ {i} [ j ]\right) \leq \sqrt {\tau_ {j} ^ {\mathbf {C}} (\mathbf {M} _ {i})} \leq \sqrt {2 ^ {- i + 1}}. \tag {B.15}
$$
Additionally, we claim that
$$
\frac {1}{3 (1 + \rho)} \boldsymbol {\sigma} _ {1} ^ {\mathbf {C}} \left(\mathbf {M} _ {i} [ j ]\right) \leq \boldsymbol {\sigma} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\mathbf {M} _ {i} [ j ]\right) \leq 3 (1 + \rho) \boldsymbol {\sigma} _ {1} ^ {\mathbf {C}} \left(\mathbf {M} _ {i} [ j ]\right). \tag {B.16}
$$
To see this, we observe that
$$
\boldsymbol {\sigma} _ {1} ^ {\mathbf {C}} \left(\mathbf {M} _ {i} [ j ]\right) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {x} ^ {\top} \mathbf {M} _ {i} [ j ] |}{\| \mathbf {C x} \| _ {1}} \geq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {x} ^ {\top} \mathbf {M} _ {i} [ j ] |}{3 \| \mathbf {A x} \| _ {1}} \geq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {x} ^ {\top} \mathbf {M} _ {i} [ j ] |}{3 (1 + \rho) \| \mathbf {S} ^ {\prime} \mathbf {A x} \| _ {1}} = \frac {\boldsymbol {\sigma} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} (\mathbf {M} _ {i} [ j ])}{3 (1 + \rho)}, \tag {B.17}
$$
where the first step is by the definition of  $\ell_1$  sensitivity of  $\mathbf{M}_i[j]$  (i.e., the  $j^{\mathrm{th}}$  row of matrix  $\mathbf{M}_i$ ) with respect to  $\mathbf{C}$ ; the second step is by the definition of  $\mathbf{C}$ , the fact that  $\mathbf{SA}$  is a 1/2-factor  $\ell_1$  subspace embedding of  $\mathbf{A}$ , and because  $\mathbf{M}$  is a subset of rows of  $\mathbf{A}$ ; the third step is because  $\mathbf{S}'\mathbf{A}$  is a  $\rho$ -approximate  $\ell_1$  subspace embedding for  $\mathbf{A}$ , and the final step is by the definition of  $\sigma_1^{\mathbf{A}}(\mathbf{M}_i[j])$ . In the other direction, we have
$$
\boldsymbol {\sigma} _ {1} ^ {\mathbf {C}} (\mathbf {M} _ {i} [ j ]) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {x} ^ {\top} \mathbf {M} _ {i} [ j ] |}{\| \mathbf {C x} \| _ {1}} \leq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {x} ^ {\top} \mathbf {M} _ {i} [ j ] |}{0 . 5 \| \mathbf {A x} \| _ {1}} \leq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {(1 + \rho) | \mathbf {x} ^ {\top} \mathbf {M} _ {i} [ j ] |}{0 . 5 \| \mathbf {S} ^ {\prime} \mathbf {A x} \| _ {1}} \leq 3 (1 + \rho) \boldsymbol {\sigma} _ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} (\mathbf {M} _ {i} [ j ]),
$$
where the second step is by the definition of  $\mathbf{C}$  as a vertical concatenation of SA and M and further using that SA is a constant factor  $\ell_{1}$  subspace embedding of A and dropping the non-negative term  $\| \mathbf{Mx}\| _1$ . Combining Inequality (B.15) and Inequality (B.16), we see that for all rows  $j\in [||\mathbf{M}_i||]$ , we have
$$
\frac {1}{3 (1 + \rho)} \sqrt {\frac {2 ^ {- i}}{| \mathbf {C} |}} \leq \sigma_ {1} ^ {\mathbf {S} ^ {\prime} \mathbf {A}} \left(\mathbf {M} _ {i} [ j ]\right) \leq 3 (1 + \rho) \sqrt {2 ^ {- i + 1}}. \tag {B.19}
$$