SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {x} ^ {\star}, i ^ {\star} = \arg \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, i \in [ | \mathbf {A} | ]} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} | ^ {p}}{\| \mathbf {A x} \| _ {p} ^ {p}} \tag {C.7}
$$
$$
\max  _ {i: \mathbf {c} _ {i} \in \mathbf {S} _ {\infty} \mathbf {A}} \sigma_ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} (\mathbf {c} _ {i}) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {c} _ {k} \in \mathbf {S} _ {\infty} \mathbf {A}} \frac {| \mathbf {c} _ {k} ^ {\top} \mathbf {x} | ^ {p}}{\| \mathbf {S} _ {p} \mathbf {A x} \| _ {p} ^ {p}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {c} _ {k} \in \mathbf {S} _ {\infty} \mathbf {A}} \frac {| \mathbf {c} _ {k} ^ {\top} \mathbf {x} | ^ {p}}{\| \mathbf {A x} \| _ {p} ^ {p}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {a} _ {i} ^ {\top} \mathbf {x} | ^ {p}}{\| \mathbf {A x} \| _ {p} ^ {p}},
$$
For  $p = 2$ , the result of [23] gives a constant factor approximation to all leverage scores in  $\mathbf{nnz}(\mathbf{A}) + d^{\omega}$  time.
where the first step is by definition of  $\ell_p$  sensitivity of  $\mathbf{S}_{\infty}\mathbf{A}$  with respect to  $\mathbf{S}_p\mathbf{A}$ , the second step is by Equation (C.6), and the third step by noting that the matrix  $\mathbf{S}_{\infty}\mathbf{A}$  is a subset of the rows of  $\mathbf{A}$  (which includes  $\mathbf{a}_{i^{\star}}$ ). By definition of  $\sigma_{p}(\mathbf{a}_{i^{\star}})$  in Equation (C.7), we have
$$
\max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left| \mathbf {a} _ {i \star} ^ {\top} \mathbf {x} \right| ^ {p}}{\left\| \mathbf {A x} \right\| _ {p} ^ {p}} = \left\| \boldsymbol {\sigma} _ {p} (\mathbf {A}) \right\| _ {\infty}. \tag {C.9}
$$
Then, combining Equation (C.8) and Equation (C.9) gives the guarantee in this case. In the other case, suppose  $\mathbf{a}_{i^*}$  is not included in  $\mathbf{S}_{\infty}\mathbf{A}$ . Then we observe that the upper bound from the preceding inequalities still holds. For the lower bound, we observe that
$$
\left\| \boldsymbol {\sigma} _ {p} ^ {\mathbf {S} _ {p} \mathbf {A}} \left(\mathbf {S} _ {\infty} \mathbf {A}\right) \right\| _ {\infty} = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}, \mathbf {c} _ {j} \in \mathbf {S} _ {\infty} \mathbf {A}} \frac {\left| \mathbf {c} _ {j} ^ {\top} \mathbf {x} \right| ^ {p}}{\left\| \mathbf {S} _ {p} \mathbf {A x} \right\| _ {p} ^ {p}} \geq \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty} ^ {p}}{\left\| \mathbf {S} _ {p} \mathbf {A x} \right\| _ {p} ^ {p}} = \Theta (1) \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty} ^ {p}}{\left\| \mathbf {A x} \right\| _ {p} ^ {p}}, \tag {C.10}
$$
where the second step is by choosing a specific vector in the numerator and the third step uses  $\| \mathbf{S}_p\mathbf{A}\mathbf{x}\| _p^p = \Theta (\| \mathbf{A}\mathbf{x}\| _p^p)$ . We further have,
$$
\max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} \right\| _ {\infty} ^ {p}}{\left\| \mathbf {A x} \right\| _ {p} ^ {p}} \geq \frac {\left\| \mathbf {S} _ {\infty} \mathbf {A x} ^ {\star} \right\| _ {\infty} ^ {p}}{\left\| \mathbf {A x} ^ {\star} \right\| _ {p} ^ {p}} \geq \frac {\left\| \mathbf {A x} ^ {\star} \right\| _ {\infty} ^ {p}}{d ^ {p / 2} \left\| \mathbf {A x} ^ {\star} \right\| _ {p} ^ {p}} \geq \frac {\left| \mathbf {a} _ {i} ^ {\top} \mathbf {x} ^ {\star} \right| ^ {p}}{d ^ {p / 2} \left\| \mathbf {A x} ^ {\star} \right\| _ {p} ^ {p}} = \frac {1}{d ^ {p / 2}} \| \boldsymbol {\sigma} _ {p} (\mathbf {A}) \| _ {\infty}, \tag {C.11}
$$
where the first step is by choosing  $\mathbf{x} = \mathbf{x}^{\star}$ , the second step is by the guarantee of  $\ell_{\infty}$  subspace embedding, and the final step is by definition of  $\sigma_p(\mathbf{a}_{i^*})$ . Combining Inequality (C.10) and Inequality (C.11) gives the claimed lower bound on  $\| \pmb{\sigma}_p^{\mathbf{S}_p\mathbf{A}}(\mathbf{S}_\infty \mathbf{A})\|_\infty$ . The runtime follows from the cost of computing  $\mathbf{S}_{\infty}\mathbf{A}$  from [35] and the cost of computing and size of  $\mathbf{S}_p\mathbf{A}$  from [23, Figure 1]
D Lower Bounds
Theorem D.1 ( $\ell_p$  Regression Reduces to  $\ell_p$  Sensitivities). Suppose that we are given an algorithm  $\mathcal{A}$ , which for any matrix  $\mathbf{A}' \in \mathbb{R}^{n' \times d'}$  and accuracy parameter  $\varepsilon' \in (0, 1)$ , computes  $(1 \pm \varepsilon')\sigma_p(\mathbf{A}')$  in time  $\mathcal{T}(n', d', \mathbf{nnz}(\mathbf{A}')$ ,  $\varepsilon'$ ). Then, there exists an algorithm that takes  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and  $\mathbf{b} \in \mathbb{R}^n$  as inputs and computes  $(1 \pm \varepsilon)\min_{\mathbf{y} \in \mathbb{R}^d} \| \mathbf{A}\mathbf{y} - \mathbf{b} \|_p^p \pm \lambda^p\varepsilon$  in time  $\mathcal{T}(n + 1, d + 1, \mathbf{nnz}(\mathbf{A}) + \mathbf{nnz}(\mathbf{b}) + 1, \varepsilon)$  for any  $\lambda > 0$ .
Proof. Given  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and  $\mathbf{b} \in \mathbb{R}^d$ , consider the matrix  $\mathbf{A}' := \begin{bmatrix} \mathbf{A} & -\mathbf{b} \\ \mathbf{0}^\top & -\lambda \end{bmatrix} \in \mathbb{R}^{(n+1) \times d}$ . Then the  $n+1$ -th  $\ell_p$  sensitivity of  $\mathbf{A}'$  is
$$
\boldsymbol {\sigma} _ {p} (\mathbf {a} _ {n + 1} ^ {\prime}) = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \langle \mathbf {e} _ {d} , \mathbf {x} \rangle | ^ {p}}{\| \mathbf {A} ^ {\prime} \mathbf {x} \| _ {p} ^ {p}} = \max  _ {\mathbf {x} \in \mathbb {R} ^ {d}: x _ {d} = \lambda^ {- 1}} \frac {1}{\| \mathbf {A} ^ {\prime} \mathbf {x} \| _ {p} ^ {p}} = \frac {1}{1 + \min  _ {\mathbf {y} \in \mathbb {R} ^ {d - 1}} \left\{\| \mathbf {A} \mathbf {y} - \lambda^ {- 1} \mathbf {b} \| _ {p} ^ {p} \right\}},
$$
where the second step is by the scale-invariance of the definition of sensitivity with respect to the variable of optimization. By scale-invariance we see that for any  $c > 0$ , and  $\min_{\mathbf{y}}\| \mathbf{A}_{: - i}\mathbf{y} - c\mathbf{A}_{:i}\| _p^p = c^p\min_{\mathbf{y}}\| \mathbf{A}_{: - i}\mathbf{y} - \mathbf{A}_{:i}\| _p^p$
Therefore, note that rearranging gives  $\min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}\mathbf{y} - \mathbf{b} \|_p^p = \frac{\lambda^p}{\sigma_p(\mathbf{a}_{n+1}') - \lambda^p}$ , which is how we can estimate the cost of the regression. Then computing  $\sigma_p(\mathbf{a}_{n+1}')$  to a multiplicative accuracy of  $\varepsilon'$  gives the value of  $\min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}\mathbf{y} - \mathbf{b} \|_p^p$  up to error  $\varepsilon\left(\frac{\lambda^p}{\sigma_p(\mathbf{a}_{n+1}')}\right) = \varepsilon(\lambda^p + \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}\mathbf{y} - \mathbf{b} \|_p^p)$ , giving our final bounds.
Using the same technique as in Theorem D.1, we can similarly extend the reduction to a multiple regression task. In particular, we show that computing the values of a family of certain regularized leave-one-out regression problems for a matrix may be obtained by simply computing leverage scores of an associated matrix; therefore, this observation demonstrates that finding fast algorithms for sensitivities is as hard as multiple regression tasks. Specifically, for some matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , we denote  $\mathbf{A}_{: -i} \in \mathbb{R}^{n \times d-1}$  as the submatrix of  $\mathbf{A}$  with its  $i$ -th column, denoted as  $\mathbf{A}_{:i}$ , removed. We show that approximate sensitivity calculations can solve  $\min_{\mathbf{y} \in \mathbb{R}^{d-1}} \|\mathbf{A}_{:-i}\mathbf{y} + \mathbf{A}_{:i}\|_2$  approximately for all  $i$ .
Lemma D.2 (Regularized Leave-One-Out  $\ell_p$  Multiregression Reduces to  $\ell_p$  Sensitivities). Suppose that we are given an algorithm  $\mathcal{A}$ , which for any matrix  $\mathbf{A}' \in \mathbb{R}^{n' \times d'}$  and accuracy parameter  $\varepsilon' \in (0,1)$ , computes  $(1 \pm \varepsilon')\sigma_p(\mathbf{A}')$  in time  $\mathcal{T}(n',d',\mathbf{nnz}(\mathbf{A}')$ ,  $\varepsilon'$ . Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  with  $n \geq d$ , let  $OPT_i := \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}_{:-i}\mathbf{y} + \mathbf{A}_{:i} \|_p^p$  and  $\mathbf{y}_i^\star := \arg \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}_{:-i}\mathbf{y} + \mathbf{A}_{:i} \|_p$  for all the  $i \in [d]$ . Then, there exists an algorithm that takes  $\mathbf{A} \in \mathbb{R}^{n \times d}$  and computes  $(1 \pm \varepsilon) OPT_i \pm \lambda^p (1 + \| \mathbf{y}_i^\star \|_p^p)$  in time  $\mathcal{T}(n+d,d,\mathbf{nnz}(\mathbf{A}),\varepsilon)$  for any  $\lambda > 0$ .
Proof. Given  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , consider the matrix  $\mathbf{A}' := \begin{bmatrix} \mathbf{A} \\ \lambda \mathbf{I} \end{bmatrix} \in \mathbb{R}^{(n + d) \times d}$  formed by vertically appending a scaled identity matrix to  $\mathbf{A}$ , with  $\lambda > 0$ . By the definition of  $\ell_p$  sensitivities, we have the following.
$$
\begin{array}{l} \boldsymbol {\sigma} _ {p} (\mathbf {a} _ {n + i} ^ {\prime}) = \max _ {\mathbf {x} \in \mathbb {R} ^ {d}} \frac {| \mathbf {x} ^ {\top} \mathbf {a} _ {n + i} ^ {\prime} | ^ {p}}{\| \mathbf {A} ^ {\prime} \mathbf {x} \| _ {p} ^ {p}} = \max _ {\mathbf {x} \in \mathbb {R} ^ {d}: x _ {i} = \lambda^ {- 1}} \frac {1}{\| \mathbf {A} ^ {\prime} \mathbf {x} \| _ {p} ^ {p}} = \max _ {\mathbf {x} \in \mathbb {R} ^ {d}: x _ {i} = \lambda^ {- 1}} \frac {1}{\| \lambda^ {- 1} \mathbf {A} _ {: i} ^ {\prime} + \mathbf {A} _ {- : i} ^ {\prime} \mathbf {x} _ {- i} \| _ {p} ^ {p}} \\ = \frac {1}{\min  _ {\mathbf {y} \in \mathbb {R} ^ {d - 1}} \left\{1 + \lambda^ {p} \| \mathbf {y} \| _ {p} ^ {p} + \| \mathbf {A} _ {: - i} \mathbf {y} + \lambda^ {- 1} \mathbf {A} _ {: i} \| _ {p} ^ {p} \right\}} \\ \end{array}
$$
Recall that  $\mathbf{y}_i^\star \coloneqq \arg \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}_{: -i}\mathbf{y} + \mathbf{A}_{:i} \|_p$ , and since the power is a monotone transform,  $\mathrm{OPT}_i \coloneqq \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}_{: -i}\mathbf{y} + \mathbf{A}_{:i} \|_p = \| \mathbf{A}_{: -i}\mathbf{y}_i^\star + \mathbf{A}_{:i} \|_p$ . By scale-invariance we see that for any  $c > 0$ ,  $c\mathbf{y}_i^\star = \arg \min_{\mathbf{y} \in \mathbb{R}^{d-1}} \| \mathbf{A}_{: -i}\mathbf{y} + c\mathbf{A}_{:i} \|_p^p$  and  $\min_{\mathbf{y}} \| \mathbf{A}_{: -i}\mathbf{y} + c\mathbf{A}_{:i} \|_p^p = c^p\mathrm{OPT}_i$ . Therefore, by plugging  $\mathbf{y} = \lambda^{-1}\mathbf{y}_i^\star$ , note that
$$
\min  _ {\mathbf {y} \in \mathbb {R} ^ {d - 1}} \left\{1 + \lambda^ {p} \| \mathbf {y} \| _ {p} ^ {p} + \| \mathbf {A} _ {: - i} \mathbf {y} + \lambda^ {- 1} \mathbf {A} _ {: i} \| _ {p} ^ {p} \right\} \leq 1 + \| \mathbf {y} ^ {\star} \| _ {p} ^ {p} + \lambda^ {- p} \mathrm {O P T} _ {i}.
$$
However, we also note that by non-negativity,
$$
\min  _ {\mathbf {y} \in \mathbb {R} ^ {d - 1}} \left\{1 + \lambda^ {p} \| \mathbf {y} \| _ {p} ^ {p} + \| \mathbf {A}: _ {- i} \mathbf {y} + \lambda^ {- 1} \mathbf {A}: _ {i} \| _ {p} ^ {p} \right\} \geq \lambda^ {- p} \mathrm {O P T} _ {i}.
$$
Combining this all together gives the claimed approximation guarantee by noting that
$$
\mathrm {O P T} _ {i} + \lambda^ {p} \| \mathbf {y} ^ {\star} \| _ {p} ^ {p} + \lambda^ {p} \geq \frac {\lambda^ {p}}{\sigma_ {p} \left(\mathbf {a} _ {n + i} ^ {\prime}\right)} \geq \mathrm {O P T} _ {i}
$$
Therefore, our algorithm is to simply return  $\lambda^p / s_p(\mathbf{a}_{n + i}')$ , where  $s$  is the  $\epsilon$ -approximate estimate of the sensitivity and our bound follows. The runtime guarantees follow immediately from assumption that the runtime of calculating the  $\ell_p$  sensitivities of  $\mathbf{A}'$ .
Corollary D.3. Assuming that leave-one-out  $\ell_p$  multi-regression with  $\Omega(d)$  instances takes poly $(d)$  nnz(A) + poly $(1/\varepsilon)$  time [44, 45], computing  $\ell_p$  sensitivities of an  $n \times d$  matrix A costs at least poly $(d)$  nnz(A) + poly $(1/\varepsilon)$ .
Here we include additional experiments on other similar datasets.
p	Total Sensitivity Upper Bound	Brute-Force	Approximation	Brute-Force Runtime (s)	Approximate Runtime (s)
1	11	4.7	3.9	1940	303
1.5	11	8.3	10.3	1980	280
2.5	20	10.3	11.8	1970	326
3	36.4	6.9	7.3	1970	371
Table 2: Runtime comparison for computing total sensitivities for the fires dataset, which has matrix shape (517, 11). We include the theoretical upper bound for the total sensitivities using lewis weights calculations.
p	Total Sensitivity Upper Bound	Brute-Force	Approximation	Brute-Force Runtime (s)	Approximate Runtime (s)
1	11	4.1	5.2	540	154
1.5	11	10.1	6.7	560	201
2.5	20	19.9	12.2	390	117
3	36.4	8.8	6.3	354	136
Table 3: Runtime comparison for computing total sensitivities for the concrete dataset, which has matrix shape (101, 11). We include the theoretical upper bound for the total sensitivities using lewis weights calculations.
Sensitivity Approximations for Fires Dataset
Figure 2: Average absolute log ratios for all  $\ell_p$  sensitivity approximations for fires.