Title: Computing Approximate $\ell_p$ Sensitivities
Venue: NeurIPS 2023
Accepted: yes
Keywords: $\ell_p$ sensitivities

Lewis weights

leverage scores

approximation algorithms

total sensitivity
Abstract: Recent works in dimensionality reduction for regression tasks have introduced the notion of sensitivity, an estimate of the importance of a specific datapoint in a dataset, offering provable guarantees on the quality of the approximation after removing low-sensitivity datapoints via subsampling. However, fast algorithms for approximating sensitivities, which we show is equivalent to approximate regression, are known for only the $\ell_2$ setting, in which they are popularly termed leverage scores. In this work, we provide the first efficient algorithms for approximating $\ell_p$ sensitivities and other summary statistics of a given matrix. In particular, for a given $n \times d$ matrix, we compute $\alpha$-approximation to its $\ell_1$ sensitivities at the cost of $n/\alpha$ sensitivity computations. For estimating the total $\ell_p$ sensitivity (i.e. the sum of $\ell_p$ sensitivities), we provide an algorithm based on importance sampling of $\ell_p$ Lewis weights, which computes a constant factor approximation at the cost of roughly $\sqrt{d}$ sensitivity computations, with no polynomial dependence on $n$. Furthermore, we estimate the maximum $\ell_1$ sensitivity up to a $\sqrt{d}$ factor in $O(d)$ sensitivity computations. We also generalize these results to $\ell_p$ norms.  Lastly, we experimentally show that for a wide class of structured matrices in real-world datasets, the total sensitivity can be quickly approximated and is significantly smaller than the theoretical prediction, demonstrating that real-world datasets have on average low intrinsic effective dimensionality.

Computing Approximate ℓp Sensitivities

Abstract

Recent works in dimensionality reduction for regression tasks have introduced the notion of sensitivity, an estimate of the importance of a specific datapoint in a dataset, offering provable guarantees on the quality of the approximation after removing low-sensitivity datapoints via subsampling. However, fast algorithms for approximating sensitivities, which we show is equivalent to approximate regression, are known for only the ℓ 2 setting, in which they are popularly termed leverage scores. In this work, we provide the first efficient algorithms for approximating ℓ p sensitivities and related summary statistics of a given matrix. In particular, for a given n × d matrix, we compute α -approximation to its ℓ 1 sensitivities at the cost of O ( n/α ) sensitivity computations. For estimating the total ℓ p sensitivity (i.e. the sum of ℓ p sensitivities), we provide an algorithm based on importance sampling of ℓ p Lewis weights, which computes a constant factor approximation to the total sensitivity at the cost of roughly O ( √ d ) sensitivity computations. Furthermore, we estimate the maximum ℓ 1 sensitivity, up to a √ d factor, using O ( d ) sensitivity computations. We generalizeall these results to ℓ p norms for p > 1 . Lastly, we experimentally show that for a wide class of matrices in real-world datasets, the total sensitivity can be quickly approximated and is significantly smaller than the theoretical prediction, demonstrating that real-world datasets have low intrinsic effective dimensionality.

1 Introduction

Many modern large-scale machine learning datasets comprise tall matrices A ∈ R n × d with d features and n ≫ d training examples, often making it computationally expensive to run them through even basic algorithmic primitives. Therefore, many applications spanning dimension reduction, privacy, and fairness aim to estimate the importance of a given datapoint as a first step. Such importance estimates afford us a principled approach to focus our attention on a subset of only the most important examples.

In view of this benefit, several sampling approaches have been extensively developed in the machine learning literature. One of the simplest such methods prevalent in practice is uniform sampling. Prominent examples of this technique include variants of stochastic gradient descent methods [1-4] such as [5-14] developed for the 'finite-sum minimization' problem

where, in the context of empirical risk minimization, each f i : X ↦→ R ≥ 0 in (1.1) measures the loss incurred by the i -th data point from the training set, and in generalized linear models, each f i represents a link function applied to a linear predictor evaluated at the i -th data point. The aforementioned algorithms randomly sample a function f i and make progress via gradient estimation techniques.

However, uniform sampling can lead to significant information loss when the number of important training examples is relatively small. Hence, importance sampling methods that assign higher sampling probabilities to more important examples have emerged both in theory [15, 16] and practice [17-21]. One such technique is based on sensitivity scores , defined [15] for each coordinate i ∈ [ n ] as:

Sampling each f i independently with probability p i ∝ σ i and scaling the sampled row with 1 /p i preserves the objective in (1.1) in expectation for every x ∈ X . Further, one gets a (1 ± ε ) -approximation to this objective by sampling ˜ O ( S dε -2 ) functions [22, Theorem 1 . 1 ], where S = ∑ n i =1 σ i is called the total sensitivity , and d is an associated VC dimension.

Numerous advantages of sensitivity sampling over ℓ p Lewis weight sampling [23] were highlighted in the recent work [24], prominent among which is that a sampling matrix built using the ℓ p sensitivities of A is significantly smaller than one built using its ℓ p Lewis weights in many important cases, e.g., when the total sensitivity S is small for p > 2 (which is seen in many structured matrices, such as combinations of Vandermonde, sparse, and low-rank matrices as studied in [25] as well as those in many naturally occuring datasets (cf. Section 4)). Additionally, sensitivity sampling has found use in constructing ϵ -approximators for shape-fitting functions, also called strong coresets [15, 16, 26-28], clustering [16, 29, 30], logistic regression [31, 32], and least squares regression [33, 34].

Despite their benefits, the computational burden of estimating these sensitivities has been known to be significant, requiring solving an expensive maximization problem for each datapoint, which is a limiting factor in their use [35]. A key exception is for f i ( x ) = ( a ⊤ i x ) 2 , when the sensitivity score of f i is exactly the leverage score of a i , which can be computed quickly in O (log( n )) sensitivity calculations, as opposed to the naive O ( n ) sensitivity calculations (cf. Section 2).

1.1 Our contributions

In this work, we initiate a systematic study of algorithms for approximately computing the ℓ p sensitivities of a matrix, generally up to a constant factor. We first define ℓ p sensitivities:

Definition 1.1. [15] Given a full-rank matrix A ∈ R n × d with n ≥ d and a scalar p ∈ (0 , ∞ ) , let a i denote the i 'th row of matrix A . Then the vector of ℓ p sensitivities of A is defined as 1 (cf. Section 2 for the notation)

̸

and the total ℓ p sensitivity is defined as S p ( A ) def = ∑ i ∈ [ n ] σ p ( a i ) .

These ℓ p sensitivities yield sampling probabilities for ℓ p regression, a canonical optimization problem that captures least squares regression, maximum flow, and linear programming, in addition to appearing in applications like low rank matrix approximation [36], sparse recovery [37], graph-based semi-supervised learning [38-40], and data clustering [41]. However, algorithms for approximating ℓ p sensitivities were known for only p = 2 (when they are called 'leverage scores'). As is typical in large-scale machine learning, we assume n ≫ d . Wenow state our contributions, which are three-fold.

(1) Approximation algorithms. We design algorithms for three prototypical computational tasks associated with ℓ p sensitivities of a matrix A ∈ R n × d . Specifically, for an approximation parameter α > 1 , we can simultaneously estimate all ℓ p sensitivities up to an additive error of O ( α p n S p ( A ) ) via O ( n/α ) individual sensitivity calculations, reducing our runtime by sacrificing some accuracy. We limit most of our results in the main text to ℓ 1 sensitivities, but extend these to all p ≥ 1 in Appendix C. To state our results for the ℓ p setting, we use nnz ( A ) to denote the number of non-zero entries of the matrix A , ω for the matrix multiplication constant, and we introduce the following piece of notation.

Definition 1.2 (Notation for ℓ p Results) . We introduce the notation LP ( m,d,p ) to denote the cost of approximating one ℓ p sensitivity of an m × d matrix up to an accuracy of a given constant factor.

̸

Theorem 1.3 ( Estimating all sensitivities ; informal version of Theorem 3.1 and Theorem C.2) . Given a matrix A ∈ R n × d and 1 ≤ α ≪ n , there exists an algorithm (Algorithm 1) which returns a vector ˜ σ ∈ R n ≥ 0 such that, with high probability, for each i ∈ [ n ] , we have

Our algorithm's runtime is ˜ O ( nnz ( A ) + n α · d ω ) . More generally, for p ≥ 1 , we can compute (via Algorithm 6) an estimate ˜ σ ∈ R n ≥ 0 satisfying σ p ( a i ) ≤ ˜ σ i ≤ O ( α p -1 σ p ( a i ) + α p n S p ( A )) with high probability for each i ∈ [ n ] , at a cost of O ( nnz ( A ) + n α · LP ( O ( d max(1 ,p/ 2) ) , d, p ) ) .

Two closely related properties arising widely in algorithms research are the total sensitivity S 1 ( A ) and the maximum sensitivity ∥ σ 1 ( A ) ∥ ∞ . As stated earlier, one important feature of the total sensitivity is that it determines the total sample size needed for function approximation, thus making its efficient estimation crucial for fast ℓ 1 regression. Additionally, it was shown in [24] that the sample complexity of sensitivity sampling is much lower than that of Lewis weight sampling in many practical applications such as when the total sensitivity is small; therefore, an approximation of the total sensitivity can be used as a fast test for whether or not to proceed with sensitivity sampling (involving the costly task of calculating all sensitivities).

While the total sensitivity helps us characterize the whole dataset (as explained above), the maximum sensitivity captures the importance of the most important datapoint and is used in, e.g., experiment design and in reweighting matrices for low coherence [42]. Additionally, it captures the maximum extent to which a datapoint can influence the objective function and is therefore used in differential privacy [43]. While one could naively estimate the total and maximum sensitivities by computing all n sensitivities using Theorem 1.3, we give faster algorithms that, strikingly, have no polynomial dependence on n . In the assumed regime of n ≫ d , this is a significant runtime improvement.

Theorem 1.4 ( Estimating the total sensitivity ; informal version of Theorem 3.5) . Given a matrix A ∈ R n × d , a scalar p ≥ 1 , and a small constant γ < 1 , there exists an algorithm (Algorithm 2), which returns a positive scalar ̂ s such that, with a probability of at least 0 . 99 , we have

Our algorithm's runtime is ˜ O ( nnz ( A ) + 1 γ 2 · d | p/ 2 -1 | LP ( O ( d max(1 ,p/ 2) , d, p ) ) .

In Algorithm 4 (presented in Appendix B.2), we give an alternate method to estimate S 1 ( A ) based on recursive leverage score sampling, without ℓ p Lewis weight calculations, which we believe to be of independent interest. All of our algorithms utilize approximation properties of ℓ p Lewis weights and subspace embeddings and recent developments in fast computation of these quantities.

Theorem 1.5 ( Estimating the maximum sensitivity ; informal version of Theorem 3.7 and Theorem C.3) . Given a matrix A ∈ R n × d , there exists an algorithm (Algorithm 3) which returns a scalar σ > 0 such that

˜

Our algorithm's runtime is ˜ O ( nnz ( A ) + d ω +1 ) . Furthermore, via Algorithm 7, this result holds for all p ∈ [1 , 2] and for p > 2 , this guarantee costs ˜ O ( nnz ( A ) + d p/ 2 · LP ( O ( d p/ 2 ) , d, p ) ) .

(2) Hardness results. In the other direction, while it is known that ℓ p sensitivity calculation reduces to ℓ p regression, we show the converse. Specifically, we establish hardness of computing ℓ p sensitivities in terms of the cost of solving multiple corresponding ℓ p regression problems, by showing that a (1 + ε ) -approximation to ℓ p -sensitivities solves ℓ p multi-regression up to an accuracy factor of 1 + ε .

Theorem 1.6 ( Leave-One-Out ℓ p Multiregression Reduces to ℓ p Sensitivities ; informal 2 ) . Suppose that we are given an sensitivity approximation algorithm A , which for any matrix A ′ ∈ R n ′ × d ′ and accuracy parameter ε ′ ∈ (0 , 1) , computes (1 ± ε ′ ) σ p ( A ′ ) in time T ( n ′ , d ′ , nnz ( A ′ ) , ε ′ ) . Given a matrix A ∈ R n × d with n ≥ d , let OPT i := min y ∈ R d -1 ∥ A : -i y + A : i ∥ p p and y ⋆ i := arg min y ∈ R d -1 ∥ A : -i y + A : i ∥ p for all the i ∈ [ d ] . Then, there exists an algorithm that takes A ∈ R n × d and computes (1 ± ε ) OPT i for all i in time T ( n + d, d, nnz ( A ) , ε ) .

̸

For p = 2 , approximating all ℓ p sensitivities to a high accuracy would therefore require improving ℓ p multi-regression algorithms with Ω( d ) instances, for which the current best high-accuracy algorithms take poly ( d ) nnz ( A ) time [44, 45]. More concretely, our hardness results imply that in general one cannot compute all sensitivities as quickly as leverage scores unless there is a major breakthrough in multi-response regression. In order to show this, we introduce a reduction algorithm that efficiently regresses columns of A against linear combinations of other columns of A , by simply adding a row to A capturing the desired linear combination and then computing sensitivities, which will reveal the cost of the corresponding regression problem. We can solve Θ( d ) of these problems by augmenting the matrix to increase the rows and columns by at most a constant factor. We defer these details to Appendix D.

(3) Numerical experiments. We consolidate our theoretical contributions with a demonstration of the empirical advantage of our approximation algorithms over the naive approach of estimating these sensitivities using the UCI Machine Learning Dataset Repository [46] in Section 4. We found that many datasets have extremely small total sensitivities; therefore, fast sensitivity approximation algorithms utilize this small intrinsic dimensionality of real-world data far better than Lewis weights sampling, with sampling bounds often a factor of 2 -5 x better. We also found these sensitivities to be easy to approximate, with the accuracy-runtime tradeoff much better than our theory suggests, with up to 40 x faster in runtime. Lastly, we show that our algorithm for estimating the total sensitivity produces accurate estimates, up to small constant factors, with a runtime speedup of at least 4 x.

1.2 Related work

Introduced in the pioneering work of [15], sensitivity sampling falls under the broad framework of 'importance sampling'. It has found use in constructing ϵ -approximators of numerical integrands of several function classes in numerical analysis, statistics (e.g., in the context of VC dimension), and computer science (e.g., in coresets for clustering). The results from [15] were refined and extended by [16], with more advances in the context of shape-fitting problems in [26-28], clustering [16, 29, 30], logistic regression [31, 32], least squares regression [33, 34, 47], principal component analysis [48], reinforcement learning [49, 50], and pruning of deep neural networks [51-53]. Sampling algorithms for ℓ p subspace embeddings have also been extensively studied in functional analysis [54-59] as well as in theoretical computer science [23, 60, 44, 45].

Another notable line of work in importance sampling uses Lewis weights to determine sampling probabilities. As we explain in Section 2, Lewis weights are closely related to sensitivities. Many modern applications of Lewis weights in theoretical computer science we introduced in [23], which gave input sparsity time algorithms for approximating Lewis weights and used them to obtain fast algorithms for solving ℓ p regression. They have subsequently been used in row sampling algorithms for data pre-processing [33, 61, 62, 60, 23], computing dimension-free strong coresets for k -median and subspace approximation [63], and fast tensor factorization in the streaming model [64]. Lewis weights are also used for ℓ 1 regression in: [65] for stochastic gradient descent pre-conditioning, [66] for quantile regression, and [67] to provide algorithms for linear algebraic problems in the sliding window model. Lewis weights have also become widely used in convex geometry [68], randomized numerical linear algebra [23, 42, 69, 64, 70], and machine learning [71-73].

2 Notation and preliminaries

We use boldface uppercase and lowercase letters for matrices and vectors respectively. We denote the i th row vector of a matrix A by a i . Given a matrix M , we use Tr( M ) for its trace, rank( M ) for its rank, M † for its Moore-Penrose pseudoinverse, and | M | for the number of its rows. When x is an integer, we use [ x ] for the set of integers 1 , 2 , . . . , x . For two positive scalars x and y and a scalar α ∈ (0 , 1) , we use x ≈ α y to denote (1 -α ) y ≤ x ≤ (1 + α ) y ; we sometimes use x ≈ y to mean (1 -c ) y ≤ x ≤ (1 + c ) y for some appropriate universal constant c . We use ˜ O to hide dependence on n o (1) . We acknowledge that there is some notation overloading between p (when used to denote the scalar in, for example, ℓ p norms) and p i (when used to denote some probabilities) but the distinction is clear from context. We defer the proofs of facts stated in this section to Appendix A.

Notation for sensitivities. We start with a slightly general definition of sensitivities: σ B p ( a i ) to denote the ℓ p sensitivity of a i with respect to B ; when computing the sensitivity with respect to the

same matrix that the row is drawn from, we omit the matrix superscript:

When referring to the vector of ℓ p sensitivities of all the rows of a matrix, we omit the subscript in the argument: so, σ B p ( A ) is the vector of ℓ p sensitivities of rows of the matrix A , each computed with respect to the matrix B (as in Equation (2.1)), and analogously, σ p ( A ) is the vector of ℓ p sensitivities of the rows of matrix A , each computed with respect to itself. Similarly, the total sensitivity is the sum of ℓ p sensitivities of the rows of A with respect to matrix B , for which we use the following notation:

where, analogous to the rest of the notation regarding sensitivities, we omit the superscript when the sensitivities are being computed with respect to the input matrix itself.

While σ B p ( a i ) could be infinite, by appending a i to B , the generalized sensitivity becomes once again contained in [0 , 1] . Specifically, a simple rearrangement of the definition gives us the identity σ B ∪ a i p ( a i ) = 1 / (1 + 1 / σ B p ( a i )) . We now define leverage scores, which are a special type of sensitivities and also amenable to efficient computation [60], which makes them an ideal 'building block' in algorithms for approximating sensitivities.

Leverage scores are ℓ 2 sensitivities. The leverage score of the i th row a i of A is τ i ( A ) def = a ⊤ i ( A ⊤ A ) † a i . The i th leverage score is also the i th diagonal entry of the orthogonal projection matrix A ( A ⊤ A ) † A ⊤ . Since the eigenvalues of an orthogonal projection matrix are zero or one, it implies τ i ( A ) = 1 ⊤ i A ( A ⊤ A ) † A ⊤ 1 i ≤ 1 for all i ∈ [ n ] and ∑ n i =1 τ i ( A ) ≤ d (see [74]).

It turns out that τ i ( A ) = σ 2 ( a i ) (see [75]). This may be verified by applying a change of basis to the variable y = Ax in the definition of sensitivity and working out that the maximizer is the vector parallel to y = A † a i . This also gives an explicit formula for the total ℓ 2 sensitivity: S 2 ( A ) = rank ( A ) .

The fastest algorithm for constant factor approximation to leverage scores is by [60], as stated next.

Fact 2.1 ([60, Lemma 7]) . Given a matrix A ∈ R n × d , we can compute constant-factor approximations to all its leverage scores in time O ( nnz ( A ) log( n ) + d ω log( d )) .

In this paper, the above is the result we use to compute leverage scores (to a constant accuracy). Indeed, since the error accumulates multiplicatively across different steps of our algorithms, a high-accuracy algorithm for leverage score computation does not serve any benefit over this constant-accuracy one. Leverage scores approximate ℓ 1 sensitivities up to a distortion factor of √ n , as seen from the below known fact we prove, for completeness, in Appendix A.

Fact 2.2 (Crude sensitivity approximation via leverage scores) . Given a matrix A ∈ R n × d , let σ 1 ( a i ) denote the i th ℓ 1 sensitivity of A with respect to A , and let τ i ( A ) denote its i th leverage

In general, ℓ p sensitivities of rows of a matrix suffer a distortion factor of at most n | 1 / 2 -1 /p | from corresponding leverage scores , which follows from Hölder's inequality. The ℓ p generalizations of leverages scores are called ℓ p Lewis weights [54, 23], which satisfy the recurrence w i = τ i ( W 1 / 2 -1 /p A ) . However, unlike when p = 2 , ℓ p Lewis weights are not equal to sensitivities, and from an approximation perspective, they provide only a one-sided bound on the sensitivities σ A p ( a i ) ≤ d max(0 ,p/ 2 -1) w A p ( a i ) [75, 76]. While we have algorithms for efficiently computing leverage scores [60] and Lewis weights [23, 77], extensions for utilizing these ideas for fast and accurate sensitivity approximation algorithms for all p have been limited, which is the gap this paper aims to fill. A key regime of interest for many downstream algorithms is a small constant factor approximation to true sensitivities, such as for subsampling [78].

Subspace embeddings. We use sampling-based constant-approximation ℓ p subspace embeddings that we denote by S p . When the type of embedding is clear from context, we omit the subscript.

Definition 2.3 ( ℓ p Subspace Embedding ) . Given A ∈ R n × d (typically n ≫ d ), we call S ∈ R r × n (typically d ≤ r ≪ n ) an ϵ -approximate ℓ p subspace embedding of A if ∥ SAx ∥ p ≈ ε ∥ Ax ∥ p ∀ x ∈ R d .

When the diagonal matrix S is defined as S ii = 1 / √ p i with probability p i = Ω( τ i ( A )) (and 0 otherwise), then with high probability SA is an ℓ 2 subspace embedding for A [33, 79]; further, S has at most ˜ O ( dε -2 ) non-zero entries. For general p , we use Lewis weights to construct ℓ p subspace embeddings fast [23, Theorem 7.1]. This fast construction has been made possible by the reduction of Lewis weight computation to a few (polynomial in p ) leverage score computations for p < 4 [23] as well as p ≥ 4 [77]. This efficient construction is the reason for our choice of this subspace embedding. We note that there has been an extensive amount of literature on designing ℓ 1 subspace embeddings for example using Cauchy [80] and exponential [81] random variables.

For p ≤ 2 , it is known that the expected sample complexity of rows is ∑ n i =1 p i = O ( d ) [34], and for p > 2 , this is known to be at most O ( d p/ 2 ) [35]. While these bounds are optimal in the worst case, one important application of faster sensitivity calculations is to compute the minimal subspace embedding dimension for average case applications, particularly those seen in practice. This gap is captured by the total sensitivity S p ( A ) , which has been used to perform more sample efficient subspace embeddings [67, 27, 28], as sampling ˜ O ( S p ( A ) d/ϵ 2 ) rows suffice to provide an ℓ p subspace embedding.

Accurate sensitivity calculations. While leverage scores give a crude approximation, constantfactor approximations of ℓ p sensitivities seem daunting since we must compute worst-case ratios over the input space. However, we can compute the cost of one sensitivity specific row by reducing it to ℓ p regression since by scale invariance, we have 1 σ B p ( a i ) = min a ⊤ i x =1 ∥ Bx ∥ p p .

This reduction allows us to use recent developments in fast approximate algorithms for ℓ p regression [44, 25, 45], which utilize subspace embeddings to first reduce the number of rows of the matrix to O ( poly ( d )) . For the specific case of p = 1 , which is the focus of the main body of our paper, we may reduce to approximate ℓ 1 regression on a O ( d ) × d matrix, which is a form of linear programming.

Fact 2.4. Given an n × d matrix, the cost of computing k of its ℓ 1 sensitivities is ˜ O ( nnz ( A ) + k · d ω ) .

3 Approximating functions of sensitivities

We first present a constant-probability algorithm approximating the ℓ 1 sensitivities in Algorithm 1. Our algorithm is a natural one: Since computing the ℓ 1 sensitivities of all the rows simultaneously is computationally expensive, we instead hash α -sized blocks of random rows into smaller (constant-sized) buckets and compute the sensitivities of the smaller, O ( n/α ) -sized matrix P so generated, computing the sensitivities with respect to SA , the ℓ 1 subspace embedding of A . Running this algorithm multiple times gives our high-probability guarantee via the standard median trick.

Theorem 3.1. Given a full-rank matrix A ∈ R n × d and an approximation factor 1 < α ≪ n , let σ 1 ( a i ) be the ℓ 1 sensitivity of the i th row of A . Then there exists an algorithm that, in time ˜ O ( nnz ( A ) + n α · d ω ) , returns ˜ σ ∈ R n ≥ 0 such that with high probability, for each i ∈ [ n ] ,

Proof Sketch of Theorem 3.1; full proof in Appendix B.1. We achieve our guarantee via Algorithm 1. To see this, first note that ∥ SAx ∥ 1 ≈ Θ( ∥ Ax ∥ 1 ) (since as per Line 1, SA is an ℓ 1 subspace embedding of A ). Combining this with the fact that every row in A is mapped, via the designed randomness, to some row in the matrix P helps establish the desired approximation guarantee. The runtime follows from using Fact 2.4 to compute ℓ 1 sensitivities with respect to SA of | P | = O ( n/α ) rows.

As seen in Appendix C.1, our techniques described above also generalize to the p ≥ 1 case. Specifically, in Theorem C.2, we show that reducing ℓ p sensitivity calculations by an α factor gives an approximation guarantee of the form O ( α p -1 σ p ( a i )) + α p n S p ( A )) .

Remark 3.2. The sensitivities returned by our algorithm are approximate, with relative and additive error, but are useful in certain settings as they preserve ℓ p regression approximation guarantees while increasing total sample complexity by only a small poly ( d ) factor compared to true sensitivities while still keeping it much smaller than that obtained via Lewis weights. To see this with a simple toy

Algorithm 1 Approximating ℓ 1 -Sensitivities: Row-wise Approximation

example, consider the case p > 2 . Note that by Theorem 1 . 5 of [24], the sample complexity using approximate sensitivities is ˜ O ( α 2 p -2 S 2 -2 /p p ( A )) , using the α p factor blowup via Theorem C.2, that using true sensitivities is ˜ O ( S 2 -2 /p p ( A )) , and that using ℓ p Lewis weights is O ( d p/ 2 ) . Suppose also that the total sensitivity S p ( A ) = d . Further assume that n = d 10 and α = n 1 10 p = d 1 /p ; then the dimension-dependence of sample complexities given by our approximate sensitivities, true sensitivities, and Lewis weights are, respectively, ˜ O ( d 4 ) , ˜ O ( d 2 ) , and ˜ O ( d p/ 2 ) (ignoring small factors in the exponent that do not affect the asymptotic analysis). Thus, our approximate sensitivities provide a tradeoff in computational cost and sample complexity between using true sensitivities and Lewis weights.

3.1 Estimating the sum of ℓ p sensitivities

In this section, we present a one-shot algorithm that provides a constant-approximation to the total ℓ p sensitivity for all p ≥ 1 . This algorithm is based on importance sampling using ℓ p Lewis weights. For our desired guarantees, we crucially need the approximation bounds of ℓ p Lewis weights, as provided by Fact 3.3, efficient computation of ℓ p Lewis weights as provided by [23] for p < 4 and by [77] for p ≥ 4 , and the fact that a random ℓ p sampling matrix S (cf. Fact 3.4) yields SA , a constant-approximation subspace embedding to A [24, Theorem 1 . 8 ].

Algorithm 2 Approximating the Sum of ℓ p -Sensitivities for p ≥ 1

Input:

Matrix A ∈ R n × d , approximation factor γ ∈ (0 . 01 , 1) , and a scalar p ≥ 1

Output:

A positive scalar that satisfies, with probability 0 . 99 , that

1: Compute all ℓ p Lewis weights of A , denoting the i th weight by w p ( a i )

2: Define the sampling vector v ∈ R n ≥ 0 by v i = w p ( a i ) d for all i ∈ [ n ]

3: Sample m = O ( d | 1 -p/ 2 | ) rows with replacement, picking the i th row with a probability of v i

5: For each sampled row i j , j ∈ [ m ] , compute r j = σ S p A p ( a i j ) v i j

4: Construct a random ℓ p sampling matrix S p A with probabilities { v i } n i =1 (cf. Fact 3.4)

6: Return 1 m ∑ m j =1 r j

Fact 3.3 (Lewis Weights Bounds Sensitivities, [42, 82]) . Given a matrix A ∈ R n × d and p ≥ 1 , the ℓ p sensitivity σ A p ( a i ) and ℓ p Lewis weight w p ( a i ) of the row a i satisfy, for all i ∈ [ n ] ,

Fact 3.4 (Sampling via Lewis Weights [23, 35]) . Given A ∈ R n × d and p > 0 . Consider a random diagonal ℓ p sampling matrix S ∈ R n × n with sampling probabilities { p i } proportional to the ℓ p Lewis weight of A , i.e., for each i ∈ [ n ] , the i th diagonal entry is independently set to be

.

Then, with high probability, S with O ( ε -2 d max(1 ,p/ 2) (log d ) 2 log( d/ε )) rows is an ℓ p subspace embedding for A (cf. Definition 2.3).

Theorem 3.5. Given a matrix A ∈ R n × d and an approximation factor γ ∈ (0 , 1) , there exists an algorithm, which returns a positive scalar ̂ s such that, with a probability 0 . 99 , we have

Our algorithm's runtime is ˜ O ( nnz ( A ) + 1 γ 2 · d | p/ 2 -1 | LP ( O ( d max(1 ,p/ 2) , d, p ) ) .

Proof. Without loss of generality, we may assume A to be full rank. Then, its Lewis weights satisfy ∑ n i =1 w p ( a i ) = d . Per Line 2 of Algorithm 2, our sampling distribution is chosen to be v i = w p ( a i ) d for all i ∈ [ n ] . We sample from A rows with replacement, with row i picked with a probability of v i . From the definition of v i in Line 2 and r j in Line 5 and the fact that S p A is a constant factor ℓ p subspace embedding of A , our algorithm's output satisfies the following unbiasedness condition:

By independence, we also have the following bound on the variance of 1 m ∑ j ∈ [ m ] r j :

with the final step holding by the choice of v i in Line 2. In the case that p ≥ 2 , we have

where the first inequality uses Fact 3.3. Therefore, applying Chebyshev's inequality on 1 m ∑ j ∈ [ m ] r j (as defined in Line 5) with m = O ( d p/ 2 S p ( A ) γ 2 ) gives us a γ -multiplicative accuracy in approximating S p ( A ) (with the desired constant probability). For p ≥ 2 , we additionally have [24, Theorem 1 . 7 ] the lower bound S p ( A ) ≥ d , which when plugged into the value of m gives the claimed sample complexity. A similar argument may be applied for the case p < 2 ; specifically, we have that

where the second step is by w p ( a i ) ≥ σ p ( a i ) from Fact 3.3. For p < 2 , we also have S p ( A ) ≥ d p/ 2 from [24, Theorem 1 . 7 ]. Applying Chebyshev's inequality with this fact gives a sample complexity of m = O ( d 1 -p/ 2 /γ 2 ) . This completes the correctness guarantee.

Runtime. We first compute all ℓ p Lewis weights of A ∈ R n × d up to a constant multiplicative accuracy, the cost of which is O ( 1 1 -| 1 -p/ 2 | log(log( n )) ) leverage score computations for p < 4 [23] and O ( p 3 log( np )) leverage score computations for p ≥ 4 [77]. Next, we compute m = O ( d | 1 -p/ 2 | ) sensitivities with respect to S p A . From [23, 35], S p has, with high probability, O ( d p/ 2 log d ) rows when p > 2 and O ( d log d ) rows when p ≤ 2 . Summing the cost of computing these m sensitivities and that of computing the Lewis weights gives the claimed runtime.

Remark 3.6. We present in Appendix B.2 an alternate algorithm, Algorithm 4, for estimating the total ℓ p sensitivity for p = 1 . Algorithm 4 uses recursive computations of leverage scores, in contrast to Algorithm 2 which uses Lewis weights in a one-shot manner, and may be of independent interest.

3.2 Approximating the maximum ℓ 1 sensitivity

In this section, we present an algorithm that approximates ∥ σ 1 ( A ) ∥ ∞ = max i ∈ [ | A | ] σ 1 ( a i ) , the maximum of the ℓ 1 sensitivities of the rows of a matrix A . As alluded to in Section 1, a first approach to this problem would be to simply estimate all n sensitivities and compute their maximum. To do better than this, one idea, inspired by the random hashing approach of Algorithm 1, is as follows.

If the matrix has a large number of high-sensitivity rows, then, intuitively, the appropriately scaled maximum sensitivity of a uniformly sampled subset of these rows should approximate ∥ σ 1 ( A ) ∥ ∞ . Specifically, assume that the matrix has at least α rows of sensitivity at least ∥ σ 1 ( A ) ∥ ∞ / √ α ; uniformly sample n/α rows and return ˜ σ a , the (appropriately scaled) maximum of σ S 1 A 1 ( a i ) , for the sampled rows i . Then, ∥ σ 1 ( A ) ∥ ∞ ≤ ˜ σ a ≤ O ( √ α ∥ σ 1 ( A ) ∥ ∞ ) with a constant probability. Here the upper bound guarantee is without any condition on the number of rows with large ℓ 1 sensitivities.

In the other case, if the matrix does not have too many high-sensitivity rows, we could estimate the maximum sensitivity by hashing rows into small buckets via Rademacher combinations of uniformly sampled blocks of α rows each (just like in Algorithm 1). Then, ˜ σ b , the scaled maximum of the ℓ 1 sensitivities of these rows satisfies ∥ σ 1 ( A ) ∥ ∞ ≤ ˜ σ b ≤ O ( √ α ∥ σ 1 ( A ) ∥ ∞ ) . Here, it is the lower bound that comes for free (i.e., without any condition on the number of rows with large ℓ 1 sensitivities).

Despite the above strategies working for each case, there is no way to combine these approaches without knowledge of whether the input matrix has a enough 'high-sensitivity' rows or not. We therefore avoid this approach and instead present Algorithm 3, where we make use of S ∞ A , an ℓ ∞ subspace embedding of A . Our algorithm hinges on the recent development [75, Theorem 1.3] on efficient construction of such an embedding with simply a subset of ˜ O ( d ) rows of A .

Algorithm 3 Approximating the Maximum of ℓ 1 -Sensitivities

Input:

Matrix A ∈

R n × d

Output: Scalar ̂ s ∈ R ≥ 0 that satisfies ∥ σ 1 ( A ) ∥ ∞ ≤ ̂ s ≤ C · √ d · ∥ σ 1 ( A ) ∥ ∞

1: Compute, for A , an ℓ ∞ subspace embedding S ∞ A ∈ R O ( d log 2 ( d )) × d such that S ∞ A is a subset of the rows of A [75, Theorem 1 . 3 ]

2: Compute, for A , an ℓ 1 subspace embedding S 1 A ∈ R O ( d ) × d √

3: Return d ∥ σ S 1 A 1 ( S ∞ A ) ∥ ∞

Theorem 3.7 ( Approximating the Maximum of ℓ 1 Sensitivities ) . Given a matrix A ∈ R n × d , there exists an algorithm, which in time ˜ O ( nnz ( A ) + d ω +1 ) , outputs a positive scalar ̂ s that satisfies

Proof Sketch of Theorem 3.7; full proof in Appendix B.3. We achieve our guarantee via Algorithm 3. Define x ⋆ and a i ⋆ as: x ⋆ , i ⋆ = arg max x ∈ R d ,i ∈ [ | A | ] | a ⊤ i x | ∥ Ax ∥ 1 . Since S 1 A is an ℓ 1 subspace embedding of A , we have, for any x ∈ R d , that ∥ S 1 Ax ∥ 1 = Θ( ∥ Ax ∥ 1 ) . If S ∞ A contains the row a i ⋆ , then ∥ S 1 Ax ∥ 1 =Θ( ∥ Ax ∥ 1 ) implies ∥ σ S 1 A 1 ( S ∞ A ) ∥ ∞ = Θ(1) ∥ σ 1 ( A ) ∥ ∞ . In the other case, suppose a i ⋆ is not included in S ∞ A . Then we observe that

where the the second step is by choosing a specific vector in the numerator and the third step uses ∥ S 1 Ax ∥ 1 = Θ( ∥ Ax ∥ 1 ) . We further have,

where the first step is by choosing x = x ⋆ , the second step is by the guarantee of ℓ ∞ subspace embedding, and the final step is by definition of σ 1 ( a i ⋆ ) . Combining Inequality (3.2) and Inequality (3.3) gives the claimed lower bound on ∥ σ S 1 A 1 ( S ∞ A ) ∥ ∞ . The runtime follows from the cost of computing S ∞ A from [75] and that of computing ˜ O ( d ) of ℓ 1 sensitivities with respect to S 1 A .

p = 1

p = 1.5

p = 2.5

p = 3

5

additive bound

average

8

7

average

additive bound

12

additive bound

average

14

16

average

additive bound

6

10

12

3

5

8

10

4

8

6

Z

3

6

4

Z

5

10

Approximation Factor (alpha)

15

20

25

30

35

40

5

10

Approximation Factor (alpha)

15

20

25

30

35

40

5

10

Approximation Factor (alpha)

15

20

25

30

35

40

5

10

Approximation Factor (alpha)

15

20

25

30

35

40

4 Numerical experiments

We demonstrate our fast sensitivity approximations on multiple real-world datasets in the UCI Machine Learning Dataset Repository [46], such as the wine and fires datasets, for different p and varying approximation parameters α . We focus on the wine dataset here, with full details in Appendix E.

Experimental Setup. For each dataset and each p , we first apply the Lewis weight subspace embedding to compute a smaller matrix SA . Then, we run our sensitivity computation Algorithm 1 and measure the average and maximum absolute log ratio | log( σ approx /σ true ) | to capture the relative error of the sensitivity approximation. Note that this is stricter than our guarantees, which give only an O ( α ) additive error; therefore we have no real upper bound on the maximum absolute log ratio due to small sensitivities. We plot an upper bound of log( α p ) , which is the worst case additive error approximation although it does provide relative error guarantees with respect to the average sensitivity. We compare our fast algorithm for computing the total sensitivity with the naive brute-force method by approximating all sensitivities and then averaging.

Analysis. In practice, we found most real-world datasets have easy-to-approximate sensitivities with total ℓ p sensitivity about 2 -5 times lower than the theoretical upper bound. Figure 1 shows that the average absolute log ratios for approximating the ℓ p sensitivities are much smaller than those suggested by the theoretical upper bound, even for large α . Specifically, when α = 40 , we incur only a 16 x accuracy deterioration in exchange for a 40 x faster algorithm. This is significantly better than the worst-case accuracy guarantee which for p = 3 would be α p = 40 3 .

More importantly, we find that empirical total sensitivities are much lower than the theoretical upper bounds suggest, often by a factor of at least 5 , especially for p > 2 . This implies that real-world structure can be utilized for improved data compression and justifies the importance of sensitivity estimation for real-world datasets. Furthermore, our novel algorithm approximates the total sensitivity up to an overestimate within a factor of 1 . 3 , in less than a quarter of the time of the brute-force algorithm. Our observation generalizes to other real-world datasets (see Appendix E).

References

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , 1951. 1

Léon Bottou and Yann Cun. Large scale online learning. Advances in neural information processing systems , 16, 2003.

Tong Zhang. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the twenty-first international conference on Machine learning , page 116, 2004.

Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade , pages 421-436. Springer, 2012. 1

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology , 2:27:1-27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm . 1

Nicolas Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. Advances in neural information processing systems , 25, 2012.

Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research , 14(2), 2013.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems , 26, 2013.

Mehrdad Mahdavi, Lijun Zhang, and Rong Jin. Mixed optimization for smooth functions. Advances in neural information processing systems , 26, 2013.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

Julien Mairal. Incremental majorization-minimization optimization with application to largescale machine learning. SIAM Journal on Optimization , 25(2), 2015.

Zeyuan Allen-Zhu and Yang Yuan. Improved svrg for non-strongly-convex or sum-of-nonconvex objectives. In International conference on machine learning , 2016.

Elad Hazan and Haipeng Luo. Variance-reduced and projection-free stochastic optimization. In International Conference on Machine Learning , 2016.

Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162(1), 2017. 1

Michael Langberg and Leonard J Schulman. Universal ε -approximators for integrals. In Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms , 2010. 2, 4

Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing , 2011. 2, 4

Deanna Needell, Rachel Ward, and Nati Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014. URL https://proceedings.neurips.cc/paper_files/ paper/2014/file/f29c21d4897f78948b91f03172341b7b-Paper.pdf . 2

Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In Proceedings of the 32nd International Conference on Machine Learning , pages 1-9, 2015.

Angelos Katharopoulos and François Fleuret. Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.00043 , 2017.

Tyler B. Johnson and Carlos Guestrin. Training deep models faster with robust, approximate importance sampling. In Advances in Neural Information Processing Systems 31 , 2018.

Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning , 2022. 2

Vladimir Braverman, Dan Feldman, Harry Lang, Adiel Statman, and Samson Zhou. New frameworks for offline and streaming coreset constructions. arXiv preprint arXiv:1612.00889 , 2016. 2

Michael B Cohen and Richard Peng. Lp row sampling by lewis weights. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , 2015. 2, 4, 5, 6, 7, 8, 28, 29

David Woodruff and Taisuke Yasuda. Sharper bounds for ℓ p sensitivity sampling. In International Conference on Machine Learning , pages 37238-37272. PMLR, 2023. 2, 3, 7, 8

Raphael Meyer, Cameron Musco, Christopher Musco, David P Woodruff, and Samson Zhou. Fast regression for structured inputs. In International Conference on Learning Representations (ICLR) , 2022. 2, 6

Kasturi Varadarajan and Xin Xiao. On the sensitivity of shape fitting problems. In 32nd International Conference on Foundations of Software Technology and Theoretical Computer Science , 2012. 2, 4

Vladimir Braverman, Dan Feldman, Harry Lang, Adiel Statman, and Samson Zhou. Efficient coreset constructions via sensitivity sampling. Proceedings of Machine Learning Research , 157 (2021), 2021. 6

Murad Tukan, Xuan Wu, Samson Zhou, Vladimir Braverman, and Dan Feldman. New coresets for projective clustering and applications. In International Conference on Artificial Intelligence and Statistics , 2022. 2, 4, 6

Olivier Bachem, Mario Lucic, and Andreas Krause. Coresets for nonparametric estimation-the case of dp-means. In International Conference on Machine Learning , 2015. 2, 4

Mario Lucic, Olivier Bachem, and Andreas Krause. Strong coresets for hard and soft bregman clustering with applications to exponential family mixtures. In Artificial intelligence and statistics , 2016. 2, 4

Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances in Neural Information Processing Systems , 29, 2016. 2, 4

Alexander Munteanu, Chris Schwiegelshohn, Christian Sohler, and David Woodruff. On coresets for logistic regression. Advances in Neural Information Processing Systems , 31, 2018. 2, 4

Petros Drineas, Michael W Mahoney, and Shan Muthukrishnan. Sampling algorithms for ℓ 2 regression and applications. In Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm , 2006. 2, 4, 6

Michael W Mahoney et al. Randomized algorithms for matrices and data. Foundations and Trends® in Machine Learning , 3(2), 2011. 2, 4, 6

David P Woodruff and Taisuke Yasuda. Online lewis weight sampling. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , 2023. 2, 6, 8, 26, 28, 29

Flavio Chierichetti, Sreenivas Gollapudi, Ravi Kumar, Silvio Lattanzi, Rina Panigrahy, and David P Woodruff. Algorithms for ℓ p low-rank approximation. In International Conference on Machine Learning , 2017. 2

Emmanuel J Candes and Terence Tao. Decoding by linear programming. IEEE transactions on information theory , 51(12), 2005. 2

Morteza Alamgir and Ulrike Luxburg. Phase transition in the family of p -resistances. Advances in neural information processing systems , 24, 2011. 2

Jeff Calder. Consistency of lipschitz learning with infinite unlabeled data and finite labeled data. SIAM Journal on Mathematics of Data Science , 1(4), 2019.

Mauricio Flores, Jeff Calder, and Gilad Lerman. Analysis and algorithms for ℓ p -based semisupervised learning on graphs. Applied and Computational Harmonic Analysis , 60, 2022. 2

Abderrahim Elmoataz, Matthieu Toutain, and Daniel Tenbrinck. On the p -laplacian and ∞ -laplacian on graphs with applications in image and data processing. SIAM Journal on Imaging Sciences , 8(4), 2015. 2

Kenneth Clarkson, Ruosong Wang, and David Woodruff. Dimensionality reduction for tukey regression. In International Conference on Machine Learning , 2019. 3, 4, 7

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014. 3

Deeksha Adil, Richard Peng, and Sushant Sachdeva. Fast, provably convergent irls algorithm for p-norm linear regression. Advances in Neural Information Processing Systems , 32, 2019. 4, 6, 30

Arun Jambulapati, Yang P Liu, and Aaron Sidford. Improved iteration complexities for overconstrained p-norm regression. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , 2022. 4, 6, 30

Arthur Asuncion and David Newman. Uci machine learning repository, 2007. 4, 10

Anant Raj, Cameron Musco, and Lester Mackey. Importance sampling via local sensitivity. In International Conference on Artificial Intelligence and Statistics , 2020. 4

Alaa Maalouf, Adiel Statman, and Dan Feldman. Tight sensitivity bounds for smaller coresets. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2051-2061, 2020. 4

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020. 4

Dingwen Kong, Ruslan Salakhutdinov, Ruosong Wang, and Lin F Yang. Online sub-sampling for reinforcement learning with general function approximation. arXiv preprint arXiv:2106.07203 , 2021. 4

Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations , 2019. 4

Murad Tukan, Loay Mualem, and Alaa Maalouf. Pruning neural networks via coresets and convex geometry: Towards no assumptions. Advances in Neural Information Processing Systems , 2022.

Ben Mussay, Dan Feldman, Samson Zhou, Vladimir Braverman, and Margarita Osadchy. Dataindependent structured pruning of neural networks via coresets. IEEE Transactions on Neural Networks and Learning Systems , 33(12):7829-7841, 2021. 4

D Lewis. Finite dimensional subspaces of ℓ p . Studia Mathematica , 1978. 4, 5

Gideon Schechtman. More on embedding subspaces of ℓ p in ℓ n r . Compositio Mathematica , 1987.

Jean Bourgain, Joram Lindenstrauss, and Vitali Milman. Approximation of zonoids by zonotopes. 1989.

Michel Talagrand. Proceedings of the American Mathematical Society , 1990.

Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes , volume 23. Springer Science & Business Media, 1991.

Gideon Schechtman and Artem Zvavitch. Embedding subspaces of ℓ p into ℓ n p , 0 < p < 1 . Mathematische Nachrichten , 2001. 4

Michael B Cohen, Yin Tat Lee, Cameron Musco, Christopher Musco, Richard Peng, and Aaron Sidford. Uniform sampling for matrix approximation. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science , 2015. 4, 5, 19

Petros Drineas, Malik Magdon-Ismail, Michael W Mahoney, and David P Woodruff. Fast approximation of matrix coherence and statistical leverage. The Journal of Machine Learning Research , 13(1), 2012. 4

Mu Li, Gary L Miller, and Richard Peng. Iterative row sampling. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , 2013. 4

Christian Sohler and David P Woodruff. Strong coresets for k-median and subspace approximation: Goodbye dimension. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , 2018. 4

Rachit Chhaya, Jayesh Choudhari, Anirban Dasgupta, and Supratim Shit. Streaming coresets for symmetric tensor factorization. In International Conference on Machine Learning , 2020. 4

David Durfee, Kevin A Lai, and Saurabh Sawlani. ℓ 1 regression using lewis weights preconditioning and stochastic gradient descent. In Conference On Learning Theory , 2018. 4

Yi Li, Ruosong Wang, Lin Yang, and Hanrui Zhang. Nearly linear row sampling algorithm for quantile regression. In Proceedings of the 37th International Conference on Machine Learning , 2020. 4

Vladimir Braverman, Petros Drineas, Cameron Musco, Christopher Musco, Jalaj Upadhyay, David P Woodruff, and Samson Zhou. Near optimal linear algebra in the online and sliding window models. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , 2020. 4, 6

Aditi Laddha, Yin Tat Lee, and Santosh Vempala. Strong self-concordance and sampling. In Proceedings of the 52nd annual ACM SIGACT symposium on theory of computing , 2020. 4

Yi Li, Ruosong Wang, Lin Yang, and Hanrui Zhang. Nearly linear row sampling algorithm for quantile regression. In International Conference on Machine Learning , 2020. 4

Arvind V Mahankali and David P Woodruff. Optimal ℓ 1 column subset selection and a fast ptas for low rank approximation. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA) , 2021. 4

Tung Mai, Cameron Musco, and Anup Rao. Coresets for classification-simplified and strengthened. Advances in Neural Information Processing Systems , 34, 2021. 4

Xue Chen and Michal Derezinski. Query complexity of least absolute deviation regression via robust uniform convergence. In Conference on Learning Theory , 2021.

Aditya Parulekar, Advait Parulekar, and Eric Price. L1 regression with lewis weights subsampling. Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques , 2021. 4

Frederic G Foster. On the stochastic matrices associated with certain queuing processes. The Annals of Mathematical Statistics , 24(3), 1953. 5

David P Woodruff and Taisuke Yasuda. High-dimensional geometric streaming in polynomial space. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 732-743, 2022. 5, 9, 28

David P Woodruff Cameron Musco, Christopher Musco, David P Woodruff, and Taisuke Yasuda. Active sampling for linear regression beyond the l2 norm. CoRR , 2021. 5

Maryam Fazel, Yin Tat Lee, Swati Padmanabhan, and Aaron Sidford. Computing lewis weights to high precision. In Proceedings of the 2022 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , 2022. 5, 6, 7, 8

David P Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science , 10(1-2), 2014. 5

Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. Journal of the ACM (JACM) , 2007. 6

Christian Sohler and David P Woodruff. Subspace embeddings for the ℓ _ 1 -norm with applications. In Proceedings of the forty-third annual ACM symposium on Theory of computing , pages 755-764, 2011. 6

David Woodruff and Qin Zhang. Subspace embeddings and ℓ _ p -regression using exponential random variables. In Conference on Learning Theory , pages 546-567. PMLR, 2013. 6

Cameron Musco, Christopher Musco, David P Woodruff, and Taisuke Yasuda. Active linear regression for ℓ p norms and beyond. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , 2022. 7

Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM) , 68(1), 2021. 17

Roman Vershynin. High-dimensional probability: An introduction with applications in data science . Cambridge university press, 2020. 17

A Omitted proofs: general technical results

Fact A.1 (Crude sensitivity approximation via leverage scores) . Given a matrix A ∈ R n × d , let σ 1 ( a i ) denote the i th ℓ 1 sensitivity of A with respect to A , and let τ i ( A ) denote its i th leverage score with respect to A . Then we have √ τ i ( A ) n ≤ σ 1 ( a i ) ≤ √ τ i ( A ) .

Proof. The proof of this claim follows from a simple application of standard norm inequalities, and we present one here for completeness. For any u ∈ R n , we have ∥ u ∥ 2 ≤ ∥ u ∥ 1 ≤ √ n ∥ u ∥ 2 . Therefore, for any x ∈ R d , we have

Let x τ be the vector that realizes the i th leverage score, and x σ be the vector that realizes the i th ℓ 1 sensitivity (i.e. they are the maximizers of sensitivity). Then, we may conclude from Inequality (A.1) that

which gives the claimed result.

Fact A.2. Given an n × d matrix, the cost of computing k of its ℓ 1 sensitivities is ˜ O ( nnz ( A )+ k · d ω ) .

Proof. Given an n × d matrix A , we first compute SA , its ℓ 1 subspace embedding of size O ( d ) × d , and compute the i th sensitivity of A as follows.

where the second step is by the definition of ℓ 1 subspace embedding (cf. Definition 2.3). To compute σ SA 1 ( a i ) , we need to compute min a ⊤ i x =1 ∥ SAx ∥ 1 , which, by introducing a new variable for each of the rows of SAx , can be transformed into a linear program with O ( d ) variables and d constraints. To see the claim on runtime, the cost of computing the subspace embedding is nnz ( A ) . Having computed this once, we can then solve k linear programs, each at cost d ω (which is the current fastest LP solver [83]). Putting together these two costs gives the claimed runtime.

Lemma A.3. Given positive numbers a 1 , a 2 , . . . , a m , let r := max i ∈ [ m ] a i min i ∈ [ m ] a i and A ( true ) := ∑ i ∈ [ m ] a i . Suppose we sample, uniformly at random with replacement, a set S ∈ [ m ] of these numbers, and let A ( est ) := m | S | · ∑ i : a i ∈ S a i . Then, if | S | ≥ 10 ( r (1+ γ ) γ 2 log ( 1 δ ) ) for a large enough absolute constant C , we can ensure with at least a probability of 1 -δ , that | A ( est ) -A ( true ) | ≤ γA ( true ) .

Proof. By construction, the expected value of a sample a i in S is 1 m A ( true ) . Denoting by P the uniform distribution over a 1 , a 2 , . . . , a m , and let σ ( P ) be the standard deviation of this distribution P . Then, we can apply Bernstein's inequality [84, Theorem 2 . 8 . 1 ] to see that the absolute error | A ( est ) -A ( true ) | satisfies the following guarantee for all t > 0 :

Since each of the a i s is positive, we note that

Combining Inequality (A.2) and Inequality (A.3) for t = γA (true) implies that

holds for the choice of | S | ≥ 10 ( m γ 2 (1 + γ ) max i ∈ [ m ] a i A ( true ) log ( 1 δ ) ) yields the claimed error guarantee.

B Omitted proofs: ℓ 1 sensitivities

B.1 Estimating all ℓ 1 sensitivities

Theorem 3.1. Given a full-rank matrix A ∈ R n × d and an approximation factor 1 < α ≪ n , let σ 1 ( a i ) be the ℓ 1 sensitivity of the i th row of A . Then there exists an algorithm that, in time ˜ O ( nnz ( A ) + n α · d ω ) , returns ˜ σ ∈ R n ≥ 0 such that with high probability, for each i ∈ [ n ] ,

Proof. Note that in Line 2 of Algorithm 1, the rows of A are partitioned into randomly created n/α blocks. Suppose a i , the i th row of A , falls into the block B ℓ . Recall, that in Line 5, the rows from B ℓ are mapped to those in P with row indices in the set J . Then, we compute the i th ℓ 1 sensitivity estimate as ˜ σ i = max j ∈ J σ SA 1 ( p j ) . We observe that for all x ∈ R d ,

We use Equation (B.1) below to establish the claimed bounds. Let x ⋆ = arg max x ∈ R d | a ⊤ i x | ∥ Ax ∥ 1 be the vector that realizes the i th ℓ 1 sensitivity of A . Further, let the j th row of P be r ( ℓ ) k B ℓ . Then, with a probability of at least 1 / 2 , we have

̸

where the first step is by definition of σ SA 1 ( p j ) , the second is by evaluating the function being maximized at a specific choice of x , the third step is by applying ∥ SAx ∥ 1 = Θ( ∥ Ax ∥ 1 ) from Equation (B.1), and the final step is by definition of x ⋆ and σ 1 ( a i ) . For the fourth step, we use the fact that | r ℓ k B ℓ x ⋆ | = | r ℓ k,i a ⊤ i x ⋆ + ∑ j = i r ℓ k,j ( B ℓ x ⋆ ) j | ≥ | a ⊤ i x ⋆ | with a probability of at least 1 / 2 since the vector r ℓ k has coordinates that are +1 or -1 with equal probability. By a union bound over the | J | independent rows that block B ℓ is mapped to, we establish the claimed lower bound in Inequality (B.2) with a probability of at least 0 . 9 . To show an upper bound on σ SA 1 ( p j ) , we observe that

where the second step is by Hölder inequality and the fact that the entries of r ( ℓ ) k are all bounded between -1 and +1 , and the third step is by Equation (B.1). The final term of the above chain of inequalities may be rewritten as Θ(1) ∑ j : a j ∈ B ℓ s j ( A ) , by the definition of B ℓ . Because B ℓ is a group of α rows selected uniformly at random out of n rows and contains the row a i , we have:

̸

̸

Therefore, Markov inequality gives us that with a probability of at least 0 . 9 , we have

The median trick then establishes the bounds in Inequality (B.2) and Inequality (B.4) with high probability. The claimed runtime is obtained by the cost of constructing SA plus O ( n/α ) computations of ℓ 1 sensitivities with respect to SA , for which we may use Fact 2.4.

B.2 Estimating the sum of ℓ 1 sensitivities

In this section, we present a randomized algorithm that approximates the total sensitivity S 1 ( A ) , up to a γ -multiplicative approximation for some γ ∈ (0 , 1) that is not too small. This algorithm is significantly less general than Algorithm 2 (which holds for all p ≥ 1 ) but it uses extremely simple facts about leverage scores and what we think is a new recursion technique in this context, which we believe could be of potential independent interest.

Theorem B.1. Given a matrix A ∈ R n × d and an approximation factor γ ∈ (0 , 1) such that γ ≥ Ω ( 1 n 3 ) , there exists an algorithm, which in time ˜ O ( nnz ( A ) + d ω γ 2 · max ( √ d, 1 γ 2 )) , returns a positive scalar ̂ s such that, with a probability of 0 . 99 , we have

Algorithm 4 Approximating the Sum of ℓ 1 -Sensitivities

Input:

Matrix A ∈ R n × d and approximation factor γ ∈ (0 , 1)

Output: A positive scalar that satisfies, with probability 0 . 99 , that

1: Compute SA ∈ R O ( d ) × d , a 1 / 2 -approximate ℓ 1 subspace embedding of A (cf. Definition 2.3)

2: Set ρ = O ( γ/ log(log( n + d ))) . Compute S ′ A , a ρ -approximate ℓ 1 subspace embedding of A

3: Construct a matrix ̂ A comprising only those rows of A with leverage scores at least 1 /n 10

4: Let s be the output of Algorithm 5 with inputs ̂ A , SA , S ′ A , n , and ρ

5: Return ̂ s := (1 + γ ) ( s + 1 n 5 ( | A | - | ̂ A | ) )

B.2.1 Proof sketch.

We achieve our guarantee via Algorithm 4 and Algorithm 5, which are based on the following key principles. The first is Fact 2.2: the i th sensitivity and i th leverage score of A ∈ R n × d satisfy √ τ i ( A ) n ≤ σ 1 ( a i ) ≤ √ τ i ( A ) . The second idea, derived from Bernstein's inequality (and formalized in Lemma A.3), is: the sum of a set of positive numbers with values bounded between a and b can be approximated to a γ -multiplicative factor using ˜ O (( b/a ) γ -2 ) uniformly sampled (and appropriately scaled) samples. The third principle is that σ 1 ( a i ) ≈ σ SA 1 ( a i ) when SA is a constant-approximation ℓ 1 subspace embedding of A .

Based on these ideas, we first divide the rows of A into B = Θ(log n ) submatrices based on which of the B buckets [1 / 2 , 1] , [1 / 4 , 1 / 2] , [1 / 8 , 1 / 4] , etc. their leverage scores (computed with respect to A ) fall into. Since each of these B submatrices comprises rows with leverage scores in the range [ a, 2 a ] for some a , per Fact 2.2, we have √ a n ≤ σ 1 ( a i ) ≤ √ 2 a . Therefore, by Lemma A.3, the total sensitivity of this submatrix may be approximated by the appropriately scaled total sensitivity of O ( √ n ) of its uniformly sampled rows. Our design choice to split rows based on leverage scores is based on the fact that leverage scores (unlike sensitivities) are efficiently computable (using [60]). Computing the total sensitivity of an ˜ O ( √ n ) -sized matrix thus costs ˜ O ( nnz ( A ) + √ n · d ω ) under this scheme.

Applying the idea sketched above recursively further reduces the cost. Suppose, after dividing the input matrix A into B submatrices M i and subsampling ˜ M i from these matrices, we try to recurse on each of these B submatrices ˜ M i . A key requirement for this idea to work is that the sum of the ℓ 1 sensitivities of the rows of ˜ M i in isolation be close enough to the true sum of ℓ 1 sensitivities of

Algorithm 5 Subroutine for Approximating the Sum of ℓ 1 -Sensitivities

Input: Matrices M ∈ R r × d , SA ∈ R O ( d ) × d , S ′ A ∈ R O ( d ) × d , scalars n and ρ ∈ (0 , 1) Output: A positive scalar ̂ s that satisfies, with probability 0 . 99 , that

2: if the number of rows of M is at most b then

4: Using Fact 2.4 on M , compute ̂ s such that S S ′ A 1 ( M ) ≤ ̂ s ≤ (1 + O ( ρ )) S S ′ A 1 ( M )

3: Let S S ′ A 1 ( M ) be the sum of ℓ 1 sensitivities of the rows of M with respect to S ′ A

5: Return s

6: else

̂

7: Construct the matrix C := [ M SA ] . Set r = O ( √ | C | (1 + ρ ) ρ -2 log( δ -1 )) .

8: Compute τ ( C ) , the vector of leverage scores of C

9: Divide [ 1 n 20 , 1 ] into B geometrically decreasing sub-intervals [1 / 2 , 1] , [1 / 4 , 1 / 2] , . . . , etc.

10: for each bucket i ∈ [ B ] created in the previous step do

11: Construct matrix M i with those rows of M whose leverage scores fall in the i th bucket

12: Construct matrix ˜ M i composed of r uniformly sampled (with replacement) rows of M i

13: Compute ˜ s i , the output of Algorithm 5 with inputs ˜ M i , SA , S ′ A , n , and ρ .

14: end for

16: end if those rows with respect to the original matrix A . In general, this is not true . However, we can meet our requirement using the ℓ 1 subspace embedding SA of the (original) matrix A . Specifically, if we vertically concatenate a matrix ˜ M i with SA and call this matrix C , then for each row ˜ M i [ j ] ∈ ˜ M i , we have σ C 1 ( ˜ M i [ j ]) ≈ σ A 1 ( ˜ M i [ j ]) . Further, | C | ≤ O ( √ n ) , and so in the step subsampling the rows of ˜ M i , we choose O ( n 1 / 4 ) samples. Recursing this exponentially decreases the row dimension of the input matrix at each level, with (log n ) k matrices at the k th level of recursion.

B.2.2 Helper lemmas for proving Theorem B.1

For our formal proof, we need Definition B.2, Fact B.3, and Definition B.4.

Definition B.2 (Notation for recursion tree of Algorithm 5) . Recall that B = Θ(log( n )) as in Line 1 of Algorithm 5. Then we have the following notation corresponding to Algorithm 5.

/caret_right We define a rooted B -ary tree T corresponding to the execution of Algorithm 5.

/caret_right We use T j to denote the subtree starting from the root node with all the nodes up to and including those at the j th level. We denote the i th node at the j th level by T ( j,i ) . Thus, every node is uniquely specified by its level in the tree and its index within that level.

/caret_right The root node T (1 , 1) at level 1 corresponds to the first call to Algorithm 5 from Algorithm 4.

/caret_right For the node T ( j,i ) , we denote by M ( j,i ) the input matrix to the corresponding recursive call; note that all the other inputs are global and remain the same throughout the recursive call.

/caret_right Analogously, we use ̂ s ( j,i ) to denote our estimate of S S ′ A 1 ( M ( j,i ) ) , the sum of ℓ 1 sensitivities of rows of the matrix M ( j,i ) in the node T ( j,i ) , defined with respect to the sketched matrix S ′ A .

Fact B.3. If event A happens with probability 1 -τ 1 and event B happens with probability 1 -τ 2 conditioned on event A , then the probability of both events A and B happening is at least 1 -τ 1 -τ 2 .

Definition B.4. We say that a node N with matrix M satisfies the ( ̂ ρ, ̂ δ ) -approximation guarantee if ̂ s , the output of Algorithm 5 on M , satisfies the following guarantee for the true sum S S ′ A 1 ( M ) of ℓ 1 sensitivities of the rows in M :

for some approximation parameter ̂ ρ ∈ (0 , 1) and an error probability parameter ̂ δ ∈ (0 , 1) .

Lemma B.5 (Partitioning of M ) . In Algorithm 5, even though we ignore the range [ 0 , 1 n 20 ] when creating the submatrices M 1 , M 2 , . . . , M B from M , every row in M is part of exactly one of the submatrices M 1 , M 2 , . . . , M B . In other words, the matrix M is partitioned into the matrices M 1 , M 2 , . . . , M B , with no row missed out.

Proof. Our proof strategy is to show that for all row indices k ∈ [ | M | ] , each of the leverage scores τ k ( C ) satisfy τ k ( C ) ≥ 1 n 20 . Therefore, creating buckets of rows with the range of leverage score starting at 1 n 20 (instead of 0 ) does not miss out any rows.

To this end, we note that for every row a k in the input M to Algorithm 5, we have:

where the first step is because in Line 3 of Algorithm 4, we discard those rows a k of A with leverage scores smaller than 1 /n 10 and there are n rows in A , so every row a k in the input matrix M to Algorithm 5 satisfies τ k ( A ) ≥ 1 n 10 ; the second step is by Fact 2.2 applied to a k ; the third step is by the definition of the ℓ 1 sensitivity of a k with respect to A ; the fourth step is by observing that ∥ Cx ∥ 1 = ∥ SAx ∥ 1 + ∥ Mx ∥ 1 ≤ 2 ∥ Ax ∥ 1 + ∥ Ax ∥ 1 = 3 ∥ Ax ∥ 1 since SA is a constant-factor (with the constant assumed 1 / 2 ) ℓ 1 subspace embedding of A and because M is a submatrix of A ; the fifth step is by the definition of ℓ 1 sensitivity of a k with respect to C ; the final step is by Fact 2.2 applied to a k within C . Observing the first and last terms of this inequality chain, we have τ k ( C ) ≥ 1 3 n 11 for all k ∈ [ | C | ] . This finishes the proof of the claim.

Lemma B.6. For the recursion tree T corresponding to Algorithm 5, let the size b of the input matrices at the leaf nodes be b = 50 D 3 log(100 B ) γ 2 · ( 2 D 3 log(100 B ) γ 2 + √ d log( d ) ) . Then, the depth of the recursion tree is less than D := 1 + log(log(2 n +2 d log( d ))) .

Proof. For now, let β be arbitrary but less than β 2 ≤ n + d ′ , where d ′ is the row dimension of the subspace embedding SA . Then, the size of the input matrix to the nodes of the recursion tree as we increase in depth evolves as

Thus, this sequence evolves as

To see the limit of this sequence, we set x = β d ′ + x, and solve for this quadratic equation as

We set the base case to be b ⋆ := 7 β ( β + d ′ ) , a constant times larger than this asymptotic limit. We now compute the depth t ⋆ at which the recursion attains the value b ⋆ is attained. To do so, we consider a sequence { y t } which overestimates the sequence { x t } of matrix sizes for a big range of t . We define { y t } as

√

and it can be checked that for t such that y t ≥ d ′ , we have y t +1 ≥ x t +1 . Moreover, for all t ≥ 1 , y t = β 1+1 / 2+ ··· +1 / 2 t -1 (2 y 0 ) 1 / 2 t = β 2 -2 1 -t (2 y 0 ) 1 / 2 t . The limit of this sequence can easily be seen to β 2 . We now split our analysis in two cases based on the limits of these two sequences.

Case 1: β 2 ≥ d ′ . Since the limit of the sequence { y t } is β 2 , it is easy to see that y t is always greater d ′ (since y t is monotonically decreasing as y 0 ≥ β 2 ). Thus, x t will always be less than y t in this case. We will now show that y t (and thus x t ) will be less than b ⋆ in t 0 = 2log log(2 y 0 ) steps. Solving the inequality for t as follows: y t ≤ β 2 (2 y 0 ) 1 / 2 t ≤ 4 β 2 ≤ b ⋆ , it suffices to ensure that (2 y 0 ) 1 / 2 t ≤ 4 , which happens if t ≥ t 0 . Thus, t ⋆ ≤ 2 log log 2 y 0 .

Case 2 : β 2 < d ′ . In this case, the sequence y t will eventually go below d ′ and thus the inequality y t ≥ x t might eventually break down. Let t = t ′ denote the first time step at which y t ≤ 5 d ′ (which will be finite in this case). Then, by definition, y t ≥ x t for all t < t ′ . We will now upper bound the value of t ′ as follows: y t ≤ β 2 (2 y 0 ) 1 / 2 t ≤ d ′ (2 y 0 ) 1 / 2 t ≤ 4 d ′ ; Thus, it suffices to ensure that (2 y 0 ) 1 / 2 t ≤ 4 , which happens if t ≥ t 0 := 2 log log(2 y 0 ) . Hence, t ′ ≤ t 0 . Moreover, the value of y t ′ ≤ 5 d ′ , implies that x t ′ -1 ≤ y t ′ -1 ≤ (5 d ′ ) 2 2 β 2 . We will now show that x t ′ +1 ≤ b ⋆ using the evolution of x t +1 = β √ d ′ + x t as follows: First, x t ′ ≤ β √ d ′ + β √ x t ′ -1 ≤ β √ d ′ +5 d ′ ≤ 6 d ′ ; Then, x t ′ +1 ≤ β √ d ′ + β √ 6 d ′ ≤ 7 β √ d ′ ≤ b ⋆ . Thus, t ⋆ ≤ 1 + 2 log log 2 y 0 .

Therefore, in both of these cases, we have that after at most D = 1 + log log(2 n + 2 d ′ ) depth, the base case of b ⋆ = 7 β ( β + √ d ′ ) is achieved at the leaf node. Recall that the algorithm sets D = 2log log( n + d ′ ) + 1 and ρ = γ/D for d ′ = d log( d ) which implies that β := C (1+ ρ ) ρ 2 log( δ -1 ) = CD ( D + γ ) γ 2 log(100 B D ) ≤ 2 D 3 log(100 B ) γ 2 . Since Algorithm 5 sets b = 50 D 3 log(100 B ) γ 2 · ( 2 D 3 log(100 B ) γ 2 + √ d log( d ) ) , which is larger than b ⋆ , we may conclude that the choice of b implies that the recursion tree T of Algorithm 5 has a depth less than D = 1 + log(log(2 n +2 d log( d ))) .

Lemma B.7. Suppose the root node T (1 , 1) of Algorithm 5 satisfies a ( ̂ ρ, ̂ δ ) -approximation guarantee as defined in Definition B.4, and denote by ̂ s the output of Algorithm 4. Then, with a probability of at least 1 -̂ δ , we have

As a result of this lemma, in the final proof of correctness of Algorithm 4 (appearing later as proof of Theorem B.1), it suffices to show that ̂ s satisfies a ( ̂ ρ, ̂ δ ) -approximation guarantee for some appropriate choices of these parameters.

Proof of Lemma B.7. Suppose the root node satisfies Definition B.4 with some ( ̂ ρ, ̂ δ ) -approximation guarantee. This means that there exists an ˜ s such that the output of Algorithm 5 on the input matrix ̂ A satisfies

Since S ′ A is, by design, a ρ -approximation ℓ 1 subspace embedding for A , it means for all x ∈ R d we have

Therefore, for every row a i ∈ ̂ A , we have by the definition of ℓ 1 sensitivity and Inequality (B.8) that the ℓ 1 sensitivity of a i computed with respect to S ′ A is a ρ -approximation of the ℓ 1 sensitivity of a i with respect to A :

Therefore, by linearity, this ρ -approximation factor transfers over in connecting the total sensitivity of ̂ A with respect to S ′ A to the total sensitivity of ̂ A with respect to A :

Therefore, chaining Equation (B.7) and Equation (B.9) yields the following approximation guarantee:

There is no change in the error probability above from Equation (B.7) because Equation (B.9) holds deterministically. Next, since in Line 3 of Algorithm 4, we dropped rows of A that have leverage scores less than 1 n 10 , for each of the dropped rows a i , we have

Therefore, the quantity we return, ̂ s , satisfies the following approximation guarantee:

where the first step is by definition of ̂ s in Algorithm 4; the second step is by Equation (B.10); the third step is by applying Inequality (B.11) to each of the rows not present in ̂ A . In the other direction,

where the first step is by definition of ̂ s in Algorithm 4; the second step is by the fact that | A | ≤ n ; the final step is by combining the assumption ρ ≥ 1 n 4 and the facts that all sensitivities are non-negative and that the total ℓ 1 sensitivity is at least one. Combining Inequality (B.12) and Inequality (B.13) finishes the proof.

Lemma B.8 (Single-Level Computation in Recursion Tree) . Consider a node N in the recursion tree corresponding to Algorithm 5, and let M be the input matrix for N . Further assume that N is not a leaf node. Denote by N 1 , N 2 , . . . , N B (where B = Θ(log( n )) is the number of recursive calls at each node in Algorithm 5) the children of node N , with each node N i with the input matrix ˜ M i (as constructed in Algorithm 5). Suppose each of these child nodes N i satisfies a ( ̂ ρ, ̂ δ ) -approximation guarantee (cf. Definition B.4). Then N satisfies a ( ̂ ρ + ρ, ( ̂ δ + δ ) · B ) -approximation guarantee.

Proof. We first reiterate the steps of Algorithm 5 that are crucial to this proof. As proved in Lemma B.5, Algorithm 5 partitions the rows of the matrix M in node N into B submatrices M 1 , M 2 , . . . , M B based on their leverage scores. Next, for each matrix M i , we uniformly sample with replacement O ( √ | C | (1 + ρ ) ρ -2 log( δ -1 )) rows and term this set of rows as matrix ˜ M i , which we then pass to Algorithm 5 recursively in node N i (recall that we assume that the node N is not a leaf node). In other words, each child node N i has as its input matrix ˜ M i .

Since we assume that all the child nodes N 1 , N 2 , . . . , N B satisfy a ( ̂ ρ, ̂ δ ) -approximation guarantee as in Definition B.4, it means that, for all i ∈ [ B ] , the output ̂ s i of Algorithm 5 on node N i satisfies the following guarantee:

Next, we recall that each M i is composed of only those rows of M whose leverage scores (computed with respect to C ) lie in the i th bucket [2 -i , 2 -i +1 ] . Combining this with Fact 2.2, this implies that the ℓ 1 sensitivity of the j th row M i [ j ] of the matrix M i , computed with respect to C , satisfies

Additionally, we claim that

To see this, we observe that

where the first step is by the definition of ℓ 1 sensitivity of M i [ j ] (i.e., the j th row of matrix M i ) with respect to C ; the second step is by the definition of C , the fact that SA is a 1 / 2 -factor ℓ 1 subspace embedding of A , and because M is a subset of rows of A ; the third step is because S ′ A is a ρ -approximate ℓ 1 subspace embedding for A , and the final step is by the definition of σ A 1 ( M i [ j ]) . In the other direction, we have

where the second step is by the definition of C as a vertical concatenation of SA and M and further using that SA is a constant factor ℓ 1 subspace embedding of A and dropping the non-negative term ∥ Mx ∥ 1 . Combining Inequality (B.15) and Inequality (B.16), we see that for all rows j ∈ [ | M i | ] , we have

Since Inequality (B.19) shows upper and lower bounds on each sensitivity σ S ′ A 1 ( M i ) of the matrix M i , we may use Lemma A.3 to approximate the sum of these sensivities. In particular, by Lemma A.3, we may construct a matrix ˜ M i composed of 20 √ | C | (1+2 ρ ) log( δ -1 ) ρ 2 rows of M i sampled uniformly at random with replacement, and we are guaranteed

Note that the output ̂ s of Algorithm 5 when applied to node N is defined as

Therefore, by union bound over the failure probability (cf. Fact B.3), we can combine Equation (B.14) and sum Equation (B.20) over all i ∈ B child nodes to conclude that ̂ s from Equation (B.21) satisfies a ( ̂ ρ + ρ, ( ̂ δ + δ ) B ) -approximation guarantee of Definition B.4.

B.2.3 Proof of Theorem B.1

Proof of Theorem B.1. We first sketch our strategy to prove Theorem B.1's correctness guarantee. Essentially, our goal is to establish Definition B.4 for the root node T (1 , 1) . That this goal suffices to prove the theorem is justified in Lemma B.7. We now show that the node T (1 , 1) satisfies a ( ̂ ρ, ̂ δ ) -approximation guarantee for some ̂ ρ and ̂ δ . We use the method of induction.

Correctness guarantee of Theorem B.1. We know that the leaf nodes all satisfy the ( ρ, 0) -approximation guarantee in Definition B.4 since at this level, the base case is triggered, where we compute a ρ -approximation estimate of the total ℓ 1 sensitivity with respect to the subspace embedding S ′ A . Having established this approximation guarantee at the leaf nodes, we can use Lemma B.8 to inductively propagate the property of ( ρ, δ ) -approximation guarantee up the recursion tree T . From Lemma B.6, we have that the recursion tree T of Algorithm 5 is of depth at most D = O (log log( n + d )) , and the number of child nodes at any given node is at most B = Θ(log( n )) ; we factor these into the approximation guarantee and error probability.

Base case. At the leaf nodes of the recursion tree, we directly compute the total sensitivities with respect to S ′ A . Recall that we denote the depth of the tree by D . Then, because S ′ A is, by definition, a ρ -approximate subspace embedding for A , we have, for each i that indexes a leaf node,

Induction proof. Consider the subtree T j truncated at (but including) the nodes at level j in the recursion tree T corresponding to Algorithm 5. Assume the induction hypothesis that all the leaf nodes in the subtree T j satisfy Definition B.4 with parameters ( ρ · ( D +1 -j ) , δ · ( B D -j +1 -B B1 )) . That is, at the matrix of each such node i , we have an estimate ̂ s ( j,i ) of its total sensitivity S S ′ A 1 ( M ( j,i ) ) such that

Then, each of the leaf nodes of the subtree T j -1 (i.e., the subtree that stops one level above T j ) satisfies the assumption in Lemma B.8 since its child nodes are either leaf nodes of the entire recursion tree T (for which this statement has been shown in Inequality (B.22)) or, if they are not

leaf nodes of the recursion tree, they satisfy Definition B.4 with parameters ̂ ρ = ρ · ( D +1 -j ) and ̂ δ = δ · ( B D -j +1 -B B1 ) . Therefore, Lemma B.8 implies that each of the leaf nodes of T j -1 satisfies ρ δ

Definition B.4 with approximation parameter ̂ and error probability ̂ defined as follows:

In other words, the outputs ̂ s ( j -1 ,k ) of Algorithm 5 on the matrices M ( j -1 ,k ) in the leaf nodes of T j -1 each satisfy:

The base case in Inequality (B.22) and the satisfaction of the induction hypothesis in Inequality (B.24) together finish the proof of the induction hypothesis.

Finishing the proof of correctness. In light of the above conclusion, for the root node T (1 , 1) , which is at the level j = 1 , we may plug in j = 1 in Inequality (B.23) and obtain that the output ̂ s (1 , 1) of Algorithm 5 at the root node satisfies

In Algorithm 5, we choose the parameters δ = 0 . 01 B D and ρ = O ( γ D ) . Plugging these parameters into Inequality (B.25), we conclude that the root node satisfies Definition B.4 with a ( γ, 0 . 01) -approximation guarantee. This finishes the proof of correctness of the claim.

Proof of runtime. Let b denote the maximum number of rows of a matrix at a leaf node of T . Further, recall from Fact 2.4 that the cost of computing one ℓ 1 sensitivity of an n × d matrix is,

Then as stated in Line 1, the algorithm computes the ρ -approximate total sensitivity at the leaf nodes; Fact 2.4, this incurs a cost of

At any other node, the computational cost is the work done at that level plus the total cost of the recursive calls at that level. Let C ( r ) denote the runtime of Algorithm 5 when the input is an r × d matrix. Then,

where b is the number of rows in the base case, and L is the cost of computing leverage scores and equals L = ˜ O ( nnz ( A ) + d ω ) from Fact 2.1. For the depth of the recursion D (where the top level is D = 1 ), plugging in our choices of δ = 0 . 01 B D and ρ = O ( γ D ) , this may be expressed as via expanding the recursion as:

for n ≥ b . We plug into this expression the values of b and D from Lemma B.6, L from Equation (B.26), C ( b ) from Equation (B.27), and B = Θ(log( n )) to get the claimed runtime.

B.3 Estimating the maximum ℓ 1 sensitivity

Theorem 3.7 ( Approximating the Maximum of ℓ 1 Sensitivities ) . Given a matrix A ∈ R n × d , there exists an algorithm, which in time ˜ O ( nnz ( A ) + d ω +1 ) , outputs a positive scalar ̂ s that satisfies

Proof. First, we have

We use Equation (B.29) to establish the claimed bounds below. First we set some notation. Define x ⋆ and a i ⋆ as follows:

Thus, x ⋆ is the vector that realizes the maximum sensitivity of the matrix A , and the row a i ⋆ is the row of A with maximum sensitivity with respect to A . Suppose the matrix S ∞ A contains the row a i ⋆ . Then we have

where the first step is by definition of ℓ 1 sensitivity of S ∞ A with respect to S 1 A , the second step is by Equation (B.29), and the third step by noting that the matrix S ∞ A is a subset of the rows of A (which includes a i ⋆ ). By definition of σ 1 ( a i ⋆ ) in Equation (B.30), we have

Then, combining Equation (B.31) and Inequality (B.32) gives the guarantee in this case. In the other case, suppose a i ⋆ is not included in S ∞ A . Then we observe that the upper bound from the preceding inequalities still holds. For the lower bound, we observe that

where the the second step is by choosing a specific vector in the numerator and the third step uses ∥ S 1 Ax ∥ 1 = Θ( ∥ Ax ∥ 1 ) . We further have,

where the first step is by choosing x = x ⋆ , the second step is by the distortion guarantee of ℓ ∞ subspace embedding [35], and the final step is by definition of σ 1 ( a i ⋆ ) . Combining Inequality (C.10) and Inequality (C.11) gives the claimed lower bound on ∥ σ S 1 A 1 ( S ∞ A ) ∥ ∞ . The runtime follows from the computational cost and row dimension of S ∞ A from [35] and the cost of computing ℓ 1 sensitivities with respect to S 1 A as per Fact 2.4.

C Omitted proofs: ℓ p sensitivities

We recall the following notation that we use for stating our results in this setting.

Definition C.1 (Notation for ℓ p Results) . We introduce the notation LP ( m,d,p ) to denote the cost of approximating one ℓ p sensitivity of an m × d matrix up to an accuracy of a given constant factor.

C.1 Estimating all ℓ p sensitivities

We generalize Algorithm 1 to the case p ≥ 1 below.

Theorem C.2 (Approximating all ℓ p sensitivities) . Given a full-rank matrix A ∈ R n × d , an approximation factor 1 < α ≪ n , and a scalar p ≥ 1 , let σ p ( a i ) be the i th ℓ p sensitivity. Then there exists an algorithm that returns a vector ˜ σ ∈ R n ≥ 0 such that with high probability, for each i ∈ [ n ] , we have

Our algorithm runs in time ˜ O ( nnz ( A ) + n α · LP ( d max(1 ,p/ 2) , d, p ) ) (cf. Definition 1.2).

Algorithm 6 Approximating ℓ p -Sensitivities: Row-wise Approximation

Inputs:

Matrix A ∈ R n × d , approximation factor α ∈ (1 , n ] , scalar p ≥ 1

Output:

Vector ˜ σ ∈ R n > 0 that satisfies, for each i ∈ [ n ] , with probability 0 . 9 , that

1: Compute, for A , an ℓ p subspace embedding S p A ∈ R ˜ O ( d max(1 ,p/ 2) ) × d

3: for the ℓ th block B ℓ , with ℓ ∈ [ n α ] do

2: Partition A into n α blocks B 1 , . . . , B n/α each comprising α randomly selected rows.

4: Sample α -dimensional independent Rademacher vectors r ( ℓ ) 1 , . . . , r ( ℓ ) 100

5: For each j ∈ [100] , compute the row vectors r ( ℓ ) j B ℓ ∈ R d

6: end for

7: Let P ∈ R 100 n α × d be the matrix of all vectors computed in Line 5. Compute σ S p A p ( P ) using Definition 1.2

8: for each i ∈ [ n ] do

9: Denote by J the set of row indices in P that a i is mapped to in Line 5

10: Set ˜ σ i = max j ∈ J ( σ S p A p ( p j ))

11: end for

12: Return σ

˜

Proof. Suppose the i th row of A falls into the bucket B ℓ . Suppose, further, that the rows from B ℓ are mapped to those in P with row indices in the set J . Then, Algorithm 6 returns a vector of sensitivity estimates ˜ σ ∈ R n with the i th coordinate defined as ˜ σ i = max j ∈ J σ p ( c j ) . For any x ∈ R d , we have

We use Equation (C.2) to establish the claimed bounds below. Let x ⋆ = arg max x ∈ R d | a ⊤ i x | p ∥ Ax ∥ p p be the vector that realizes the i th ℓ p sensitivity of A . Further, let the j th row of P be r ( ℓ ) k B ℓ . Then, with probability of at least 1 / 2 , we have

̸

where the first step is by definition of σ p ( p j ) , the second is by evaluating the function being maximized at a specific choice of x , the third step is by Equation (C.2), and the final step is by definition of x ⋆ and σ p ( a i ) . For the fourth step, we use the fact that | r ( ℓ ) k B ℓ x ⋆ | p = | r ( ℓ ) k,i a ⊤ i x ⋆ + ∑ j = i r ( ℓ ) k,j ( Bx ⋆ ) j | p ≥ | a ⊤ i x ⋆ | p with a probability of at least 1 / 2 since the vector r ( ℓ ) k has coordinates that are -1 or +1 with equal probability. By a union bound over j ∈ J independent rows that block B ℓ is mapped to, we establish the claimed lower bound in Inequality (C.3) with probability at least 0 . 9 . To show an upper bound on σ S p A p ( p j ) , we observe that

where the second step is by Equation (C.2) and opening up ∥ B ℓ x ∥ p p in terms of the rows in B ℓ , and the final step is by definition of B ℓ . To see the first step, we observe that

where 1 p + 1 q = 1 , the first step is by Hölder's inequality, the second is because each entry of r ( ℓ ) k is either +1 or -1 and r ( ℓ ) k ∈ R α , and the third is because p q = p -1 . Because B ℓ is a group of α rows

selected uniformly at random out of n rows and contains the row a i , we have:

̸

̸

Therefore, Markov inequality gives us that with a probability of at least 0 . 9 , we have

The median trick then establishes Inequality (C.4) and Inequality (C.5) with high probability. To establish the runtime, note that we first construct the subspace embedding S p A and then compute the ℓ p sensitivities of P (with O ( n/α ) rows) with respect to S p A . The size of the subspace embedding S p A is O ( d 1 ∨ p/ 2 × d ) (see [75, Table 1 ]). This completes the runtime claim.

C.2 Estimating the maximum of ℓ p sensitivities

In Section 3.2 and Appendix B.3, we showed our result for estimating the maximum ℓ 1 sensitivity. In this section, we show how to extend this result to all p ≥ 1 . Algorithm 7 generalizes Algorithm 3.

Algorithm 7 Approximating the Maximum of ℓ p -Sensitivities

̸

Input: Matrix A ∈ R n × d and p ≥ 1 (with p = 2

)

Output:

Scalar ̂ s ∈ R ≥ 0 that satisfies that

1: Compute, for A , an ℓ ∞ subspace embedding S ∞ A ∈ R O ( d log 2 ( d )) × d such that S ∞ A is a subset of the rows of A [35]

2: Compute, for A , an ℓ p subspace embedding S p A ∈ { R O ( d ) × d p ∈ [1 , 2) R O ( d p/ 2 ) × d p > 2

3: Return d p/ 2 ∥ σ S p A p ( S ∞ A ) ∥ ∞

̸

Theorem C.3 (Approximating the maximum of ℓ p sensitivities) . Given a matrix A ∈ R n × d and p ≥ 1 (with 3 p = 2 ), there exists an algorithm, which outputs a positive scalar ̂ s that satisfies

The runtime of the algorithm is ˜ O ( nnz ( A ) + d ω +1 ) for p ∈ [1 , 2) and ˜ O ( nnz ( A ) + d p/ 2 · LP ( O ( d p/ 2 ) , d, p )) for p > 2 .

Proof. First, we have

We use Equation (C.6) to establish the claimed bounds below. First we set some notation. Define x ⋆ and a i ⋆ as follows:

Thus, x ⋆ is the vector that realizes the maximum sensitivity of the matrix A , and the row a i ⋆ is the row of A with maximum sensitivity with respect to A . Suppose the matrix S ∞ A contains the row a i ⋆ . Then we have

where the first step is by definition of ℓ p sensitivity of S ∞ A with respect to S p A , the second step is by Equation (C.6), and the third step by noting that the matrix S ∞ A is a subset of the rows of A (which includes a i ⋆ ). By definition of σ p ( a i ⋆ ) in Equation (C.7), we have

Then, combining Equation (C.8) and Equation (C.9) gives the guarantee in this case. In the other case, suppose a i ⋆ is not included in S ∞ A . Then we observe that the upper bound from the preceding inequalities still holds. For the lower bound, we observe that

where the the second step is by choosing a specific vector in the numerator and the third step uses ∥ S p Ax ∥ p p = Θ( ∥ Ax ∥ p p ) . We further have,

where the first step is by choosing x = x ⋆ , the second step is by the guarantee of ℓ ∞ subspace embedding, and the final step is by definition of σ p ( a i ⋆ ) . Combining Inequality (C.10) and Inequality (C.11) gives the claimed lower bound on ∥ σ S p A p ( S ∞ A ) ∥ ∞ . The runtime follows from the cost of computing S ∞ A from [35] and the cost of computing and size of S p A from [23, Figure 1]

D Lower Bounds

Theorem D.1 ( ℓ p Regression Reduces to ℓ p Sensitivities ) . Suppose that we are given an algorithm A , which for any matrix A ′ ∈ R n ′ × d ′ and accuracy parameter ε ′ ∈ (0 , 1) , computes (1 ± ε ′ ) σ p ( A ′ ) in time T ( n ′ , d ′ , nnz ( A ′ ) , ε ′ ) . Then, there exists an algorithm that takes A ∈ R n × d and b ∈ R n as inputs and computes (1 ± ε ) min y ∈ R d ∥ Ay -b ∥ p p ± λ p ε in time T ( n +1 , d +1 , nnz ( A )+ nnz ( b )+ 1 , ε ) for any λ > 0 .

Proof. Given A ∈ R n × d and b ∈ R d , consider the matrix A ′ := [ A -b 0 ⊤ -λ ] ∈ R ( n +1) × d . Then the n +1 -th ℓ p sensitivity of A ′ is

where the second step is by the scale-invariance of the definition of sensitivity with respect to the variable of optimization. By scale-invariance we see that for any c > 0 , and min y ∥ A : -i y -c A : i ∥ p p = c p min y ∥ A : -i y -A : i ∥ p p .

Therefore, note that rearranging gives min y ∈ R d -1 ∥ Ay -b ∥ p p = λ p σ p ( a ′ n +1 ) -λ p , which is how we can estimate the cost of the regression. Then computing σ p ( a ′ n +1 ) to a multiplicative accuracy of ε ′ gives the value of min y ∈ R d -1 ∥ Ay -b ∥ p p up to error ε ( λ p σ p ( a ′ n +1 ) ) = ε ( λ p +min y ∈ R d -1 ∥ Ay -b ∥ p p ) , giving our final bounds.

Using the same technique as in Theorem D.1, we can similarly extend the reduction to a multiple regression task. In particular, we show that computing the values of a family of certain regularized leave-one-out regression problems for a matrix may be obtained by simply computing leverage scores of an associated matrix; therefore, this observation demonstrates that finding fast algorithms for sensitivities is as hard as multiple regression tasks. Specifically, for some matrix A ∈ R n × d , we denote A : -i ∈ R n × d -1 as the submatrix of A with its i -th column, denoted as A : i , removed. We show that approximate sensitivity calculations can solve min y ∈ R d -1 ∥ A : -i y + A : i ∥ 2 approximately for all i .

Lemma D.2 ( Regularized Leave-One-Out ℓ p Multiregression Reduces to ℓ p Sensitivities ) . Suppose that we are given an algorithm A , which for any matrix A ′ ∈ R n ′ × d ′ and accuracy parameter ε ′ ∈ (0 , 1) , computes (1 ± ε ′ ) σ p ( A ′ ) in time T ( n ′ , d ′ , nnz ( A ′ ) , ε ′ ) . Given a matrix A ∈ R n × d with n ≥ d , let OPT i := min y ∈ R d -1 ∥ A : -i y + A : i ∥ p p and y ⋆ i := arg min y ∈ R d -1 ∥ A : -i y + A : i ∥ p for all the i ∈ [ d ] . Then, there exists an algorithm that takes A ∈ R n × d and computes (1 ± ε ) OPT i ± λ p (1 + ∥ y ⋆ i ∥ p p ) in time T ( n + d, d, nnz ( A ) , ε ) for any λ > 0 .

Proof. Given A ∈ R n × d , consider the matrix A ′ := [ A λ I ] ∈ R ( n + d ) × d formed by vertically appending a scaled identity matrix to A , with λ > 0 . By the definition of ℓ p sensitivities, we have the following.

Recall that y ⋆ i := arg min y ∈ R d -1 ∥ A : -i y + A : i ∥ p , and since the power is a monotone transform, OPT i := min y ∈ R d -1 ∥ A : -i y + A : i ∥ p = ∥ A : -i y ⋆ i + A : i ∥ p . By scale-invariance we see that for any c > 0 , c y ⋆ i = arg min y ∈ R d -1 ∥ A : -i y + c A : i ∥ p p and min y ∥ A : -i y + c A : i ∥ p p = c p OPT i . Therefore, by plugging y = λ -1 y ⋆ i , note that

However, we also note that by non-negativity,

Combining this all together gives the claimed approximation guarantee by noting that

Therefore, our algorithm is to simply return λ p /s p ( a ′ n + i ) , where s is the ϵ -approximate estimate of the sensitivity and our bound follows. The runtime guarantees follow immediately from assumption that the runtime of calculating the ℓ p sensitivities of A ′ .

Corollary D.3. Assuming that leave-one-out ℓ p multi-regression with Ω( d ) instances takes poly ( d ) nnz ( A ) + poly (1 /ε ) time [44, 45], computing ℓ p sensitivities of an n × d matrix A costs at least poly ( d ) nnz ( A ) + poly (1 /ε ) .

E Additional Experiments

Here we include additional experiments on other similar datasets.

SensitivityApproximationsforFiresDataset

p=1

p =1.5

additivebound

8

additive bound

7

average

average

7

6

6

5

5

4

4

3

3

2

5

10

15

520253035

40

5

10

15

202530

35

40

ApproximationFactor(alpha)

ApproximationFactor(alpha)

p = 2.5

p=3

additive bound

16

14

average

14

12

12

10

10

additivebound

average

8

8

6

9

t

5

10

15

20

25

30

35

40

5

10

15

20

25

30

35

40

Approximation Factor(alpha)

ApproximationFactor(alpha)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose randomized algorithms to approximate $\\ell_p$ sensitivity functions for $p \\in [1,\\infty)$, which extend leverage scores beyond the $\\ell_2$ norm. The functions they consider are: 1. Estimating all sensitivities, 2. Estimating total sensitivity, and 3. Estimating maximum sensitivity. They provide different types of approximations for each task.\n\nFor task 1, they give an additive error, constant factor approximation. For task 2, they give a relative error, $(1+O(\\gamma))$ approximation ($\\gamma \\in (0,1)$). And for task 3, they give a constant factor relative error approximation.\n\nThey demonstrate their algorithm initially for $\\ell_1$ and then generalize it to all $\\ell_p$ norms with $p > 1$. They also prove a hardness result by reducing $\\ell_p$ regression to $\\ell_p$ sensitivity estimation. Additionally, they implement their algorithm to estimate all sensitivities on 2 existing datasets and compare the average and maximum approximation ratios with the theoretical results.\n\nThe main techniques used in the algorithms involve hashing using Rademacher combinations, computing sensitivities with respect to subspace embeddings, splitting matrix rows based on leverage score intervals, and utilizing existing results for $\\ell_\\infty$ subspace embeddings.\n\nSTRENGTHS:\nThe problem of sensitivity estimation is useful for regression problems and has received limited attention for general $p$. Approximating total sensitivity is particularly significant in obtaining the sample complexity of learning arbitrary functions. \n\nThe algorithms presented are an interesting combination of known results from sensitivity sampling framework and RNLA. Specifically, their result for approximating total sensitivity is interesting because the computational complexity does not depend polynomially on the number of rows of the matrix, which is usually very large. \n\nFinally, the analysis of the proposed algorithms is largely clear, helping to understand the guarantees of their algorithms. Overall, this paper is a significant contribution to the field of sensitivity sampling and dimensionality reduction.\n\nWEAKNESSES:\n- Algorithm 4 does not appear to be significantly novel, except for integrating generalized sensitivity to an existing $\\ell_\\infty$ subspace embedding technique, but the task is coherent with the other tasks mentioned.\n- While the motivation for total sensitivity is clear, the authors have not motivated the problem of estimating maximum sensitivity. However, this might be because I am not familiar with it. \n- I noticed some typos, missing definitions.. In Algorithm 3, it is unclear where the vector of leverage scores $\\tau(C)$ is being used. Additionally, $\\omega$ is not defined, which I am assuming to be the matrix multiplication exponent.\n\nQUESTIONS:\nI would like to understand the comparison between approximate $\\ell_p$ sensitivities and Lewis weights. From what I understand, it is known that Lewis weights cover sensitivities, at least for $p \\in [1,2]$. The authors mentioned that Lewis weights are a crude approximation to sensitivities, although Lewis weights are used as a subroutine to obtain subspace embedding in their proposed algorithms. I am curious to see the benefits that an additive error approximation to sensitivities have over Lewis weight sampling.\n\nLIMITATIONS:\nIt would be beneficial if the authors consider adding a section addressing the open problems and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives algorithms to compute, approximately, the individual sensitivity scores, total sensitivity and maximum sensitivity, for $\\ell_p $ norms. The approximation for the individual sensitivity scores is additive while a relative error approximation is obtained for both total and maximum sensitivity. Calculating exact sensitivity scores often being a computationally expensive problem, the authors provide faster algorithms that use techniques of multiplication with Rademacher vectors and hashing. The authors also empirically validate their results on real datasets.\n\nSTRENGTHS:\nStrengths of the paper\n1) Methods of constructing coresets for various ML problems rely heavily on good approximation to sensitivity scores. As such the paper is important and will be of interest to the community.\n2) Empirical Results for various values of $p$ which are not usually seen in literature.\n\nWEAKNESSES:\nTo me the main weakness appears to be with the writing and clarity of the paper (may be because of space constraints) and also to an extent comparison and discussion with some related works. Here I list out some of the questions/ suggestions that I have: \n1) The authors mention Lewis weights as crude over approximation to sensitivity scores for $\\ell_p$. However, another way to sample rows for $\\ell_p$ subspace embeddings is using the row norms of a well-conditioned basis. The authors have not discussed much from the area. It would be interesting to compare if for some well-conditioned basis, the row norms correspond to some approximation of sensitivity scores.\n2) In all the algorithms the authors rely on constructing SA which is an $\\ell_1$ subspace embedding. There is good amount of work on $\\ell_1$ subspace embeddings using 1-stable Cauchy random variables, exponential random variables etc. It has not been compared with. It would be useful to give a table that compares these methods in terms of time, approximation factors and no. of rows required etc. also how is SA calculated in this work and what is the time taken for it? Please clarify. \n3) Also, algorithm 2 requires computation of leverage scores of A. Are they calculated exactly in which case the time required will be $O(nd^2)$ and if they are also approximated, how does the approximation factor figure in your guarantees?\n4) The implications of the lower bound are not clarified. Please elaborate.\n5) It would be useful to give some motivation as to why maximum of the sensitivity scores is important.\n\nOverall, the paper uses some known techniques in Randomized numerical linear algebra literature to calculate the approximations to $\\ell_p$ sensitivities. However improved writing, better clarification of exact contributions and comparison with existing literature will strengthen it.\n\nQUESTIONS:\nPlease see weaknesses section.\n\nLIMITATIONS:\nPlease see weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a randomized algorithm for efficiently approximating the $\ell_p$ sensitivities, with a constant approximation parameter guaranteed.

STRENGTHS:
This paper presents several novel randomized algorithms for approximating $\ell_p$ sensitivities and related statistics, based on two key ideas:

(1) Using subspace embeddings to efficiently approximate $\|Ax\|_p$ that avoids computing $Ax$ on the whole dataset

(2) Randomly hashing the dataset into small subsets and computing the sensitivities for each subset separately

Given the above, this work provides efficient approximations of the sensitivities of all data samples, the total sensitivity, and the maximum one. The resulting algorithms are good contributions to the problem of estimating sensitivities that fill in the blank of computing $\ell_p$ sensitivities efficiently. 

The theoretical guarantees appear to be solid and correct, which is further verified by several experiments.

QUESTIONS:
I am a bit confused about the definition of $\boldsymbol{S}$ in line 191, where the size of $\boldsymbol{S}$ is $r$ by $d$, should it be $r$ by $n$? Also, in lines 189 and 193 it is said $\boldsymbol{S}$ to be 'diagonal' while this seems to contradict its definition, please let me know if I miss anything.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven a matrix $A \\in \\mathbb{R}^{n \\times d}$ and $p \\in (0, \\infty)$, the $l_p$ sensitivity of a vector $a \\in \\mathbb{R}^d$ with respect to $A$ is defined as $\\sigma_p(a) := \\max_x |a^\\top x|^p / |Ax|_p^p$. It is known that by sampling each row $a_i$ of $A$ with probability proportional to (an upper bound on) $\\sigma_p(a_i)$, we can obtain a coreset of $A$ with respect to the $l_p$ loss. Therefore, estimating $\\sigma_p(a_i)$ quickly is desirable. This work presents three fast approximation algorithms for $l_p$ sensitivities. The main focus is on the case where $p = 1$, so I will state the results specifically for that case:\n\n- An algorithm that provides an estimate $\\tilde{\\sigma}$ such that $\\sigma_1(a_i) \\leq \\tilde{\\sigma} \\leq \\sigma_1(a_i) + \\alpha/n \\mathfrak{S}_1(A)$, where $\\mathfrak{S}_1(A)$ is the sum of $l_1$ sensitivities. The algorithm's running time is $O(n/\\alpha (nnz(A) + d^\\omega) + n)$, where $\\omega$ is the matrix multiplication exponent.\n- An algorithm that provides an estimate $\\tilde{\\sigma}$ such that $\\mathfrak{S}_1(A) \\leq \\tilde{\\sigma} \\leq (1 + \\gamma)\\mathfrak{S}_1(A)$. The running time of this algorithm is $O(\\sqrt{d} (nnz(A) + d^\\omega))$, significantly faster than the naive bound of $O(n (nnz(A) + d^\\omega))$.\n- An algorithm that provides an estimate $\\tilde{\\sigma}$ such that $\\max_i \\sigma_1(a_i) \\leq \\tilde{\\sigma} \\leq \\sqrt{d} \\max_i \\sigma_1(a_i)$. The running time for this algorithm is $O(d(nnz(A) + d^\\omega))$, also faster than the naive bound.\n\nThe authors showed that $l_p$ regression reduces to $l_p$ sensitivity calculation, implying that designing a fast algorithm for $l_p$ sensitivity requires a fast algorithm for $l_p$ regression. Experimental results are provided to validate the theoretical bounds.\n\nSTRENGTHS:\nThe algorithm effectively utilizes $l_p$ subspace embedding to enhance its speed. Additionally, the recursive approach employed for computing total sensitivity is intriguing.\n\nWEAKNESSES:\n- There are reservations about the usefulness of Theorem 1.2. It is possible to estimate sensitivities in $O(n (nnz(A) + d^\\omega))$ time, as indicated in Fact 2.4. This means that, to achieve a significant speedup, $\\alpha$ needs to be a function of $n$, which then results in a substantial additive error.\n- The usefulness of Theorem 1.3 is unclear. Although the sum provides a bound on the coreset's size, it is not evident when one would solely desire the bound without the coreset itself.\n- Similarly, the usefulness of Theorem 1.4 is not clear.\n\nQUESTIONS:\n- This is just a comment, but \"additive error of $O(\\alpha^p)$\" should be \"additive error of $\\alpha^p/n \\cdot \\mathfrak{S}_p(A)$\".\n- What is the meaning of the \"$\\rho$-factor subspace embedding\" in Algorithm 2? Should it be \"$\\rho$-approximate subspace embedding\"?\n\nLIMITATIONS:\nThe limitations are not clearly mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}