Title: Tight Bounds for Volumetric Spanners and Applications
Venue: NeurIPS 2023
Accepted: yes
Keywords: volumetric spanner

well-conditioned basis

determinant maximization

minimum volume enclosing ellipsoid
Abstract: Given a set of points of interest, a volumetric spanner is a subset of the points using which all the points can be expressed using "small" coefficients (measured in an appropriate norm). Formally, given a set of vectors $X = [v_1, v_2, \dots, v_n]$, the goal is to find $T \subseteq [n]$ such that every $v \in X$ can be expressed as $\sum_{i\in T} \alpha_i v_i$, with $\Vert \alpha \Vert$ being small.  This notion, which has also been referred to as a well-conditioned basis, has found several applications, including bandit linear optimization, determinant maximization, and matrix low rank approximation. In this paper, we give almost optimal bounds on the size of volumetric spanners for all $\ell_p$ norms, and show that they can be constructed using a simple local search procedure. We then show the applications of our result to other tasks and in particular the problem of finding coresets for the Minimum Volume Enclosing Ellipsoid (MVEE) problem.

Tight Bounds for Volumetric Spanners and Applications

Abstract

Given a set of points of interest, a volumetric spanner is a subset of the points using which all the points can be expressed using 'small' coefficients (measured in an appropriate norm). Formally, given a set of vectors X = { v 1 , v 2 , . . . , v n } , the goal is to find T ⊆ [ n ] such that every v ∈ X can be expressed as ∑ i ∈ T α i v i , with ∥ α ∥ being small. This notion, which has also been referred to as a well-conditioned basis, has found several applications, including bandit linear optimization, determinant maximization, and matrix low rank approximation. In this paper, we give almost optimal bounds on the size of volumetric spanners for all ℓ p norms, and show that they can be constructed using a simple local search procedure. We then show the applications of our result to other tasks and in particular the problem of finding coresets for the Minimum Volume Enclosing Ellipsoid (MVEE) problem.

1 Introduction

In many applications in machine learning and signal processing, it is important to find the right 'representation' for a collection of data points or signals. As one classic example, in the column subset selection problem (used in applications like feature selection, [Boutsidis et al., 2008]), the goal is to find a small subset of a given set of vectors that can represent all the other vectors via linear combinations. In the sparse coding problem, the goal is to find a basis or dictionary under which a collection of vectors admit a sparse representation (see [Olshausen and Field, 1997]).

In this paper, we focus on finding 'bases' that allow us to represent a given set of vectors using small coefficients. A now-classic example is the notion of an Auerbach basis. Auerbach used an extremal argument to prove that for any compact subset X of R d , there exists a basis of size d (that is a subset of X ) such that every v ∈ X can be expressed as a linear combination of the basis vectors using coefficients of magnitude ≤ 1 (see, e.g., [Lindenstrauss and Tzafriri, 2013]). This notion was rediscovered in the ML community in the well-known work of Awerbuch and Kleinberg [2008], and subsequently in papers that used such a basis as directions of exploration in bandit algorithms. The term barycentric spanner has been used to refer to Auerbach bases. More recently, the paper of Hazan and Karnin [2016] introduced an ℓ 2 version of barycentric spanners, which they called volumetric spanners , and use them to obtain improved bandit algorithms.

The same notion has been used in the literature on matrix sketching and low rank approximation, where it has been referred to as a 'well-conditioned basis' (or a spanning subset ); see Dasgupta et al. [2009]. These works use well-conditioned bases to ensure that every small norm vector (in some normed space) can be expressed as a combination of the vectors in the basis using small coefficients.

Sepideh Mahabadi

Recently, Woodruff and Yasuda [2023] used the results of [Todd, 2016] and [Kumar and Yildirim, 2005] on minimum volume enclosing ellipsoids (MVEE) to show the existence of a well-conditioned spanning subset of size O ( d log log d ) . (Note that this bound was already superseded by the work of Hazan and Karnin [2016], who used different techniques.) Moreover, [Woodruff and Yasuda, 2023] demonstrated the use of well-conditioned bases for a host of matrix approximation problems (in offline and online regimes) such as low-rank approximation problems, ℓ p subspace embedding, and ℓ p column subset selection.

Our main contribution in this paper is to show that a simple local search algorithm yields volumetric spanners with parameters that improve both lines of prior work by Hazan and Karnin [2016] and Woodruff and Yasuda [2023]. Our arguments also allow us to study the case of having a general ℓ p norm bound on the coefficients. Thus, we obtain a common generalization with the results of Awerbuch and Kleinberg [2008] on barycentric spanners (which correspond to the case p = ∞ ). Further our results can be plugged in, following the same approach of Woodruff and Yasuda [2023], to obtain improvements on a range of matrix approximation problems in these settings.

One application we highlight is the following. Volumetric spanners turn out to be closely related to another well-studied problem, that of finding the minimum volume enclosing ellipsoid (MVEE) for a given set of points, or more generally, for a given convex body K . This is a classic problem in geometry [Welzl, 1991, Khachiyan and Todd, 1990]. The celebrated result of Fitz John (e.g., see [Ball, 1992]) characterized the optimal solution for general K . Computationally, the MVEE can be computed using a semidefinite programming relaxation [Boyd et al., 2004], and more efficient algorithms have subsequently been developed; see [Cohen et al., 2019]. Coresets for MVEE (defined formally below) were used to construct well-conditioned spanning subsets in the recent work of Woodruff and Yasuda [2023]. We give a result in the opposite direction, and show that the local search algorithm for finding well-conditioned spanning sets can be used to obtain a coreset of size O ( d/ϵ ) . This quantitatively improves upon prior work, as we now discuss.

1.1 Our Results

We start with some notation. Suppose X = { v 1 , v 2 , . . . , v n } is a set of vectors in R d . We say that a subset S ⊆ [ n ] is a volumetric spanner [Hazan and Karnin, 2016] or a well-conditioned spanning subset [Woodruff and Yasuda, 2023], if for all j ∈ [ n ] , we can write v j = ∑ i ∈ S α i v i , with ∥ α ∥ 2 ≤ 1 . More generally, we will consider the setting in which we are given parameters c, p , and we look to satisfy the condition ∥ α ∥ p ≤ c (refer to Section 2) for a formal definition. Our main results here are the following.

Volumetric spanners via local search. For the ℓ 2 case, we show that there exists a volumetric spanner as above with | S | ≤ 3 d . Moreover, it can be found in polynomial time via a single-swap local search procedure (akin to ones studied in the context of determinant maximization [Madan et al., 2019]). Furthermore, our approach demonstrates the existence of a volumetric spanner of size 2 d -1 , marking a substantial progress in addressing the question posed by Hazan and Karnin [2016]. This helps to narrow the gap between the obvious d +1 lower bound and their 12 d upper bound.

In the case of p = 2 , there are two main prior works. The first is the work of Hazan and Karnin [2016]. They obtain a linear sized basis, similar to our result. However, their result is weaker in terms of running time (by a factor roughly d 2 ), as well as the constants in the size of the basis. Moreover, our algorithm is much simpler, it is simply local search with an appropriate objective, while theirs involves running a spectral sparsification subroutine (that involves keeping track of barrier potentials, etc.) followed by a rounding step.

The second work is the recent result of Woodruff and Yasuda [2023] which utilizes the prior work by Todd [2016]. Their algorithm is simple, essentially employing a greedy approach. However, it incurs an additional log log d factor in the size of the spanner due to the analysis of the greedy algorithm. Removing this factor is interesting conceptually, as it shows that local search with an appropriate objective can achieve a stronger guarantee than the best known result for greedy.

General p norms. For the case of general ℓ p norms, we show that a local search algorithm can still be used to find the near-optimal sized volumetric spanners. However, the optimal size exhibits three distinct behaviors:

For p = 1 , we show that there exist a set X of size n = exp( d ) for which any ℓ 1 volumetric spanner of size strictly smaller than n can only achieve ∥ α ∥ 1 = ˜ Ω( √ n ) . See Theorem 3.8 for the formal statement.

For p ∈ (1 , 2) , we show that ℓ p volumetric spanners that can achieve ∥ α ∥ p ≤ 1 exist, but require | S | = Ω ( ( d log n ) p 2 p -2 ) . For strictly smaller sized S , we show a lower bound akin to the one above for p = 1 . See Theorem 3.10.

For p > 2 , an ℓ p volumetric spanner (achieving ∥ α ∥ p ≤ 1 ) of size 3 d exists trivially because of the corresponding result for p = 2 . See Theorem 3.6 and Corollary 3.12.

Our results show that one-swap local search yields near-optimal sized volumetric spanners for all ℓ p norms. We note that a unified treatment of the ℓ p volumetric spanner (for general p ), along with matching lower bounds have not been done in any of the prior work. Existing results treated the cases p = 2 and p = ∞ , using different techniques.

Coresets for MVEE. While well-conditioned spanning subsets have several applications [Woodruff and Yasuda, 2023], we highlight one in particular as it is a classic problem. Given a symmetric convex body K ∈ R d , the minimum volume enclosing ellipsoid (MVEE) of K , denoted MVEE ( K ) , is defined as the ellipsoid E that satisfies E ⊃ K , while minimizing vol( E ) . We show that for any K , there exists a subset S of O ( d ϵ ) points of K , such that

This result is formally stated in Theorem 4.2. We define such a set S to be a coreset, and while it is weaker than notions of coresets considered for other problems (see the discussion in Section 2.1), it is the one used in the earlier works of Todd [2016], Kumar and Yildirim [2005]. We thus improve the size of the best known coreset constructions for this fundamental problem, indeed, by showing that a simple local search yields the desired coreset. This is conceptually interesting, because it matches the coreset size for a much simpler object, that is the axis-parallel bounding box of a set of points (which always has a coreset of size ≤ 2 d , obtained by taking the two extreme points along each axis).

Other applications. Our result can be used as a black-box to improve other results in the recent work of Woodruff and Yasuda [2023], such as entrywise Huber low rank approximation, average top k subspace embeddings and cascaded norm subspace embeddings and oblivious ℓ p subsapce embdeddings. In particular, we show that the local search algorithm provides a simple existential proof of oblivious ℓ p subsapce embdeddings for all p > 1 . In this application, the goal is to find a small size 'spanning subset' of a whole subspace of points (i.e., given a matrix A , the subspace is { x |∥ Ax ∥ p = 1 } ), rather than a finite set. Our results for oblivious ℓ p subsapce embdedding improves the bounds of non-constructive solution of Woodruff and Yasuda [2023] by shaving a factor of log log d in size.

1.2 Related work

In the context of dealing with large data sets, getting simple algorithms based on greedy or local search strategies has been a prominent research direction. A large number of works have been on focusing to prove theoretical guarantees for these simple algorithms (e.g. [Madan et al., 2019, Altschuler et al., 2016, Mahabadi et al., 2019, Civril and Magdon-Ismail, 2009, Mirzasoleiman et al., 2013, Anari and Vuong, 2022]). Our techniques are inspired by these works, and contribute to this literature.

Well-conditioned bases are closely related to 'column subset selection' problem, where the goal is to find a small (in cardinality) subset of columns whose combinations can be used to approximate all the other columns of a matrix. Column subset selection has been used for matrix sketching, 'interpretable' low rank approximation, and has applications to streaming algorithms as well. The classic works of [Frieze et al., 2004, Drineas et al., 2006, Boutsidis et al., 2009] all study this question, and indeed, the work of Boutsidis et al. [2009] exploit the connection to well-conditioned bases.

More broadly, with the increasing amounts of available data, there has been a significant amount of work on data summarization, where the goal is to find a small size set of representatives for a data set. Examples include subspace approximation [Achlioptas and McSherry, 2007], projective clustering [Deshpande et al., 2006, Agarwal and Procopiuc, 2003, Agarwal and Mustafa, 2004], determinant

maximization [Civril and Magdon-Ismail, 2009, Gritzmann et al., 1995, Nikolov, 2015], experimental design problems [Pukelsheim, 2006], sparsifiers [Batson et al., 2009], and coresets [Agarwal et al., 2005], which all have been extensively studied in the literature. Our results on coresets for MVEE are closely related to a line of work on contact points of the John Ellipsoid (these are the points at which an MVEE for a convex body touches the body). Srivastava [2012], improving upon a work of Rudelson [1997], showed that any convex K in R d can be well-approximated by another body K ′ that has at most O ( d ϵ 2 ) contact points with its corresponding MVEE (and is thus 'simpler'). While this result implies a coreset for K , it has a worse dependence on ϵ than our results.

2 Preliminaries and Notation

Definition 2.1 ( ℓ p -volumetric spanner) . Given a set of n ≥ d vectors { v i } i ∈ [ n ] ⊂ R d and p ≥ 1 , a subset of vectors indexed by S ⊂ [ n ] is an c -approximate ℓ p -volumetric spanner of size | S | if for every j ∈ [ n ] , v j can be written as v j = ∑ i ∈ S α i v i where ∥ α ∥ p ≤ c .

In particular, when c = 1 the set is denoted as an ℓ p -volumetric spanner of { v 1 , · · · , v n } .

Determinant and volume. For a set of vectors { v 1 , v 2 , . . . , v d } ∈ R d , det ( ∑ d i =1 v i v T i ) is equal to the square of the volume of the parallelopiped formed by the vectors v 1 , v 2 , . . . , v d with the origin.

The determinant maximization problem is defined as follows. Given n vectors v 1 , v 2 , . . . , v n ∈ R d , and a parameter k , the goal is to find S ⊆ [ n ] with | S | = k , so as to maximize det (∑ i ∈ S v i v T i ) . In this paper, we will consider the case when k ≥ d .

Fact 2.2 (Cauchy-Binet formula) . Let v 1 , · · · , v n ∈ R d , with n ≥ d . Then

Lemma 2.3 (Matrix Determinant Lemma) . Suppose A is an invertible square matrix and u, v are column vectors, then

̸

Lemma 2.4 (Sherman-Morrison formula) . Suppose A is an invertible square matrix and u, v are column vectors. Then, A + uv ⊤ is invertible iff 1 + v ⊤ A -1 u = 0 . In this case,

We will also use the following inequality, which follows from the classic Hölder's inequality.

Lemma 2.5. For any 1 ≤ p ≤ q and x ∈ R n , ∥ x ∥ p ≤ n 1 /p -1 /q ∥ x ∥ q .

2.1 Coresets for MVEE

As discussed earlier, for a set of points X ⊂ R d , we denote by MVEE ( X ) the minimum volume enclosing ellipsoid (MVEE) of X . We say that S is a coreset for MVEE on X if

Strong vs. weak coresets. The notion above agrees with prior work, but it might be more natural (in the spirit of strong coresets considered for problems such as clustering; see, e.g., Cohen-Addad et al. [2021]) to define a coreset as a set S such that for any E ⊃ S , (1 + ϵ ) E ⊃ X . Indeed, this guarantee need not hold for the coresets we (and prior work) produce. An example is shown in Figure 1.

3 Local Search Algorithm for Volumetric Spanners

We will begin by describing simple local search procedures LocalSearch-R and LocalSearch-NR . The former allows 'repeating' vectors (i.e., choosing vectors that are already in the chosen set), while the latter does not.

LocalSearch-NR will be used for constructing well-conditioned bases, and LocalSearch-R will be used to construct coresets for the minimum volume enclosing ellipsoid problem.

Algorithm 1 Procedure LocalSearch-NR

1: Input: Set of vectors { v 1 , v 2 , . . . , v n } ⊆ R d , parameter δ > 0 , integer r ≥ d

2: Output: Set of indices S

3: Initialize S of size r using the greedy procedure described in the text

4: Define M = ∑ i ∈ S v i v T i

5: while ∃ i ∈ S and j ∈ [ n ] \ S such that det( M -v i v T i + v j v T j ) > (1 + δ ) det( M ) do

6: Set S ← S \ { i } ∪ { j }

7: M ← M -v i v T i + v j v T j

8: Return S

Initialization. The set S is initialized as the output of the standard greedy algorithm for volume maximization [Civril and Magdon-Ismail, 2009] running for d iterations, and then augmented with a set of ( r -d ) arbitrary vectors from { v 1 , . . . , v n } .

Procedure LocalSearch-R . The procedure LocalSearch-R (where we allow repetitions) is almost identical to Algorithm 1. It uses the same initialization, however, the set S that is maintained is now a multiset. More importantly, when finding j in the local search step, LocalSearch-R looks over all j ∈ [ n ] (including potentially j ∈ S ). Also in this case, removing i from S in Line 6 corresponds to removing 'one copy' of i .

3.1 Running time of Local Search

We will assume throughout that the dimension of span ( { v 1 , v 2 , . . . , v n } ) is d (i.e., the given vectors span all of R d ; this is without loss of generality, as we can otherwise restrict to the span).

The following lemma bounds the running time of local search in terms of the parameters r, δ . We note that we only focus on bounding the number of iterations of local search. Each iteration involves potentially computing nr determinants, and assuming the updates are done via the matrix determinant lemma and the Sherman-Morrison formula, the total time is roughly O ( nrd 2 ) . This can be large for large n, r , and it is one of the well-known drawbacks of local search.

Lemma 3.1. The number of iterations of the while loop in the procedures LocalSearch-R and LocalSearch-NR is bounded by

Proof. In every iteration of the while loop, the determinant of the maintained M increases by at least a (1 + δ ) factor. Thus, suppose we define S ∗ to be the (multi-)set of [ n ] that maximizes det( M ∗ ) , where M ∗ := ∑ i ∈ S ∗ v i v T i . We claim that for the S used by the algorithm at initialization (and the corresponding M ), we have

This follows from two observations. First, let T ∗ be the (multi-)set of [ n ] that has size exactly d , and maximizes det( ∑ i ∈ T ∗ v i v T i ) . Indeed, such a set will not be a multi-set, as a repeated element will reduce the rank. From the bound of Civril and Magdon-Ismail [2009], we have that at initialization, M satisfies

Next, by the Cauchy-Binet formula, we can decompose det( M ∗ ) into a sum over sub-determinants of d -sized subsets of the columns. Thus there are ( r d ) terms in the summation. Each such sub-determinant is at most det( ∑ i ∈ T ∗ v i v T i ) , as T ∗ is the maximizer. This proves (1).

Next, since the determinant increases by a factor (1 + δ ) in every iteration, the number of iterations is at most

where we have used the standard bound of ( r d ) ≤ ( er d ) d . This completes the proof.

3.2 Analysis of Local Search

We now prove some simple properties of the Local Search procedures. Following the notation of Madan et al. [2019], we define the following. Given a choice of S in the algorithm (which defines the corresponding matrix M ), let

Note that M is invertible. The determinant of M does not decrease in the local search update (see condition in line 5 ) and the selected M in the initialization step is invertible. This easily follows from (a) the greedy algorithm is maximizing volume (which is proportional to determinant) and (b) the full set of vectors span a d dimensional space (which is an assumption we can make without loss of generality, as we can always work with the span of the vectors).

Weremark that τ i is often referred to as the leverage score. We have the following (proof in Section B):

Lemma3.2. Let v 1 , . . . , v n ∈ R d and let S be a (multi-)set of indices in [ n ] . Define M = ∑ i ∈ S v i v T i , and suppose M has full rank. Then,

The following key lemma lets us analyze how the determinant changes when we perform a swap.

Lemma 3.3. Let S be a (multi-)set of indices and let M = ∑ i ∈ S v i v T i be full-rank. Let i, j be any two indices. We have

Remark. Note that the proof will not use any additional properties about i, j . They could be equal to each other, and i, j may or may not already be in S .

Proof. By the matrix determinant lemma (Lemma 2.3),

Next, we apply Sherman-Morrison formula (Lemma 2.4) to get

Combining the above two expressions, we get

Simplifying this yields the lemma.

The following lemma shows the structural property we have when the local search procedure ends.

Lemma 3.4. Let S be a (multi-)set of indices and M = ∑ i ∈ S v i v T i as before. Let j ∈ [ n ] , and suppose that for all i ∈ S , det( M -v i v T i + v j v T j ) < (1 + δ ) det( M ) . Then we have

Once again, the lemma does not assume anything about j being in S .

Proof. First, observe that for any j ∈ [ n ] , we have

Combining this observation with Lemma 3.3 and summing over i ∈ S (with repetitions, if S is a multi-set), we have

and simplifying this completes the proof of the lemma.

Since Lemma 3.4 does not make any additional assumptions about j , we immediately have:

Corollary 3.5. The following properties hold for the output of the Local search procedures.

For LocalSearch-NR , the output S satisfies: for all j ∈ [ n ] \ S , τ j < d + rδ r -d +1 .

For LocalSearch-R , the output S satisfies: for all j ∈ [ n ] , τ j < d + rδ r -d +1 .

3.3 Volumetric Spanners: Spanning Subsets in the ℓ 2 Norm

We use Lemma 3.1 and Corollary 3.5 to obtain the following.

Theorem 3.6 ( ℓ 2 -volumetric spanner) . For any set X = { v 1 , v 2 , . . . , v n } of n ≥ d vectors in R d and parameter r ≥ d , LocalSearch-NR outputs a (max { 1 , ( d + rδ r -d +1 ) 1 / 2 ) -approximate ℓ 2 -volumetric spanner of X of size r in O ( d δ log r ) iterations of Local Search.

In particular, setting r = 3 d and δ = 1 / 3 , LocalSearch-NR returns an ℓ 2 -volumetric spanner of size 3 d in O ( d log d ) iterations of Local Search. Furthermore, setting r = 2 d -1 and δ = 0 , LocalSearch-NR implies the existence of an ℓ 2 -volumetric spanner of size 2 d -1 .

Proof. Let S be the output of LocalSearch-NR with the parameters r, δ on X . Let U be the matrix whose columns are { v i : i ∈ S } . We show how to express any v j ∈ X as Uα , where α ∈ R r is a coefficient vector with ∥ α ∥ 2 being small.

For any j ∈ S , v j can be clearly written with α being a vector that is 1 in the row corresponding to v j and 0 otherwise, thus ∥ α ∥ = 1 . For any j ̸∈ S , by definition, the solution to Uα = v j is α = U † v j , where U † is the Moore-Penrose pseudoinverse. Thus, we have

Here we are using standard properties of the pseudoinverse. (These can be proved easily using the SVD). Hence, by Corollary 3.5, we have ∥ α ∥ 2 ≤ ( d + rδ r -d +1 ) 1 / 2 .

3.4 Spanning Subsets in the ℓ p Norm

We now extend our methods above for all ℓ p -norms, for p ∈ [1 , ∞ ) . As outlined in Section 1.1, we see three distinct behaviors. We begin now with the lower bound for p = 1 .

ℓ 1 -volumetric spanner. For the case p = 1 , we show that small sized spanning subsets do not exist for non-trivial approximation factors.

Our construction is based on 'almost orthogonal' sets of vectors.

Lemma 3.7. There exists a set of m = exp(Ω( d )) unit vectors v 1 , . . . , v m ∈ R d such that for every pair of i, j ∈ [ m ] , |⟨ v i , v j ⟩| ≤ c √ log m d for some fixed constant c .

An example construction of almost orthogonal vectors is a collection of random vectors where each coordinate of each vector is picked uniformly at random from { 1 √ d , -1 √ d } (e.g., see [Dasgupta et al., 2009]).

Theorem 3.8 (Lower bound for ℓ 1 -volumetric spanners) . For any n ≤ exp(Ω( d )) , there exists a set of n vectors in R d that has no o ( √ d log n ) -approximate ℓ 1 -volumetric spanner of size at most n -1 .

In other words, unless the spanning subset contains all vectors, it is not possible to get an ℓ 1 -volumetric spanner with approximation factor o ( √ d log n ) .

Proof. Let X = { v 1 , . . . , v n } be a set of n almost orthonormal vectors as in Lemma 3.7. Suppose for the sake of contradiction, that there exists a spanning subset indexed by S that is a strict subset of [ n ] . Note that for every i ∈ [ n ] \ S and j ∈ S , |⟨ v i , v j ⟩| ≤ c √ log n d . So, for any representation of v i in terms of vectors in S , i.e., v i = ∑ α j v j , j ∈ S

Hence, ∥ α ∥ 1 ≥ 1 c √ d log n , as long as | S | < n .

Note that the lower bound nearly matches the easy upper bound that one obtains from ℓ 2 volumetric spanners (Theorem 3.6), described below:

Corollary 3.9. For any set of vectors X = { v 1 , v 2 , . . . , v n } , an ℓ 2 -volumetric spanner is also a 2 √ d -approximate ℓ 1 -volumetric spanner. Consequently, such a spanner of size O ( d ) exists and can be found in O ( d log d ) iterations of Local Search.

√

The proof follows from the fact that if ∥ α ∥ 2 ≤ 1 , ∥ α ∥ 1 ≤ 3 d , for α ∈ R 3 d (which is a consequence of the Cauchy-Schwarz inequality). Note that the existence and construction of an ℓ 2 volumetric spanner of size 3 d was shown in Theorem 3.6.

ℓ p -volumetric spanner for p ∈ (1 , 2) . Next, we apply the same argument as above for the case p ∈ (1 , 2) . Here, we see that the lower bound is not so strong: one can obtain a trade-off between the size of the spanner and the approximation. Once again, the solution returned by LocalSearch-NR is an almost optimal construction for spanning subsets in the ℓ p norm. The proofs are deferred to Section B of the Supplement.

Theorem 3.10 (Lower bound for ℓ p -volumetric spanners for p ∈ (1 , 2) ) . For any value of n ≤ e Ω( d ) and 1 < p < 2 , there exists a set of n vectors in R d that has no o ( r 1 p -1 · ( d log n ) 1 2 ) -approximate ℓ p -volumetric spanner of size at most r .

In particular, a ( 1 -approximate) ℓ p -volumetric spanner of V , has size Ω(( d log n ) p 2 p -2 ) .

Next, we show that local search outputs almost optimal ℓ p -volumetric spanners.

Theorem 3.11 (Construction of ℓ p -volumetric spanners for p ∈ (1 , 2) ) . For any set of vectors X = { v 1 , v 2 , . . . , v n } ⊂ R d and p ∈ (1 , 2) , LocalSearch-NR outputs an O ( r 1 p -1 · d 1 2 ) -approximate ℓ p -volumetric spanner of X of size r .

In particular, the local search algorithm outputs a 1 -approximate ℓ p -volumetric spanner when r = O ( d p 2 p -2 ) .

ℓ p -volumetric spanner for p > 2 . The result for p > 2 simply follows from the results for ℓ 2 -norm and the fact that ∥ x ∥ p ≤ ∥ x ∥ 2 for any p ≥ 2 when ∥ x ∥ 2 ≤ 1 .

Corollary 3.12 ( ℓ p -volumetric spanner for p > 2 ) . For any set of n vectors X = { v 1 , v 2 , . . . , v n } ⊂ R d , LocalSearch-NR outputs a 1 -approximate ℓ p -volumetric spanner of X of size r = 3 d in O ( d δ log d ) iterations of Local Search.

4 Applications of Local Search and Volumetric Spanners

We now give an application of our Local Search algorithms and volumetric spanners to the problem of finding coresets for the MVEE problem. For other applications, refer to Section A.

Definition 4.1 (Minimum volume enclosing ellipsoid (MVEE)) . Given a set of points X = { v 1 , v 2 , . . . , v n } ⊆ R d , define E ( X ) to be the ellipsoid of the minimum volume containing the points X ∪ ( -X ) , where ( -X ) := {-v : v ∈ X } .

While the MVEE problem is well-defined for general sets of points, we are restricting to sets that are symmetric about the origin. It is well-known (see Todd [2016]) that the general case can be reduced to the symmetric one. Thus for any X , E ( X ) is centered at the origin. Since E is convex, one can also define E ( X ) to be the ellipsoid of the least volume containing conv ( ± v 1 , ± v 2 , . . . , ± v n ) , where conv ( · ) refers to the convex hull.

As defined in Section 2.1, a coreset is a subset of X that preserves the volume of the MVEE.

Theorem 4.2. Consider a set of vectors X = { v 1 , · · · , v n } ⊂ R d . Let S be the output of the algorithm LocalSearch-R on X , with

Then S is a coreset for the MVEE problem on X .

To formulate the MVEE problem, recall that any ellipsoid E can be defined using a positive semidefinite (PSD) matrix H , as

and for E defined as such, we have vol( E ) = det( H -1 ) , up to a factor that only depends on the dimension d (i.e., is independent of the choice of the ellipsoid). Thus, to find E , we can consider the following optimization problem.

It is well known (e.g., Boyd et al. [2004]) that this is a convex optimization problem. For any λ ∈ R n with λ i ≥ 0 for all i ∈ [ n ] , the Lagrangian for this problem can be defined as:

Let OPT be the optimal value of the problem MVEE defined above. For any λ with non-negative coordinates, we have

where the minimization is over the feasible set for MVEE ; this is because over the feasible set, the second term of the definition of L ( H ; λ ) is ≤ 0 . We can then remove the feasibility constraint, and conclude that

as this only makes the minimum smaller. For any given λ with non-negative coordinates, the minimizing H can now be found by setting the derivative to 0 ,

There is a mild technicality here: if λ is chosen such that ∑ i λ i v i v T i is not invertible, then min H ⪰ 0 ( H ; λ ) = -∞ . We will only consider λ for which this is not the case. Thus, we have that for any λ with non-negative coordinates for which ∑ i λ i v i v T i is invertible,

We are now ready to prove Theorem 4.2 on small-sized coresets.

Proof of Theorem 4.2. Let X = { v 1 , v 2 , . . . , v n } be the set of given points, and let S be the output of the algorithm LocalSearch-R on X , with r, δ chosen later in (5). By definition, S is a multi-set of size r , and we define T to be its support, supp ( S ) . We prove that T is a coreset for the MVEE problem on X .

To do so, define OPT X and OPT T to be the optimum values of the optimization problem MVEE defined earlier on sets X and T respectively. Since the problem on T has fewer constraints, we have OPT T ≤ OPT X , and thus we focus on showing that OPT X ≤ d ln(1 + ϵ ) + OPT T . This will imply the desired bound on the volumes.

Let S be the multi-set returned by the algorithm LocalSearch-R , and let M := ∑ i ∈ S v i v T i . Define λ i = n i /r , where n i is the number of times i appears in S . By definition, we have that ∑ i ∈ [ n ] λ i = 1 . Further, if we define H := ( ∑ i ∈ [ n ] λ i v i v T i ) -1 , we have H -1 = 1 r · M .

Now, using Corollary 3.5, we have that for all j ∈ [ n ] ,

Our choice of parameters will be such that both the terms in the parentheses are (1 + ϵ/ 4) . For this, we can choose r, δ as in (5).

Thus, we have that H ′ = H (1+ ϵ ) is a feasible solution to the optimization problem MVEE on X . This implies that OPT X ≤ -ln det( H/ (1 + ϵ )) = d ln(1 + ϵ ) -ln det( H ) . The last equality follows because det( H/c ) = det( H ) /c d for a d -dimensional H , and any constant c .

Next, using the fact that the λ i are supported on T = supp ( S ) , we can use (6) to conclude that OPT T ≥ ln det( H -1 ) = -ln det( H ) , where we also used the fact that ∑ i λ i = 1 .

Together, these imply that OPT X ≤ d ln(1 + ϵ ) + OPT T , as desired.

5 Conclusion

We show that a one-swap local search procedure can be used to obtain an efficient construction of volumetric spanners, also known as well-conditioned spanning subsets. This improves (and simplifies) two lines of work that have used this notion in applications ranging from bandit algorithms to matrix sketching and low rank approximation. We then show that the local search algorithm also yields nearly tight results for an ℓ p analog of volumetric spanners. Finally, we obtain O ( d/ϵ ) sized coresets for the classic problem of minimum volume enclosing ellipsoid, improving previous results by a d log log d term.

An interesting open problem arising from our work is that of developing an efficient algorithm to find a ℓ 2 volumetric spanner of size 2 d -1 . While our local search approach demonstrates the existence of such spanner, it does not find it in polynomial time. Another interesting question is about further improving the gap: is it possible to obtain a volumetric spanner of size smaller than 2 d -1 for all instances?

Acknowledgments and Disclosure of Funding

The first author thanks the National Science Foundation for supporting his work via awards CCF2008688 and CCF-2047288.

References

Achlioptas and F. McSherry. Fast computation of low-rank matrix approximations. Journal of the ACM (JACM) , 54(2):9-es, 2007.

K. Agarwal and N. H. Mustafa. k -means projective clustering. In Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems , pages 155-165, 2004.

K. Agarwal and C. M. Procopiuc. Approximation algorithms for projective clustering. Journal of Algorithms , 46(2):115-139, 2003.

K. Agarwal, S. Har-Peled, K. R. Varadarajan, et al. Geometric approximation via coresets. Combinatorial and computational geometry , 52(1):1-30, 2005.

Altschuler, A. Bhaskara, G. Fu, V. Mirrokni, A. Rostamizadeh, and M. Zadimoghaddam. Greedy column subset selection: New bounds and distributed algorithms. In International conference on machine learning , pages 2539-2548, 2016.

Anari and T.-D. Vuong. From sampling to optimization on discrete domains with applications to determinant maximization. In Conference on Learning Theory , pages 5596-5618, 2022.

Awerbuch and R. Kleinberg. Online linear optimization and adaptive routing. Journal of Computer and System Sciences , 74(1):97-114, 2008.

Ball. Ellipsoids of maximal volume in convex bodies. Geometriae Dedicata , 41(2):241-250, 1992.

D. Batson, D. A. Spielman, and N. Srivastava. Twice-ramanujan sparsifiers. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 255-262, 2009.

Boutsidis, M. W. Mahoney, and P. Drineas. Unsupervised feature selection for principal components analysis. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 61-69, 2008.

Boutsidis, M. W. Mahoney, and P. Drineas. An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms , pages 968-977, 2009.

Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization . Cambridge university press, 2004.

Civril and M. Magdon-Ismail. On selecting a maximum volume sub-matrix of a matrix and related problems. Theoretical Computer Science , 410(47-49):4801-4811, 2009.

B. Cohen, B. Cousins, Y. T. Lee, and X. Yang. A near-optimal algorithm for approximating the john ellipsoid. In Conference on Learning Theory , pages 849-873, 2019.

Cohen-Addad, D. Saulpic, and C. Schwiegelshohn. A new coreset framework for clustering. In S. Khuller and V. V. Williams, editors, 53rd Annual ACM SIGACT Symposium on Theory of Computing (STOC) , pages 169-182, 2021.

Dasgupta, P. Drineas, B. Harb, R. Kumar, and M. W. Mahoney. Sampling algorithms and coresets for ℓ p regression. SIAM Journal on Computing , 38(5):2060-2078, 2009.

Deshpande, L. Rademacher, S. S. Vempala, and G. Wang. Matrix approximation and projective clustering via volume sampling. Theory of Computing , 2(1):225-247, 2006.

Drineas, M. W. Mahoney, and S. Muthukrishnan. Subspace sampling and relative-error matrix approximation: Column-based methods. In 9th International Workshop on Approximation Algorithms for Combinatorial Optimization Problems, APPROX 2006 and 10th International Workshop on Randomization and Computation, RANDOM 2006 , pages 316-326. Springer, 2006.

Frieze, R. Kannan, and S. Vempala. Fast monte-carlo algorithms for finding low-rank approximations. Journal of the ACM (JACM) , 51(6):1025-1041, 2004.

Gritzmann, V. Klee, and D. Larman. Largest j -simplices in n -polytopes. Discrete & Computational Geometry , 13:477-515, 1995.

Hazan and Z. Karnin. Volumetric spanners: an efficient exploration basis for learning. Journal of Machine Learning Research , 2016.

G. Khachiyan and M. J. Todd. On the complexity of approximating the maximal inscribed ellipsoid for a polytope. Technical report, Cornell University Operations Research and Industrial Engineering, 1990.

Kindler, A. Naor, and G. Schechtman. The UGC hardness threshold of the L p grothendieck problem. Mathematics of Operations Research , 35(2):267-283, 2010.

Kumar and E. A. Yildirim. Minimum-volume enclosing ellipsoids and core sets. Journal of Optimization Theory and applications , 126(1):1-21, 2005.

Lindenstrauss and L. Tzafriri. Classical Banach Spaces I: Sequence Spaces . Ergebnisse der Mathematik und ihrer Grenzgebiete. 2. Folge. Springer Berlin Heidelberg, 2013.

Madan, M. Singh, U. Tantipongpipat, and W. Xie. Combinatorial algorithms for optimal design. In Conference on Learning Theory , pages 2210-2258, 2019.

Mahabadi, P. Indyk, S. O. Gharan, and A. Rezaei. Composable core-sets for determinant maximization: A simple near-optimal algorithm. In International Conference on Machine Learning , pages 4254-4263, 2019.

Mirzasoleiman, A. Karbasi, R. Sarkar, and A. Krause. Distributed submodular maximization: Identifying representative elements in massive data. Advances in Neural Information Processing Systems , 26, 2013.

Nikolov. Randomized rounding for the largest simplex problem. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 861-870, 2015.

A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research , 37(23):3311-3325, 1997. ISSN 0042-6989.

Pukelsheim. Optimal design of experiments . SIAM, 2006.

Rudelson. Contact points of convex bodies. Israel Journal of Mathematics , 101:93-124, 1997.

Srivastava. On contact points of convex bodies. In Geometric Aspects of Functional Analysis: Israel Seminar 2006-2010 , pages 393-412, 2012.

J. Todd. Minimum-volume ellipsoids: Theory and algorithms . SIAM, 2016.

Welzl. Smallest enclosing disks (balls and ellipsoids). In New Results and New Trends in Computer Science , volume 555 of Lecture Notes in Computer Science , pages 359-370. Springer, 1991.

P. Woodruff and T. Yasuda. New subset selection algorithms for low rank approximation: Offline and online. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 1802-1813, 2023.

A Other Applications of Volumetric Spanners

We now show some direct applications of our construction of volumetric spanners.

A.1 Oblivious ℓ p Subspace Embeddings

Oblivious subspace embeddings (OSEs) are a well studied tool in matrix approximation, where the goal is to show that there exist sketching matrices that preserve the norm (say the ℓ p norm) of all vectors in an unknown subspace with high probability. The constructions and analyses of OSEs rely on the existence of a well-conditioned spanning set for the vectors of interest. The following follows directly from Theorem 3.6 (note that we are only using our result for ℓ 2 ).

Theorem A.1 (Improvement of Theorem 1.11 in [Woodruff and Yasuda, 2023]) . Let p ∈ (1 , ∞ ) and let A ∈ R n × d . There exists a matrix R ∈ R d × s for s = 3 d such that ∥ ARe i ∥ p = 1 for every i ∈ [ s ] , and for every x ∈ R d , ∥ Ax ∥ p = 1 , there exists a y ∈ R s such that Ax = ARy and ∥ y ∥ 2 ≤ 1 .

The Theorem follows by considering the set

and considering a well conditioned spanning subset in the ℓ 2 norm. Theorem 3.6 shows the existence of such a subset with s = 3 d , thus the theorem follows.

However, note that the proof is non-constructive. In order to make it polynomial time, we need to show that the local search procedure can be implemented efficiently. For p ≥ 2 , this may be possible via the classic result of Kindler et al. [2010] on ℓ p variants of the Gröthendieck inequality, but we note that the applications in [Woodruff and Yasuda, 2023] require only the existential statement.

A.2 Entrywise Huber Low Rank Approximation

The Huber loss is a classic method introduced as a robust analog to the least squares error. A significant amount of work has been dedicated to finding low-rank approximations of a matrix with the objective of minimizing the entry-wise Huber loss. The following offers a slight improvement over [Woodruff and Yasuda, 2023].

Theorem A.2. Let A ∈ R n × d and let k ≥ 1 . There exists a polynomial time algorithm that outputs a subset S ⊂ [ d ] of columns in A of size O ( k log d ) and X ∈ R S × d such that

where A | S denotes the matrix whose columns are the columns of A indexed by S and ∥ · ∥ H denotes the entrywise Huber loss.

Note that the size of S is reduced from O ( k log log k log d ) to O ( k log d ) . The proof of Theorem A.2 follows from Theorem 1.6 in [Woodruff and Yasuda, 2023] and our improved construction for ℓ 2 -volumetric spanner, i.e., O (1) -approximate spanning subset of size O ( d ) (see Theorem 3.6).

A.3 Average Top k Subspace Embedding

For a given vector v ∈ R n , the average top k loss is defined as

where v i denotes the i th largest coordinate in v .

Using the result of [Woodruff and Yasuda, 2023] relating the problem of average top k subspace embedding to ℓ 2 -volumetric spanners as a black-box, we have the following theorems for small k (i.e., k ≤ 3 d ) and large k (i.e., k > 3 d ) respectively.

Theorem A.3 (small k ) . Let A ∈ R n × d and let k ≤ 3 d . There exists a set S ⊂ [ n ] of size O ( d ) such that for all x ∈ R d ,

where A | S denotes the set of rows in A indexed by S .

Theorem A.4 (large k ) . Let A ∈ R n × d and let k ≥ k 0 where k 0 = O ( d + 1 δ ) . Let P 1 , . . . , P k t be a random partition of [ n ] into k t groups where t = O ( d +log 1 δ ) . For every i ∈ [ k t ] , there exists a set S i ⊂ N i of size O ( d ) such that with probability at least 1 -δ , for all x ∈ R d ,

where S := ⋃ i ∈ [ k t ] S i and A | S denotes the set of rows in A indexed by S .

In both regimes, compared to [Woodruff and Yasuda, 2023], our improved bounds for ℓ 2 -volumetric spanner saves a factor of log log d in the number of rows and a factor of √ log log d in the distortion. The proofs of above theorems respectively follows from Theorem 3.11 and 3.12 of [Woodruff and Yasuda, 2023] together with our Theorem 3.6.

A.4 Cascaded Norm Subspace Embedding

Next, we explore the implications of the improved bound of ℓ 2 -volumetric spanner (i.e., Theorem 3.6) for embedding a subspace of matrices under ( ∥∥ ∞ , ∥∥ ) -cascaded norm, which first evaluates an arbitrary norm of the rows and then return the maximum value over the n rows.

The following is a consequence of Theorem 3.13 in [Woodruff and Yasuda, 2023] and our Theorem 3.6. We describe our result for the ( ∥∥ ∞ , ∥∥ ) -cascaded norm, which first evaluates an arbitrary norm of rows and then return the maximum value over the n rows.

Theorem A.5 ( ( ∥∥ ∞ , ∥∥ ) -subspace embedding) . Let A ∈ R n × d and let ∥∥ be any norm on R m . There exists a set S ⊂ [ n ] of size at most 3 d such that for every X ∈ R d × m ,

B Missing Proofs

B.1 Proof of Lemma 3.2

Proof. Note that the second part follows from the symmetry of M (and thus also M -1 ). To see the first part, note that we can write v T i M -1 v i = ⟨ M -1 , v i v T i ⟩ , where ⟨ U, V ⟩ refers to the entry-wise inner product between matrices U, V , which also equals Tr ( U T V ) . Using this,

In the last equality, we used the symmetry of M .

B.2 Proof of Theorem 3.10

Proof. The proof follows from the same argument as before. Consider a set of n > r almost orthonormal vectors X = { v 1 , · · · , v n } ⊂ R d from Lemma 3.7.

Consider an index i ∈ [ n ] \ S and let v i = ∑ j ∈ S α j v j . By Lemma 2.5, for any p > 1 ,

In particular, to get a 1 -approximate ℓ p -volumetric spanner, i.e., ∥ α ∥ p = 1 , the spanning subset must have size r = Ω(( d log n ) p 2 p -2 ) .

B.3 Proof of Theorem 3.11

Proof. By Corollary 3.5, the local search outputs a set of vectors in X indexed by the set S ⊂ [ n ] of size r > d such that for every i ∈ [ n ] \ S , v i can be written as a linear combination of the vectors in

the spanner, v i = ∑ j ∈ S α j v j , such that ∥ α ∥ 2 ≤ ( d + rδ r -d +1 ) 1 2 . By Lemma 2.5 and setting δ = d/r , for any 1 < p < 2 ,

In particular, if we set r = O ( d p 2 -2 p ) , the subset of vectors S returned by LocalSearch-NR is an (exact) ℓ p -volumetric spanner; i.e., for every i ∈ [ n ] \ S , ∥ α ∥ p ≤ 1 .

Finally, the runtime analysis follows immediately from Lemma 3.1.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper researches the $\\ell_2$-volumetric spanner (or $\\ell_2$-well-conditioned spanning subset) for a given dataset $X$ based on the local search algorithm, in which each iteration implements a single swap to update the volumetric spanner $S$. More generally, the results are extended to $\\ell_p$-norm for any $p \\ge 1$. Furthermore, the proposed algorithm can be applied to build coresets of size $O(d/\\epsilon)$, which is independent of the number of data points $n$, for the minimum volume enclosing ellipsoid (MVEE) problem.\n\nSTRENGTHS:\n(1) The proposed algorithm is very easy to understand and the paper is well-written.  \n(2) This paper presents strict proof for the proposed algorithm.  \n(3) The coresets for the minimum volume enclosing ellipsoid problem are of size $O(d / \\epsilon)$ which is independent of the number of points $n$.\n\nWEAKNESSES:\n1. For the $\\ell_2$-volumetric spanner case  \n(1) In the paper [Hazan, Karnin, and Meka'13] (see Theorem 1.1), the  $1$-approximate spanner $S$ has size $|S| = 12 d$ and the running time is $O(n^{3.5} + n^3 d + d^5)$.  \n(2) In the paper [Woodruff and Yasuda'23] (see Theorem 3.4 and Corollary 3.5), the $(1+\\epsilon)$-approximate spanner $S$ has size $|S| = O(d \\log\\log{d} + d / \\epsilon)$ and the running time is $\\widetilde{O}((\\text{nnz}(A) + d^2) d / \\epsilon)$, where $\\text{nnz}(A) = O(nd)$.  \n(3) In this paper, setting $\\delta = O(\\epsilon)$ gives the $(1+\\epsilon)$-approximate spanner $S$ of size $|S| = O(d)$ and the running time $O(n d^4 \\log{d} / \\epsilon)$.  \nFrom the above comparison, we can see that the running time is much worse than that of [Woodruff and Yasuda'23] taking $\\epsilon \\in (0, 1)$ as a constant.  \n2. This paper gives the coresets for MVEE with size $|S| = O(d / \\epsilon)$ and running time $O(n d^4 \\log (d/\\epsilon) / \\epsilon^2)$. The size of coreset is impressive, but the running time and approximation ratio $(1+\\epsilon)^d$ are not satisfactory. In the paper [Cohen, Cousins, Lee, and Yang'19] (see Theorem 1.1), the $\\sqrt{d}$-approximate ellipsoid can be obtained in time $O(n d^2 \\log (n/d) / \\epsilon)$. The coreset is a proxy of dataset $X$, but computing the coreset of $X$ in this paper takes much more time than computing the approximate ellipsoid of $X$ directly in the paper of Cohen et al. For me, this is a critical drawback.\n\nQUESTIONS:\nIn Line 37-39, this paper claims that it improves both lines of prior work [Hazan, Karnin, and Meka'13] and [Woodruff and Yasuda'23]. What are the aspects of improvements?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

comments: SUMMARY:
This paper studies the problem of constructing small volumetric spanners. Given a set S of points in R^d, (the l2 version of) the problem is to find a subset of S so that every point in S can be written as a linear combination of the subset points, with the l2 norm of the coefficients bounded above by 1. Analogous problems can be defined for other lp norms.

There is a simple and efficient greedy algorithm that by folklore results gets a spanner of size O(d log d); there is an improvement due to Todd that gets O(d log log d), and Hazan, Karnin, and Meka get 12d via methods involving the John ellipsoid and spectral sparsification. The main result of this paper is that a slight modification of the greedy algorithm actually computes a spanner of size 3d.

STRENGTHS:
This paper gives a simpler and (slightly) better algorithm for computing an l2 volumetric spanner: specifically, a natural greedy approach works. It also gives simple extensions to other lp norms, and matching lower bounds.

WEAKNESSES:
The improvement in the spanner size is modest compared to prior work, and the algorithm is quite similar to known approaches (the only difference is that instead of iteratively adding elements to the spanner, each time we add a new element we also remove one), as is the analysis. I would not be entirely sure that this algorithm/analysis is not already present in the literature. However, since I could not find a reference I would err on the side of acceptance.

It would have been good to explicitly mention the bound obtained by the prior work (Hazan, Karnin, and Meka 2013).

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at the problem of identifying volumetric spanners from a set of points  \\in \\RR^d. The paper shows a method of local search which can be extended to find near optimal bounds of volumetric spanners, improving on previous algorithms. The authors also apply it to an application called MVEE, where they provide better rates for finding coresets of MVEE than previously in the literature. \n\nThe authors start with the generalization of volumetric spanners to the c-approximate use-case and analyze their local search method to show that for an L2 volumetric spanner of size 3d, the local search would only take O(d log d) time. They also extend their results for general l_p cases for p = 1, p \\in (1,2) and p>2, the result holding trivially for the latter. \n\nThey analyze the coresets of MVEE problem to show that they can obtain coresets of size O(d/\\epsilon) using local search method, which is an improvement over previous methods which have O(d/\\epsilon^2) bounds.\n\nSTRENGTHS:\nI have not completely checked the proofs, but the results are strong if the proofs hold. \nParticularly important is their usage in the spanning subset setting/usecase, which is a frequent need in today's age of very large sized datasets and having a small sized subset spanning the majority of the data points  can open doors to a lot of efficient analysis with higher dimensional data, as well as help us perform matrix vector operations on such datasets more efficiently. \nThe problem is very easy to describe and motivate and the algorithm is easy to follow.\n\nWEAKNESSES:\nThe comparison of results is quite difficult to follow from the paper. In particular, it could be pretty easily explained by having a table or having a clear description of the best possible previous result and how much it was improved by through the local search method. I am also not clear about the degree of optimality, which can be more clearly stated in the description itself. It would also be useful to have a description of the degree of optimality of the previous attempts to solve this problem.\n\nQUESTIONS:\nPlease look at the weakness section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops and analyzes a local-search algorithm for finding volumetric spanners under norms in the regime where the number of given vectors $n$ is at least as high as the dimension $d$ of these vectors. Moreover, a runtime of the algorithm is given and the size of the algorithm's output is compared to existing lower bounds. Finally, this paper presents applications of the algorithm to the minimum volume enclosing ellipsoid (MVEE) problem (in the paper's body) and other lower rank problems (in the appendices).\n\nSTRENGTHS:\nThe main novelty of the paper is the development of a (relatively) simple algorithm that obtains near-optimal results under different norms. A minor novelty is the improvement in the $\\ell_2$ case where the existing upper bound for computing well-conditioned coresets was improved from $O(d\\log\\log d)$ to $O(d)$. A particular point of originality is the application of the proposed algorithm to generate a concrete solution to $\\ell_p$ subspace embedding problem, in contrast to existing non-constructive solutions.\n\nWEAKNESSES:\nThis paper suffers from three major weaknesses: (i) unfocused comparisons of the paper's results to existing literature, (ii) technical issues with the mathematical results, and (iii) the lack of important details in several parts of the paper. Below, I outline some specific instances:\n\n(i.1) [Woodruff and Yasuda, 2023] - How does the notion of “distortion” in Definition 1.2 of [Woodruff and Yasuda, 2023] factor into the analysis of this paper. For example, do the coreset generated by Algorithm have high distortion when applied to the well-conditioned $\\ell_2$ coreset problem?\n\n(i.1) Runtime and Solution Size - It is difficult for the reader to find how the proposed algorithm compares to existing literature, as comments are generally scattered throughout the paper. A table that compares runtime and solution size would help in improving this issue.\n\n(ii.1) Proof of Thm. 4.2 - Assuming that indeed $OPT_X \\leq (1+\\epsilon)d + OPT_T$, the only conclusion is that $vol(MVEE(X)) \\leq \\exp((1+\\epsilon)d) \\times vol(MVEE(S))$, which does not coincide with the definition at the beginning of Subsection 2.1, as claimed.\n\n(ii.2) Lemma 3.4 - The analysis seems to break down when $r\\leq d+1$, but $r$ does not appear to be restricted in the algorithm (indeed, the “Initialization” discussion following Algorithm 1 seems to suggest that $r=d, d+1$ are valid cases.\n\n(ii.3) Section 3.2 - How do we know that $M$ remains invertible throughout the entire runtime of the algorithm. None of the results (including Lemma 3.3) seem to suggest that $\\det M \\neq 0$.\n\n(iii.1) Algorithm 1 - What is the significance of the parameter $r$? Can I set it to $d$ to be optimal?\n\n(iii.2) Algorithm 1 - A description of the initialization subroutine and its properties is missing. This is important for the reader verify that the presented algorithm is valid, e.g., $M$ starts with full rank.\n\n(iii.3) Proof of Thm 4.2 - Why is $\\sum n_i = r$? \n\n(iii.4) Proof of Thm 4.2 - Why is $H^{-1} = M / r$? Specifically, why is $n_i = 1$ for all $i$ (which is needed for this identity to hold in general).\n\nQUESTIONS:\nAside from the issues and questions pointed out in the “Weaknesses” section, I have a few minor suggestions/questions below:\n\n(1) Please use either $u_i$ or $\\lambda_i$ for the dual variables in the proof of Theorem, but do not use both.\n\n(2) Please make the use of parameter $r$ somewhere in Algorithm 1. At first glance, it appears to be an unused parameter, e.g., setting $r=d$ and $r=1000d$ has no effect on the result of the algorithm.\n\n(3) In equation (4), $r$ can be fractional. This doesn't seem to make sense in the context of the “Initialization” discussion after Algorithm 1.\n\nLIMITATIONS:\nYes, the authors have sufficiently addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}

comments: SUMMARY:
This paper considers the problem of constructing a volumetric spanner: a subset of vectors which allows to represent every other vector via a linear combination with coefficients whose lp-norm is small. The algorithms are based on local search and work for any lp-norm for p>=1. As a representative result (Theorem 3.6) one can find for a set X of size n>=d a subset of size at most 3d which can be used to represent all vectors in X via linear combinations with l2-norm of the coefficients <= 1. This requires O(d log d) iterations of local search. For l1-norm an exponential lower bound is given (Theorem 3.8). The result for p>2 follows trivially from Theorem 3.6 and for 1<p<2 a construction with certain parameters is given.

STRENGTHS:
The paper has clear results and is well-written. I think I understand all the main results and techniques quite well.

WEAKNESSES:
The applications to machine learning problems are somewhat weak (a few examples are given in the intro to column subset selection and sparse coding, but I am not exactly sure what the implications of the results in this paper are for the applications).

The claim that MVEE is an application is that we get an improvement on the size of the best known coreset construction. What is the magnitude of this improvement? Does this yield an algorithmic speedup?

There is an application mentioned to [Woodruff, Yasuda’23] where a log log d factor can be shaved. Can you elaborate on the implications of this as well? The discussion in Appendix A.1 is hard to follow. Can you restate the WY’23 result and explain what the contribution here is?

QUESTIONS:
– Line 17: sparse coding or problem?
– See a question regarding applications below.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-08-01