SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aditya Bhaskara
bhaskaraaditya@gmail.com
Microsoft Research-Redmond
Toyota Technological Institute at Chicago (TTIC)
Given a set of points of interest, a volumetric spanner is a subset of the points using which all the points can be expressed using "small" coefficients (measured in an appropriate norm). Formally, given a set of vectors  $X = \{v_{1}, v_{2}, \ldots, v_{n}\}$ , the goal is to find  $T \subseteq [n]$  such that every  $v \in X$  can be expressed as  $\sum_{i \in T} \alpha_{i} v_{i}$ , with  $\| \alpha \|$  being small. This notion, which has also been referred to as a well-conditioned basis, has found several applications, including bandit linear optimization, determinant maximization, and matrix low rank approximation. In this paper, we give almost optimal bounds on the size of volumetric spanners for all  $\ell_{p}$  norms, and show that they can be constructed using a simple local search procedure. We then show the applications of our result to other tasks and in particular the problem of finding coresets for the Minimum Volume Enclosing Ellipsoid (MVEE) problem.
In many applications in machine learning and signal processing, it is important to find the right "representation" for a collection of data points or signals. As one classic example, in the column subset selection problem (used in applications like feature selection, [Boutsidis et al., 2008]), the goal is to find a small subset of a given set of vectors that can represent all the other vectors via linear combinations. In the sparse coding problem, the goal is to find a basis or dictionary under which a collection of vectors admits a sparse representation (see [Olshausen and Field, 1997]).
In this paper, we focus on finding "bases" that allow us to represent a given set of vectors using small coefficients. A now-classic example is the notion of an Auerbach basis. Auerbach used an extremal argument to prove that for any compact subset  $X$  of  $\mathbb{R}^d$ , there exists a basis of size  $d$  (that is a subset of  $X$ ) such that every  $v \in X$  can be expressed as a linear combination of the basis vectors using coefficients of magnitude  $\leq 1$  (see, e.g., [Lindenstrauss and Tzafriri, 2013]). This notion was rediscovered in the ML community in the well-known work of Auerbuch and Kleinberg [2008], and subsequently in papers that used such a basis as directions of exploration in bandit algorithms. The term barycentric spanner has been used to refer to Auerbach bases. More recently, the paper of Hazan and Karnin [2016] introduced an  $\ell_2$  version of barycentric spanners, which they called volumetric spanners, and use them to obtain improved bandit algorithms.
The same notion has been used in the literature on matrix sketching and low rank approximation, where it has been referred to as a "well-conditioned basis" (or a spanning subset); see Dasgupta et al. [2009]. These works use well-conditioned bases to ensure that every small norm vector (in some normed space) can be expressed as a combination of the vectors in the basis using small coefficients.
Recently, Woodruff and Yasuda [2023] used the results of [Todd, 2016] and [Kumar and Yildirim, 2005] on minimum volume enclosing ellipsoids (MVEE) to show the existence of a well-conditioned spanning subset of size  $O(d\log \log d)$ . (Note that this bound was already superseded by the work of Hazan and Karnin [2016], who used different techniques.) Moreover, [Woodruff and Yasuda, 2023] demonstrated the use of well-conditioned bases for a host of matrix approximation problems (in offline and online regimes) such as low-rank approximation problems,  $\ell_p$  subspace embedding, and  $\ell_p$  column subset selection.
Our main contribution in this paper is to show that a simple local search algorithm yields volumetric spanners with parameters that improve both lines of prior work by Hazan and Karnin [2016] and Woodruff and Yasuda [2023]. Our arguments also allow us to study the case of having a general  $\ell_p$  norm bound on the coefficients. Thus, we obtain a common generalization with the results of Awerbuch and Kleinberg [2008] on barycentric spanners (which correspond to the case  $p = \infty$ ). Further our results can be plugged in, following the same approach of Woodruff and Yasuda [2023], to obtain improvements on a range of matrix approximation problems in these settings.
One application we highlight is the following. Volumetric spanners turn out to be closely related to another well-studied problem, that of finding the minimum volume enclosing ellipsoid (MVEE) for a given set of points, or more generally, for a given convex body  $K$ . This is a classic problem in geometry [Welzl, 1991, Khachiyan and Todd, 1990]. The celebrated result of Fitz John (e.g., see [Ball, 1992]) characterized the optimal solution for general  $K$ . Computationally, the MVEE can be computed using a semidefinite programming relaxation [Boyd et al., 2004], and more efficient algorithms have subsequently been developed; see [Cohen et al., 2019]. Coresets for MVEE (defined formally below) were used to construct well-conditioned spanning subsets in the recent work of Woodruff and Yasuda [2023]. We give a result in the opposite direction, and show that the local search algorithm for finding well-conditioned spanning sets can be used to obtain a coreset of size  $O(d / \epsilon)$ . This quantitatively improves upon prior work, as we now discuss.
We start with some notation. Suppose  $X = \{v_{1}, v_{2}, \ldots, v_{n}\}$  is a set of vectors in  $\mathbb{R}^d$ . We say that a subset  $S \subseteq [n]$  is a volumetric spanner [Hazan and Karnin, 2016] or a well-conditioned spanning subset [Woodruff and Yasuda, 2023], if for all  $j \in [n]$ , we can write  $v_{j} = \sum_{i \in S} \alpha_{i} v_{i}$ , with  $\| \alpha \|_2 \leq 1$ . More generally, we will consider the setting in which we are given parameters  $c, p$ , and we look to satisfy the condition  $\| \alpha \|_p \leq c$  (refer to Section 2) for a formal definition. Our main results here are the following.
Volumetric spanners via local search. For the  $\ell_2$  case, we show that there exists a volumetric spanner as above with  $|S| \leq 3d$ . Moreover, it can be found in polynomial time via a single-swap local search procedure (akin to ones studied in the context of determinant maximization [Madan et al., 2019]). Furthermore, our approach demonstrates the existence of a volumetric spanner of size  $2d - 1$ , marking a substantial progress in addressing the question posed by Hazan and Karnin [2016]. This helps to narrow the gap between the obvious  $d + 1$  lower bound and their  $12d$  upper bound.
In the case of  $p = 2$ , there are two main prior works. The first is the work of Hazan and Karnin [2016]. They obtain a linear sized basis, similar to our result. However, their result is weaker in terms of running time (by a factor roughly  $d^2$ ), as well as the constants in the size of the basis. Moreover, our algorithm is much simpler, it is simply local search with an appropriate objective, while theirs involves running a spectral sparsification subroutine (that involves keeping track of barrier potentials, etc.) followed by a rounding step.
The second work is the recent result of Woodruff and Yasuda [2023] which utilizes the prior work by Todd [2016]. Their algorithm is simple, essentially employing a greedy approach. However, it incurs an additional  $\log \log d$  factor in the size of the spanner due to the analysis of the greedy algorithm. Removing this factor is interesting conceptually, as it shows that local search with an appropriate objective can achieve a stronger guarantee than the best known result for greedy.
General  $p$  norms. For the case of general  $\ell_p$  norms, we show that a local search algorithm can still be used to find the near-optimal sized volumetric spanners. However, the optimal size exhibits three distinct behaviors:
- For  $p = 1$ , we show that there exist a set  $X$  of size  $n = \exp(d)$  for which any  $\ell_1$  volumetric spanner of size strictly smaller than  $n$  can only achieve  $\| \alpha \|_1 = \widetilde{\Omega}(\sqrt{n})$ . See Theorem 3.8 for the formal statement.
- For  $p \in (1, 2)$ , we show that  $\ell_p$  volumetric spanners that can achieve  $\| \alpha \|_p \leq 1$  exist, but require  $|S| = \Omega \left( \left( \frac{d}{\log n} \right)^{\frac{p}{2p - 2}} \right)$ . For strictly smaller sized  $S$ , we show a lower bound akin to the one above for  $p = 1$ . See Theorem 3.10.
- For  $p > 2$ , an  $\ell_p$  volumetric spanner (achieving  $\| \alpha \|_p \leq 1$ ) of size  $3d$  exists trivially because of the corresponding result for  $p = 2$ . See Theorem 3.6 and Corollary 3.12.
Our results show that one-swap local search yields near-optimal sized volumetric spanners for all  $\ell_p$  norms. We note that a unified treatment of the  $\ell_p$  volumetric spanner (for general  $p$ ), along with matching lower bounds have not been done in any of the prior work. Existing results treated the cases  $p = 2$  and  $p = \infty$ , using different techniques.
Coresets for MVEE. While well-conditioned spanning subsets have several applications [Woodruff and Yasuda, 2023], we highlight one in particular as it is a classic problem. Given a symmetric convex body  $K \in \mathbb{R}^d$ , the minimum volume enclosing ellipsoid (MVEE) of  $K$ , denoted  $\mathsf{MVEE}(K)$ , is defined as the ellipsoid  $\mathcal{E}$  that satisfies  $\mathcal{E} \supset K$ , while minimizing  $\mathrm{vol}(\mathcal{E})$ . We show that for any  $K$ , there exists a subset  $S$  of  $O\left(\frac{d}{\epsilon}\right)$  points of  $K$ , such that
$$
\operatorname {v o l} (\mathbf {M V E E} (K)) \leq (1 + \epsilon) ^ {d} \cdot \operatorname {v o l} (\mathbf {M V E E} (S)).
$$
This result is formally stated in Theorem 4.2. We define such a set  $S$  to be a coreset, and while it is weaker than notions of coresets considered for other problems (see the discussion in Section 2.1), it is the one used in the earlier works of Todd [2016], Kumar and Yildirim [2005]. We thus improve the size of the best known coreset constructions for this fundamental problem, indeed, by showing that a simple local search yields the desired coreset. This is conceptually interesting, because it matches the coreset size for a much simpler object, that is the axis-parallel bounding box of a set of points (which always has a coreset of size  $\leq 2d$ , obtained by taking the two extreme points along each axis).
Other applications. Our result can be used as a black-box to improve other results in the recent work of Woodruff and Yasuda [2023], such as entrywise Huber low rank approximation, average top  $k$  subspace embeddings and cascaded norm subspace embeddings and oblivious  $\ell_p$  subspace embeddings. In particular, we show that the local search algorithm provides a simple existential proof of oblivious  $\ell_p$  subspace embeddings for all  $p > 1$ . In this application, the goal is to find a small size "spanning subset" of a whole subspace of points (i.e., given a matrix  $A$ , the subspace is  $\{x \| Ax \|_p = 1\}$ ), rather than a finite set. Our results for oblivious  $\ell_p$  subspace embedding improve the bounds of non-constructive solution of Woodruff and Yasuda [2023] by shaving a factor of  $\log \log d$  in size.
In the context of dealing with large data sets, getting simple algorithms based on greedy or local search strategies has been a prominent research direction. A large number of works have been on focusing to prove theoretical guarantees for these simple algorithms (e.g. [Madan et al., 2019, Altschuler et al., 2016, Mahabadi et al., 2019, Civil and Magdon-Ismail, 2009, Mirzasoleiman et al., 2013, Anari and Vuong, 2022]). Our techniques are inspired by these works, and contribute to this literature.
Well-conditioned bases are closely related to "column subset selection" problem, where the goal is to find a small (in cardinality) subset of columns whose combinations can be used to approximate all the other columns of a matrix. Column subset selection has been used for matrix sketching, "interpretable" low rank approximation, and has applications to streaming algorithms as well. The classic works of [Frieze et al., 2004, Drineas et al., 2006, Boutsidis et al., 2009] all study this question, and indeed, the work of Boutsidis et al. [2009] exploit the connection to well-conditioned bases.
More broadly, with the increasing amounts of available data, there has been a significant amount of work on data summarization, where the goal is to find a small size set of representatives for a data set. Examples include subspace approximation [Achlioptas and McSherry, 2007], projective clustering [Deshpande et al., 2006, Agarwal and Procopiuc, 2003, Agarwal and Mustafa, 2004], determinant
maximization [Civril and Magdon-Ismail, 2009, Gritzmann et al., 1995, Nikolov, 2015], experimental design problems [Pukelsheim, 2006], sparsifiers [Batson et al., 2009], and coresets [Agarwal et al., 2005], which all have been extensively studied in the literature. Our results on coresets for MVEE are closely related to a line of work on contact points of the John Ellipsoid (these are the points at which an MVEE for a convex body touches the body). Srivastava [2012], improving upon a work of Rudelson [1997], showed that any convex  $K$  in  $\mathbb{R}^d$  can be well-approximated by another body  $K'$  that has at most  $O\left(\frac{d}{\epsilon^2}\right)$  contact points with its corresponding MVEE (and is thus "simpler"). While this result implies a coreset for  $K$ , it has a worse dependence on  $\epsilon$  than our results.
Definition 2.1 ( $\ell_p$ -volumetric spanner). Given a set of  $n \geq d$  vectors  $\{v_i\}_{i \in [n]} \subset \mathbb{R}^d$  and  $p \geq 1$ , a subset of vectors indexed by  $S \subset [n]$  is an  $c$ -approximate  $\ell_p$ -volumetric spanner of size  $|S|$  if for every  $j \in [n]$ ,  $v_j$  can be written as  $v_j = \sum_{i \in S} \alpha_i v_i$  where  $\| \alpha \|_p \leq c$ .
In particular, when  $c = 1$  the set is denoted as an  $\ell_p$ -volumetric spanner of  $\{v_1, \dots, v_n\}$ .
Determinant and volume. For a set of vectors  $\{v_{1}, v_{2}, \ldots, v_{d}\} \in \mathbb{R}^{d}$ ,  $\det \left(\sum_{i=1}^{d} v_{i} v_{i}^{T}\right)$  is equal to the square of the volume of the parallelepiped formed by the vectors  $v_{1}, v_{2}, \ldots, v_{d}$  with the origin.
The determinant maximization problem is defined as follows. Given  $n$  vectors  $v_{1}, v_{2}, \ldots, v_{n} \in \mathbb{R}^{d}$ , and a parameter  $k$ , the goal is to find  $S \subseteq [n]$  with  $|S| = k$ , so as to maximize  $\det \left( \sum_{i \in S} v_{i} v_{i}^{T} \right)$ . In this paper, we will consider the case when  $k \geq d$ .
Fact 2.2 (Cauchy-Binet formula). Let  $v_{1}, \dots, v_{n} \in \mathbb{R}^{d}$ , with  $n \geq d$ . Then
$$
\det  \left(\sum_ {i = 1} ^ {n} v _ {i} v _ {i} ^ {T}\right) = \sum_ {S \subset [ n ], | S | = d} \det  \left(\sum_ {i \in S} v _ {i} v _ {i} ^ {T}\right)
$$
Lemma 2.3 (Matrix Determinant Lemma). Suppose  $A$  is an invertible square matrix and  $u, v$  are column vectors, then
$$
\det  (A + u v ^ {T}) = (1 + v ^ {T} A ^ {- 1} u) \det  (A).
$$
Lemma 2.4 (Sherman-Morrison formula). Suppose  $A$  is an invertible square matrix and  $u, v$  are column vectors. Then,  $A + uv^{\top}$  is invertible iff  $1 + v^{\top}A^{-1}u \neq 0$ . In this case,
$$
\left(A + u v ^ {T}\right) ^ {- 1} = A ^ {- 1} - \frac {A ^ {- 1} u v ^ {T} A ^ {- 1}}{1 + v ^ {T} A ^ {- 1} u}
$$
We will also use the following inequality, which follows from the classic Hölder's inequality.
Lemma 2.5. For any  $1 \leq p \leq q$  and  $x \in \mathbb{R}^n$ ,  $\| x \|_p \leq n^{1/p - 1/q} \| x \|_q$ .
2.1 CoresetsofMVEE
As discussed earlier, for a set of points  $X \subset \mathbb{R}^d$ , we denote by  $\mathrm{MVEE}(X)$  the minimum volume enclosing ellipsoid (MVEE) of  $X$ . We say that  $S$  is a coreset for MVEE on  $X$  if
$$
\operatorname {v o l} (\mathsf {M V E E} (X)) \leq (1 + \epsilon) ^ {d} \cdot \operatorname {v o l} (\mathsf {M V E E} (S)).
$$
Strong vs. weak coresets. The notion above agrees with prior work, but it might be more natural (in the spirit of strong coresets considered for problems such as clustering; see, e.g., Cohen-Addad et al. [2021]) to define a coreset as a set  $S$  such that for any  $\mathcal{E} \supset S$ ,  $(1 + \epsilon)\mathcal{E} \supset X$ . Indeed, this guarantee need not hold for the coresets we (and prior work) produce. An example is shown in Figure 1.
3 Local Search Algorithm for Volumetric Spanners
We will begin by describing simple local search procedures LocalSearch-R and LocalSearch-NR. The former allows "repeating" vectors (i.e., choosing vectors that are already in the chosen set), while the latter does not.
LocalSearch-NR will be used for constructing well-conditioned bases, and LocalSearch-R will be used to construct coresets for the minimum volume enclosing ellipsoid problem.
Figure 1: Suppose  $X$  is the set of all points (blue and black), and let  $S$  be the set of black points. While  $\mathsf{MVEE}(X) = \mathsf{MVEE}(S)$ , there can be ellipsoids like the one in red, that contain  $S$  but not  $X$  even after scaling up by a small constant.
Algorithm 1 Procedure LocalSearch-NR
1: Input: Set of vectors  $\{v_{1}, v_{2}, \ldots, v_{n}\} \subseteq \mathbb{R}^{d}$ , parameter  $\delta > 0$ , integer  $r \geq d$
2: Output: Set of indices  $S$
3: Initialize  $S$  of size  $r$  using the greedy procedure described in the text
4: Define  $M = \sum_{i\in S}v_{i}v_{i}^{T}$
5: while  $\exists i\in S$  and  $j\in [n]\setminus S$  such that  $\operatorname *{det}(M - v_{i}v_{i}^{T} + v_{j}v_{j}^{T}) > (1 + \delta)\operatorname *{det}(M)$  do
6: Set  $S \gets S \setminus \{i\} \cup \{j\}$
7:  $M\gets M - v_{i}v_{i}^{T} + v_{j}v_{j}^{T}$
8: Return S
Initialization. The set  $S$  is initialized as the output of the standard greedy algorithm for volume maximization [Civril and Magdon-Ismail, 2009] running for  $d$  iterations, and then augmented with a set of  $(r - d)$  arbitrary vectors from  $\{v_{1},\ldots ,v_{n}\}$ .
Procedure LocalSearch-R. The procedure LocalSearch-R (where we allow repetitions) is almost identical to Algorithm 1. It uses the same initialization, however, the set  $S$  that is maintained is now a multiset. More importantly, when finding  $j$  in the local search step, LocalSearch-R looks over all  $j \in [n]$  (including potentially  $j \in S$ ). Also in this case, removing  $i$  from  $S$  in Line 6 corresponds to removing "one copy" of  $i$ .
3.1 Running time of Local Search
We will assume throughout that the dimension of  $\operatorname{span}(\{v_1, v_2, \ldots, v_n\})$  is  $d$  (i.e., the given vectors span all of  $\mathbb{R}^d$ ; this is without loss of generality, as we can otherwise restrict to the span).
The following lemma bounds the running time of local search in terms of the parameters  $r, \delta$ . We note that we only focus on bounding the number of iterations of local search. Each iteration involves potentially computing  $nr$  determinants, and assuming the updates are done via the matrix determinant lemma and the Sherman-Morrison formula, the total time is roughly  $O(nrd^2)$ . This can be large for large  $n, r$ , and it is one of the well-known drawbacks of local search.
Lemma 3.1. The number of iterations of the while loop in the procedures LocalSearch-R and LocalSearch-NR is bounded by
$$
O \left(\frac {d}{\delta} \cdot \log r\right).
$$
Proof. In every iteration of the while loop, the determinant of the maintained  $M$  increases by at least a  $(1 + \delta)$  factor. Thus, suppose we define  $S^{*}$  to be the (multi-)set of  $[n]$  that maximizes  $\operatorname{det}(M^{*})$ , where  $M^{*} := \sum_{i \in S^{*}} v_{i} v_{i}^{T}$ . We claim that for the  $S$  used by the algorithm at initialization (and the corresponding  $M$ ), we have
$$
\det  \left(M ^ {*}\right) \leq \binom {r} {d} d! \cdot \det  (M). \tag {1}
$$
This follows from two observations. First, let  $T^{*}$  be the (multi-)set of  $[n]$  that has size exactly  $d$ , and maximizes  $\operatorname{det}\left(\sum_{i\in T^{*}}v_{i}v_{i}^{T}\right)$ . Indeed, such a set will not be a multi-set, as a repeated element will reduce the rank. From the bound of Civil and Magdon-Ismail [2009], we have that at initialization,  $M$  satisfies
$$
\det  \left(\sum_ {i \in T ^ {*}} v _ {i} v _ {i} ^ {T}\right) \leq d! \cdot \det  (M).
$$
Next, by the Cauchy-Binet formula, we can decompose  $\operatorname*{det}(M^{*})$  into a sum over sub-determinants of  $d$ -sized subsets of the columns. Thus there are  $\binom{r}{d}$  terms in the summation. Each such sub-determinant is at most  $\operatorname*{det}(\sum_{i\in T^*}v_iv_i^T)$ , as  $T^{*}$  is the maximizer. This proves (1).
Next, since the determinant increases by a factor  $(1 + \delta)$  in every iteration, the number of iterations is at most
$$
O \left(\frac {1}{\delta}\right) \cdot \left[ d \log d + d \log (e r / d) \right],
$$
where we have used the standard bound of  $\binom{r}{d} \leq \left( \frac{er}{d} \right)^d$ . This completes the proof.
3.2 Analysis of Local Search
We now prove some simple properties of the Local Search procedures. Following the notation of Madan et al. [2019], we define the following. Given a choice of  $S$  in the algorithm (which defines the corresponding matrix  $M$ ), let
$$
\tau_ {i} := v _ {i} ^ {T} M ^ {- 1} v _ {i}, \quad \tau_ {i j} := v _ {i} ^ {T} M ^ {- 1} v _ {j}. \tag {2}
$$
Note that  $M$  is invertible. The determinant of  $M$  does not decrease in the local search update (see condition in line 5) and the selected  $M$  in the initialization step is invertible. This easily follows from (a) the greedy algorithm is maximizing volume (which is proportional to determinant) and (b) the full set of vectors span a  $d$  dimensional space (which is an assumption we can make without loss of generality, as we can always work with the span of the vectors).
We remark that  $\tau_{i}$  is often referred to as the leverage score. We have the following (proof in Section B):
Lemma 3.2. Let  $v_{1}, \ldots, v_{n} \in \mathbb{R}^{d}$  and let  $S$  be a (multi-)set of indices in  $[n]$ . Define  $M = \sum_{i \in S} v_{i} v_{i}^{T}$ , and suppose  $M$  has full rank. Then,
-  $\sum_{i\in S}\tau_i = d,$
- For any  $i, j \in [n]$ ,  $\tau_{ij} = \tau_{ji}$ .
The following key lemma lets us analyze how the determinant changes when we perform a swap.
Lemma 3.3. Let  $S$  be a (multi-)set of indices and let  $M = \sum_{i\in S}v_{i}v_{i}^{T}$  be full-rank. Let  $i,j$  be any two indices. We have
$$
\det (M - v _ {i} v _ {i} ^ {T} + v _ {j} v _ {j} ^ {T}) = \det (M) \left[ (1 - \tau_ {i}) (1 + \tau_ {j}) + \tau_ {i j} ^ {2} \right].
$$
Remark. Note that the proof will not use any additional properties about  $i, j$ . They could be equal to each other, and  $i, j$  may or may not already be in  $S$ .
Proof. By the matrix determinant lemma (Lemma 2.3),
$$
\begin{array}{l} \det  (M + v _ {j} v _ {j} ^ {T} - v _ {i} v _ {i} ^ {T}) = \det  (M + v _ {j} v _ {j} ^ {T}) (1 - v _ {i} ^ {T} (M + v _ {j} v _ {j} ^ {T}) ^ {- 1} v _ {i}) \\ = \det  (M) \left(1 + v _ {j} ^ {T} M ^ {- 1} v _ {j}\right) \left(1 - v _ {i} ^ {T} \left(M + v _ {j} v _ {j} ^ {T}\right) ^ {- 1} v _ {i}\right). \tag {3} \\ \end{array}
$$
Next, we apply Sherman-Morrison formula (Lemma 2.4) to get
$$
1 - v _ {i} ^ {T} \left(M + v _ {j} v _ {j} ^ {T}\right) ^ {- 1} v _ {i} = 1 - v _ {i} ^ {T} \left(M ^ {- 1} - \frac {M ^ {- 1} v _ {j} v _ {j} ^ {T} M ^ {- 1}}{1 + v _ {j} ^ {T} M ^ {- 1} v _ {j}}\right) v _ {i} = 1 - \tau_ {i} + \frac {\tau_ {i j} ^ {2}}{1 + \tau_ {j}}. \tag {4}
$$
Combining the above two expressions, we get
$$
\det  (M + v _ {j} v _ {j} ^ {T} - v _ {i} v _ {i} ^ {T}) = \det  (M) (1 + \tau_ {j}) \left[ 1 - \tau_ {i} + \frac {\tau_ {i j} ^ {2}}{1 + \tau_ {j}} \right].
$$
Simplifying this yields the lemma.
The following lemma shows the structural property we have when the local search procedure ends.
Lemma 3.4. Let  $S$  be a (multi-)set of indices and  $M = \sum_{i\in S}v_{i}v_{i}^{T}$  as before. Let  $j\in [n]$ , and suppose that for all  $i\in S$ ,  $\operatorname*{det}(M - v_i v_i^T + v_j v_j^T) < (1 + \delta)\operatorname*{det}(M)$ . Then we have