SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \det  \left(\left. B ^ {\prime} \right.) ^ {T} B ^ {\prime}\right) = \det  \left(C \left(W ^ {\prime}\right) ^ {T} W ^ {\prime} C ^ {T}\right) \\ = \det  (C) ^ {2} \det  (D ^ {\prime}) \\ = \det  (C) ^ {2} \prod_ {j = i} ^ {k} \| v _ {j} ^ {\prime} \| _ {2} ^ {2} \\ \end{array}
$$
It remains to show that  $|\operatorname{det}(C)| \leq (1 + \sqrt{k})$ . We may assume that  $\alpha_{i}^{k+1} \geq 0$  by taking the negative of the first column if necessary. This does not affect the magnitude of the determinant. Note that all eigenvalues of  $C'$  and  $(C')^{-1}$  are 1. Further,
$$
\left\| x \right\| _ {2} ^ {2} \leq k - i + 1 \leq k \tag {12}
$$
$$
\begin{array}{l} | \det  (C) | = | \left(1 + x ^ {T} \left(C ^ {\prime}\right) ^ {- 1} e _ {1}\right) | \cdot | \det  \left(C ^ {\prime}\right) | \quad [ \text {b y} \\ = | 1 + x ^ {T} (C ^ {\prime}) ^ {- 1} e _ {1} | \\ \leq 1 + \sqrt {k} \lambda_ {\max} ((C ^ {\prime}) ^ {- 1}) \\ = 1 + \sqrt {k} \\ \end{array}
$$
We now provide a simple example where the output of the greedy algorithm is at best  $\sqrt{k}$ -locally optimal, thus demonstrating that the locality result for greedy is optimal up to the constant 1.
Theorem 9 (Tightness of Local Optimality). There exists a point set  $P = \{v_{1},\ldots ,v_{k},v_{k + 1}\}$  from which the greedy algorithm picks  $V = \{v_{1},\dots,v_{k}\}$ , and
$$
\frac {\operatorname {v o l} \left(V - v _ {1} + v _ {k + 1}\right)}{\operatorname {v o l} (V)} = \sqrt {k} \tag {13}
$$
Proof. Let  $P = \{v_{1},\ldots ,v_{k},v_{k + 1}\}$  where  $v_{1}\in \mathbb{R}^{k}$  is the vector of all ones and  $v_{i} = \sqrt{k} e_{i - 1}$  for  $i = 2,\dots ,k + 1$ . Since the magnitude of every vector in  $P$  is  $\sqrt{k}$ , the greedy algorithm could start by picking  $v_{1}$ . The greedy algorithm will then pick any  $k - 1$  of the remaining  $k$  vectors. Without loss in generality, assume that the algorithm picks  $V = \{v_{1},\dots ,v_{k}\}$ . Then  $\mathrm{vol}(V) = (\sqrt{k})^{k - 1}$ . On the other hand,  $\mathrm{vol}(V - v_1 + v_{k + 1}) = (\sqrt{k})^k$ . The result follows.
3 Application to Standard Determinant Maximization
The greedy algorithm for volume maximization was shown to have an approximation factor of  $k!$  in [CMI09]. We provide a completely new proof for this result with a slightly improved approximation factor.
Theorem 10. Let  $P$  be a point set,  $\mathrm{Greedy}(P) = \{v_1, \dots, v_k\}$  the output of the greedy algorithm, and  $\mathrm{MAXVOL}_k(P)$  the maximum volume of any subset of  $k$  vectors from  $P$ . Then
$$
\operatorname {v o l} (\text {G r e e d y} (P)) \geq \frac {\operatorname {M A X V O L} _ {k} (P)}{\prod_ {i = 2} ^ {k} \left(1 + \sqrt {i}\right)} \tag {14}
$$
Proof. Let  $S \subseteq P$  be the set of  $k$  vectors with maximum volume. Without loss of generality and for simplicity of exposition, we assume that  $\mathrm{Greedy}(P) \cap S = \emptyset$  (the proof still goes through if this is not the case). We will order  $S$  in a convenient manner.
Consider the set  $W_{1} = \{v_{1}\} \cup S$  with  $k + 1$  elements. Perform the greedy algorithm on  $W_{1}$  with  $k$  steps. Clearly, greedy will choose  $v_{1}$  first and then some  $k - 1$  of the remaining vectors. Label the left out vector  $w_{1}$ .
Inductively define  $W_{i+1} = \{v_1, \ldots, v_i, v_{i+1}\} \cup (S - \{w_1, \ldots, w_i\})$ , which has size  $k + 1$ . Perform greedy on  $W_{i+1}$  with  $k$  steps. The first  $i + 1$  vectors chosen will be  $v_1, \ldots, v_i, v_{i+1}$  by definition. Call the left out vector  $w_{i+1}$ . We now have an ordering for  $S = \{w_1, \ldots, w_k\}$ .
Starting with the greedy solution, we will now perform  $k$  swaps to obtain the optimal solution. Each swap will increase the volume by a factor of at most  $1 + \sqrt{k}$ . Initially, our solution starts with  $\text{Greedy}(P) = \{v_1, \ldots, v_k\}$ . Note that this is also the output of greedy when applied to the set  $\text{Greedy}(P) \cup \{w_k\} = W_k$ . Swapping in  $w_k$  in place of  $v_k$  increases our volume by a factor of at most  $1 + \sqrt{k}$ .
Our current set of vectors is now  $\{v_{1},\ldots ,v_{k - 1},w_{k}\}$ . By the ordering on  $S$ , this is also the greedy output on the set  $W_{k - 1} = \{v_{1},\dots ,v_{k - 1},w_{k - 1},w_{k}\}$ . Therefore, we may swap in  $w_{k - 1}$  in place of  $v_{k - 1}$  in our current set of vectors by increasing the volume by at most a factor of  $(1 + \sqrt{k})$ . Proceeding in this manner, we can perform  $k$  swaps to obtain the optimal solution from the greedy solution by increasing our volume by a factor of at most  $(1 + \sqrt{k})^{k}$ .
To obtain the slightly better approximation factor in the theorem statement, we observe that in the proof of Theorem 5, swapping out the  $i^{\mathrm{th}}$  vector from the greedy solution for a vector that was not chosen increases the volume only by a factor of  $(1 + \sqrt{k + 1 - i})\leq 1 + \sqrt{k}$  (Eq. (10),Eq. (12)), and that swapping out the  $k^{\mathrm{th}}$  vector does not increase the volume at all. Therefore, the approximation factor of greedy is at most
$$
\prod_ {i = 1} ^ {k - 1} (1 + \sqrt {k + 1 - i}) = \prod_ {i = 2} ^ {k} (1 + \sqrt {i})
$$
Remark 11. Note that  $\prod_{i=2}^{k}(1 + \sqrt{i}) < 2^k\sqrt{k!}$  for  $k \geq 7$ , which is  $(k!)^{\frac{1}{2} + o(1)}$ . While the improvement in the approximation factor is quite small, we emphasize that the proof idea is very different from the  $k!$  guarantee obtained in [CMI09].
In this section, we measure the local optimality parameter for the greedy algorithm empirically. We use two real world datasets, both of which were used as benchmarks for determinant maximization in immediately related work ([MIGR19, LJS16]:
- MNIST [LBBH98], which has 60000 elements, each representing a 28-by-28 bitmap image of a hand-drawn digit;
GENES [BQK+14], which has 10000 elements, with each representing the feature vector of a gene. The data set was initially used in identifying a diverse set of genes to predict breast cancer tumors. After removing the elements with some unknown values, we have around 8000 points.
We measure the local optimality parameter both as a function of  $k$ , and as a function of the data set size as explained in the next two subsections.
4.1 Local Optimality for Real and Random Datasets as a Function of  $k$
Experiment Setup: For both MNIST and GENES, we consider a collection of  $m = 10$  data sets, each with  $n = 3000$  points chosen uniformly at random from the full dataset. We ran the greedy algorithm for  $k$  from 1 to 20 and measured the local optimality value  $(1 + \epsilon)$  as a function of  $k = 2, 4, \ldots, 20$  for each of the 10 data sets in the collection. More precisely, for each such  $k$ , we took the maximum value of  $(1 + \epsilon)$  over every data set in the collection. The reason we take the worst value of  $(1 + \epsilon)$ , is that in the context of composable coresets, we require the guarantee to hold for each individual data set to be  $(1 + \epsilon)$ -locally optimal. We repeated this process for 5 iterations and took the average. We plot this value as a function of  $k$ .
Further, to compare against a random data set, for both MNIST and GENES, we repeated the above experiment against a set of random points of the same dimension sampled uniformly at random from the unit sphere.
Results: As shown in Fig. 1, while the real world data sets have local optimality value  $(1 + \epsilon)$  higher than the random data sets, they are both significantly lower than (less than 1.4) the theoretical bound of  $(1 + \sqrt{k})$ . This suggests that real world data sets behave much more nicely and are closer to random than the worst case analysis would suggest, which explains why greedy does so well in practice.
For the purpose of diversity maximization, the regime of interest is when  $k \ll n$ . However, we wanted to verify that the local optimality value does not increase much even when  $k$  is much larger and closer to  $n$ . Since measuring local optimality is expensive when both  $k$  and  $n$  are large, we ran the same experiment again, except with  $n = 300$  points per point set, and measuring the local optimality at  $k = 1, 50, 100, \ldots, 300$  in steps of 50. Again, as seen in Fig. 2, local optimality stays much below  $1 + \sqrt{k}$  (in fact less than 1.5) for larger values of  $k$  as well.
4.2 Local Optimality as a Function of the Size of Point Sets
Experiment Setup: Here, we fix the value of  $k \in \{5, 10, 15, 20\}$  and compute the local optimality value  $(1 + \epsilon)$  while increasing the size of the point sets. The point set size is varied from 500 to 4000 in intervals of 500. For each point set size, we chose a stream of 10 random point sets from the dataset and took the maximum value over 10 iterations. Once again, we did this on MNIST and GENES and took the average of 5 iterations.
Results: As shown in Fig. 3, the local optimality parameter remains very low (lower than 1.2) regardless of the number of points in the data set, which is much smaller than  $(1 + \sqrt{k})$ .
In this work, we provided an almost tight analysis of the greedy algorithm for determinant maximization in the composable coreset setting: we improve upon the previous known bound of  $C^{k^2}$  to  $O(k)^{3k}$ , which is optimal up to the factor 3 in the exponent. We do this by proving a result on the local optimality of the greedy algorithm for volume maximization. We also support our theoretical analysis by measuring the local optimality of greedy over real world data sets. It remains an interesting question to tighten the constant in the exponent or otherwise provide a lower bound showing that 3 is in fact optimal.
[MKSK13] Baharan Mirzasoleiman, Amin Karbasi, Rik Sarkar, and Andreas Krause. Distributed submodular maximization: Identifying representative elements in massive data. In Advances in Neural Information Processing Systems, pages 2049-2057, 2013.
[MZ15] Vahab Mirrokni and Morteza Zadimoghaddam. Randomized composable core-sets for distributed submodular maximization. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 153–162. ACM, 2015.
[Nik15] Aleksandar Nikolov. Randomized rounding for the largest simplex problem. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 861-870. ACM, 2015.
[PJG+14] Xinghao Pan, Stefanie Jegelka, Joseph E Gonzalez, Joseph K Bradley, and Michael I Jordan. Parallel double greedy submodular maximization. In Advances in Neural Information Processing Systems, pages 118-126, 2014.
[WIB14] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Fast multi-stage submodular maximization. In International conference on machine learning, pages 1494–1502, 2014.
[YFZ+16] Jin-ge Yao, Feifan Fan, Wayne Xin Zhao, Xiaojun Wan, Edward Y Chang, and Jianguo Xiao. Tweet timeline generation with determinantal point processes. In AAAI, pages 3080-3086, 2016.