SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sepideh Mahabadi*
Redmond, WA, USA
Stojan Trajanovski*
sttrajan@microsoft.com
We study core-set construction algorithms for the task of Diversity Maximization under fairness/partition constraint. Given a set of points  $P$  in a metric space partitioned into  $m$  groups, and given  $k_{1},\ldots ,k_{m}$ , the goal of this problem is to pick  $k_{i}$  points from each group  $i$  such that the overall diversity of the  $k = \sum_{i}k_{i}$  picked points is maximized. We consider two natural diversity measures: sum-of-pairwise distances and sum-of-nearest-neighbor distances, and show improved core-set construction algorithms with respect to these measures. More precisely, we show the first constant factor core-set w.r.t. sum-of-pairwise distances whose size is independent of the size of the dataset and the aspect ratio. Second, we show the first core-set w.r.t. the sum-of-nearest-neighbor distances. Finally, we run several experiments showing the effectiveness of our core-set approach. In particular, we apply constrained diversity maximization to summarize a set of timed messages that takes into account the messages' recency. Specifically, the summary should include more recent messages compared to older ones. This is a real task in one of the largest communication platforms, affecting the experience of hundreds of millions daily active users. By utilizing our core-set method for this task, we achieve a 100x speed-up while losing the diversity by only a few percent. Moreover, our approach allows us to improve the space usage of the algorithm in the streaming setting.
Data summarization problem is a class of tasks where a small subset of items must be selected as a summary to represent the whole data. Typically in applications, the selected summary is required to fulfill various criteria such as diversity. In this paper, we focus on Diversity Maximization (DM) which is a topic that has attracted significant attention over the past decades [11, 33, 3, 8, 9, 10, 31, 30, 24, 19, 20, 25, 14]. The goal of this line of research is to provide a summary of a large dataset that preserves the diversity of the data as much as possible. This has many applications in various domains including summarization, recommendation systems, search, and facility location [15, 43, 40, 1, 5, 13, 2, 21, 39, 33]. Formally, in this task, given a universe  $P$  of  $n$  items, its goal is to choose a small subset of size  $k$  that maximizes a given pre-specified measure of diversity.
Diversity Maximization under Partition Constraints is a variant of the problem that has been studied to find a diverse summary while satisfying some additional orthogonal constraints [7, 31, 3, 28, 4, 10, 8, 9, 6, 26]. Here, the items in the input are partitioned into  $m$  disjoint groups  $P = P_{1} \cup \dots \cup P_{m}$  and additionally, one is given a pre-specified number of desired results from each group  $k_{1}, \ldots, k_{m}$ , and the goal is to return  $k_{i}$  objects from each group  $P_{i}$  such that the overall diversity among all  $k = \sum_{i} k_{i}$  points is maximized. This formulation allows to control the number of objects from each category in the output, e.g., the number of movies from each genre shown to a user in a recommendation system, or to bound the number of old messages included in a summary of a user's feed. Moreover, this
*This is an equal contribution paper.
formulation has been studied in the context of fairness (e.g. see [28, 4, 10]), where one wishes to control the number of results from each population in the produced summary. We will therefore refer to the Diversity Maximization under Partition Constraints as Fair Diversity Maximization (FDM), and use  $\mathrm{div}_{k_1,\dots,k_m}(P)$  to refer to the optimal achievable diversity under the fairness constraint. In this work, we consider fair diversity maximization with respect to three common diversity measures.
Diversity Measures. Several measures (or objective functions) have been proposed in the literature to model the notion of diversity [11, 3, 6, 25, 28, 9, 20, 16, 10, 31, 30]. A main category of such measures are based on pairwise distances (see [11] for a complete list). In this work, we will consider three of the most common pairwise distance-based measures: (1) the minimum-pairwise distance, where for a subset of points  $S \subseteq P$ , their diversity is measured as the minimum pairwise distance between the points in the subset, i.e.,  $\min_{p,q \in S} \mathrm{dist}(p, q)$ ; (2) sum-of-pairwise distances between the points in the subset, i.e.,  $\sum_{p,q \in S} \mathrm{dist}(p, q)$ ; and (3) the sum-of-nearest-neighbor distances which is an interpolation between the first two measures, and is defined formally as  $\sum_{p \in S} \min_{q \in S \setminus \{p\}} \mathrm{dist}(p, q)$ . We refer to these measures respectively as MIN-PAIRWISE DIST, SUM-PAIRWISE DIST, and SUM-NN DIST. All these three measures have been used previously to model diversity (e.g. see [11, 20, 6, 25], where they are respectively referred to as remote-edge, remote-clique, and remote-pseudoforest). In this work, we study Fair Diversity Maximization (FDM) over large datasets with respect to these three diversity measures.
Core-sets for Diversity Maximization. As one of the main applications of diversity maximization is related to data summarization, there has been a large body of work to solve diversity maximization in massive data models of computation, and in particular the design of core-sets [20, 19, 24, 27, 28, 8, 9, 41, 25, 26, 14]. Core-set is a small subset of the data which is sufficient for computing an approximate solution of a pre-specified optimization problem on the whole data.
More specifically, in this work we focus on construction of composable core-sets[20]: we present a summarization algorithm  $\mathcal{A}$  that processes each group  $P_{i}$  independently and produces a small subset of it as its summary  $S_{i} = \mathcal{A}(P_{i})\subseteq P_{i}$ , with the property that the fair diversity of the data is approximately preserved, i.e.,  $\mathrm{div}_{k_1,\dots,k_m}(S)\geq \frac{1}{\alpha}\cdot \mathrm{div}_{k_1,\dots,k_m}(P)$ , where  $S$  is defined to be the union of all core-sets, i.e.,  $S = \bigcup_{i}S_{i}$ . Composable core-sets are in particular useful for big data models of computation such as distributed and streaming settings as discussed in details in [20]. For example, in a distributed setting where the dataset is partitioned over multiple machines based on groups, each machine can compute a core-set for its own dataset, and only send this small summary over to a single aggregator. The aggregator then processes the union of the summaries and outputs the solution. We provide further applications of the core-sets for processing timed datasets in the experiments sections of our paper.
1.1 Prior Work
Table 1 shows a summary of prior and our results.
Fair Diversity Maximization. Moumoulidou et al. [28] studied fair diversity maximization under MIN-PAIRWISE DIST and gave an  $O(m)$  approximation algorithm for the problem. The bound was later improved [4] to  $m + 1$ . However, it is not known whether a linear dependence on  $m$  is necessary. In fact the problem admits an  $O(1)$  approximation in the unconstrained version [17, 33]. For SUM-PAIRWISE DIST the problem admits a  $(1/2 + \epsilon)$  approximation [3, 7], which matches the performance of the best algorithm for the unconstrained diversity maximization. Finally, for SUM-NN DIST, Bhaskara et al. [6] presented a randomized  $O(1)$  approximation algorithm based on solving an LP for both DM and FDM.
Core-sets for Fair Diversity Maximization. Core-set construction algorithms for the unconstrained DM has been shown in [20] with respect to all three diversity measures. For FDM, Moumoulidou et al. [28] showed that running a greedy algorithm on each group independently provides a core-set with a constant approximation factor, with respect to the MIN-PAIRWISE DIST. For the SUM-PAIRWISE DIST, Ceccarello et al. [8] provides a core-set with the almost optimal  $(1 - \epsilon)$  approximation factor. However, the size of the core-set depends on a parameter similar to the aspect ratio of the dataset. In particular, they show that for the case of doubling metrics, the core-set size could have an exponential dependence on the doubling dimension. Under SUM-NN DIST, there has been no prior work on core-sets for FDM.
Table 1: The summary of prior and our results for FDM. Here  $k$  is the size of the solution and  $m$  is the number of groups. All the core-set results mentioned in the table have size  $poly(k)$ .
MIN-PAIRWISE DIST	SUM-PAIRWISE DIST	SUM-NN DIST
FDM	O(m) [28, 4]	O(1) [3]	O(1) [6]
Core-set for FDM	O(1) [28]	O(1) Section 2	O(m·log k) Section 4
Theoretical results. We show the following theoretical results in this paper.
- We show the first core-set construction algorithm for FDM under SUM-PAIRWISE DIST, with constant approximation factor whose size is independent of the number of points and the aspect ratio. In fact, we show a core-set of size only  $k_i^2$  for each group  $P_i$ , that together achieve a constant factor approximation (see Theorem 2.1).
- We also show the first core-set construction algorithm for the FDM under the SUM-NN DIST notion of diversity. We provide a core-set of size  $O(k^2)$  for each group  $P_i$  that together achieve an  $O(m \cdot \log k)$  approximation factor (see Theorem 4.1). We remark that although the approximation factor is probably not optimal, we see in the experiments section that the algorithm performs well on real data.
To get the above core-set algorithm, we first show in Section 3, an approximation algorithm for FDM under the SUM-NN DIST. More precisely, we show an  $O(m^2 \cdot \log k)$  approximation with polynomial running time (see Theorem 3.4) and  $O(m \cdot \log k)$  approximation with an exponential runtime in  $k$ , (see Theorem 3.5). This algorithm might be of independent interest as the best previous algorithm by [6], despite having the ideal  $O(1)$  approximation, is randomized and based on solving an LP, whereas our algorithm is deterministic and based on a greedy approach (see Algorithm 1). In fact all of our algorithms are simple to implement and thus practical as we show next.
- Finally, in Section C of the Appendices we show how to apply our results to the setting where the partitioning is not necessarily done according to the colors and thus get a fully composable core-set for these notions.
Experiments. We perform experiments for all three diversity measures. Our first set of experiments uses FDM as a tool to account for recency in a summary computed on timed datasets. Here, we group a dataset of messages by their created time. The goal is to compute a diverse summary of the messages, where we additionally require to include less number of messages in the summary from the older batches of messages, and more from the recent batches. This is a real task in one of the largest commercial communication platforms. We get the following experiments and results.
- First, we show that running diversity maximization algorithms without the group constraint does not satisfy our requirement and reports about equally number of old messages and the new ones.
- Next, we compute the price of fairness (i.e., price of balancedness in this context), that is we compute how much we lose diversity by imposing the grouping requirement. Our experiments on a Reddit dataset shows that the diversity only decreases by around  $1\%$  for sum-of-pairwise distances; few percent up to no more than  $20\%$  for sum-of-NN distances and around  $50\%$  for minimum-pairwise distances (due to its fragility) metrics.
- We use the core-set construction algorithms to summarize each group first, and then run our diversity maximization algorithms on the union of the core-sets. Our experiments show that using core-sets, the runtime of our algorithm improves on average by factor of  $100 \times$ , while only losing diversity by few percent. We further remark that using core-sets in this context has an additional benefit: it removes the need to recompute the summary on the whole data when new messages arrive: once we summarize a batch of old message, we no longer need to process the batch and only need to work with the core-set that is computed once.
We further show experiments that uses FDM as a tool for controlling the desired contribution of each genre in a movie recommendation system.
Finally, we run experiments to compare our proposed core-set of Section 2 to the prior algorithm of [8] for core-set construction. We see that while the core-set produced by our algorithm is on average smaller by a factor of  $200\mathrm{x}$ , its performance is only worse by  $1.3\%$ .
1.3 Preliminaries
Throughout this work, we assume that we are given a point set  $P = P_{1} \cup \dots \cup P_{m}$  in a metric space  $(\mathcal{X}, \mathrm{dist})$ . Each point in  $P$  comes from one of the  $m$  groups in  $[m]$ , which we also refer to as colors. We use  $P_{i}$  to denote the points of color  $i$ . We use  $n$  to denote  $|P|$  and  $n_{i}$  to denote  $|P_{i}|$ . We denote the distance between two points  $p, q$  by  $\mathrm{dist}(p, q)$  and for a set of points  $S$ , we use  $\mathrm{dist}(p, S)$  to denote  $\min_{q \in S} \mathrm{dist}(p, q)$ .
1.3.1 Optimization problem
The optimization problem considered in this paper is Fair Diversity Maximization as defined below.
Definition 1.1 (Fair Diversity Maximization (FDM)). Given a colored point set  $P = \bigcup_{i} P_{i}$ , and  $k_{1}, \ldots, k_{m}$ , the goal is to pick subsets  $S_{i} \subseteq P_{i}$ , such that first  $|S_{i}| = k_{i}$ , and second  $\mathrm{div}(S)$  is maximized where  $S = \bigcup_{i} S_{i}$ . We will use  $\mathrm{div}_{k_{1}, \ldots, k_{m}}(P)$  to denote the optimal diversity one can achieve this way, i.e.,
$$
\operatorname{div}_{k_{1},\dots ,k_{m}}(P) = \max_{S_{1}\subseteq P_{1},\dots ,S_{m}\subseteq P_{m}:|S_{i}| = k_{i}}\operatorname {div}(\bigcup_{i\leq m}S_{i})
$$
We will use  $k$  to denote  $\sum_{i} k_{i}$ , and thus  $|S| = k$ .
Definition 1.2 (Diversity Measures). For a set of points  $S$ , we consider the following diversity measures in this work: i) MIN-PAIRWISE DIST:  $\min_{p,q\in S}\mathrm{dist}(p,q)$ ; ii) SUM-PAIRWISE DIST:  $\sum_{p,q\in S}\mathrm{dist}(p,q)$ ; iii) SUM-NN DIST:  $\sum_{p\in S}\min_{q\in S\setminus \{p\}}\mathrm{dist}(p,q)$ .
1.3.2 Summarization task
The goal in our paper is to provide an intermediate summarization (i.e., a core-set) algorithm such that the union of the core-sets contains a good solution relative to the whole data. Let us formally define this notion.
Definition 1.3 (Core-set). Given a point set  $P$ , a subset  $T \subseteq P$  is called an  $\alpha$ -approximate core-set with respect to an optimization function  $f$  if the optimal value of  $f$  over  $T$  is within a factor  $\alpha$  of the optimal value of  $f$  over  $P$ .
It is desired that the size of the core-set is small. In this paper we focus on the optimization function  $f$  being the diversity maximization function. Further, we emphasize that our algorithm for constructing core-sets for  $P$ , processes each group  $P_{i}$  independent of other groups. Therefore, it provides a form of composability property as defined below.
Definition 1.4 (Color-Abiding Composable Core-set). An algorithm  $\mathcal{A}$  is said to construct an  $\alpha$ -approximate color-abiding composable core-set, if it produces a subset  $T_{i} = \mathcal{A}(P_{i}) \subseteq P_{i}$  independent of the points in other colors, s.t.  $\mathrm{div}_{k_1,\ldots,k_m}(T) \geq \frac{1}{\alpha} \mathrm{div}_{k_1,\ldots,k_m}(P)$ , where  $T = \bigcup_{i} T_{i}$ .
Again, we note that although it is desirable that the size of the core-set  $|T_i|$  is small, it does not necessarily need to be  $k_{i}$ . Later on, as a post-processing, one can use any exact or approximation algorithm on the union of the computed core-sets, i.e.,  $T$  to get a final solution. However the focus of this work is on the core-set computation algorithm  $\mathcal{A}$ .
Throughout this work, for brevity we use **composable core-set** to refer to Definition 1.4. However, we remark that the above definition of color-abiding composable core-set is only applicable when points are partitioned based on the colors, and differs from the standard notion of **composable core-sets** defined in [20], where the partitioning is done arbitrarily. Later in Section C of the Appendices, we show how to employ our algorithms and get fully composable core-sets (that is not necessarily color-abiding) as defined in [20].
1.3.3 The GMM algorithm
In this work we will use the greedy algorithm of [17, 33] which we denote by GMM and is depicted in Algorithm 1. Let us also state three well-known properties of GMM:
1.  $r_i$ 's are decreasing, i.e., for  $j > i$ , we have  $r_i \geq r_j$ .
2. For any  $i < k$ , let  $S_{i} = \{p_{1},\ldots ,p_{i}\}$ . Then for any  $p\in P\setminus S_i$ , we have that  $\mathrm{dist}(p,S_i)\leq r_{i + 1}$
3. Let  $T \subset_k P$  be any subset of  $k$  points in  $P$ . Then the minimum pairwise distance in  $T$  is at most  $2 \cdot r_k$ .
Algorithm 1 The GMM Algorithm
Input a point set  $P$  , and  $k$    
Output subset  $S = \{p_{1},\ldots ,p_{k}\} \subseteq P$  , and radii  $r_1,\dots ,r_k$  1:  $S\gets$  an arbitrary point from  $P$  , and  $r_1\gets \infty$    
2: for  $i = 2$  to  $k$  do   
3:  $p_i\gets$  the farthest point in  $P$  from  $S$  , i.e.,  $\mathrm{argmax}_{p\in P}\mathrm{dist}(p,S)$    
4:  $S\gets S\cup p_i$    
5:  $r_i\gets$  minimum pairwise distance in  $S$  , i.e.,  $\min_{p_1,p_2\in S}\mathrm{dist}(p_1,p_2)$    
6: end for   
7: return  $S$
2 Core-set for FDM under SUM-PAIRWISE DIST
In this section we show an algorithm for constructing a core-set for FDM with respect to the SUM-PAIRWISE DIST notion of diversity. Given a colored point set  $P = P_{1} \cup \dots \cup P_{m}$ , and  $k_{1}, \ldots, k_{m}$ , the goal is to come up with a core-set construction algorithm  $\mathcal{A}$  that independently summarizes each point set  $P_{i}$  and produces a small subset of its  $S_{i} = \mathcal{A}(P_{i}) \subseteq P_{i}$ , such that  $\mathrm{div}_{k_1,\ldots,k_m}(S) \geq \frac{1}{\alpha} \cdot \mathrm{div}_{k_1,\ldots,k_m}(P)$ , where again  $S = \bigcup_{i} S_{i}$ .
Overview of the algorithm. Suppose that we want to find a solution that maximizes FDM under SUM-PAIRWISE DIST notion of diversity. Consider the optimal solution OPT and let  $\mathrm{OPT}_i$  be the set of points from color  $i$ . Our algorithm for constructing a core-set proceeds by running GMM for  $k_i$  iterations on  $P_i$ , to get a seed of size  $k_i$ . It is then a property of GMM that if each point  $p \in \mathrm{OPT}_i$  is mapped (i.e. moved) to the closest seed in  $P_i$ , its distance does not change by more than  $r_i$  (as defined in GMM). Now the points in different colors can have different  $r_i$  and therefore the movements can be of different scales. For two points  $p, q \in \mathrm{OPT}$ , if the scales of their movements w.r.t. their original distance are small, then the new distance is still large compared to before. Otherwise, if the scale of these movements are large, we show how to charge the loss in the diversity of this pair onto other pairs that have relatively small movements. If there are not enough such pairs, one can then show that the solution returned by the seeds themselves have had large diversity. Finally, in order to make the map injective (so that no two points are mapped to the same core-set point), for each of the seed points, the algorithm also stores in the core-set, the closest  $k_i$  points to the seed, and thus increasing the size of the core-set to  $k_i^2$ .
There is one hard case in the above approach and that is when  $k_{i} = 1$ . In this case, the GMM algorithm can return any arbitrary point as the seed and that could provably be a bad core-set. To handle this case, we show that it is enough for the core-set algorithm to find at least a minimum of two seeds. Then we show that it is always possible to pick one of the seeds and get a reasonably good approximation. This modification is proved to work in Appendix B.
Algorithm description. For simplicity, let us assume that for all  $i$ , we have  $k_{i} \geq 2$ . Otherwise, we show in Appendix B that a slight modification of the algorithm works (although the proof becomes more involved). The core-set construction algorithm is shown in Algorithm 2. The algorithm proceeds by running GMM for  $k_{i}$  iterations to compute a set of  $k_{i}$  centers. Then for each of these  $k_{i}$  centers such as  $p$ , the algorithm stores at most  $k_{i}$  points from  $P_{i}$  whose closest center among all  $k_{i}$  centers is  $p$ .
Theorem 2.1. Algorithm 2 produces a core-set with size  $O(k_i^2)$  and a constant factor approximation:  $\mathrm{div}_{k_1,\dots,k_m}(S) \geq \frac{1}{C} \cdot \mathrm{div}_{k_1,\dots,k_m}(P)$  for a constant  $C$ . (Proof in Appendix A.1)
Algorithm 2 Core-set Construction Algorithm for SUM-PAIRWISE
Input a point set  $P_{i}$  together with parameters  $k_{i}$  and  $k$  (where  $k = k_{1} + \dots +k_{m})$    
Output a subset  $S_{i}\subseteq P_{i}$    
1:  $S_{i} = \{p_{1},\ldots ,p_{k_{i}}\} \gets \mathrm{GMM}(P_{i},k_{i})$    
2:  $T\gets \emptyset$    
3: for  $p\in S_i$  do   
4: for  $j = 1$  to  $k_{i}$  do   
5:  $T\gets T\cup$  any point  $p_j\in P_i\setminus T$  s.t.  $\mathrm{argmin}_{q\in S_i}\mathrm{dist}(p_j,q) = p.$    
6: end for   
7: end for   
8:  $S_{i}\gets S_{i}\cup T$    
9: return  $S_{i}$
3 Approximation Algorithm for FDM under SUM-NN DIST
As mentioned in the results section, in order to obtain our core-set construction algorithm under SUM-NN DIST in Section 4, first in this section we show an approximation algorithm for FDM under the SUM-NN DIST notion of diversity.
Overview of the algorithm. Consider the optimal solution OPT that maximizes FDM under SUM-NN DIST notion of diversity, and let  $\mathrm{OPT}_i$  be the set of points from color  $i$ . Now each point  $p \in \mathrm{OPT}$  has a contribution towards the diversity, i.e.,  $\mathrm{dist}(p, \mathrm{OPT} \setminus \{p\})$ , and thus the contribution of each group  $\mathrm{OPT}_i$  towards the optimal solution is well-defined. Let us call the color with the maximum contribution as the most significant color. Clearly if we find a solution SOL whose value is at least as large as the contribution of the most significant color, then we are within a factor of  $m$  of the optimal solution.
Next, suppose that the most significant color is  $i$  and let  $d_{1} \geq \dots \geq d_{k_{i}}$  be the contributions of the points in the optimal solution that are of color  $i$ . It is not hard to prove (as we will show later) that  $\max_{j}(j \cdot d_{j})$  approximates  $\sum_{j=1}^{k_{i}} d_{j}$  up to a factor of  $\log k$ . So we will construct a solution whose value is at least  $j \cdot d_{j}$  and this will be within a factor of  $(m \cdot \log k)$  of the optimal solution as promised.
More concretely, we will construct a solution such that it contains  $j$  points of color  $i$  where no other point (from any color) lies within a distance  $d_j$  of them (in reality, we find  $O(j)$  points such that no other point lies within a distance  $O(d_j)$  of them). Of course, we do not know  $i$ ,  $j$ , and  $d_j$  a priori. We can enumerate  $i$  and  $j$ . Further, if we run the GMM algorithm on color  $i$  for  $j$  iterations, then the minimum pairwise distance between the retrieved points is an approximation to  $d_j$ .
This itself is enough to pick  $O(j)$  points of color  $i$  that are  $O(d_{j})$  far from each other. Let us call these points the seeds. However, the remaining challenge is to pick the rest of the points (that is the  $k_{i} - O(j)$  points of color  $i$ , as well as  $k_{\ell}$  points of color  $\ell$  for all  $\ell \neq i$ ) away from these seeds.