SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Eduardo S. Laber
Department of Computer Science PUC-Rio
Rio de Janeiro, RJ - Brazil
laber@inf.puc-rio.br
Lucas Murtinho
lmurtinho@aluno.puc-rio.br
Internal measures that are used to assess the quality of a clustering usually take into account intra-group and/or inter-group criteria. There are many papers in the literature that propose algorithms with provable approximation guarantees for optimizing the former. However, the optimization of inter-group criteria is much less understood.
Here, we contribute to the state-of-the-art of this literature by devising algorithms with provable guarantees for the maximization of two natural inter-group criteria, namely the minimum spacing and the minimum spanning tree spacing. The former is the minimum distance between points in different groups while the latter captures separability through the cost of the minimum spanning tree that connects all groups. We obtain results for both the unrestricted case, in which no constraint on the clusters is imposed, and for the constrained case where each group is required to have a minimum number of points. Our constraint is motivated by the fact that the popular single-linkage, which optimizes both criteria in the unrestricted case, produces clusterings with many tiny groups.
To complement our work, we present an empirical study with 10 real datasets, providing evidence that our methods work very well in practical settings.
Data clustering is a fundamental tool in machine learning that is commonly used in exploratory analysis and to reduce the computational resources required to handle large datasets. For comprehensive descriptions of different clustering methods and their applications, we refer to Jain et al. (1999); Hennig et al. (2015). In general, clustering is the problem of partitioning a set of items so that similar items are grouped together and dissimilar items are separated. Internal measures that are used to assess the quality of a clustering (e.g. Silhouette coefficient Rousseeuw (1987) and Davies-Bouldin index Davies & Bouldin (1979)) usually take into account intra-group and inter-group criteria. The former considers the cohesion of a group while the latter measures how separated the groups are.
There are many papers in the literature that propose algorithms with provable approximation guarantees for optimizing intra-group criteria. Algorithms for  $k$ -center,  $k$ -medians and  $k$ -means cost functions Gonzalez (1985); Charikar et al. (2002); Ahmadian et al. (2020) are some examples. However, optimizing inter-group criteria is much less understood. Here, we contribute to the state-of-the-art by considering the maximization of two natural inter-group criteria, namely the minimum spacing and the minimum spanning tree spacing.
Our Results. The spacing between two groups of points, formalized in Section 2, is the minimum distance between a point in the first and a point in the second group. We consider two criteria for
capturing the inter-group distance of a clustering, the minimum spacing (Min-Sp) and minimum spanning tree spacing (MST-Sp). The former is given by the spacing of the two groups with the smallest spacing in the clustering. The latter, as the name suggests, measures the separability of a clustering according to the cost of the minimum spanning tree (MST) that connects its groups; the largest the cost, the most separated the clustering is.
We first show that single-linkage, a procedure for building hierarchical clustering, produces a clustering that maximizes the MST spacing. This result contributes to a better understanding of this very popular algorithm since the only provable guarantee that we are aware of (in terms of optimizing some natural cost function) is that it maximizes the minimum spacing of a clustering Kleinberg & Tardos (2006)[Chap 4.7]. Our guarantee [Theorem 3.3] is stronger than the existing one in the sense that any clustering that maximizes the MST spacing also maximizes the minimum spacing. Figure 1 shows an example where the minimum spacing criterion does not characterize well the behavior of single-linkage.
Despite its nice properties, single-linkage tends to build clusterings with very small groups, which may be undesirable in practice. This can be seen in our experiments (see Figure 2) and is related to the well-documented fact that single-linkage suffers from the chaining effect (Jain et al., 1999)[Chap 3.2].
To mitigate this problem we consider the optimization of the aforementioned criteria under size constraints. Let  $L$  be a given positive integer that determines the minimum size that every group in a clustering should have. A  $(k, L)$ -clustering is a clustering with  $k$  groups in which the smallest group has at least  $L$  elements. For the Min-Sp criterion we devise an algorithm that builds clustering with at least  $L(1 - \epsilon)$  points per group while guaranteeing that its minimum spacing is not smaller than that of an optimal  $(k, L)$ -clustering. This result is the best possible in the sense that, unless  $P = NP$ , it is not possible to obtain an approximation with subpolynomial factor for the case in which the clustering is required to satisfy the hard constraint of  $L$  points per group. For the MST-Sp criterion we also devise an algorithm with provable guarantees. It produces a clustering whose MST-Sp is at most a log  $k$  factor from the optimal one and whose groups have each at least  $\rho L(1 - \epsilon)/2$  points, where  $\rho$  is a number in the interval [1, 2] that depends on the ratio  $n/kL$ . We also prove that the maximization of this criterion is APX-Hard for any fixed  $k$ .
Figure 1: Partitions with 3 groups (defined by colors) that maximize the minimum spacing. The rightmost one is built by single-linkage, but both of them maximize the minimum spacing - showing that this condition alone is insufficient to properly characterize single-linkage's behavior.
We complement our investigation with an experimental study where we compare the behaviour of the clustering produced by our proposed algorithms with those produced by  $k$ -means and single-linkage for 10 real datasets. Our algorithms, as expected, present better results than  $k$ -means for the criteria under consideration while avoiding the issue of small groups observed for single-linkage.
Related Work. We are only aware of a few works that propose clustering algorithms with provable guarantees for optimizing separability (inter-group) criteria. We explain some of them.
The maximum  $k$ -cut problem is a widely studied problem in the combinatorial optimization community and its solution can be naturally viewed as a clustering that optimizes a separability criterion. Given an edge-weighted graph  $G$  and an integer  $k \geq 2$ , the maximum  $k$ -cut problem consists of finding a  $k$ -cut (partition of the vertices of the graph into  $k$  groups) with maximum weight, where the weight of a cut is given by the sum of the weights of the edges that have vertices in different groups of the cut. The weight of the  $k$ -cut can be seen as a separability criterion, where the distance between two groups is given by the sum of the pairwise distances of its points. It is well-known that a random
assignment of the vertices (points) yields an  $(1 - 1 / k)$ -approximation algorithm. This bound can be slightly improved using semi-definitive programming Frieze & Jerrum (1997).
The minimum spacing, one of the criteria we studied here, also admits an algorithm with provable guarantees. In fact, as we already mentioned, it can be maximized in polynomial time via the single-linkage algorithm Kleinberg & Tardos (2006)[Chap 4.7]. In what follows, we discuss works that study this algorithm as well as works that study problems and/or methods related to it.
single-linkage has been the subject of a number of researches (Zahn, 1971; Kleinberg, 2002; Carlsson & Memoli, 2010; Hofmeyr, 2020). Kleinberg (2002) presents an axiomatic study of clustering, the main result of which is a proof that it is not possible to define clustering functions that simultaneously satisfy three desirable properties introduced in the paper. However, it was shown that by choosing a proper stopping rule for the single-linkage it satisfies any two of the three properties. Carlsson & Memoli (2010) replaces the property of Kleinberg (2002) that the clustering function must output a partition with the property that it must generate a tree (dendogram). Then, it was established that single-linkage satisfies the new set of properties. In more recent work, Hofmeyr (2020) establishes a connection between minimum spacing and spectral clustering. While the aforementioned works prove that single-linkage has important properties, in practice, it is reported that sometimes it presents poor performance due to the so-called chaining effect (Jain et al. (1999)[Chap 3.2]).
single-linkage belongs to the family of algorithms that are used to build Hierarchical Agglomerative Clustering (HAC). Dasgupta (2016) frames the problem of building a hierarchical clustering as a combinatorial optimization problem, where the goal is to output a hierarchy/tree that minimizes a given cost function. The paper proposes a cost function that has desirable properties and an algorithm to optimize it. This result was improved and extended by a series of papers (Roy & Pokutta, 2016; Charikar & Chatziafratis, 2017; Cohen-Addad et al., 2019). The algorithms discussed in these papers do not seem to be employed in practice. Recently, there has been some effort to analyse HAC algorithms that are popular in practice, such as the Average Link (Moseley & Wang, 2017; Cohen-Addad et al., 2019). Our investigation of single-linkage can be naturally connected with this line of research.
Finally, in a recent work, Ahmadi et al. (2022) studied the notion of individual preference stability (IP-stability) in which a point is IP-stable if it is closer to its group (on average) than to any other group (on average). The clustering produced by single-linkage presents a kind of individual preference stability in the sense that each point is closer to some point in its group than to any point in some other group.
Potential Applications. We discuss two cases in which the maximization of inter-group criteria via our algorithms may be relevant: ensuring data diversity when training machine learning algorithms and population diversity in candidate solutions for genetic algorithms.
When training a machine learning model, ensuring data diversity may be crucial for achieving good results (Gong et al., 2019). In situations in which all available data cannot be used for training (e.g. training in the cloud with budget constraints), it is important to have a method for selecting a diverse subset of the data, and our algorithms can be used for this: to select  $n = kL$  elements, one can partition the full data set into  $k$  clusters, all of them containing at least  $L$  members, and then select  $L$  elements from each cluster. Using single-linkage to create  $kL$  groups and then picking up one element per group is also a possibility but it would increase the probability of an over-representation of outliers in the obtained subset, as these outliers would likely be clustered as singletons (see Figure 2 in Section 5).
Note that our algorithms can be used to create not only one but several diverse and disjoint subsets, which might be relevant to generate partitions for cross-validation or for evaluating a model's robustness. For that, each subset is obtained by picking exactly one point per group.
For genetic algorithms, maintaining diversity over the iterations is important to ensure a good exploration of the search space (Gupta & Ghafir, 2012). If all candidate solutions become too similar, the algorithm will become too dependent on mutations for improvement, as the offspring of two solutions will likely be similar to its two parents; and mutation alone may not be enough to fully explore the search space. We can apply our algorithms in a similar manner as mentioned above, by partitioning, at each iteration, the solutions into  $k$  clusters of minimum size  $L$  and selecting the best  $L$  solutions from each cluster, according to the objective function of the underlying optimization problem, to maintain
the solution population simultaneously optimized and diverse. Using single-linkage could lead to several poor solutions being selected to remain in the population, in case they are clustered as singletons.
We finally note that an algorithm that optimizes intra-group measures (e.g.  $k$ -means or  $k$ -medians) would not necessarily guarantee diversity for the aforementioned applications, as points from different groups can be close to each other.
Let  $\mathcal{X}$  be a set of  $n$  points and let  $\mathrm{dist}:\mathcal{X}\times \mathcal{X}\mapsto \mathbb{R}^{+}$  be a distance function that maps every pair of points in  $\mathcal{X}$  into a non-negative real.
Given a  $k$ -clustering  $\mathcal{C} = (C_1, \ldots, C_k)$ , we define the spacing between two distinct groups  $C_i$  and  $C_j$  of  $\mathcal{C}$  as
$$
\operatorname {s p a c i n g} \left(C _ {i}, C _ {j}\right) := \min  _ {x \in C _ {i}, y \in C _ {j}} \left\{\operatorname {d i s t} (x, y) \right\}.
$$
Then, the minimum spacing of  $\mathcal{C}$  is given by
$$
\operatorname {Min - Sp}(\mathcal{C}):= \min_{\substack{C_{i},C_{j}\in \mathcal{C}\\ i\neq j}}\{\operatorname {spacing}(C_{i},C_{j})\}
$$
A  $k$ -clustering  $\mathcal{C}$  induces on  $(\mathcal{X}, \mathrm{dist})$  an edge-weighted complete graph  $G_{\mathcal{C}}$  whose vertices are the groups of  $\mathcal{C}$  and the weight  $w(e_{ij})$  of the edge  $e_{ij}$  between groups  $C_i$  and  $C_j$  is given by  $w(e_{ij}) = \mathrm{spaning}(C_i, C_j)$ . For a set of edges  $S \in G_{\mathcal{C}}$  we define  $w(S) := \sum_{e \in S} w(e)$ .
The minimum spanning tree spacing of  $\mathcal{C}$  (MST-Sp( $\mathcal{C}$ )) is defined as the sum of the weights of the edges of the minimum spanning tree (MST $(G_{\mathcal{C}})$ ) for  $G_{\mathcal{C}}$ . In formulae,
$$
\operatorname {M S T} - \operatorname {S p} (\mathcal {C}) := \sum_ {e \in M S T (G _ {\mathcal {C}})} w (e).
$$
Here, we will be interested in the problems of finding partitions with maximum Min-Sp and maximum MST-Sp both in the unrestricted case in which no constraint on the groups is imposed and in the constrained case where each group is required to have at least  $L$  points.
Single-Linkage. We briefly explain single-linkage. The algorithm starts with  $n$  groups, each of them consisting of a point in  $\mathcal{X}$ . Then, at each iteration, it merges the two groups with minimum spacing into a new group. Thus, by the end of the iteration  $n - k$  it obtains a clustering with  $k$  groups. In Kleinberg & Tardos (2006) it is proved that the single-linkage obtains a  $k$ -clustering with maximum minimum spacing.
Theorem 2.1 (Kleinberg & Tardos (2006), chap 4.7). The single-linkage algorithm obtains the  $k$ -clustering with maximum Min-Sp for instance  $(\mathcal{X}, \text{dist})$ .
single-linkage and minimum spanning trees are closely related since the former can be seen as the Kruskal's algorithm for building MST's with an early stopping rule. To analyze our algorithms we make use of well-known properties of MST's as the cut property[Kleinberg & Tardos (2006), Property 4.17] and the cycle property[Kleinberg & Tardos (2006), Property 4.20]. Their statements can be found in the appendix.
For ease of presentation, we assume that all values of dist are distinct. We note, however, that our results hold if this assumption is dropped.
3 Relating Min-Sp and MST-Sp criteria
We show that single-linkage finds the clustering with maximum MST-Sp and the maximization of MST-Sp implies the maximization of Min-Sp. These results are a consequence of Lemma 3.1 that generalizes the result of Theorem 2.1. The proof of this lemma can be found in the appendix.
Fix an instance  $I = (\mathcal{X}, \mathrm{dist}, k)$ . In what follows,  $\mathcal{C}_{SL}$  is a  $k$ -clustering obtained by single-linkage for instance  $I$  and  $T_{SL}$  is a MST for  $G_{\mathcal{C}_{SL}}$ . Moreover,  $w_i^{SL}$  is the weight of the  $i$ -th smallest weight of  $T_{SL}$ .
Lemma 3.1. Let  $\mathcal{C}$  be a  $k$ -clustering for  $I$  and let  $w_{i}$  be the weight of the  $i$ -th smallest weight in a MST  $T$  for the graph  $G_{\mathcal{C}}$ . Then,  $w_{i}^{SL} \geq w_{i}$ .
Theorem 3.2. The clustering  $\mathcal{C}_{SL}$  returned by single-linkage for instance  $(\mathcal{X}, \mathrm{dist}, k)$  maximizes the MST-Sp criterion.
Proof. Let  $\mathcal{C}$  be a  $k$ -clustering for  $(\mathcal{X}, \mathrm{dist}, k)$  and let  $w_{i}$  be the weight of the  $i$ -th cheapest edge of the MST for  $G_{\mathcal{C}}$ . Since  $w_{i}^{SL} \geq w_{i}$  for  $i = 1, \ldots, k-1$ , we have that
$$
\mathrm {M S T - S p} (\mathcal {C} _ {S L}) = \sum_ {i = 1} ^ {k - 1} w _ {i} ^ {S L} \geq \sum_ {i = 1} ^ {k - 1} w _ {i} = \mathrm {M S T - S p} (\mathcal {C})
$$
Theorem 3.3. Let  $\mathcal{C}^*$  be a clustering that maximizes the MST-Sp criterion for instance  $(\mathcal{X}, \mathrm{dist}, k)$ . Then, it also maximizes Min-Sp for this same instance.
Proof. Let us assume that  $\mathcal{C}^*$  maximizes the MST-Sp criterion but it does not maximize the Min-Sp criterion. Thus,  $w_{1}^{SL} > w_{1}^{*}$ , where  $w_{1}^{*}$  is the minimum spacing of  $\mathcal{C}^*$ . It follows from the previous lemma that
$$
\operatorname {M S T} - \operatorname {S p} \left(\mathcal {C} _ {S L}\right) = \sum_ {i = 1} ^ {k - 1} w _ {i} ^ {S L} > \sum_ {i = 1} ^ {k - 1} w _ {i} ^ {*} = \operatorname {M S T} - \operatorname {S p} \left(\mathcal {C} ^ {*}\right),
$$
which contradicts the assumption that  $\mathcal{C}^*$  maximizes the MST-Sp criterion.
The next example (in the spirit of Figure 1) shows that a partition that maximizes the Min-Sp criterion may have a poor result in terms of the MST-Sp criterion.
Example 3.4. Let  $D$  be a positive number much larger than  $k$ . Moreover, let  $[t]$  be the set of the  $t$  first positive integers and  $S = \{(D \cdot i, j) | i, j \in [k - 1]\} \cup (D, k)$  be a set of  $(k - 1)^2 + 1$  points in  $\mathbb{R}^2$ .
single-linkage builds a  $k$ -clustering with Min-Sp 1 and MST-Spl +  $(k - 2)D$  for  $S$ .
However, the  $k$ -clustering  $(C_1, \ldots, C_k)$ , where  $C_j = \{(D \cdot i, j) | i = 1, \ldots, k - 1\}$ , for  $j < k$  and  $C_k = \{(D, k)\}$  has Min-Sp 1 and MST-Sp =  $(k - 1)$ ,
4 Avoiding small groups
In this section, we optimize our criteria under the constraint that all groups must have at least  $L$  points, where  $L$  is a positive integer not larger than  $n / k$  that is provided by the user. Note that the problem is not feasible if  $L > n / k$ .
We say that an algorithm has  $(\beta, \gamma)$ -approximation for a criterion  $\kappa \in \{\mathrm{Min - Sp}, \mathrm{MST - Sp}\}$  if for all instances  $I$  it obtains a clustering  $\mathcal{C}_I$  such that  $\mathcal{C}_I$  is a  $(k, \lfloor \beta L \rfloor)$ -clustering and the value of  $\kappa$  for  $\mathcal{C}_I$  is at least  $\gamma \cdot OPT$ , where  $OPT$  is the maximum possible value of  $\kappa$  that can be achieved for a  $(k, L)$ -clustering.
We first show how to obtain a  $(1 - \epsilon, 1)$  for the Min-Sp criterion.
4.1 The Min-Sp criterion
We start with the polynomial-time approximation scheme for the Min-Sp criterion. Our method uses, as a subroutine, an algorithm for the max-min scheduling problem with identical machines Csrik et al. (1992); Woeginger (1997). Given  $m$  machines and a set of  $n$  jobs, with processing times  $p_1, \ldots, p_n$ , the problem consists of finding an assignment of jobs to the machines so that the load of the machine with minimum load is maximized. This problem admits a polynomial-time approximation scheme Woeginger (1997).
Let MaxMinSched  $(P, k, \epsilon)$  be a routine that implements this scheme. It receives as input a parameter  $\epsilon > 0$ , an integer  $k$  and a list of numbers  $P$  (corresponding to processing times). Then, it returns a partition of  $P$  into  $k$  lists (corresponding to machines) such that the sum of the numbers of the list
with minimum sum is at least  $(1 - \epsilon)OPT$ , where  $OPT$  is the minimum load of a machine in an optimal solution of for the max-min scheduling when the list of processing times is  $P$  and the number of machines is  $k$ .
Algorithm 1, as proved in the next theorem, obtains a  $(k,L(1 - \epsilon))$ -clustering whose Min-Sp is at least the Min-Sp of an optimal  $(k,L)$ -clustering. For that, it looks for the largest integer  $t$  for which the clustering  $\mathcal{A}_t$  obtained by executing  $t$  steps of single-linkage and then combining the resulting groups into  $k$  groups (via MaxMinSched) is a  $(k,L(1 - \epsilon))$ -clustering. We assume that MaxMinSched, in addition to returning the partition of the sizes, also returns the group associated to each size.
Algorithm 1 AlgoMinSp  $(\mathcal{X};$  dist;  $k;\epsilon >0;L)$
$t\gets n - k$    
while  $t\geq 0$  do Run  $t$  merging steps of the single-linkage for input  $\mathcal{X}$  Let  $C_1,\ldots ,C_{n - t}$  the groups obtained by the end of the  $t$  steps  $P\leftarrow (|C_1|,\dots ,|C_{n - t}|\right)$ $A_{t}\gets \mathrm{MaxMinSched}(P,k,\epsilon)$  if the smallest group in  $A_{t}$  has size greater than or equal to  $L(1 - \epsilon)$  then Return  $A_{t}$  else  $t\gets t - 1$
Theorem 4.1. Fix  $\epsilon >0$ . The clustering  $\mathcal{A}_t$  returned by the Algorithm 1 is a  $(k,(1 - \epsilon)L)$ -clustering that satisfies Min-Sp  $(\mathcal{A}_t) \geq \text{Min-Sp}(\mathcal{C}^*)$ , where  $\mathcal{C}^*$  is the  $(k,L)$ -clustering with maximum Min-Sp.
Proof. By design,  $\mathcal{A}_t$  has  $k$  groups with at least  $L(1 - \epsilon)$  points in each of them.
For the sake of contradiction, let us assume that  $\mathrm{Min - Sp}(\mathcal{A}_t) < \mathrm{Min - Sp}(\mathcal{C}^*)$ .
Let  $\mathcal{C} = (C_1, \ldots, C_{n - t})$  be the list of  $n - t$  groups obtained when  $t$  merging steps of single-linkage are performed. We assume w.l.o.g. that  $C_1$  and  $C_2$  are the two groups with a minimum spacing in this list, so that  $\operatorname{Min-Sp}(\mathcal{C}) = \operatorname{spacing}(C_1, C_2)$ . Since  $\mathcal{A}_t$  is a  $k$ -clustering that is obtained by merging groups in  $\mathcal{C}$  we have  $\operatorname{Min-Sp}(\mathcal{A}_t) \geq \operatorname{Min-Sp}(\mathcal{C}) = \operatorname{spacing}(C_1, C_2)$ .
For  $i = 1,\dots ,n - t$  we have that  $C_i\subseteq C$  for some  $C\in \mathcal{C}^*$ , otherwise we would have  $\operatorname {Min - Sp}(\mathcal{A}_t)\geq$ $\operatorname {Min - Sp}(\mathcal{C}^*)$ . In addition, we must have  $C_1\cup C_2\subseteq C$  for some  $C\in \mathcal{C}^*$ , otherwise, again, we would have  $\operatorname {Min - Sp}(\mathcal{A}_t)\geq \operatorname {Min - Sp}(\mathcal{C}^*)$ .
We can conclude that there is a feasible solution with minimum load not smaller than  $L$  for the max-min scheduling problem with processing times  $P' = (|C_1 \cup C_2|, |C_3|, \ldots, |C_{n - t}|)$  and  $K$  machines. Thus, by running  $t + 1$  steps of single-linkage followed by MaxMinSched( $P', k, \epsilon$ ), we would get a  $k$ -clustering whose smallest group has at least  $L(1 - \epsilon)$  points. This implies that the algorithm would have stopped after performing  $t + 1$  merging steps, which is a contradiction.
Algorithm 1, as presented, may run single-linkage  $n - k$  times, which may be quite expensive. Fortunately, it admits an implementation that runs single-linkage just once, and performs an inexpensive binary search to find a suitable  $t$ .
The next theorem shows that Algorithm 1 has essentially tight guarantees under the hypothesis that  $P \neq NP$ . The proof can be found in the appendix
Theorem 4.2. Unless  $P = NP$ , for any  $\alpha = poly(n)$ , the problem of finding the  $(k, L)$ -clustering that maximizes the Min-Sp criterion does not admit a  $(1, \frac{1}{\alpha})$ -approximation.
4.2 The MST-Sp criterion