Title: Optimization of Inter-group criteria for clustering with minimum size constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: Single Link

clustering

approximation algorithms

complexity

inter-group criterion
Abstract: Internal measures that are used to assess the quality of a clustering usually take into account intra-group and/or inter-group criteria.
There are many papers in the literature that propose algorithms with provable approximation guarantees for optimizing the former. However,  the optimization of inter-group criteria is much less understood.

Here, we contribute to the state-of-the-art of this literature by devising algorithms with provable guarantees for the maximization of two natural inter-group criteria, namely the minimum spacing and the minimum spanning tree spacing. The former is the minimum distance between points in different groups while the latter captures separability through the cost of the minimum spanning tree that connects all groups. We obtain results for both the unrestricted case, in which no constraint on the clusters is imposed, and for the constrained case where each group is required to have a minimum number of points. Our constraint is motivated by the fact that the popular Single-Linkage, which optimizes both criteria in the unrestricted case, produces clustering with many tiny groups.

To complement our work, we present an empirical study with 10 real datasets that provides evidence that our methods work very well in practical settings.

Optimization of Inter-group Criteria for Clustering with Minimum Size Constraints

Abstract

Internal measures that are used to assess the quality of a clustering usually take into account intra-group and/or inter-group criteria. There are many papers in the literature that propose algorithms with provable approximation guarantees for optimizing the former. However, the optimization of inter-group criteria is much less understood.

Here, we contribute to the state-of-the-art of this literature by devising algorithms with provable guarantees for the maximization of two natural inter-group criteria, namely the minimum spacing and the minimum spanning tree spacing. The former is the minimum distance between points in different groups while the latter captures separability through the cost of the minimum spanning tree that connects all groups. We obtain results for both the unrestricted case, in which no constraint on the clusters is imposed, and for the constrained case where each group is required to have a minimum number of points. Our constraint is motivated by the fact that the popular single-linkage , which optimizes both criteria in the unrestricted case, produces clusterings with many tiny groups.

To complement our work, we present an empirical study with 10 real datasets, providing evidence that our methods work very well in practical settings.

1 Introduction

Data clustering is a fundamental tool in machine learning that is commonly used in exploratory analysis and to reduce the computational resources required to handle large datasets. For comprehensive descriptions of different clustering methods and their applications, we refer to Jain et al. (1999); Hennig et al. (2015). In general, clustering is the problem of partitioning a set of items so that similar items are grouped together and dissimilar items are separated. Internal measures that are used to assess the quality of a clustering (e.g. Silhouette coefficient Rousseeuw (1987) and Davies-Bouldin index Davies & Bouldin (1979)) usually take into account intra-group and inter-group criteria. The former considers the cohesion of a group while the latter measures how separated the groups are.

There are many papers in the literature that propose algorithms with provable approximation guarantees for optimizing intra-group criteria. Algorithms for k -center, k -medians and k -means cost functions Gonzalez (1985); Charikar et al. (2002); Ahmadian et al. (2020) are some examples. However, optimizing inter-group criteria is much less understood. Here, we contribute to the stateof-the-art by considering the maximization of two natural inter-group criteria, namely the minimum spacing and the minimum spanning tree spacing.

Our Results. The spacing between two groups of points, formalized in Section 2, is the minimum distance between a point in the first and a point in the second group. We consider two criteria for

capturing the inter-group distance of a clustering, the minimum spacing ( Min-Sp ) and minimum spanning tree spacing ( MST-Sp ). The former is given by the spacing of the two groups with the smallest spacing in the clustering. The latter, as the name suggests, measures the separability of a clustering according to the cost of the minimum spanning tree (MST) that connects its groups; the largest the cost, the most separated the clustering is.

We first show that single-linkage , a procedure for building hierarchical clustering, produces a clustering that maximizes the MST spacing. This result contributes to a better understanding of this very popular algorithm since the only provable guarantee that we are aware of (in terms of optimizing some natural cost function) is that it maximizes the minimum spacing of a clustering Kleinberg & Tardos (2006)[Chap 4.7]. Our guarantee [Theorem 3.3] is stronger than the existing one in the sense that any clustering that maximizes the MST spacing also maximizes the minimum spacing. Figure 1 shows an example where the minimum spacing criterion does not characterize well the behavior of single-linkage .

Despite its nice properties, single-linkage tends to build clusterings with very small groups, which may be undesirable in practice. This can be seen in our experiments (see Figure 2) and is related to the well-documented fact that single-linkage suffers from the chaining effect (Jain et al., 1999)[Chap 3.2].

To mitigate this problem we consider the optimization of the aforementioned criteria under size constraints. Let L be a given positive integer that determines the minimum size that every group in a clustering should have. A ( k, L ) -clustering is a clustering with k groups in which the smallest group has at least L elements. For the Min-Sp criterion we devise an algorithm that builds clustering with at least L (1 -ϵ ) points per group while guaranteeing that its minimum spacing is not smaller than that of an optimal ( k, L ) -clustering. This result is the best possible in the sense that, unless P = NP , it is not possible to obtain an approximation with subpolynomial factor for the case in which the clustering is required to satisfy the hard constraint of L points per group. For the MST-Sp criterion we also devise an algorithm with provable guarantees. It produces a clustering whose MST-Sp is at most a log k factor from the optimal one and whose groups have each at least ρL (1 -ϵ ) / 2 points, where ρ is a number in the interval [1 , 2] that depends on the ratio n/kL . We also prove that the maximization of this criterion is APX-Hard for any fixed k .

We complement our investigation with an experimental study where we compare the behaviour of the clustering produced by our proposed algorithms with those produced by k -means and single-linkage for 10 real datasets. Our algorithms, as expected, present better results than k -means for the criteria under consideration while avoiding the issue of small groups observed for single-linkage .

Related Work. We are only aware of a few works that propose clustering algorithms with provable guarantees for optimizing separability (inter-group) criteria. We explain some of them.

The maximum k -cut problem is a widely studied problem in the combinatorial optimization community and its solution can be naturally viewed as a clustering that optimizes a separability criterion. Given an edge-weighted graph G and an integer k ≥ 2 , the maximum k -cut problem consists of finding a k -cut (partition of the vertices of the graph into k groups) with maximum weight, where the weight of a cut is given by the sum of the weights of the edges that have vertices in different groups of the cut. The weight of the k -cut can be seen as a separability criterion, where the distance between two groups is given by the sum of the pairwise distances of its points. It is well-known that a random

assignment of the vertices (points) yields an (1 -1 /k ) -approximation algorithm. This bound can be slightly improved using semi-definitive programming Frieze & Jerrum (1997).

The minimum spacing, one of the criteria we studied here, also admits an algorithm with provable guarantees. In fact, as we already mentioned, it can be maximized in polynomial time via the single-linkage algorithm Kleinberg & Tardos (2006)[Chap 4.7]. In what follows, we discuss works that study this algorithm as well as works that study problems and/or methods related to it.

single-linkage has been the subject of a number of researches (Zahn, 1971; Kleinberg, 2002; Carlsson & Mémoli, 2010; Hofmeyr, 2020). Kleinberg (2002) presents an axiomatic study of clustering, the main result of which is a proof that it is not possible to define clustering functions that simultaneously satisfy three desirable properties introduced in the paper. However, it was shown that by choosing a proper stopping rule for the single-linkage it satisfies any two of the three properties. Carlsson & Mémoli (2010) replaces the property of Kleinberg (2002) that the clustering function must output a partition with the property that it must generate a tree (dendogram). Then, it was established that single-linkage satisfies the new set of properties. In more recent work, Hofmeyr (2020) establishes a connection between minimum spacing and spectral clustering. While the aforementioned works prove that single-linkage has important properties, in practice, it is reported that sometimes it presents poor performance due to the so-called chaining effect (Jain et al. (1999)[Chap 3.2]).

single-linkage belongs to the family of algorithms that are used to build Hierarchical Agglomerative Clustering (HAC). Dasgupta (2016) frames the problem of building a hierarchical clustering as a combinatorial optimization problem, where the goal is to output a hierarchy/tree that minimizes a given cost function. The paper proposes a cost function that has desirable properties and an algorithm to optimize it. This result was improved and extended by a series of papers (Roy & Pokutta, 2016; Charikar & Chatziafratis, 2017; Cohen-Addad et al., 2019). The algorithms discussed in these papers do not seem to be employed in practice. Recently, there has been some effort to analyse HAC algorithms that are popular in practice, such as the Average Link (Moseley & Wang, 2017; Cohen-Addad et al., 2019). Our investigation of single-linkage can be naturally connected with this line of research.

Finally, in a recent work, Ahmadi et al. (2022) studied the notion of individual preference stability (IP-stability) in which a point is IP-stable if it is closer to its group (on average) than to any other group (on average). The clustering produced by single-linkage presents a kind of individual preference stability in the sense that each point is closer to some point in its group than to any point in some other group.

Potential Applications. We discuss two cases in which the maximization of inter-group criteria via our algorithms may be relevant: ensuring data diversity when training machine learning algorithms and population diversity in candidate solutions for genetic algorithms.

When training a machine learning model, ensuring data diversity may be crucial for achieving good results (Gong et al., 2019). In situations in which all available data cannot be used for training (e.g. training in the cloud with budget constraints), it is important to have a method for selecting a diverse subset of the data, and our algorithms can be used for this: to select n = kL elements, one can partition the full data set into k clusters, all of them containing at least L members, and then select L elements from each cluster. Using single-linkage to create kL groups and then picking up one element per group is also a possibility but it would increase the probability of an over-representation of outliers in the obtained subset, as these outliers would likely be clustered as singletons (see Figure 2 in Section 5).

Note that our algorithms can be used to create not only one but several diverse and disjoint subsets, which might be relevant to generate partitions for cross-validation or for evaluating a model's robustness. For that, each subset is obtained by picking exactly one point per group.

For genetic algorithms, maintaining diversity over the iterations is important to ensure a good exploration of the search space (Gupta & Ghafir, 2012). If all candidate solutions become too similar, the algorithm will become too dependent on mutations for improvement, as the offspring of two solutions will likely be similar to its two parents; and mutation alone may not be enough to fully explore the search space. We can apply our algorithms in a similar manner as mentioned above, by partitioning, at each iteration, the solutions into k clusters of minimum size L and selecting the best L solutions from each cluster, according to the objective function of the underlying optimization problem, to maintain

the solution population simultaneously optimized and diverse. Using single-linkage could lead to several poor solutions being selected to remain in the population, in case they are clustered as singletons.

We finally note that an algorithm that optimizes intra-group measures (e.g. k -means or k -medians) would not necessarily guarantee diversity for the aforementioned applications, as points from different groups can be close to each other.

2 Preliminaries

Let X be a set of n points and let dist : X × X ↦→ R + be a distance function that maps every pair of points in X into a non-negative real.

Given a k -clustering C = ( C 1 , . . . , C k ) , we define the spacing between two distinct groups C i and C j of C as

Then, the minimum spacing of C is given by

̸

A k -clustering C induces on ( X , dist ) an edge-weighted complete graph G C whose vertices are the groups of C and the weight w ( e ij ) of the edge e ij between groups C i and C j is given by w ( e ij ) = spacing ( C i , C j ) . For a set of edges S ∈ G C we define w ( S ) := ∑ e ∈ S w ( e ) .

The minimum spanning tree spacing of C ( MST-Sp ( C )) is defined as the sum of the weights of the edges of the minimum spanning tree (MST ( G C )) for G C . In formulae,

Here, we will be interested in the problems of finding partitions with maximum Min-Sp and maximum MST-Sp both in the unrestricted case in which no constraint on the groups is imposed and in the constrained case where each group is required to have at least L points.

Single-Linkage . We briefly explain single-linkage . The algorithm starts with n groups, each of them consisting of a point in X . Then, at each iteration, it merges the two groups with minimum spacing into a new group. Thus, by the end of the iteration n -k it obtains a clustering with k groups. In Kleinberg & Tardos (2006) it is proved that the single-linkage obtains a k -clustering with maximum minimum spacing.

Theorem 2.1 (Kleinberg & Tardos (2006), chap 4.7) . The single-linkage algorithm obtains the k -clustering with maximum Min-Sp for instance ( X , dist ) .

single-linkage and minimum spanning trees are closely related since the former can be seen as the Kruskal's algorithm for building MST's with an early stopping rule. To analyze our algorithms we make use of well-known properties of MST's as the cut property[Kleinberg & Tardos (2006), Property 4.17] and the cycle property[Kleinberg & Tardos (2006), Property 4.20]. Their statements can be found in the appendix.

For ease of presentation, we assume that all values of dist are distinct. We note, however, that our results hold if this assumption is dropped.

3 Relating Min-Sp and MST-Sp criteria

We show that single-linkage finds the clustering with maximum MST-Sp and the maximization of MST-Sp implies the maximization of Min-Sp . These results are a consequence of Lemma 3.1 that generalizes the result of Theorem 2.1. The proof of this lemma can be found in the appendix.

Fix an instance I = ( X , dist , k ) . In what follows, C SL is a k -clustering obtained by single-linkage for instance I and T SL is a MST for G C SL . Moreover, w SL i is the weight of the i -th smallest weight of T SL .

Lemma 3.1. Let C be a k -clustering for I and let w i be the weight of the i -th smallest weight in a MST T for the graph G C . Then, w SL i ≥ w i .

Theorem 3.2. The clustering C SL returned by single-linkage for instance ( X , dist , k ) maximizes the MST-Sp criterion.

Proof. Let C be a k -clustering for ( X , dist , k ) and let w i be the weight of the i -th cheapest edge of the MST for G C . Since w SL i ≥ w i for i = 1 , . . . , k -1 , we have that

Theorem 3.3. Let C ∗ be a clustering that maximizes the MST-Sp criterion for instance ( X , dist , k ) . Then, it also maximizes Min-Sp for this same instance.

Proof. Let us assume that C ∗ maximizes the MST-Sp criterion but it does not maximize the Min-Sp criterion. Thus, w SL 1 > w ∗ 1 , where w ∗ 1 is the minimum spacing of C ∗ . It follows from the previous lemma that

∗

which contradicts the assumption that C maximizes the MST-Sp criterion.

The next example (in the spirit of Figure 1) shows that a partition that maximizes the Min-Sp criterion may have a poor result in terms of the MST-Sp criterion.

Example 3.4. Let D be a positive number much larger than k . Moreover, let [ t ] be the set of the t first positive integers and S = { ( D · i, j ) | i, j ∈ [ k -1] } ∪ ( D,k ) be a set of ( k -1) 2 +1 points in R 2 .

single-linkage builds a k -clustering with Min-Sp 1 and MST-Sp 1 + ( k -2) D for S .

However, the k -clustering ( C 1 , . . . , C k ) , where C j = { ( D · i, j ) | i = 1 , . . . , k -1 } , for j < k and C k = { ( D,k ) } has Min-Sp 1 and MST-Sp = ( k -1) ,

4 Avoiding small groups

In this section, we optimize our criteria under the constraint that all groups must have at least L points, where L is a positive integer not larger than n/k that is provided by the user. Note that the problem is not feasible if L > n/k .

We say that an algorithm has ( β, γ ) -approximation for a criterion κ ∈ { Min-Sp , MST-Sp } if for all instances I it obtains a clustering C I such that C I is a ( k, ⌊ βL ⌋ ) -clustering and the value of κ for C I is at least γ · OPT , where OPT is the maximum possible value of κ that can be achieved for a ( k, L ) -clustering.

We first show how to obtain a (1 -ϵ, 1) for the Min-Sp criterion.

4.1 The Min-Sp criterion

We start with the polynomial-time approximation scheme for the Min-Sp criterion. Our method uses, as a subroutine, an algorithm for the max-min scheduling problem with identical machines Csirik et al. (1992); Woeginger (1997). Given m machines and a set of n jobs, with processing times p 1 , . . . , p n , the problem consists of finding an assignment of jobs to the machines so that the load of the machine with minimum load is maximized. This problem admits a polynomial-time approximation scheme Woeginger (1997).

Let MaxMinSched ( P, k, ϵ ) be a routine that implements this scheme. It receives as input a parameter ϵ > 0 , an integer k and a list of numbers P (corresponding to processing times). Then, it returns a partition of P into k lists (corresponding to machines) such that the sum of the numbers of the list

with minimum sum is at least (1 -ϵ ) OPT , where OPT is the minimum load of a machine in an optimal solution of for the max-min scheduling when the list of processing times is P and the number of machines is k .

Algorithm 1, as proved in the next theorem, obtains a ( k, L (1 -ϵ )) -clustering whose Min-Sp is at least the Min-Sp of an optimal ( k, L ) -clustering. For that, it looks for the largest integer t for which the clustering A t obtained by executing t steps of single-linkage and then combining the resulting groups into k groups (via MaxMinSched ) is a ( k, L (1 -ϵ )) -clustering. We assume that MaxMinSched , in addition of returning the partition of the sizes, also returns the group associated to each size.

Proof. By design, A t has k groups with at least L (1 -ϵ ) points in each of them.

For the sake of contradiction, let us assume that Min-Sp ( A t ) < Min-Sp ( C ∗ ) .

Let C = ( C 1 , . . . , C n -t ) be the list of n -t groups obtained when t merging steps of single-linkage are performed. We assume w.l.o.g. that C 1 and C 2 are the two groups with a minimum spacing in this list, so that Min-Sp ( C ) = spacing ( C 1 , C 2 ) . Since A t is a k -clustering that is obtained by merging groups in C we have Min-Sp ( A t ) ≥ Min-Sp ( C ) = spacing ( C 1 , C 2 ) .

For i = 1 , . . . , n -t we have that C i ⊆ C for some C ∈ C ∗ , otherwise we would have Min-Sp ( A t ) ≥ Min-Sp ( C ∗ ) . In addition, we must have C 1 ∪ C 2 ⊆ C for some C ∈ C ∗ , otherwise, again, we would have Min-Sp ( A t ) ≥ Min-Sp ( C ∗ ) .

We can conclude that there is a feasible solution with minimum load not smaller than L for the max-min scheduling problem with processing times P ′ = ( | C 1 ∪ C 2 | , | C 3 | , . . . , | C n -t | ) and K machines. Thus, by running t +1 steps of single-linkage followed by MaxMinSched ( P ′ , k, ϵ ) , we would get a k -clustering whose smallest group has at least L (1 -ϵ ) points. This implies that the algorithm would have stopped after performing t +1 merging steps, which is a contradiction.

Algorithm 1, as presented, may run single-linkage n -k times, which may be quite expensive. Fortunately, it admits an implementation that runs single-linkage just once, and performs an inexpensive binary search to find a suitable t .

̸

The next theorem shows that Algorithm 1 has essentially tight guarantees under the hypothesis that P = NP . The proof can be found in the appendix

Theorem 4.2. Unless P = NP , for any α = poly ( n ) , the problem of finding the ( k, L ) -clustering that maximizes the Min-Sp criterion does not admit a (1 , 1 α ) -approximation.

4.2 The MST-Sp criterion

Now, we turn to the MST-Sp criterion. Let ρ := min { n/k L , 2 } . Our main contribution is Algorithm 2, it obtains a ( ρ (1 -ϵ ) 2 , 1 H k -1 ) approximation for this criterion, where H k -1 = ∑ k -1 i =1 1 i is the ( k -1) -th Harmonic number. Note that H k -1 is Θ(log k ) .

In high level, for each ℓ = 2 , . . . , k , the algorithm calls AlgoMinSp (Algorithm 1) to build a clustering A ′ ℓ with ℓ groups and then it transforms A ′ ℓ (lines 5-13) into a clustering A ℓ with k groups. In the end, it returns the clustering, among the k -1 considered, with maximum MST-Sp .

Algorithm 2 Constrained-MaxMST ( X ; dist ; k ; L ; ϵ ) 1: for ℓ = 2 , . . . , k do 2: A ′ ℓ ← AlgoMinSp ( X , dist , ℓ , ϵ , L ) 3: NonVisited ←A ′ ℓ 4: A ℓ ←∅ 5: for each A ′ in A ′ ℓ , iterating from the largest group to the smallest do 6: NonVisited ← NonVisited -A ′ 7: SplitNumber ← ⌊ 2 | A ′ | ρ (1 -ϵ ) L ⌋ 8: if |A ℓ | + | NonVisited | + SplitNumber < k then 9: Split A ′ into SplitNumber as balanced as possible groups and add them to A ℓ 10: else 11: Split A ′ into k -|A ℓ |-| NonVisited | as balanced as possible groups; add them to A ℓ 12: Add all groups in NonVisited to A ℓ 13: Break 14: Return the clustering A ℓ , among the k -1 obtained, that has the maximum MST-Sp

We remark that we do not need to scan the groups in A ′ ℓ by non-increasing order of their sizes to establish our guarantees presented below. However, this rule tends to avoid groups with sizes smaller than L .

Proof. The groups that are added to A ℓ in line 12 have at least (1 -ϵ ) L points while the number of points of those that are added at either line 11 or 9 is at least

Moreover, if the For is not interrupted by the Break command, the total number of groups in A ℓ is

Since the For is interrupted as soon as k groups can be obtained then, A ℓ has k groups.

For the next results, we use C ∗ to denote the ( k, L ) -clustering with maximum MST-Sp and w ∗ i to denote the cost of the i -th cheapest edge in the MST for G C ∗ . Our first lemma can be seen as a generalization of Theorem 4.1. Its proof can be found in the appendix.

A simple consequence of the previous lemma is that the MST-Sp of clustering A ′ ℓ is at least ( ℓ -1) · w ∗ k -ℓ +1 . The next lemma shows that this bound also holds for the clustering A ℓ . The proof consists of showing that each edge of a MST for A ′ ℓ is also an edge of a MST for A ℓ .

Proof. Let T ℓ and T ′ ℓ be, respectively, the MST for G A ℓ and G A ′ ℓ . By the previous lemma, each of the ( ℓ -1) edges of T ′ ℓ has cost at least Min-Sp ( A ′ ℓ ) ≥ w ∗ k -ℓ +1 . Thus, to establish the result, it is enough to argue that each edge of T ′ ℓ also belongs to T ℓ .

We say that a group A ∈ A ℓ is is generated from a group A ′ ∈ A ′ ℓ if A = A ′ or A is one of the balanced groups that is generated when A ′ is split in the internal For of Algorithm 2. We say that a

vertex x in G A ℓ is generated from a vertex x ′ in G A ′ ℓ if the group corresponding to x is generated by the corresponding to x ′ .

Let e ′ = u ′ v ′ be an edge in T ′ ℓ and let S ′ be a cut in graph G A ′ ℓ whose vertices are those from the connected component of T ′ ℓ \ e ′ that includes u ′ . We define the cut S of G A ℓ as follows S = { x ∈ G A ℓ | x is generated from some x ′ ∈ S ′ } .

Let u and v be vertices generated from u ′ and v ′ , respectively, that satisfy w ( uv ) = w ( u ′ v ′ ) . It is enough to show that uv is the cheapest edge that crosses S since by the cut property [Kleinberg & Tardos (2006), Property 4.17.] this implies that uv ∈ T ℓ . We prove it by contradiction. Let us assume that there is another edge f = yz that crosses S and has weight smaller than w ( uv ) . Let y ′ and z ′ be vertices in G A ′ ℓ that generate y and z , respectively, and let f ′ = y ′ z ′ . Thus, w ( f ′ ) ≤ w ( f ) < w ( uv ) = w ( u ′ v ′ ) = w ( e ′ ) . However, this contradicts the cycle property of MST's [Kleinberg & Tardos (2006), Property 4.20] because it implies that the edge with the largest weight in the cycle of G A ′ ℓ comprised by edge f ′ and the path in T ′ ℓ the connects y ′ to z ′ belongs to the T ′ ℓ .

The next theorem is the main result of this section.

Theorem 4.6. Fix ϵ > 0 . Algorithm 2 is a ( (1 -ϵ ) ρ 2 , 1 H k -1 ) -approximation for the problem of finding the ( k, L ) -clustering that maximizes the MST-Sp criterion.

Proof. Let C be the clustering returned by Algorithm 2. Lemma 4.3 guarantees that C is a ( k, ⌊ (1 -ϵ ) ρL 2 ⌋ ) -clustering.

Thus, we just need to argue about MST-Sp ( C ) . We have that

and, due to Lemma 4.5, MST-Sp ( C ) ≥ max { ( ℓ -1) · w ∗ k -ℓ +1 | 2 ≤ ℓ ≤ k } .

Let ˜ ℓ be the value ℓ that maximizes ( ℓ -1) · w ∗ k -ℓ +1 . It follows that w ∗ k -i +1 ≤ (( ˜ ℓ -1) / ( i -1)) w ∗ k -˜ ℓ +1 . for i = 2 , . . . , k . Thus,

We end this section by showing that the optimization of MST-Sp is APX-HARD (for fixed k ) when a hard constraint on the number of points per group is imposed. The proof can be found in the appendix.

Theorem 4.7. Unless P = NP , for any α = poly ( n ) , there is no (1, k -2 k -1 + 1 α ( k -1) )-approximation for the problem of finding the ( k, L ) -clustering that maximizes the MST-Sp criterion.

5 Experiments

To evaluate the performance of Algorithms 1 and 2, we ran experiments with 10 different datasets, comparing the results with those of single-linkage and of the traditional k -means algorithm from Lloyd (1982) with a ++ initialization (Arthur & Vassilvitskii, 2007). For the implementation of routine MaxMinSched , employed by Algorithm 1, we used the Longest Processing Time rule. This rule has the advantage of being fast while guaranteeing a 3 / 4 approximation for the max-min scheduling problem (Csirik et al., 1992). The code for running the algorithms can be found at https://github.com/lmurtinho/SizeConstrainedSpacing .

Our first experiment investigates the size of the groups produced by single-linkage for the 10 datasets, whose dimensions can be found in the first two columns of Table 1. Figure 2 shows the proportion of singletons for each dataset with the growth of k . For all datasets but Vowel and Mice

the majority of groups are singletons, even for small values of k . This undesirable behavior motivates our constraint on the minimum size of a group.

1.0

proportion of singletons

0.8

newsgroups

pendigits

0.6

digits

0.4

anuran

avila

vowel

0.2

sensorless

letter

0.0

mice

0.0

0.2

0.4

0.6

0.8

1.0

k (normalized by n)

In our second experiment, we compare the values of Min-Sp and MST-Sp achieved by our algorithms with those of k -means. While k -means is not a particularly strong competitor in the sense that it was not designed to optimize our criteria, the motivation to include it is that k -means is a very popular choice among practitioners. Moreover, for datasets with well-separated groups, the minimization of the squared sum of errors (pursued by k -means) should also imply the maximization of inter-group criteria.

Table 1 presents the results of this experiment. The values chosen for k are the numbers of classes (dependent variable) in the datasets, while for the dist function we employed the Euclidean distance. The values associated with the criteria are averages of 10 executions, with each execution corresponding to a different seed provided to k -means. To set the value of L for the i -th execution of our algorithms, we take the size of the smallest group generated by k -means for this execution and multiply it by 4 / 3 . This way, we guarantee that the size of the smallest group produced by our methods, for each execution, is not smaller than that of k -means, which makes the comparison among the optimization criteria fairer.

With respect to the Min-Sp criterion, Algorithm 1 is at least as good as Algorithm 2 for every dataset (being superior on 6) and both Algorithm 1 and 2 outperform k -means on all datasets. On the other hand, with respect to the MST-Sp criterion, Algorithm 2 is at least as good as Algorithm 1 for every dataset (being better on 6) and, again, both algorithms outperform k -means for all datasets. In the appendix, we present additional information regarding this experiment. In particular, we show that

the MST-Sp achieved by Algorithm 2 is on average 81 % of the upper bound ∑ k ℓ =2 Min-Sp ( A ′ ℓ ) that follows from Lemma 4.4. This is much better than the 1 /H k -1 ratio given by the theoretical bound.

Finally, Table 2 shows the running time of our algorithms for the datasets that consumed more time. We observe that the overhead introduced by Algorithm 1 w.r.t. single-linkage is negligible while Algorithm 2, as expected, is more costly. In the appendix, we show that a strategy that only considers values of ℓ that can be written as ⌈ k/ 2 t ⌉ , for t = 0 , . . . , ⌊ log k ⌋ , in the first loop of Algorithm 2 provides a significant gain of running time while incurring a small loss in the MST-Sp . We note that the log k bound of Theorem 4.6 is still valid for this strategy.

6 Final Remarks

We have endeavored in this paper to expand the current knowledge on clustering methods for optimizing inter-cluster criteria. We have proved that the well-known single-linkage produces partitions that maximize not only the minimum spacing between any two clusters, but also the MST spacing, a stronger guarantee. We have also studied the task of maximizing these criteria under the constraint that each group of the clustering has at least L points. We provided complexity results and algorithms with provable approximation guarantees.

One potential limitation of our proposed algorithms is their usage on massive datasets (in particular Algorithm 2) since they execute single-linkage one or many times. If the dist function is explicitly given, then the Ω( n 2 ) time spent by single-linkage is unavoidable. However, if the distances can be calculated from the set of points X then faster algorithms might be obtained.

The main theoretical question that remained open in our work is whether there exist constant approximation algorithms for the maximization of MST-Sp . In addition to addressing this question, interesting directions for future research include handling different inter-group measures as well as other constraints on the structure of clustering.

Acknowledgments and Disclosure of Funding

The authors thank BigDataCorp ( https://bigdatacorp.com.br/ ) for providing computational power for the experiments of an initial version of the paper.

The work of the authors is partially supported by the Air Force Office of Scientific Research (award number FA9550-22-1-0475).

The work of the first author is partially supported by CNPq (grant 310741/2021-1).

References

Ahmadi, S., Awasthi, P., Khuller, S., Kleindessner, M., Morgenstern, J., Sukprasert, P., and Vakilian, A. Individual preference stability for clustering. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 197-246. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ahmadi22a.html .

Ahmadian, S., Norouzi-Fard, A., Svensson, O., and Ward, J. Better guarantees for k-means and euclidean k-median by primal-dual algorithms. SIAM J. Comput. , 49(4), 2020. doi: 10.1137/ 18M1171321. URL https://doi.org/10.1137/18M1171321 .

Arthur, D. and Vassilvitskii, S. K-means++: The advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '07, pp. 1027-1035, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 9780898716245. URL https://dl.acm.org/doi/10.5555/1283383.1283494 .

Carlsson, G. E. and Mémoli, F. Characterization, stability and convergence of hierarchical clustering methods. J. Mach. Learn. Res. , 11:1425-1470, 2010.

Charikar, M. and Chatziafratis, V. Approximate hierarchical clustering via sparsest cut and spreading metrics. In Klein, P. N. (ed.), Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2017, Barcelona, Spain, Hotel Porta Fira, January 16-19 , pp. 841-854. SIAM, 2017. doi: 10.1137/1.9781611974782.53. URL https://doi.org/10.1137/ 1.9781611974782.53 .

Charikar, M., Guha, S., Tardos, É., and Shmoys, D. B. A constant-factor approximation algorithm for the k-median problem. J. Comput. Syst. Sci. , 65(1):129-149, 2002. doi: 10.1006/jcss.2002.1882. URL https://doi.org/10.1006/jcss.2002.1882 .

Cohen-Addad, V., Kanade, V., Mallmann-Trenn, F., and Mathieu, C. Hierarchical clustering: Objective functions and algorithms. J. ACM , 66(4):26:1-26:42, 2019. doi: 10.1145/3321386. URL https://doi.org/10.1145/3321386 .

Csirik, J., Kellerer, H., and Woeginger, G. J. The exact lpt-bound for maximizing the minimum completion time. Oper. Res. Lett. , 11(5):281-287, 1992. doi: 10.1016/0167-6377(92)90004-M. URL https://doi.org/10.1016/0167-6377(92)90004-M .

Dasgupta, S. A cost function for similarity-based hierarchical clustering. In Wichs, D. and Mansour, Y. (eds.), Proceedings of the 48th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2016, Cambridge, MA, USA, June 18-21, 2016 , pp. 118-127. ACM, 2016. doi: 10.1145/ 2897518.2897527. URL https://doi.org/10.1145/2897518.2897527 .

Davies, D. L. and Bouldin, D. W. A Cluster Separation Measure. IEEE Transactions on Pattern Analysis and Machine Intelligence , PAMI-1(2):224-227, April 1979. ISSN 1939-3539. doi: 10.1109/TPAMI.1979.4766909. Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.

Frieze, A. M. and Jerrum, M. Improved approximation algorithms for MAX k-cut and MAX BISECTION. Algorithmica , 18(1):67-81, 1997. doi: 10.1007/BF02523688. URL https: //doi.org/10.1007/BF02523688 .

Garey, M. R. and Johnson, D. S. Computers and Intractability: A Guide to the Theory of NP-Completeness (Series of Books in the Mathematical Sciences) . W. H. Freeman, first edition edition, 1979. ISBN 0716710455. URL http://www.amazon. com/Computers-Intractability-NP-Completeness-Mathematical-Sciences/dp/ 0716710455 .

Gong, Z., Zhong, P., and Hu, W. Diversity in machine learning. Ieee Access , 7:64323-64350, 2019.

Gonzalez, T. F. Clustering to minimize the maximum intercluster distance. Theor. Comput. Sci. , 38:293-306, 1985. doi: 10.1016/0304-3975(85)90224-5. URL https://doi.org/10.1016/ 0304-3975(85)90224-5 .

Gupta, D. and Ghafir, S. An overview of methods maintaining diversity in genetic algorithms. International journal of emerging technology and advanced engineering , 2(5):56-60, 2012.

Hennig, C., Meila, M., Murtagh, F., and Rocci, R. Handbook of Cluster Analysis . Chapman and Hall/CRC, 2015.

Hofmeyr, D. P. Connecting spectral clustering to maximum margins and level sets. The Journal of Machine Learning Research , 21(1):630-664, 2020.

Jain, A. K., Murty, M. N., and Flynn, P. J. Data clustering: A review. ACM Comput. Surv. , 31(3): 264-323, September 1999. ISSN 0360-0300.

Kleinberg, J. M. An impossibility theorem for clustering. In Advances in neural information processing systems , 2002.

Kleinberg, J. M. and Tardos, É. Algorithm design . Addison-Wesley, 2006. ISBN 978-0-321-37291-8.

Lloyd, S. Least squares quantization in pcm. IEEE transactions on information theory , 28(2): 129-137, 1982.

Moseley, B. and Wang, J. R. Approximation bounds for hierarchical clustering: Average linkage, bisecting k-means, and local search. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pp. 3094-3103, 2017. URL https://proceedings. neurips.cc/paper/2017/hash/d8d31bd778da8bdd536187c36e48892b-Abstract.html .

Rousseeuw, P. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. , 20(1):53-65, 1987. ISSN 0377-0427. doi: http://dx.doi.org/10.1016/ 0377-0427(87)90125-7. URL http://portal.acm.org/citation.cfm?id=38772 .

Roy, A. and Pokutta, S. Hierarchical clustering via spreading metrics. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pp. 2316-2324, 2016. URL https://proceedings.neurips.cc/ paper/2016/hash/4d2e7bd33c475784381a64e43e50922f-Abstract.html .

Woeginger, G. J. A polynomial-time approximation scheme for maximizing the minimum machine completion time. Oper. Res. Lett. , 20(4):149-154, 1997. doi: 10.1016/S0167-6377(96)00055-7. URL https://doi.org/10.1016/S0167-6377(96)00055-7 .

Zahn, C. T. Graph-theoretical methods for detecting and describing gestalt clusters. IEEE Transactions on computers , 100(1):68-86, 1971.

A Properties of Minimum Spanning Trees

Theorem A.1 (Cut Property, Kleinberg & Tardos (2006), Property 4.17.) . Let G = ( V, E ) be a graph with distinct weights on its edges. Let S ⊂ V be a non-empty cut in G , If e is the edge with minimum cost among those that have one endpoint in S and the other one in V \ S , then e belongs to every MST for G

Theorem A.2 (Cycle Property, Kleinberg & Tardos (2006), Property 4.20.) . Let G = ( V, E ) be a graph with distinct weights on its edges and let C be a cycle in G . Then, the edge with the largest weight in C does not belong to any minimum spanning tree for G .

The following characterization of MST's will be useful. Its correctness follows directly from the cycle property.

Theorem A.3. Let G = ( V, E ) be a graph with weights on its edges. A spanning tree T for G is a MST for G if and only if for each edge e = uv in E , the weight w ( e ) of e satisfies w ( e ) ≥ w ( e ′ ) for every edge e ′ in the path that connects u to v in T .

B Proof of Lemma 3.1

We need the following proposition.

Proposition B.1. Let C ′ be a k -clustering for instance I and let T ′ be a MST for G C ′ . Moreover, let C ′ i and C ′ j be groups of C ′ such that spacing ( C ′ i , C ′ j ) = Min-Sp ( C ′ ) . Then, the tree T a that results from the contraction of the nodes C ′ i and C ′ j in T ′ is a MST for G C a , where C a is the ( k -1) -clustering obtained from C ′ by merging C ′ i and C ′ j .

Proof. We show that T a satisfies the conditions of Theorem A.3 when G = G C a . For that, we will use the fact that T ′ satisfies the conditions of Theorem A.3 when G = G C ′ .

Let x and y be nodes of T a . For the sake of contradiction, we assume that edge xy does not satisfy the conditions of Theorem A.3 when G = G C a and T = T a . Let w ∗ be the weight of edge xy and let e be an edge in the path that connects x to y in T a such that w ( e ) > w ∗ . We have two cases:

̸

Case 1) x = C ′ i ∪ C ′ j and y = C ′ i ∪ C ′ j . Then, e is also an edge in the path that connects x to y in T ′ . This implies that xy does not satisfy the required conditions when G = G C ′ and T = T ′ , which is a contradiction.

̸

̸

Case 2) x = C ′ i ∪ C ′ j or y = C ′ i ∪ C ′ j . Let us assume w.l.o.g. that x = C ′ i ∪ C ′ j . Let w ′ i and w ′ j be, respectively, the weights of the edges ( y, C ′ i ) and ( y, C ′ j ) in G C ′ . We have that w ∗ = min { w ′ i , w ′ j } .

Let us assume w.l.o.g. that w ∗ = w ′ i . Then, e is also an edge in the path that connects y to C ′ i in T ′ . Again, this implies that xy does not satisfy the required conditions when G = G C ′ and T = T ′ , which is a contradiction.

Proof of Lemma 3.1 .

It follows from the previous proposition hat the tree T i -1 that is obtained by contracting the i -1 cheapest edges of T is a MST for G C i -1 , where C i -1 is a clustering for instance I that contains ( k -( i -1)) groups. The cheapest edge of T i -1 is exactly the i -th cheapest edge of T . Thus, Min-Sp ( C i -1 ) = w i .

Similarly, w SL i is exactly the minimum spacing of a clustering, with ( k -( i -1) groups, that is obtained by single-linkage for instance I . Thus, it follows from Theorem 2.1 that w SL i ≥ w i .

C Proofs of Section 4

C.1 Proof of Lemma 4.4

Proof. Let T ∗ be the MST for G C ∗ . If we remove the ℓ -1 most expensive edges of T ∗ we obtain a forest F with ℓ connected components. The clustering C ∗ ℓ comprised by ℓ groups in

which the i th group corresponds to the i th connected component of F is a ( ℓ, L ) -clustering and Min-Sp ( C ∗ ℓ ) = w ∗ k -ℓ +1 .

Let OPT be the Min-Sp of the ( ℓ, L ) -clustering with maximum Min-Sp. Thus, by Theorem 4.1 Min-Sp ( A ′ ℓ ) ≥ OPT ≥ Min-Sp ( C ∗ ℓ ) = w ∗ k -ℓ +1

C.2 Proof of Theorem 4.2

Proof. We make a reduction from the ( T/ 4 , T/ 2) -restricted 3-PARTITION problem. Given a multiset S = { s 1 , . . . , s 3 m } of positive integers that satisfies ∑ i s i = mT , the 3-PARTITION problem consists of deciding whether or not there exists a partition of S into m triples such that the sum of the numbers in each one is equal to T . In the ( T/ 4 , T/ 2) -restricted 3-PARTITION problem, there is an additional requirement that each number of S should be in the interval ( T/ 4 , T/ 2) . This problem is strongly NP-COMPLETE Garey & Johnson (1979)

The instance I = ( X,k,L, dist ) for our clustering problem is built as follows: we set L = T , k = m ; for i = 1 , . . . , 3 m let X i be a set with s i points so that the distance between points in the same group X i is 1 while the distance between points in different groups is α + 1 . We set X = X 1 ∪ . . . ∪ X 3 m . Note that we are employing a pseudo-polynomial reduction but this is fine since the ( T/ 4 , T/ 2) -restricted 3-PARTITION problem is strongly NP-COMPLETE.

Let us assume that there is an (1 , 1 /α ) -approximation for our problem and let C be the clustering returned by this algorithm for instance I . We argue that the answer to the 3-PARTITION problem is 'YES' if and only if Min-Sp ( C ) = α +1 .

First, we show that if the answer is 'YES', there is a k -clustering C ∗ for I with Min-Sp ( C ∗ ) = α +1 . In fact, let S 1 , . . . , S m be a solution of the 3-PARTITION problem and let { s i 1 , s i 2 , s i 3 } the numbers in S i . Let C ∗ be a k -clustering where the i th group is comprised by all points in X i 1 ∪ X i 2 ∪ X i 3 . Clearly, each group has L points and the Min-Sp of this clustering is α +1 . Since our algorithm is a (1, 1 /α )-approximation it returns a clustering C with Min-Sp ( C ) ≥ ( α +1) /α . Since the Min-Sp of any clustering for instance I is either 1 or α +1 we have that Min-Sp ( C )= α +1

On the other hand, if the clustering has Min-Sp α +1 then all points in X i , for each i , must be in the same group. Moreover, due to the restriction that every |X i | = s i ∈ ( L/ 4 , L/ 2) , we should have exactly 3 X i 's in each of the m groups. Thus, the answer is 'YES'.

C.3 Proof of Theorem 4.7

The proof is very similar to that of Theorem 4.2.

Proof. We make a reduction from the ( T/ 4 , T/ 2) -restricted 3-PARTITION problem. Given a multiset S = { s 1 , . . . , s 3 m } of positive integers that satisfies ∑ i s i = mT , the 3-PARTITION problem consists of deciding whether or not there exists a partition of S into m triples such that the sum of the numbers in each one is equal to T . In the ( T/ 4 , T/ 2) -restricted 3-PARTITION problem, there is an additional requirement that each number of S should be in the interval ( T/ 4 , T/ 2) . This problem is strongly NP-COMPLETE Garey & Johnson (1979)

The instance I = ( X,k,L, dist ) for our clustering problem is built as follows: we set L = T , k = m ; for i = 1 , . . . , 3 m let X i be a set with s i points so that the distance between points in the same group X i is 1/2 while the distance between points in different groups is α . We set X = X 1 ∪ . . . ∪X 3 m .

Let us assume that there is an (1 , k -2 k -1 + 1 α ( k -1) ) -approximation for our problem and let C be the clustering returned by this algorithm for instance I . We argue that the answer to the 3-PARTITION problem is 'YES' if and only if MST-Sp ( C ) = ( k -1) α .

First, we show that if the answer is 'YES', there is a k -clustering C ∗ for I with MST-Sp ( C ∗ ) = ( k -1) α . In fact, let S 1 , . . . , S m be a solution of the 3-PARTITION problem and let { s i 1 , s i 2 , s i 3 } the numbers in S i . Let C ∗ be a k -clustering where the i th group is comprised by all points in X i 1 ∪ X i 2 ∪ X i 3 . Clearly, each group has L points and the MST-Sp of this clustering is ( k -1) α . Since our algorithm is a (1 , k -2 k -1 + 1 α ( k -1) ) -approximation it returns a clustering C with MST-Sp ( C )

> ( k -2) α +1 . Since the MST-Sp of any clustering for instance I is either ( k -1) α or at most ( k -2) α +1 / 2 we have that MST-Sp ( C )= ( k -1) α

On the other hand, if the clustering has Min-Sp ( k -1) α then all points in X i , for each i , must be in the same group. Moreover, due to the restriction that every |X i | = s i ∈ ( L/ 4 , L/ 2) , we should have exactly 3 X i 's in each of the m groups. Thus, the answer is 'YES'.

D Experiments: Additional Information

Experiments were run in an Ubuntu 20.04.5 LTS with 40 cores and 115 GB RAM. The repository of the project at https://anonymous.4open.science/r/SizeConstrainedSpacing-B260 contains the code and instructions needed to generate the experimental data analyzed in the paper.

D.1 MST-Sp : comparison between empirical results and upper bound of Algorithm 2

As mentioned in Section 5, Algorithm 2 can in practice obtain clusterings that are much closer to the optimal MST-Sp than the prediction guaranteed by Theorem 4.6. In Table 3, we present, for each dataset: the average MST-Sp obtained by Algorithm 2; the upper bound on the MST-Sp given by the sum of the Min-Sp for all partitions found by an execution of the algorithm; the approximation ratio of Algorithm 2, given by its MST-Sp divided by the upper bound; and the theoretical approximation ratio 1 /H k -1 from Theorem 4.6.

For all 10 datasets, Algorithm 2 performs significantly better than its theoretical approximation ratio. The smallest gap between theoretical and empirical result occurs for avila dataset, in which the algorithm is 21 percentage points closer to the optimal MST-Sp than Theorem 4.6 guarantees; On the other extreme, for dataset newsgroups , it actually achieves the best possible MST-Sp . These results increase our confidence that Algorithm 2 is a good option for finding separated groups.

D.2 Average size of smallest clusters

Table 4 presents the average size of the smallest group generated by Algorithms 1 and 2 and k -means. Values tend to be close across all algorithms, and for all iterations of the experiments the smallest group returned by Algorithms 1 and 2 is at least as large as the smallest group from the corresponding k -means clustering; recall that L is set as 4 s/ 3 , where s is the size of the smallest group produced by k -means. In particular, thanks to the rule of iterating from the largest cluster to the smallest when building our k -clustering from an ℓ -clustering, the theoretical possibility that the smallest group induced by Algorithm 2 is 1 / 2 of the desired size does not appear to happen in practice.

D.3 Fast version of Algorithm 2

As mentioned in Section 5, the log k bound of Theorem 4.6 is still valid for Algorithm 2 if, instead of investigating all values of ℓ from 2 to k , it considers only the values that can be written as ⌈ k/ 2 t ⌉ , for t = 0 , . . . , ⌊ log k ⌋ . In Table 5 we compare the results of this fast version of the algorithm with those of the full version.

Even considering the overhead of running the single-linkage algorithm, which cannot be avoided for both versions of Algorithm 2, we see a reduction of at least 30% in the algorithm's running time when using the fast version. The loss in terms of MST-Sp , on the other hand, is less than 10% in the worst scenario, and in 5 of the 10 datasets analyzed both versions return the same clustering.

D.4 Distribution of results for Min-Sp and MST-Sp

Figures 3 and 4 show the boxplots for the Min-Sp and the MST-Sp , respectively, per dataset and algorithm. Algorithm 2 presents some large variations, when compared to both k-means and Algorithm 1, in terms of Min-Sp (Figure 3) for some datasets; as it is designed to maximize the MST-Sp , this behavior in the other metric presented here should not be too concerning. Also in terms of Min-Sp , both algorithms presented in the paper clearly outperform k-means in almost all datasets, even considering the variation in results. The same can be said for the MST-Sp (Figure 4), in which, additionally, the range of results returned by Algorithm 2 is much more in line with those returned by the two other algorithms.

D.5 Trade-off between size of smallest cluster and inter-group separability criteria

In Figure 5 we present scatterplots for our 5 smallest datasets showing how the quality of the clusterings generated by Algorithms 1 and 2 (considering, respectively, the Min-Sp and the MST-Sp as criteria) increases as we allow for clusters of smaller sizes. For all datasets, as expected, allowing for smallest clusters leads to higher Min-Sp and MST-Sp . It is still noteworthy that the algorithms presented in this paper can be used not only to find a good partition with a hard limit on the size of the smallest cluster, but also to find the best balance between minimum size and a good separation of clusters.

algorithm

0.25

km

algo1

0.20

algo2

0.15

0.10

0.05

0.00

anuran

avila

letter

sensorless

dataset

1.0

algorithm

km

algo1

0.8

algo2

0.6

0.4

0.2

mice

newsgroups

vowel

dataset

algorithm

22.5

km

algo1

20.0

algo2

17.5

15.0

12.5

10.0

7.5

5.0

digits

pendigits

dataset

6

algorithm

km

5

algo1

algo2

3

2

1

anuran

avila

letter

sensorless

dataset

algorithm

17.5

km

algo1

15.0

algo2

12.5

10.0

7.5

5.0

2.5

mice

newsgroups

vowel

dataset

algorithm

220

km

algo1

200

algo2

180

160

140

120

digits

pendigits

dataset

(a) Algorithm 1:

smallest cluster vs.

Min-Sp

.

smallest cluster vs.

MST-Sp

.

dataset=anuran

dataset=anuran

0.8

8

0.7

Min-Sp

0.6

MST-Sp

6

0.5

0.4

4

0.3

0.2

2

0

100

200

400

0

100

200

300

400

dataset=vowel

dataset=vowel

0.70

F

8.0

7.5

0.65

7.0

0.60

MST-Sp

6.5

Min-s

6.0

0.55

5.5

0.50

5.0

0

10

20

30

40

50

60

0

10

20

30

40

50

60

dataset= collins

dataset = collins

18

0.550

17

0.525

16

0.500

0.475

MST-Sp

15

Min-

14

0.450

0.425

13

0.400

12

0.375

11

2

4

6

8

10

12

14

2

4

6

10

12

14

dataset = digits

dataset = digits

281

260

26

240

Min-Sp

24

MST-Sp

220

22

200

20

180

0

20

40

60

80

100

0

20

40

60

80

100

dataset=mice

dataset=mice

1.4

12

1.3

11

1.2

10

Min-Sp

MST-Sp

1.1

9

1.0

8

0.9

7

0.8

十

6

0

10

20

30

40

50

0

10

20

30

40

50

smallest_cluster

smallest_cluster

D.6 Effect of randomness on Algorithm 2's results

While Algorithm 1 is fully deterministic, in Algorithm 2 the split of clusters from an ℓ -clustering to turn it into a k -clustering is performed randomly. In practice, however, this does not affect the results of the algorithm.

For each dataset, we ran 10 seeded iterations of Algorithm 2 for each value of L used in the experiments. We then calculate the standard deviation of the MST-Sp for each value of L . As shown in Table 6, for 8 of the datasets analyzed the MST-Sp of the clustering returned by Algorithm 2 is always the same for a given value of L ; for letter and sensorless , there is some variation, but it is very small compared to the average MST-Sp returned by the algorithm.

D.7 Relative quadratic loss for Algorithms 1 and 2

In Table 7 we present the quadratic loss of both Algorithm 1 and Algorithm 2 as a multiple of the loss incurred by the k -means algorithm, which is specifically designed to minimize this loss. As expected, since both algorithms were devised for maximizing inter-group criteria, they perform poorly in light of this intra-group loss function - with the sole exception of the newsgroups dataset, for which both algorithms incur a loss only 5% above that of k -means. Across datasets, the performance of both algorithms is similar for this loss, with only small variations.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper consider metric clustering problems, where the goal is to maximize inter-clustering objectives. Such objectives measure how \"distinct\" different clusters are. The first objective they consider is the minimum spacing one (MS). According to this, the goal is to maximize the minimum distance between any resulting clusters, where the minimum distance between clusters is naturally defined as the minimum distance between any of their points. The second objective they consider is maximizing the minimum spanning tree (MST) distance. Here, the MST distance is defined on the complete graph representing the resulting clusters. I would say that the latter objective captures a more general/global notion of connectivity/similarity compared to the earlier one.\n\nAt first, the authors state that the single-linkage algorithm by Kleinberg and Tardos maximizes the MS objective, and this is something already known in the literature. However, they also show that this algorithm satisfies MST as well. More interestingly, they show that any algorithm satisfying MST will also satisfy MS.\n\nMoving forward, the authors tackle a constrained version of the problem, where besides minimizing the appropriate objective, they want to create clusters of size at least L, for some given L. For the constrained versions of the their problems, they give their most algorithmically involved results, which consist of bicriteria approximation algorithms followed by corresponding lower bounds.\n\nSTRENGTHS:\n1) The paper is technically solid and well-written.\n\n2) The inter-clustering objective is cute and quite novel. From the theoretical perspective it opens up a new direction for clustering research.\n\n3) The results are pretty tight. The characterization of Theorem 3.3 is solid and impressive and the lower bound algorithms are accompanied by the proper inapproximability results.\n\nWEAKNESSES:\n1) From a practical motivational perspective, I am not convinced about the use-cases of such a model. The authors should have elaborated more there, giving more fitting applications. This is honestly my only concern for this paper. I can easily see this paper getting into a pure algorithms conference like APPROX-RANDON or ICALP, but I feel like for NeurIPS standards there should have been a more detailed discussion on how this inter-group model (with the objective functions considered) could fly in real-life scenarios (perhaps by giving concrete specific use-cases). Also, from a practical perspective I can't see the necessity of the lower bound constraint. \n\nIf it wasn't for the lack of motivation my score would most likely be higher.\n\nQUESTIONS:\n1) I think that the following works are a bit related to yours. Under the broader umbrella of inter-group clustering. The goal in those papers is to make sure that on average, each point is closer to points in its own cluster compared to other clusters. To me this feels like an inter-group objective.\n\nhttps://arxiv.org/abs/2006.04960\nhttps://arxiv.org/abs/2207.03600\n\n2) In Figure 1 where you say that the left clustering is not natural, it is not really clear what you mean. I suppose that you want to show that solving MS can negatively impact MST.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem where we want to cluster $n$ points into $k$ clusters with the clustering objective trying to maximize the minimum distances between clusters (either the actual minimum distance or the weight of an MST defined between clusters). The authors show that greedy agglomerative clustering (stopping when $k$ clusters are formed) with single-linkage optimizes both objectives. The authors also consider the scenario where the user specifies a minimum cluster size as an additional constraint. Approximation algorithms for this setting are presented and hardness of approximation results are shown. Empirical results are shown on publicly available datasets.\n\nSTRENGTHS:\n* The paper is well-written\n* The problem is interesting from a theoretical perspective\n* The theoretical results are sound, albeit not surprising\n\nWEAKNESSES:\n* The motivation for the problem setting is very weak. It is unclear where this would be used in practice.\n* The results presented are probably not of high interest to the NeurIPS community. The theoretical results are solid, but not surprising, and the applications of this method are not presented. Could this be more appropriate for a theory venue?\n* The experimental results do not convey much about the performance of the algorithms except for their relative performance, which is not very dramatic. Are there other experiments that could be run to show why someone would want to use Algorithm 1 or 2 over another approach?\n\nQUESTIONS:\n* What is another example application other than the diversity example given on lines 34-38? \n* What benefits do Min-Sp and MST-Sp have over other clustering objectives? Why would a user choose this objective over another objective?\n* How might one select the minimum cluster size? Could this be an issue just as difficult as selecting the number of clusters $k$?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the following problem: given a metric space on a set of $n$ points and a parameter $k$, partition the points into $k$ clusters $C_1, \\ldots, C_k$. The paper considers two objective functions: consider a metric  with vertices $X$ as the set of clusters and distance between two clusters being the minimum distance between two points belonging to these clusters respectively. The first objective is to maximize the minimum distance between any two points in $X$, and the second objective is to maximize the cost of the min. spanning tree on $X$. It is easy to show that the greedy algorithm for MST can be modified to solve these two problems in polynomial time (essentially run the Kruskal's algorithm till only $k$ components remain). \n\nHowever, the authors observe that this can lead to skewed clusters with some clusters being too small. So they propose the following problem: consider the same settings as above, but now each cluster must have at least $L$ points, where $L$ is another parameter provided in the input. They give the following polynomial time guarantees for the two problems: (i) for the max min. cluster separation objective, they give an algorithm of optimal value but each cluster has size at least $L(1-\\epsilon)$, where $\\epsilon > 0$ is any constant, and (ii) for the MST objective, they give an algorithm of objective value at least $1/\\log k$ times the optimal value, and each cluster size is at least $L(1-\\epsilon)$. They show that  modifications of the greedy heuristic along with known results for a schedling problem work for the first problem, whereas the second one requires more details.\n\nSTRENGTHS:\n1. Introduce an interesting problem. \n2. Give near-tight bounds for the max min cluster separation problem, and leave an interesting open problem for the max min spanning tree problem.\n\nWEAKNESSES:\nIt seems to me that the two problems can be solved in a simpler manner without appealing to the greedy heuristic. Assume that the metric on the $n$ points is given by a weighted graph $G$. \n\n1. First consider the max min cluster separation problem: suppose we know the optimal value $T$ (which we can guess by binary search). Now remove all edges of length less than $T$ from the $G$:  these edges cannot appear between two distinct clusters. Now consider the components formed -- they should be more than $k$. Now merge these components using the scheduling problem used in the paper. \n\n2. For the max. mst problem: again suppose we know the optimal value, say $T$. Now consider the optimal tree on the $k$ vertices $X$ -- call this tree $F$. Ignore the edges of cost less than $T/2k$ from $F$ -- the total cost of such edges is at most $T/2$. Now consider the edges in $F$ whose cost lies in the range $[T/2k, T]$. Divide this range into $\\log k$ buckets of exponentially increasing range. So there is an index $i$ such that the edges in $F$ (call these edges $E_i$) with cost in the range $[\\frac{T}{2^{i+1} k}, \\frac{T}{2^i k}]$ have total cost at least $\\frac{T}{2 \\log k}$. Now remove all edges of length at least $\\frac{T}{2^{i+1} k}$ from $G$. It is not difficult to show that if $E_i$ has $a$ edges, then removing such edges from $G$ will create at least $a$ components. Now we form clusters so that no cluster contains points from two distinct components.\n\nQUESTIONS:\n1. It is not clear if the problems are non-trivial or follow from known ideas/techniques. \n\n2. Please explain if instead of placing lower bound of $L$, placing an upper bound of $L$ will change the problem significantly.\n\nLIMITATIONS:\nNo negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a body of theoretical results addressing the problems of minimum-spacing and minimum spanning tree clustering; both based on the concept of finding clustering solutions which maximise the inter-cluster distance(s). The former maximises the distance between the pair of closest clusters while the latter maximises the total distances between clusters along a tree connecting them (where set distances are defined as the minimum pairwise distances between elements in different sets).\n\nThe single-linkage agglomerative method is known to maximise inter-cluster distances but is also known to have poor properties empirically (notably the chaining effect and isolation of \"outliers\" and \"noise\"  over the construction of entire clusters). Based on this the authors propose imposing size-constraints which bound the size of the smallest cluster.\n\nThe authors show that hard constraints of this type for the two objectives studied are APX-hard, and so propose relaxations which allow a (1-\\epsilon) factor on the minimum cluster size but admit polynomial time algorithms for the minimum spacing, and in the case of the minimum spanning tree relaxations which also involve the average cluster size relative to the \"hard constraint size\" (being a more substantial relaxation the more balanced the cluster sizes need to be) as well as an approximation factor on the objective function given by the inverse of the k-1 th harmonic number (k being the number of clusters).\n\nThe paper concludes with some experiments documenting the performance of their proposed algorithms.\n\nSTRENGTHS:\nThe work is extremely thorough and addresses an important problem, which is that maximum inter-cluster distances are very intuitive as objectives for clustering but have poor robustness. The size constraint proposed is natural and intuitive, and the authors have clearly done more than their due diligence in studying the extending the problem.\n\nThe paper is also, subject to some technical knowledge, clear and easy to follow.\n\nWEAKNESSES:\nNothing substantial. Arguably some missed pieces of literature:\n- Early work on MST clustering: Zahn, Charles T. \"Graph-theoretical methods for detecting and describing gestalt clusters.\" IEEE Transactions on computers 100.1 (1971): 68-86.\n- Connection between Min-Sp and spectral clustering: Hofmeyr, David P. \"Connecting spectral clustering to maximum margins and level sets.\" The Journal of Machine Learning Research 21.1 (2020): 630-664.\n- Maximum Margin Clustering: Xu, Linli, James Neufeld, Bryce Larson, and Dale Schuurmans. \"Maximum margin clustering.\" Advances in neural information processing systems 17 (2004).\n\nTypos and similar:\n- Line 111: plural of vertex is \"vertices\"\n- Lines 126-7: \"the the cut property... and the cut property\"???\n- Include the explicit version of PTAS\n\nQUESTIONS:\nNone which I can think of. Good work\n\nLIMITATIONS:\nNone noted", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}