Learning to Link

1 INTRODUCTION . Overview . Clustering is an important component of modern data analysis . For example , we might cluster emails as a pre-processing step for spam detection , or we might cluster individuals in a social network in order to suggest new connections . There are a myriad of different clustering algorithms , and it is not always clear what algorithm will give the best performance on a specific clustering task . Similarly , we often have multiple different ways to measure distances between data points , and it is not obvious which distance metric will lead to the best performance . In this work , we study data-driven algorithm selection and metric learning for clustering problems , where the goal is to use data to simultaneously learn the best algorithm and metric for a specific application such as clustering emails or users of a social network . An application is modeled as a distribution over clustering tasks , we observe an i.i.d . sample of clustering instances drawn from that distribution , and our goal is to choose an approximately optimal algorithm from a parameterized family of algorithms ( according to some well-defined loss function ) . This corresponds to settings where we repeatedly solve clustering instances ( e.g. , clustering the emails that arrive each day ) and we want to use historic instances to learn the best clustering algorithm and metric . The family of clustering algorithms we learn over consists of parameterized linkage based procedures and includes single and complete linkage , which are widely used in practice and optimal in many cases ( Awasthi et al. , 2014 ; Saeed et al. , 2003 ; White et al. , 2010 ; Awasthi et al. , 2012 ; Balcan and Liang , 2016 ; Grosswendt and Roeglin , 2015 ) . The family of distance metrics we learn over consists of convex combinations of base distance functions . We design efficient learning algorithms that receive samples from an application-specific distribution over clustering instances and simultaneously learn both a near-optimal distance metric and clustering algorithm from these classes . We contribute to a recent line of work that provides learning-theoretical guarantees for data-driven algorithm configuration ( Gupta and Roughgarden , 2017 ; Balcan et al. , 2017 ; 2018a ; b ; 2019 ) . These papers analyze the intrinsic complexity of parameterized algorithm families in order to provide sample complexity guarantees , that is , bounds on the number of sample instances needed in order to find an approximately optimal algorithm for a given application domain . Our results build on the work of Balcan et al . ( 2017 ) , who studied the problem of learning the best clustering algorithm from a class of linkage based procedures , but did not study learning the best metric . In addition to our sample complexity guarantees , we develop a number of algorithmic tools that enable learning application specific clustering algorithms and metrics for realistically large clustering instances . We use our efficient implementations to conduct comprehensive experiments on clustering domains derived from both real-world and synthetic datasets . These experiments demonstrate that learning application-specific algorithms and metrics can lead to significant performance improvements over standard algorithms and metrics . Our Results . We study linkage-based clustering algorithms that take as input a clustering instance S and output a hierarchical clustering of S represented as a binary cluster tree . Each node in the tree represents a cluster in the data at one level of granularity , with the leaves corresponding to individual data points and the root node corresponding to the entire dataset . Each internal node represents a cluster obtained by merging its two children . Linkage-based clustering algorithms build a cluster tree from the leaves up , starting with each point belonging to its own cluster and repeatedly merging the “ closest ” pair of clusters until only one remains . The parameters of our algorithm family control both the metric used to measure pointwise distances , as well as how the linkage algorithm measures distances between clusters ( in terms of the distances between their points ) . This work has two major contributions . Our first contribution is to provide sample complexity guarantees for learning effective application-specific distance metrics for use with linkage-based clustering algorithms . The key challenge is that , if we fix a clustering algorithm from the family we study and a single clustering instance S , the algorithm output is a piecewise constant function of our metric family ’ s parameters . This implies that , unlike many standard learning problems , the loss we want to minimize is very sensitive to the metric parameters and small perturbations to the optimal parameters can lead to high loss . Our main technical insight is that for any clustering instance S , we can partition the parameter space of our metric family into a relatively small number of regions such that the ordering over pairs of points in S given by the metric is constant on each region . The clustering output by all algorithms in the family we study only depends on the ordering over pairs of points induced by the metric , and therefore their output is also a piecewise constant function of the metric parameters with not too many pieces . We leverage this structure to bound the intrinsic complexity of the learning problem , leading to uniform convergence guarantees . By combining our results with those of Balcan et al . ( 2017 ) , we show how to simultaneously learn both an application-specific metric and linkage algorithm . Our second main contribution is a comprehensive empirical evaluation of our proposed methods , enabled by new algorithmic insights for efficiently learning application-specific algorithms and metrics from sample clustering instances . For any fixed clustering instance , we show that we can use an execution tree data structure to efficiently construct a coarse partition of the joint parameter space so that on each region the output clustering is constant . Roughly speaking , the execution tree compactly describes all possible sequences of merges the linkage algorithm might make together with the parameter settings for the algorithm and metric that lead to that merge sequence . The learning procedure proposed by Balcan et al . ( 2017 ) takes a more combinatorial approach , resulting in partitions of the parameter space that have many unnecessary regions and increased overall running time . Balcan et al . ( 2018a ) and Balcan et al . ( 2018b ) also use an execution tree approach for different algorithm families , however their specific approaches to enumerating the tree are not efficient enough to be used in our setting . We show that using a depth-first traversal of the execution tree leads to significantly reduced memory requirements , since in our setting the execution tree is shallow but very wide . Using our efficient implementations , we evaluate our learning algorithms on several real world and synthetic clustering applications . We learn the best algorithm and metric for clustering applications derived from MNIST , CIFAR-10 , Omniglot , Places2 , and a synthetic rings and disks distribution . Across these different tasks the optimal clustering algorithm and metric vary greatly . Moreover , in most cases we achieve significant improvements in clustering quality over standard clustering algorithms and metrics . Related work . Gupta and Roughgarden ( 2017 ) introduced the theoretical framework for analyzing algorithm configuration problems that we study in this work . They provide sample complexity guarantees for greedy algorithms for canonical subset selection problems including the knapsack problem , maximum weight independent set , and machine scheduling . Some recent works provide sample complexity guarantees for learning application-specific clustering algorithms . Balcan et al . ( 2017 ) consider several parameterized families of linkage based clustering algorithms , one of which is a special case of the family studied in this paper . Their sample complexity results are also based on showing that for a single clustering instance , we can find a partitioning of the algorithm parameter space into regions where the output clustering is constant . The families of linkage procedures they study have a single parameter , while our linkage algorithm and metric families have multiple . Moreover , they suppose we are given a fixed metric for each clustering instance and do not study the problem of learning an application-specific metric . Balcan et al . ( 2018b ) study the related problem of learning the best initialization procedure and local search method to use in a clustering algorithm inspired by Lloyd ’ s method for k-means clustering . Their sample complexity results are again based on demonstrating that for any clustering instance , there exists a partitioning of the parameter space on which the algorithm ’ s output is constant . The parameter space partitions in both of these related works are defined by linear separators . Due to the interactions between the distance metric and the linkage algorithm , our partitions are defined by quadratic functions . The procedures proposed by prior work for finding an empirically optimal algorithm for a collection of problem instances roughly fall into two categories : combinatorial approaches and approaches based on an execution-tree data structure . Gupta and Roughgarden ( 2017 ) and Balcan et al . ( 2017 ) are two examples of the combinatorial approach . They show that the boundaries in the constant-output partition of the algorithm parameter space always occur at the solutions to finitely many equations that depend on the problem instance . To find an empirically optimal algorithm , they find all solutions to these problem-dependent equations to explicitly construct a partition of the parameter space . Unfortunately , only a small subset of the solutions are actual boundaries in the partition . Consequently , their partitions contain many extra regions and suffer from long running times . The execution-tree based approaches find the coarsest possible partitioning of the parameter space such that the algorithm output is constant . Balcan et al . ( 2018b ) and Balcan et al . ( 2018a ) both use execution trees to find empirically optimal algorithm parameters for different algorithm families . However , the specific algorithms used to construct and enumerate the execution tree are different from those explored in this paper and are not suitable in our setting . 2 LEARNING CLUSTERING ALGORITHMS . The problem we study is as follows . Let X be a data domain . A clustering instance consists of a point set S = { x1 , . . . , xn } ⊂ X and an ( unknown ) target clustering Y = ( C1 , . . . , Ck ) , where the sets C1 , . . . , Ck partition S into k clusters . Linkage-based clustering algorithms output a hierarchical clustering of the input data , represented by a cluster tree . We measure the agreement of a cluster tree T with the target clustering Y = ( C1 , . . . , Ck ) in terms of the Hamming distance between Y and the closest pruning of T into k clusters ( i.e. , k disjoint subtrees that contain all the leaves of T ) . More formally , we define the loss ` ( T , Y ) = minP1 , ... , Pk minσ∈Sn 1|S| ∑k i=1 |Ci \ Pσi | , where A \ B denotes set difference , the first minimum is over all prunings P1 , . . . , Pk of the cluster tree T , and the second minimum is over all permutations of the k cluster indices . This formulation allows us to handle the case where each clustering task has a different number of clusters , and where the desired number might not be known in advance . Our analysis applies to any loss function ` measuring the quality of the output cluster tree T , but we focus on the Hamming distance for simplicity . Given a distribution D over clustering instances ( i.e. , point sets together with target clusterings ) , our goal is to find the algorithm A from a family A with the lowest expected loss for an instance sampled from D. As training data , we assume that we are given an i.i.d . sample of clustering instances annotated with their target clusterings drawn from the application distribution D. We study linkage-based clustering algorithms . These algorithms construct a hierarchical clustering of a point set by starting with each point belonging to a cluster of its own and then they repeatedly merge the closest pair of clusters until only one remains . There are two distinct notions of distance at play in linkage-based algorithms : first , the notion of distance between pairs of points ( e.g. , Euclidean distance between feature vectors , edit distance between strings , or the Jaccard distance between sets ) . Second , these algorithms must define a distance function between clusters , which we refer to as a merge function to avoid confusion . A merge function D defines the distance between a pair of clusters A , B ⊂ X in terms of the pairwise distances given by a metric d between their points . For example , single linkage uses the merge function Dmin ( A , B ; d ) = mina∈A , b∈B d ( a , b ) and complete linkage uses the merge function Dmax ( A , B ; d ) = maxa∈A , b∈B d ( a , b ) . Our parameterized family of linkage-based clustering algorithms allows us to vary both the metric used to measure distances between points , as well as the merge function used to measure distances between clusters . To vary the metric , we suppose we have access to L metrics d1 , . . . , dL defined on our data universe X , and our goal is to find the best convex combination of those metrics . That is , for any parameter vector β ∈ ∆L = { β ∈ [ 0 , 1 ] L | ∑ i βi = 1 } , we define a metric dβ ( x , x′ ) = ∑ i βi · di ( x , x′ ) . This definition is suitable across a wide range of applications , since it allows us to learn the best combination of a given set of metrics for the application at hand . Similarly , for varying the merge function , we suppose we have L′ merge functions D1 , . . . , DL′ . For any parameter α ∈ ∆L′ , define the merge function Dα ( A , B ; d ) = ∑ i αiDi ( A , B ; d ) . For each pair of parameters β ∈ ∆L and α ∈ ∆L′ , we obtain a different clustering algorithm ( i.e. , one that repeatedly merges the pair of clusters minimizing Dα ( · , · ; dβ ) ) . Pseudocode for this method is given in Algorithm 1 . In the pseudocode , clusters are represented by binary trees with leaves corresponding to the points belonging to that cluster . For any clustering instance S ⊂ X , we letAα , β ( S ) denote the cluster tree output by Algorithm 1 when run with parameter vectors α and β. Algorithm 1 Linkage Clustering Input : Metrics d1 , . . . , dL , merge functions D1 , . . . , DL′ , points x1 , . . . , xn ∈ X , parameters α and β . 1 . Let N = { Leaf ( x1 ) , . . . , Leaf ( xn ) } be the initial set of nodes ( one leaf per point ) . 2 . While |N | > 1 ( a ) Let A , B ∈ N be the clusters in N minimizing Dα ( A , B ; dβ ) . ( b ) Remove clusters A and B from N and add Node ( A , B ) to N . 3 . Return the cluster tree ( the only element of N ) . First , we provide sample complexity results that hold for any collection of metrics d1 , . . . , dL and any collection of merge functions D1 , . . . , DL′ that belong to the following family : Definition 1 . A merge function D is 2-point-based if for any pair of clusters A , B ⊂ X and any metric d , there exists a pair of points ( a , b ) ∈ A×B such that D ( A , B ; d ) = d ( a , b ) . Moreover , the pair of points defining the merge distance must depend only on the ordering of pairwise distances . More formally , if d and d′ are two metrics s.t . for all a , a′ ∈ A and b , b′ ∈ B , we have d ( a , b ) ≤ d ( a′ , b′ ) if and only if d′ ( a , b ) ≤ d′ ( a′ , b′ ) , then D ( A , B ; d ) = d ( a , b ) implies that D ( A , B ; d′ ) = d′ ( a , b ) . For example , both single and complete linkage are 2-point-based merge functions , since they output the distance between the closest or farthest pair of points , respectively . Theorem 1 . Fix any metrics d1 , . . . , dL , 2-point-based merge functionsD1 , . . . , DL′ , and distributionD over clustering instances with at most n points . For any parameters > 0 and δ > 0 , let ( S1 , Y1 ) , . . . , ( SN , YN ) be an i.i.d . sample of N = O ( 1 2 ( ( L′ + L ) 2L′ log ( ( L′+L ) 2L′n 2 ) + log ( 1 δ ) ) ) = Õ ( ( L′+L ) 2L′ 2 ) clustering instances with target clusterings drawn from D. Then with probability at least 1− δ over the draw of the sample , we have sup ( α , β ) ∈∆L′×∆L ∣∣∣∣∣ 1N N∑ i=1 ` ( Aα , β ( Si ) , Yi ) − E ( S , Y ) ∼D [ ` ( Aα , β ( S ) , Y ) ] ∣∣∣∣∣ ≤ . The key step in the proof of Theorem 1 is to show that for any clustering instance S with target clustering Y , the function ( α , β ) 7→ ` ( Aα , β ( S ) , Y ) is piecewise constant with not too many pieces and where each piece is simple . Intuitively , this guarantees that for any collection of clustering instances , we can not see too much variation in the loss of the algorithm on those instances as we vary over the parameter space . In Appendix A we use this fact to bound the empirical Rademacher complexity of the learning problem and obtain the uniform convergence guarantee in Theorem 1 . In the remainder of this section , we prove the key structural property . We let ζ = ( α , β ) ∈ ∆L′ ×∆L denote a pair of parameter vectors for Algorithm 1 , viewed as a vector in RL ′+L . Our parameter space partition will be induced by the sign-pattern of M quadratic functions . Definition 2 ( Sign-pattern Partition ) . The sign-pattern partition of Rp induced M functions f1 , . . . , fM : Rp → R is defined as follows : Let F : Rp → { ±1 } M be the function F ( ζ ) = ( sign ( f1 ( ζ ) ) , . . . , sign ( fM ( ζ ) ) ) . Two points ζ , ζ′ ∈ Rp belong to the same region in the partition iff F ( ζ ) = F ( ζ′ ) . Each region is of the form Z = { ζ ∈ Rp|F ( ζ ) = b } , for some sign-pattern vector b ∈ { ±1 } M . We show that for any fixed metrics d1 , . . . , dL and clustering instance S = { x1 , . . . , xn } ⊂ X , we can find a signpattern partitioning of ∆L induced by linear functions such that , on each region , the ordering over pairs of points in S induced by the metric dβ is constant . An important consequence of this result is that for each region Z in this partitioning of ∆L , the following holds : For any 2-point-based merge function D and any pair of clusters A , B ⊂ S , there exists a pair of points ( a , b ) ∈ A×B such that D ( A , B ; dβ ) = dβ ( a , b ) for all β ∈ Z . In other words , restricted to β parameters belonging to Z , the same pair of points ( a , b ) defines the D-merge distance for the clusters A and B. Lemma 1 . Fix any metrics d1 , . . . , dL and a clustering instance S ⊂ X . There exists a set H of O ( |S|4 ) linear functions mapping RL to R with the following property : if two metric parameters β , β′ ∈ ∆L belong to the same region in the sign-pattern partition induced byH , then the ordering over pairs of points in S given by dβ and dβ′ are the same . That is , for all points a , b , a′ , b′ ∈ S we have dβ ( a , b ) ≤ dβ ( a′ , b′ ) iff dβ′ ( a , b ) ≤ dβ′ ( a′ , b′ ) . Proof sketch . For any pair of points a , b ∈ S , the distance dβ ( a , b ) is a linear function of the parameter β . Therefore , for any four points a , b , a′ , b′ ∈ S , we have that dβ ( a , b ) ≤ dβ ( a′ , b′ ) iff ha , b , a′ , b′ ( β ) ≤ 0 , where ha , b , a′ , b′ is the linear function given by ha , b , a′ , b′ ( β ) = dβ ( a , b ) − dβ ( a′ , b′ ) . Let H = { ha , b , a′ , b′ | a , b , a′ , b′ ∈ S } be the collection of all such linear functions arising from any subset of 4 points . On each region of the sign-pattern partition induced byH , all comparisons of pairwise distances in S are fixed , implying that the ordering over pairs of points in S is fixed . Building on Lemma 1 , we now prove the main structural property of Algorithm 1 . We argue that for any clustering instance S ⊂ X , there is a partition induced by quadratic functions of ∆L′ ×∆L ⊂ RL ′+L over α and β into regions such that on each region , the ordering over all pairs of clusters according to the merge distance Dα ( · , · ; dβ ) is fixed . This implies that for all ( α , β ) in one region of the partition , the output of Algorithm 1 when run on S is constant , since the algorithm output only depends on the ordering over pairs of clusters in S given by Dα ( · , · ; dβ ) . Lemma 2 . Fix any metrics d1 , . . . , dL , any 2-point-based merge functions D1 , . . . , DL′ , and clustering instance S ⊂ X . There exists a set Q of O ( |S|4L′ ) quadratic functions defined on RL′+L so that if parameters ( α , β ) and ( α′ , β′ ) belong to the same region of the sign-pattern partition induced by Q , then the ordering over pairs of clusters in S given by Dα ( · , · ; dβ ) and Dα′ ( · , · ; dβ′ ) is the same . That is , for all clusters A , B , A′ , B′ ⊂ S , we have that Dα ( A , B ; dβ ) ≤ Dα ( A′ , B′ ; dβ ) iff Dα′ ( A , B ; dβ′ ) ≤ Dα′ ( A′ , B′ ; dβ′ ) . Proof sketch . LetH be the linear functions constructed in Lemma 1 and fix any region Z in the sign-pattern partition induced by H. For any i ∈ [ L′ ] , since the merge function Di is 2-point-based and the ordering over pairs of points according to dβ ( · , · ) in the region Z is fixed , for any clusters A , B , A′ , B′ ⊂ S we can find points ai , bi , a′i , b′i such that Di ( A , B ; dβ ) = dβ ( ai , bi ) and Di ( A′ , B′ ; dβ ) = dβ ( a′i , b ′ i ) for all β ∈ R. Therefore , expanding the definition of Dα , we have that Dα ( A , B ; dβ ) ≤ Dα ( A′ , B′ ; dβ ) iff qA , B , A′ , B′ ( α , β ) ≤ 0 , where qA , B , A′ , B′ ( α , β ) =∑ i ∑ j αiβj ( dj ( ai , bi ) − dj ( a′i , b′i ) ) . Observe that the coefficients of each quadratic function depend only on 4L′ points in S , so there are onlyO ( |S|4L′ ) possible quadratics of this from collected across all regions Z in the sign-pattern partition induced byH and all subsets of 4 clusters . Together withH , this set of quadratic functions partitions the joint parameter space into regions where the ordering over all pairs of clusters is fixed . A consequence of Lemma 2 is that for any clustering instance S with target clustering Y , the function ( α , β ) 7→ ` ( Aα , β ( S ) , Y ) is piecewise constant , where the constant partitioning is the sign-pattern partition induced by O ( |S|4L ′ ) quadratic functions . The proof of Theorem 1 now follows from using this fact to bound the empirical Rademacher complexity of the learning problem . Extensions . The above analysis can be extended to handle several more general settings . First , we can accommodate many specific merge functions that are not included in the 2-point-based family , at the cost of increasing the number of quadratic functions |Q| needed in Lemma 2 . For example , if one of the merge functions is average linkage , Davg ( A , B ; d ) = 1 |A|·|B| ∑ a∈A , b∈B d ( a , b ) , then |Q| will be exponential in the dataset size n. Fortunately , our sample complexity analysis depends only on log ( |Q| ) , so this still leads to non-trivial sample complexity guarantees ( though the computational algorithm selection problem becomes harder ) . We can also extend the analysis to more intricate methods for combining the metrics and merge functions . For example , our analysis applies to polynomial combinations of metrics and merges at the cost of increasing the complexity of the functions defining the piecewise constant partition .

Summary: This paper studies the problem of learning both the distance metric and a linkage rule from clustering examples. Suppose we have L metrics d_1, …, d_L and L’ linkage rules for hierarchical agglomerative clustering, D_1, …, D_L’ where each rule is a 2-point-based merge function (i.e. computes the distance between some two points in the clusters, examples of such functions are single-linkage and complete-linkage). The paper considers the problem of finding the convex combination of the distance functions and linkage rules which best fits the data. The main result (Theorem 1) is an \tildeO((L’ + L)^2 L’ /eps^2) uniform convergence bound on the number of clustering instances which are required to learn up to expected loss \eps the best possible convex combination. The key technical part of the proof is showing that for any fixed clustering the loss function is piecewise-constant with a small number of simple pieces. The overall approach is based on Balcan et al.’17 who solve the case when the distance metric is known but the linkage rule is to be learned and Balcan et al. ‘19 who give techniques for the piecewise constant case. Some further results are given which are specific to learning a mix of two merge functions under a single distance metric and the best combination of two metrics when using the complete linkage merge function. Experimental results are given on MNIST, CIFAR-10 and some other fairly small datasets.