On Margin-Based Cluster Recovery with Oracle Queries

1 Introduction . This work investigates the problem of exact cluster recovery using oracle queries , in the well-known framework introduced by Ashtiani et al . [ 2016 ] . We are given a set X of n points from some domain X ( e.g. , from the Euclidean m-dimensional space Rm ) and an oracle answering to samecluster queries of the form “ are these two points in the same cluster ? ” or label queries of the form “ which cluster does this point belong to ? ” . The oracle answers are consistent with some clustering C = ( C1 , . . . , Ck ) of X unknown to the algorithm , where k is a fixed constant . The goal is to design an algorithm that recovers C by using as few queries as possible . Clearly , if there are no restrictions on C , then any algorithm needs n queries in the worst case . Thus , our goal is to understand when C can be recovered efficiently ; ideally , in polynomial time , and by making O ( log n ) queries . A natural attempt is to consider clusterings with well-separated clusters , since this is what is often considered a “ good clustering ” . The existing work goes precisely in this direction , starting with the following result of Ashtiani et al . [ 2016 ] for the case X ✓ Rm . If every cluster Ci is separated from X \ Ci by a ball centered in the center of mass of Ci , and that ball does not intersect other clusters even if expanded by a factor of 1 + , then with high probability 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . poly ( k , m , 1/ ) log n queries are sufficient to recover C in polynomial time . This is called “ spherical margin ” condition , and is called the margin . Unfortunately , the spherical margin condition is not very realistic , since it imposes a very precise shape upon the clusters . In a generalization attempt , Bressan et al . [ 2020 ] showed that one can actually recover clusters with ellipsoidal separators with arbitrary centers , by increasing the number of queries to poly ( k , m , 1/ ) ( m/ ) m log n. This result is achieved via boosted one-sided error learning , which works as follows . Suppose that , by making f ( k , m , ) queries , we could identify correctly ( with zero mistakes ) a constant fraction of the points in some cluster Ci . Then , we could label those points as i , remove them from the dataset , and repeat . It is not hard to show that , after O ( k log n ) rounds , we will have correctly labeled all the input points with high probability . The difficult task is , of course , to learn a constant fraction of some cluster Ci with one-sided error ( one-sided error means that any point of X predicted to be in Ci must be in Ci ) . The key insight in [ Bressan et al. , 2020 ] is that , if the clusters have margin with respect to their ellipsoidal separators , then roughly ( m/ ) m queries are sufficient . This leads to the following question : how much can this approach be extended ? In this work we provide several answers , revealing an interesting connection between margin-based cluster recovery and one-sided error learning . Our main contributions are as follows . 1 . We introduce a new notion of margin in Rm , that we call “ convex hull margin ” ( Definition 2 ) . This is a strict generalization of the margins of Bressan et al . [ 2020 ] , Ashtiani et al . [ 2016 ] and of the usual SVM margin , and allows the clusters to have any shape whatsoever as long as they are convex . Under the convex hull margin , we develop a novel technique for learning with one-sided error that we call convex hull expansion trick . It essentially amounts to sampling many points from a single cluster and “ inflate ” their convex hull by a factor of ( 1 + ) . This technique yields a polynomial-time exact cluster recovery algorithm that uses poly ( k , m , 1/ ) ( 1 + 1/ ) m log n queries ( Theorem 1 ) . The ( 1 + 1/ ) m dependence on and m is significantly better than that of Bressan et al . [ 2020 ] , and closer to their lower bound of order ( 1 + 1/ ) m/2 . 2 . We introduce a notion of cluster margin for general pseudometric spaces called one-versus-all margin ( Definition 3 ) . This notion of margin is strictly more general than convex hull margin , and captures , as special cases , standard notions of stability for clustering problems such as k-means or k-centers . We show that , if a clustering has one-versus-all margin , then it can be recovered with M ( ) poly ( k ) log n queries via a pure learning-theoretic approach ( Theorem 3 ) , where M ( ) is a quantity related to the packing numbers of the pseudometric space . We show that the dependence on M ( ) is essentially optimal , thus characterizing the recoverability of clusterings in this setting . 3 . Finally , we show a connection between margin-based learning and exact active cluster recoverability , when clusters are realized by some concept class H ( that is , when for each cluster Ci there is a concept hi 2 H such that X \ hi = Ci ) . We show that if a certain combinatorial parameter , the coslicing dimension cosl ( H ) , is bounded , then one can learn clusterings with cosl ( H ) poly ( k ) log n label queries ; otherwise , ⌦ ( n ) queries are needed in the worst case ( Theorem 4 ) . Moreover we show that , for all concept classes in Rm that are closed under affine transformations and well-behaved in a natural sense , finite coslicing dimension and positive one-versus-all margin are equivalent ( Theorem 5 ) . Note that actively learning a clustering is equivalent , up to a relabeling of the classes , to actively learning a multiclass classifier in the transductive realizable case . Hence our results apply to that case , too . In particular , our O ( log n ) query bounds imply eO ( log 1/ '' ) query bounds for pool-based active learning [ McCallum and Nigam , 1998 ] of multiclass classifiers , where `` is the generalization error . To see this , draw a set X of ⇥ `` 1 ( K log 1/ '' + log 1/ ) unlabeled samples from the underlying distribution , where K is the relevant measure of capacity ( e.g. , the Natarajan dimension ) , run our algorithms over X , and compute a hypothesis consistent with the recovered labeling C. These kinds of reductions are standard in active learning , see for instance [ Kane et al. , 2017 ] . On the other hand , our results do not apply to actively learning subsets h ✓ X ( that is , to active learning in the standard sense ) if our margin conditions are only enjoyed by the set of positives X+ = X \ h. To see this , let X ⇢ R2 and suppose X+ contains a single point . Then X+ satisfies our conditions with unbounded margin w.r.t . the Euclidean distance , but any algorithm needs ⌦ ( n ) label queries to recover it . This does not happen with clustering because , in that case , every class enjoys the margin property , and in particular both the “ positives ” and the “ negatives ” for k = 2 . Table 1 compares the bounds resulting from the different notions of margins known . where dW ( a , b ) = p ha b , W ( a b ) i ; note that for ⌧ 1 this can be thought of as dW ( y , ci ) > ( 1 + ) dW ( x , ci ) . Related work . Same-cluster queries were introduced formally in [ Ashtiani et al. , 2016 ] together with the active cluster recovery problem . Those queries are natural to implement in crowd-sourcing systems , and for this reason they have been extensively studied both in theory [ Ailon et al. , 2018a , b , Gamlath et al. , 2018 , Huleihel et al. , 2019 , Mazumdar and Pal , 2017 , Mazumdar and Saha , 2017b , a , Saha and Subramanian , 2019 , Vitale et al. , 2019 ] and in practice [ Firmani et al. , 2018 , Gruenheid et al. , 2015 , Verroios and Garcia-Molina , 2015 , Verroios et al. , 2017 ] . Various notions of margin are central in both active learning and cluster recovery [ Xu et al. , 2004 , Balcan et al. , 2007 , Balcan and Long , 2013 , Kane et al. , 2017 , Bressan et al. , 2021 ] . Our coslicing dimension is similar to the slicing dimension of Kivinen [ 1995 ] and the star number of Hanneke and Yang [ 2015 ] . Our lower bounds , like many others , are inspired from a construction by Dasgupta [ 2004 ] . Our arguments based on packing numbers are similar to those based on the inference dimension of Kane et al . [ 2017 ] or the lossless sample compression of Hopkins et al . [ 2021 ] , as we can not infer the label of a point only when it is far from already-labeled points . A query bound similar to the one given by our convex hull expansion trick , but worse by a factor roughly 2m , can be inferred by adapting arguments of Hopkins et al . [ 2020b ] . Combinatorial characterizations of multiclass learning have been proposed in the passive case by Ben-David et al . [ 1995 ] , Rubinstein et al . [ 2009 ] , Daniely and Shalev-Shwartz [ 2014 ] . Other learning settings related to one-sided and active learning are RPU learning [ Rivest and Sloan , 1988 ] and perfect selective classification [ El-Yaniv and Wiener , 2012 ] — see [ Hopkins et al. , 2020a ] for a discussion . 2 Preliminaries and notation . All missing proofs can be found in the supplementary material . The input is a pair ( X , O ) , where X is a set of n points from some domain X , and O is a label oracle that , when queried on any x 2 X , returns the cluster id C ( x ) of x . The oracle O is consistent with a latent clustering C = ( C1 , . . . , Ck ) of X , i.e. , a k-tuple of pairwise disjoint sets whose union is X .1 We allow clusters to be empty . Our goal is to recover C by making as few queries as possible to O. Queries can be made adaptively , that is , the j-th point to be queried can be chosen as a function of the answers to the first j 1 queries . We express the number of queries as a function of k , n , and other parameters to be introduced later . This setting is essentially equivalent to the semi-supervised active clustering ( SSAC ) framework of Ashtiani et al . [ 2016 ] , where the oracle answers same-cluster queries SCQ ( x , y ) that , for any two points x , y 2 X , return TRUE iff C ( x ) = C ( y ) . We use label queries instead of SCQ queries for simplicity ; any SCQ query can be emulated with two label queries . Conversely , the label of any point can be learned with k SCQ queries , up to a relabeling of the clusters , so our bounds hold for an SCQ oracle as well if multiplied by k. For conciseness , we state query bounds in the form f ( k , m , n , ) , 1In line with previous works , we assume k is fixed and known . for instance O k2 ( 1 + 1/ ) m log n . These bounds should be thought of as min n , f ( k , m , n , ) , since obviously we never need to query the same point twice . We often assume a metric or pseudometric d over X ( a pseudometric allows two distinct points to have distance 0 ) . For x 2 X and r 0 , the closed ball of radius r centered at x is B ( x , r ) = { y 2 X : d ( x , y ) r } . For any X ⇢ X , we denote by d ( X ) = supx , x02X d ( x , x0 ) the diameter of X measured by d , and we define d ( ; ) = 0 . For any two sets U , S ⇢ X , we denote by d ( U , S ) = infx2U , y2S d ( U , S ) their distance according to d , and we define d ( U , ; ) = 1 . For any X ⇢ Rm , we write conv ( X ) for the convex hull of X . The unit sphere in Rm is Sm 1 = { x 2 Rm : kxk2 = 1 } . We recall some learning-theoretic facts . Let H be an arbitrary collection of subsets of X ( i.e. , a concept class ) . The intersection class of H is I ( H ) = S i2N { h1 \ . . .\ hi : h1 , . . . , hi 2 H } . Given any S ⇢ X and any S0 ✓ S realized by some h ? 2 H , the smallest concept in I ( H ) consistent with S0 is defined as h = T { h 2 H : h \ S = S0 } . Note that h ✓ h ? . Finally , we recall the definition of learning with one-sided error : Definition 1 ( Kivinen [ 1995 ] , Definition 4.4 ) . An algorithm A learns H with one-sided error `` and confidence with r examples if , for any target concept h ? 2 H and any probability measure P over X , by drawing r independent labeled examples from P , the algorithm outputs a concept h ✓ h ? such that P ( h ? \ h ) `` with probability at least 1 .

Summary: This work examines a cluster recovery problem where the algorithm has access to an oracle that can return whether two points lie in the same cluster. It builds on a line of work [Ashtiani et al. ‘16] [Bressan et al. ‘20] that does so with only O(log n) queries but under a margin property (the dependence on the margin is hidden in the number of queries). [Ashtiani et al. ‘16] did so with a spherical margin property: there is a sphere around each cluster separating it from all other points, and this had to continue to hold as the sphere was expanded by a (1 + gamma) factor. [Bressan et al. ‘20] generalized this assumption to ellipsoidal margins, and this work further generalizes that to “convex hull” margins and even further to “one-versus-all” margins.