On Margin-Based Cluster Recovery with Oracle Queries

1 Introduction . This work investigates the problem of exact cluster recovery using oracle queries , in the well-known framework introduced by Ashtiani et al . [ 2016 ] . We are given a set X of n points from some domain X ( e.g. , from the Euclidean m-dimensional space Rm ) and an oracle answering to samecluster queries of the form “ are these two points in the same cluster ? ” or label queries of the form “ which cluster does this point belong to ? ” . The oracle answers are consistent with some clustering C = ( C1 , . . . , Ck ) of X unknown to the algorithm , where k is a fixed constant . The goal is to design an algorithm that recovers C by using as few queries as possible . Clearly , if there are no restrictions on C , then any algorithm needs n queries in the worst case . Thus , our goal is to understand when C can be recovered efficiently ; ideally , in polynomial time , and by making O ( log n ) queries . A natural attempt is to consider clusterings with well-separated clusters , since this is what is often considered a “ good clustering ” . The existing work goes precisely in this direction , starting with the following result of Ashtiani et al . [ 2016 ] for the case X ✓ Rm . If every cluster Ci is separated from X \ Ci by a ball centered in the center of mass of Ci , and that ball does not intersect other clusters even if expanded by a factor of 1 + , then with high probability 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . poly ( k , m , 1/ ) log n queries are sufficient to recover C in polynomial time . This is called “ spherical margin ” condition , and is called the margin . Unfortunately , the spherical margin condition is not very realistic , since it imposes a very precise shape upon the clusters . In a generalization attempt , Bressan et al . [ 2020 ] showed that one can actually recover clusters with ellipsoidal separators with arbitrary centers , by increasing the number of queries to poly ( k , m , 1/ ) ( m/ ) m log n. This result is achieved via boosted one-sided error learning , which works as follows . Suppose that , by making f ( k , m , ) queries , we could identify correctly ( with zero mistakes ) a constant fraction of the points in some cluster Ci . Then , we could label those points as i , remove them from the dataset , and repeat . It is not hard to show that , after O ( k log n ) rounds , we will have correctly labeled all the input points with high probability . The difficult task is , of course , to learn a constant fraction of some cluster Ci with one-sided error ( one-sided error means that any point of X predicted to be in Ci must be in Ci ) . The key insight in [ Bressan et al. , 2020 ] is that , if the clusters have margin with respect to their ellipsoidal separators , then roughly ( m/ ) m queries are sufficient . This leads to the following question : how much can this approach be extended ? In this work we provide several answers , revealing an interesting connection between margin-based cluster recovery and one-sided error learning . Our main contributions are as follows . 1 . We introduce a new notion of margin in Rm , that we call “ convex hull margin ” ( Definition 2 ) . This is a strict generalization of the margins of Bressan et al . [ 2020 ] , Ashtiani et al . [ 2016 ] and of the usual SVM margin , and allows the clusters to have any shape whatsoever as long as they are convex . Under the convex hull margin , we develop a novel technique for learning with one-sided error that we call convex hull expansion trick . It essentially amounts to sampling many points from a single cluster and “ inflate ” their convex hull by a factor of ( 1 + ) . This technique yields a polynomial-time exact cluster recovery algorithm that uses poly ( k , m , 1/ ) ( 1 + 1/ ) m log n queries ( Theorem 1 ) . The ( 1 + 1/ ) m dependence on and m is significantly better than that of Bressan et al . [ 2020 ] , and closer to their lower bound of order ( 1 + 1/ ) m/2 . 2 . We introduce a notion of cluster margin for general pseudometric spaces called one-versus-all margin ( Definition 3 ) . This notion of margin is strictly more general than convex hull margin , and captures , as special cases , standard notions of stability for clustering problems such as k-means or k-centers . We show that , if a clustering has one-versus-all margin , then it can be recovered with M ( ) poly ( k ) log n queries via a pure learning-theoretic approach ( Theorem 3 ) , where M ( ) is a quantity related to the packing numbers of the pseudometric space . We show that the dependence on M ( ) is essentially optimal , thus characterizing the recoverability of clusterings in this setting . 3 . Finally , we show a connection between margin-based learning and exact active cluster recoverability , when clusters are realized by some concept class H ( that is , when for each cluster Ci there is a concept hi 2 H such that X \ hi = Ci ) . We show that if a certain combinatorial parameter , the coslicing dimension cosl ( H ) , is bounded , then one can learn clusterings with cosl ( H ) poly ( k ) log n label queries ; otherwise , ⌦ ( n ) queries are needed in the worst case ( Theorem 4 ) . Moreover we show that , for all concept classes in Rm that are closed under affine transformations and well-behaved in a natural sense , finite coslicing dimension and positive one-versus-all margin are equivalent ( Theorem 5 ) . Note that actively learning a clustering is equivalent , up to a relabeling of the classes , to actively learning a multiclass classifier in the transductive realizable case . Hence our results apply to that case , too . In particular , our O ( log n ) query bounds imply eO ( log 1/ '' ) query bounds for pool-based active learning [ McCallum and Nigam , 1998 ] of multiclass classifiers , where `` is the generalization error . To see this , draw a set X of ⇥ `` 1 ( K log 1/ '' + log 1/ ) unlabeled samples from the underlying distribution , where K is the relevant measure of capacity ( e.g. , the Natarajan dimension ) , run our algorithms over X , and compute a hypothesis consistent with the recovered labeling C. These kinds of reductions are standard in active learning , see for instance [ Kane et al. , 2017 ] . On the other hand , our results do not apply to actively learning subsets h ✓ X ( that is , to active learning in the standard sense ) if our margin conditions are only enjoyed by the set of positives X+ = X \ h. To see this , let X ⇢ R2 and suppose X+ contains a single point . Then X+ satisfies our conditions with unbounded margin w.r.t . the Euclidean distance , but any algorithm needs ⌦ ( n ) label queries to recover it . This does not happen with clustering because , in that case , every class enjoys the margin property , and in particular both the “ positives ” and the “ negatives ” for k = 2 . Table 1 compares the bounds resulting from the different notions of margins known . where dW ( a , b ) = p ha b , W ( a b ) i ; note that for ⌧ 1 this can be thought of as dW ( y , ci ) > ( 1 + ) dW ( x , ci ) . Related work . Same-cluster queries were introduced formally in [ Ashtiani et al. , 2016 ] together with the active cluster recovery problem . Those queries are natural to implement in crowd-sourcing systems , and for this reason they have been extensively studied both in theory [ Ailon et al. , 2018a , b , Gamlath et al. , 2018 , Huleihel et al. , 2019 , Mazumdar and Pal , 2017 , Mazumdar and Saha , 2017b , a , Saha and Subramanian , 2019 , Vitale et al. , 2019 ] and in practice [ Firmani et al. , 2018 , Gruenheid et al. , 2015 , Verroios and Garcia-Molina , 2015 , Verroios et al. , 2017 ] . Various notions of margin are central in both active learning and cluster recovery [ Xu et al. , 2004 , Balcan et al. , 2007 , Balcan and Long , 2013 , Kane et al. , 2017 , Bressan et al. , 2021 ] . Our coslicing dimension is similar to the slicing dimension of Kivinen [ 1995 ] and the star number of Hanneke and Yang [ 2015 ] . Our lower bounds , like many others , are inspired from a construction by Dasgupta [ 2004 ] . Our arguments based on packing numbers are similar to those based on the inference dimension of Kane et al . [ 2017 ] or the lossless sample compression of Hopkins et al . [ 2021 ] , as we can not infer the label of a point only when it is far from already-labeled points . A query bound similar to the one given by our convex hull expansion trick , but worse by a factor roughly 2m , can be inferred by adapting arguments of Hopkins et al . [ 2020b ] . Combinatorial characterizations of multiclass learning have been proposed in the passive case by Ben-David et al . [ 1995 ] , Rubinstein et al . [ 2009 ] , Daniely and Shalev-Shwartz [ 2014 ] . Other learning settings related to one-sided and active learning are RPU learning [ Rivest and Sloan , 1988 ] and perfect selective classification [ El-Yaniv and Wiener , 2012 ] — see [ Hopkins et al. , 2020a ] for a discussion . 2 Preliminaries and notation . All missing proofs can be found in the supplementary material . The input is a pair ( X , O ) , where X is a set of n points from some domain X , and O is a label oracle that , when queried on any x 2 X , returns the cluster id C ( x ) of x . The oracle O is consistent with a latent clustering C = ( C1 , . . . , Ck ) of X , i.e. , a k-tuple of pairwise disjoint sets whose union is X .1 We allow clusters to be empty . Our goal is to recover C by making as few queries as possible to O. Queries can be made adaptively , that is , the j-th point to be queried can be chosen as a function of the answers to the first j 1 queries . We express the number of queries as a function of k , n , and other parameters to be introduced later . This setting is essentially equivalent to the semi-supervised active clustering ( SSAC ) framework of Ashtiani et al . [ 2016 ] , where the oracle answers same-cluster queries SCQ ( x , y ) that , for any two points x , y 2 X , return TRUE iff C ( x ) = C ( y ) . We use label queries instead of SCQ queries for simplicity ; any SCQ query can be emulated with two label queries . Conversely , the label of any point can be learned with k SCQ queries , up to a relabeling of the clusters , so our bounds hold for an SCQ oracle as well if multiplied by k. For conciseness , we state query bounds in the form f ( k , m , n , ) , 1In line with previous works , we assume k is fixed and known . for instance O k2 ( 1 + 1/ ) m log n . These bounds should be thought of as min n , f ( k , m , n , ) , since obviously we never need to query the same point twice . We often assume a metric or pseudometric d over X ( a pseudometric allows two distinct points to have distance 0 ) . For x 2 X and r 0 , the closed ball of radius r centered at x is B ( x , r ) = { y 2 X : d ( x , y ) r } . For any X ⇢ X , we denote by d ( X ) = supx , x02X d ( x , x0 ) the diameter of X measured by d , and we define d ( ; ) = 0 . For any two sets U , S ⇢ X , we denote by d ( U , S ) = infx2U , y2S d ( U , S ) their distance according to d , and we define d ( U , ; ) = 1 . For any X ⇢ Rm , we write conv ( X ) for the convex hull of X . The unit sphere in Rm is Sm 1 = { x 2 Rm : kxk2 = 1 } . We recall some learning-theoretic facts . Let H be an arbitrary collection of subsets of X ( i.e. , a concept class ) . The intersection class of H is I ( H ) = S i2N { h1 \ . . .\ hi : h1 , . . . , hi 2 H } . Given any S ⇢ X and any S0 ✓ S realized by some h ? 2 H , the smallest concept in I ( H ) consistent with S0 is defined as h = T { h 2 H : h \ S = S0 } . Note that h ✓ h ? . Finally , we recall the definition of learning with one-sided error : Definition 1 ( Kivinen [ 1995 ] , Definition 4.4 ) . An algorithm A learns H with one-sided error `` and confidence with r examples if , for any target concept h ? 2 H and any probability measure P over X , by drawing r independent labeled examples from P , the algorithm outputs a concept h ✓ h ? such that P ( h ? \ h ) `` with probability at least 1 .

Summary: This work studies query efficient cluster recovery with label queries. In particular, given a k-clustering $(C_1,\ldots,C_k)$ of an $n$ point set $X$, the learner may ask, adaptively, for any $x \in X$, “which cluster contains x?” The goal is to recover the clustering on all n points using only $O_k(\log(n))$ queries. This is a very natural problem, variants of which have seen significant study both within the clustering (e.g. Ashtiani et al. 2016) and active learning literature (e.g. Kane et al. 2017).  The authors prove a number of interesting results to this effect. In the Euclidean setting, the authors introduce a novel complexity measure called the “Convex Hull Margin” that non-trivially generalizes previous sufficient conditions for query efficient cluster recovery such ball and ellipsoid-based margins and the SVM margin. They show that small convex hull margin implies computational and query efficient cluster recovery (with near optimal query complexity), even improving previous special cases such as the ellipsoid margin. The authors also study an extension to non-Euclidean spaces which are only assumed to have some fixed set of pseudometrics. In this case, they introduce a novel measure called “one-versus-all” margin. They prove this measure generalizes a number of known cluster recovery results such as center proximity and perturbation stability, and also gives a sufficient condition for query (but not computationally) efficient cluster recovery.  Finally, the authors also address an interesting variant of cluster recovery where the clusters must be realized by some fixed concept class H. In this scenario, the authors introduce coslicing dimension which captures a well-known lower bound of Dasgupta showing that efficient learning is impossible when every point can be cutoff from the rest. They then proceed to show that this is in fact the only barrier to efficient cluster recovery—proving that finite coslicing dimension is necessary and sufficient.