Fuzzy Clustering with Similarity Queries

1 Introduction . The k-means objective for clustering is perhaps the most ubiquitous of all unsupervised learning paradigms . It is extremely well studied , with Lloyd ’ s algorithm being the most popular solution method , while the problem in general being NP Hard [ 11 , 7 ] . A variety of approximate solutions for k-means exists ( such as , [ 20 , 3 ] ) . In recent times , there have been efforts to bring in a flavor of active learning in k-means clustering ; by allowing the learner to make a limited number of carefully chosen label/membership queries [ 4 , 2 , 18 ] . In this setting , the main objective is to show that a polynomial-time solution exists provided that small number of such queries to an oracle is permitted . Note that , an alternative to membership query ( i.e. , query of the form “ does the ith element belong to the jth cluster ” ) is the same-cluster/similarity query ( i.e. , “ do elements i and j belong to the same cluster ” ) . Given representatives from each cluster is available , one can simulate any membership query with at most k same cluster queries . Hence whatever is achievable with membership queries , can be also achieved by same cluster queries by asking at most k times as many queries [ 4 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . How realistic is to allow the learner to make such label queries , and how realistic is the oracle assumption ? It turns out that many clustering tasks , primarily entity-resolution , are being delegated to crowdsourcing models . One can easily assume that the crowd is playing the role of an oracle here . It is natural in these models to assign crowd workers with sub-tasks , which can either be answering the membership query or the same cluster query [ 5 , 49 , 50 , 38 , 23 ] . Fuzzy k-means or soft k-means is a generalized model of the k-means objective that covers lot more grounds being applicable to dataset where datapoints show affinity to multiple labels , the clustering criteria are vague and data features are unavailable [ 8 ] . In fuzzy clustering , instead of an element being part of one fixed cluster , it is assumed that it has a membership value between 0 and 1 to each of the clusters . The objective of the clustering is to recover all of the membership values for each element or datapoint . This relaxation in membership values from { 0 , 1 } to [ 0 , 1 ] makes the fuzzy k-means an attractive model for overlapping clusters [ 52 ] , multilabel classification [ 30 ] and numerous other applications such as , pattern recognition [ 8 ] , computational biology [ 6 ] , bioinformatics [ 47 ] , image segmentation [ 1 , 19 ] and so on . What is more interesting is that solving fuzzy clustering is equivalent to approximate solution of nonnegative symmetric matrix factorization problems [ 22 ] . Given such applications , it is somewhat surprising that the theoretical results known about fuzzy clustering is quite limited . As is even the computational tractability of fuzzy k-means is only conjecturally known . In fact algorithmic study of fuzzy k-means algorithm seems to have been only recently started [ 9 , 10 , 31 ] . Membership vs. similarity queries . As mentioned above , in the hard-clustering case , one can simulate any membership query with at most k same-cluster ( or , similarity ) queries [ 4 ] . In fuzzy clustering , we are given a set of n vectors ( items ) in the d-dimensional Euclidean space , and we are tasked the problem of determining the k-dimensional membership vector for each of the items . The clustering algorithm has access to the set of vectors , and can also make queries about the underlying soft clustering . As in hard-clustering , these queries can take at least two forms . Specifically , membership queries refer to the existence of a membership-oracle which whenever queried , replies with the membership value of the ith item to the jth cluster , i.e. , the jth entry of the ith membership vector . One could argue that querying the membership values , and specifically , the existence of such a membership-oracle , is not realistic , and often impractical in real-world settings since it requires knowledge of the relevant clusters . Instead , the similarity query model that takes two or three elements as input and ask “ How similar are all these elements ? '' is much easier to implement in practice . Accordingly , in a similar vein to hard-clustering , we show that membership queries can be simulated by similarity queries , measured by the inner-product between pairs ( and triplets ) of membership vectors . Finally , as will be explained in more detail , the responses of the oracle or the target solution do not need to correspond to the optimum solution of the fuzzy k-means objective , but instead need to satisfy a few mild conditions . Accordingly , the target solution can represent noisy/quantized version of the optimum solution of the fuzzy k-means objective function making our set-up highly practical in crowdsourcing settings . A natural question that one can ask about fuzzy clustering following the lead of [ 4 ] is how many membership/similarity queries one may ask so that the clustering can be efficiently solved with a good approximation of the target solution ? In this paper , our main motivation is to answer this rather basic question . It has to be noted that revealing a fractions of labels during clustering algorithm in fuzzy k-means has been studied with some heuristics in [ 42 ] , but not only the model was different from ours , a rigorous theoretical study was also lacking in the past literature . Our contributions and techniques . We design several algorithms and analyze their query and computational complexities . Specifically , the first algorithm we propose works in two phases : in the first phase , we estimate the center of each cluster using non-adaptive queries , and in the second phase we recover the membership of each element to each cluster using the estimated centers obtained in the first phase . We show with this algorithm it is possible to get an approximation of the membership vectors by making only O ( poly ( k/ ) log n ) queries , where is ratio of the volume of the smallest cluster to the average volume , informally speaking . Then , we show that by allowing one to ask adaptive/sequential queries in the first phase , the polynomial dependence on of the query complexity can be further improved . Despite the strong dependency of the query complexity on , for a large regime of parameters , our query complexity is non-trivial and sublinear in n. Interestingly , for the special case of k = 2 , we can completely get rid of the dependency of the query complexity on , and it suffices to ask O ( log2 n ) queries only . For our theoretical results to hold , we make an assumption on the dataset that quantifies the clusterability of the data . We would like to emphasize that our assumption is only needed for deriving the theoretical guarantees ; our algorithms are demonstrated to perform well on real-world datasets irrespective of whether our assumption holds or not . The computational complexity of all the algorithms scales as d times the number of queries . It is to be noted that the analysis of the aforementioned algorithms are technically much more demanding than solving the analogous query complexity problem for hard k-means . For the case of hard k-means , a general principle would be to sample a small set of items and estimate the centre of the largest cluster from the samples ( see , [ 4 ] ) . Subsequently all the items closest to the estimated centre can be assigned to it and the cluster can be removed from consideration . One can iterate over this algorithm to find all the clusters . However the size of a cluster is not well-defined in the fuzzy setting ; and also it is not possible to assign items to a cluster and remove them from consideration , since they can very well have nonzero memberships to other clusters . To tackle these difficulties , we have to use more challenging technical tools and analysis methods than the hard k-means literature . In a nutshell , since at any step of the algorithm we can not remove items from consideration , we propose sophisticated random sub-sampling algorithms that explore the space of items efficiently . Specifically , having reliable approximations for the centers and memberships of some processed clusters , we partition the space of items into bins that have the same approximated sum of membership weights to unprocessed clusters . Sampling equally from every bin ensures that we obtain enough samples from items which have high membership weight to those clusters which have not yet been approximated . Also , contrary to hard k-means where a simple ( unbiased ) empirical mean estimator is used to approximate the means , in the fuzzy setting , we end up with a form of self-normalized importance sampling estimators . While these estimators are biased , we managed to bound their deviations in an elegant way using concentration inequalities . We tested our algorithms over both synthetic and real-world datasets , and illustrate the tradeoff and effect of different values of and k on the estimation error and query complexity . Related work . Clustering with limited label-queries have mostly followed two different lines of works . In the first line of work , queries were used to make the objective based clustering , such as k-means , polynomial time solvable [ 4 , 2 , 18 , 12 , 13 , 14 ] . The problem being considered here is most closely related to this line of work , albeit , instead of hard k-means , we focus on a soft objective . In the second line of work , instead of an objective based clustering , it is assumed that there exists a ground-truth clustering , and by label-querying , one gets a ( noisy ) answer from an oracle about this ground-truth . The aim of this line of work has been to give statistical-computational guarantees on recovery of the ground truth [ 36 , 37 , 35 , 2 , 15 , 46 , 43 ] . The work in this canon that is closest to our problem is [ 28 ] , where an overlapping clustering groundtruth was considered . At a high level , we also consider overlapping clusters , however , focus on objective-based clustering therein . It is worth noting that clustering with same cluster , or other label queries , has found connections with and application to correlation clustering [ 15 , 43 ] , community detection [ 37 , 17 ] , heavy-hitters [ 44 ] and possibly other areas , that we might have missed . Our problem can be seen as approximately equivalent to non-negative symmetric matrix factorization [ 40 ] with limited access to the entries of the factors ( via queries ) [ 22 ] . Organization . The rest of the paper is organized as follows . In Section 2 we present our model and the learning problem . Then , in Section 3 we present our algorithms and their theoretical guarantees . The main theoretical results can be found in Theorems 1,2,3 . Due to space limitation , proofs , some pseudocodes , experimental results over both synthetic and real-world datasets , and conclusions are relegated to the supplementary material .

Summary: Given a set of points X in R^d, a fuzzy k-means clustering of X is an assignments of k weights in [0,1] to every point in X, so that the i-th weight represents "how much" that point belongs to the i-th cluster. The problem tackled in this work is to recover a hidden fuzzy k-mean clustering of a given set X by querying some similarity oracle that answers queries in the form "how similar are the weight vectors of these two points?". The authors develop an algorithm to approximately recover the hidden clustering (in a well-defined optimization sense) with a number of queries that grows as log |X|, and that depends polynomially on other parameters (including the desired accuracy).  EDIT: I think the authors for their response; after reading it, I will leave my rating unchanged.