Meta-k: Towards Unsupervised Prediction of Number of Clusters

1 INTRODUCTION . Clustering is an important task in machine learning , and it has a wide range of applications ( Lung et al . ( 2004 ) ; Aminzadeh & Chatterjee ( 1984 ) ; Gan et al . ( 2007 ) ) . Clustering often consists of two steps : the feature extraction step and the clustering step . There have been numerous works on clustering ( Xu & Tian ( 2015 ) ) , and among the proposed algorithms , K-Means ( Bock ( 2007 ) ) is renowned for its simplicity and performance . Despite its popularity , K-Means has several shortcomings discussed in ( Ortega et al . ( 2009 ) ; Shibao & Keyun ( 2007 ) ) . In particular , with an increase in the dimensionality of the input data , K-Means ’ performance decreases ( Prabhu & Anbazhagan ( 2011 ) ) . This phenomenon is called the curse of dimensionality ( Bellman ( 2015 ) ) . Dimensionality reduction and feature transformation methods have been used to minimize this effect . These methods map the original data into a new feature space , in which the new data-points are easier to be separated and clustered ( Min et al . ( 2018 ) ) . Some examples of existing data transformation methods are : PCA ( Wold et al . ( 1987 ) ) , kernel methods ( Hofmann et al . ( 2008 ) ) and spectral methods ( Ng et al . ( 2002 ) ) . Although these methods are effective , a highly complex latent structure of data can still challenge them ( ( Saul et al. , 2006 ; Min et al. , 2018 ) ) . Due to the recent enhancements in deep neural networks ( Liu et al . ( 2017 ) ) and because of their inherent property of non-linear transformations , these architectures have the potential to replace classical dimensionality reduction methods . In the research field of deep clustering , popularized by the seminal paper ” Unsupervised Deep Embedding for Clustering Analysis ” ( Xie et al . ( 2016 ) ) , deep neural networks are adopted as the feature extractor and are combined with a clustering algorithm to perform the clustering task . A unique loss function is defined which updates the model . Deep clustering methods typically take k , the number of clusters , as a hyper-parameter . In real-world scenarios , where datasets are not labeled , assigning a wrong value to this parameter can reduce the overall accuracy of the model . Meta-learning , a framework that allows a model to use information from its past tasks to learn a new task quickly or with little data , has been adopted by a handful of papers ( Ferrari & de Castro ( 2012 ) ; Ferrari & De Castro ( 2015 ) ; Garg & Kalai ( 2018 ) ; Kim et al . ( 2019 ) ; Jiang & Verma ( 2019 ) ) to improve the performance of clustering tasks . Closest to our work is the approach proposed by Garg & Kalai ( 2018 ) that tries to predict the number of clusters in K-Means clustering using meta-information . To solve the same issue , we propose Meta-k , a gradient-based method for finding the optimal number of clusters and an attempt to have a self-supervised approach for clustering . Our work is based on the observation that a network can take input points and learn parameters to predict the best number of clusters , k. The predicted k , along with the points in the dataset , are the inputs to the clustering algorithm K-Means . For the clusters created , the silhouette score ( Rousseeuw ( 1987 ) ) is calculated . Using this metric value as the reward signal , we can compute the policy gradient to update the controller . As a result , in the next iteration , the controller gives higher probabilities to the k that causes better ( closer to 1 ) silhouette scores to be calculated . To be able to perform optimized clustering on both low- and high-dimensional spaces , we augment our model with a feature extraction module , a deep auto-encoder . In this regard , our work is related to the idea of learning to learn or meta-learning , a general framework to use the information that is learned in one task for improving a future task . Figure 1 shows the diagram of our model for lowand high-dimensional data . We evaluate our method in multiple scenarios on different clustering tasks using synthetic and computer vision datasets , and we show that our approach can predict the number of clusters in most settings with an insignificant error . Our contributions are : • A novel self-supervised approach for predicting the number of clusters using policy gradient methods . • Extensive evaluation on synthetic scikit-learn ( Pedregosa et al . ( 2011 ) ) datasets and wellknown vision datasets MNIST ( LeCun et al . ( 2010 ) ) and Fashion-MNIST ( Xiao et al . ( 2017 ) ) . • Our results show that our approach is able to predict the number of clusters in most scenarios identical or very close to the real number of data clusters . • We plan to release the source code of this work upon its acceptance . 2 RELATED WORK . There is a vast amount of unlabeled data in many scientific fields that can be used for training neural networks . Unsupervised learning makes use of these data , and clustering is one of the most important tasks in unsupervised learning . For unsupervised clustering , the classical K-means algorithm ( Lloyd ( 1982 ) ) has been used extensively due to its simplicity and effectiveness . If the data is distributed compactly around distinctive centroids , then the K-Means algorithm works well , but in real life , such scenarios are rare . Therefore , research has focused on transforming the data into a lowerdimensional space in which K-Means can perform successfully . If our data points are small images , then PCA ( Wold et al . ( 1987 ) ) is commonly used for this transformation . Other methods include non-linear transformation such as kernel methods ( Hofmann et al . ( 2008 ) ) and spectral methods ( Ng et al . ( 2002 ) ) . Deep learning has also been used for unsupervised clustering specifically because it can process high dimensional data effectively by learning embedding spaces ( Schmidhuber ( 2015 ) ) . Deep clustering includes two phases : the feature extraction phase and the clustering phase . Although the extracted features can be fed directly to standard clustering algorithms , deep learning models usually optimize further over specific clustering losses . Xie et al . ( 2016 ) use a loss , based on student t-distribution and can accommodate for soft clustering . They train a stacked denoising auto-encoder as their feature extraction architecture . Caron et al . ( 2018 ) is an unsupervised training method that employs pseudo labels generated by the K-Means algorithm applied to the output of a convolutional network for the task of classification . Even though ( Xie et al. , 2016 ; Caron et al. , 2018 ) achieve promising results , the number of clusters in K-Means and the classifier architecture remain as hyper-parameters needed to be tuned . To optimize the task of clustering , we can leverage the meta-learning framework . According to Vanschoren ( 2018 ) , ” Meta-learning , or learning to learn , is the science of systematically observing how different machine learning approaches perform on a wide range of learning tasks , and then learning from this experience , or meta-data , to learn new tasks much faster than otherwise possible. ” Meta-learning is closely related to one-shot or few-shot learning , and it has shown promising results in supervised learning tasks . When data is limited or quick weight optimization is a must , meta-learning can benefit standard classification and also challenges in the area of Reinforcement Learning . To our knowledge , there are only a few works that focus on using a meta-learning framework for unsupervised learning tasks . Ferrari & de Castro ( 2012 ) ; Ferrari & De Castro ( 2015 ) estimate which of the pre-existing clustering algorithms works well for a new clustering task . Their approach is limited to the algorithms that the user must provide and the number of clusters in K-Means is a hyper-parameter . Garg & Kalai ( 2018 ) Focuses on the theoretical foundations for meta-clustering and uses meta-attributes for learning the best number of clusters . The 399 datasets provided for training a binary similarity function are all labeled , and this means that during training , the number of clusters for each dataset is known . Kim et al . ( 2019 ) proposes a novel meta-learner called MLCNet that mimics numerous clustering tasks during the training to learn an effective embedding space for new clustering tasks . In this regard , their work has a resemblance to metric-based meta-learning algorithms , and learning the best number of clusters is out of their focus . In Jiang & Verma ( 2019 ) , a framework is introduced , which finds the cluster structure directly without having to choose a specific cluster loss for each clustering problem . They propose a stacked-LSTM ( Hochreiter & Schmidhuber ( 1997 ) ) as their architecture , and the number of clusters is a hyper-parameter of this network . 3 META-K. Our proposed method , Meta-K , is an attempt to self-supervise the unsupervised clustering task by automating the process of finding the best number of clusters , k. We adopted the meta-learning framework and the policy gradient algorithm in Meta-K. Our pipeline has two phases , training and inference . During the training phase , given we have high dimensional input x , we train our feature extractor φ ( x ) using reconstruction loss as shown in Equation 1 : L ( x , x̂ ) = ‖x− x̂‖2 ( 1 ) When the auto-encoder is fully-trained , a low dimensional latent representation z is learned by the model . It is shown in previous research that the K-Means algorithm , in particular , performs better when the inputs are of lower dimensions ; therefore , we use this representation as input to the clustering algorithm . The extracted features from the encoder ( z ) or low dimensional input x′ are fed to both the K-Means algorithm ( with a randomly sampled k value from a predefined range with Gaussian distribution ) and the controller network θ ( x′ ) which has the job of predicting k. The controller network consists of multiple fully connected layers with a final layer , with fixed output length that shows the range of the k values predicted by the controller . The length of this vector is denoted by n. We update the parameters of θ ( x′ ) using the policy gradient method for the chosen k value . In a standard reinforcement learning setting , the gradient is considered 1 when a certain action is taken and it is set to 0 for the other actions . In our pipeline , we use the value of silhouette score s ( C ) for the gradients . In other words , the goal of the controller is to find the value of k which gives us the highest silhouette score . Silhouette score is a metric to evaluate the quality of clusters when we do not have access to the labels . Equation 2 shows the calculation of silhouette score . s ( C ) = 1 | ∪ C| ∑ x∈∪C b ( x ) − a ( x ) max { a ( x ) , b ( x ) } ( 2 ) Where b is the mean distance to the points in the nearest cluster , and a is the mean intra-cluster distance to all the points . C is the clustering output from K-Means and x is each data point belonging to an assigned cluster . The output of s ( C ) is in the range of ( −1 , 1 ) with 1 showing the perfect clustering . During the inference phase , the inputs are fed to the controller . The number of clusters is predicted by the network , and K-means clustering is performed using the predicted k value . Figure 1 shows the outline of our method . Over the next sections , we explain our approach in detail .

Summary: The paper uses policy gradients in a bandit setting to learn the optimal number of clusters, k,  in k-means clustering based on the silhouette score. Finding k that leads to the highest silhouette score is a more specific problem that what the paper title promises. The approach is well-described and supported by experiments on simulated and real-world data.