Meta-k: Towards Unsupervised Prediction of Number of Clusters

1 INTRODUCTION . Clustering is an important task in machine learning , and it has a wide range of applications ( Lung et al . ( 2004 ) ; Aminzadeh & Chatterjee ( 1984 ) ; Gan et al . ( 2007 ) ) . Clustering often consists of two steps : the feature extraction step and the clustering step . There have been numerous works on clustering ( Xu & Tian ( 2015 ) ) , and among the proposed algorithms , K-Means ( Bock ( 2007 ) ) is renowned for its simplicity and performance . Despite its popularity , K-Means has several shortcomings discussed in ( Ortega et al . ( 2009 ) ; Shibao & Keyun ( 2007 ) ) . In particular , with an increase in the dimensionality of the input data , K-Means ’ performance decreases ( Prabhu & Anbazhagan ( 2011 ) ) . This phenomenon is called the curse of dimensionality ( Bellman ( 2015 ) ) . Dimensionality reduction and feature transformation methods have been used to minimize this effect . These methods map the original data into a new feature space , in which the new data-points are easier to be separated and clustered ( Min et al . ( 2018 ) ) . Some examples of existing data transformation methods are : PCA ( Wold et al . ( 1987 ) ) , kernel methods ( Hofmann et al . ( 2008 ) ) and spectral methods ( Ng et al . ( 2002 ) ) . Although these methods are effective , a highly complex latent structure of data can still challenge them ( ( Saul et al. , 2006 ; Min et al. , 2018 ) ) . Due to the recent enhancements in deep neural networks ( Liu et al . ( 2017 ) ) and because of their inherent property of non-linear transformations , these architectures have the potential to replace classical dimensionality reduction methods . In the research field of deep clustering , popularized by the seminal paper ” Unsupervised Deep Embedding for Clustering Analysis ” ( Xie et al . ( 2016 ) ) , deep neural networks are adopted as the feature extractor and are combined with a clustering algorithm to perform the clustering task . A unique loss function is defined which updates the model . Deep clustering methods typically take k , the number of clusters , as a hyper-parameter . In real-world scenarios , where datasets are not labeled , assigning a wrong value to this parameter can reduce the overall accuracy of the model . Meta-learning , a framework that allows a model to use information from its past tasks to learn a new task quickly or with little data , has been adopted by a handful of papers ( Ferrari & de Castro ( 2012 ) ; Ferrari & De Castro ( 2015 ) ; Garg & Kalai ( 2018 ) ; Kim et al . ( 2019 ) ; Jiang & Verma ( 2019 ) ) to improve the performance of clustering tasks . Closest to our work is the approach proposed by Garg & Kalai ( 2018 ) that tries to predict the number of clusters in K-Means clustering using meta-information . To solve the same issue , we propose Meta-k , a gradient-based method for finding the optimal number of clusters and an attempt to have a self-supervised approach for clustering . Our work is based on the observation that a network can take input points and learn parameters to predict the best number of clusters , k. The predicted k , along with the points in the dataset , are the inputs to the clustering algorithm K-Means . For the clusters created , the silhouette score ( Rousseeuw ( 1987 ) ) is calculated . Using this metric value as the reward signal , we can compute the policy gradient to update the controller . As a result , in the next iteration , the controller gives higher probabilities to the k that causes better ( closer to 1 ) silhouette scores to be calculated . To be able to perform optimized clustering on both low- and high-dimensional spaces , we augment our model with a feature extraction module , a deep auto-encoder . In this regard , our work is related to the idea of learning to learn or meta-learning , a general framework to use the information that is learned in one task for improving a future task . Figure 1 shows the diagram of our model for lowand high-dimensional data . We evaluate our method in multiple scenarios on different clustering tasks using synthetic and computer vision datasets , and we show that our approach can predict the number of clusters in most settings with an insignificant error . Our contributions are : • A novel self-supervised approach for predicting the number of clusters using policy gradient methods . • Extensive evaluation on synthetic scikit-learn ( Pedregosa et al . ( 2011 ) ) datasets and wellknown vision datasets MNIST ( LeCun et al . ( 2010 ) ) and Fashion-MNIST ( Xiao et al . ( 2017 ) ) . • Our results show that our approach is able to predict the number of clusters in most scenarios identical or very close to the real number of data clusters . • We plan to release the source code of this work upon its acceptance . 2 RELATED WORK . There is a vast amount of unlabeled data in many scientific fields that can be used for training neural networks . Unsupervised learning makes use of these data , and clustering is one of the most important tasks in unsupervised learning . For unsupervised clustering , the classical K-means algorithm ( Lloyd ( 1982 ) ) has been used extensively due to its simplicity and effectiveness . If the data is distributed compactly around distinctive centroids , then the K-Means algorithm works well , but in real life , such scenarios are rare . Therefore , research has focused on transforming the data into a lowerdimensional space in which K-Means can perform successfully . If our data points are small images , then PCA ( Wold et al . ( 1987 ) ) is commonly used for this transformation . Other methods include non-linear transformation such as kernel methods ( Hofmann et al . ( 2008 ) ) and spectral methods ( Ng et al . ( 2002 ) ) . Deep learning has also been used for unsupervised clustering specifically because it can process high dimensional data effectively by learning embedding spaces ( Schmidhuber ( 2015 ) ) . Deep clustering includes two phases : the feature extraction phase and the clustering phase . Although the extracted features can be fed directly to standard clustering algorithms , deep learning models usually optimize further over specific clustering losses . Xie et al . ( 2016 ) use a loss , based on student t-distribution and can accommodate for soft clustering . They train a stacked denoising auto-encoder as their feature extraction architecture . Caron et al . ( 2018 ) is an unsupervised training method that employs pseudo labels generated by the K-Means algorithm applied to the output of a convolutional network for the task of classification . Even though ( Xie et al. , 2016 ; Caron et al. , 2018 ) achieve promising results , the number of clusters in K-Means and the classifier architecture remain as hyper-parameters needed to be tuned . To optimize the task of clustering , we can leverage the meta-learning framework . According to Vanschoren ( 2018 ) , ” Meta-learning , or learning to learn , is the science of systematically observing how different machine learning approaches perform on a wide range of learning tasks , and then learning from this experience , or meta-data , to learn new tasks much faster than otherwise possible. ” Meta-learning is closely related to one-shot or few-shot learning , and it has shown promising results in supervised learning tasks . When data is limited or quick weight optimization is a must , meta-learning can benefit standard classification and also challenges in the area of Reinforcement Learning . To our knowledge , there are only a few works that focus on using a meta-learning framework for unsupervised learning tasks . Ferrari & de Castro ( 2012 ) ; Ferrari & De Castro ( 2015 ) estimate which of the pre-existing clustering algorithms works well for a new clustering task . Their approach is limited to the algorithms that the user must provide and the number of clusters in K-Means is a hyper-parameter . Garg & Kalai ( 2018 ) Focuses on the theoretical foundations for meta-clustering and uses meta-attributes for learning the best number of clusters . The 399 datasets provided for training a binary similarity function are all labeled , and this means that during training , the number of clusters for each dataset is known . Kim et al . ( 2019 ) proposes a novel meta-learner called MLCNet that mimics numerous clustering tasks during the training to learn an effective embedding space for new clustering tasks . In this regard , their work has a resemblance to metric-based meta-learning algorithms , and learning the best number of clusters is out of their focus . In Jiang & Verma ( 2019 ) , a framework is introduced , which finds the cluster structure directly without having to choose a specific cluster loss for each clustering problem . They propose a stacked-LSTM ( Hochreiter & Schmidhuber ( 1997 ) ) as their architecture , and the number of clusters is a hyper-parameter of this network . 3 META-K. Our proposed method , Meta-K , is an attempt to self-supervise the unsupervised clustering task by automating the process of finding the best number of clusters , k. We adopted the meta-learning framework and the policy gradient algorithm in Meta-K. Our pipeline has two phases , training and inference . During the training phase , given we have high dimensional input x , we train our feature extractor φ ( x ) using reconstruction loss as shown in Equation 1 : L ( x , x̂ ) = ‖x− x̂‖2 ( 1 ) When the auto-encoder is fully-trained , a low dimensional latent representation z is learned by the model . It is shown in previous research that the K-Means algorithm , in particular , performs better when the inputs are of lower dimensions ; therefore , we use this representation as input to the clustering algorithm . The extracted features from the encoder ( z ) or low dimensional input x′ are fed to both the K-Means algorithm ( with a randomly sampled k value from a predefined range with Gaussian distribution ) and the controller network θ ( x′ ) which has the job of predicting k. The controller network consists of multiple fully connected layers with a final layer , with fixed output length that shows the range of the k values predicted by the controller . The length of this vector is denoted by n. We update the parameters of θ ( x′ ) using the policy gradient method for the chosen k value . In a standard reinforcement learning setting , the gradient is considered 1 when a certain action is taken and it is set to 0 for the other actions . In our pipeline , we use the value of silhouette score s ( C ) for the gradients . In other words , the goal of the controller is to find the value of k which gives us the highest silhouette score . Silhouette score is a metric to evaluate the quality of clusters when we do not have access to the labels . Equation 2 shows the calculation of silhouette score . s ( C ) = 1 | ∪ C| ∑ x∈∪C b ( x ) − a ( x ) max { a ( x ) , b ( x ) } ( 2 ) Where b is the mean distance to the points in the nearest cluster , and a is the mean intra-cluster distance to all the points . C is the clustering output from K-Means and x is each data point belonging to an assigned cluster . The output of s ( C ) is in the range of ( −1 , 1 ) with 1 showing the perfect clustering . During the inference phase , the inputs are fed to the controller . The number of clusters is predicted by the network , and K-means clustering is performed using the predicted k value . Figure 1 shows the outline of our method . Over the next sections , we explain our approach in detail .

Summary: The reviewed paper presents a completely unsupervised framework Meta-K for predicting the number of clusters. The approach advocated in the paper comprises two main parts: autoencoder for feature extraction and multilayer perceptron (MLP) for predicting the number of clusters. Autoencoder is used if necessary to decrease the dimensionality of the input data. The MLP is trained using policy gradient optimization schema to predict the best (according to silhouette score) number of clusters k in the given dataset. Overall, the authors show that their approach achieves near-optimum results on both a number of synthetic datasets as well as on two well-known computer vision datasets: MNIST and FMNIST.