Title: HUMAN-LIKE CLUSTERING WITH DEEP CONVOLU-TIONAL NEURAL NETWORKS

Abstract: Classification and clustering have been studied separately in machine learning and computer vision. Inspired by the recent success of deep learning models in solving various vision problems (e.g., object recognition, semantic segmentation) and the fact that humans serve as the gold standard in assessing clustering algorithms, here, we advocate for a unified treatment of the two problems and suggest that hierarchical frameworks that progressively build complex patterns on top of the simpler ones (e.g., convolutional neural networks) offer a promising solution. We do not dwell much on the learning mechanisms in these frameworks as they are still a matter of debate, with respect to biological constraints. Instead, we emphasize on the compositionality of the real world structures and objects. In particular, we show that CNNs, trained end to end using back propagation with noisy labels, are able to cluster data points belonging to several overlapping shapes, and do so much better than the state of the art algorithms. The main takeaway lesson from our study is that mechanisms of human vision, particularly the hierarchal organization of the visual ventral stream should be taken into account in clustering algorithms (e.g., for learning representations in an unsupervised manner or with minimum supervision) to reach human level clustering performance. This, by no means, suggests that other methods do not hold merits. For example, methods relying on pairwise affinities (e.g., spectral clustering) have been very successful in many cases but still fail in some cases (e.g., overlapping clusters).

1. INTRODUCTION:

Clustering, a.k.a unsupervised classification or nonparametric density estimation, is central to many data-driven domains and has been studied heavily in the past. The task in clustering is to group a given collection of unlabeled patterns into meaningful clusters such that objects within a cluster are more similar to each other than they are to objects in other clusters. Clustering provides a summary representation of data at a coarse level and is used widely in many disciplines (e.g., computer version, bioinformatics, text processing) for exploratory data analysis (a.k.a pattern mining) as well as representation learning (e.g., bag of words). Despite the introduction of thousands of clustering algorithms in the past Aggarwal & Reddy (2013) , some challenges still remain. For instance, existing algorithms fall short in dealing with different cluster shapes, high dimensions, automatically determining the number of clusters or other parameters, large amounts of data, choosing the appropriate similarity measure, incorporating domain knowledge, and cluster evaluation. Further, no clustering algorithm can consistently win over other algorithms, handle all test cases, and perform at the level of humans.

Deep neural networks have become a dominant approach to solve various tasks across many fields. They have been proven successful in several domains including computer vision Krizhevsky et al. (2012) , natural language processing Collobert et al. (2011) , and speech recognition Dahl et al. (2012) for tasks such as scene and object classification Krizhevsky et al. (2012) , pixel-level labeling for image segmentation Long et al. (2015) ; Zheng et al. (2015) , modeling attention Borji & Itti (2013) ; Borji et al. (2013) , image generation Goodfellow et al. (2014) , robot arm control Levine et al. (2015) , speech recognition Graves & Jaitly (2014) , playing Atari games Mnih et al. (2015) and beating the Go champion. Further natural objects are compositional. These facts allow applying the same filters across spatial locations (and hence share weights), and build complex filters from simpler ones to detect highlevel patterns (e.g., object parts, objects). We advocate that these properties are highly appealing when dealing with clustering problems. For instance, the classic two half moons example can be solved by applying a filter that is selective to each half moon. Or, when two clusters with different shapes overlap, the problem can be solved by having filters responding to each shape. Solving these cases is very challenging by just looking at local regions around points and being blind to the high-level patterns. Incorporating domain knowledge, while working in some cases, does not give a general solution for solving all clustering problems. The human visual system easily solves these 2D problems because it is a general system with a rich set of learned or evolved filters. We believe that deep CNNs, although imperfect models of the human vision as they lack feedback and lateral connections carry a huge promise for solving clustering tasks. Further, as we will argue, they offer a unified solution to both classification and clustering tasks.

The current demarcation between classification and clustering becomes murky when we notice that researchers often refer to human judgments in evaluating the outcomes of clustering algorithms. Indeed, humans learn quite a lot about the visual world during their life time. Moreover, the structure of the visual system has been fine-tuned through the evolution. Thus, certainly, there is a learning component involved which has been often neglected in formulating clustering algorithms. While this is sensible from an application point of view (e.g., pattern mining), not only it limits the pursuit for stronger algorithms but also narrows our understanding of human vision.

Learning techniques have been utilized for clustering in the past (e.g., Bach & Jordan (2004) ; Pinheiro et al. (2016) ), for example for tuning parameters (e.g., Bach & Jordan (2004) ). Deep networks have also been exploited for clustering (e.g., Hsu & Kira (2015) ; Hershey et al. (2016) ; Wang et al. (2016) ). However, to our knowledge, while CNNs have been already adopted for image segmentation, so far they have not been exploited for generic clustering. Our goal is to investigate such possibility. To this end, instead of borrowing from clustering to do image segmentation, we follow the opposite direction and propose a deep learning based approach to clustering.

Our method builds on the fully convolutional network literature, in particular, recent work on edge detection and semantic segmentation which utilize multi-scale local and non-local cues Ronneberger et al. (2015) . Thanks to a high volume of labeled data, high capacity of deep networks, powerful optimization algorithms, and high computational power, deep models win on these tasks. We are also strongly inspired by the works showing the high resemblance between human vision mechanisms and CNNs from behavioral, electrophysiological, and computational aspects (e.g., Yamins et al. (2014); DiCarlo & Cox (2007) ; LeCun et al. (1998) ; Krizhevsky et al. (2012) ; Borji & Itti (2014) . Our study enriches our understanding of the concept of clustering and its relation to classification.

2. MODEL DESCRIPTION:

We are motivated by three observations. First, CNNs have been very successful over a variety of vision problems such as semantic segmentation, edge detection, and recognition. Second, CNNs learn representations through several stages of non-linear processing, akin to how the cortex adapts to represent the visual world Yamins et al. (2014) . Similar to other biological models of visual cortex (e.g., HMAX Serre et al. (2007) ), these models capture aspects of the organization of the visual ventral stream. Third, clustering methods are often evaluated against human perception motivating biologically inspired solutions.

Our strategy parallels recent work in using CNNs for semantic segmentation. A crucial difference, however, is that cluster identities (class labels) are not important here. For instance, if a triangle and a circle exist in the image, the shape labels can be anything as long as clusters are correctly separated. Unlike previous work, instead of learning embeddings, we use back propagation via stochastic gradient descent to optimize a clustering objective to learn the mapping, which is parameterized by a deep neural network. In this way, there is no need to specify parameters like number of clusters, distance measure, scale, cluster centers, etc.

2.1. THE NETWORK ARCHITECTURE:

Figure 1 shows the proposed deep network architecture which is based on the U-Net Ronneberger et al. (2015) ; an encoder-decoder with skip connections. The input is a binary image with a single channel, also shown in Figure 2 . Input is fed to five stacks of [convolution, convolutions, pooling] modules. These are followed by five stacks of decoder modules [convolution, convolution, upsampling] . Skip connections from mirrored layers in the encoder are fed to decoder stacks. Such skip connections recover the spatial image information which might have been lost through successive convolution and pooling operations in the encoder. Finally, three 1 × 1 filters are applied to collapse the convolutional maps to three channels that can cluster three objects (one channel per cluster). Each convolution layer in the decoder module has 16 filters and is followed by a ReLU layer except the last convolution layer which uses Sigmoid activation. Then, the pointwise multiplication of the output and the input is calculated to generate the final cluster map (denoted as the output map in Figure 2 ). This multiplication removes the background pixels from the output, giving us only the prediction for points that are of interest.

2.2. TRAINING PROCEDURE AND GROUND TRUTH:

To implement our model, we use the Keras Chollet (2015) platform. We use 128 × 128 binary images as inputs (see Figure 2 ). The corresponding ground truth map for each input is generated as follows: the points belonging to the topmost cluster are assigned label 0, descending top-down in the image, points belonging to the next cluster are assigned label 1, and so on. This process is repeated until all clusters are labeled. This way the labels are independent of the object shapes. This makes our training scheme different from classification and segmentation methods where each object is always assigned the exact same label. Notice that here we are mainly interested in separating the objects from each other rather than correctly classifying them. Despite this, as we will show later, the network is able to cluster objects with the same shape successfully. We use the mean squared error loss and train the network with the Adam optimizer Kingma & Ba (2014) . Batch size is set to 16 and learning rate to 0.001. To generate an image, first a number k ∈ O, indicating the number of objects is randomly drawn. The following is then repeated k times. Random density and scales are chosen for clusters. The cluster is randomly rotated θ degrees (θ ∈ [0, 2π]) and shifted such that it remains within the image boundary. The output is then saved as a 128 × 128 pixels binary image (1 for shape pixels and 0 for background) to be fed to CNN. Ground truth of clusters is saved for training and evaluation. Some sample generated images are shown in Figure 2 . While it seems easy for humans to find the clusters in these images, as will be shown in the next section, our stimuli poses a serious challenge to current best clustering algorithms. In particular, current clustering algorithms fail when clusters occlude each other. The reason is that they lack a global understanding of the patterns.

3. EXPERIMENTAL SETUP:



3.2. EVALUATION METRIC:

Since our purpose is to do clustering, and not classification, prediction score is not sensible to evaluate the performance. Instead, we use rand index which is used to evaluate clustering methods Rand (1971) . The metric is as follows: Given n points in the image, a binary matrix of size n 2 is formed where each element indicates whether two points belong to the same cluster or not. A similar ma-trix is created for the prediction of each model. Notice that the order of points is preserved when constructing these matrices. Then, the Hamming distance between the ground truth matrix and the prediction matrix is calculated which determines the fraction of cases they do not agree with each other (i.e., error rate).

3.3. BENCHMARK ALGORITHMS:

We used the following classic and state of the art clustering methods as the benchmark algorithms: k-Means (KM) Lloyd (1982) , minimizes the sum of squared errors between data points and their nearest cluster centers. It is a simple and widely used method.

Fuzzy C-Means (FCM) Dunn ( 1973) assigns soft labels to data points meaning that each data point can belong to more than one cluster with different degrees of membership.

Spectral Clustering. We employ two spectral clustering algorithms. These algorithms perform a spectral analysis of the matrix of point-to-point similarities instead of estimating an explicit model of data distribution (as in k-Means). The first one is the Normalized Cut (NC) proposed by Shi and Malik Shi & Malik (2000) . Here, first, a graph is built from the image (pixels as nodes, edges as similarities between pixels). Then, algorithm cuts the graph into two subgraphs. The second algorithm, known as Ng-Jordan-Weiss (NJW) Ng et al. ( 2002) is a simple and significant example of spectral clustering which analysis the eigenvectors of the Laplacian of the similarity matrix. Spectral clustering has been shown to work well on data that is connected but not necessarily compact or clustered within convex boundaries.

Mean shift (MS) Comaniciu & Meer (2002) iteratively seeks the modes of a density function from discrete samples of that function. Mean Shift performs as follows. First, it fixes a window around each data point. Then, computes the mean of data within each window. Finally, shifts the window to the mean and repeats till convergence.

Clustering by fast search and find of density peaks (CFSFDP) Rodriguez & Laio (2014) method seeks the modes or peaks of a distribution. It works as follows: 1) For each point, its local density is computed (i.e., number of points in its neighborhood), 2) For each point, its distance to all the points with higher density is computed and the minimum value is sought, 3) A plot is made showing the minimum distance for a point as a function of the density. The outliers in this plot are the cluster centers, 5) Finally, each point is assigned to the same cluster of its nearest neighbor of higher density. The input to this algorithms is the pairwise distance matrix. To find outliers (i.e., cluster centers), we find the point [max x max y ] in this plot and then find q closest points to this point with q being equal to number of ground truth clusters. We use the Gaussian cut off kernel.

All algorithms are provided with the actual number of clusters that exist in the input image, except the CNN and MS algorithm which are supposed to automatically determine the number of clusters in the process of clustering. Euclidean distance is used in both k-Means and FCM as the distance measure. All algorithms are optimized for their best performance in each experiment (e.g., by varying the type of affinity, scale, and normalization). The last three algorithms have been very successful for solving the perceptual grouping problem in computer vision.

3.4. EXPERIMENTS AND RESULTS:

We run a total of nine experiments. The first six experiments aim to evaluate the performance of the proposed method with respect to the benchmark algorithms and contain different set-ups with geometric shape and Gaussian mixture data. The last three experiments test the robustness of the proposed method.

Experiment 1: We generate images with 2 objects randomly chosen from 5 shapes. We use 1800 training and 200 testing images. The goal here is to study the effect of cluster heterogeneity on the results. Results of the experiment 1, the first row of Table 1 , show that CNN is able to successfully cluster the data over the easy cases when two different objects are in the image. Yet, these images challenge the other algorithms.

Experiment 2: We generate 200 test images with 2 objects of the same shape type (one of the 5 shapes). We use the network trained in our 1st experiment. This experiment concentrates on the proposed method's ability to cluster shapes that look very similar to each other. This case is Stimuli: We used the GMM stimuli for the user study. Cases where all three methods performed better than 95% were discarded. Eventually, 300 stimuli were chosen (122 with 2 clusters and 178 with three).

Result: Fig. 6 .B shows the fraction of the cases where subjects chose models (averaged over all subjects). As it shows, subjects favored CNN output more frequently than the other two methods (p=2.52e -06 CNN vs. FCM; p=0.045 CNN vs. CFSFDP; using t-test over subjects; n=14). This also holds over stimuli with two and three clusters. The average accuracies of the three clustering algorithms are shown in Fig. 6 .C. As it can be seen CNN does better than other methods over the 300 stimuli. Results of the user study indicate that our method produces clusters that are more aligned with human clustering judgments. Fraction of the cases where subjects chose models and break downs over clusters (all, 2, or 3), and C) Performance of the models over the stimuli.

We argued that deep neural networks, especially CNNs, hold a great promise for data clustering. We are motivated by the fact that human vision (and learning) is a general system capable of solving both classification and clustering tasks thus blurring the current dichotomy in treating these problems. Our results show that CNNs can successfully handle complex and occluded clusters much better than other algorithms. This means that a learning mechanism, unsupervised or with minimal supervision, seems inevitable in capturing complex cluster shapes.

While our formulation is supervised, feeding the labels to the network is not always consistent. This is where our work differs from semantic segmentation and instance level segmentation. We exploited the mean squared loss to train the network. It might be possible to define other loss functions to teach the network more efficiency using less number of training data or even with weaker labels. One possibility is the pairwise accuracy that we used here for evaluation. Instead of correctly classifying labels, the emphasis can be placed on correctly predicting whether two points belong to the same cluster, regardless of cluster identities (i.e., class labels may vary).

Notice that while here we focused on synthetic stimuli, variations of the proposed CNN architecture, have been successfully applied to natural image segmentation. Thus, CNNs offer a unified solution that can be applied to different data modalities and even to higher dimensional data. Further work is needed to extend this line of work to higher dimensions, and more versatile types of cluster shapes (e.g., free form curves, Gestalt examples, density-based clusters). In this regard, adopting CNNs trained on natural images containing a rich set of intermediate-and high-level patterns can give invaluable insights.

In sum, our results provide encouragement for researchers seeking unified theoretical explanations for supervised and unsupervised categorization but raise a range of challenging theoretical questions.

We emphasized more on the capacity of hierarchical frameworks and compositionality to capture complex structures rather than the learning algorithms. Indeed, further discussion and research are needed for training CNNs in unsupervised or weakly supervised manners. Some new works on learning representations from videos (e.g., for predicting future frames) are particularly interesting. Further, research on unsupervised training of spiking neural networks Masquelier & Thorpe (2007) and CNNs (e.g., using Hebb rule Wadhwa & Madhow (2016) ), along with computational modeling (e.g., Serre et al. (2007) ; LeCun et al. (2015) ), and experimental studies on mechanisms of human vision (e.g., Yamins et al. ( 2014)), will hopefully converge to computational vision algorithms that are capable of solving a variety of tasks, including classification and clustering.

:



Figures and Tables:

Figure 1: U-Net architecture Ronneberger et al. (2015) adopted in this work.

3.1 SYNTHETIC DATA Geometric shape stimuli. Objects are parametrized using several variables including S = {Circle, Ring, Square, SquareRing, Bar}, O = {1, . . . , m}, D = [200 300], and SC = [10 30]. They, in order, denote the set of possible shapes, the set of the possible number of objects to place in the image, the interval of point densities for an object, and the interval of possible object scales.

Figure 2: Examples stimuli. Parameters include number of objects, shape type, point density, object scale, rotation, and location. Parameter ranges are set to increase the chance of overlap among objects. (a) Stimuli generated with shapes. Shapes include circle(filled), ring, square(filled), square ring, and bar. (b) Stimuli generated with Gaussian distributions. Gaussian clusters are shown with different markers for the illustration purpose.

Figure 6: A) A sample trial in the user study, B)

Quantitative comparison of clustering algorithms over shape stimuli. The best one is highlighted in bold. Rows in each experiment show means and standard deviations, respectively. To measure the lower bound, we also test a randomly initialized network that has not seen any training sample. Results are shown in Figure 4. The left and right panels in this figure show models' predictions for shapes and Gaussian clusters respectively. Expectedly, it shows that the model fails without training (i.e., model with its weights randomly

Table content: <table><tr><td>Model</td><td>CNN kM FCM NJW SC</td><td colspan="2">MS CFSFDP</td></tr><tr><td>Exp 1</td><td colspan="2">0.916 0.850 0.858 0.816 0.805 0.748</td><td>0.771</td></tr><tr><td/><td colspan="2">0.131 0.173 0.165 0.220 0.213 0.181</td><td>0.207</td></tr><tr><td>Exp 2</td><td colspan="2">0.895 0.849 0.859 0.813 0.811 0.749</td><td>0.756</td></tr><tr><td/><td colspan="2">0.151 0.176 0.168 0.224 0.217 0.192</td><td>0.212</td></tr><tr><td>Exp 3</td><td colspan="2">0.895 0.770 0.771 0.721 0.760 0.714</td><td>0.728</td></tr><tr><td/><td colspan="2">0.098 0.107 0.103 0.138 0.139 0.116</td><td>0.134</td></tr><tr><td>Exp 4</td><td colspan="2">0.871 0.798 0.804 0.703 0.739 0.714</td><td>0.729</td></tr><tr><td/><td colspan="2">0.117 0.120 0.117 0.190 0.153 0.167</td><td>0.130</td></tr></table>

Peer review (iclr2018):
 OVERALL The paper validates the idea that deep convolutional neural networks could learn to cluster input data better than other clustering methods. This is explained by noting the ability of a deep network to interpret the context of every input point due to a large field of view. However, this explanation is not explored. Also, the clustering method used is supervised and is compared against unsupervised methods. The utility of supervised clustering, as distinct from segmentation, is not explained. Furthermore, the possibility of using weak labels is put forth but not explained and no ideas on how this may be done are proposed. The experimental setup is fairly solid. However the thesis of this work should be better motivated and explained. The range of experiments may also need to be expanded to support this work*s thesis. EXPERIMENTS The use of supervised convolutional neural networks, as opposed to unsupervised networks, should be motivated. It is already clear that autoencoders (and of course convolutional autoencoders) learn semantic clusters at the bottleneck. Also, deep belief networks learn a similar partitioning of the data in the latent domain. These methods are unsupervised and are trained on data -- they should be compared against the proposed method, for clustering. It should also be made clear why clustering with a supervised objective in the fully convolutional network (FCN) is preferable to using unsupervised autoencoders or deep belief networks. It would also be helpful to test soft k-means, alongside the already tested k-means algorithm (this should perform similarly to an autoencoder with a single hidden layer). The improved performance of the FCN over the tested unsupervised clustering methods is put down to the compositional structure of the FCN and the high field of view attained from depth. This should be tested by varying the depth of the network while keeping the same number of parameters. Especially since better performance could also be attributed to this method being the only one which is trained on the dat -- and in a supervised manner, at that. The experiment that evaluates sensitivity to noise could optionally be made more interesting by including a class that models unintersting points (such as noise). The network ould then be trained with a low (but non-zero) amount of noise and then tested on varying amounts of noise, while its ability to classify the noise as noise would be evaluated. Also, while an accuracy-like measure (rand index) is explored while varying noise, there is no evaluation of a precision-like measure -- this would be useful. TEXT AND STYLE The text is fairly well written and could use some polishing to remove typos and a some awkward phrasing. However the introduction needs to be improved to better motivate the work. The conclusion should also be improved as it does not make the significance of the work or the results clear. In tables 1 and 2, the metric used should be noted in their descriptions. In the text, the statistical significance of the results should also be noted. It would be helpful to include an equation in the paper for computing the rand index since it is explained but only with words. The terminology should also be consistent, referring always to rand index rather than to the rand index, accuracy, or pairwise accuracy, interchangeably. The in-text citation format should be fixed as currently, author names are placed in sentences with no linking words or separating punctuation.