Unsupervised Deep Embedding for Clustering Analysis — A Summary
Tags: Clustering, Machine Learning, Deep Learning, Unsupervised Learning, Deep Embedded Clustering

Recently I came across a very interesting idea during my research for a project at work. I am currently dealing with a large dataset of images which has been captured when the light exits a laser. The goal is to understand the data better so that the data can be leveraged to provide insights for servicing these laser devices. So I decided to try unsupervised clustering on images. That is when I came across this paper.

Unsupervised Deep Embedding for Clustering Analysis — Junyuan Xie, Ross Girshick, Ali Farhadi

I really enjoyed reading the paper and decided to make some notes for future reference. I have used some text or images directly from the paper as relevant.

Abstract

The uniqueness of the paper comes from the fact that it focuses on a methodology which simultaneously learns feature representations and clustering assignments using deep neural networks. Unlike the traditional clustering algorithms (which focus on distance functions and grouping algorithms), DEC first the given data space into a lower-dimensional feature space. It then optimizes a clustering objective in this lower dimensional space.

Introduction

Data clustering techniques rely on the ideas such as distance or dissimilarity. However, these concepts are not very effective when it comes to individual pixels in image data. So the authors explore a data driven approach to solve for the feature space and cluster memberships jointly.

To this end they define a parameterized non-linear mapping from the data space X to a lower-dimensional feature space Z, where the clustering objective is optimized. Stochastic Gradient Descent (SGD) via back propagation on a clustering objective has been used to learn the mapping, which is parameterized by a deep neural network.

The authors propose an approach to simultaneously solve for cluster assignment and the underlying feature representation. This is challenging because unlike supervised learning, the deep neural networks cannot be trained with labelled data. So the proposal is to iteratively refine clusters with an auxiliary target distribution derived from the current soft cluster assignment. This process gradually improves the clustering as well as the feature representation. Authors, based on their experiments, additionally note that DEC is significantly less sensitive to the choice of hyperparameters compared to state-of-the-art methods.

The paper’s major contributions are:

Joint optimization of deep embedding and clustering A novel iterative refinement via soft assignment State-of-the-art clustering results in terms of clustering accuracy and speed

Related Work

Clustering methodology has been studied under many names: feature selection, distance functions, grouping methods. I am yet to go through the comprehensive survey that has been referred to in the paper (Aggarwal & Reddy, 2013).

Some of the methods mentioned in the paper are:

1. K-means and Gaussian Mixture Models (GMM):

Very popular

Fast and applicable to a wide range of problems

Distance metrics are limited to the original data space

Tend to be ineffective when input dimensionality is high

2. Variants of k-means to address higher-dimensional input space issues:

Some methods perform joint dimensionality reduction and clustering by first clustering the data with k-means and then projecting the data into a lower dimensions where the inter-cluster variance is maximized (De la Torre & Kanade (2006); Ye et al. (2008)). This process is repeated in EM-style iterations until convergence. This framework is limited to linear embedding. (The benefit DEC method is that it employs dep neural networks to perform non-linear embedding that is necessary for more complex data).

3. Spectral clustering and its variants:

These methods allow more flexible distance metrics and generally perform better than k-means.

Some methods combine spectral clustering and embedding. One of the methods proposes an algorithm based on spectral clustering, but replaces eigen value decomposition with deep autoencoder. This improves performance but further increases memory consumption.

Most spectral clustering algorithms need to compute full graph Laplacian matrix and therefore have quadratic or super quadratic complexities in the number of data points. This means they need specialized machines with large memory for any dataset larger than a few tens of thousands of points. Thus, in order to scale spectral clustering to large datasets, approximate algorithms were invented to trade off performance for speed

However, DEC method is linear in the number of data points and scales gracefully to large datasets

To overcome the drawbacks (such as need for large amount of memory and computational power) of the methods reviewed above, the authors have proposed a method inspired by parametric t-SNE. They define a centroid-based probability distribution and minimize its KL divergence to an auxiliary target distribution to simultaneously improve clustering assignment and feature representation. (This is different from minimizing KL divergence to produce an embedding that is faithful to distances in the original data space.)

Note: The method minimizes the KL divergence between a data distribution and an embedded distribution for data visualization and dimensionality reduction. Parametric variant of t-SNE uses deep neural network to parameterize the embedding.

I found a good explanation of KL Divergence here. In short, with KL divergence, we can calculate exactly how much information is lost when we approximate one distribution with another.

Deep Embedded Clustering

To summarize, the authors propose to first transform the data space X into a latent feature space Z (using a non-linear mapping — DNN) where Z is typically much smaller than X. The DEC therefore clusters the data by simultaneously learning a set of k cluster centers in the feature space Z and the parameters theta of the DNN that maps data points into Z.

The authors propose two phase method:

Phase 1: Parameter initialization with a deep autoencoder Phase 2: Parameter optimization (i.e., clustering with KL divergence)

Thus, in this method, we iterate between:

Computing an auxiliary target distribution Minimizing KL divergence to the computed auxiliary target distribution

Phase 2 has been defined first.

Phase 2: Clustering with KL divergence:

Given:

Initial estimate of non-linear mapping fθ Initial cluster centroids

Phase 2 consists of two steps. These steps are repeated until a convergence criterion is met.

Step 1: SOFT ASSIGNMENT: Compute a soft assignment between the embedded points and the cluster centroids

Student’s t-distribution is used as a kernel to measure the similarity between embedded point and centroid.

Note the definition of qij : probability of assigning sample i to cluster j

Step 2: KL DIVERGENCE MINIMIZATION: Update the deep mapping fθ and refine the cluster centroids by learning from current high confidence assignments using an auxiliary target distribution. Specifically, the model is trained by matching the soft assignment to the target distribution.

The objective is defined as a KL divergence loss between the soft assignments qi and the auxiliary distribution pi as:

Choosing the right target distribution P is very important. It is chosen to have the following properties:

Strengthen predictions (i.e., improve cluster purity) Put more emphasis on data points assigned with high confidence Normalize loss contribution of each centroid to prevent large clusters from distorting the hidden feature space

So, pi is calculated as:

Note: The training strategy can be seen as a form of self-training (Nigam & Ghani, 2000). As in self-training, we take an initial classifier and an unlabeled dataset, then label the dataset with the classifier in order to train on its own high confidence predictions. Indeed, in experiments we observe that DEC improves upon the initial estimate in each iteration by learning from high confidence predictions, which in turn helps to improve low confidence ones.

Optimization:

The method jointly optimizes cluster centers and DNN parameters theta using Stochastic Gradient Descent (SDG) with momentum.

The gradients ∂L/∂zi are then passed down to the DNN and used in standard backpropagation to compute the DNN’s parameter gradient ∂L/∂θ. For the purpose of discovering cluster assignments, the procedure is stopped when less than tol% of points change cluster assignment between two consecutive iterations.

Phase 1: Parameter Initialization

Phase 2 assumed some given initial estimates of DNN parameters (theta) and the cluster centroids {µj}. The initialization of these parameters are discussed here.

DEC initialized with a stacked autoencoder (SAE) because recent research has shown that they consistently produce semantically meaningful and well-separated representations on real-world datasets. Thus the unsupervised representation learned by SAE naturally facilitates the learning of clustering representations with DEC.

The SAE network is initialized layer by layer with each layer being a denoising autoencoder trained to reconstruct the previous layer’s output after random corruption. General network structure is given below:

Note: A denoising autoencoder is a two layer neural network is defined as:

After a greedy layer-wise training, all encoder layers are concatenated followed by all decoder layers, in reverse layer-wise training order, to form a deep autoencoder and then finetune it to minimize reconstruction loss. The final result is a multilayer deep autoencoder with a bottleneck coding layer in the middle. The decoder layers are then discarded and only encoder layers are used as initial mapping between the data space and the feature space.

To initialize the cluster centers, the data is passed through the initialized DNN to get embedded data points and then k-means clustering is performed in the feature space Z to obtain k initial centroids {µj} k j=1.

Overall Approach

I created this diagram summarizing the entire approach.

Experiments

The authors tested this methodology on 3 different datasets and found good results. They used the following datasets:

MNIST — image data STL-10 — image data Reuters — text data

They used the following evaluation metric (ACC — Unsupervised Clustering Accuracy):

Intuitively, this method takes a cluster assignment from an unsupervised algorithm and a ground truth assignment and then finds the best matching between them.

The results across the different datasets have been compiled in the table below:

Implementation Details

Determining hyperparameters by cross-validation on a validation set is not an option in unsupervised clustering. Thus the authors stuck to commonly used parameters for DNNs and avoided dataset specific tuning as much as possible.

Specifically, inspired by van der Maaten (2009), they set network dimensions to d–500–500–2000–10 for all datasets, where d is the data-space dimension, which varies between datasets. All layers are densely (fully) connected.

During greedy layer-wise pretraining the weights to random numbers were drawn from a zero-mean Gaussian distribution with a stddev of 0.01

Each layer was pretrained for 50000 iterations with a dropout rate of 20%

The entire deep autoencoder was further finetuned for 100000 iterations without dropout

For both layer-wise pretraining and end-to-end finetuning of the autoencoder the minibatch size is set to 256, LR = 0.1 which is divided by 10 every 20000 iterations and weight decay = 0

All of the above parameters are set to achieve a reasonably good reconstruction loss and are held constant across all datasets. The authors opine that dataset specific settings might improve the performance on each dataset.

To initialize centroids, they ran k-means with 20 restarts and selected the best solution.

In the KL divergence minimization phase, they trained with a constant learning rate of 0.01. The convergence threshold was set to tol = 0.1%.

Discussion

1.Assumptions and objectives: The underlying assumption of DEC is that the initial classifier’s high confidence predictions are mostly correct

2. Contribution of Autoencoder Initialization: To better understand the contribution of each component, the authors show the performance of all algorithms with autoencoder features in Table 3. You can observe that SEC and LDMGI’s performance do not change significantly with autoencoder feature, while k-means improved but is still below DEC. This demonstrates the power of deep embedding and the benefit of fine-tuning with the proposed KL divergence objective.

3. Performance on imbalanced data: DEC seems to perform fairly well with imbalanced data.

4. Number of clusters: Authors define 2 metrics to determine number of clusters:

The standard metric: NMI — Normalized Mutual Information for evaluating clustering results with different cluster number:

Generalizability — G, which is defined as the ration between training and validation loss. G is small when training loss is lower than validation loss, which indicate a high degree of overfitting.

Conclusion